Multistage image colorization network system and method based on lightweight attention structure

By adopting a multi-level feature extraction and fusion method of lightweight attention structure in the image colorization network, combined with the classification loss function, the color brown, unsaturation and overflow problems of the image colorization algorithm in the prior art is solved, and the color is improved The accuracy and coherence of color images optimizes the training difficulty, making the generated color images more vivid and realistic.

CN119006655APending Publication Date: 2024-11-22CHANGCHUN INST OF OPTICS FINE MECHANICS & PHYSICS CHINESE ACAD OF SCI
View PDF 0 Cites 3 Cited by

Patent Information

Application Number
CN202411205504.7
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-30
Publication Date
2024-11-22

AI Technical Summary

Technical Problem

The existing image colorization algorithm has problems such as brownish color, unsaturation, inability to accurately identify objects and scene categories, color overflow and incoherence, and the training is difficult, resulting in poor results of the generated color images.

Method used

Using a multi-level image colored network system based on lightweight attention structure, multiple features of different sizes are acquired through feature extraction branches, and feature fusion and color matching are performed in the color processing branches, and the model is trained in combination with the classification loss function. to generate more vivid and coherent color images.

Benefits of technology

It significantly improves the accuracy and coherence of image colorization, reduces the phenomenon of color overflow, and optimizes the difficulty of network training, so that the generated color images are more vivid and realistic.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119006655A_ABST
    Figure CN119006655A_ABST
Patent Text Reader

Abstract

The invention relates to the technical field of image processing, in particular to a multilevel image colorization network system and method based on a lightweight attention structure, and the system comprises a feature extraction branch and a color processing branch. Wherein an input grayscale image enters the feature extraction branch to obtain a plurality of features with different sizes; a plurality of features with different sizes are input into a color processing branch, firstly, multi-scale input features are fused, the phenomenon of color overflow can be greatly inhibited, and then color matching is performed on the fused features to obtain color features corresponding to a grayscale image; and finally, splicing the grayscale image and the color features to obtain a color image corresponding to the grayscale image. The feature extraction branch is used for densely connecting multi-layer space attention and channel attention information while reducing the calculation amount, so that the coloring accuracy and coherence are remarkably improved; in the method, a classification loss function is adopted to train an image colorization model to obtain a brighter coloring effect.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of image processing technology, and in particular relates to a multi-level image colorization network system and method based on a lightweight attention structure. Background Art

[0002] Image colorization refers to the process of adding appropriate color channels to a single-channel gray image with only brightness to turn it into a multi-channel color image. Compared with grayscale images, color images not only have more detailed features but also are more in line with human visual perception. Although image colorization methods have been greatly improved in recent years and have been widely used in many advanced fields. However, due to the uncertainty and diversity of image colorization, image colorization algorithms are still a challenging topic.

[0003] In recent years, with the continuous development of deep learning and computer vision, a large number of automatic colorization methods based on neural networks have been proposed. The automatic colorization method only needs to input the image to be colored into the image colorization network with the optimal model to quickly obtain the corresponding color image. Cheng et al. first proposed an image colorization algorithm based on convolutional neural networks (CNN) in the paper "Deep Colorization" and used bilateral filtering to improve the colorization effect. On this basis, Larsson et al. analyzed the multimodal color distribution of pixels in the training set image in the paper "Learning Representations for Automatic Colorization" and output the color distribution histogram of pixels in the image to be colored to complete the colorization. Isola et al.'s published paper "Image-to-Image Translation with Conditional Adversarial Networks" and Nazeri et al.'s published paper "Image Colorization using Generative Adversarial Networks" proposed the use of generative adversarial networks (GAN) for image colorization tasks, which improved the colorization effect to a certain extent. In the paper "Colorful Image Colorization", Zhang et al. used classification loss function to train the colorization network according to the manually divided color categories, and obtained a bright and rich colorization effect. In the paper "Let there be Color!: Joint End-to-end Learning of Global and Local Image Priors for Automatic Image Colorization with Simultaneous Classification", Iizuka et al. proposed a dual-channel CNN structure that integrates global features and local features and added image category branches to improve the accuracy of colorization. In the paper "ChromaGAN: Adversarial Picture Colorization with Semantic Class Distribution", Vitoria et al. introduced an image classification subnetwork and used the image category loss function to guide the image colorization network colorization.In the paper "Focusing on Persons: Colorizing Old Images Learning from Modern Historical Movies", Jin et al. proposed to add a semantic segmentation subnetwork based on Vitoria et al. to realize the colorization of black and white military images. In the paper "Instance-aware Image Colorization", Su et al. cropped the objects in the image to be colored and colored them separately, and finally used the weighted fusion of each colored object and the overall image. In the paper "COLORIZATION TRANSFORMER", Kumar et al. proposed to design a Colorization Transformer to color high-resolution images. The algorithm achieves excellent colorization effects for high-resolution images. In the paper "ColorFormer: Image Colorization via Color Memory assisted Hybrid-attention Transformer", Ji et al. disclosed the fusion of global and local features of grayscale images in the encoder, and guided the colorization in the decoder according to the stored semantics and color mapping relationship to generate more realistic color images.

[0004] Although the above-mentioned automatic colorization image colorization methods have been able to achieve satisfactory colorization effects, these image colorization methods still have some defects that have not been improved. First, most of these algorithms use regression loss functions, which causes the color of the generated images to be brown and unsaturated. Secondly, when the image colorization method cannot accurately identify the categories of objects and scenes in the image to be colored, the image colorization method cannot give accurate colors to the objects and scenes. Secondly, when the image colorization algorithm does not accurately grasp the overall information of the image to be colored and does not identify the boundary information of each object, it will cause color overflow and incoherence in the generated image. Finally, most of these algorithms use complex network structures, which aggravates the training difficulty of the automatic colorization colorization method and greatly prolongs the network training time. Summary of the invention

[0005] In view of this, the present invention aims to provide a multi-level image colorization network system and method based on a lightweight attention structure, in which multiple features of different sizes are obtained in the feature extraction branch, which significantly improves the accuracy and consistency of coloring; multiple features of different sizes are input into the color processing branch for feature fusion and color matching to obtain color features, which can greatly suppress the phenomenon of color overflow; finally, the grayscale image and the color features are spliced ​​to obtain a color image corresponding to the grayscale image; the method uses a classification loss function to train the image colorization model to obtain a more vivid coloring effect.

[0006] To achieve the above object, the technical solution created by the present invention is implemented as follows: A multi-level image colorization network system includes a feature extraction branch and a color processing branch; wherein the feature extraction branch performs N-level feature extraction on an input grayscale image to obtain N features of different sizes, where N is not less than 3; the N features of different sizes are input into the color processing branch for feature fusion and color matching to obtain color features corresponding to the grayscale image; finally, the grayscale image and the color features are spliced ​​to obtain a color image corresponding to the grayscale image.

[0007] Furthermore, in the feature extraction branch, the grayscale image is first input into the feature extraction module for processing to obtain the first-level features; the previous-level features are downsampled and pooled, and then the pooled features are input into the feature extraction module for processing to obtain the current-level features; the above operation is repeated N-1 times to obtain N-1 features of different sizes.

[0008] Furthermore, the color processing branch includes a LAATF module, a feature extraction module and a CR module; among them, N features of different sizes are input into the LAATF module for feature fusion; the fused features are first subjected to a convolution operation of a 1×1 convolution layer and then input into the feature extraction module for feature extraction; the features output by the feature extraction module are then input into the CR module for color matching to obtain color features.

[0009] Furthermore, in the LAATF module, N features of different sizes are input into their corresponding global pooling layers to obtain corresponding pooling features, and the corresponding pooling features are input into their corresponding fully connected layers to obtain corresponding fully connected features; the N fully connected features are spliced, and the spliced ​​features are sequentially input into the ReLU activation function layer, the 1×1 convolution layer, and the Sigmoid activation function layer to obtain weight features; the N features of different sizes are respectively element-wise multiplied with the weight features, and then the N multiplied features are scaled to obtain N features of the same size; finally, the N features of the same size are spliced ​​and the spliced ​​features are output.

[0010] Furthermore, the feature extraction module includes a CADC submodule and a LIRA submodule; wherein: the input feature is first passed through the CADC submodule to obtain the CADC feature; and the CADC feature is then input into the LIRA submodule to obtain the extracted feature corresponding to the input feature.

[0011] Furthermore, in the CADC submodule, the separable convolution layer performs a convolution operation on the input features, and the features obtained after the convolution operation are then subjected to a lightweight channel attention operation to obtain the CADC features.

[0012] Furthermore, in the LIRA sub-module, the CADC features are first subjected to lightweight spatial attention operations and lightweight channel attention operations in sequence to obtain primary LIRA features; the primary LIRA features are then subjected to lightweight spatial attention operations and lightweight channel attention operations in sequence to obtain secondary LIRA features; the CADC features, primary LIRA features and secondary LIRA features are added together to obtain the extracted features.

[0013] A multi-level image colorization method, applicable to the image colorization network system based on a lightweight attention structure proposed in the present invention, specifically comprises the following steps: S1: Obtain pairs of grayscale images to be processed and reference color images from the dataset, and establish training sets and validation sets; S2: Constructing the image colorization network system based on the lightweight attention structure provided by the present invention; S3: input the training set and the validation set obtained in step S1 into the image colorization network system obtained in step S2, and perform training according to the classification loss function to obtain an image colorization network model; S4: Input the test grayscale image into the trained image colorization network model to obtain a predicted color image corresponding to the test grayscale image.

[0014] Furthermore, the classification loss function used in the training in step S3 is:

[0015] in, represents the classification loss function, represents the reference color image in the training set, It indicates that the image colorization network model predicts the corresponding predicted color image based on the grayscale image to be processed in the training set; represents the color category of the reference color image, represents the color balance weight of the reference color image, represents the color category probability distribution in the reference color image, Represents the predicted color category probability distribution in a color image; h, w, and q are the width, height, and color category of the pixel, respectively.

[0016] Furthermore, the color balance weight of the reference color image is obtained by the following formula: :

[0017]

[0018] in, Represents the color category probability distribution of all pixels in the image in the training set and the inverse of the average distribution of all classes, Represents the color category probability distribution of all pixels in the image in the training set and the average distribution of all categories are weighted values ​​of the two distributions, Represents the total number of color categories of all pixels in the data set in step S1; Represents the color category probability distribution of all pixels in the data set, Indicates the percentage of pixels of color category q in the total pixels. Represents probability distribution The probability of being in color category q.

[0019] Compared with the prior art, the invention can achieve the following beneficial effects: (1) In the multi-level image colorization network system created by the present invention, the feature extraction branch obtains multiple features of different sizes in the form of multi-level feature extraction, and inputs the multiple features of different sizes into the color processing branch for feature fusion and color matching to obtain color features, which can greatly suppress the color overflow phenomenon; finally, the grayscale image and the color features are spliced ​​to obtain a color image corresponding to the grayscale image; (2) In the multi-level image colorization network system created by the present invention: the CADC module greatly reduces the parameters of the model and optimizes network training; the LIRA module densely connects multiple layers of spatial attention and channel attention information while reducing the amount of calculation, significantly improving the accuracy and coherence of colorization; the LAAF module integrates multi-scale input features while reducing the amount of calculation, greatly suppressing the phenomenon of color overflow. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The drawings constituting part of the present invention are used to provide a further understanding of the present invention. The exemplary embodiments and descriptions of the present invention are used to explain the present invention and do not constitute an improper limitation on the present invention. In the drawings: Figure 1 A schematic diagram of the structure of the multi-level image colorization network system according to an embodiment of the present invention; Figure 2 A schematic diagram of the structure of the CADC submodule described in the embodiment of the present invention; Figure 3 A schematic diagram of the structure of the LIRA submodule according to an embodiment of the present invention; Figure 4 A schematic diagram of the structure of the LAAF module described in the embodiment of the present invention; Figure 5A schematic diagram of a process flow of a multi-level image colorization method according to an embodiment of the present invention; Figure 6 A comparison chart of the image colorization effects of the multi-level image colorization method described in the embodiment of the present invention and other methods. DETAILED DESCRIPTION

[0021] In order to make the purpose, technical solution and advantages of the invention more clear, the invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described here are only used to explain the invention and do not constitute a limitation of the invention.

[0022] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other.

[0023] In the description of the present invention, it should be understood that the terms "center", "longitudinal", "lateral", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside" and the like indicate positions or positional relationships based on the positions or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention. In addition, the terms "first", "second", etc. are only used for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, features defined as "first", "second", etc. may explicitly or implicitly include one or more of the features. In the description of the present invention, unless otherwise specified, "multiple" means two or more.

[0024] In the description of the invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installation", "connection" and "connection" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the invention can be understood according to specific circumstances.

[0025] The present invention will be described in detail below with reference to the accompanying drawings and in combination with embodiments.

[0026] The multi-level image colorization network system described in the embodiment of the invention is calculated in the Lab color space, where L, a, and b represent the brightness of the Lab color space, the range from red to green, and the range from yellow to blue. The multi-level image colorization network system described in the embodiment of the invention is to convert a grayscale image Predict the remaining a and b color value channels in its Lab channel , convert the single channel grayscale image With color characteristics Stitching to get three-channel color image . Where h and w are the height and width of the image respectively.

[0027] like Figure 1 As shown, the multi-level image colorization network system described in the embodiment of the present invention includes a feature extraction branch and a color processing branch. The feature extraction branch processes the input grayscale image. Perform N-level feature extraction to obtain N features of different sizes, where N is not less than 3. N features of different sizes are input into the color processing branch for feature fusion and color matching to obtain the color features corresponding to the grayscale image. ; Finally, the grayscale image With color features Splice to get the color image corresponding to the grayscale image .

[0028] In the embodiment of the present invention, N is 4, that is, the feature extraction branch in the embodiment of the present invention is a grayscale image input Perform 4-level feature extraction to obtain 4 features of different sizes. The 4 features of different sizes are input into the color processing branch for feature fusion and color matching to obtain the color features corresponding to the grayscale image. Finally, the grayscale image is concatenated with the color features to obtain the color image corresponding to the grayscale image. .

[0029] Specifically, in the feature extraction branch, the grayscale image is first The input feature is processed in the feature extraction module to obtain the first-level feature; the previous-level feature is downsampled and pooled, and the pooled feature is input into the feature extraction module for processing to obtain the current-level feature; the above operation is repeated N-1 times to obtain N-1 features of different sizes.

[0030] In the embodiment of the present invention, N is 4, and each downsampling pooling adopts a pooling step of 2. Therefore, in the feature extraction branch of the embodiment of the present invention, the grayscale image is firstly The input feature extraction module is processed to obtain the first-level feature (the number of channels is 64 and the size is h×w); the first-level feature is downsampled and pooled with a step size of 2, and the pooled feature is input into the feature extraction module for processing to obtain the second-level feature (the number of channels is 128 and the size is h / 2×w / 2); the second-level feature is downsampled and pooled with a step size of 2, and the pooled feature is input into the feature extraction module for processing to obtain the third-level feature with 256 channels and a size of h / 4×w / 4); finally, the third-level feature is downsampled and pooled with a step size of 2, and the pooled feature is input into the feature extraction module for processing to obtain the fourth-level feature (the number of channels is 512 and the size is h / 8×w / 8).

[0031] Among them, the feature extraction module in the feature extraction branch includes CADC (Channel Attention Depthwise Convolutional, channel attention depth separable convolutional layer) submodule and LIRA (Lightweight Intensive Residual Attention, lightweight dense residual attention) submodule. Among them, the input feature (the number of channels is Cin, the size is H×W, H represents the length of the feature, and W represents the width of the feature) is first obtained by the CADC submodule to obtain the CADC feature (the number of channels is Cout, the size is H×W), and the CADC feature is then input into the LIRA submodule to obtain the extracted feature corresponding to the input feature.

[0032] like Figure 2 As shown, in the CADC submodule, the separable convolution layer (Depthwise Convolution, DW) performs a convolution operation on the input features. Among them, the parameter amount and calculation amount of the separable convolution layer are only 1 / Cout of the two-dimensional convolution kernel. Although the separable convolution layer greatly reduces the training difficulty and calculation time, the structure does not consider the connection between the channels of the input features. Therefore, the lightweight channel attention (LCA) operation is used to process the features obtained after the convolution of the separable convolution layer. The lightweight channel attention operation can obtain the importance of each channel in the feature, so that the image colorization model pays more attention to the input features that are helpful for image colorization when extracting features, effectively improving the model performance. Finally, after the features (the number of channels is Cin and the size is H×W) obtained after the lightweight channel attention operation is processed by the convolution operation of the 1×1 convolution layer, the number of channels of the features changes, and the CADC features are obtained. In an embodiment of the present invention, the global pooling layer in the lightweight channel attention operation is a maximum pooling layer.

[0033] The specific process of the lightweight channel attention operation is that the input features (the number of channels is Cin, the size is H×W, H represents the length of the feature, and W represents the width of the feature) are first input into the global pooling layer to obtain the pooled features (the number of channels is Cin, the size is 1×1). The pooled features are sequentially passed through the 1×1 convolution layer, the ReLU activation function layer, the 1×1 convolution layer, and the Sigmoid activation function layer to obtain the channel weights (the number of channels is Cin, the size is 1×1). The channel weights are multiplied by the corresponding elements of the input features to obtain the features processed by the lightweight channel attention operation (the number of channels is Cin, the size is H×W).

[0034] In the traditional convolution module, the number of parameters of the two-dimensional convolution kernel is (k represents the size of the convolution kernel), the amount of calculation is , while the parameter quantity of CADCM provided by the embodiment of the present invention is only , the computational effort is only It can be seen that the CADC submodule proposed in the embodiment of the present invention can effectively reduce the parameters and calculation amount of the model.

[0035] like Figure 3 As shown, in the LIRA submodule (such as Figure 3 (a) The input CADC features are first subjected to the Lightweight Spatial Attention (LSA) and Lightweight Channel Attention operations to obtain the primary LIRA features. The primary LIRA features are then subjected to the Lightweight Spatial Attention and Lightweight Channel Attention operations to obtain the secondary LIRA features. The CADC features, primary LIRA features, and secondary LIRA features are added together to obtain the extracted features.

[0036] The lightweight channel attention operation in the LIRA submodule (such as Figure 3 (b) is consistent with the lightweight channel attention operation in the CADC submodule and will not be repeated here. The lightweight spatial attention operation in the LIRA submodule (e.g. Figure 3 The specific process of (c) is as follows: The input features (number of channels is C, size is H×W) are first input into the channel maximum pooling layer to obtain the pooled features (number of channels is 1, size is H×W). The pooled features are sequentially passed through the 1×1 convolution layer, the ReLU activation function layer, the 1×1 convolution layer, and the Sigmoid activation function layer to obtain the spatial weights (number of channels is 1, size is H×W). The spatial weights are multiplied by the corresponding elements of the input features to obtain the features processed by the lightweight spatial attention operation (number of channels is C, size is H×W).

[0037] The LIRA submodule combines spatial attention and channel attention. Spatial attention takes into account the importance of each pixel of the input feature, and combined with the attention weights of each channel obtained by channel attention, it can extract effective information of the input feature more comprehensively and accurately. In addition, the LIRA submodule connects the inferred attention weights of each layer to the subsequent attention weights through a residual structure, which not only maintains the feedforward nature of the model, but also greatly optimizes network training. The dense residual attention fusion module enables the image colorization model to extract objects in the image more comprehensively and accurately, significantly improving the accuracy and coherence of coloring. The parameter amount of the LIRA submodule provided in an embodiment of the present invention is , the calculation amount is .

[0038] The color processing branch includes a LAATF (Lightweight Asymmetric Attention Fusion Module), a feature extraction module and a CR (Color Recovery) module. Features of N different sizes are input into the LAATF module for feature fusion. In an embodiment of the present invention, primary features, secondary features, tertiary features and quaternary features are input into the LAATF module for feature fusion. The fused features are first subjected to a convolution operation of a 1×1 convolutional layer and then input into the feature extraction module for feature extraction. The features output by the feature extraction module are then input into the CR module for color matching to obtain color features. In an embodiment of the present invention, the structure of the feature extraction module in the color processing branch is consistent with that of the feature extraction module in the feature extraction branch.

[0039] like Figure 4 As shown in the figure, in the LAATF module, N features of different sizes are input into their respective global pooling layers to obtain corresponding pooling features, and the corresponding pooling features are input into their respective fully connected layers to obtain corresponding fully connected features. The N fully connected features are concatenated, and the concatenated features are sequentially input into the ReLU activation function layer, the 1×1 convolution layer, and the Sigmoid activation function layer to obtain weight features. After the N features of different sizes are element-wise multiplied with the weight features, the N multiplied features are scaled to obtain N features of the same size. Finally, the N features of the same size are concatenated and the concatenated features are output.

[0040] In the embodiment of the present invention, the first-level feature, the second-level feature, the third-level feature and the fourth-level feature are input into the corresponding global maximum pooling layer to obtain the corresponding pooling feature. At this time, the number of channels of the pooling feature corresponding to the first-level feature is 64, and the size is 1×1. The number of channels of the pooling feature corresponding to the second-level feature is 128, and the size is 1×1. The number of channels of the pooling feature corresponding to the third-level feature is 256, and the size is 1×1. The number of channels of the pooling feature corresponding to the fourth-level feature is 512, and the size is 1×1. The four pooling features are input into the corresponding fully connected layers to obtain the corresponding fully connected features. The number of channels of each fully connected feature is 1, and the size is 1×1. The four fully connected features are spliced, and the spliced ​​features are sequentially input into the ReLU activation function layer, the 1×1 convolution layer and the Sigmoid activation function layer to obtain the weighted features with 4 channels and a size of 1×1. After the first-level features, second-level features, third-level features, and fourth-level features are respectively element-wise multiplied with the weight features, the four multiplied features are scaled to obtain four features of size h / 4×w / 4. Finally, the four features of the same size are spliced ​​and the spliced ​​features are output. The number of channels of the feature is 960, and the size is h / 4×w / 4. In an embodiment of the present invention, the four features are scaled by corresponding upsampling and downsampling operations. Specifically, the features smaller than h / 4×w / 4 are upsampled, and the features larger than h / 4×w / 4 are downsampled, so that the four features are unified to the size of h / 4×w / 4 after scaling.

[0041] In the LAATF module, the attention weights of each input feature are first learned using channel attention, and then the input features are scaled to the required resolution. Finally, the fusion of each input feature is guided according to the attention weights of the input features and the corresponding features are spliced ​​by channel. The LAATF module makes up for the feature differences between high-scale and low-scale features, allowing the model to obtain more accurate fusion features and effectively improve the ability of the image colorization algorithm to suppress color overflow.

[0042] In the CR module, the color category probability distribution in the predicted color image generated by the image colorization network system provided by the embodiment of the present invention is Select the color category of each pixel ,Right now:

[0043] in, Represents the probability distribution of color categories The color category with the highest probability, color category probability distribution Related to the data set for training the image colorization network system provided by the embodiment of the present invention. T is the annealing parameter, which is 0.38 here. Next, use the inverse color category matrix Get the matrix ,matrix Corresponding to channel a and channel b, that is:

[0044] Among them, the inverse color category matrix Related to the data set for training the image colorization network system provided by the embodiment of the present invention. Finally, upsample and amplify it to obtain the final color feature ,Right now:

[0045] in, represents n-fold upsampling. Since the feature size of the input CR module is h / 4×w / 4, n is set to 4 here, that is, the matrix Perform 4 times upsampling to make the color features The size and grayscale image The same size.

[0046] The embodiment of the present invention also provides a multi-level image colorization method, which is applicable to the multi-level image colorization network system proposed by the present invention, such as Figure 5 As shown, the specific steps include: S1: Obtain pairs of grayscale images to be processed and reference color images from the dataset, and establish training sets and validation sets.

[0047] In an embodiment of the present invention, a training set and a validation set are obtained from the ImageNet data set, specifically, 1,281,167 color images are obtained from the ImageNet training set, and these color images are grayed to obtain 1,281,167 gray images to be processed and reference color images, and these gray images to be processed and reference color images are organized into a training set. Then, 50,000 color images are obtained from the ImageNet validation set, and these color images are grayed to obtain 50,000 gray images to be processed and reference color images, and these gray images to be processed and reference color images are organized into a validation set. In an embodiment of the present invention, the image sizes in the training set and the validation set are both 224×224.

[0048] S2: Construct the multi-level image colorization network system provided by the present invention.

[0049] Among them, since the training set and test set in the embodiment of the present invention are from the ImageNet dataset, in the implementation of the present invention, the color category of the image is based on the channels of all pixels in the ImageNet training set (about 1.28 million images). a and Channel b A total of 313 color categories are constructed, and their corresponding relationships are shown in the conversion table Q shown in Table 1.

[0050] Table 1:

[0051]

[0052]

[0053]

[0054] According to the conversion table Q, the color category of each pixel in the real color image can be obtained. . Corresponding to the color category probability distribution in the CR module , the inverse color category matrix That is to a and b The channel value and color category correspond to the value and index of table Q, that is, .in, For color categories The channel corresponding to the center point of the small square a and Channel b The numerical value of .

[0055] S3: Input the training set and the validation set obtained in step S1 into the image colorization network system obtained in step S2, and perform training according to the classification loss function to obtain an image colorization network model.

[0056] The classification loss function used in training in step S3 is:

[0057] in, represents the classification loss function, represents the reference color image in the validation set, It indicates that the image colorization network model predicts the corresponding predicted color image based on the grayscale image to be processed in the validation set; represents the color category of the reference color image in the validation set, represents the color balance weight of the reference color image, represents the color category probability distribution in the reference color image, Represents the predicted color category probability distribution in the color image; q represents the color category of the corresponding pixel.

[0058] The color balance weight of the reference color image is obtained by the following formula :

[0059]

[0060] in, Represents the color category probability distribution of all pixels in the 128w images in the ImageNet training set and the inverse of the average distribution of all categories (here there are 313 categories, each category is 1 / 313), Represents the color category probability distribution of all pixels in the 128w images in the ImageNet training set The weighted value of the two distributions, , and the average distribution of all categories, is set to 0.3 in the embodiment of the present invention. represents the total number of color categories of all pixels in the data set in step S1, that is, S=313; Represents the color category probability distribution of all pixels in the data set, Indicates the percentage of pixels of color category q in the total pixels. Represents probability distribution The probability of being in color category q.

[0061] In the training process of step S3, the embodiment of the present invention uses the Xavier normal function to initialize the image colorization network system, and uses the SGD (Batch gradient descent) optimizer to train the image colorization network system. The initial learning rate Set to 10 -3 , the momentum parameter is set to 0.9, and the weight decay is set to 10 -4 , the batch size is set to 64, and the training is 10 cycles. Among them, the learning rate decays with training according to the following formula:

[0062]

[0063]

[0064] in, is the number of training cycles, is the total number of training cycles, is the current number of training times, is the exponential parameter, here it is 0.9, is the current learning rate.

[0065] S4: Input the test grayscale image into the trained image colorization network model to obtain a predicted color image corresponding to the test grayscale image.

[0066] The test grayscale image in step S4 comes from the ImageNet dataset, and the process of obtaining the test grayscale image is the same as the process of obtaining the grayscale image to be processed.

[0067] In order to verify the effectiveness of the multi-level image colorization method provided by the embodiment of the present invention for color restoration, the multi-level image colorization method provided by the embodiment of the present invention is compared with those of Larsson et al., Zhang et al., Vitoria et al., Su et al., and Ji et al., and the resolution of the color images generated by all methods is adjusted to 256×256. The comparison results are shown in FIG. Figure 6 As shown, it can be seen that the multi-level image colorization method provided by the embodiment of the present invention can generate a more accurate, full, bright color image without obvious color overflow.

[0068] At the same time, in order to quantitatively analyze and compare the difference in color restoration between the method provided by the embodiment of the present invention and the comparison method, the embodiment of the present invention uses the peak signal-to-noise ratio (PSNR), structural similarity (SSIM) and color vividness (Colorful) indicators to compare the difference between the real color image corresponding to the test grayscale image and the predicted color image corresponding to the test grayscale image. The comparison results are shown in Table 2.

[0069] Among them, the formula for peak signal-to-noise ratio PSNR is:

[0070] in, Indicates the maximum value that an image pixel can take. Represents the mean square error between the predicted color image and the corresponding true color image. The larger the PSNR value, the smaller the image distortion and the better the image quality.

[0071] The formula for structural similarity SSIM is:

[0072] in, represents the mean of the predicted color image, represents the mean of the real color image, represents the standard deviation of the predicted color image, represents the standard deviation of the true color image, represents the covariance between the predicted color image and the true color image, and There are two variables used to prevent the denominator from being zero and increase the stability of the formula. When the value of the structural similarity SSIM is closer to 1, it means that the predicted color image and the true color image are more similar in structure.

[0073] Colorfulness is an empirical formula for evaluating the color vividness of color images proposed by David et al. in the paper "Measuring colorfulness in natural images", specifically:

[0074] in, and Respectively represent the variance of the matrix corresponding to channel a and the matrix corresponding to channel b in the color image, represents the mean of the matrix c, and the matrix The larger the value of Colorful, the more pixels of bright colors there are in the image, and the brighter the color of the image.

[0075] Table 2:

[0076] It can be seen from Table 2 that the PSNR value of the predicted color image obtained by the multi-level image colorization method provided by the embodiment of the present invention is the highest, indicating that the predicted color image has the smallest image distortion and the best image quality. The SSIM value of the predicted color image is closest to 1, indicating that the predicted color image is more similar in structure to the corresponding true color image. The Colorful value of the predicted color image can reflect The color image generated by the image colorization algorithm provided by the present invention has a more vivid coloring effect. Although the Colorful value of the algorithm of Zhang et al. is higher than the Colorful value of the image colorization algorithm provided by the present invention, the image colorization algorithm of Zhang et al. does not take into account the outline of the object in the image, the category and the consistency with the background, resulting in oversaturation or bright colors that should not exist in the generated color image, making the Colorful of Zhang et al. even higher than the real image. It can be seen from the intuitive comparison of images and the comparison of evaluation index values ​​that the multi-level image colorization method provided by the embodiment of the present invention can restore the color better.

[0077] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the disclosure of the present invention can be performed in parallel, sequentially or in different orders, as long as the desired results of the technical solution disclosed in the present invention can be achieved, and this document does not limit this.

[0078] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A multi-level image colorization network system, characterized by: It includes a feature extraction branch and a color processing branch; wherein the feature extraction branch performs N-level feature extraction on the input grayscale image to obtain N features of different sizes, N being not less than 3; the N features of different sizes are input into the color processing branch for feature fusion and color matching to obtain the color features corresponding to the grayscale image; finally, the grayscale image is spliced ​​with the color features to obtain the color image corresponding to the grayscale image.

2. The multi-level image colorization network system according to claim 1, characterized in that: In the feature extraction branch, the grayscale image is first input into the feature extraction module for processing to obtain the first-level features; the previous-level features are downsampled and pooled, and then the pooled features are input into the feature extraction module for processing to obtain the current-level features; the above operation is repeated N-1 times to obtain N-1 features of different sizes.

3. The multi-level image colorization network system according to claim 1, characterized in that: The color processing branch includes a LAATF module, a feature extraction module and a CR module; wherein N features of different sizes are input into the LAATF module for feature fusion; the fused features are first subjected to a convolution operation of a 1×1 convolution layer and then input into the feature extraction module for feature extraction; the features output by the feature extraction module are then input into the CR module for color matching, the color features.

4. The multi-level image colorization network system according to claim 1, characterized in that: In the LAATF module, N features of different sizes are input into the corresponding global pooling layers to obtain corresponding pooling features, and the corresponding pooling features are input into the corresponding fully connected layers to obtain corresponding fully connected features; Concatenate N fully connected features, and input the concatenated features into the ReLU activation function layer, 1×1 convolution layer, and Sigmoid activation function layer in sequence to obtain the weighted features; After multiplying the corresponding elements of N features of different sizes by the weighted features, the N multiplied features are scaled to obtain N features of the same size; Finally, N features of the same size are spliced ​​and the spliced ​​features are output.

5. The multi-level image colorization network system according to any one of claims 2 to 4, characterized in that: The feature extraction module includes a CADC submodule and a LIRA submodule; wherein: the input feature is first passed through the CADC submodule to obtain a CDC feature; and the CADC feature is then input into the LIRA submodule to obtain an extraction feature corresponding to the input feature.

6. The multi-level image colorization network system according to claim 5, characterized in that: In the CADC submodule, the separable convolution layer performs a convolution operation on the input features, and the features obtained after the convolution operation are further subjected to a lightweight channel attention operation to obtain the CADC features.

7. The multi-level image colorization network system according to claim 6, characterized in that: In the LIRA sub-module, the CADC feature is first subjected to a lightweight spatial attention operation and a lightweight channel attention operation in sequence to obtain a primary LIRA feature; the primary LIRA feature is then subjected to a lightweight spatial attention operation and a lightweight channel attention operation in sequence to obtain a secondary LIRA feature; the CADC feature, the primary LIRA feature and the secondary LIRA feature are added together to obtain the extracted feature.

8. A multi-level image colorization method, characterized in that: The specific steps include: S1: Obtain pairs of grayscale images to be processed and reference color images from the dataset, and establish training sets and validation sets; S2: Constructing an image colorization network system based on a lightweight attention structure as described in any one of claims 1 to 7; S3: input the training set and the validation set obtained in step S1 into the image colorization network system obtained in step S2, and perform training according to the classification loss function to obtain an image colorization network model; S4: Input the test grayscale image into the trained image colorization network model to obtain a predicted color image corresponding to the test grayscale image.

9. The multi-level image colorization network method according to claim 8, characterized in that: The classification loss function used in training in step S3 is: in, represents the classification loss function, represents the reference color image in the training set, It indicates that the image colorization network model predicts the corresponding predicted color image according to the grayscale image to be processed in the training set; represents the color category of the reference color image, represents the color balance weight of the reference color image, represents the color category probability distribution in the reference color image, represents the color category probability distribution in the predicted color image; h and w represent the position of the pixel in the image, and q represents the color category of the pixel.

10. The multi-level image colorization network method according to claim 9, characterized in that: The color balance weight of the reference color image is obtained by the following formula : in, Represents the color category probability distribution of all pixels in the image in the training set and the inverse of the average distribution of all classes, Represents the color category probability distribution of all pixels in the image in the training set and the average distribution of all categories are weighted values ​​of the two distributions, Represents the total number of color categories of all pixels in the data set in step S1; represents the color category probability distribution of all pixels in the data set, Indicates the percentage of pixels of color category q in the total pixels. Represents probability distribution The probability of being in color category q.

Citation Information

Cited By

  • Gray scale image coloring method based on combination of dual-scale convolutional network and attention fusion strategy

    CN119515700A

  • Low-illumination grayscale image colorization method

    CN120510070A

  • Low-illumination grayscale image colorization method

    CN120510070B