A deep color gamut compression method and device based on lightweight convolutional network

Abstract

The present invention relates to a deep color gamut compression method and device based on a lightweight convolutional network, and belongs to the field of image processing technology. A pre-trained deep color gamut compression model is obtained, the model includes a plurality of down-sampling encoding modules with different sampling rates connected in series along a forward propagation direction and a plurality of up-sampling modules with different sampling rates; each down-sampling encoding module includes a refocused depth separable convolution unit, a refocused depth convolution unit, and a pooling unit connected in series along the forward propagation direction; each up-sampling module includes an up-sampling unit, a refocused depth separable convolution unit, and a refocused depth convolution unit connected in series along the forward propagation direction; the image to be compressed is input into a plurality of down-sampling encoding modules with different sampling rates connected in series along the forward propagation direction for feature extraction, and a target feature map is output; the target feature map is input into a plurality of up-sampling modules with different sampling rates connected in series along the forward propagation direction for pixel restoration, and a compressed target image is output.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a deep color gamut compression method, device and computer-readable storage medium based on a lightweight convolutional network. Background Art

[0002] Color gamut is used to describe the range of colors that a given image or device can display or capture. Due to the differences in color gamuts among various devices, color gamut compression is required in a connected color space to ensure color consistency across devices. Color gamut compression is a necessary means for cross-device color reproduction. The algorithm establishes a mapping relationship between the color gamuts of two different devices to perform color mapping between different media, thereby presenting similar color effects on different devices. How to accurately reproduce color images that exceed the color gamut range while minimizing the loss of key image details and color is the core problem that color gamut compression technology needs to solve.

[0003] Existing color gamut compression methods include point-by-point color gamut compression, compensatory color gamut compression, and iterative color gamut compression; point-by-point color gamut compression is mainly performed in the brightness, hue, and chroma space, and the brightness and chroma values ​​are mapped by keeping the hue angle unchanged. This algorithm operates on a single pixel point, and each pixel point has a fixed mapping formula, which can achieve a very high processing speed. However, since the pixels are completely independent, the spatial feature information of the image is not considered, resulting in serious loss of image details after color gamut compression; the compensatory color gamut compression algorithm belongs to the spatial color gamut compression algorithm, which performs high- and low-frequency image compression by separating high and low frequencies. After color gamut compression, the row is merged with the high-frequency image and the merged image is remapped. However, this type of algorithm directly adds the high-frequency and low-frequency images, which will cause halos and excessive artificial textures in the compression result. The iterative color gamut compression algorithm defines the problem as a constrained optimization problem and uses several metrics as the objective function to minimize the human eye perception of the original image and the target image. The image quality after color gamut compression using this algorithm is relatively high, but the optimization criterion required by the algorithm depends on a complex image quality evaluation model. Each iteration needs to use the image quality evaluation model to calculate the quality of the current iteration result, which takes a long time and requires a lot of calculation.

[0004] In summary, the existing color gamut compression methods cannot take into account both the target image quality after color gamut compression and the processing speed of the color gamut compression algorithm. Summary of the invention

[0005] Therefore, the technical problem to be solved by the present invention is to overcome the problem that the color gamut compression method in the prior art cannot take into account both the quality and processing speed of the target image obtained after the color gamut compression.

[0006] In order to solve the above technical problems, the present invention provides a deep color gamut compression method based on a lightweight convolutional network, comprising:

[0007] Acquire a pre-trained deep color gamut compression model; the deep color gamut compression model comprises a plurality of down-sampling encoding modules with different sampling rates and a plurality of up-sampling modules with different sampling rates which are sequentially connected in series along a forward propagation direction; each down-sampling encoding module comprises a refocused depth separable convolution unit, a refocused depth convolution unit and a pooling unit which are sequentially connected in series along the forward propagation direction; each up-sampling module comprises an up-sampling unit, a refocused depth separable convolution unit and a refocused depth convolution unit which are sequentially connected in series along the forward propagation direction;

[0008] The image to be compressed is input into a plurality of down-sampling coding modules with different sampling rates connected in series in the forward propagation direction for feature extraction, and a target feature map is output;

[0009] The target feature map is input into a plurality of up-sampling modules with different sampling rates which are connected in series in a forward propagation direction to perform pixel restoration, and a compressed target image is output.

[0010] Preferably, the image to be compressed is input into a plurality of down-sampling encoding modules with different sampling rates which are sequentially connected in series along the forward propagation direction for feature extraction, and the output target feature map includes:

[0011] Input the image to be compressed into the refocused depth-separable convolution unit in the first downsampling encoding module for convolution, and output a first feature map;

[0012] Inputting the first feature map into a refocusing depth convolution unit in a first downsampling encoding module for convolution, and outputting a second feature map;

[0013] Inputting the second feature map into the pooling unit in the first downsampling encoding module for maximum pooling, and outputting a pooled feature map;

[0014] The pooled feature map is used as the input of the next down-sampling encoding module, and the output of the pooling unit in the last down-sampling encoding module is used as the target feature map.

[0015] Preferably, the target feature map is input into a plurality of up-sampling modules with different sampling rates connected in series in the forward propagation direction for pixel restoration, and the compressed target image is outputted, comprising:

[0016] Inputting the target feature map into an upsampling unit in a first upsampling module for upsampling, and outputting a first target feature map;

[0017] Inputting the first target feature map into a refocused depth-separable convolution unit in a first upsampling module for convolution, and outputting a second target feature map;

[0018] Input the second target feature map into the refocusing depth convolution unit in the first upsampling module for convolution, and output a third target feature map;

[0019] The third target feature map is used as the input of the next upsampling module, and the output of the refocusing depth convolution unit in the last upsampling module is used as the compressed target image.

[0020] Preferably, the training process of the deep color gamut compression model includes:

[0021] Get image samples in the training set;

[0022] Input the image sample into a plurality of down-sampling encoding modules with different sampling rates connected in series in a forward propagation direction to extract features, and output a target feature map;

[0023] Input the target feature map into a plurality of upsampling modules with different sampling rates connected in series in a forward propagation direction for pixel restoration, and construct a deep color gamut compression loss function based on the output of the last upsampling module;

[0024] Iteratively train multiple down-sampling coding modules with different sampling rates and multiple up-sampling modules with different sampling rates using image samples in a training set until the value of the deep color gamut compression loss function is minimized, thereby obtaining multiple trained down-sampling coding modules with different sampling rates and multiple up-sampling modules with different sampling rates;

[0025] A trained deep color gamut compression model is obtained based on a plurality of trained down-sampling encoding modules with different sampling rates and a plurality of trained up-sampling modules with different sampling rates.

[0026] Preferably, constructing a deep color gamut compression loss function based on the output of the last upsampling module comprises:

[0027] Based on the output of the last upsampling module and the label image of the image sample, construct the brightness mean absolute error loss function, the chroma mean absolute error loss function and the hue mean absolute error loss function;

[0028] Based on the brightness similarity, contrast similarity and structural similarity between the output of the last upsampling module and the label image of the image sample, a multi-scale structural similarity loss function of brightness visual perception, a multi-scale structural similarity loss function of chroma visual perception and a multi-scale structural similarity loss function of hue visual perception are constructed;

[0029] Based on the brightness mean absolute value error loss function and the brightness visual perception multi-scale structural similarity loss function, construct a loss function of the image sample on the brightness channel;

[0030] Based on the chroma mean absolute error loss function and the chroma visual perception multi-scale structural similarity loss function, construct a loss function of the image sample on the chroma channel;

[0031] Based on the hue mean absolute value error loss function and the hue visual perception multi-scale structural similarity loss function, construct a loss function of the image sample on the hue channel;

[0032] A loss function of the image sample on the lightness channel, a loss function on the chroma channel, and a loss function on the hue channel are weightedly summed to obtain a deep color gamut compression loss function.

[0033] Preferably, the brightness mean absolute error loss function is:

[0034] ,

[0035] in, represents the brightness mean absolute error loss function, Represents the pixel representation of the target image on the brightness channel after the image sample output of the last upsampling module is compressed. The pixel representation of the label image representing the image sample on the brightness channel, The label image representing the image sample is in the brightness channel The value of the pixel, Indicates the target image after image sample compression on the brightness channel The value of the pixel, Represents the number of pixels of the image sample;

[0036] The chroma mean absolute error loss function is:

[0037] ,

[0038] in, represents the chroma mean absolute error loss function, The pixel representation of the target image on the chroma channel after the image sample output of the last upsampling module is compressed. The pixel representation of the label image representing the image sample on the chroma channel, The label image representing the image sample is in the chroma channel. The value of the pixel, Indicates the target image after image sample compression on the chroma channel The value of the pixel;

[0039] The hue mean absolute error loss function is:

[0040] ,

[0041] in, represents the hue mean absolute value error loss function, The pixel representation of the target image on the hue channel after the image sample output of the last upsampling module is compressed. The pixel representation of the label image representing the image sample on the hue channel, The label image representing the image sample is the first The value of the pixel, Indicates the target image after image sample compression on the hue channel The value of the pixel;

[0042] The multi-scale structural similarity loss function of brightness visual perception is:

[0043] ,

[0044] in, Represents the multi-scale structural similarity loss function of brightness visual perception, express and The brightness similarity between To express The parameters of importance, express and In the Contrast similarity at scales, represents the scale of the image sample, To express The parameters of importance, express and In the The structural similarity at each scale is To express Parameters of importance;

[0045] The multi-scale structural similarity loss function of chromaticity visual perception is:

[0046] ,

[0047] in, represents the multi-scale structural similarity loss function of chromaticity visual perception, express and The brightness similarity between To express The parameters of importance, express and In the Contrast similarity at scales, To express The parameters of importance, express and In the The structural similarity at each scale is To express Parameters of importance;

[0048] The multi-scale structural similarity loss function of hue visual perception is:

[0049] ,

[0050] in, represents the multi-scale structural similarity loss function of hue visual perception, express and The brightness similarity between To express The parameters of importance, express and In the Contrast similarity at scales, To express The parameters of importance, express and In the The structural similarity at each scale is To express Parameters of importance.

[0051] Preferably, the loss function of the image sample on the brightness channel is:

[0052] ,

[0053] in, Represents the loss function of the image sample on the brightness channel, express The weight of represents the Gaussian weight matrix;

[0054] The loss function of the image sample on the chroma channel is:

[0055] ,

[0056] in, Represents the loss function of the image sample on the chroma channel, express The weight of

[0057] The loss function of the image sample on the hue channel is:

[0058] ,

[0059] in, represents the loss function of the image sample on the hue channel, express The weight of .

[0060] Preferably, the depth color gamut compression loss function is:

[0061] ,

[0062] in, represents the deep color gamut compression loss function, Represents the weight of the loss function of the image sample on the brightness channel, represents the weight of the loss function of the image sample on the chroma channel, Represents the weight of the loss function for image samples on the hue channel.

[0063] Preferably, the label image of each image sample in the training set is an image obtained by performing color gamut compression on the image sample using an iterative color gamut compression algorithm.

[0064] The present invention also provides a deep color gamut compression device based on a lightweight convolutional network, comprising:

[0065] A model acquisition module, used to acquire a pre-trained deep color gamut compression model; the deep color gamut compression model includes a plurality of downsampling encoding modules with different sampling rates and a plurality of upsampling modules with different sampling rates connected in series in a forward propagation direction; each downsampling encoding module includes a refocused depth separable convolution unit, a refocused depth convolution unit and a pooling unit connected in series in a forward propagation direction; each upsampling module includes an upsampling unit, a refocused depth separable convolution unit and a refocused depth convolution unit connected in series in a forward propagation direction;

[0066] A feature extraction module is used to extract features from the image to be compressed input into a plurality of down-sampling encoding modules with different sampling rates connected in series in a forward propagation direction, and output a target feature map;

[0067] The pixel restoration module is used to input the target feature map into a plurality of up-sampling modules with different sampling rates connected in series in a forward propagation direction to perform pixel restoration and output a compressed target image.

[0068] The deep color gamut compression method based on a lightweight convolutional network provided in the present application applies deep learning to the field of color gamut compression of images, uses multiple downsampling coding modules with different sampling rates to condense the information of the input image, gradually extracts image feature information, obtains global information within a larger receptive field through a downsampling coding module with a lower sampling rate, and extracts local detail features in the image using a downsampling coding module with a higher sampling rate; then uses multiple upsampling modules with different sampling rates to perform pixel restoration on the target feature map obtained by the downsampling coding module, and realizes step-by-step refinement and restoration of the target feature map through upsampling modules with different sampling rates connected in series, restoring the target feature map to a higher resolution, thereby better preserving the details and texture information of the original image, and ensuring the image quality of the target image obtained after color gamut compression; at the same time, the present application replaces the conventional convolution units in the downsampling coding module and the upsampling module with a refocusing depth separable convolution unit and a refocusing depth convolution unit, thereby achieving lightweight of the model, reducing the number of parameters and the amount of calculation of the model, improving the running speed of the model, and making the model easier to deploy and expand on different hardware and platforms. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] In order to make the content of the present invention more clearly understood, the present invention is further described in detail below according to specific embodiments of the present invention in conjunction with the accompanying drawings, wherein:

[0070] Figure 1 Schematic diagram of the color reproduction process provided for this application;

[0071] Figure 2 A flow chart of the deep color gamut compression method based on a lightweight convolutional network provided in this application;

[0072] Figure 3 A schematic diagram of the deep color gamut compression model structure provided for this application;

[0073] Figure 4 Schematic diagram of the DSC convolution structure provided for this application;

[0074] Figure 5 Schematic diagram of the lightweight convolution design process provided for this application; wherein, Figure 5 (a) is a schematic diagram of a conventional double-layer convolution structure. Figure 5 (b) is a schematic diagram of a lightweight double-layer convolution structure. Figure 5 (c) is a schematic diagram of the re-parameterized double-layer convolution structure;

[0075] Figure 6 A schematic diagram of the improved refocusing of the DW convolution kernel provided in this application;

[0076] Figure 7Schematic diagram of the deep color gamut compression model training process provided for this application;

[0077] Figure 8 A schematic diagram of an image after color gamut compression obtained by using different color gamut compression methods provided in the present application; wherein, Figure 8 (a) in the figure is the image before color gamut compression. Figure 8 (b) is the color gamut mapping algorithm used to Figure 8 The image shown in (a) is the image after color gamut compression. Figure 8 (c) in the figure is the iterative color gamut compression algorithm (iCID) Figure 8 The image shown in (a) is the image after color gamut compression. Figure 8 (d) is to use the method provided by this application to Figure 8 (a) is the image after color gamut compression;

[0078] Fig. 9 This is another schematic diagram of an image after color gamut compression obtained by using different color gamut compression methods provided by the present application; wherein, Fig. 9 (a) in the figure is the image before color gamut compression. Fig. 9 (b) is the color gamut mapping algorithm used to Fig. 9 The image shown in (a) is the image after color gamut compression. Fig. 9 (c) in the figure is the iterative color gamut compression algorithm (iCID) Fig. 9 The image shown in (a) is the image after color gamut compression. Fig. 9 (d) is to use the method provided by this application to Fig. 9 The image shown in (a) is the image after color gamut compression;

[0079] Fig.10 A schematic diagram of the application of the method provided in this application in a practical scenario;

[0080] Fig.11 A schematic diagram of the structure of a deep color gamut compression device based on a lightweight convolutional network provided in this application. DETAILED DESCRIPTION

[0081] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments so that those skilled in the art can better understand the present invention and implement it, but the embodiments are not intended to limit the present invention.

[0082] See also Figure 1Taking the color reproduction process from display to printer device as an example, color gamut mapping is at the center of the conversion process. First, the RGB color space is converted to the device-independent LAB color space through the characterization model, and then the color gamut mapping is performed in the LAB color space. This process aims to ensure that the color of the mapped image falls completely within the target color gamut and minimize the visual difference between the original image and the mapped image. Finally, the mapping result is converted to the CMYK color gamut of the printing device and printed using the printer's characterization model.

[0083] Based on the above-mentioned color gamut mapping principle and the problems existing in the color gamut compression algorithm in the prior art, this application applies deep learning to the color gamut compression of images for the first time, so as to take into account both the image quality and processing speed after color gamut compression.

[0084] See also Figure 2 , Figure 2 The flowchart of the deep color gamut compression method based on the lightweight convolutional network provided by the present application is shown, and the method specifically includes:

[0085] S10: Acquire a pre-trained deep color gamut compression model; the deep color gamut compression model includes a plurality of downsampling encoding modules with different sampling rates and a plurality of upsampling modules with different sampling rates connected in series in a forward propagation direction; each downsampling encoding module includes a refocused depth separable convolution unit, a refocused depth convolution unit, and a pooling unit connected in series in a forward propagation direction; each upsampling module includes an upsampling unit, a refocused depth separable convolution unit, and a refocused depth convolution unit connected in series in a forward propagation direction;

[0086] S20: Input the image to be compressed into a plurality of downsampling encoding modules with different sampling rates connected in series in a forward propagation direction for feature extraction, and output a target feature map;

[0087] S30: Input the target feature map into a plurality of up-sampling modules with different sampling rates connected in series in a forward propagation direction to perform pixel restoration, and output a compressed target image.

[0088] The deep color gamut compression method based on a lightweight convolutional network provided in the present application applies deep learning to the field of color gamut compression of images, uses multiple downsampling coding modules with different sampling rates to condense the information of the input image, gradually extracts image feature information, obtains global information within a larger receptive field through a downsampling coding module with a lower sampling rate, and extracts local detail features in the image using a downsampling coding module with a higher sampling rate; then uses multiple upsampling modules with different sampling rates to perform pixel restoration on the target feature map obtained by the downsampling coding module, and realizes step-by-step refinement and restoration of the target feature map through upsampling modules with different sampling rates connected in series, restoring the target feature map to a higher resolution, thereby better preserving the details and texture information of the original image, and ensuring the image quality of the target image obtained after color gamut compression; at the same time, the present application replaces the conventional convolution units in the downsampling coding module and the upsampling module with a refocusing depth separable convolution unit and a refocusing depth convolution unit, thereby achieving lightweight of the model, reducing the number of parameters and the amount of calculation of the model, improving the running speed of the model, and making the model easier to deploy and expand on different hardware and platforms.

[0089] Specifically, step S20 is implemented as follows:

[0090] S200: Inputting the image to be compressed into the refocusing depth separable convolution unit in the first downsampling encoding module for convolution, and outputting a first feature map;

[0091] S201: Inputting the first feature map into the refocusing depth convolution unit in the first downsampling encoding module for convolution, and outputting a second feature map;

[0092] S202: Input the second feature map into the pooling unit in the first down-sampling encoding module for maximum pooling, and output the pooling feature map;

[0093] S203: Using the pooled feature map as the input of the next downsampling encoding module, and using the output of the pooling unit in the last downsampling encoding module as the target feature map.

[0094] Specifically, step S30 is implemented as follows:

[0095] S300: inputting the target feature map into an upsampling unit in a first upsampling module for upsampling, and outputting a first target feature map;

[0096] S301: Inputting the first target feature map into the refocusing depth separable convolution unit in the first upsampling module for convolution, and outputting a second target feature map;

[0097] S302: Input the second target feature map into the refocusing depth convolution unit in the first upsampling module for convolution, and output a third target feature map;

[0098] S303: Using the third target feature map as the input of the next upsampling module, and using the output of the refocusing depth convolution unit in the last upsampling module as the compressed target image.

[0099] For example, Figure 3 As shown is a schematic diagram of the structure of the deep color gamut compression model provided by the present application, the size of the image to be compressed is 64*64*3, which is input into the deep color gamut compression model, and two down-sampling encoding modules with gradually decreasing sampling rates connected in series are used to extract features of the image to be compressed, and a target feature map is output, and a double-layer convolution module is used to convolve the target feature map, and then two up-sampling modules with gradually increasing sampling rates connected in series are used to restore pixels of the feature map output by the double-layer convolution module. The model makes full use of the high-rank feature information obtained by the deep network and the image context information contained in the shallow network, and adopts a copy and superposition jump connection method for feature fusion. Finally, a channel transformation module is used to perform channel transformation on the output of the second up-sampling module to output a compressed target image with a size of 64*64*3.

[0100] Specifically, the present application uses a refocused depth separable convolution unit and a refocused depth convolution unit to replace the conventional convolution units in the downsampling encoding module and the upsampling module, thereby achieving lightweight models, improving the efficiency and ability of the model to process image and video data, and meeting the limitations of memory and power consumption. The following provides the principles and process of lightweight design of the model in the present application:

[0101] like Figure 4 As shown in the figure, the common lightweight convolution depth separable convolution is a combination of channel-by-channel convolution and point-by-point convolution. Channel-by-channel convolution uses an independent convolution kernel for feature extraction for each channel of the feature map. Since channel-by-channel convolution cannot change the number of output channels, the entire convolution process also requires subsequent point-by-point 1*1 convolution kernels to change the number of output channels and fuse channel information.

[0102] A channel-by-channel convolution can be regarded as a series connection of multiple independent two-dimensional convolution kernels (kernel channels). Since a specific kernel channel is not associated with the kernel channels corresponding to other feature maps, the representation ability of the model will be limited. Although information interaction between channels has been performed through channel-by-channel convolution in depth-separable convolution, this design is overly dependent on the effect of channel-by-channel convolution, and the generalization of the model is not high. Based on this, this application designs a lightweight two-layer convolution block to replace the conventional two-layer convolution block in the downsampling encoding module and the upsampling module. The specific design process is as follows:

[0103] See also Figure 5 , Figure 5The schematic diagram of the lightweight convolution design process provided for this application is as follows: Figure 5 (a) in FIG. 1 shows a conventional double-layer convolution structure, which includes two convolution blocks (convolution, Conv), has many parameters and is not easy to deploy. Based on the defects of the conventional double-layer convolution block, this application first performs a preliminary lightweighting of the structure, such as Figure 5 (b) in FIG. 1 shows a lightweight two-layer convolution structure, which includes a depthwise separable convolution block (DSC) and a channel-by-channel convolution block (DW). Furthermore, in order to maintain the inference accuracy of the model on the basis of lightweight, the present application refocuses the parameters of all DW convolution blocks, and finally obtains the following: Figure 5 The re-parameterized two-layer convolution structure shown in (c) in Fig. 1 includes a refocusing depthwise separable convolution block (RefDSC) and a refocusing depthwise convolution block (RefDW).

[0104] Specifically, Figure 6 As shown is a schematic diagram of the present application refocusing the parameters of the DW convolution block. In the RefDW convolution block, the DW convolution block parameters in the upper model are input into a trainable refocusing parameter change mechanism to establish a connection between the parameters. Refocusing learning allows the input feature map to refocus on other parts of the model instead of only focusing on the input features of the model. RefDW improves the priori of the existing model structure, uses the representation encoded in the pre-trained parameters as a priori, and refocuses on these parameters to learn richer deep features, thereby further enhancing the representation ability of the model.

[0105] Specifically, the embodiment of the present application also provides a training process of a deep color gamut compression model, such as Figure 7 As shown, the process specifically includes:

[0106] Get image samples in the training set;

[0107] The image samples are input into a plurality of down-sampling encoding modules with different sampling rates connected in series in the forward propagation direction for feature extraction, and a target feature map is output;

[0108] The target feature map is input into multiple upsampling modules with different sampling rates connected in series along the forward propagation direction for pixel restoration, and a deep color gamut compression loss function is constructed based on the output of the last upsampling module;

[0109] Iteratively train multiple down-sampling coding modules with different sampling rates and multiple up-sampling modules with different sampling rates using image samples in a training set until the value of the deep color gamut compression loss function is minimized, thereby obtaining multiple trained down-sampling coding modules with different sampling rates and multiple up-sampling modules with different sampling rates;

[0110] A trained deep color gamut compression model is obtained based on a plurality of trained down-sampling encoding modules with different sampling rates and a plurality of trained up-sampling modules with different sampling rates.

[0111] Optionally, in some embodiments of the present application, the deep color gamut compression loss function may adopt a mean square error (MSE) loss function commonly used in existing color gamut compression algorithms. The MSE loss function can accurately limit the source color gamut within the target color gamut, while expanding the color gamut boundary value to a wider color gamut, retaining most of the key pixel information.

[0112] However, during the color gamut compression process, the mean square error loss function will act on the pixels of the entire image, resulting in a large amount of detail loss. In fact, since the human visual system is extremely sensitive to changes in brightness and color, a loss function that only considers color features and ignores spatial features will destroy the spatial characteristics and thus affect the visual effect. However, although the introduction of structural loss can improve this problem, it may cause pixels to exceed the color gamut. Therefore, it is necessary to find a balance between color features and spatial features.

[0113] Based on the above reasons, as a preferred embodiment, in addition to focusing on pixel-level error loss, the present application also introduces a lightweight visual perception multi-scale structural similarity loss (Multi Scale Structural Similarity IndexMeasure, MS-SSIM); at the same time, considering that in the color gamut compression process, the hue (Hue, H) channel, lightness (L) channel and chroma (C) channel will affect the mapping quality, therefore, the present application weights the loss of the image on the three channels as a deep color gamut compression loss function.

[0114] Specifically, constructing the deep color gamut compression loss function based on the output of the last upsampling module includes:

[0115] Based on the output of the last upsampling module and the label image of the image sample, construct the brightness mean absolute error loss function, the chroma mean absolute error loss function and the hue mean absolute error loss function;

[0116] Based on the brightness similarity, contrast similarity and structural similarity between the output of the last upsampling module and the label image of the image sample, a multi-scale structural similarity loss function of brightness visual perception, a multi-scale structural similarity loss function of chroma visual perception and a multi-scale structural similarity loss function of hue visual perception are constructed;

[0117] Based on the brightness mean absolute value error loss function and the brightness visual perception multi-scale structural similarity loss function, the loss function of the image sample in the brightness channel is constructed;

[0118] Based on the chroma mean absolute error loss function and the chroma visual perception multi-scale structural similarity loss function, the loss function of the image sample in the chroma channel is constructed;

[0119] Based on the hue mean absolute value error loss function and the hue visual perception multi-scale structural similarity loss function, a loss function of the image sample on the hue channel is constructed;

[0120] The loss function of the image sample on the lightness channel, the loss function on the chroma channel, and the loss function on the hue channel are weighted summed to obtain the deep color gamut compression loss function.

[0121] Specifically, the brightness mean absolute error loss function is:

[0122] ,

[0123] in, represents the brightness mean absolute error loss function, Represents the pixel representation of the target image on the brightness channel after the image sample output of the last upsampling module is compressed. The pixel representation of the label image representing the image sample on the brightness channel, The label image representing the image sample is in the brightness channel The value of the pixel, Indicates the target image after image sample compression on the brightness channel The value of the pixel, Represents the number of pixels of the image sample;

[0124] The chroma mean absolute error loss function is:

[0125] ,

[0126] in, represents the chroma mean absolute error loss function, The pixel representation of the target image on the chroma channel after the image sample output of the last upsampling module is compressed. The pixel representation of the label image representing the image sample on the chroma channel, The label image representing the image sample is in the chroma channel. The value of the pixel, Indicates the target image after image sample compression on the chroma channel The value of the pixel;

[0127] The hue mean absolute error loss function is:

[0128] ,

[0129] in, represents the hue mean absolute value error loss function, The pixel representation of the target image on the hue channel after the image sample output of the last upsampling module is compressed. The pixel representation of the label image representing the image sample on the hue channel, The label image representing the image sample is the first The value of the pixel, Indicates the target image after image sample compression on the hue channel The value of the pixel;

[0130] The multi-scale structural similarity loss function of brightness visual perception is:

[0131] ,

[0132] in, Represents the multi-scale structural similarity loss function of brightness visual perception, express and The brightness similarity between To express The parameters of importance, express and In the Contrast similarity at scales, represents the scale of the image sample, To express The parameters of importance, express and In the The structural similarity at each scale is To express Parameters of importance;

[0133] The multi-scale structural similarity loss function of chromaticity visual perception is:

[0134] ,

[0135] in, represents the multi-scale structural similarity loss function of chromaticity visual perception, express and The brightness similarity between To express The parameters of importance, express and In the Contrast similarity at scales, To express The parameters of importance, express and In the The structural similarity at each scale is To express Parameters of importance;

[0136] The multi-scale structural similarity loss function of hue visual perception is:

[0137] ,

[0138] in, represents the multi-scale structural similarity loss function of hue visual perception, express and The brightness similarity between To express The parameters of importance, express and In the Contrast similarity at scales, To express The parameters of importance, express and In the The structural similarity at each scale is To express Parameters of importance;

[0139] Specifically, the calculation formula for the brightness similarity between images is:

[0140] ,

[0141] in, Representing images and images The brightness similarity between Representing images The average value of Representing images The average value of A constant introduced to avoid zero denominator;

[0142] The calculation formula of contrast similarity is:

[0143] ,

[0144] in, Representing images and images The contrast similarity between Representing images The standard deviation of Representing images The standard deviation of A constant introduced to avoid the denominator being zero; the calculation formula for structural similarity is:

[0145] ,

[0146] in, Representing images and images The structural similarity between Representing images and images The covariance between A constant introduced to avoid the denominator being zero; further, the loss function of the image sample on the brightness channel is:

[0147] ,

[0148] in, Represents the loss function of the image sample on the brightness channel, express The weight of represents the Gaussian weight matrix;

[0149] The loss function of the image sample on the chroma channel is:

[0150] ,

[0151] in, Represents the loss function of the image sample on the chroma channel, express The weight of

[0152] The loss function of the image sample on the hue channel is:

[0153] ,

[0154] in, represents the loss function of the image sample on the hue channel, express The weight of

[0155] The deep color gamut compression loss function is:

[0156] ,

[0157] in, represents the deep color gamut compression loss function, Represents the weight of the loss function of the image sample on the brightness channel, represents the weight of the loss function of the image sample on the chroma channel, Represents the weight of the loss function for image samples on the hue channel.

[0158] Optionally, since the iterative color gamut compression algorithm regards color gamut compression as a constrained optimization problem, iCID, which has a strong correlation with image difference perception, is selected as the optimization function, and multiple iterative optimizations are performed through the iCID indicator to largely retain the contrast, structure and color of the original image. Therefore, the compressed target image obtained by the iterative color gamut compression algorithm usually has a higher quality. Based on this, in some embodiments of the present application, the label image of each image sample in the training set is an image obtained after color gamut compression of the image sample using the iterative color gamut compression algorithm.

[0159] In order to verify the effectiveness of the color gamut compression method provided by this application, this application also provides a schematic diagram of image comparison after color gamut compression using different color gamut compression methods, such as Figure 8 and Fig. 9 As shown, Figure 8 (a) in the figure is the image to be compressed. Figure 8 (b) in the figure is the target image after the image is compressed using the point-by-point color gamut compression (SGCK) algorithm. Figure 8 (c) in the figure is the target image after color gamut compression using the iCID iterative algorithm. Figure 8 (d) is the target image after the image is subjected to color gamut compression by the deep color gamut compression method based on a lightweight convolutional network provided in this application; Fig. 9 (a) in the figure is another image to be compressed. Fig. 9 (b) in the figure is the target image after color gamut compression using the SGCK algorithm. Fig. 9 (c) in the figure is the target image after color gamut compression using the iCID iterative algorithm. Fig. 9(d) is the target image after the image is compressed using the deep color gamut compression method based on a lightweight convolutional network provided in this application.

[0160] As can be seen from the figure, the target image compressed by the color gamut compression method provided by the present application approximates the iterative color gamut compression algorithm, and has better visual effects and higher image quality than the point-by-point color gamut compression algorithm.

[0161] At the same time, in order to verify that the color gamut compression method provided by the present application has a faster processing speed, the present application also provides a comparison result of the running time when different color gamut compression algorithms perform color gamut compression on the same image, as shown in Table 1:

[0162] Table 1

[0163] algorithm iCID SSIM CPU running time (s) HPMINDE 0.5944 0.6179 0.069 SGCK 0.5915 0.6448 0.071 Saturation enhancement algorithm 0.5928 0.6494 0.070 Compensation algorithm 0.4990 0.6280 278.0 iCID Iteration Method 0.4673 0.7069 272.1 This algorithm 0.4905 0.6790 0.208

[0164] From the data in Table 1, it can be seen that the running time of the color gamut compression method provided by the present application is close to that of the point-by-point color gamut compression algorithm. Compared with the SGCK algorithm, which has the best effect among the point-by-point color gamut compression algorithms, the iCID value is reduced by 17.08%, while the SSIM is improved by about 5.30%, and the running time is only increased by 0.137s. Compared with the iCID iterative algorithm with the best effect, the present application is slightly lower in the SSIM indicator, but the running time is significantly better than the iCID iterative algorithm, and the efficiency is improved by 1308 times.

[0165] From the above experimental results, it can be seen that the color gamut compression method provided by the present application has a visual effect close to that of an iterative color gamut compression algorithm, and at the same time has a processing speed close to that of a point-by-point color gamut compression algorithm, and can take into account both image quality and processing speed.

[0166] like Fig.10 The figure shows a schematic diagram of the application of a color gamut compression algorithm provided by an embodiment of the present application in an actual scenario. In the actual printing process, it is first necessary to determine the relationship between the printer device space and the connection color space. The establishment of the relationship often requires printing several test color block images through the printer. After printing, the color blocks are scanned by a scanner, and then the relationship is fitted through the scanned data to obtain the relationship between the connection color space and the printer color space. For the relationship between the display and the connection color space, it is also established in a similar way. The color gamut of the printer can be obtained by scanning the data, and a color gamut compression data set is produced according to the color gamut of the printer through an iterative color gamut compression algorithm. The deep color gamut compression model is trained using the data in the data set. After the training is completed, a deep color gamut compression operation is performed under the connection color space, and the connection color space data obtained by color conversion is data mapped and compressed into the printer color gamut. Finally, the mapped connection color space data is converted into a color space to obtain data that the printer can recognize.

[0167] Based on the deep color gamut compression method based on a lightweight convolutional network provided in the above embodiment, the present application also provides a deep color gamut compression device based on a lightweight convolutional network, such as Fig.11 As shown, the device specifically includes:

[0168] The model acquisition module 10 is used to acquire a pre-trained deep color gamut compression model; the deep color gamut compression model includes a plurality of down-sampling encoding modules with different sampling rates and a plurality of up-sampling modules with different sampling rates connected in series in a forward propagation direction; each down-sampling encoding module includes a refocused depth separable convolution unit, a refocused depth convolution unit and a pooling unit connected in series in a forward propagation direction; each up-sampling module includes an up-sampling unit, a refocused depth separable convolution unit and a refocused depth convolution unit connected in series in a forward propagation direction;

[0169] The feature extraction module 20 is used to input the image to be compressed into a plurality of down-sampling encoding modules with different sampling rates connected in series in a forward propagation direction for feature extraction, and output a target feature map;

[0170] The pixel restoration module 30 is used to input the target feature map into a plurality of up-sampling modules with different sampling rates connected in series in the forward propagation direction to perform pixel restoration and output a compressed target image.

[0171] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0172] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0173] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0174] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0175] Obviously, the above embodiments are merely examples for the purpose of clear explanation and are not intended to limit the implementation methods. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the implementation methods here. The obvious changes or modifications derived therefrom are still within the scope of protection of the present invention.

Claims

1. A deep color gamut compression method based on a lightweight convolutional network, characterized in that: include: Obtain a pre-trained deep color gamut compression model; The deep color gamut compression model includes a plurality of downsampling encoding modules with different sampling rates and a plurality of upsampling modules with different sampling rates which are sequentially connected in series along the forward propagation direction; each downsampling encoding module includes a refocused depth separable convolution unit, a refocused depth convolution unit and a pooling unit which are sequentially connected in series along the forward propagation direction; each upsampling module includes an upsampling unit, a refocused depth separable convolution unit and a refocused depth convolution unit which are sequentially connected in series along the forward propagation direction; wherein the refocused depth separable convolution unit is obtained by refocusing the parameters of the depth separable convolution block, the refocused depth convolution unit is obtained by refocusing the parameters of the channel-by-channel convolution block, and the refocused depth separable unit and the refocused depth convolution unit are used to extract deep features of the input feature map; The image to be compressed is input into a plurality of down-sampling coding modules with different sampling rates connected in series in the forward propagation direction for feature extraction, and a target feature map is output; specifically, the following steps are included: Input the image to be compressed into the refocused depth-separable convolution unit in the first downsampling encoding module for convolution, and output a first feature map; Inputting the first feature map into a refocusing depth convolution unit in a first downsampling encoding module for convolution, and outputting a second feature map; Inputting the second feature map into the pooling unit in the first downsampling encoding module for maximum pooling, and outputting a pooled feature map; Using the pooled feature map as the input of the next downsampling encoding module, and using the output of the pooling unit in the last downsampling encoding module as the target feature map; The target feature map is input into a plurality of up-sampling modules with different sampling rates connected in series in the forward propagation direction to perform pixel restoration, and a compressed target image is output; the method specifically includes: Inputting the target feature map into an upsampling unit in a first upsampling module for upsampling, and outputting a first target feature map; Inputting the first target feature map into a refocused depth-separable convolution unit in a first upsampling module for convolution, and outputting a second target feature map; Input the second target feature map into the refocusing depth convolution unit in the first upsampling module for convolution, and output a third target feature map; Using the third target feature map as the input of the next upsampling module, and using the output of the refocusing depth convolution unit in the last upsampling module as the compressed target image; Among them, the training process of the deep color gamut compression model includes: Get image samples in the training set; Input the image sample into a plurality of down-sampling encoding modules with different sampling rates connected in series in a forward propagation direction to extract features, and output a target feature map; The target feature map is input into a plurality of upsampling modules with different sampling rates connected in series in the forward propagation direction for pixel restoration, and a deep color gamut compression loss function is constructed based on the output of the last upsampling module; which specifically includes: Based on the output of the last upsampling module and the label image of the image sample, construct the brightness mean absolute error loss function, the chroma mean absolute error loss function and the hue mean absolute error loss function; Based on the brightness similarity, contrast similarity and structural similarity between the output of the last upsampling module and the label image of the image sample, a multi-scale structural similarity loss function of brightness visual perception, a multi-scale structural similarity loss function of chroma visual perception and a multi-scale structural similarity loss function of hue visual perception are constructed; Based on the brightness mean absolute value error loss function and the brightness visual perception multi-scale structural similarity loss function, construct a loss function of the image sample on the brightness channel; Based on the chroma mean absolute error loss function and the chroma visual perception multi-scale structural similarity loss function, construct a loss function of the image sample on the chroma channel; Based on the hue mean absolute value error loss function and the hue visual perception multi-scale structural similarity loss function, construct a loss function of the image sample on the hue channel; Performing weighted summation on the loss function of the image sample on the brightness channel, the loss function on the chroma channel, and the loss function on the hue channel to obtain a deep color gamut compression loss function; Iteratively train multiple down-sampling coding modules with different sampling rates and multiple up-sampling modules with different sampling rates using image samples in a training set until the value of the deep color gamut compression loss function is minimized, thereby obtaining multiple trained down-sampling coding modules with different sampling rates and multiple up-sampling modules with different sampling rates; A trained deep color gamut compression model is obtained based on a plurality of trained down-sampling encoding modules with different sampling rates and a plurality of trained up-sampling modules with different sampling rates.

2. The deep color gamut compression method based on a lightweight convolutional network according to claim 1, characterized in that: The brightness mean absolute error loss function is: ， in, represents the brightness mean absolute error loss function, The pixel representation of the target image on the brightness channel after the image sample output from the last upsampling module is compressed. The pixel representation of the label image representing the image sample on the brightness channel, The label image representing the image sample is in the brightness channel The value of the pixel, Indicates the target image after image sample compression on the brightness channel The value of the pixel, Represents the number of pixels in the image sample; The chroma mean absolute error loss function is: ， in, represents the chroma mean absolute error loss function, The pixel representation of the target image on the chroma channel after the image sample output of the last upsampling module is compressed. The pixel representation of the label image representing the image sample on the chroma channel, The label image representing the image sample is in the chroma channel. The value of the pixel, Indicates the target image after image sample compression on the chroma channel The value of the pixel; The hue mean absolute error loss function is: ， in, represents the hue mean absolute value error loss function, The pixel representation of the target image on the hue channel after the image sample output of the last upsampling module is compressed. The pixel representation of the label image representing the image sample on the hue channel, The label image representing the image sample is the first The value of the pixel, Indicates the target image after image sample compression on the hue channel The value of the pixel; The multi-scale structural similarity loss function of brightness visual perception is: ， in, Represents the multi-scale structural similarity loss function of brightness visual perception, express and The brightness similarity between To express The parameters of importance, express and In the Contrast similarity on scales, represents the scale of the image sample, To express The parameters of importance, express and In the The structural similarity at each scale is To express Parameters of importance; The multi-scale structural similarity loss function of chromaticity visual perception is: ， in, represents the multi-scale structural similarity loss function of chromaticity visual perception, express and The brightness similarity between To express The parameters of importance, express and In the Contrast similarity on scales, To express The parameters of importance, express and In the The structural similarity at each scale is To express Parameters of importance; The multi-scale structural similarity loss function of hue visual perception is: ， in, represents the multi-scale structural similarity loss function of hue visual perception, express and The brightness similarity between To express The parameters of importance, express and In the Contrast similarity on scales, To express The parameters of importance, express and In the The structural similarity at each scale is To express Parameters of importance.

3. The deep color gamut compression method based on a lightweight convolutional network according to claim 2, characterized in that: The loss function of the image sample on the brightness channel is: ， in, Represents the loss function of the image sample on the brightness channel, express The weight of represents the Gaussian weight matrix; The loss function of the image sample on the chroma channel is: ， in, Represents the loss function of the image sample on the chroma channel, express The weight of The loss function of the image sample on the hue channel is: ， in, represents the loss function of the image sample on the hue channel, express The weight of .

4. The deep color gamut compression method based on a lightweight convolutional network according to claim 3, characterized in that: The depth color gamut compression loss function is: ， in, represents the deep color gamut compression loss function, Represents the weight of the loss function of the image sample on the brightness channel, represents the weight of the loss function of the image sample on the chroma channel, Represents the weight of the loss function for image samples on the hue channel.

5. The deep color gamut compression method based on a lightweight convolutional network according to claim 1, characterized in that: The label image of each image sample in the training set is an image obtained by performing color gamut compression on the image sample using an iterative color gamut compression algorithm.

6. A deep color gamut compression device based on a lightweight convolutional network, characterized in that: include: A model acquisition module, used to acquire a pre-trained deep color gamut compression model; The deep color gamut compression model includes a plurality of downsampling encoding modules with different sampling rates and a plurality of upsampling modules with different sampling rates which are sequentially connected in series along the forward propagation direction; each downsampling encoding module includes a refocused depth separable convolution unit, a refocused depth convolution unit and a pooling unit which are sequentially connected in series along the forward propagation direction; each upsampling module includes an upsampling unit, a refocused depth separable convolution unit and a refocused depth convolution unit which are sequentially connected in series along the forward propagation direction; wherein the refocused depth separable convolution unit is obtained by refocusing the parameters of the depth separable convolution block, the refocused depth convolution unit is obtained by refocusing the parameters of the channel-by-channel convolution block, and the refocused depth separable unit and the refocused depth convolution unit are used to extract deep features of the input feature map; The feature extraction module is used to extract features from the image to be compressed input into a plurality of down-sampling encoding modules with different sampling rates connected in series in the forward propagation direction, and output a target feature map; it specifically includes: Input the image to be compressed into the refocused depth-separable convolution unit in the first downsampling encoding module for convolution, and output a first feature map; Inputting the first feature map into a refocusing depth convolution unit in a first downsampling encoding module for convolution, and outputting a second feature map; Inputting the second feature map into the pooling unit in the first downsampling encoding module for maximum pooling, and outputting a pooled feature map; Using the pooled feature map as the input of the next downsampling encoding module, and using the output of the pooling unit in the last downsampling encoding module as the target feature map; A pixel restoration module is used to input the target feature map into a plurality of up-sampling modules with different sampling rates connected in series in a forward propagation direction to perform pixel restoration, and output a compressed target image; it specifically includes: Inputting the target feature map into an upsampling unit in a first upsampling module for upsampling, and outputting a first target feature map; Inputting the first target feature map into a refocused depth-separable convolution unit in a first upsampling module for convolution, and outputting a second target feature map; Input the second target feature map into the refocusing depth convolution unit in the first upsampling module for convolution, and output a third target feature map; Using the third target feature map as the input of the next upsampling module, and using the output of the refocusing depth convolution unit in the last upsampling module as the compressed target image; Among them, the training process of the deep color gamut compression model includes: Get image samples in the training set; Input the image sample into a plurality of down-sampling encoding modules with different sampling rates connected in series in a forward propagation direction to extract features, and output a target feature map; The target feature map is input into a plurality of upsampling modules with different sampling rates connected in series in the forward propagation direction for pixel restoration, and a deep color gamut compression loss function is constructed based on the output of the last upsampling module; which specifically includes: Based on the output of the last upsampling module and the label image of the image sample, construct the brightness mean absolute error loss function, the chroma mean absolute error loss function and the hue mean absolute error loss function; Based on the brightness similarity, contrast similarity and structural similarity between the output of the last upsampling module and the label image of the image sample, a multi-scale structural similarity loss function of brightness visual perception, a multi-scale structural similarity loss function of chroma visual perception and a multi-scale structural similarity loss function of hue visual perception are constructed; Based on the brightness mean absolute value error loss function and the brightness visual perception multi-scale structural similarity loss function, construct a loss function of the image sample on the brightness channel; Based on the chroma mean absolute error loss function and the chroma visual perception multi-scale structural similarity loss function, construct a loss function of the image sample on the chroma channel; Based on the hue mean absolute value error loss function and the hue visual perception multi-scale structural similarity loss function, construct a loss function of the image sample on the hue channel; Performing weighted summation on the loss function of the image sample on the brightness channel, the loss function on the chroma channel, and the loss function on the hue channel to obtain a deep color gamut compression loss function; Iteratively train multiple down-sampling coding modules with different sampling rates and multiple up-sampling modules with different sampling rates using image samples in a training set until the value of the deep color gamut compression loss function is minimized, thereby obtaining multiple trained down-sampling coding modules with different sampling rates and multiple up-sampling modules with different sampling rates; A trained deep color gamut compression model is obtained based on a plurality of trained down-sampling encoding modules with different sampling rates and a plurality of trained up-sampling modules with different sampling rates.

Images