A color image color transfer method based on saliency features

Abstract

A color transfer method for color images based on saliency features is disclosed. The method obtains saliency feature maps of the original image and a reference image, respectively. Based on thresholding of the saliency feature maps, the original image is divided into saliency regions and non-saliency regions, and the reference image is also divided into saliency regions and non-saliency regions. Within the saliency regions and non-saliency regions, color information from the reference image is transferred to the input image based on a weighted average of variance and saliency values, respectively. This invention can further improve the color transfer effect of color images.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, and relates to color transfer of color images, and particularly to a color transfer method for color images based on salient features. Background Technology

[0002] Image processing technology has gained widespread attention in many application fields and achieved significant pioneering achievements, making it a remarkable and promising new discipline. One of the most common problems in image processing is changing the colors in an image, now known as color transfer and style transfer. Image color transfer transfers color information from a reference image to an input image while preserving the original image's texture information, resulting in an output image with similar color statistics to the reference image. For example, color transfer can be used to convert red areas in the original image to blue areas based on a reference image, and the output image will display a color pattern similar to the reference image. Photographs contain rich and well-structured visual information. In human visual perception, edges are effective and expressive stimuli, crucial for neural interpretation to optimally perceive scenes. High-level reasoning regarding salient structures is highly valued in manipulating and understanding images. Early color transfer methods relied heavily on manual adjustments by the user, requiring high time complexity and extensive user interaction. In recent years, much effort has been made to address these issues, yielding significant results. Examples include color transfer based on statistical information (such as color mean and variance), and user-assisted color transfer methods such as doodles and samples. Deep learning-based color transfer methods emerge by using deep learning models to learn the relationship between source and target images. While this type of algorithm has good transfer performance, it requires a relatively large dataset, the learning network is more complex, and the entire process is time-consuming.

[0003] Significant progress has been made in the fields of image color transfer and style transfer to date. However, some problems still need to be solved, such as unnatural output results, long transfer times, and region blending issues. Summary of the Invention

[0004] In order to overcome the shortcomings of the prior art, the present invention aims to provide a color image color transfer method based on salient features, so as to further improve the effect of color image color transfer.

[0005] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0006] A color transfer method for color images based on salient features includes the following steps:

[0007] Step 1: Obtain the saliency feature maps M of the original images. sw Saliency feature map M of the reference image rw The original image refers to the one that needs to preserve texture and accepts the colors of the reference image; the reference image refers to the one that provides color information to the original image.

[0008] Step 2: Based on the threshold processing of the saliency feature map, the original image is divided into saliency regions and non-saliency regions, and the reference image is also divided into saliency regions and non-saliency regions.

[0009] Step 3: In salient and non-salient regions, the color information of the reference image is transferred to the input image based on the weighted average of the variance and saliency values, respectively.

[0010] Compared with the prior art, the beneficial effects of the present invention are:

[0011] First, in obtaining the saliency feature map, this invention employs a saliency value method based on histogram contrast. This improves computational speed by reducing the number of colors to consider, and a color smoothing term is added to enhance the saliency map's effectiveness. Second, addressing the color region mixing problem during color transfer, this invention uses a region segmentation method, dividing the original and reference images into saliency and non-saliency regions. This ensures that color transfer between image regions occurs between regions with similar visual attention, resulting in a better visual effect in the output. Finally, during the color transfer process in corresponding regions, this invention uses the saliency values ​​of the corresponding regions in the input image as weights to calculate the mean, increasing the influence of important parts, focusing attention on prominent areas, and reducing the influence of unimportant parts. Attached Figure Description

[0012] Figure 1 This is a schematic diagram of the overall process of the present invention.

[0013] Figure 2 It is the saliency feature map of the input image.

[0014] Figure 3 This is the original image.

[0015] Figure 4 This is a reference image.

[0016] Figure 5 This is a diagram showing the color transfer result. Detailed Implementation

[0017] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings and examples.

[0018] This invention is a color transfer method for color images based on salient features, such as... Figure 1As shown, the present invention is mainly divided into three parts: the first part is the part for obtaining the salient feature map of the input image, the second part is the part for dividing the color transfer region, and the third part is the part for color image color transfer.

[0019] The first part calculates the contrast of the original image and the reference image to obtain saliency feature maps; the second part divides the original image and the reference image into saliency regions and non-saliency regions respectively based on the threshold processing of the saliency feature maps; the third part performs color transfer in the saliency regions and non-saliency regions respectively based on the weighted average of variance and saliency values.

[0020] In this invention, the original image refers to the one that needs to retain texture and accepts the color of the reference image; the reference image refers to the one that provides color information to the original image.

[0021] The following is a detailed description of each part.

[0022] The first part uses color statistics from the original and reference images to define the saliency value of a pixel. The saliency of a pixel is defined using its color contrast with all other pixels in the image, i.e., pixel I in image I. k The significance value is defined as:

[0023]

[0024] D(I k ,I i )=||I k -I i ||

[0025] Where D(I) k ,I i ) is pixel I in CIEL*a*b* k and I i The color distance metric between pixels is N, where N is the total number of pixels in image I. However, under this definition, pixels with the same color value have the same saliency value because this metric ignores spatial relationships. Therefore, items with the same color value are grouped into one type, resulting in n types. The saliency value for each color is then calculated as follows:

[0026]

[0027] Where f j The pixel color x in image I j The probability of . A simple evaluation of the above equation shows that the significance value of each image requires o(N) + o(n). 2 The computational time required is too long, so the key to speeding up the process is to reduce the number of pixel colors in the image.

[0028] To reduce the number of colors to consider, each color channel in the RGB color space is first quantized from 256 to have 12 distinct values, reducing the number of colors to 12. 3 =1728. Considering that colors in natural images typically only cover a small portion of the entire color space, the number of colors is further reduced by ignoring less frequent colors. More frequent colors are selected, ensuring they cover more than 95% of the image pixels, with less than 5% of pixels replaced by the closest color in the histogram.

[0029] However, quantization itself can introduce artifacts, and some similar colors may be quantized to different values. To reduce the noisy significance results caused by this randomness, a smoothing procedure is used to refine the significance value of each color, replacing the significance value of each color with a weighted average of the significance values ​​of similar colors.

[0030]

[0031] Based on the above formula, if the input image is the original image, then the saliency feature map M of the original image is obtained. sw If a reference image is selected as the input image, the saliency feature map M of the reference image will be obtained. rw That is, M'(x) represents M sw Or M rw x represents the color that needs to be refined, and m represents the number of closest colors selected for refining color x. i Let x represent the i-th color among the m nearest neighbor colors, and T represent the sum of the distances between color x and its m nearest neighbor colors.

[0032] In the formula, D(x,x i ) represents x and x i The Euclidean distance between them, M(x) i The saliency value of color x is represented by TD(x, x). Typically, the m = n / 4 closest colors are selected to refine the saliency value of color x. A linearly varying smooth weight TD(x, x) is used. i Assign greater weight to colors closer to x in the color feature space. Quantize in RGB space while simultaneously measuring distance in Lab color space. Figure 2 It is the salient feature map obtained from the input image.

[0033] The second part involves obtaining the saliency feature map M of the reference image. rw Saliency feature map M of the original image sw Then, the threshold θ of the original image is obtained according to the following formula. s Threshold θ of the reference image r :

[0034]

[0035]

[0036] Where N s N represents the number of pixels in the original image. r Given the number of pixels in the reference image, the saliency value is thresholded to segment the pixels in both the reference and original images into salient and insignificant pixels, respectively. The region consisting of salient pixels is the salient region, and the region consisting of insignificant pixels is the insignificant region. The formula is as follows:

[0037] I ry =I r (M rw ≥θ r )

[0038] I rn =I r (M rw <θ r )

[0039] I sy =I s (M sw ≥θ s )

[0040] I sn =I s (M sw <θ s )

[0041] Among them I ry It is the salient region of the reference image, I rn It is a non-salient region of the reference image, I sy It is the salient region of the original image, I sn It is a non-salient region of the original image.

[0042] The meaning of the above formulas:

[0043] I r M represents the pixel value of the reference image. rw ≥θ r This indicates the condition for classifying pixels in a reference image as salient regions. The salient feature value of the reference image is used to define the salient regions. When this condition is met, the pixel values ​​of the reference image can be classified as salient regions. rw <θ r The condition for classifying the pixels of a reference image as a non-salient region is the salient feature value of the reference image. When the condition is met, the pixel values ​​of the reference image can be classified as non-salient regions.

[0044] I s M represents the pixel values ​​of the original image. sw ≥θ s This refers to the condition for classifying the pixels of the original image into salient regions. The salient feature value of the reference image is used as the salient value; when this condition is met, the pixel values ​​of the reference image can be classified as salient regions. M sw <θ s The condition for classifying the pixels of the original image into non-salient regions is represented by the salient feature value of the reference image. When the condition is met, the pixel values ​​of the reference image can be classified into non-salient regions.

[0045] In the third part, in both salient and non-salient regions, the color information of the reference image is transferred to the input image based on a weighted average of the variance and saliency values, respectively. Ultimately, this invention can transfer the color information of the reference image to the input image while preserving the original image's texture information, resulting in an output image with color statistical information similar to the reference image.

[0046] Specifically, in the YUV color space, the standard deviation of each channel in each region of the input image is calculated, and the saliency values ​​of the input image are normalized according to the regions:

[0047]

[0048]

[0049] The input images here include the original image and the reference image. Specifically, W s (i) represents the weight of the saliency value of the original image at the i-th pixel, M sw (i) represents the saliency value of the saliency feature map of the original image at the i-th pixel, min(M sw ) represents the minimum saliency value of the original image, max(M) sw W represents the maximum saliency value of the original image; r (i) represents the weight of the saliency value of the reference image at the i-th pixel, M rw (i) represents the saliency value of the saliency feature map of the reference image at the i-th pixel, min(M rw ) represents the minimum saliency value of the reference image, max(M) rw () represents the maximum saliency value of the reference image.

[0050] The salient regions of the output image are calculated as follows:

[0051] Calculate the weighted average of the significant regions using the normalized significance values ​​according to the following formula:

[0052]

[0053]

[0054] in This represents the weighted average of the salient regions of the original image at the i-th pixel. This represents the pixel value in channel c at the i-th pixel location in the salient region of the original image. This represents the weighted average of the salient regions of the reference image at the i-th pixel. The pixel value in channel c at the i-th pixel represents the salient region of the reference image.

[0055] N sy N represents the total number of pixels in the salient regions of the original image. ry The total number of pixels in the salient region of the reference image is replaced by a weighted average value instead of the mean value in the original Reinhard method, as shown in the following formula:

[0056]

[0057] in This represents the pixel value in channel c at the i-th pixel in the salient region of the output image, where c represents the channel and c∈{Y,U,V}. This represents the standard deviation of each channel in the salient region of the reference image. This represents the standard deviation of each channel in the salient region of the original image.

[0058] Similarly, the non-salient regions of the output image are calculated as follows:

[0059] The weighted average of the non-significant regions is calculated using the normalized significance values ​​according to the following formula:

[0060]

[0061]

[0062] in This represents the weighted average of the non-salient regions in the original image at the i-th pixel. This represents the pixel value in channel c at the i-th pixel location in the non-salient region of the original image. This represents the weighted average of the non-salient regions of the reference image at the i-th pixel. This represents the pixel value in channel c at the i-th pixel location in the non-salient region of the reference image.

[0063] N sn N represents the total number of pixels in the non-salient regions of the original image.rn The total number of pixels in the non-salient regions of the reference image is replaced by a weighted average value, as shown in the following formula:

[0064]

[0065] in This represents the pixel value in channel c at the i-th pixel location in the non-salient region of the output image. This represents the standard deviation of each channel in the non-significant region of the reference image. The standard deviation of each channel in the salient region of the original image;

[0066] The final result, obtained by adding the results for the salient regions and the results for the non-salient regions, is the output image.

[0067]

[0068] This increases the impact of important parts, focusing attention on prominent areas and reducing the influence of unimportant parts. It also allows the output image to produce colors similar to those in the reference image while preserving the structure of objects in the original image.

[0069] In one specific embodiment of the present invention, the kitten is the original image as shown below. Figure 3 As shown, the fox is the reference image. Figure 4 As shown, the color transfer result obtained by the method of the present invention is as follows: Figure 5 As shown, it can be seen that the present invention significantly speeds up the entire process due to the improvement in color quantization in the saliency map acquisition stage, and the improvement on the Reinhard method results in a better and more natural color transfer map of a color image.

Claims

1. A color transfer method for color images based on salient features, characterized in that, Includes the following steps: Step 1: Obtain the saliency feature maps M of the original images. sw Saliency feature map M of the reference image rw The original image refers to the one that needs to preserve texture and accepts the colors of the reference image; the reference image refers to the one that provides color information to the original image. Step 2: Based on the threshold processing of the saliency feature map, the original image is divided into saliency regions and non-saliency regions, and the reference image is also divided into saliency regions and non-saliency regions. Step 3: In salient and non-salient regions, the color information of the reference image is transferred to the input image based on the weighted average of variance and significance values, respectively. The implementation method is as follows: In the YUV color space, the standard deviation of each channel in each region of the input image is calculated, and the saliency values ​​of the input image are normalized according to the regions: The input image includes the original image and the reference image; W s (i) represents the weight of the saliency value of the original image at the i-th pixel, M sw (i) represents the saliency value of the saliency feature map of the original image at the i-th pixel, min(M sw ) represents the minimum saliency value of the original image, max(M) sw W represents the maximum saliency value of the original image; r (i) represents the weight of the saliency value of the reference image at the i-th pixel, M rw (i) represents the saliency value of the saliency feature map of the reference image at the i-th pixel, min(M rw ) represents the minimum saliency value of the reference image, max(M) rw () represents the maximum saliency value of the reference image; The salient regions of the output image are calculated as follows: Calculate the weighted average of the significant regions using the normalized significance values ​​according to the following formula: in This represents the weighted average of the salient regions of the original image at the i-th pixel. This represents the pixel value in channel c at the i-th pixel location in the salient region of the original image. This represents the weighted average of the salient regions of the reference image at the i-th pixel. The pixel value in channel c at the i-th pixel represents the salient region of the reference image. N sy N represents the total number of pixels in the salient regions of the original image. ry The total number of pixels in the salient region of the reference image is replaced by a weighted average value instead of the mean value in the original Reinhard method, as shown in the following formula: in This represents the pixel value in channel c at the i-th pixel location in the salient region of the output image. This represents the standard deviation of each channel in the salient region of the reference image. The standard deviation of each channel in the salient region of the original image; The non-salient regions of the output image are calculated as follows: The weighted average of the non-significant regions is calculated using the normalized significance values ​​according to the following formula: in This represents the weighted average of the non-salient regions in the original image at the i-th pixel. This represents the pixel value in channel c at the i-th pixel location in the non-salient region of the original image. This represents the weighted average of the non-salient regions of the reference image at the i-th pixel. This represents the pixel value in channel c at the i-th pixel location in the non-salient region of the reference image. N sn N represents the total number of pixels in the non-salient regions of the original image. rn The total number of pixels in the non-salient regions of the reference image is replaced by a weighted average value, as shown in the following formula: in This represents the pixel value in channel c at the i-th pixel location in the non-salient region of the output image. This represents the standard deviation of each channel in the non-significant region of the reference image. The standard deviation of each channel in the salient region of the original image; The final result, obtained by adding the results for the salient regions and the results for the non-salient regions, is the output image.

2. The color transfer method for color images based on salient features according to claim 1, characterized in that, In step 1, the saliency feature map M of the original image sw Saliency feature map M of the reference image rw All are calculated using the following formula: M'(x) represents M sw Or M rw x represents the color that needs to be refined, and m represents the number of closest colors selected for refining color x. i Let x represent the i-th color among the m nearest neighbors of color x, and T represent the sum of the distances between color x and its m nearest neighbors. D(x,x i ) represents x and x i The Euclidean distance between them, M(x) i ) represents the saliency value of each color in the image.

3. The color transfer method for color images based on salient features according to claim 1, characterized in that, In step 2, the threshold θ of the original image is obtained according to the following formula. s Threshold θ of the reference image r : Where N s N represents the number of pixels in the original image. r The number of pixels in the reference image; The pixels in the reference image and the original image are segmented into salient pixels and insalient pixels, respectively. The region composed of salient pixels is the salient region, and the region composed of insalient pixels is the insalient region, as shown in the following formula: I ry =I r (M rw ≥θ r ) I rn =I r (M rw <θ r ) I sy =I s (M sw ≥θ s ) I sn =I s (M sw <θ s ) Among them I ry It is the salient region of the reference image, I rn It is a non-salient region of the reference image, I sy It is the salient region of the original image, I sn These are non-salient regions of the original image; I r M represents the pixel value of the reference image. rw ≥θ r This indicates the condition for classifying pixels in a reference image as salient regions. The salient feature value of the reference image is used to define the salient regions. When this condition is met, the pixel values ​​of the reference image can be classified as salient regions. rw <θ r The condition for classifying the pixels of a reference image as a non-salient region is the salient feature value of the reference image. When the condition is met, the pixel values ​​of the reference image can be classified as non-salient regions. I s M represents the pixel values ​​of the original image. sw ≥θ s This refers to the condition for classifying the pixels of the original image into salient regions. The salient feature value of the reference image is used as the salient value; when this condition is met, the pixel values ​​of the reference image can be classified as salient regions. M sw <θ s The condition for classifying the pixels of the original image into non-salient regions is represented by the salient feature value of the reference image. When the condition is met, the pixel values ​​of the reference image can be classified into non-salient regions.

Images