An image-driven method for generating harmonious color combinations

Abstract

The present invention proposes an image-driven visualization harmonious color generation method, which is characterized in that it uses a picture-driven method to automatically match colors for visualization, specifically including: selecting an image as an input for color scheme reference, performing saliency detection on the input image, and obtaining feature maps of different perception levels; cropping the original image to obtain sub-images including different image area ranges, and performing super-pixel segmentation on the sub-images as preprocessing; using a deep neural network to calculate the super-pixel segmentation results and extract candidate colors; using the calculated visual importance to assign weights to the candidate colors, and considering the visualization context, generating a final color scheme to color-code the data. Compared with the prior art, the present invention uses a picture-driven method to automatically match colors for visualization, generates a harmonious and beautiful color scheme, and supports different visualization context environments in practical applications.

Description

Technical Field

[0001] The invention relates to the technical field of color matching generation in visual design, and in particular to an image-driven visual harmonious color matching generation method. Background Art

[0002] Colors are widely used in the field of visualization to encode data. An effective color scheme contains colors that are easy to distinguish from each other and match harmoniously, so that visualization examples can accurately and beautifully convey information. For ordinary users and even professional design practitioners, designing an effective color scheme is a time-consuming and challenging process. Considering the matching of colors from the perspective of independent colors is quite complicated and requires professional knowledge. However, it is easy for people to distinguish whether an image is beautiful or not. Using images as a reference for visual style will greatly facilitate the color design of visualization. Actual visualization design is often used in different scenarios and has different contexts to consider. Pre-defined color schemes not only lack the flexibility of custom color schemes, but also ignore the usage environment of visualization examples.

[0003] The automatic generation methods in the prior art often focus more on the distinction between colors and ignore the harmony between colors. They cannot guarantee the aesthetics of the visual color design and it is difficult to provide style constraints for the generated colors. Summary of the invention

[0004] The purpose of the present invention is to design an image-driven visualization harmonious color generation method to address the deficiencies of the prior art. It uses a picture-driven approach to automatically match colors for visualization, uses human cognition and perception of images to assist visualization design, uses deep neural networks to process input images, and considers the visualization context to generate a final color scheme to color-code the data, generate a harmonious and beautiful color scheme, and support different visualization contexts in practical applications.

[0005] The object of the present invention is achieved by: an image-driven visualization harmonious color matching generation method, which is characterized in that the visualization is automatically matched in a picture-driven manner, and the generation of harmonious color matching specifically includes the following steps:

[0006] Step 1: Input the original image I as a reference for the final color matching, with a resolution of w*h, where w represents the number of pixels in each row of the image, and h represents the number of pixels in each column of the image.

[0007] Step 2: Use the saliency detection model to perform saliency detection on the original image I and generate a visual saliency feature map I Sa , represented as a grayscale image with a resolution of w*h, the grayscale value of pixel p is Grayp Indicates the visual importance of the pixel, Gray p The value range is [0,1].

[0008] Step 3: Use the following formula to identify the significant feature map I Sa The gray value of each pixel in is linearly transformed:

[0009] Gray p =(Gray p +0.5) 2 (a);

[0010] Among them: Gray p The value range is [0,1].

[0011] For the transformed salient feature map I Sa The threshold θ is separated by the following formula b to obtain the significant feature map I Sa Binarized segmented image B p :

[0012]

[0013] Among them: threshold θ = 0.1, 0.3, 0.5, 0.8.

[0014] Extract segmented image B p The minimum bounding box of the white area in the image is used to obtain the attention areas with different visual saliency levels.

[0015] Step 4: According to the regions of interest with different significance levels obtained in step 3, the original image I is divided into N sub-regions by the following formula c:

[0016] p={p0,…,p N} (c);

[0017] Where: p0 represents the original image I;

[0018] Step 5: Use the superpixel segmentation model to perform superpixel segmentation on all sub-regions according to the above formula c, and obtain the segmentation result S represented by the following formula d p :

[0019]

[0020] in: Indicates p i Superpixel segmentation results; Contains M i Segmented areas;

[0021] Step 6: Select the representative color of each segmented area in the segmentation result of step 5 to recolor the entire area. Let Sk The segmentation result The color value c of each pixel is calculated by the following formula k :

[0022]

[0023] By c k The average and median Calculate the color c of the pixel point according to the following formula f i,j :

[0024]

[0025] Among them: the values ​​of parameters α1 and α2 are calculated by the following g formula:

[0026]

[0027] Where: σ() is the covariance calculation, and the other parameters are constants: γ1=20, γ2=40, λ=1.2.

[0028] Step 7: The segmented image S after color processing in step 6 p They are input into the color encoding network respectively. The encoding network uses a 4×4 convolution kernel, a step size of 2, a border padding of 0, and LeakyReLU as the activation function. The convolution layer is followed by an adaptive average pooling layer to adjust the input images of different resolutions to a 1×1 multi-channel representation. Then, the output of the last convolution layer is connected to 3 fully connected layers, and the features are converted to 1×1×18 encoding, corresponding to the CIE-LAB values ​​of 6 colors. The dataset used for training is a paired data set of exquisite pictures crawled from professional design websites and color schemes designed by designers. There are 1,300 samples in total, 950 for training and 350 for testing. Data augmentation is applied to the training dataset for expansion.

[0029] The encoding network is trained using the following formula as the loss function:

[0030] Loss function l(P) = l mse (P,G)+ωl mse (P,I) (i);

[0031] Among them: P is the predicted result; G is the real label; I is the color range of the original image; l mse () indicates MSELoss; l mse (P, G) Calculate the difference between the predicted result and the true label; mse(P,I) is used to penalize the generated results that exceed the color range contained in the original image; the parameter ω is used to balance the weights of the two items. The encoding network extracts 6 colors from each image and combines the extraction results of all images to generate a set of candidate colors.

[0032] Step 8: Calculate the contrast between candidate colors using the following formula:

[0033]

[0034] Among them: Lumi() represents the brightness of the color.

[0035] In order to select the final K colors required, first exclude colors with a contrast ratio of less than 3 with the background color, then perform median clustering, using the contrast between colors as the distance to divide them into K categories. Using the visual saliency feature map in step 2, calculate the saliency weight ω of each candidate color, use weighted random sampling to select a sample color from each cluster, and combine them to obtain the final color scheme.

[0036] Compared with the prior art, the present invention extracts features from input images and considers the context in actual applications to adaptively generate appropriate, harmonious and beautiful color schemes to meet different visualization design requirements and application scenarios. It allows users to provide a style reference for the final generated color scheme through pictures and pays more attention to the aesthetic quality of visualization color design without affecting the visualization data representation. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 It is a schematic diagram of the process of the present invention;

[0038] Figure 2 Schematic diagram of the implementation process. DETAILED DESCRIPTION

[0039] See attached Figure 1 The present invention performs image-driven visualization and harmonious color matching generation according to the following steps:

[0040] Step 1: Input a reference image I;

[0041] Step 2: Use convolutional neural network to extract the saliency detection of I and output the visual saliency feature map I Sa ;

[0042] Step 3: I Sa Perform linear transformation and threshold segmentation to obtain regions with different visual saliency levels;

[0043] Step 4: Segment the corresponding area in the original image to obtain N sub-areas;

[0044] Step 5: Perform superpixel segmentation on each of the N sub-regions through a fully convolutional network;

[0045] Step 6: Extract the representative color of each area of ​​the superpixel segmentation result and recolor the entire area to output image S p ;

[0046] Step 7: Use the encoder model to extract the N S p Extract characteristic colors and generate a candidate color set;

[0047] Step 8: Consider the visualization context and select k target colors from the candidate color set to generate a visualization color scheme.

[0048] The present invention is further described in detail below by taking the generation of a visual color scheme from a certain image as an example.

[0049] Example 1

[0050] See attached Figure 2 The present invention performs image-driven visualization and harmonious color matching generation according to the following steps:

[0051] Step 1: Input image I as a reference for the final color scheme, with a resolution of w*h, where w represents the number of pixels in each row of the image and h represents the number of pixels in each column of the image.

[0052] Step 2: Use the saliency detection model to perform saliency detection on the input image I and generate a visual saliency feature map I Sa , represented as a grayscale image with a resolution of w*h, the grayscale value of pixel p is Gray p Indicates the visual importance of the pixel, Gray p The value range is [0,1].

[0053] Step 3: Use the following formula to identify the significant feature map I Sa The gray value of each pixel in is linearly transformed:

[0054] Gray p =(Gray p +0.5) 2 (a);

[0055] Among them: Gray p The value range of is [0,1];

[0056] Then the transformed salient feature map I is transformed by the following formula b Sa Perform threshold θ separation:

[0057]

[0058] Get I Sa Binarized segmented image B p , where the threshold θ is θ=0.1, 0.3, 0.5, 0.8, and finally the segmented image B is intercepted p The minimum bounding box of the white area in the image is used to obtain the attention areas with different visual saliency levels.

[0059] Step 4: According to the regions of interest with different significance levels obtained in step 3, the original image I is divided into N sub-regions by the following formula c:

[0060] p={p0,…,p N} (c);

[0061] Where: p0 is the original image I.

[0062] Step 5: Use the superpixel segmentation model to perform superpixel segmentation on all sub-regions according to the above formula c, and obtain the segmentation result S represented by the following formula d p :

[0063]

[0064] in, Indicates p i The superpixel segmentation result is Contains M i A segmented area.

[0065] Step 6: Select the representative color of each segmented area in the segmentation result of step 5 to recolor the entire area. Let S k The segmentation result The color value c of each pixel is calculated by the following formula k :

[0066]

[0067] By c k The average and median Calculate the color c of the pixel point according to the following formula f i,j :

[0068]

[0069] Among them: the values ​​of parameters α1 and α2 are calculated by the following g formula:

[0070]

[0071] Where: σ() is the covariance calculation, and the other parameters are constants: γ1=20, γ2=40, λ=1.2.

[0072] Step 7: Input the segmented images after color processing in step 6 into the color encoding network respectively. The encoding network uses a 4×4 convolution kernel, a step size of 2, a border padding of 0, and LeakyReLU as the activation function. The convolution layer is followed by an adaptive average pooling layer to adjust the input images of different resolutions to a 1×1 multi-channel representation. Then, the output of the last convolution layer is connected to 3 fully connected layers, and the features are converted to 1×1×18 encoding, corresponding to the CIE-LAB values ​​of 6 colors. The dataset used for training is a paired data set of beautiful pictures crawled from professional design websites and color schemes designed by designers. There are 1,300 samples in total, 950 for training and 350 for testing. Data augmentation is applied to the training dataset for expansion.

[0073] The encoding network is trained using the following formula as the loss function:

[0074] Loss function l(P) = l mse (P,G)+ωl mse (P,I) (i);

[0075] Among them: P is the predicted result; G is the real label; I is the color range of the original image; l mse () indicates MSELoss; l mse (P, G) Calculate the difference between the predicted result and the true label; mse (P,I) is used to penalize the generated results that exceed the color range contained in the original image; the parameter ω is used to balance the weights of the two items. The encoding network extracts 6 colors from each image and combines the extraction results of all images to generate a set of candidate colors.

[0076] Step 8: Calculate the contrast between candidate colors using the following formula:

[0077]

[0078] Among them: Lumi() represents the brightness of the color.

[0079] In order to select the final k colors required, first exclude colors with a contrast ratio of less than 3 with the background color, then perform median clustering, using the contrast between colors as the distance to divide them into k categories. Using the visual saliency feature map in step 2, calculate the saliency weight ω of each candidate color, use weighted random sampling to select a sample color from each cluster, and combine them to get the final color scheme.

[0080] The above is only a further explanation of the present invention and is not intended to limit this patent. Equivalent implementations that do not deviate from the spirit and scope of the present invention should be included in the scope of the claims of this patent.

Claims

1. An image-driven visual harmonious color matching generation method, characterized in that The image-driven approach is used to automatically match colors for visualization. The generation of harmonious color matching includes the following steps: Step 1: Input the original image as a reference for the final color matching, with a resolution of w*h, where w represents the number of pixels in each row of the image; h represents the number of pixels in each column of the image; Step 2: Use the saliency detection model to perform saliency detection on the original image and generate a visual saliency feature map I Sa ; Step 3: Use the following formula to identify the significant feature map I Sa The gray value of each pixel in is linearly transformed: Gray p =(Gray p +0.5) 2 (a); Among them: Gray p The value range of is [0, 1]; The transformed salient feature map I is obtained by the following formula b: Sa Separate by threshold θ to obtain the binary segmented image B p : Where: threshold θ = 0.1, 0.3, 0.5, 0.8; Extract segmented image B p The minimum bounding box of the white area in the middle, so as to obtain the focus areas with different visual saliency levels; Step 4: According to the regions of interest with different significance levels obtained in step 3, the original image is divided into N sub-regions by the following formula c: p={p0,...,p N } (c); Where: p0 is the original image I; Step 5: Use the superpixel segmentation model to perform superpixel segmentation on all sub-regions according to the above formula c, and obtain the segmentation result S represented by the following formula d p : in: Indicates p i Superpixel segmentation results; Contains M i segmented areas; Step 6: Select the representative color of each segmented area in the segmentation result of step 5 to recolor the entire area. Let S k The segmentation result The color value c of each pixel is calculated by the following formula k : By c k The average and median Calculate the color c of the pixel point according to the following formula f i，j : Among them: the values ​​of parameters α1 and α2 are calculated by the following g formula: Among them: σ() is the covariance calculation, and the other parameters are constants: γ1=20, γ2=40, λ=1.2; Step 7: Input the segmented images after color processing in step 6 into the color encoding network respectively, extract 6 colors from each image, and combine the extraction results of all images to generate a candidate color set; Step 8: Calculate the contrast between candidate colors using the following formula: Among them: Lumi() represents the brightness of the color; Step 9: Adjust the Contrast The color is removed, and then median clustering is performed, using the contrast between colors as the distance to divide them into k categories, using the visual saliency feature map I in step 2 Sa , calculate the significance weight ω of each candidate color, select a sample color from each cluster using weighted random sampling, and combine them to get the final color scheme; The color coding network in step 7 uses a convolutional layer to extract characteristic colors from the image, and the training of the convolutional layer specifically includes the following steps: Step a: Using a 4×4 convolution kernel, a stride of 2, a border padding of 0, and LeakyReLU as the activation function, the convolution layer is followed by an adaptive average pooling layer to adjust the input images of different resolutions to a 1×1 multi-channel representation. Then, the output of the last convolution layer is connected to three fully connected layers, and the features are converted into 1×1×18 encoding, corresponding to the CIE-LAB values ​​of 6 colors; Step b: Crawl the exquisite pictures of professional design websites and the paired data sets of color schemes as the data sets for training and testing, and apply data enhancement to expand the training data set; Step c: Use the following formula i as the loss function to evaluate the prediction effect and train the encoding network: Loss function l(P) = l mse (P,G)+ωl mse (P, I) (i); Among them: P is the predicted result; G is the real label; I is the color range of the original image; l mse () indicates MSE Loss; l mse (P, G) Calculate the difference between the predicted result and the true label; l mse (P, I) is used to penalize the generated results that exceed the color range contained in the original image; the parameter ω is used to balance the weights of the two items.

Images