Image Color Transfer Method, Device, Electronic Device, and Storage Medium

Abstract

The present invention provides an image color transfer method, apparatus, electronic device, and storage medium. The method includes: obtaining a source image and a target color palette; constructing a color space convex hull based on the color component values of each pixel point in the source image; determining a source color palette of the source image based on the color space convex hull; determining a coefficient matrix corresponding to the source color palette based on the relationship between the color component values of each pixel point and the color space convex hull; and performing color transfer on the source image based on the source color palette, the coefficient matrix corresponding to the source color palette, and the target color palette. The image color transfer method, apparatus, electronic device, and storage medium provided by the present invention enable the transferred image to retain the color of each pixel point, solve the problems of rough color and unnatural transition in the color transfer image, and improve the quality of the color transfer image.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular, to an image color transfer method, apparatus, electronic device, and storage medium. Background Art

[0002] For the requirements of personalized recommendation of Internet advertisements, it is usually necessary to modify the color matching of a large number of advertising creative materials for various placement environments. Image color transfer is a feasible solution.

[0003] Image color transfer means that given an image to be transferred and a color transfer target, the image to be transferred has a specified color style while retaining its own image content.

[0004] Existing image color transfer methods are mostly implemented based on automatic clustering algorithms, which usually cause large information loss, resulting in relatively rough color granularity of the image after color transfer and unnatural color transition positions. Summary of the Invention

[0005] The present invention provides an image color transfer method, apparatus, electronic device, and storage medium to solve the defect that the color granularity of the image after color transfer in the prior art is relatively rough and the color transition position is unnatural.

[0006] The present invention provides an image color transfer method, including:

[0007] Obtain a source image and a target color palette;

[0008] Construct a color space convex hull based on the color component values of each pixel point in the source image;

[0009] Determine the source color palette of the source image based on the color space convex hull;

[0010] Determine a coefficient matrix corresponding to the source color palette based on the relationship between the color component values of each pixel point and the color space convex hull;

[0011] Perform color transfer on the source image based on the source color palette, the coefficient matrix corresponding to the source color palette, and the target color palette.

[0012] According to the image color transfer method provided by the present invention, the determining the coefficient matrix corresponding to the source color palette based on the relationship between the color component values of each pixel point and the color space convex hull includes:

[0013] Determine the coefficient of each pixel point relative to the source color palette based on the linear relationship between the color component values of each pixel point and the color component values represented by the color space convex hull, where the source color palette corresponds to the color component values represented by the color space convex hull;

[0014] Determine the coefficient matrix corresponding to the source color palette based on the coefficients of each pixel point relative to the source color palette.

[0015] According to the image color transfer method provided by the present invention, the determining the source color palette of the source image based on the convex hull of the color space includes:

[0016] Determine the initial vertex set of the convex hull of the color space;

[0017] Perform vertex merging on the initial vertex set until a merging termination condition is reached, where the merging termination condition is determined based on the merging error of the vertices to be merged and / or the number of vertices after merging;

[0018] Determine the source color palette of the source image based on the vertex set of the convex hull of the color space after vertex merging.

[0019] According to the image color transfer method provided by the present invention, the performing color transfer on the source image based on the source color palette, the coefficient matrix corresponding to the source color palette, and the target color palette includes:

[0020] Convert the source color palette and the target color palette to the target color space;

[0021] Transfer each color component other than the luminance component in the target color palette to the source color palette respectively to obtain a transferred color palette; convert the transferred color palette from the target color space to the initial color space;

[0022] Determine the target image based on the transferred color palette in the initial color space and the coefficient matrix corresponding to the source color palette.

[0023] According to the image color transfer method provided by the present invention, the transferring each color component other than the luminance component in the target color palette to the source color palette respectively to obtain a transferred color palette includes:

[0024] Transfer each color component other than the luminance component in the target color palette to the source color palette in sequence according to the importance sorting order of the source color palette and the order of the target color palette to obtain a transferred color palette;

[0025] The importance sorting order of the source color palette is determined based on the coefficient matrix corresponding to the source color palette.

[0026] According to the image color transfer method provided by the present invention, the target color palette is obtained based on the following steps:

[0027] Obtain a reference image and determine the color palette of the reference image;

[0028] Determine the color palette of the reference image as the target color palette.

[0029] According to the image color transfer method provided by the present invention, the target color palette is obtained based on the following steps:

[0030] Obtain the number of source images and target images;

[0031] Convert the color palette of the source image to the target color space;

[0032] In the target color space, sample the color palette of the source image in each dimension except the brightness dimension by the number of target images to obtain the number of sets of sampled color palettes of the target image;

[0033] Determine the color palette obtained by converting the number of sets of sampled color palettes of the target image from the target color space to the initial color space as the target color palette.

[0034] The present invention also provides an image color transfer device, including:

[0035] An image acquisition unit for acquiring a source image and a target color palette;

[0036] A convex hull construction unit for constructing a color space convex hull based on the color component values of each pixel point in the source image;

[0037] A source color palette determination unit for determining the source color palette of the source image based on the color space convex hull;

[0038] A coefficient matrix determination unit for determining the coefficient matrix corresponding to the source color palette based on the relationship between the color component values of each pixel point and the color space convex hull;

[0039] A color transfer unit for performing color transfer on the source image based on the source color palette, the coefficient matrix corresponding to the source color palette, and the target color palette.

[0040] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, the steps of any of the above-mentioned image color transfer methods are implemented.

[0041] The present invention also provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned image color transfer methods are implemented.

[0042] The present invention also provides a computer program product, including a computer program. When the computer program is executed by a processor, the steps of any of the above-mentioned image color transfer methods are implemented.

[0043] The image color transfer method, device, electronic device, and storage medium provided by the present invention first determine the source color palette of the source image based on the constructed color space convex hull. According to the relationship between the color component values of each pixel point and the color space convex hull, the coefficient matrix corresponding to the source color palette is determined, so that the color of each pixel point in the source image can be accurately represented by the source color palette, and the deviation between the color of each pixel point and the color shown by the source color palette is retained, without causing information loss. Then, based on the source color palette, the coefficient matrix corresponding to the source color palette, and the target color palette, color transfer is performed on the source image. The resulting transferred image retains the deviation between the color of each pixel point and the color shown by the source color palette, solves the problem of rough color and unnatural transition in the color transfer image, and improves the quality of the color transfer image. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] To more clearly illustrate the technical solutions in the present invention or the prior art, the following will briefly introduce the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings in the following description are some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0045] Figure 1 is one of the flow diagrams of the image color transfer method provided by the present invention;

[0046] Figure 2 is the flow diagram of step 140 in the image color transfer method provided by the present invention;

[0047] Figure 3 is a schematic diagram of the convex hull theory;

[0048] Figure 4 is the flow diagram of step 130 in the image color transfer method provided by the present invention;

[0049] Figure 5 is the flow diagram of step 150 in the image color transfer method provided by the present invention;

[0050] Figure 6 is one of the flow diagrams of the target color palette acquisition method provided by the present invention;

[0051] Figure 7 is the second flow diagram of the target color palette acquisition method provided by the present invention;

[0052] Figure 8 is the second flow diagram of the image color transfer method provided by the present invention;

[0053] Figure 9 is the structural diagram of the image color transfer device provided by the present invention;

[0054] Figure 10 It is a schematic structural diagram of the electronic device provided by the present invention. Specific embodiments

[0055] In order to make the objectives, technical solutions and advantages of the present invention clearer, the technical solutions in the present invention will be clearly and completely described below with reference to the accompanying drawings in the present invention. Apparently, the described embodiments are some, but not all, of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in the present invention without making creative efforts shall fall within the protection scope of the present invention.

[0056] The general process of image color transfer is to first extract the image color palette and the corresponding coefficient matrix. Given the color palette P and the coefficient matrix W of the image, the image I = WP can be obtained, that is, image reconstruction. When performing color transfer, the color palette is changed to the specified color P target , and then the target image can be obtained by restoring the image through the coefficient matrix: I target = WP target .

[0057] The existing image color transfer methods are mostly as follows: First, input the image to be processed, then extract several main colors of the image as the color palette of the image based on a certain automatic clustering algorithm, determine the coefficient matrix according to the clustering center to which each pixel point belongs, and finally change the color palette according to the task objective of color transfer and perform image reconstruction to obtain the target image.

[0058] The target image obtained thereby usually causes relatively large information loss. The reason is that the coefficient matrix determined according to the clustering center usually only assigns pixel points to a certain clustering center, resulting in the reconstructed image having only several colors on the color palette. Specifically, the color granularity of the image after color transfer is relatively rough and the color transition positions are not natural.

[0059] In view of this, how to improve the image color transfer method to improve the quality of the image after color transfer is an urgent problem to be solved in the field of image processing.

[0060] Figure 1 It is a schematic flow diagram of the image color transfer method provided by the present invention. As Figure 1 shown, the image color transfer method provided by the present invention can be applied to various common image color transfer scenarios, such as the color matching modification scenario of advertising creative materials or the digital image restoration scenario, etc. The method includes the following steps:

[0061] Step 110, obtain the source image and the target color palette.

[0062] Specifically, the source image refers to the image to be subjected to color transfer. For example, the source image can be a landscape image, a portrait image, etc. captured by an image acquisition device, or a colored image drawn through an image editing software. Of course, it can also be a specified image selected by the user from the cloud or a database for color transfer. The embodiments of the present invention do not make specific limitations thereto.

[0063] The target color palette refers to the color palette used for color transfer processing. For example, the target color palette can be a color palette specified by a designer with a specific color matching scheme, or a color palette with a gradient color theme or a warm and cold color theme obtained from the cloud or a database. Of course, the target color palette can also be obtained by extracting the color palette from any image. The embodiments of the present invention do not make specific limitations thereto. The number of colors in the target color palette can be one or more, and the embodiments of the present invention do not make specific limitations thereto. For example, the target color palette can be a user-specified target color palette represented in hexadecimal RGB, such as: 056A6B, 992F88, FFFFFF, 987654.

[0064] During an image color transfer process, color transfer is performed based on the source image and the target color palette, and a new image is output. The new image has the same structural information and shape information as the source image, and at the same time has color information similar to or the same as the target color palette.

[0065] Step 120: Construct a color space convex hull based on the color component values of each pixel point in the source image.

[0066] Specifically, the source image can be an image in any color space. A color space refers to a system or method for digitally describing colors through a corresponding mathematical model. For example, the color space can be the RGB color space (the RGB color space is a color model composed of a red channel R, a green channel G, and a blue channel B), or the HSV color space (the HSV color space is a color model composed of hue, saturation, and value). Of course, it can also be other types of hybrid color spaces or intensity / saturation / hue color spaces, such as the CMY / CMYK color space, the HIS / HSL color space, etc. The embodiments of the present invention do not make specific limitations thereto.

[0067] The color component values of each pixel point correspond to different values according to different color spaces. Taking the RGB color space as an example, the color component values include: a first component value corresponding to red, a second component value corresponding to green, and a third component value corresponding to blue. Another example is the HSV color space, where the color component values include: a first component value corresponding to chroma, a second component value corresponding to saturation, and a third component value corresponding to brightness.

[0068] It should be noted that a color space convex hull is constructed according to the color component values of each pixel point in the source image, where the color component values of each pixel point should be the corresponding color component values in the same color space.

[0069] For example, if the color component values of any pixel point on the source image in the RGB color space are (r i , g i , b i ), and this color component value can correspond to a point in three-dimensional space, then an RGB color space convex hull can be constructed according to the color component values (r i , g i , b i ) corresponding to all pixel points on the source image in the RGB color space. This color space convex hull covers the color component values corresponding to all pixel points on the source image.

[0070] Step 130: Based on the color space convex hull, determine the source color palette of the source image.

[0071] Specifically, since this color space convex hull covers the color component values corresponding to all pixel points on the source image, according to the convex hull theory, any point inside the convex hull can be linearly represented by the vertices of the convex hull. Therefore, the vertices of the color space convex hull can be used as the source color palette of the source image, and the colors of all pixel points on the source image except the source color palette (i.e., any point inside the convex hull) can be linearly represented by the source color palette.

[0072] For example, the vertices of the color space convex hull can be used as the source color palette of the source image, and there can be a one-to-one correspondence between the source color palette and each vertex. The coordinate values corresponding to each vertex are the color component values of the source color palette. In some cases where the number of vertices is large, to reduce the computational amount, multiple vertices can also correspond to one source color palette. For example, the average value of the coordinate values corresponding to multiple vertices is used as the color component value of the source color palette.

[0073] Step 140: Based on the relationship between the color component values of each pixel point and the color space convex hull, determine the coefficient matrix corresponding to the source color palette.

[0074] Specifically, the coefficient matrix corresponding to the source color palette refers to the coefficient matrix that can restore the colors of each pixel point on the source image according to the source color palette, and it can represent the weights of the colors in the source color palette on the color component values of each pixel point.

[0075] Since any point within the convex hull can be linearly represented by the vertices of the convex hull, the coefficient matrix corresponding to the source color palette can be solved based on the linear relationship between the color component values of each pixel and the convex hull of the color space. For example, any point within the convex hull can represent the color component values of each pixel, and the vertices of the convex hull can represent the source color palette, then the linear relationship between the color component values of each pixel and the source color palette, that is, the coefficient matrix corresponding to the source color palette, can be solved.

[0076] Compared with the traditional method of extracting several main colors of an image as the source color palette of the image based on a clustering algorithm and determining the coefficient matrix according to the cluster center to which each pixel belongs, the method provided by the embodiments of the present invention determines the source color palette of the source image based on the convex hull of the color space, and determines the coefficient matrix corresponding to the source color palette according to the relationship between the color component values of each pixel and the convex hull of the color space. The colors of each pixel can be accurately represented by the source color palette and the coefficients corresponding to the source color palette. This method retains the deviation between the colors of each pixel on the source image and the colors shown by the source color palette and will not cause information loss.

[0077] Step 150, perform color transfer on the source image based on the source color palette, the coefficient matrix corresponding to the source color palette, and the target color palette.

[0078] Specifically, when the source color palette, the coefficient matrix corresponding to the source color palette, and the target color palette are obtained, color transfer can be performed on the source image. During color transfer, the source color palette can be changed to the target color palette with a specified color, and then the image can be restored through the coefficient matrix corresponding to the source color palette to obtain the color-transferred image, so that the color-transferred image has the same structural information and shape information as the source image, and at the same time has color information similar to or the same as the target color palette.

[0079] During color transfer, changing the source color palette to the target color palette with a specified color can be changing the color palette in the initial color space corresponding to the source color palette; or the source color palette and the target color palette can be both subjected to color space conversion and simultaneously converted to the target color space. For example, the RGB color space can be converted to the HSV color space, and then the color palette is changed in the target color space. The embodiments of the present invention do not make specific limitations on this.

[0080] The image color transfer method provided by the embodiment of the present invention first determines the source color palette of the source image based on the constructed color space convex hull, and determines the coefficient matrix corresponding to the source color palette according to the relationship between the color component values of each pixel point and the color space convex hull, so that the color of each pixel point in the source image can be accurately represented by the source color palette, and the deviation between the color of each pixel point and the color shown in the source color palette is retained, without causing information loss; then, based on the source color palette, the coefficient matrix corresponding to the source color palette, and the target color palette, color transfer is performed on the source image, and the resulting transferred image retains the deviation between the color of each pixel point and the color shown in the source color palette, solving the problem that the color of the color transferred image is rough and the transition is not natural, and improving the quality of the color transferred image.

[0081] Based on the above embodiment, Figure 2 is a schematic flowchart of step 140 in the image color transfer method provided by the present invention, as Figure 2 shown, step 140 specifically includes:

[0082] Step 141, determine the coefficient of each pixel point relative to the source color palette based on the linear relationship between the color component value of each pixel point and the color component value represented by the color space convex hull, and the source color palette corresponds to the color component value represented by the color space convex hull;

[0083] Step 142, determine the coefficient matrix corresponding to the source color palette based on the coefficient of each pixel point relative to the source color palette.

[0084] Specifically, the linear relationship between the color component value of each pixel point and the color component value represented by the color space convex hull can be solved based on the convex hull theory. It can be known from the convex hull theory that any point inside the convex hull can be linearly represented by the vertices of the convex hull. Since the source color palette corresponds to the color component value represented by the color space convex hull, the linear relationship between the color component value of each pixel point and the color component value represented by the color space convex hull can be solved according to the convex hull theory to obtain the coefficient of each pixel point relative to the source color palette.

[0085] It can be understood that the coefficient of each pixel point relative to the source color palette can represent the weight of the source color palette at this pixel point, that is, adding the color component values represented by the source color palette with this coefficient as the weight can obtain the color component value of this pixel point.

[0086] After obtaining the coefficient of each pixel point relative to the source color palette, the coefficient matrix corresponding to the source color palette can be obtained according to the matrix composed of the coefficients of each pixel point relative to the source color palette.

[0087] Based on the above embodiment, the method for determining the source color palette and the coefficient matrix corresponding to the source color palette is as follows:

[0088] Figure 3It is a schematic diagram of the convex hull theory, as shown in Figure 3 shown Figure 3 The midpoint {i 1 , i 2 , i 3 , i 4} represents the point set I of the image RGB . The polygon P is the convex hull P. Assume that the point set of all pixel points of the source image in the RGB color space is I RGB . The convex hull of this point set is P. According to the convex hull theory, any point inside the convex hull can be linearly represented by the vertices of the convex hull, as shown in formula (1). Therefore, the point set I RGB can be linearly represented by the vertices of the convex hull, that is, formula (2), where P RGB is the vertex set of the convex hull P

[0089] Taking P RGB as the color palette of the image and W as the corresponding coefficient matrix, the above process is the basic principle of the method for extracting the image color palette and solving the coefficient matrix based on the convex hull theory. The convex hull of the point set and the coefficient matrix can be solved through the ConvexHull and Delaunay modules of the open-source machine learning library Scipy respectively

[0090]

[0091] I RGB = WP RGB (2)

[0092] For example, in formula (1), i 1 is the first point inside the convex hull P, which can represent the color component value of the first pixel point in the source image; p 1 to p 7 are the vertices of the convex hull P, which can represent the source color palette of the source image; w 11 is the linear relationship between the first point i 1 inside the convex hull P and the vertex p 1 of the convex hull, which can be understood as the coefficient of the color component value of the first pixel point relative to the source color palette corresponding to the vertex p 1 ; similarly, w 15 is the coefficient of the color component value of the first pixel point relative to the source color palette corresponding to the vertex p 5 ; w 21 is the coefficient of the color component value of the second pixel point relative to the source color palette corresponding to the vertex p 1

[0093] ​It should be noted that in formula (1), the value of w can be any value between 0 and 1, such as 0.2, 0.35, etc. The image after color transfer retains the deviation between the color of each pixel point and the color shown in the source color palette, without causing information loss; while when the color component values of the pixel points are attributed to the coefficient matrix determined by a certain clustering center (color palette), the value of w is only simply 0 or 1, resulting in only several colors on the color palette in the reconstructed image.

[0094] The method provided by the embodiment of the present invention determines the coefficient matrix corresponding to the source color palette through the linear relationship between the color component values of each pixel point and the color component values represented by the vertices of the convex hull of the color space. Compared with only attributing the pixel points to the coefficient matrix determined by a certain clustering center, it more accurately determines the weights of various colors in the source color palette on each pixel point, thereby improving the quality of the color transfer image.

[0095] Based on the above embodiments, Figure 4 is a schematic flowchart of step 130 in the image color transfer method provided by the present invention, as Figure 4 shown, step 130 specifically includes:

[0096] Step 131, determining the initial vertex set of the convex hull of the color space;

[0097] Step 132, performing vertex merging on the initial vertex set until the merging termination condition is reached, and the merging termination condition is determined based on the merging error of the vertices to be merged and / or the number of vertices after merging;

[0098] Step 133, determining the source color palette of the source image based on the vertex set of the convex hull of the color space after vertex merging.

[0099] Specifically, in actual application, since the number of directly obtained convex hull vertices is usually multiple, such as several hundred, directly using it as the source color palette of the source image will lead to an overly complex subsequent color transfer process. Therefore, it is necessary to reduce the number of convex hull vertices, and the number of vertices can be reduced by merging the convex hull vertices to meet the actual usage requirements.

[0100] The specific method of vertex merging can directly merge two adjacent vertices, or first perform pre-merging on two adjacent vertices, and then merge the two vertices with the smallest pre-merging error. The embodiment of the present invention does not make specific limitations on this.

[0101] When the merging termination condition is reached, the vertex merging is terminated to form a new convex hull of the color space after merging. The merging termination condition can be determined according to the merging error of the vertices to be merged. For example, when the merging error is greater than the pre-set error threshold, the merging is terminated.

[0102] It is also possible to determine the merging termination condition based on the number of vertices after merging. For example, when the number of vertices after merging is less than or equal to a preset number threshold, the merging is terminated.

[0103] Of course, the above two termination conditions can also be comprehensively considered during merging. If both of the above two termination conditions are met, the merging is terminated.

[0104] Determine the source color palette of the source image according to the vertex set of the convex hull of the merged color space. For example, the coordinate values of each vertex can be directly used as the color component values of each color in the source color palette.

[0105] Based on any of the above embodiments, the following method can be used to merge adjacent vertices of the convex hull:

[0106] First, determine the initial vertex set according to the constructed convex hull of the color space. For example, the initial vertex set can be expressed as

[0107] Next, find two adjacent vertices p x and p y with the smallest merging error after merging for merging. When two adjacent vertices of the convex hull are merged, the color component values of some pixel points may fall outside the convex hull, resulting in a merging error. The merging error can be the shortest distance from the pixel points falling outside the convex hull to the convex hull after merging the vertices.

[0108] At the same time, update the reconstruction error according to the merging error of the two initial vertices to be merged. Here, the reconstruction error can be the average merging error, representing the average value of the merging error. For example, it can be represented by e sum / count p where e sum is the cumulative merging error and count p is the number of pixel points in the source image.

[0109] During merging, it is necessary to determine whether the merging termination condition is reached. If the merging termination condition is reached, the merging is terminated; if the merging termination condition is not reached, continue merging and update the reconstruction error at the same time. The merging termination condition can be set according to actual needs. For example, the merging termination condition can be that the reconstruction error is greater than the first threshold. When the reconstruction error is greater than the first threshold, the vertex merging of the convex hull is completed; the merging termination condition can also be that the number of vertices of the convex hull of the color space is less than or equal to the second threshold. When the number of vertices of the convex hull of the color space is less than or equal to the second threshold, the vertex merging of the convex hull is completed. Here, both the first threshold and the second threshold are preset.

[0110] After the vertex merging of the convex hull is completed, a new convex hull vertex set P' RGB, the source color palette of the source image can be determined according to the new convex hull vertex set, that is, the new convex hull vertices can be used as the source color palette of the source image.

[0111] In the method provided by the embodiment of the present invention, considering that the number of directly obtained convex hull vertices is large, the number of vertices is reduced by merging two adjacent vertices of the convex hull, thereby reducing the computational complexity of the subsequent color transfer process and improving the calculation speed.

[0112] Based on the above embodiments, Figure 5 is a schematic flowchart of step 150 in the image color transfer method provided by the present invention, as Figure 5 shown, step 150 specifically includes:

[0113] Step 151, converting the source color palette and the target color palette to the target color space;

[0114] Step 152, migrating each color component except the luminance component in the target color palette to the source color palette respectively to obtain a migrated color palette;

[0115] Step 153, converting the migrated color palette from the target color space to the initial color space;

[0116] Step 154, determining the target image based on the migrated color palette in the initial color space and the coefficient matrix corresponding to the source color palette.

[0117] Specifically, in the case of obtaining the source color palette, the coefficient matrix corresponding to the source color palette, and the target color palette, color transfer can be performed on the source image.

[0118] Among them, the initial color spaces of the source color palette and the target color palette can be the same. For example, the initial color spaces of both the source color palette and the target color palette are the RGB color space, or they can be different. For example, the initial color space of the source color palette is the RGB color space, and the initial color space of the target color palette is the HSV color space.

[0119] The target color space here can be any color space. For example, it can be the RGB color space or the HSV color space. It should be noted that after color space conversion, the source color palette and the target color palette are in the same color space, that is, the target color space.

[0120] In addition, through a large number of experiments and studies, it is found that when performing color transfer, if the brightness distribution of the source image changes, it will cause visual incoordination, thereby affecting the quality of the transferred image.

[0121] Taking the above two points into consideration at the same time, in the color transfer method provided by the embodiment of the present invention, when performing color transfer in the target color space, the luminance component is not migrated, and only each color component except the luminance component in the target color palette is migrated to the source color palette.

[0122] Taking the HSV color space as the target color space as an example, the components of the HSV color space are: hue H, saturation S, and value V.

[0123] When performing color transfer in the HSV color space, the H component and the S component of the target color palette can be transferred to the source color palette respectively to obtain a transferred color palette. Only changing the H and S components of the source color palette here can avoid changing the brightness distribution of the source image, thereby avoiding visual disharmony.

[0124] When performing color transfer, the number of colors of the source color palette and the target color palette can be the same or different. When the number of colors of the source color palette and the target color palette is different, the priority of color transfer can be determined according to the order of the source color palette and the target color palette.

[0125] Then, the transferred color palette is converted back to the initial color space again, and according to the formula I target = WP target , multiplying the transferred color palette by the coefficient matrix W corresponding to the source color palette can obtain the target image, and the target image here is the image obtained after color transfer.

[0126] The image color transfer method provided by the embodiments of the present invention can avoid changing the brightness distribution of the source image and thus avoid visual disharmony, and further improve the quality of the color transfer image by converting both the source color palette and the target color palette from the initial color space to the target color space, performing color transfer in the target color space, and only changing the color components other than the brightness component of the source color palette.

[0127] Based on the above embodiments, step 152 specifically includes: sequentially transferring each color component other than the brightness component in the target color palette to the source color palette according to the importance sorting order of the source color palette and the order of the target color palette to obtain a transferred color palette;

[0128] The importance sorting order of the source color palette is determined based on the coefficient matrix corresponding to the source color palette.

[0129] Specifically, when performing color transfer, considering that the number of colors of the source color palette and the target color palette may not match. For example, the source color palette has 5 colors while the target color palette has only 4 colors. At this time, it is necessary to sort the source color palette and the target color palette, and sequentially transfer each color component other than the brightness component in the target color palette to the source color palette according to the order of the source color palette and the target color palette to obtain a transferred color palette.

[0130] Among them, the sorting can be performed according to the importance of the source color palette. The higher the importance of the color, the higher the transfer priority during color transfer; the lower the importance of the color, the lower the transfer priority or no transfer during color transfer.

[0131] The importance of the source color palette is determined according to the coefficient matrix corresponding to the source color palette. The importance of the source color palette can be characterized by the sum of all coefficients corresponding to any one color in the source color palette. The larger the sum, the higher the importance, and the higher the migration priority during color migration.

[0132] For example, the target color space is the HSV color space. The five colors of the source color palette are A, B, C, D, and E respectively, and the four colors of the target color palette are F, G, J, and K respectively. The importance of the source color palette determined according to the coefficient matrix corresponding to the source color palette are: 1.2, 0.7, 0.9, 1.9, 2.2. Then the order of the source color palette after sorting according to importance is E, D, A, C, B.

[0133] Since the number of colors in the target color palette is 4, during color migration, the first four source color palettes after sorting are preferentially migrated, that is, E, D, A, C, and the color palette B does not perform color migration. And to avoid causing changes in the brightness distribution, only the H and S components are migrated. For example, the H and S components of the F color palette can be sequentially migrated to the E color palette, the H and S components of the G color palette are migrated to the D color palette, the H and S components of the J color palette are migrated to the A color palette, and the H and S components of the K color palette are migrated to the C color palette.

[0134] It should be noted that after sorting the importance of the source color palette, the order of the coefficient matrix corresponding to the source color palette also needs to be adjusted accordingly to be consistent with the sorted source color palette.

[0135] The method provided by the embodiments of the present invention first determines the importance of the source color palette according to the coefficient matrix corresponding to the source color palette, and then performs color migration according to the sorting order of the importance of the source color palette and the order of the target color palette, ensuring that the higher the importance of the color, the higher the migration priority during color migration, and further improving the quality of the color-migrated image.

[0136] Based on any of the above embodiments, Figure 6 is one of the flow diagrams of the method for obtaining the target color palette provided by the present invention, as Figure 6 shown, step 110 specifically includes:

[0137] Step 111a, obtain a reference image and determine the color palette of the reference image;

[0138] Step 112a, determine the color palette of the reference image as the target color palette.

[0139] Specifically, the reference image here refers to an image used to provide color information for color transfer. The reference image can be a landscape image, a portrait image, etc. captured by an image acquisition device, or an image pre-drawn by a designer with a specific color matching scheme. Of course, it can also be an image selected by a user obtained from the cloud or a database. The embodiments of the present invention do not make specific limitations on this.

[0140] Considering that in practical applications, it is often necessary to transfer the color style in the reference image to the source image so that the source image has a color distribution similar to that of the reference image without changing the content distribution. Therefore, the color palette of the reference image can be extracted first to obtain the color palette of the reference image. Among them, the method for extracting the color palette of the reference image is the same as the method for determining the source color palette described in the above embodiments, and will not be elaborated here.

[0141] After the color palette of the reference image is determined, the color palette of the reference image can be determined as the target color palette, and based on the source color palette, the coefficient matrix corresponding to the source color palette, and the target color palette, color transfer is performed on the source image. The transferred image has a content distribution similar to or the same as that of the source image, and also has a color distribution similar to or the same as that of the reference image.

[0142] The method provided by the embodiments of the present invention obtains a reference image and determines the color palette of the reference image as the target color palette. This method can perform color transfer on the source image based on the source image and the reference image. The transferred image has a content distribution similar to or the same as that of the source image, and also has a color distribution similar to or the same as that of the reference image.

[0143] Based on any of the above embodiments, Figure 7 is the second flow diagram of the method for obtaining the target color palette provided by the present invention. As Figure 7 shown, step 110 specifically includes:

[0144] Step 111b, obtaining the number of source images and target images;

[0145] Step 112b, converting the color palette of the source image to the target color space;

[0146] Step 113b, sampling the color palette of the source image in each dimension except the brightness dimension in the target color space for the number of target images to obtain the number of sets of sampled color palettes of the target image;

[0147] Step 114b, determining the color palette obtained by converting the number of sets of sampled color palettes of the target image from the target color space to the initial color space as the target color palette.

[0148] Specifically, the number of target images here refers to the number of target images obtained after color transfer. The number of target images can be flexibly set by the user according to actual needs. For example, it can be 2 or 4. The method for obtaining the source image is the same as that in the above embodiment and will not be elaborated here.

[0149] Considering the actual application, another common color transfer application scenario is that the user only provides the source image and the number of target images to be generated by migration, and the algorithm needs to automatically select colors in the color space for transfer. The method provided in the embodiments of the present invention can determine the target color palette based on the source image and the number of target images, and then perform color transfer based on the source image and the target color palette.

[0150] First, the color palette of the source image can be extracted to obtain the color palette of the source image. The method for color palette extraction is the same as the method for determining the source color palette in the above embodiment. Then, the color palette of the source image is converted to the target color space, and the color palette of the source image is sampled in each dimension except the brightness dimension in the target color space for the number of target images to obtain the number of sets of sampled color palettes corresponding to the number of target images. The sampling here can be random sampling or uniform sampling, and the embodiments of the present invention do not make specific limitations on this.

[0151] Here, only the values of each color component except the brightness component are changed, which is also to avoid causing brightness changes and resulting in visual incoordination problems.

[0152] Finally, the color palette obtained by converting the number of sets of sampled color palettes from the target color space to the initial color space is determined as the target color palette. The number of target color palettes is consistent with the number of target images.

[0153] After obtaining the corresponding number of target color palettes, one target image after color transfer can be generated based on each set of target color palettes and the source image, and finally the corresponding number of target images can be obtained.

[0154] The method provided in the embodiments of the present invention samples the color palette of the source image in each dimension except the brightness dimension in the target color space for the number of target images to obtain the number of sets of sampled color palettes, and then determines the target color palette based on the number of sets of sampled color palettes. Through this method, the source image can be automatically color transferred according to the source image and the number of target images to be generated by migration.

[0155] Based on any of the above embodiments, Figure 8 is the second flowchart of the image color transfer method provided by the present invention. As Figure 8 shown, the method includes:

[0156] Step 810, obtain the source image and the transfer target.

[0157] Here, the migration target is divided into: the target color palette, the reference image, or the number of target images according to different migration modes.

[0158] Step 820: Solve the source color palette and the corresponding coefficient matrix based on the convex hull theory.

[0159] The basic process of the method for extracting the image color palette and solving the coefficient matrix based on the convex hull theory is similar to the traditional color palette extraction method. Both use a certain method to layer the image, and similar colors are divided into the same layer. Different color layers form the color palette of the image, and at the same time, solve the coefficient matrix that can restore the image according to the color palette. When performing subsequent color migration, only need to change the color palette to the specified color and multiply it by the coefficient matrix to obtain the color-migrated image. The difference is that traditional color palette extraction mostly uses methods such as color clustering for color palette extraction, while the method proposed in the embodiments of the present invention completes color palette extraction based on the convex hull theory.

[0160] The coefficient matrix determined according to the clustering center usually only assigns pixel points to a certain clustering center, resulting in only several colors on the color palette in the reconstructed image. Specifically, the color granularity of the image after color migration is relatively rough and the color transition positions are not natural. The method for extracting the image color palette and solving the coefficient matrix based on the convex hull theory greatly reduces the problem of rough and unnatural colors in the color-migrated image.

[0161] Step 830: Perform color migration on the source image based on the source color palette, the corresponding coefficient matrix of the source color palette, and the migration target.

[0162] Specifically, according to the common usage scenarios of image color migration, the embodiments of the present invention provide three migration modes according to the migration target, namely, the image color migration mode based on the target color palette, the image color migration mode based on the reference image, and the full-automatic image color migration mode based on the number of target images.

[0163] Among them, the basic steps of the image color migration based on the reference image are basically the same as those of the image color migration based on the target color palette. The difference is that the input of the image color migration based on the reference image is not the target color palette but the reference image. Therefore, it is necessary to first extract the target color palette from the reference image. The color palette extraction method is the same as that described above. The color migration steps after the target color palette extraction are the same as those of the migration mode based on the target color palette.

[0164] In the full-automatic image color migration mode based on the number of target images, the user only needs to provide the source image and the expected number of generated images N. The full-automatic image color migration algorithm will automatically select N sets of appropriate target color palettes and generate the target images.

[0165] The image color transfer device provided by the present invention will be described below. The image color transfer device described below can be referred to in correspondence with the image color transfer method described above.

[0166] Based on any of the above embodiments, Figure 9 is a schematic structural diagram of the image color transfer device provided by the present invention. As Figure 9 shown, the device includes:

[0167] An image and color palette acquisition unit 910, configured to acquire a source image and a target color palette;

[0168] A convex hull construction unit 920, configured to construct a color space convex hull based on the color component values of each pixel point in the source image;

[0169] A source color palette determination unit 930, configured to determine the source color palette of the source image based on the vertices of the color space convex hull;

[0170] A coefficient matrix determination unit 940, configured to determine a coefficient matrix corresponding to the source color palette based on the relationship between the color component values of each pixel point and each vertex of the color space convex hull;

[0171] A color transfer unit 950, configured to perform color transfer on the source image based on the source color palette, the coefficient matrix corresponding to the source color palette, and the target color palette.

[0172] For the image color transfer device provided by the embodiments of the present invention, first, based on the constructed color space convex hull, the source color palette of the source image is determined. According to the relationship between the color component values of each pixel point and the color space convex hull, the coefficient matrix corresponding to the source color palette is determined, so that the color of each pixel point in the source image can be accurately represented by the source color palette, and the deviation between the color of each pixel point and the color shown by the source color palette is retained, without causing information loss; then, based on the source color palette, the coefficient matrix corresponding to the source color palette, and the target color palette, color transfer is performed on the source image. The resulting transferred image retains the deviation between the color of each pixel point and the color shown by the source color palette, solves the problem of rough color and unnatural transition of the color transfer image, and improves the quality of the color transfer image.

[0173] Based on any of the above embodiments, the coefficient matrix determination unit 940 is further configured to:

[0174] Determine the coefficient of each pixel point relative to the source color palette based on the linear relationship between the color component values of each pixel point and the color component values represented by the color space convex hull, where the source color palette corresponds to the color component values represented by the color space convex hull;

[0175] Determine the coefficient matrix corresponding to the source color palette based on the coefficients of each pixel point relative to the source color palette.

[0176] Based on any of the above embodiments, the source color palette determination unit 930 is further configured to:

[0177] Determine an initial vertex set of the convex hull of the color space;

[0178] Perform vertex merging on the initial vertex set until a merging termination condition is reached, where the merging termination condition is determined based on the merging error of the vertices to be merged and / or the number of vertices after merging;

[0179] Based on the vertex set of the convex hull of the color space after vertex merging, determine the source color palette of the source image.

[0180] Based on any of the above embodiments, the color transfer unit 950 is further configured to:

[0181] Convert the source color palette and the target color palette to the target color space;

[0182] Transfer each color component other than the luminance component in the target color palette to the source color palette respectively to obtain a transferred color palette;

[0183] Convert the transferred color palette from the target color space to the initial color space;

[0184] Based on the transferred color palette in the initial color space and the coefficient matrix corresponding to the source color palette, determine the target image.

[0185] Based on any of the above embodiments, the color transfer unit 950 is further configured to:

[0186] According to the importance sorting order of the source color palette and the order of the target color palette, sequentially transfer each color component other than the luminance component in the target color palette to the source color palette to obtain a transferred color palette;

[0187] The importance sorting order of the source color palette is determined based on the coefficient matrix corresponding to the source color palette.

[0188] Based on any of the above embodiments, the image and color palette acquisition unit 910 is further configured to:

[0189] Acquire a reference image and determine the color palette of the reference image;

[0190] Determine the color palette of the reference image as the target color palette.

[0191] Based on any of the above embodiments, the image and color palette acquisition unit 910 is further configured to:

[0192] Acquire the number of source images and target images;

[0193] Convert the color palette of the source image to the target color space;

[0194] In the target color space, sample the target image in the number of copies for each dimension of the color palette of the source image except the luminance dimension to obtain the number of sets of sampled color palettes of the target image.

[0195] Determine the color palette obtained after converting the number of sets of sampled color palettes of the target image from the target color space to the initial color space as the target color palette.

[0196] Figure 10 An example of a schematic physical structure diagram of an electronic device is shown as Figure 10 shown. The electronic device may include: a processor 1010, a communication interface 1020, a memory 1030, and a communication bus 1040. Among them, the processor 1010, the communication interface 1020, and the memory 1030 complete mutual communication through the communication bus 1040. The processor 1010 may call the logical instructions in the memory 1030 to execute an image color migration method, and the method includes: obtaining a source image and a target color palette; constructing a color space convex hull based on the color component values of each pixel point in the source image; determining the source color palette of the source image based on the vertices of the color space convex hull, and determining the coefficient matrix corresponding to the source color palette based on the relationship between the color component values of each pixel point and the vertices of the color space convex hull; performing color migration on the source image based on the source color palette, the coefficient matrix corresponding to the source color palette, and the target color palette.

[0197] In addition, when the logical instructions in the above-mentioned memory 1030 are implemented in the form of a software functional unit and sold or used as an independent product, they may be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of the technical solution, may be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The foregoing storage medium includes: various media such as a USB flash drive, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk, or an optical disc that can store program codes.

[0198] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the image color transfer method provided by the above-mentioned various methods. The method includes: obtaining a source image and a target color palette; constructing a color space convex hull based on the color component values of each pixel point in the source image; determining the source color palette of the source image based on the color space convex hull; determining a coefficient matrix corresponding to the source color palette based on the relationship between the color component values of each pixel point and the color space convex hull; and performing color transfer on the source image based on the source color palette, the coefficient matrix corresponding to the source color palette, and the target color palette.

[0199] In another aspect, the present invention also provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it realizes the image color transfer method provided by the above-mentioned various methods. The method includes: obtaining a source image and a target color palette; constructing a color space convex hull based on the color component values of each pixel point in the source image; determining the source color palette of the source image based on the color space convex hull; determining a coefficient matrix corresponding to the source color palette based on the relationship between the color component values of each pixel point and the color space convex hull; and performing color transfer on the source image based on the source color palette, the coefficient matrix corresponding to the source color palette, and the target color palette.

[0200] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art can understand and implement it without creative efforts.

[0201] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on such an understanding, the above technical solution, in essence, or the part that contributes to the prior art can be embodied in the form of a software product. The computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0202] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements on some of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. An image color transfer method, characterized in that, it includes: Obtain a source image and a target color palette; Based on the color component values of each pixel point in the source image, construct a color space convex hull; Based on the color space convex hull, determine the source color palette of the source image; Based on the relationship between the color component values of each pixel point and the color space convex hull, determine the coefficient matrix corresponding to the source color palette; Based on the source color palette, the coefficient matrix corresponding to the source color palette, and the target color palette, perform color transfer on the source image; wherein, the performing color transfer on the source image based on the source color palette, the coefficient matrix corresponding to the source color palette, and the target color palette includes: Convert the source color palette and the target color palette to the target color space; Transfer each color component other than the luminance component in the target color palette to the source color palette respectively to obtain a transferred color palette; Convert the transferred color palette from the target color space to the initial color space; Based on the transferred color palette in the initial color space and the coefficient matrix corresponding to the source color palette, determine the target image.

2. The image color transfer method according to claim 1, characterized in that, the determining the coefficient matrix corresponding to the source color palette based on the relationship between the color component values of each pixel point and the color space convex hull includes: Based on the linear relationship between the color component values of each pixel point and the color component values represented by the color space convex hull, determine the coefficient of each pixel point relative to the source color palette, and the source color palette corresponds to the color component values represented by the color space convex hull; Based on the coefficients of each pixel point relative to the source color palette, determine the coefficient matrix corresponding to the source color palette.

3. The image color transfer method according to claim 1, characterized in that, the determining the source color palette of the source image based on the color space convex hull includes: Determine the initial vertex set of the color space convex hull; Perform vertex merging on the initial vertex set until the merging termination condition is reached, and the merging termination condition is determined based on the merging error of the vertices to be merged and / or the number of vertices after merging; Based on the vertex set of the color space convex hull after vertex merging, determine the source color palette of the source image.

4. The image color transfer method according to claim 1, characterized in that, the transferring each color component other than the luminance component in the target color palette to the source color palette respectively to obtain a transferred color palette includes: According to the importance sorting order of the source color palette and the order of the target color palette, sequentially transfer each color component other than the luminance component in the target color palette to the source color palette to obtain a transferred color palette; The importance sorting order of the source color palette is determined based on the coefficient matrix corresponding to the source color palette.

5. The image color transfer method according to any one of claims 1-4, characterized in that, the target color palette is obtained based on the following steps: Obtain a reference image and determine the color palette of the reference image; Determine the color palette of the reference image as the target color palette.

6. The image color transfer method according to any one of claims 1-4, characterized in that, The target color palette is obtained based on the following steps: Obtain the number of source images and target images; Convert the color palette of the source image to the target color space; Under the target color space, sample the color palette of the source image by the number of target images for each dimension except the brightness dimension to obtain the number of sets of sampled color palettes equal to the number of target images; Determine the color palette obtained by converting the number of sets of sampled color palettes from the target color space to the initial color space as the target color palette.

7. An image color transfer device, characterized in that, it includes: An image and color palette acquisition unit for acquiring a source image and a target color palette; A convex hull construction unit for constructing a color space convex hull based on the color component values of each pixel point in the source image; A source color palette determination unit for determining the source color palette of the source image based on the color space convex hull; A coefficient matrix determination unit for determining the coefficient matrix corresponding to the source color palette based on the relationship between the color component values of each pixel point and the color space convex hull; A color transfer unit for performing color transfer on the source image based on the source color palette, the coefficient matrix corresponding to the source color palette, and the target color palette; wherein, the device is further configured to: Convert the source color palette and the target color palette to the target color space; Transfer each color component except the brightness component in the target color palette to the source color palette respectively to obtain a transferred color palette; Convert the transferred color palette from the target color space to the initial color space; Determine the target image based on the transferred color palette in the initial color space and the coefficient matrix corresponding to the source color palette.

8. An electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, when the processor executes the program, it implements the steps of the image color transfer method according to any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium, on which a computer program is stored, characterized in that, when the computer program is executed by a processor, it implements the steps of the image color transfer method according to any one of claims 1 to 6.

Images