Image recoloring method and system based on palette

Abstract

The present invention provides a palette-based image recoloring method and system, and relates to the field of image processing technology. The proposed method includes: extracting the convex hull of the RGB space corresponding to the input image; clustering the cluster centers of the image pixels in the convex hull of the RGB space, and selecting the cluster centers as representative colors, and the representative colors corresponding to the convex hull vertices and the cluster centers together form a palette; tetrahedralizing the vertex set corresponding to the palette, and calculating the interpolation weights of the pixels with respect to the palette colors; modifying the color of the palette, and using the interpolation weights to implement image recoloring editing. The extracted palette of the present invention has better representativeness and editing locality, which is convenient for users to perform local editing on images.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a palette-based image recoloring method and system. Background Art

[0002] Recoloring is a popular research topic in the field of visual media editing. It is widely used in film production and art design, and plays an important role in color editing of images and videos.

[0003] At present, there are two main types of image recoloring methods based on palettes: extracting image palettes by clustering and extracting image palettes by calculating convex hulls. Among them, extracting image palettes by calculating convex hulls is the latest technology in this field. At present, the method of extracting image palettes by calculating convex hulls is to regard the image as a point set in RGB color space, calculate the convex hull of the point set and further simplify it. Then, the weight of each pixel with respect to the palette is obtained by the MVC interpolation method, and finally the image recoloring is achieved by modifying the color of the palette. This type of method can extract hidden representative colors, but some representative colors in the image may be located inside the convex hull and ignored. That is, the existing palette-based image recoloring methods ignore the representative colors located inside the convex hull, making the palette lack of representativeness as a whole, which is not conducive to subsequent image editing. Summary of the invention

[0004] 1. Technical issues to be solved

[0005] In view of the deficiencies of the prior art, the present invention provides a palette-based image recoloring method and system, which solves the technical problem that the prior methods ignore the representative colors located inside the convex hull.

[0006] (II) Technical solution

[0007] To achieve the above objectives, the present invention is implemented through the following technical solutions:

[0008] (III) Beneficial effects

[0009] The present invention provides a palette-based image recoloring method and system. Compared with the prior art, it has the following beneficial effects:

[0010] The present invention discloses a method and system for image recoloring based on a palette. The method first extracts the convex hull of the RGB space corresponding to the input image; then clusters the image pixel points in the convex hull of the RGB space to obtain the cluster center, and selects the cluster center as a representative color. The convex hull vertices and the representative colors corresponding to the cluster center together form a palette; tetrahedron is performed on the vertex set corresponding to the palette, and the interpolation weight of the pixel point with respect to the palette color is calculated; finally, the color of the palette is modified, and the interpolation weight is used to realize image recoloring editing. The extracted palette of the present invention has better representativeness and editing locality, which is convenient for users to perform local editing on the image. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0012] Figure 1 is a flowchart of an embodiment of the present invention;

[0013] Figure 2 It is an algorithm framework diagram of an embodiment of the present invention;

[0014] Figure 3 A simplified schematic diagram of a convex hull according to an embodiment of the present invention;

[0015] Figure 4a and Figure 4b All of them are effect diagrams of verification experiments, wherein the first column is the input, the second and third columns are effect diagrams of existing algorithms, and the fourth column is the effect diagram of the method of an embodiment of the present invention. DETAILED DESCRIPTION

[0016] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention are clearly and completely described. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0017] The embodiments of the present application provide a palette-based image recoloring method and system, which solves the technical problem that the existing methods ignore the representative colors located inside the convex hull, and achieves that the extracted palette has better representativeness and editing locality, and the recoloring effect is excellent.

[0018] The technical solution in the embodiment of the present application is to solve the above technical problems, and the overall idea is as follows:

[0019] At present, there are two main types of image recoloring methods based on palettes: the first type of method extracts the image palette through clustering, and then obtains the weight of each pixel about the cluster center through RBF interpolation for image recoloring editing. The representative colors extracted by this type of method come completely from the image itself, which well represents the color distribution of the image. However, the representative colors in the image may not appear explicitly on the image. The second type of method extracts the image palette by calculating the convex hull. This type of method regards the image as a point set in the RGB color space, calculates the convex hull of the point set and further simplifies it. Then, the weight of each pixel about the palette is obtained by the MVC interpolation method, and finally the image recoloring is achieved by modifying the color of the palette. This type of method can extract hidden representative colors, but some representative colors in the image may be located inside the convex hull and ignored. To solve the above problems, the embodiments of the present invention enhance the expressiveness of the palette by adding internal representative vertices (colors) based on the simplified convex hull of the image, spatially divides the palette colors (vertices), and interpolates pixels within each tetrahedron to improve the sparsity of the interpolation weights, etc., so that the extracted palette has better representativeness, the interpolation weights are sparser, and supports more precise local editing.

[0020] In order to better understand the above technical solution, the above technical solution will be described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0021] The embodiment of the present invention provides a palette-based image recoloring method, such as Figures 1-2 As shown, including:

[0022] S1, extract the convex hull of the RGB space corresponding to the input image;

[0023] S2, clustering the image pixels in the convex hull of the RGB space to obtain the cluster center, and selecting the cluster center as a representative color, wherein the representative colors corresponding to the convex hull vertices and the cluster center together constitute a color palette;

[0024] S3, tetrahedroning the vertex set corresponding to the palette, and calculating the interpolation weight of the pixel point with respect to the palette color;

[0025] S4. Modify the color of the palette and use the interpolation weight to implement image recoloring editing.

[0026] The following is a detailed description of each step:

[0027] In step S1, the convex hull of the RGB space corresponding to the input image is extracted. The specific implementation process is as follows:

[0028] The input image is regarded as a point set in RGB space, and the original three-dimensional convex hull of the point set is calculated. The original three-dimensional convex hull often contains hundreds or thousands of vertices and cannot be directly used as a palette to edit the image. It needs to be simplified. The simplification process is as follows:

[0029] like Figure 3 As shown, the edge (v i , v j ) is reduced to a vertex v. Specifically, we need to find a vertex v outside the convex hull such that v connects (v i , v j )'s adjacent points form a new convex hull that expands outward and completely encloses (v i , v j ), so that the edge (v i , v j ) are deleted from the original convex hull, and the topological simplification of the convex hull is achieved while the convex hull volume is expanded. The expression of v is as follows:

[0030]

[0031] The right side of the equation represents the volume increase of the convex hull caused by the shrinkage, and A represents (v i , v j ) is the area of ​​each triangle associated with the image, n represents the normal of each triangle, and v is the vertex coordinate to be solved. In actual operation, all the edges of the convex hull are enumerated each time and the edge with the smallest volume increase is selected for contraction. When the volume of the convex hull grows to a certain extent, the algorithm ends (in the specific implementation process, as the volume of the convex hull increases, the reconstruction error increases accordingly. When the reconstruction error is greater than 2 / 255, the convex hull expansion is stopped). After the algorithm ends, a convex hull with a small number of vertices is obtained, that is, the convex hull in RGB space.

[0032] In step S2, clustering is performed in the convex hull of the RGB space to obtain the cluster center of the image pixels, and the cluster center is selected as the representative color. The representative colors corresponding to the convex hull vertices and the cluster center together form a color palette. The specific implementation process is as follows:

[0033] In step S1, as the shrinking continues, the volume of the convex hull will continue to increase, and those representative colors inside the convex hull will not be added to the palette and cannot participate in subsequent image editing tasks. In order to make the palette more representative, the embodiment of the present invention further optimizes the initial palette by introducing those ignored representative colors inside the convex hull through clustering.

[0034] The embodiment of the present invention uses the MeanShift algorithm to cluster image pixels to extract representative colors inside the image. It should be noted that clustering can also be performed by other methods such as hierarchical clustering. The embodiment of the present invention introduces in detail the use of the MeanShift algorithm to cluster image pixels.

[0035] Each pixel in the convex hull of the RGB space is regarded as an independent cluster center. Then, mean shift is performed on each pixel. After multiple rounds of mean shift, each pixel will eventually converge to a fixed position. Finally, pixels with close positions are classified into the same class. Specifically, for any pixel, assume that its coordinates in the RGB space are c0=p0(R0, G0, B0), make a ball S0 with r as radius, and then calculate the mean of all pixels falling in S0 c1=(R1, G1, B1). Next, make a ball S1 with c1 as the center and r as radius and calculate the mean of the pixels in the ball again. Repeat this process until S kt1 The center of the sphere c k+1 Distance S k The center of the sphere c k If the value is close enough, the algorithm is considered to have converged and the mean drift has stopped. The formal representation is as follows:

[0036]

[0037] Among them, p1, p2, ..., p M is located at the center of the sphere c k All pixels within a sphere with radius r.

[0038] The estimation of radius r is particularly important for MeanShift clustering. Generally, a smaller radius results in more clusters, and a larger radius results in fewer clusters. The method for selecting the radius in the embodiment of the present invention includes:

[0039] First, randomly select several pixel points in the image, calculate the distances of all point pairs and sort them, and mark the sorted sequence as D. Assuming that the number of all point pairs is N, the distance r = D [Nq] at position Nq in D is selected as the radius of the mean shift.

[0040] The efficiency of performing MeanShift clustering directly on all pixels is very low. In order to improve the speed, the embodiment of the present invention samples the original image, and then only considers the points on the sampling point set as the initial clustering centers for mean shift. After clustering, some new representative colors are obtained. These colors are basically located inside the convex hull of the RGB space obtained in S1. At the same time, some colors may be very close to the convex hull vertex, making the palette redundant. To this end, all colors of the initial palette are retained and those colors similar to the initial palette in the clustering results are deleted. Specifically, the algorithm deletes those colors whose distance from the convex hull vertex is less than r to obtain representative colors.

[0041] The representative colors corresponding to the convex hull vertices and cluster centers together form a color palette.

[0042] In step S3, the vertex set corresponding to the palette is tetrahedronized, and the interpolation weight of the pixel point with respect to the palette color is calculated. The specific implementation process is as follows:

[0043] All cluster center points and convex hull vertices corresponding to representative colors are organized together to form a three-dimensional vertex set, and the three-dimensional vertex set is divided into Delaunay tetrahedrons. In this way, the image pixel point set will be naturally divided into multiple tetrahedrons.

[0044] For interpolation weights Calculation is currently divided into optimization-based and interpolation-based methods. The interpolation weights obtained by the optimization-based method have good smoothness, but the calculation efficiency is often low. It often takes tens of minutes to calculate a single image. The current interpolation-based method requires a large number of control points. Therefore, when the number of palettes is small, it is difficult for existing algorithms to achieve sparse interpolation, that is, the interpolation weights of pixels for most palette colors are greater than 0. Modifying certain palette colors may cause large-area color changes in the image, making it difficult to achieve effective local editing.

[0045] After tetrahedron decomposition, each pixel in the embodiment of the present invention is only related to the four vertices of the tetrahedron. Therefore, the interpolation weight of each point has at most four elements with values ​​greater than 0, which naturally meets the sparsity requirement and achieves better local editing.

[0046] After the palette color vertex space is partitioned, a set of spatial tetrahedrons are obtained, and the interpolation of tetrahedrons has a good definition. Specifically, inside a tetrahedron ABCD after the palette color vertex space is partitioned, the interpolation weight of any pixel point p corresponding to vertex A can be simply expressed as:

[0047]

[0048] Among them, T(pBCD) represents the volume of the tetrahedron composed of 4 vertices of pBCD, and T(ABCD) represents the volume of tetrahedron ABCD. Similarly, the interpolation weights of point p with respect to the other 3 vertices of the tetrahedron can be obtained. and

[0049] In step S4, the color of the palette is modified, and the image recoloring editing is implemented using the interpolation weights. The specific implementation process is as follows:

[0050] For ease of description, the color set of the palette is marked as V, and the pixel set of the input image is marked as P. In order to implement image editing, all pixels of the input image are interpolated. As shown in the formula

[0051]

[0052] Represent any pixel point p∈P as a convex combination of the palette V. Among them, Represents pixel p with respect to palette color V i The interpolation weights satisfy In the actual image editing process, the interpolation weight is only calculated once at the beginning and remains unchanged afterwards. The user only needs to modify one or more colors of the palette V and substitute the modified palette V' into the formula You can get the edited image color.

[0053] In order to verify the effectiveness of the embodiment of the present invention, the following verification is performed:

[0054] The experimental environment is: Win10, AMD R7-5800H, 16GB memory. In order to verify the sensitivity of the parameters, the parameter q used by the MeanShift algorithm was first verified; then, in order to illustrate the excellent sparsity of the interpolation weights of the algorithm of the embodiment of the present invention, the method of the embodiment of the present invention was compared with the existing image recoloring algorithm based on K-Means; next, in order to illustrate that the algorithm of the embodiment of the present invention supports better local editing, the image recoloring effect of the method of the embodiment of the present invention and the existing image recoloring algorithm based on convex hull optimization were compared. The experimental results all show that the color palette extracted by the algorithm of the embodiment of the present invention has better representativeness. Compared with the current best algorithm, the interpolation weights of the embodiment of the present invention are sparser, more conducive to local editing of the image, and achieve better recoloring effect. As shown in Figure 4 (a), the user's editing intention is to change the color of the plant in the upper right corner from green to brown. The method shown in the second column changes the color of the flower and the color of the stamen while modifying the color of the plant. The method shown in the third column causes the color of the ground in the image to change unnaturally. The embodiment of the present invention well realizes the editing intention and hardly affects other areas of the image. As shown in Figure 4(b), the user's editing intention is to change the lady's brown hair to blue to further highlight the hair dyeing effect. Existing methods all introduce unnatural facial color changes, while the method of the embodiment of the present invention accurately achieves the hair dyeing effect.

[0055] The embodiment of the present invention further provides a palette-based image recoloring system, the system comprising:

[0056] Convex hull acquisition module, used to extract the convex hull of the RGB space corresponding to the input image;

[0057] A clustering module, used for clustering image pixels in the convex hull of the RGB space to obtain cluster centers, and selecting the cluster centers as representative colors, wherein the representative colors corresponding to the convex hull vertices and cluster centers together constitute a color palette;

[0058] A partitioning module is used to perform tetrahedral partitioning on the vertex set corresponding to the palette and calculate the interpolation weight of the pixel point with respect to the palette color;

[0059] The recoloring module is used to modify the color of the palette and implement image recoloring editing using the interpolation weights.

[0060] It can be understood that the palette-based image recoloring system provided in the embodiment of the present invention corresponds to the above-mentioned palette-based image recoloring method, and the explanations, examples, beneficial effects and other parts of its relevant contents can refer to the corresponding contents in the palette-based image recoloring method, which will not be repeated here.

[0061] An embodiment of the present invention further provides a computer-readable storage medium storing a computer program for palette-based image recoloring, wherein the computer program enables a computer to execute the palette-based image recoloring method as described above.

[0062] An embodiment of the present invention further provides an electronic device, including:

[0063] one or more processors;

[0064] Memory; and

[0065] One or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the programs comprising instructions for executing the palette-based image recoloring method as described above.

[0066] In summary, compared with the prior art, the present invention has the following beneficial effects:

[0067] 1. The color palette extracted in the embodiment of the present invention has better representativeness and editing locality, which facilitates users to perform local editing on the image.

[0068] 2. The embodiment of the present invention spatially divides the palette vertex set and uses barycentric coordinates to interpolate image pixels, which significantly improves the sparsity of interpolation weights. Compared with existing methods, the interpolation weights are sparser and support more precise local editing.

[0069] It should be noted that, in the embodiments of the present invention, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or equipment including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or equipment. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the existence of other identical elements in the process, method, article or equipment including the elements.

[0070] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A palette-based image recoloring method, characterized in that: include: S1, extract the convex hull of the RGB space corresponding to the input image; S2, clustering the image pixels in the convex hull of the RGB space to obtain the cluster center, and selecting the cluster center as a representative color, and the representative colors corresponding to the convex hull vertices and the cluster center together form a color palette; S3, tetrahedroning the vertex set corresponding to the palette, and calculating the interpolation weight of the pixel point with respect to the palette color; S4, modifying the color of the palette, and implementing image recoloring editing using the interpolation weight; The method of tetrahedroning the vertex set corresponding to the palette includes: All cluster center points and convex hull vertices of representative colors are organized together to form a vertex set, and the vertex set is subjected to Delaunay tetrahedron decomposition; The calculating the interpolation weight of the pixel point with respect to the color palette includes: Inside the tetrahedron ABCD, the interpolation weight of any pixel point p corresponding to vertex A can be simply expressed as: Where, T(pBCD) represents the volume of the tetrahedron formed by the 4 vertices of pBCD, and T(ABCD) represents the volume of the tetrahedron ABCD; The modifying the color of the palette comprises: The color set of the palette is marked as V, and the pixel set of the input image is marked as P; Interpolate all pixels of the input image, as shown in the formula: Represent any pixel point p∈P as a convex combination of the palette V; where Represents pixel p with respect to palette color V i The interpolation weights satisfy Modify one or more colors of the palette V and substitute the modified palette V′ into the formula You can get the edited image color.

2. The palette-based image recoloring method according to claim 1, characterized in that: The extracting the convex hull of the RGB space corresponding to the input image includes: The input image is regarded as a point set in RGB space, and the original three-dimensional convex hull of the point set is calculated; The edges of the original three-dimensional convex hull (v i , v j ) is reduced to a vertex v, and the convex hull of the RGB space is obtained. The calculation formula is as follows: The right side of the equation represents the volume increase of the convex hull caused by the shrinkage, and A represents (v i , v j ) is the area of ​​each triangle face associated with the image, n represents the normal of each triangle face, and v is the vertex coordinate to be solved. In actual operation, all the edges of the convex hull are enumerated each time and the edges with the smallest volume increase are selected for contraction. As the volume of the convex hull increases, the reconstruction error increases accordingly. When the reconstruction error exceeds the preset threshold, the algorithm ends and the convex hull in RGB space is obtained.

3. The palette-based image recoloring method according to claim 1, characterized in that: The step of clustering the image pixels in the convex hull of the RGB space to obtain the cluster center includes: Each pixel in the convex hull of the RGB space is regarded as an independent cluster center. Next, mean shift is performed on each pixel. After multiple rounds of mean shift, each pixel converges to a fixed position, and pixels with close positions are classified into the same class.

4. The palette-based image recoloring method according to claim 1, characterized in that: The screening of the cluster center as a representative color comprises: Delete the cluster centers whose distance from the convex hull vertex is less than r, where r is the radius of the mean shift.

5. A palette-based image recoloring system, characterized in that: The system comprises: Convex hull acquisition module, used to extract the convex hull of the RGB space corresponding to the input image; A clustering module, used for clustering image pixels in the convex hull of the RGB space to obtain cluster centers, and selecting the cluster centers as representative colors, wherein the representative colors corresponding to the convex hull vertices and the cluster centers together form a color palette; A partitioning module is used to perform tetrahedral partitioning on the vertex set corresponding to the palette and calculate the interpolation weight of the pixel point with respect to the palette color; A recoloring module, used for modifying the color of the palette and implementing image recoloring editing using the interpolation weights; The method of tetrahedroning the vertex set corresponding to the palette includes: All cluster center points and convex hull vertices of representative colors are organized together to form a vertex set, and the vertex set is subjected to Delaunay tetrahedron decomposition; The calculating the interpolation weight of the pixel point with respect to the color palette includes: Inside the tetrahedron ABCD, the interpolation weight of any pixel point p corresponding to vertex A can be simply expressed as: Where, T(pBCD) represents the volume of the tetrahedron formed by the 4 vertices of pBCD, and T(ABCD) represents the volume of the tetrahedron ABCD; The modifying of the color of the palette comprises: The color set of the palette is marked as V, and the pixel set of the input image is marked as P; Interpolate all pixels of the input image, as shown in the formula: Represent any pixel point p∈P as a convex combination of the palette V; where Represents pixel p with respect to palette color V i The interpolation weights satisfy Modify one or more colors of the palette V and substitute the modified palette V′ into the formula You can get the edited image color.

6. A computer-readable storage medium, characterized in that: The computer program for palette-based image recoloring is stored therein, wherein the computer program enables a computer to execute the palette-based image recoloring method according to any one of claims 1 to 4.

7. An electronic device, characterized in that: include: one or more processors; Memory; as well as One or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the programs comprising instructions for executing the palette-based image recoloring method according to any one of claims 1 to 4.

Images