Color transfer method combining global and superpixel segmentation

Abstract

The present invention firstly performs segmentation processing on the source image and the target image by adopting the simple linear iterative clustering method of gradient descent, then searches for the most similar superpixel in the target image for each superpixel in the source image, and then performs color migration between the matched superpixel pairs, and finally eliminates the erroneous color migration and unnatural brightness transition problems caused by the mismatching generated by the over-segmentation processing. Firstly, a fuzzy matrix is ​​introduced in the first matching migration process, and then the global statistical information of the migration result obtained for the first time is used to perform secondary color migration on the source image to obtain a more natural color migration effect; on the one hand, the unnatural migration result problem caused by the erroneous color migration of certain areas generated by over-segmentation is corrected; on the other hand, the application scope of the algorithm is improved, the robustness, robustness and generalization ability of the method are improved, and the color migration effect is improved while improving the efficiency.

Description

Technical Field

[0001] The present invention relates to a method for color migration between images, and in particular to a method for color migration that integrates global and superpixel segmentation, and belongs to the technical field of color migration methods. Background Art

[0002] Color is a very important attribute of an image. There are many ways to change the color distribution and type of an image. However, there is a great correlation between the three channels of the RGB space of image pixels. To change the color information of an image, the three channel components of the image pixels must be changed at the same time to maintain the natural visual effect of the image itself. In order to eliminate the strong correlation between image channels, the prior art has proposed an orthogonal independent color space Lαβ space theory through statistical research on the perception of a large number of natural images and statistical analysis of the distribution law of color information of these images. In the Lαβ space theory, the L channel represents the brightness information of the color, and the α and β channels represent the color information of the color. The Lαβ color space is more in line with the visual perception characteristics of the human eye, and the correlation between channels is minimized. The Lαβ color space theory lays the theoretical foundation for the color migration method.

[0003] Color migration between images refers to the process of obtaining a new image by learning the color information of another or multiple images while maintaining its own structural content information. Color migration can not only realize the mutual conversion of colors between color images, but also be used to colorize grayscale images. Therefore, it has important applications in film and television editing, medical image processing, image color rendering and other fields.

[0004] The global color migration methods in the prior art are all global adaptive color migration methods in a statistical sense. When processing natural images with complex color information, it is easy to cause color erroneous migration, resulting in the effect of color migration not meeting the expected goal. The prior art proposes a color migration algorithm between color images based on the global statistical information of the image. The successful application of this algorithm has promoted the development of color migration technology. The theoretical basis is that images with the same or similar statistical information should have the same or similar visual effects. This view is a law obtained by statistical analysis of a large number of natural images. Therefore, by modifying the pixel mean variance information of the shape image to align it with these statistical features of the color image, the color information of the color image can be migrated to the shape image, that is, the shape image can obtain the color information of the color image by learning the basic statistical information of the color image. After the color migration algorithm was proposed, color migration technology has attracted widespread attention and has been developed and improved from various angles.

[0005] At present, the existing color migration algorithms can be roughly divided into three categories, including: migration technology based on global statistical characteristics of images, migration technology based on interactive outlining, and automatic migration technology based on sub-regions based on image segmentation theory.

[0006] First, color migration based on global statistical features. This type of color migration algorithm starts with the global color migration algorithm. Specifically, it counts the mean and variance of the pixels in each channel of the source image and the target image respectively, and then migrates the mean and variance statistical information of the target image to the shape image, so that the source image has the color distribution characteristics of the target image. The color migration algorithm based on global statistical information is simple and fast to implement, and can achieve better migration effects for color migration between images with relatively simple color gradations. However, for color migration between images with rich color details, over-learning is prone to occur, resulting in the color transition of the image learned by the shape image being not natural enough. It is not enough to just transfer low-order statistical information. The distribution of pixels at the boundaries such as the edge contour of the image has certain non-Gaussianity. In order to improve the robustness of global color migration between images, the existing technology also migrates the high-order moment slope and kurtosis of the image. Specifically, the third-order moment and fourth-order moment of the source image pixel data are adjusted through power transformation and modular transformation to align them with the high-order statistical information of the target image. As a result, the high-order moment information of the color image is transferred to the shape image, so that the color migration effect of the image is better; or an image pyramid is introduced in the color migration, and the sub-band components of the image in different directions at multiple scales are obtained by convolution operation on the shape image and the color image, and then the statistical information is migrated between the corresponding sub-bands according to the local statistical information of each sub-band. Different color migration effects are obtained according to different selections. In general, the color migration algorithm based on global statistical information is relatively simple, and can achieve better results when the shape image and the color image are similar and the structure content is relatively simple. Otherwise, the problem of excessive color migration will occur.

[0007] Second, based on the user interactive color migration algorithm, the color of some pixels of the source image is first migrated during the color migration. The specific method is that the user adds some colored lines as the initial seed points, and then performs color diffusion migration according to the neighborhood similarity of the image. By introducing manual interaction to select the type of color migration of the shape image, it is possible to select a suitable color for the image and avoid the uncertainty of the color migration effect of the global color migration algorithm. However, this type of method is prone to excessive color diffusion at the boundary of the region. Compared with the color migration algorithm based on global statistical information, this type of interactive method can increase the controllability of the color migration results and achieve better color migration effects. However, this interactive migration method requires manual participation and requires the user to have a high level of professional and knowledge background, which is obviously not in line with the development trend of automated batch processing required by image processing work.

[0008] Third, the color migration algorithm based on image segmentation, based on the color perception characteristics of the human visual system, the existing technology roughly classifies the colors that are obviously perceived by the human eye into 11 categories through statistical analysis of the color perception of a large number of natural images, and then obtains a more ideal color migration result by classifying the shape image and the color image according to these 11 color categories, and then performing partitioned color migration in these 11 color areas. The color migration algorithm based on image segmentation takes into account the differences in image structure, pays more attention to the local structural information of the image, and replaces the global statistical information with local statistical information migration. In terms of effect, it can achieve a more stable migration effect than the global migration method, but the algorithm is inefficient in the segmentation and matching process, is prone to over-segmentation problems, and increases system overhead.

[0009] The existing technology still does not solve the problem of color migration between images. The difficulties of the existing technology and the problems solved by the present invention are mainly concentrated in the following aspects:

[0010] First, there are many global methods and methods based on local similarity of images in the existing color migration methods. Among them, the global method is simple to apply, but when the color gradation of the image is complex, it is easy to over-render the color. The local similarity-based methods mainly include interactive local migration with manual intervention and local similarity adaptive matching methods. The interactive migration method with manual intervention can achieve better migration effect than the global method, but this method increases the workload of users and is not suitable for batch processing of images. Therefore, the local similarity adaptive matching method is particularly important. There is no color migration method with excellent performance, stable color migration effect and strong generalization ability in the existing technology.

[0011] Second, the global color transfer methods in the prior art are all global adaptive color transfer methods in a statistical sense. When processing natural images with complex color information, it is easy to cause color transfer errors, resulting in the color transfer effect not meeting the expected goal, and it is easy to cause over-learning, resulting in the color of the image learned from the shape image being too unnatural. The color transfer algorithm based on global statistical information is relatively simple, and can achieve relatively good results when the shape image and the color image are similar and the structure content is relatively simple. Otherwise, the problem of over-color transfer will occur. In some cases, the global color transfer method in the prior art will become very unstable and cannot achieve the theoretical effect.

[0012] Third, the existing methods are not ideal for color migration between images with complex tones and textures. In the field of image processing, most image processing operations are performed on a single image pixel, without considering the spatial information between pixels, which leads to unsatisfactory processing effects and low efficiency of the algorithm. In the process of regional color migration between matching superpixels, the existing technology first colors the seed point of each superpixel in the source image, and then diffuses and migrates other pixels in the superpixel based on similarity. However, this method is complex and prone to excessive edge diffusion problems. The texture structure and color of each region after the source image and the target image are segmented are relatively simple, and the difference in structure between the source image and the target image is too large, resulting in unnatural color migration.

[0013] Fourth, the existing technology is based on user interactive color migration algorithms, which can easily cause excessive color diffusion at the boundaries of the region. Compared with the color migration algorithms based on global statistical information, this type of interactive method can increase the controllability of the color migration results and achieve better color migration effects. However, this interactive migration method requires manual participation and requires the user to have a high level of professional and knowledge background, which is obviously not in line with the development trend of automated batch processing required by image processing work.

[0014] Fifth, the existing color migration algorithm based on image segmentation can achieve a more stable migration effect than the global migration method, but the algorithm is inefficient in the segmentation and matching process, and is prone to over-segmentation problems. The color migration results are not ideal and the system overhead is increased. Summary of the invention

[0015] In view of the shortcomings of the prior art, the present invention, based on an in-depth analysis of classic color migration methods and corresponding improved algorithms, analyzes and verifies the advantages and disadvantages of these algorithms through experimental tests. From the perspective of local image similarity, based on an in-depth theoretical analysis of image superpixel theory, an improved scheme is proposed to integrate the local color migration method based on image segmentation and matching with the global color migration method, which improves the universality and operation efficiency of the algorithm, has strong feasibility in practical applications, and is a simple, efficient and practical color migration method that integrates global and superpixel segmentation.

[0016] In order to achieve the above technical effects, the technical solutions adopted by the present invention are as follows:

[0017] The color migration method that combines global and superpixel segmentation is based on the local similarity of the image. The local color migration and global color migration are merged in stages. When the color of the image is complex, the image is segmented to make the color texture of each area single. Applying local color migration in such a small area can improve the migration effect of the image color. Image segmentation divides the image into several parts according to the similarity rule. The pixels of each part meet a certain similarity, but the difference between classes is large. When the color gradation of the image is complex, the image is segmented according to a certain standard to make the color gradation of each part relatively single, and then the global color migration method is applied to each part.

[0018] Color migration by integrating global algorithm and superpixel segmentation: First, the source image and the target image are segmented by using the simple linear iterative clustering method of gradient descent, and then the most similar superpixel is searched in the target image for each superpixel in the source image, and then color migration is performed between the matched superpixel pairs, and finally the color erroneous migration and unnatural brightness transition problems caused by the mismatching caused by the over-segmentation process are eliminated. The present invention first introduces a fuzzy matrix in the first matching migration process, and then uses the global statistical information of the migration result obtained initially to perform a secondary color migration on the source image to obtain a natural color migration effect. The specific process is as follows:

[0019] Step 1: Transform the color space: First, transform the pixel data information of the source image and the target image in the RGB space to the lαβ space to maintain the independence between channels;

[0020] Step 2: Use the gradient descent simple linear iterative clustering method to segment the source image and the target image respectively. The specific number of segmentations is adaptively determined according to the structural similarity of the two images. Then, the features of these superpixels are extracted to construct a feature vector space. For all superpixels in the source image, the best matching superpixels are found in the target image. Many-to-one matching is allowed. For each superpixel area in the source image, a corresponding area with similar texture is found in the target image.

[0021] Step 3. After the matching of superpixel pairs is completed, traverse each superpixel of the source image and migrate the color information of the superpixel in the target image that matches it. The source image obtains color information similar to the target image. In the process of color migration, the fuzzy membership matrix of each superpixel in the source image is calculated for each pixel of the source image, so that the edge of the migrated result image is smoother. The calculation of the membership of the non-seed pixel in a single superpixel to the superpixel considers its color and distance relationship with the center of the superpixel, while the calculation of the membership of other superpixels only considers its color distance relationship with the center of other superpixels; the membership of the seed pixel to the superpixel area where it is located is 1, and the membership of other superpixels is 0;

[0022] Step 4: Using the source image that has acquired the required target image color information as the target image to perform global color migration on the initial source image;

[0023] In step 5, the changed source image data is transformed back to the RGB space to realize the color migration process of the source image.

[0024] A color migration method that combines global and superpixel segmentation. Further, a local similarity measurement of an image: a segmentation method is used in the color migration preprocessing stage to segment the image, and then a local similarity adaptive matching method is used to find a corresponding matching area for each segmented area of ​​the source image. The local similarity adaptive matching method matches according to the similarity features of the local areas of the image. The local similarity of the image mainly measures the texture similarity of the local areas after the image is segmented. The present invention adopts a method of calculating statistics to analyze the texture from the aspects of the spatial distribution of image pixels, boundary distribution and grayscale dependency of image pixels, and extracts the texture features of the image from the grayscale information of the image. Specifically, the grayscale co-occurrence moment is used, which considers the spatial position relationship of pixel grayscale data when describing the texture. It is a function of the spatial position and angle between adjacent pixels. The grayscale co-occurrence matrix table It shows the probability that a certain pixel gray level deviates from a certain position in a certain direction and becomes another gray level. Conversely, the gray level can be restored to its original gray level by moving the same distance in the opposite direction. The gray level co-occurrence matrix data is symmetrically distributed. By selecting a direction and distance relationship, a gray level co-occurrence matrix of the image can be calculated. According to the gray level co-occurrence matrix, various statistics of the image are calculated to measure the representation and extract several texture features of the image. Among them, energy, contrast, inverse difference, entropy and autocorrelation features describe the texture feature parameters of the image. These features extracted according to the gray level co-occurrence moment are comprehensively represented by a feature vector. This feature vector includes five aspects of features in four directions and a total of 20 dimensions. The local similarity of the image is measured according to the gray level co-occurrence moment characteristics of the local area to measure the texture features. The more similar the local area is, the smaller the Euclidean distance between its feature vectors is.

[0025] After local feature extraction of the image, the similarity of these feature vectors is calculated. For the local feature similarity measurement of the image, the present invention adopts cosine similarity. In the same feature space, the local feature vectors of the image are considered to be a directed line segment starting from the origin. The more similar the feature vectors of two regions are, the closer their end points are. On the contrary, the more dissimilar they are, the farther their end points are. When the two vectors are in the same direction, the distance is the shortest, and when they are in opposite directions, the distance is the farthest. The cosine value of the angle between the two vectors just reflects this distance relationship, and the size range of the cosine value is just between 0 and 1. The cosine value of the angle between the two vectors is used as an indicator to measure the similarity of the two feature vectors. The closer the cosine value of the two vectors is to 0, the more dissimilar they are. On the contrary, the closer it is to 1, the more similar they are.

[0026] A color migration method integrating global and superpixel segmentation, and further, superpixel image segmentation matching: Based on the local similarity features of the image and the image segmentation method, the present invention proposes to perform over-segmentation processing on the image during the image color migration process, and uses superpixels to perform effective over-segmentation processing on the image;

[0027] Superpixel image segmentation: Superpixels are used as the basic processing unit in image processing. Several adjacent pixels with similar texture structures, similar color information, and similar brightness information characteristics in the image are combined into a set as superpixels. These small areas can retain the effective information of the image and combine image pixels by using the similarity of features between pixels to reduce the redundancy between image pixels.

[0028] The present invention adopts a simple linear iterative clustering method of gradient descent to generate superpixels of uniform size and regular shape based on color and distance similarity, and performs superpixel segmentation on the image to extract local features of the image well;

[0029] First, the image data is transformed from RGB space to Lαβ space, and then the corresponding three-channel data and XY coordinates are fused to form a five-dimensional feature space for each pixel. The five-dimensional feature vector is then locally clustered according to the metric scale. The segmentation result of the image superpixel segmentation depends on the number of superpixels to be generated, that is, the segmentation accuracy, which is input by the user in advance. The specific generation steps are as follows:

[0030] The first step is to initialize the seed points. If there are M pixels in the image to be processed, the image is divided into W uniform superpixels. The average number of pixels in each superpixel is M / W, and the distance between superpixel centers is approximately equal to In order to prevent the seed point from interfering with the subsequent superpixel segmentation process when it is located at the boundary of the superpixel, the seed point is moved to the position with the minimum pixel gradient value within its 3×3 neighborhood, and then a label is attached to each of these seed points to indicate their category affiliation;

[0031] The second step is to calculate pixel similarity. We traverse all pixels in the entire image and compare their distances with W seed points by statistics, and assign the label category to the seed point closest to it.

[0032]

[0033]

[0034]

[0035] where a Lαβ Represents the distance relationship between pixel colors, a xyIt measures the positional relationship between pixels. i It measures the proximity between two pixels. i The value of is the weighted sum of the color distance and spatial distance of two pixels. Its value indicates the similarity of the two pixels. c represents the average spatial distance of any two sub-points. n is the weight adjustment factor for adjusting the color value difference and position difference in the similarity measurement.

[0036] The third step is to continuously iterate and update until the final constraint function converges. To speed up the search process, when searching for similar pixels to all seed points to complete the region division, only the 2c×2c neighborhood of each seed pixel is searched for pixels close to the seed point.

[0037] The color migration method combines global and superpixel segmentation, and further extracts superpixel features: based on the image structure and texture features, first select Gabor features and SURF features that are insensitive to brightness changes and have stability, and then use the grayscale image colorization method to extract the brightness features and variance features of the low-order statistical features of the image.

[0038] The color migration method integrated with global and superpixel segmentation, further, superpixel brightness feature extraction: The superpixel brightness feature extracted by the present invention not only counts the neighborhood brightness mean information of the seed point in each superpixel, but also considers the spatial relationship of the superpixels, that is, the average brightness of the neighborhood of the seed point in the superpixel area adjacent to the superpixel. The neighborhood is the 3×3 window area of ​​the corresponding seed point. Assume that c pi is the i-th superpixel of the image, and c pi The central neighborhood of is composed of n = 9 pixels, c pi The first dimension feature of brightness is the average brightness of the pixels in the neighborhood of the seed point that constitutes the superpixel, and the calculation formula is:

[0039]

[0040] Where J(x,y) represents the brightness value of the pixel at (x,y), c pi The second dimension brightness feature is c pi The one-dimensional feature mean of the neighborhood superpixel center and the neighborhood pixel brightness is expressed as:

[0041]

[0042] In the formula Represents c pi The area composed of the neighborhood superpixels of , w represents the number of superpixels in the area;

[0043] The superpixel brightness feature extracted by the present invention is a combination of these two aspects, that is, the first-dimensional superpixel feature and the second-dimensional superpixel feature constitute a two-dimensional feature of a single superpixel brightness, which not only measures the brightness feature of the superpixel itself, but also takes into account the environment in which it is located.

[0044] The color migration method integrated with global and superpixel segmentation, further, superpixel standard deviation feature extraction: the superpixel standard deviation feature is the variance mean of the pixels in the superpixel and its adjacent superpixels. The first dimension standard deviation feature of the image superpixel extracted by the present invention is the three-channel standard deviation mean of all pixels in the superpixel, and is expressed by c pi represents the i-th superpixel of the image, c pi The standard deviation is c pi The mean of the three-channel standard deviation of all pixels in the area is calculated as follows:

[0045]

[0046] Among them J i (x,y) is the single channel value at (x,y), m is c pi The number of pixels contained in c pi The second dimension characteristic expression of the standard deviation is:

[0047]

[0048] where ω is c pi The set of neighborhood superpixels of , M represents the number of superpixels within ω, and these two-dimensional features constitute the two-dimensional feature vector of a single superpixel.

[0049] A color migration method integrating global and superpixel segmentation, further, superpixel Gabor feature extraction: superpixel Gabor features separate the directional features and scale features of an image, the localized frequency representation requires a window of a specific size in the spatial domain, and the frequency domain bandwidth is limited to a quantitative size. A set of filters of different scales are required to detect local features at different scales of the image, and the Gabor transform can well determine the function of the lower bound of the uncertainty relationship between the time domain and the frequency domain, and can well represent the image space domain and frequency domain features in the case of two-dimensional uncertainty, and the obtained different filters are convolved with the image to be processed to obtain the corresponding Gabor feature values ​​of the pixels, first traversing the entire image to calculate the Gabor features of all pixels, including five scales, wherein each scale selects 8 directions (0, π / 8, ..., 7 / 8π), and obtains a 40-dimensional feature vector, and then takes the mean of the Gabor features of all pixel points contained in each superpixel or the Gabor features of the iterated seed points of each superpixel as the overall Gabor feature vector of each superpixel, and the present invention extracts the Gabor features of each seed point.

[0050] The color migration method integrating global and superpixel segmentation, further, superpixel SURF feature extraction: The SURF operator first searches for the interest points of the image. The present invention does not directly detect the interest points of the image but directly takes the center of each superpixel as the interest point of the image, and then constructs its feature descriptor as the feature description of the entire superpixel. The process of extracting the SURF feature of the image based on SURF is as follows:

[0051] Step 1: Determine the main direction of the pixel feature. First, draw a circle with the pixel as the center. Then count the Harr wavelet features of each pixel in the horizontal and vertical directions, and assign corresponding Gaussian weight values ​​to these feature values ​​to ensure that the distance from the current pixel is positively correlated with the Harr feature weights of these pixels. Then, count the Harr features of all pixels in the 60° area centered on the pixel in the circle in both directions and sum them up to obtain a vector of the area. Compare these vectors and select the one with the largest modulus to determine the main direction of the central pixel feature.

[0052] Step 2: Generate pixel feature descriptors. To ensure the rotation stability of the feature, adjust the direction of the coordinate axis to keep it the same as the main direction of the current pixel feature. Then select a square window with the current pixel as the center in the image, and then divide the window into 4×4 uniform parts. Calculate the harr feature a of the pixel points in each small area. x 、a y , and for a x 、a y Assign different Gaussian weights and then accumulate the statistics of a for all pixels in each area x and a y Value, a x Absolute value, a y The sum of absolute values, these values ​​constitute the feature description of each area. For each small square area, there is a four-dimensional feature. The neighborhood of each pixel in the image has 4×4 small areas that constitute a 64-dimensional feature descriptor.

[0053] Step three, expand the feature description vector obtained by the above process to obtain a more accurate pixel feature description. This step is an extension of the statistical results of the previous step. When performing feature statistics based on the Harr wavelet features, the sum of the features greater than 0 and less than 0 in the two directions are counted respectively. The feature vector of each small area becomes eight-dimensional, and the 4×4 small areas form a 128-dimensional feature vector of the current pixel.

[0054] The color migration method that integrates global and superpixel segmentation, further, quickly matches superpixels: first traverse all superpixels of the source image, and for any superpixel of it, find the superpixel that is most similar to it in the target image. The process of fast multi-level superpixel matching adopted by the present invention is:

[0055] Process 1: Search for the closest superpixels in the target image for all superpixels in the source image based on Gabor features, where the similarity between superpixels is measured by the cosine value of their Gabor feature vectors. The smaller the cosine value, the less similar it is, and vice versa. Then, for each superpixel in the source image, search for the W superpixels with the largest cosine value based on the cosine value of the Gabor features of each superpixel in the target image.

[0056] In the second step, all superpixels of the source image are traversed, and the W / 2 most similar superpixels are searched again using the SURF features of the superpixels from the W most similar superpixels of each superpixel obtained by the comparison search in the first step;

[0057] In the third step, all superpixels of the source image are traversed again, and the W / 2×1 / 2 most similar superpixels are found in the W / 2 most similar superpixels of each superpixel according to the cosine similarity of the m-order moment of the superpixel histogram, where the first-dimensional brightness feature and the second-dimensional standard deviation feature in the m-order moment are both two-dimensional vectors; then W / 2×1 / 2×1 / 2 relatively close superpixels are selected again according to the gray-level co-occurrence moment feature of each superpixel, and finally the set of these superpixels is combined with ε i It means that, so with Q i The most similar superpixels satisfy:

[0058] ε a =arg min G(ε b , r i ), ε b ∈ε i

[0059] Where:

[0060] G(ε b , r i ) = k 1 S 1 (ε b , r i )+k 2 S 2 (ε b , r i )+k 3 S 3 (ε b , r i )+k 4 S 4 (εb , r i )

[0061] Where S 1 (ε b , r i ), S 2 (ε b r i ), S 3 (ε b r i ) and S 4 (ε b , r i ) represents ε b and r i The Euclidean distance of Gabor, SURF, gray level histogram m-order moment and gray level co-occurrence moment features, k 1 , k 2 , k 3 and k 4 are the weight coefficients corresponding to these distances;

[0062] In feature extraction, the Gabor features and SURF features are first matched, and then the brightness and variance features are matched.

[0063] The color migration method integrated with global and super-pixel segmentation, further, determines the number of super-pixel segmentation: for the rough determination of the optimal number of super-pixel segmentation, the present invention adaptively determines it according to the structural complexity of the source image itself and the structural similarity of the two images, and the measurement index of the structural similarity between the source image and the target image adopts structural similarity, which is mainly composed of three parts of similarity statistics comprehensive measurement: brightness difference, contrast difference and correlation difference of the two signals;

[0064] (1) Brightness similarity: Brightness similarity is mainly based on the statistical calculation of the mean brightness of the two images. The specific calculation formula is:

[0065]

[0066]

[0067] Where S 1 In order to adjust the factor and avoid division by zero in the calculation of h(x, y);

[0068] (2) Contrast difference: The brightness information variance of the two images is compared. The variance is calculated as:

[0069]

[0070] The contrast function of brightness is calculated as:

[0071]

[0072] S 2 It is also the adjustment factor, M represents the number of superpixels;

[0073] (3) Calculation of structural similarity between two signals:

[0074]

[0075] in:

[0076]

[0077] The measurement formula for structural similarity is:

[0078] JGXSD=h(x,y)*c(x,y)*s(x,y)

[0079] The present invention determines the optimal number of image superpixel segmentations according to the size of the structural similarity. When calculating the structural similarity, the brightness distribution of the source image is first adjusted to align with the target image. If the structural similarity between the source image and the target image is below 65%, it is considered that the similarity is poor and requires fine segmentation. At this time, the number of segmentations is set to 1000; if the structural similarity is above 65%, it is considered that the structural similarity is good and does not require excessively fine segmentation. At this time, the number of segmentations is set to 200.

[0080] Compared with the prior art, the contribution and innovation of the present invention are:

[0081] First, the present invention improves the local migration algorithm. Based on the local similarity of the image, the local color migration and the global color migration are integrated in stages, thereby obtaining a better color migration effect. At the same time, the generalization ability of the method is also significantly improved. When the color of the image is complex, the color texture of each area is made single by segmenting the image, so that the application of local color migration in such a small area can improve the migration effect of the image color. Image segmentation is to divide the image into several parts according to the similarity rule. The pixels of each part meet a certain similarity, and the difference between classes is large. When the color gradation of the image is complex, the color gradation of each part is relatively single by segmenting the image according to a certain standard, and then the global color migration method is applied to each part to obtain a better color migration result.

[0082] Second, based on an in-depth analysis of classic color migration methods and corresponding improved algorithms, the present invention analyzes and verifies the advantages and disadvantages of these algorithms through experimental tests. From the perspective of local image similarity, based on an in-depth theoretical analysis of image superpixel theory, an improved scheme is proposed to integrate the local color migration method based on image segmentation and matching with the global color migration method, thereby improving the universality and operating efficiency of the algorithm, and having strong feasibility in practical applications. It is a simple, efficient and practical color migration method that integrates global and superpixel segmentation.

[0083] Third, the present invention uses superpixels as the basic processing unit in image processing. Several neighboring pixels in the image with similar texture structures, similar color information, and similar brightness information characteristics form a set as superpixels. These small areas can retain the effective information of the image, and use the similarity of the features between pixels to combine the image pixels, reduce the redundancy between the image pixels, facilitate the subsequent work of image processing, and accelerate the calculation efficiency; based on summarizing the traditional color migration algorithm based on global statistical information and the traditional clustering segmentation algorithm, the present invention proposes a migration algorithm that integrates these two color migration ideas in a process, thereby improving the accuracy of color migration and the generalization ability of the algorithm;

[0084] Fourth, the present invention firstly uses the gradient descent simple linear iterative clustering method to segment the source image and the target image, then searches for the most similar superpixel in the target image for each superpixel in the source image, and then performs color migration between the matching superpixel pairs, and finally eliminates the color erroneous migration and unnatural brightness transition problems caused by the mismatching caused by the over-segmentation process. The present invention first introduces a fuzzy matrix in the first matching migration process, and then uses the global statistical information of the migration result obtained initially to perform secondary color migration on the source image, so as to obtain a more natural color migration effect;

[0085] Fifth, the present invention integrates the global migration method on the basis of regional segmentation and matching. Through regional segmentation, the similarities of the local regions of the source image and the target image are fully considered to obtain a more suitable color migration direction. By using the global-based color migration method, on the one hand, the unnatural migration result caused by the erroneous migration of colors in certain regions caused by the previous over-segmentation is corrected; on the other hand, the application scope of the algorithm is improved, the generalization ability of the algorithm is improved, and a more ideal color migration effect can be achieved. It is basically possible to migrate the color information of the target image on the basis of maintaining the structural content of the source image itself, which significantly improves the robustness and robustness of the method, and improves the color migration effect while improving efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0086] Figure 1 This is a schematic diagram of the superpixel segmentation effect using the gradient descent linear iterative clustering method.

[0087] Figure 2 It is a schematic diagram of the method for determining the main direction of the current pixel of the present invention.

[0088] Figure 3 It is a schematic diagram comparing the effects of the method of the present invention and superpixel segmentation.

[0089] Figure 4 It is a schematic diagram of the effect of the color migration method of the present invention. DETAILED DESCRIPTION

[0090] The following is combined with the accompanying drawings to further describe the technical solution of the color migration method that integrates global and superpixel segmentation provided by the present invention, so that technical personnel in the field can better understand the present invention and implement it.

[0091] Color migration between images refers to an image learning the color information of another or multiple images while maintaining its own structural content information, thereby obtaining a new image. Color migration can not only realize the mutual conversion of colors between color images, but also can be used to colorize grayscale images. Therefore, it has important applications in the fields of film and television editing, medical image processing, image color rendering, etc. Among the color migration methods in the prior art, there are many global methods and methods based on local similarity of images. Among them, the global method is simple to apply, but when the color gradation of the image is more complex, it is easy to over-render the color. The local similarity-based method mainly includes interactive local migration with manual intervention and local similarity adaptive matching method. The interactive migration method with manual intervention can achieve better migration effect than the global method, but this method increases the workload of users and is not suitable for batch processing of images. Therefore, the local similarity adaptive matching method is particularly important. The present invention improves the local migration algorithm: based on the local similarity of the image, the local color migration and the global color migration are merged in stages, and a better color migration effect is obtained, and the generalization ability of the method is also significantly improved.

[0092] The global color migration methods in the prior art are all global adaptive color migration methods in a statistical sense. When processing natural images with complex color information, erroneous color migration is prone to occur, resulting in the color migration effect not meeting the expected goal. When the color of the image is complex, the image can be segmented to make the color texture of each area single, so that applying local color migration in such a small area can improve the image color migration effect. Image segmentation is to divide the image into several parts according to the similarity rule. The pixels of each part meet a certain similarity, while the differences between classes are large. The global migration method can achieve better results when performing color migration of images with simple color gradations. When the color gradation of the image is complex, the image can be segmented according to a certain standard to make the color gradation of each part relatively single, and then the global color migration method can be applied to each part to achieve better color migration results.

[0093] 1. Local Similarity Measurement of Images

[0094] In the color migration preprocessing stage, a segmentation method is used to segment the image, and then a local similarity adaptive matching method is used to find a corresponding matching area for each segmented area of ​​the source image. The local similarity adaptive matching method matches according to the similarity characteristics of the local area of ​​the image, and the image area characteristics are mainly manifested as texture characteristics and color characteristics. Color migration is to change the color visual effect of the image. In the area matching process, the texture characteristics of the image are mainly considered. Therefore, the local similarity of the image mainly measures the texture similarity of the local area after the image segmentation. Texture is the change of pixels in the local area of ​​the image, which is expressed by the contrast change between pixels. It is closely related to the environment in which a single pixel is located. The coarse and fine distribution, directionality and change period of the texture are the most significant features of the image structure, and are the main basis for distinguishing between textures. Due to the complexity and diversity of textures, the present invention adopts a method of calculating statistics to calculate the texture of the local area from the spatial distribution of image pixels, boundary classification, etc. The texture is analyzed from the aspects of grayscale dependence of the image pixels, and the texture features of the image are extracted from the grayscale information of the image. Specifically, the grayscale co-occurrence moment is used, which considers the spatial position relationship of the pixel grayscale data when describing the texture. It is a function of the spatial position and angle between adjacent pixels. The grayscale co-occurrence matrix represents the probability that the grayscale of a pixel deviates from a certain position in a certain direction and becomes another grayscale level. Conversely, the grayscale level only needs to move the same distance in the opposite direction to restore the original grayscale level. The grayscale co-occurrence matrix data is symmetrically distributed. A grayscale co-occurrence matrix of the image can be calculated by selecting a direction and distance relationship. In the embodiment, the distance is selected as a pixel unit, and the direction angles are mainly 0°, 45°, 90° and 135°. Various statistics of the image are calculated according to the grayscale co-occurrence matrix to measure the representation, and several texture features of the image are extracted, among which energy, contrast, inverse difference, entropy and autocorrelation features describe the texture feature parameters of the image. These features extracted based on the grayscale co-occurrence moment are comprehensively represented by a feature vector, which includes five features in four directions and a total of 20 dimensions. The local similarity of the image is measured based on the grayscale co-occurrence moment features of the local area to measure the texture features. The more similar the local areas are, the smaller the Euclidean distance between their feature vectors.

[0095] After extracting local features of the image, the similarity of these feature vectors is calculated next. For the local feature similarity measurement of the image, the present invention adopts cosine similarity. In the same feature space, the local feature vectors of the image are considered to be a directed line segment starting from the origin. The more similar the feature vectors of two regions are, the closer their end points are. On the contrary, the more dissimilar they are, the farther their end points are. When the two vectors are in the same direction, the distance is shortest, and when they are in opposite directions, the distance is farthest. The cosine value of the angle between the two vectors just reflects this distance relationship, and the size range of the cosine value is just between 0 and 1. The cosine value of the angle between the two vectors is used as an indicator to measure the similarity of the two feature vectors. The closer the cosine value of the two vectors is to 0, the more dissimilar they are. On the contrary, the closer it is to 1, the more similar they are.

[0096] 2. Superpixel Image Segmentation and Matching

[0097] The methods of the prior art are not ideal for color migration between images with complex tones and textures. Based on the local similarity features of images and image segmentation methods, the present invention proposes to over-segment the image during the image color migration process. In order to avoid the blindness of segmentation and to make good use of the local features of the image, the present invention uses superpixels to effectively over-segment the image.

[0098] 1. Superpixel Image Segmentation

[0099] In the field of image processing, most of the processing operations on images are based on a single image pixel, without considering the spatial information between pixels, which leads to unsatisfactory processing effects and low efficiency of the algorithm. When the human eye observes an image to obtain image information, most of the information comes from a region composed of multiple pixels. The human eye is not sensitive to a single pixel and a single pixel cannot provide meaningful visual information. Only a collection of multiple pixels has a specific meaning. In image processing, superpixels are used as the basic processing unit. Several adjacent pixels with similar texture structures, similar color information, and similar brightness information in the image are combined into a set as superpixels. These small areas can retain the effective information of the image, and the image pixels are combined by using the similarity of the features between pixels, reducing the redundancy between image pixels, facilitating the subsequent work of image processing, and accelerating the efficiency of calculation.

[0100] The present invention adopts a simple linear iterative clustering method based on gradient descent to generate superpixels of uniform size and regular shape based on color and distance similarity. Moreover, these generated superpixels can well maintain the edge of the image, and the local features of the image can be well extracted by performing superpixel segmentation on the image.

[0101] First, the image data is transformed from RGB space to Lαβ space, and then the corresponding three-channel data and XY coordinates are fused to form a five-dimensional feature space for each pixel. The five-dimensional feature vector is then locally clustered according to the metric scale. The segmentation result of the image superpixel segmentation depends on the number of superpixels to be generated, that is, the segmentation accuracy, which is input by the user in advance. The specific generation steps are as follows:

[0102] The first step is to initialize the seed points. If there are M pixels in the image to be processed, the image is divided into W uniform superpixels. The average number of pixels in each superpixel is M / W, and the distance between superpixel centers is approximately equal to In order to prevent the seed point from interfering with the subsequent superpixel segmentation process when it is located at the boundary of the superpixel, the seed point is moved to the position with the minimum pixel gradient value within its 3×3 neighborhood, and then a label is attached to each of these seed points to indicate their category affiliation;

[0103] The second step is to calculate pixel similarity. We traverse all pixels in the entire image and compare their distances with W seed points by statistics, and assign the label category to the seed point closest to it.

[0104]

[0105]

[0106]

[0107] where a Lαβ Represents the distance relationship between pixel colors, a xy It measures the positional relationship between pixels. i It measures the proximity between two pixels. i The value of is the weighted sum of the color distance and spatial distance of two pixels. Its value indicates the similarity of the two pixels. c represents the average spatial distance of any two sub-points. n is the weight adjustment factor for adjusting the color value difference and position difference in the similarity measurement.

[0108] The third step is to continuously iterate and update until the final constraint function converges. To speed up the search process, when searching for similar pixels to all seed points to complete the region division, only the 2c×2c neighborhood of each seed pixel is searched for pixels close to the seed point.

[0109] (II) Superpixel image segmentation results

[0110] The image to be processed is segmented using a simple linear iterative clustering method with gradient descent to obtain a result image consisting of W superpixel regions. The different values ​​of W control the accuracy of image segmentation. The larger the W value, the finer the segmentation and the better the edge details of the image are preserved. However, the selection of the W value will affect the speed of superpixel segmentation. The more segmentations there are, the greater the amount of computational effort in the segmentation process and the longer the time taken. In addition, when the segmentation is too fine, the extraction of regional features is not utilized. Figure 1 The superpixel segmentation effect is the result of applying the gradient descent linear iterative clustering method to the image.

[0111] 3. Extracting Superpixel Features

[0112] In the color migration preprocessing stage, the source image and the target image are finely segmented, and then the features of a single superpixel are extracted for matching. The selection of features will affect the matching results, and then affect the accuracy of subsequent color migration. Therefore, feature selection is related to the success or failure of color migration. Based on image structure and texture features, the present invention first selects Gabor features and SURF features that are insensitive to brightness changes and have stability, and then adopts the brightness features and variance features of the low-order statistical features of the image using the grayscale image colorization method.

[0113] 1. Superpixel brightness feature extraction

[0114] The superpixel brightness feature extracted by the present invention not only counts the neighborhood brightness mean information of the seed point in each superpixel, but also considers the spatial relationship of the superpixels, that is, the average brightness of the seed point neighborhood of the superpixel area adjacent to the superpixel. The neighborhood is the 3×3 window area of ​​the corresponding seed point. Assuming c pi is the i-th superpixel of the image, and c pi The central neighborhood of is composed of n = 9 pixels, c pi The first dimension feature of brightness is the average brightness of the pixels in the neighborhood of the seed point that constitutes the superpixel, and the calculation formula is:

[0115]

[0116] Where J(x,y) represents the brightness value of the pixel at (x,y), c pi The second dimension brightness feature is c pi The one-dimensional feature mean of the neighborhood superpixel center and the neighborhood pixel brightness is expressed as:

[0117]

[0118] In the formula Represents c pi The area composed of the neighborhood superpixels of , w represents the number of superpixels in the area;

[0119] The superpixel brightness feature extracted by the present invention is a combination of these two aspects, that is, the first-dimensional superpixel feature and the second-dimensional superpixel feature constitute a two-dimensional feature of a single superpixel brightness, which not only measures the brightness feature of the superpixel itself, but also takes into account the environment in which it is located.

[0120] (II) Superpixel standard deviation feature extraction

[0121] The superpixel standard deviation feature is the variance mean of the pixels in the superpixel and its adjacent superpixels. The first-dimensional standard deviation feature of the image superpixel extracted by the present invention is the three-channel standard deviation mean of all pixels in the superpixel, expressed as c pi represents the i-th superpixel of the image, c pi The standard deviation is c pi The mean of the three-channel standard deviation of all pixels in the area is calculated as follows:

[0122]

[0123] Among them J i (x,y) is the single channel value at (x,y), m is c pi The number of pixels contained in c pi The second dimension characteristic expression of the standard deviation is:

[0124]

[0125] where ω is c pi The set of neighborhood superpixels of , M represents the number of superpixels within ω, and these two-dimensional features constitute the two-dimensional feature vector of a single superpixel.

[0126] (III) Superpixel Gabor feature extraction

[0127] The superpixel Gabor feature separates the directional features and scale features of the image and is insensitive to changes in brightness. The localized frequency representation requires a window of a specific size in the spatial domain, and the frequency domain bandwidth is limited to a quantitative size. However, the localized frequency description cannot fully describe all local features of the image. A set of filters of different scales are required to detect local features at different scales of the image. The Gabor transform can well determine the function of the lower bound of the uncertainty relationship between the time domain and the frequency domain. It can well describe the image space domain and frequency domain features in the case of two-dimensional uncertainty. The obtained different filters are convolved with the image to be processed to obtain the corresponding Gabor feature value of the pixel. First, the entire image is traversed to calculate the Gabor features of all pixels, including five scales in total, where each scale selects 8 directions (0, π / 8, ..., 7 / 8π) to obtain a 40-dimensional feature vector. Then, the mean of the Gabor features of all pixel points contained in each superpixel or the Gabor features of the iterated seed points of each superpixel are taken as the overall Gabor feature vector of each superpixel. The present invention extracts the Gabor features of each seed point.

[0128] (IV) Superpixel SURF feature extraction

[0129] SURF has a fast operation speed and can maintain better robustness when matching feature points in multiple images, and is more adaptable to changes in image brightness. SURF features are accelerated features that contain a certain degree of stability. The SURF operator first searches for points of interest in the image. The present invention does not directly detect the points of interest in the image, but directly uses the center of each superpixel as the point of interest in the image, and then constructs its feature descriptor as the feature description of the entire superpixel. The process of extracting SURF features of an image based on SURF is as follows:

[0130] Step 1: Determine the main direction of the pixel feature. First, draw a circle with the pixel as the center. Then count the Harr wavelet features of each pixel in the horizontal and vertical directions, and assign corresponding Gaussian weight values ​​to these feature values ​​to ensure that the distance from the current pixel is positively correlated with the Harr feature weights of these pixels. Then, count the Harr features of all pixels in the 60° area centered on the pixel in the circle in both directions and sum them up to obtain a vector of the area. Compare these vectors and select the one with the largest modulus to determine the main direction of the central pixel feature. Figure 2 Determines the method for the main direction of the current pixel.

[0131] Step 2: Generate pixel feature descriptors. To ensure the rotation stability of the feature, adjust the direction of the coordinate axis to keep it the same as the main direction of the current pixel feature. Then select a square window with the current pixel as the center in the image, and then divide the window into 4×4 uniform parts. Calculate the harr feature a of the pixel points in each small area.x 、a y , and for a x 、a y Assign different Gaussian weights and then accumulate the statistics of a for all pixels in each area x and a y Value, a x Absolute value, a y The sum of absolute values, these values ​​constitute the feature description of each area. For each small square area, there is a four-dimensional feature. The neighborhood of each pixel in the image has 4×4 small areas that constitute a 64-dimensional feature descriptor.

[0132] Step three, expand the feature description vector obtained by the above process to obtain a more accurate pixel feature description. This step is an extension of the statistical results of the previous step. When performing feature statistics based on the Harr wavelet features, the sum of the features greater than 0 and less than 0 in the two directions are counted respectively. The feature vector of each small area becomes eight-dimensional, and the 4×4 small areas form a 128-dimensional feature vector of the current pixel.

[0133] 4. Fast Matching Superpixels

[0134] First, all superpixels of the source image are traversed, and for any superpixel, the most similar superpixel is found in the target image. Since the number and dimension of superpixels are high, direct comparison one by one is computationally intensive and slow. In order to reduce the computational complexity of matching search, the present invention adopts the process of fast multi-level superpixel matching as follows:

[0135] Process 1: Search for the closest superpixel in the target image for all superpixels in the source image based on Gabor features, where the similarity between superpixels is measured by the cosine value of their Gabor feature vectors. The smaller the cosine value, the less similar it is, and vice versa. Then, for each superpixel in the source image, search for W superpixels with the largest cosine value based on the cosine value of the Gabor feature of each superpixel in the target image. In the embodiment, W=20.

[0136] In the second step, all superpixels of the source image are traversed, and the W / 2 most similar superpixels are searched again using the SURF features of the superpixels from the W most similar superpixels of each superpixel obtained by the comparison search in the first step;

[0137] In the third step, all superpixels of the source image are traversed again, and the W / 2×1 / 2 most similar superpixels are found in the W / 2 most similar superpixels of each superpixel according to the cosine similarity of the m-order moment of the superpixel histogram, where the first-dimensional brightness feature and the second-dimensional standard deviation feature in the m-order moment are both two-dimensional vectors; then W / 2×1 / 2×1 / 2 relatively close superpixels are selected again according to the gray-level co-occurrence moment feature of each superpixel, and finally the set of these superpixels is combined with ε iIt means that, so with Q i The most similar superpixels satisfy:

[0138] ε a =arg min G(ε b , r i ), ε b ∈ε i

[0139] Where:

[0140] G(ε b r i ) = k 1 S 1 (ε b , r i )+k 2 S 2 (ε b , r i )+k 3 S 3 (ε b , r i )+k 4 S 4 (ε b , r i )

[0141] Where S 1 (ε b , r i ), S 2 (ε b , r i ), S 3 (ε b , r i ) and S 4 (ε b , r i ) represents ε b and r i The Euclidean distance of Gabor, SURF, gray level histogram m-order moment and gray level co-occurrence moment features, k 1 , k 2 , k 3 and k 4 are weight coefficients corresponding to these distances, which are assigned values ​​of 0.3, 0.4, 0.2 and 0.1 during the calculation process of the embodiment;

[0142] In the grayscale image colorization algorithm, when the number of image segmentation is small, using grayscale mean and variance for matching can achieve better results. However, when the number of segmentation is large when the image is over-segmented, using only these two features for matching is prone to more mismatches. The reason is that there are many areas with similar brightness but different colors in the image, while Gabor features and SURF features are not sensitive to changes in brightness and can extract finer features of different scales and directions. Therefore, in feature extraction, matching is first performed based on Gabor features and SURF features, and then based on brightness and variance features.

[0143] 5. Color Migration by Fusion of Global Algorithm and Superpixel Segmentation

[0144] First, the source image and the target image are segmented by using a simple linear iterative clustering method using gradient descent. Then, the most similar superpixel is searched in the target image for each superpixel in the source image. Then, color migration is performed between the matched superpixel pairs. Finally, the color erroneous migration and unnatural brightness transition caused by the mismatching caused by the over-segmentation process are eliminated. The present invention first introduces a fuzzy matrix in the first matching migration process, and then uses the global statistical information of the migration result obtained initially to perform a secondary color migration on the source image to obtain a more natural color migration effect. The specific process is as follows:

[0145] Step 1: Transform the color space: First, transform the pixel data information of the source image and the target image in the RGB space to the lαβ space to maintain the independence between channels;

[0146] Step 2: Use the gradient descent simple linear iterative clustering method to segment the source image and the target image respectively. The specific number of segmentations is adaptively determined according to the structural similarity of the two images. Then, the features of these superpixels are extracted to construct a feature vector space. For all superpixels in the source image, the best matching superpixels are found in the target image. Many-to-one matching is allowed. For each superpixel area in the source image, a corresponding area with similar texture is found in the target image.

[0147] Step 3. After the matching of superpixel pairs is completed, traverse each superpixel of the source image and migrate the color information of the superpixel in the target image that matches it. The source image obtains color information similar to the target image. In the process of color migration, the fuzzy membership matrix of each superpixel in the source image is calculated for each pixel of the source image, so that the edge of the migrated result image is smoother. The calculation of the membership of the non-seed pixel in a single superpixel to the superpixel considers its color and distance relationship with the center of the superpixel, while the calculation of the membership of other superpixels only considers its color distance relationship with the center of other superpixels; the membership of the seed pixel to the superpixel area where it is located is 1, and the membership of other superpixels is 0;

[0148] Step 4: Using the source image that has acquired the required target image color information as the target image to perform global color migration on the initial source image;

[0149] In step 5, the changed source image data is transformed back to the RGB space to realize the color migration process of the source image.

[0150] Figure 3 This is a comparison diagram of the effects of the method of the present invention and superpixel segmentation.

[0151] (I) Determine the number of superpixel segmentations

[0152] For the determination of the optimal number of superpixel segmentation in the preprocessing stage of image segmentation by gradient descent simple linear iterative clustering method, the number of superpixels for image segmentation is directly related to the image size, and an optimal number of segmentation is determined for the same image. By observing the result image, when the number of superpixels is too small, the superpixel segmentation method does not maintain the image edge well, and the image color migration effect is not good, which means that when the number of segmentation is too small, the extracted features are inaccurate when the image is segmented by superpixel, so in order to achieve better segmentation effect, more refined segmentation processing should be performed, but the number of segmentation should not be too much. When the number of segmentation is too large, the image is cut too finely, and each too small segmentation area is more similar, which is not conducive to the effective extraction of features and increases the relevant calculation amount, affecting the operation efficiency of the algorithm.

[0153] For the rough determination of the optimal number of superpixel segmentations, the present invention adaptively determines the structural similarity of the source image and the target image based on the structural complexity of the source image itself and the structural similarity of the two images. The structural similarity SSIM is used as the measurement index of the structural similarity between the source image and the target image, which is mainly composed of three similarity statistical comprehensive measurements: brightness difference, contrast difference and correlation difference between the two signals.

[0154] (1) Brightness similarity: Brightness similarity is mainly based on the statistical calculation of the mean brightness of the two images. The specific calculation formula is:

[0155]

[0156]

[0157] Where S 1 In order to adjust the factor and avoid division by zero in the process of calculating h(x,y), S is taken in the embodiment. 1 =0.01;

[0158] (2) Contrast difference: The brightness information variance of the two images is compared. The variance is calculated as:

[0159]

[0160] The contrast function of brightness is calculated as:

[0161]

[0162] S 2 It is also an adjustment factor. In the embodiment, S 2 =0.02;

[0163] (3) Calculation of structural similarity between two signals:

[0164]

[0165] in:

[0166]

[0167] The measurement formula for structural similarity is:

[0168] JGXSD=h(x,y)*c(x,y)*s(x,y)

[0169] For images with large structural complexity or poor structural similarity between two images, more refined segmentation is performed to improve the accuracy of superpixel matching and the accuracy of color migration. The measurement of the structural complexity of the source image itself is based on its grayscale co-occurrence moment information to count its energy, contrast, inverse difference moment and entropy, and then compared with the corresponding empirical critical value. For the structural similarity of the two images, the more dissimilar the two images are, the more refined segmentation is required. Therefore, the present invention determines the optimal number of image superpixel segmentations according to the size of the structural similarity. In order to eliminate the influence of the excessive difference in brightness of the two images on the similarity results, when calculating the structural similarity, the brightness distribution of the source image is first adjusted to align with the target image. If the structural similarity between the source image and the target image is below 65%, it is considered that the similarity is poor and needs fine segmentation. At this time, the number of segmentations is set to 1000; if the structural similarity is above 65%, it is considered that the structural similarity is good and does not require excessive fine segmentation. At this time, the number of segmentations is set to 200.

[0170] (II) Correcting incorrect matching areas

[0171] The selection of color migration in the process of regional color migration between matching superpixels is to first color the seed point of each superpixel in the source image, and then diffuse and migrate other pixel points in the superpixel according to similarity. However, this method is complicated and prone to the problem of excessive edge diffusion, and the texture structure and color of each region after the source image and the target image are over-segmented are relatively simple. Therefore, the color migration processing method of the present invention is as follows: first, superpixels are used for over-segmentation processing. Due to incomplete feature extraction, matching errors of local superpixels will occur, resulting in local color migration errors of regional color migration based on feature matching. In order to obtain a migration result with more natural color transition, the result image is subsequently corrected to eliminate the unnatural problem of color migration. The solution to this problem is that the areas in the source image with the same color before and after migration are also the same after migration. A consistent color visual effect should be obtained, and clustering migration should be performed according to the main color of the image. The specific method is to first adaptively obtain the number of main colors of the image according to the main color histogram of the image, and then use k-means clustering processing to count the category to which each pixel in the source image belongs and label it accordingly. Then, the source image is applied with a superpixel segmentation and matching algorithm. The color statistical mean and variance of the original label class in the migration result initially obtained are migrated to the label area corresponding to the source image to obtain an ideal and natural color migration effect. However, this method can only be used to correct erroneous migration caused by matching errors in only a small number of areas in the source image. Another idea for correcting erroneous migration of local area colors is the method of the present invention, which is to directly migrate the statistical information of the color migration result of the source image obtained initially to the source image. This method is relatively simple and can obtain ideal experimental results.

[0172] Compared with the clustering-based color transfer algorithm, the superpixel-based image segmentation is an over-segmentation. Each superpixel in the source image and the target image can be regarded as a feature point. The metric matching process of the superpixel is the matching process of the feature points. These feature points of the target image are automatically adaptively acquired using superpixels to avoid the arbitrariness and subjectivity of manual random selection. Compared with the global transfer method, the color transfer algorithm based on superpixel segmentation selects similarities based on texture information when matching, and does not require one-to-one matching. When applying the global color transfer algorithm, only part of the color information of the target image is used, not all of the color information. In addition, the color transfer process of the color transfer algorithm based on superpixel matching is an indirect color transfer method, which to a certain extent avoids the problem of unnatural color transfer caused by the large difference in the structure of the source image and the target image when applying the global color transfer algorithm.

[0173] The present invention integrates a global migration method on the basis of region segmentation and matching. Through region segmentation, the similarities of local regions of the source image and the target image are fully considered to obtain a more suitable color migration direction. By using a global-based color migration method, on the one hand, the unnatural migration result caused by the erroneous migration of colors in certain regions caused by the previous over-segmentation can be corrected; on the other hand, the application scope of the algorithm can be improved and the generalization ability of the algorithm can be improved.

[0174] (III) Experimental results of the method of the present invention

[0175] Figure 4 To demonstrate the effect of image color migration, it can be found from the test result graph that the present invention can achieve a relatively ideal color migration effect, and can basically migrate the color information of the target image on the basis of maintaining the structural content of the source image itself. The present invention analyzes and tests the principle of the color migration algorithm, aims at the shortcomings of the existing algorithm, and improves and optimizes it from the perspective of local image similarity, which significantly improves the robustness and robustness of the method. The present invention is committed to an adaptive and automated color migration method, and proposes a color migration method that integrates global and superpixel segmentation. In view of the problems that the color migration algorithm in the prior art is prone to more color over-rendering, and the low efficiency and narrow application range of the segmentation matching algorithm, an algorithm that integrates the local over-segmentation migration and the global migration method is proposed, which improves the color migration effect while improving the efficiency, and the generalization ability is significantly enhanced.

Claims

1. Color migration method combining global and superpixel segmentation, It is characterized in that Based on the local similarity of the image, the local color migration and global color migration are merged in stages. When the color of the image is complex, the image is segmented to make the color texture of each area single. Applying local color migration in such a small area can improve the migration effect of the image color. Image segmentation divides the image into several parts according to the similarity rule. The pixels of each part meet a certain similarity, and the difference between classes is large. When the color gradation of the image is complex, the image is segmented according to a certain standard to make the color gradation of each part relatively single, and then the global color migration method is applied to each part. Color migration by integrating global algorithm and superpixel segmentation: First, the source image and the target image are segmented by using the simple linear iterative clustering method of gradient descent. Then, the most similar superpixel is searched in the target image for each superpixel in the source image. Then, color migration is performed between the matched superpixel pairs. Finally, the color erroneous migration and unnatural brightness transition caused by the mismatch caused by over-segmentation are eliminated. First, the fuzzy matrix is ​​introduced in the first matching migration process, and then the global statistical information of the migration result obtained initially is used to perform secondary color migration on the source image to obtain a natural color migration effect. The specific process is as follows: Step 1: Transform the color space: First, transform the pixel data information of the source image and the target image in the RGB space to the lαβ space to maintain the independence between channels; Step 2: Use the gradient descent simple linear iterative clustering method to segment the source image and the target image respectively. The specific number of segmentations is adaptively determined according to the structural similarity of the two images. Then, the features of these superpixels are extracted to construct a feature vector space. For all superpixels in the source image, the best matching superpixels are found in the target image. Many-to-one matching is allowed. For each superpixel area in the source image, a corresponding area with similar texture is found in the target image. Step 3. After the matching of superpixel pairs is completed, traverse each superpixel of the source image and migrate the color information of the superpixel in the target image that matches it. The source image obtains color information similar to the target image. In the process of color migration, the fuzzy membership matrix of the superpixel is calculated for each pixel of the source image, and the edge of the migrated result image is smoother. The calculation of the membership of non-seed pixels in a single superpixel considers the color and distance relationship of the superpixel center, while the calculation of the membership of other superpixels only considers the color distance relationship between it and the center of other superpixels; the membership of the superpixel area where the seed pixel is located is 1, and the membership of other superpixels is 0; Step 4: Using the source image that has acquired the required target image color information as the target image to perform global color migration on the initial source image; Step 5, transform the changed source image data back to RGB space to realize the color migration process of the source image; Calculate the number of superpixel segmentations: According to the structural complexity of the source image itself and the structural similarity of the two images, the structural similarity between the source image and the target image is measured by structural similarity, which is a statistical comprehensive measure of the three similarities: brightness difference, contrast difference, and correlation difference between the two signals. When calculating the structural similarity, the brightness distribution of the source image is first adjusted to align with the target image. If the structural similarity between the source image and the target image is below 65%, it is considered that the similarity is poor and fine segmentation is required. In this case, the number of segmentations is set to 1000. If the structural similarity is above 65%, it is considered that the structural similarity is good and excessive fine segmentation is not required. In this case, the number of segmentations is set to 200. Correct the wrong matching area: According to the fact that the areas in the source image with the same color before and after migration should also obtain consistent color visual effects after migration, cluster migration is performed according to the main color of the image. The specific approach is to first adaptively obtain the number of main colors of the image based on the main color histogram of the image, and then use clustering to count the category to which each pixel in the source image belongs and label it accordingly. Then, the color statistical mean and variance of the original label class in the migration result initially obtained by applying the superpixel segmentation and matching algorithm to the source image are migrated to the corresponding label area of ​​the source image to obtain an ideal and natural color migration effect, and correct the erroneous migration caused by matching errors in only a small number of areas in the source image. To correct the erroneous migration of local area colors, the statistical information of the initially obtained source image color migration result is directly migrated to the source image.

2. The color migration method combining global and superpixel segmentation according to claim 1, It is characterized in that Local similarity measurement of images: In the color migration preprocessing stage, the segmentation method is used to segment the image, and then the local similarity adaptive matching method is used to find the corresponding matching area for each segmented area of ​​the source image. The local similarity adaptive matching method matches according to the similarity characteristics of the local area of ​​the image. The local similarity of the image mainly measures the texture similarity of the local area after the image is segmented. The method of calculating statistics is used to analyze the texture from the spatial distribution of image pixels, boundary distribution and grayscale dependence of image pixels. The texture features of the image are extracted from the grayscale information of the image. Specifically, the grayscale co-occurrence matrix is ​​used. It considers the spatial position relationship of pixel grayscale data when describing the texture. It is a function of the spatial position and angle between adjacent pixels. The grayscale co-occurrence matrix represents the deviation of the grayscale of a pixel in a certain direction. The probability of changing to another gray level after moving from a certain position. Conversely, the gray level can be restored to its original gray level by moving the same distance in the opposite direction. The gray level co-occurrence matrix data is symmetrically distributed. By selecting a direction and distance relationship, a gray level co-occurrence matrix of the image can be calculated. According to the gray level co-occurrence matrix, various statistics of the image are calculated to measure the representation and extract several texture features of the image. Among them, energy, contrast, inverse difference, entropy and autocorrelation features describe the texture feature parameters of the image. These features extracted according to the gray level co-occurrence moment are comprehensively represented by a feature vector. This feature vector includes five features in four directions and a total of 20 dimensions. The local similarity of the image is measured according to the gray level co-occurrence moment characteristics of the local area to measure the texture features. The more similar the local area is, the smaller the Euclidean distance between its feature vectors is. After local feature extraction of the image, the similarity of these feature vectors is calculated. For the local feature similarity measurement of the image, cosine similarity is used. In the same feature space, the local feature vectors of the image are considered to be a directed line segment starting from the origin. The more similar the feature vectors of the two regions are, the closer their end points are. On the contrary, the more dissimilar they are, the farther their end points are. When the two vectors are in the same direction, the distance is shortest, and when they are in opposite directions, the distance is farthest. The cosine value of the angle between the two vectors just reflects this distance relationship, and the range of the cosine value is just between 0 and 1. The cosine value of the angle between the two vectors is used as an indicator to measure the similarity of the two feature vectors. The closer the cosine value of the two vectors is to 0, the more dissimilar they are. On the contrary, the closer it is to 1, the more similar they are.

3. The color migration method combining global and superpixel segmentation according to claim 1, It is characterized in that Superpixel image segmentation and matching: Based on the local similarity features of images and image segmentation methods, it is proposed to perform over-segmentation on images during the image color migration process, and use superpixels to perform effective over-segmentation on images; Superpixel image segmentation: Superpixels are used as the basic processing unit in image processing. Several adjacent pixels with similar texture structures, similar color information, and similar brightness information characteristics in the image are combined into a set as superpixels. These small areas can retain the effective information of the image and combine image pixels by using the similarity of features between pixels to reduce the redundancy between image pixels. The gradient descent simple linear iterative clustering method is used to generate superpixels of uniform size and regular shape based on color and distance similarity. The image is segmented into superpixels to extract local features of the image well. First, the image data is transformed from RGB space to Lαβ space, and then the corresponding three-channel data and XY coordinates are fused to form a five-dimensional feature space for each pixel. The image pixels are then locally clustered in the five-dimensional feature space according to the metric scale. The segmentation result of the image superpixel segmentation depends on the number of superpixels to be generated input by the user in advance, that is, the segmentation accuracy. The specific generation steps are: The first step is to initialize the seed points. If there are M pixels in the image to be processed, the image is divided into W uniform superpixels. The average number of pixels in each superpixel is M / W, and the spacing between superpixel centers is approximately equal to In order to prevent the seed point from interfering with the subsequent superpixel segmentation process when it is located at the boundary of the superpixel, the seed point is moved to the position with the minimum pixel gradient value within its 3×3 neighborhood, and then a label is attached to each of these seed points to indicate their category affiliation; The second step is to calculate pixel similarity. We traverse all pixels in the entire image and compare their distances with W seed points by statistics, and assign the label category to the seed point closest to it. where a Lαβ Represents the distance relationship between pixel colors, a xy It measures the positional relationship between pixels. i It measures the proximity between two pixels. i The value of is the weighted sum of the color distance and spatial distance of two pixels. Its value indicates the similarity of the two pixels. c represents the average spatial distance of any two sub-points. n is the weight adjustment factor for adjusting the color value difference and position difference in the similarity measurement. The third step is to continuously iterate and update until the final constraint function converges. To speed up the search process, when searching for similar pixels to all seed points to complete the region division, only the 2c×2c neighborhood of each seed pixel is searched for pixels close to the seed point.

4. The color migration method combining global and superpixel segmentation according to claim 1, It is characterized in that Extract superpixel features: Based on the image structure and texture features, first select Gabor features and SURF features that are insensitive to brightness changes and have stability, and then use the grayscale image colorization method to extract the brightness features and variance features of the low-order statistical features of the image.

5. The color migration method combining global and superpixel segmentation according to claim 4, It is characterized in that Superpixel brightness feature extraction: The extracted superpixel brightness feature not only counts the neighborhood brightness mean information of the seed point in each superpixel, but also considers the spatial relationship of the superpixels, that is, the average brightness of the seed point neighborhood in the superpixel area adjacent to the superpixel. The neighborhood is the 3×3 window area of ​​the corresponding seed point. Assume that c pi is the i-th superpixel of the image, and c pi The central neighborhood of is composed of n = 9 pixels, c pi The first dimension feature of brightness is the average brightness of the pixels in the neighborhood of the seed point that constitutes the superpixel, and the calculation formula is: Where J(x,y) represents the brightness value of the pixel at (x,y), c pi The second dimension brightness feature is c pi The one-dimensional feature mean of the neighborhood superpixel center and the neighborhood pixel brightness is expressed as: In the formula Represents c pi The area composed of the neighborhood superpixels of , w represents the number of superpixels in the area; The extracted superpixel brightness feature is a combination of these two features, that is, the first-dimensional superpixel feature and the second-dimensional superpixel feature constitute a two-dimensional feature of a single superpixel brightness. This feature not only measures the brightness feature of the superpixel itself, but also takes into account the environment in which it is located.

6. The color migration method combining global and superpixel segmentation according to claim 4, It is characterized in that Superpixel standard deviation feature extraction: The superpixel standard deviation feature is the variance mean of the pixels in the superpixel and its adjacent superpixels. The first-dimensional standard deviation feature of the extracted image superpixel is the mean of the three-channel standard deviations of all pixels in the superpixel, expressed as c pi represents the i-th superpixel of the image, c pi The standard deviation is c pi The mean of the three-channel standard deviation of all pixels in the area is calculated as follows: Among them J i (x,y) is the single channel value at (x,y), m is c pi The number of pixels contained in c pi The second dimension characteristic expression of the standard deviation is: where ω is c pi The set of neighborhood superpixels of , M represents the number of superpixels within ω, and these two-dimensional features constitute the two-dimensional feature vector of a single superpixel.

7. The color migration method combining global and superpixel segmentation according to claim 4, It is characterized in that Superpixel Gabor feature extraction: Superpixel Gabor features separate the directional features and scale features of the image. The localized frequency representation requires a window of a specific size in the spatial domain, and the frequency domain bandwidth is limited to a quantitative size. A set of filters of different scales are required to detect local features at different scales of the image. The Gabor transform can well determine the function of the lower bound of the uncertainty relationship between the time domain and the frequency domain. It can well represent the spatial domain and frequency domain features of the image in the case of two-dimensional uncertainty. The different filters obtained are convolved with the image to be processed to obtain the corresponding Gabor feature values ​​of the pixels. First, the entire image is traversed to calculate the Gabor features of all pixels, including five scales, where each scale selects 8 directions (0, π / 8, ..., 7 / 8π) to obtain a 40-dimensional feature vector. Then, the mean of the Gabor features of all pixels contained in each superpixel or the Gabor features of the iterated seed points of each superpixel are taken as the overall Gabor feature vector of each superpixel. The Gabor features of each seed point are extracted.

8. The color migration method combining global and superpixel segmentation according to claim 4, It is characterized in that Superpixel SURF feature extraction: The SURF operator first searches for interest points in the image. Instead of directly detecting interest points in the image, it directly takes the center of each superpixel as the interest point of the image, and then constructs its feature descriptor as the feature description of the entire superpixel. The process of extracting SURF features of the image based on SURF is as follows: Step 1: Determine the main direction of the pixel feature. First, draw a circle with the pixel as the center. Then count the Harr wavelet features of each pixel in the horizontal and vertical directions, and assign corresponding Gaussian weight values ​​to these feature values ​​to ensure that the distance from the current pixel is positively correlated with the Harr feature weights of these pixels. Then, count the Harr features of all pixels in the 60° area centered on the pixel in the circle in both directions and sum them up to obtain a vector of the area. Compare these vectors and select the one with the largest modulus to determine the main direction of the central pixel feature. Step 2: Generate pixel feature descriptors. To ensure the rotation stability of the feature, adjust the direction of the coordinate axis to keep it the same as the main direction of the current pixel feature. Then select a square window with the current pixel as the center in the image, and then divide the window into 4×4 uniform parts. Calculate the harr feature a of the pixel points in each small area. x 、a y , and for a x 、a y Assign different Gaussian weights and then accumulate the statistics of a for all pixels in each area x and a y Value, a x Absolute value, a y The sum of absolute values, these values ​​constitute the feature description of each area. For each small square area, there is a four-dimensional feature. The neighborhood of each pixel in the image has 4×4 small areas that constitute a 64-dimensional feature descriptor. Step three, expand the feature description vector obtained by the above process to obtain a more accurate pixel feature description. This step is an extension of the statistical results of the previous step. When performing feature statistics based on the Harr wavelet features, the sum of the features greater than 0 and less than 0 in the two directions are counted respectively. The feature vector of each small area becomes eight-dimensional, and the 4×4 small areas form a 128-dimensional feature vector of the current pixel.

9. The color migration method combining global and superpixel segmentation according to claim 1, It is characterized in that Fast matching of superpixels: First, traverse all superpixels of the source image, and for any superpixel, find the most similar superpixel in the target image. The process of fast multi-level superpixel matching is as follows: Process 1: Search for the closest superpixels in the target image for all superpixels in the source image based on Gabor features, where the similarity between superpixels is measured by the cosine value of their Gabor feature vectors. The smaller the cosine value, the less similar it is, and vice versa. Then, for each superpixel in the source image, search for the W superpixels with the largest cosine value based on the cosine value of the Gabor features of each superpixel in the target image. In the second step, all superpixels of the source image are traversed, and the W / 2 most similar superpixels are searched again using the SURF features of the superpixels from the W most similar superpixels of each superpixel obtained by the comparison search in the first step; In the third step, all superpixels of the source image are traversed again, and the W / 2×1 / 2 most similar superpixels are found in the W / 2 most similar superpixels of each superpixel according to the cosine similarity of the m-order matrix of the superpixel histogram, where the first-dimensional brightness feature and the second-dimensional standard deviation feature in the m-order matrix are both two-dimensional vectors; then W / 2×1 / 2×1 / 2 relatively close superpixels are selected again according to the gray level co-occurrence moment feature of each superpixel, and finally the set of these superpixels is combined with ε i It means that, so with Q i The most similar superpixels satisfy: e a =arg min G(e b ,r i ),e b ∈e i Where: G(e b ,r i )=k 1 S 1 (e b ,r i )+k 2 S 2 (e b ,r i )+k 3 S 3 (e b ,r i )+k 4 S 4 (e b ,r i ) Where S 1 (ε b , r i ), S 2 (ε b , r i ), S 3 (ε b , r i ) and S 4 (ε b , r i ) represents ε b and r i Gabor, SURF, gray level histogram m-order matrix and gray level co-occurrence moment feature Euclidean distance, k 1 , k 2 , k 3 and k 4 are the weight coefficients corresponding to these distances; In feature extraction, the Gabor features and SURF features are first matched, and then the brightness and variance features are matched.

10. The color migration method combining global and superpixel segmentation according to claim 1, It is characterized in that Rough determination of the optimal number of superpixel segmentations: (1) Brightness similarity: comparison is based on the statistical calculation of the mean brightness of the two images. The specific calculation formula is: Where S 1 In order to adjust the factor and avoid division by zero in the calculation of h(x,y); (2) Contrast difference: The brightness information variance of the two images is compared. The variance is calculated as: The contrast function of brightness is calculated as: S 2 It is also the adjustment factor, M represents the number of superpixels; (3) Calculation of structural similarity between two signals: in: The measurement formula for structural similarity is: JGXSD=h(x,y)* c(x, y) *s(x,y) The optimal number of superpixel segmentation of the image is determined according to the size of the structural similarity.

Images