An old image inpainting method fusing change detection and color transfer

Abstract

The application discloses an old image repairing method combining change detection and color migration, and comprises the following steps: first, for the content complete gray old image and the content missing color image, the two images are geometrically corrected, matched and aligned; second, change detection is carried out through average brightness, gray level co-occurrence matrix entropy, variance, LBP and Gabor features, and the unchanged area with high confidence is determined; third, the color of the high-confidence corresponding point is used for color migration of the original gray image, so that a preliminary colored image is obtained; fourth, the color of the low-confidence corresponding point is diffused according to the preliminary colored image, so that a repaired image is obtained. The old image can be color restored by the application, so that people can enjoy the original gorgeous color of the old image again.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of computer vision enhancement, and particularly relates to old image content and color restoration. BACKGROUND

[0002] For old images, especially images such as mural images, due to the limitations of acquisition equipment, the content is complete but there is no color information, while existing images are rich in color information but the content information is missing due to erosion, pests and diseases. Traditional methods such as image inpainting, color transfer and image enhancement have achieved certain results in restoring old images and have been widely used, but for specific digital images such as digital murals, painting cultural heritage digital images, color and content restoration has the following difficulties: (1) the existing method of using a inpainting model to restore images based on content missing color images has low accuracy for images with complex texture and rich color details, especially for large-area content missing images; (2) the traditional color transfer model based on reference images has difficulty in finding accurate color feature matching points, and the color transfer process is prone to local region boundary color transition smoothing, resulting in color transfer errors and limitations in the restoration result; (3) the color of old images, especially digital murals, is strongly subjective in image creation, and existing deep learning color transfer methods cannot learn accurate and true color information of murals, resulting in failure of image color restoration. How to fuse the information of gray old images and content missing color images to restore old image information is still a challenging problem. SUMMARY

[0003] To solve the above problems, the purpose of the present application is to provide an old image restoration technology that fuses change detection and color transfer methods. This method has three significant features: (1) an old image restoration framework that fuses change detection and color transfer methods is constructed, a geometrically registered image pair of gray and color images is established, and the change information of the registered images is realized by a change detection model to transfer the color information of the color image to the unchanged area, and a color diffusion model is used to realize the overall coloring of the gray image; (2) in the geometric registration stage, the Chanel Feature of Orientated Gradient (CFOG) method is used to realize the alignment of the same point of the gray image and the color image, and the geometric registration of the image is completed; (3) in the preliminary color transfer stage, the average brightness, gray co-occurrence matrix entropy, variance, Local Binary Patterns (LBP) and Gabor feature are fused to detect the change information of the gray image and the color image, and the unchanged area is determined as the high confidence area of color transfer, and the color transfer is realized.

[0004] To achieve the above object, the application adopts the following technical scheme: an old image restoration method combining change detection and color transfer, comprising the following steps:

[0005] Step 1: obtaining original old image and reference color image, and matching and aligning the two images by CFOG algorithm;

[0006] Step 2: based on average brightness, gray co-occurrence matrix entropy, variance, LBP and Gabor features, calculating feature difference of same point of matched gray image and color image to realize change detection;

[0007] Step 3: using color of high-confidence corresponding point to perform color transfer on original gray image to obtain preliminary colored image;

[0008] Step 4: obtaining color of low-confidence corresponding point according to preliminary colored image to obtain restored image.

[0009] Further, the specific implementation of step 1 is as follows:

[0010] Step 1.1: in order to realize uniform distribution of feature points, the original old image and the reference color image are divided into local small windows respectively, and a certain number of Harris corner points are selected in each window, and the corner points are used as pixel-level features for subsequent image matching;

[0011] Step 1.2: obtaining x-direction gradient size and y-direction gradient size of each corner point pixel under Cartesian coordinates, and obtaining gradient size in 9 bins according to x-direction gradient size and y-direction gradient size, and the calculation formula is:

[0012] g0=|cosθ×g x +sinθ×g y | (1)

[0013] Wherein g x ,g y are x-direction gradient and y-direction gradient, and θ is the central angle of bin, and when 180° is divided into 9 bins, the central angle of each bin is 0, 20, 40,..., 180, so that 9 feature maps with size (H, W) are obtained, H is the height of the image, and W is the width;

[0014] Step 1.3: performing 3-dimensional Gaussian convolution on the 9 feature maps with size (H, W) to generate CFOG descriptor, wherein the three-dimensional Gaussian convolution kernel functions as that two-dimensional XY direction convolution can replace three-linear interpolation to generate direction histogram, and Z direction convolution operates in gradient direction, which can reduce the influence of local geometric distortion on feature vector;

[0015] Step 1.4: Match using the sum of squared differences of CFOG, in order to speed up the operation, the SSD operation is accelerated using the fast Fourier transform (FFT) in the actual operation, the sum of squared differences of CFOG is SSD, wherein the pixel-level features of the two images are represented as D1 and D2 respectively, wherein x is the position of a pixel in the three-dimensional feature representation, T i (x) is a mask on D1(x), wherein T i (x) = 1, and T i (x) = 0; in addition, S i (v) is the SSD similarity function between D1 and D2 in the template window by the vector v, v is the difference allowed by the matching of D1 and D2, and the matching between D1 and D2 is achieved by minimizing S i (v);

[0016]

[0017] The matching function is further defined as:

[0018] V i = argmin v (∑ x [D1(x)-D2(x-v)] 2 T i (x)) (3)

[0019] The above formula (2) represents the position of the minimum SSD of the two matching windows, and after expansion, it can be obtained:

[0020]

[0021] Since the first term in formula (4) is a constant, the minimum value solving can be simplified as:

[0022] V i = argmax(∑ x D1(x)·D2(x-v)T i (x)) (5)

[0023] The product operation in the spatial domain can be converted to the frequency domain, and the fast Fourier transform and inverse transform are used to accelerate the calculation, and the final solving formula is:

[0024]

[0025] wherein 3DF, 3DF -1 represent the fast Fourier transform and inverse transform respectively.

[0026] Further, the specific implementation manner of step 2 is as follows:

[0027] Step 2.1, calculate the feature difference value of the matched image, the features include average brightness, entropy, variance, LBP and Gabor feature, and normalize, calculate the feature difference of the same point of the original old image and the reference image after matching:

[0028]

[0029] Wherein, Δf is the feature distance, ω k is the weight of the kth feature, D k (x i ,y j ) represents the similarity of the feature k of the matching point pair (x i ,y j ), indicates the normalized Euclidean distance, N represents the number of selected feature categories;

[0030] Step 2.2, set the confidence threshold, the area higher than the threshold is the changed area, and the area lower than the threshold is the unchanged area;

[0031]

[0032] Wherein, S change is the change detection result image, δ is the change detection threshold constant, and the unchanged area is the high confidence area.

[0033] Further, in step 3, the matching image is changed in CIELab color space, and the (a, b) value of the unchanged area color image is migrated to the gray image, and the brightness of the gray image is kept unchanged, and the initial color migration is completed.

[0034] Further, in step 4, the colored point is taken as a marked coloring sample, and the Levin optimization method is used to adaptively diffuse the color to the whole image.

[0035] Further, the specific implementation method of the Levin optimization method is as follows:

[0036] A minimum objective function is set as:

[0037] J(x)=∑ r (x(r)-∑ s∈N(r) W rs x(s)) 2 (9)

[0038] In the formula, s is the adjacent pixel of r, N(r) is the set of adjacent pixels, X(r) is the chroma value of pixel r, W rs is a weighting function, which satisfies:

[0039]

[0040] Where Y(r) and Y(s) are the pixel brightness values. X is the variance of the brightness values ​​of the neighborhood of pixel r, where ∝ represents the proportional sign; since formula (9) is difficult to minimize, the image is represented in vector form, X T =[x1,x2,...,x n ], where n represents the total number of pixels, and equation (9) can be expressed as:

[0041] J(x)=X T AX (11)

[0042] In the formula, A is the weight matrix, consisting of weight coefficients W at different positions. rs The resulting symmetric matrix has a coloring point b. T =[b1,b2,...,b n ], and the element at the color marker position in b is set to a non-zero value, represented as:

[0043] b T X = b T b (12)

[0044] Minimize using the Lagrange multiplier method:

[0045] P(x) = X T AX + λ(b T bb T X) (13)

[0046] Differentiating the above equation with respect to x and setting the derivative to 0, we get:

[0047] AX = λb / 2 (14)

[0048] According to the principle of grayscale image colorization, the color values ​​of high-confidence regions should not change. Therefore, A needs to be corrected. Let i be the pixel in the high-confidence region, and the result needs to satisfy x. i =b i Therefore, let w in matrix A ij The value must be 0, 1≤j≤n, i≠j, and λ=2 to obtain x. i =b i Let the corrected A be denoted as A. r ,Right now:

[0049] A r X = b (15)

[0050] Thus, the problem of colorizing a grayscale image becomes a simple matrix equation problem; by solving X, the restored image can be obtained.

[0051] Compared with the prior art, the old image restoration technology fusing change detection and color migration provided by the embodiment of the present application has the beneficial effect that a digital mural and the like are constructed to be suitable for an image restoration framework, a change detection and color migration method are fused, a geometric registration image pair of a gray-scale and color image is established, change information of the registration image at different time phases is realized by a change detection model, color information of the color image is migrated to an unchanged area, and overall color coloring of the gray-scale image is realized by a color diffusion model. BRIEF DESCRIPTION OF DRAWINGS

[0052] Figure 1 The image correction method in the embodiment of the present application is provided.

[0053] Figure 2 The CFOG descriptor calculation method in the embodiment of the present application is provided.

[0054] Figure 3 The corrected image restoration method in the embodiment of the present application is provided. DETAILED DESCRIPTION

[0055] It should be noted that the embodiments and the features in the embodiments in the present application can be combined with each other without conflict. The present application will be described in detail below with reference to the drawings and in combination with the embodiments.

[0056] In order to enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor should belong to the protection scope of the present application.

[0057] It should be noted that the terms "first", "second" and the like in the specification and claims of the present application and the above-described drawings are used to distinguish similar objects, and do not necessarily indicate a specific order or a chronological sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device including a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to the process, method, product or device.

[0058] The old image restoration method fusing change detection and color migration provided by the present application includes the following steps:

[0059] Step 1, obtain the original old image and the reference color image, and match and align the two images by using the CFOG algorithm;

[0060] Step 2, change detection is performed by using average brightness, gray co-occurrence matrix entropy, variance, LBP and Gabor features to determine the high-confidence unchanged area;

[0061] Step 3, the original old image is color transferred by using the color of the high-confidence corresponding point to obtain a preliminary colored image;

[0062] Step 4, the color of the low-confidence corresponding point is obtained by diffusion according to the preliminary colored image to obtain a repaired image;

[0063] Step 1 includes:

[0064] Step 1.1: In order to realize the uniform distribution of feature points, the two images (the original old image and the reference color image) are respectively divided into local small windows, and a certain number of Harris corner points are selected in each window. The basic idea of Harris algorithm is to use a fixed window to slide in any direction on the image, compare the two cases before and after sliding, and if there is a large gray scale change in any direction, it can be considered that there is a corner point in the window, which is used as a pixel-level feature for subsequent image matching.

[0065] Step 1.2: Obtain the x-direction gradient size and y-direction gradient size of each corner point pixel in Cartesian coordinates, and obtain the gradient size in 9 bins according to the x, y direction gradient size. The calculation formula is:

[0066] g0=|cosθ×g x +sinθ×g y | (1)

[0067] where g x ,g y are the x, y direction gradients, and θ is the central angle of the bin (for example, when 180° is divided into 9 bins, the central angle of each bin is 0, 20, 40,..., 180). Thus, 9 feature maps with size (H, W) are obtained, where H is the image height and W is the width.

[0068] Step 1.3: Perform 3D Gaussian convolution on the 9 feature maps with size (H, W) to generate CFOG descriptors. The three-dimensional Gaussian convolution kernel is used to replace the three linear interpolation to generate the direction histogram in the two-dimensional XY direction, and the convolution in the Z direction is operated in the gradient direction, which can reduce the influence of local geometric distortion on the feature vector.

[0069] Step 1.4: The matching is done using the sum of squared differences (SSD) of CFOG. To speed up the computation, the SSD operation is accelerated using the fast Fourier transform (FFT) in practice, where the pixel-level features of the two images are denoted as D1and D2, respectively, where x is the position of a pixel in the three-dimensional feature representation, T i (x) is a mask on D1(x) where T i (x) = 1, and T i (x) = 0 otherwise. In addition, S i (v) is the SSD similarity function between D1and D2at the template window with a vector v, which is the allowed difference between D1and D2for matching. By minimizing S i (v), we can achieve the matching between D1and D2.

[0070] S i (v) =∑ x [D1(x)-D2(x-v)] 2 T i (x) (2)

[0071] Therefore, the matching function can be further defined as:

[0072] V i = argmin v (∑ x [D1(x)-D2(x-v)] 2 T i (x)) (3)

[0073] The above equation (2) is to find the position of the minimum SSD of two matching windows, which can be expanded as:

[0074]

[0075] Since the first term in equation (4) is a constant, the minimization can be simplified as:

[0076] V i = argmax(∑ x D1(x)·D2(x-v)T i (x)) (5)

[0077] The product operation in the spatial domain can be converted to the frequency domain using the Fourier transform and inverse transform to speed up the computation, and the final solution is:

[0078]

[0079] where 3DF,3DF -1 represent the Fourier transform and inverse transform, respectively.

[0080] Step 2 includes:

[0081] Step 2.1: Calculate the feature difference value of the matched image respectively, the features include average brightness, entropy, variance, LBP and Gabor feature, and normalize, calculate the feature difference of the same point of the matched original old image and the reference color image:

[0082]

[0083] Wherein, Δf is the feature distance, ω k is the weight of the kth feature (generally, the weight is set according to the importance of different features), D k (x i ,x j ) represents the similarity of the feature k of the matching point pair (x i ,y j ), which refers to the normalized Euclidean distance, and N represents the number of selected feature categories.

[0084] Step 2.2: Based on the previous result, set the confidence threshold through experiment, the area higher than the threshold is the changed area, and the area lower than the threshold is the unchanged area.

[0085]

[0086] Wherein, S change is the change detection result image, δ is the change detection threshold constant, and the unchanged area is the high confidence area.

[0087] Step 3 includes: performing CIELab color space change on the matched image, migrating the (a, b) value in the Lab color space of the unchanged area color image to the gray image, and keeping the brightness of the gray image unchanged, to complete the initial color migration.

[0088] Step 4 includes: taking the colored point as a marked coloring sample, and using Levin optimization method to adaptively diffuse the color to the whole image.

[0089] Levin et al. set a minimum objective function as:

[0090] J(x) = ∑ r (x(r)-∑ s∈N(r) W rs x(s)) 2 (9)

[0091] In the formula, s is the adjacent pixel of r, N(r) is the set of adjacent pixels, X(r) is the chroma value of pixel r, and W rs is a weighting function, which satisfies:

[0092]

[0093] Where Y(r) and Y(s) are the pixel brightness values. X is the variance of the brightness values ​​of the neighborhood of pixel r, where ∝ indicates a proportional sign. Since formula (9) is difficult to minimize, the image is represented as a vector, X T =[x1,x2,...,x n ], where n represents the total number of pixels. Equation (9) can be expressed as:

[0094] J(x)=X T AX (11)

[0095] In the formula, A is the weight matrix, consisting of weight coefficients W at different positions. rs The resulting symmetric matrix. The colored point is b. T =[b1,b2,...,b n ], and the element at the color marker position in b is set to a non-zero value, represented as:

[0096] b T X = b T b (12)

[0097] Minimize using the Lagrange multiplier method:

[0098] P(x) = X T AX+λ(b T bb T X) (13)

[0099] Differentiating the above equation with respect to X and setting the derivative to 0, we get:

[0100] AX = λb / 2 (14)

[0101] According to the principle of grayscale image colorization, the color values ​​of high-confidence regions should not change, therefore, A needs to be corrected. The pixels in the high-confidence regions are labeled i, and the result needs to satisfy X. i =b i Therefore, let w in matrix A ij The value of (1≤j≤n, i≠j) must be 0, and λ=2 must be set to obtain X. i =b i Let the corrected A be denoted as A. r ,Right now:

[0102] A r X = b (15)

[0103] Thus, the problem of colorizing a grayscale image becomes a simple matrix equation problem. Solving for X yields the restored image, which is then output as a color image.

[0104] Please refer to Figure 1 , Figure 1 is an image geometric correction method in the method, comprising the following steps:

[0105] First, extract the feature points of the gray image, and calculate the CFOG descriptor of the gray and color image;

[0106] Second, obtain the matching window of the gray and RGB image;

[0107] Third, calculate the SSD similarity to obtain the corrected image.

[0108] Please refer to Figure 2 , Figure 2 is Figure 1 the specific steps of the CFOG method, comprising the following steps:

[0109] First, calculate the gradient features of the image in multiple directions, arrange them in the Z direction to form a three-dimensional multi-directional gradient feature map.

[0110] Second, construct an approximate three-dimensional Gaussian convolution kernel to convolve the feature map in the first step to generate the CFOG descriptor.

[0111] Please refer to Figure 3 , Figure 3 is a corrected image repair method in the method, comprising the following steps:

[0112] First, calculate the feature map of the corrected color image and the original old image, normalize the feature values, determine the feature weights, and then calculate the feature difference;

[0113] Second, set a confidence threshold, and the area above the threshold is the changed area, and the area below the threshold is the unchanged area.

[0114] Third, realize color migration in the unchanged area according to the confidence distribution map to obtain an initial colored image;

[0115] Fourth, through the Levin color diffusion method, diffuse the color in the initial colored image to the global to obtain the repaired image.

Claims

1. An aged image restoration method fusing change detection and color transfer methods, characterized in that, The method comprises the following steps: Step 1, obtaining an original old image and a reference color image, and matching and aligning the two images by using a CFOG algorithm; Step 2, calculating feature difference of same-named points of the original old image and the reference color image after matching based on average brightness, gray level co-occurrence matrix entropy, variance, LBP and Gabor features to realize change detection; The specific implementation mode of step 2 is as follows: Step 2.1, calculating feature difference values of the matched images respectively, the features including average brightness, entropy, variance, LBP and Gabor features, and performing normalization, and calculating feature difference of same-named points of the original old image and the reference color image after matching: （3） in, For feature distance, For the first The weights of each feature Indicates matching point pairs Features The similarity refers to the normalized Euclidean distance. Indicates the number of feature categories selected; Step 2.2, setting a confidence threshold, regions higher than the threshold are regarded as changed regions, and regions lower than the threshold are regarded as unchanged regions; （4） wherein, is a change detection result map, is a change detection threshold constant, and unchanged regions are high confidence regions; Step 3, performing color migration on the original gray image by using colors of high-confidence corresponding points to obtain a preliminary colored image; In step 3, the matched images are transformed using the CIELab color space, and the unchanged areas are converted to color images. The color values ​​are transferred to the grayscale image while the brightness of the grayscale image remains unchanged, thus completing the initial color transfer. Step 4, colors of low-confidence corresponding points are diffused according to the preliminary colored image to obtain a repaired image; In step 4, the colored points are regarded as marked coloring samples, and a Levin optimization method is used to adaptively diffuse colors to the whole image.

2. A method of restoring old images by fusing change detection and color transfer method as claimed in claim 1, wherein: The specific implementation mode of step 1 is as follows: Step 1.1, in order to realize uniform distribution of feature points, the original old image and the reference color image are respectively divided into local small windows, and a certain number of Harris corner points are selected in each window, and the corner points are regarded as pixel-level features for subsequent image matching; Step 1.2, obtaining x-direction gradient size and y-direction gradient size of each corner point pixel under Cartesian coordinates, and obtaining gradient sizes in 9 bins according to the x-direction gradient size and the y-direction gradient size, and the calculation formula is as follows: （1） Where g x ,g y θ is the gradient in the x and y directions, and θ is the center angle of the bin. When 180° is divided into 9 bins, the center angle of each bin is 0, 20, 40...180, thus obtaining 9 feature maps of size (H, W), where H is the image height and W is the width. Step 1.3: performing 3-dimensional Gaussian convolution on 9 feature maps with a size of (H, W) to generate CFOG descriptors, wherein the three-dimensional Gaussian convolution kernel is used to replace the three-linear interpolation to generate a direction histogram in the two-dimensional XY direction, and is used to operate in the gradient direction in the Z direction, so that the influence of local geometric distortion on the feature vector can be reduced; Step 1.4: SSD matching using CFOG, in practice, the SSD operation is accelerated using FFT, CFOG's SSD is defined as SSD, where D1 and D2 are two image pixel-level features, x is the position of a pixel in the three-dimensional feature representation, T i (x) is a mask on D1(x), where T i (x) = 1, otherwise T i (x) = 0; in addition, S i (v) is the SSD similarity function between D1 and D2 with a vector v on the template window, v is the allowed difference between D1 and D2 for matching, the matching between D1 and D2 is achieved by minimizing S i (v). （2） The matching function is further defined as: （3） The formula (2) is expressed as a position where the SSD of two matching windows is minimum, and after expansion, the following formula (4) can be obtained: ) （4） Since the first term in formula (4) is a constant, the minimum value solving can be simplified as: （5） The product operation in the spatial domain can be converted to the frequency domain, and Fourier forward transformation and inverse transformation are used to accelerate the calculation, and finally the formula for solving is as follows: （6） where 3DF, respectively denote the forward and inverse Fourier transform.

3. The aged image restoration method that fuses change detection and color transfer methods as claimed in claim 1, characterized in that: The specific implementation mode of the Levin optimization method is as follows: A minimum objective function is set as: （9） where s is a neighboring pixel of r, N(r) is a set of neighboring pixels, X(r) is the chrominance value of pixel r, W rs is a weighting function satisfying: （10） where Y(r), Y(s) are the luminance values of the pixels, is the variance of the neighborhood luminance values of pixel r, denotes the proportional symbol; since formula (9) is difficult to minimize, the image is represented in the form of a vector, X T = [x1, x2,..., xn], where n represents the total number of pixels, formula (9) can be expressed as: n Y(r) = Y(s) - k * (X - Xs) （11） In the formula, A is a weight matrix composed of weight coefficients W of different positions rs The coloring point is b T =[b1,b2,...,b n ], and the element of the color mark position in b is set to a non-zero value, represented as: X = 0 b (12) The Lagrange multiplier method is used to minimize: （13） Derivation is performed on x in the above formula, and the derivative is 0, and the following formula (5) is obtained: （14） According to the principle of grayscale image colorization, the color value of the high confidence area does not change, so A needs to be corrected, the pixel of the high confidence area is i, and the result needs to satisfy x i = b i , w ij = 0, 1≤j≤n, i≠j in the matrix A, and set = 2, so that x i = b i , and the corrected A is denoted as A r , that is: （15） In this way, the colorization problem of the gray image is changed into a simple matrix equation solving problem, and after solving X, the repaired image can be obtained.

Images