A color harmonization method, apparatus, and related components for image fusion.

Abstract

This invention discloses a color harmonization method, apparatus, and related components for image fusion. The method includes: processing the input original image to obtain a mask image; obtaining the sample mean values ​​of the target sky background image and the original image, and calculating the sample difference between them; assigning corresponding weight coefficients to the sample difference between the target sky background image and the original image based on preset weighting rules; and performing color harmonization processing on the original image according to the sample difference and weight coefficients to obtain a harmonized image; and superimposing the harmonized image with the mask image to obtain the target image. This method first obtains the color difference between the original image and the target sky background image, and then increases the weight coefficients, allowing users to choose the tendency of the generated result according to the specific application scenario, so that the colors of the generated target image are more closely matched to the colors of the original image or the target sky background image, thus making the combination of the harmonized image and the mask image more realistic and natural, enhancing the user experience.

Description

Technical Field

[0001] This invention relates to the field of image processing, and more particularly to a color harmonization method, apparatus, and related components for image fusion. Background Technology

[0002] With the rapid development of artificial intelligence algorithms, image matting and segmentation techniques have become increasingly mature. However, how to harmoniously and naturally integrate the segmented target with the new background has become a pressing issue. For example, in a scene where the sky changes between two images, if one image has a sunny background... Figure 1 As shown, the background of this image needs to be replaced with another image background featuring lightning and thunder. The current practice is usually to simply segment out the sky part of the image and then replace it with another image background.

[0003] However, images obtained through the above methods will have abrupt and extremely unnatural visual effects, greatly reducing the user experience. Summary of the Invention

[0004] The purpose of this invention is to provide a color harmonization method, apparatus and related components for image fusion, which aims to solve the problem that images obtained by directly replacing images through image matting have poor visual effects.

[0005] To solve the above-mentioned technical problems, the objective of this invention is achieved through the following technical solution: providing a color harmonization method for image fusion, comprising:

[0006] The original input image is processed to obtain the corresponding mask image;

[0007] Obtain the sample mean values ​​of the target sky background image and the original image respectively, and calculate the sample difference between the two;

[0008] Based on preset weighting rules, the sampling difference between the target sky background image and the original image is assigned a corresponding weighting coefficient, and the original image is color harmonized according to the sampling difference and the weighting coefficient to obtain the corresponding harmonized image.

[0009] The target image is obtained by overlaying the harmony image and the mask image.

[0010] Furthermore, the technical problem to be solved by the present invention is to provide a color harmonization device for image fusion, comprising:

[0011] The original image processing unit is used to process the input original image to obtain the corresponding mask image;

[0012] The difference acquisition unit is used to acquire the sample mean of the target sky background image and the original image respectively, and calculate the sample difference between the two;

[0013] The color harmonization processing unit is used to assign corresponding weight coefficients to the sampling difference between the target sky background image and the original image based on a preset weight assignment rule, and to perform color harmonization processing on the original image according to the sampling difference and the weight coefficients to obtain the corresponding harmonious image.

[0014] The overlay unit is used to overlay the harmonic image and the mask image to obtain the target image.

[0015] In addition, this embodiment of the invention provides a computer device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the color harmonization method for image fusion described in the first aspect above.

[0016] In addition, embodiments of the present invention also provide a computer-readable storage medium storing a computer program that, when executed by a processor, causes the processor to perform the color harmonization method for image fusion described in the first aspect.

[0017] This invention discloses a color harmonization method, apparatus, and related components for image fusion. The method includes: processing an input original image to obtain a corresponding mask image; obtaining the sampling mean of a target sky background image and the original image, and calculating the sampling difference between them; assigning corresponding weight coefficients to the sampling difference between the target sky background image and the original image based on a preset weighting rule, and performing color harmonization processing on the original image according to the sampling difference and the weight coefficients to obtain a corresponding harmonized image; and superimposing the harmonized image with the mask image to obtain a target image. This method first obtains the color difference between the original image and the target sky background image, and then increases the weight coefficients, allowing users to choose the tendency of the generated result according to the specific application scenario. That is, the color of the generated target image can be more closely matched to the color difference of the original image, or more closely matched to the color of the target sky background image, so that the combination of the harmonized image and the mask image is more realistic and natural, enhancing the user experience. Attached Figure Description

[0018] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a schematic diagram of the effect generated using existing technology;

[0020] Figure 2 This is a schematic flowchart of a color harmonization method for image fusion provided in an embodiment of the present invention;

[0021] Figure 3 A schematic diagram illustrating the effect of the color harmonization method for image fusion provided in this embodiment of the invention;

[0022] Figure 4 A schematic block diagram of a color harmonization device for image fusion provided in an embodiment of the present invention;

[0023] Figure 5 A schematic block diagram of a computer device provided for an embodiment of the present invention. Detailed Implementation

[0024] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0025] It should be understood that, when used in this specification and the appended claims, the terms "comprising" and "including" indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0026] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0027] It should also be further understood that the term "and / or" as used in this specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0028] In existing technologies, image fusion schemes are mainly divided into traditional image processing techniques and deep learning image processing techniques. Traditional image processing techniques include cut-and-paste, alpha fusion, multi-band fusion, and Poisson fusion. Among them, cut-and-paste is the simplest cut-and-paste technique, which segments the target from an image and then overlays it onto the target background as needed. This approach is simple and convenient; however, when the foreground and background colors and brightness differ significantly, the resulting image may appear unnatural. Alpha fusion, multi-band fusion, and Poisson fusion all suffer from undesirable effects such as halo artifacts or have high computational complexity. In other words, existing image fusion methods are not suitable for widespread application.

[0029] Please see Figure 2 , Figure 2 This is a schematic flowchart of a color harmonization method for image fusion provided in an embodiment of the present invention;

[0030] like Figure 2 As shown, the method includes steps S101 to S104.

[0031] S101. Perform image processing on the input original image to obtain the corresponding mask image;

[0032] S102. Obtain the sample mean values ​​of the target sky background image and the original image respectively, and calculate the sample difference between the two;

[0033] S103. Based on the preset weighting rules, assign corresponding weight coefficients to the sampling difference between the target sky background image and the original image, and perform color harmonization processing on the original image according to the sampling difference and the weight coefficients to obtain the corresponding harmonized image.

[0034] S104. Overlay the harmonic image with the mask image to obtain the target image.

[0035] In this embodiment, the input original image is first processed to obtain a corresponding mask image. It should be noted that the size of the mask image is the same as the size of the original image, and the old sky background area and the image area to be extracted are segmented in the mask image. Then, the sampling mean of the target sky background image and the original image are obtained respectively, and the sampling difference between the two is calculated. Through simple calculation, the color difference (color difference, brightness difference) between the target sky background image and the original image is obtained. Then, the original image is harmonized. Specifically, to make the obtained harmonized image closer to the color of the target sky background image, a larger weight coefficient is assigned, and to make the obtained harmonized image less affected by the color of the target sky background image, a smaller weight coefficient is assigned. Finally, the harmonized image and the mask image are superimposed to form the target image, that is, the new sky background area and the image area to be extracted are combined together.

[0036] Compared to existing technologies that directly use image cutout and replacement to obtain the target image (such as... Figure 1 As shown), the method of this application generates a more realistic and natural target image. Users can choose the tendency of the generated result according to the specific application scenario, so that the colors are more in line with the colors of the original image, or more in line with the colors of the new background (target sky background image) (e.g., Figure 3 As shown in the figure, this application can achieve the harmonization result during image fusion through simple calculation, which is convenient, fast and has strong practicality.

[0037] In one specific embodiment, step S101 includes the following steps:

[0038] S10. Divide the original image into a sky region and a non-sky region, set the pixel value of all pixels in the sky region to 0, and set the pixel value of all pixels in the non-sky region to 1, to obtain the corresponding mask image.

[0039] In this embodiment, by setting the pixel values ​​of all pixels in the sky area to 0 and the pixel values ​​of all pixels in the non-sky area to 1, the sky area and non-sky area of ​​the original image are obtained. It should be noted that the non-sky area of ​​the original image is the image area to be extracted, and the sky area of ​​the original image is the image area to be replaced.

[0040] In one specific embodiment, step S10 includes the following steps:

[0041] S11. Use the UET segmentation algorithm to divide the original image into sky regions and non-sky regions.

[0042] In this embodiment, the UET segmentation algorithm is used to divide the original image into sky and non-sky regions. However, in actual operation, other image segmentation algorithms can also be used to segment the original image into regions, so this embodiment will not elaborate further.

[0043] In one specific embodiment, step S102 includes the following steps:

[0044] S20. According to the preset step size, the color space of the original image src, the target sky background image skybg, and the mask image mask are sampled at pixel points respectively, and the corresponding sampling results are as follows: src_samp, skybg_samp, and mask_samp;

[0045] S21. Obtain the sampled mean value of the target sky background image, skybg_mean, using the following formula:

[0046] skybg_mean=mean(skybg_samp);

[0047] Where mean() represents the average value;

[0048] S22. Obtain the sample mean src_mean of the original image using the following formula:

[0049] src_mean=sum(src_samp*(1-mask_samp)) / (sum(1-mask_samp)+1e -10 );

[0050] Where sum() represents the summation operation;

[0051] S23. Calculate the sampling difference diff between the target sky background image and the original image using the following formula:

[0052] diff = skybg_mean - src_mean.

[0053] In this embodiment, mean(skybg_samp) represents the average value of the sampling results of the target sky background image in the color space; in practical applications, the step size can be obtained according to the following formula:

[0054] step = src.h / threshold, where src.h represents the height of the original image and threshold represents the preset sampling interval threshold.

[0055] Specifically, threshold can be set to 20, but it should be understood that threshold can also be set to other values. Different thresholds will result in different step sizes.

[0056] After obtaining the step size, pixel sampling is performed on the color spaces of the original image (src), the target sky background image (skybg), and the mask image (mask) according to the step size, obtaining the corresponding sampling results. Then, the corresponding sampling mean is obtained. It should be noted that in the formula "src_mean=sum(src_samp*(1-mask_samp)) / (sum(1-mask_samp)+1e-10)", 1e... -10 This is to avoid calculation errors when the denominator is 0.

[0057] In one specific embodiment, the color space includes RGB three channels or LAB three channels.

[0058] In this embodiment, the color space is either RGB three-channel or LAB three-channel.

[0059] In one specific embodiment, step S103 includes the following steps:

[0060] S30. Perform color harmonization processing on the original image using the following formula to obtain the corresponding harmonized image img_recolor:

[0061] img_recolor=src+w*diff

[0062] Where w represents the weighting coefficient of the influence of the color in the target sky background image on the harmony image.

[0063] In this embodiment, w represents the weighting coefficient of the original image being affected by the color in the target sky background image. The larger the weighting coefficient w, the closer the obtained harmonious image is to the color in the target sky background image; the smaller the weighting coefficient w, the less the obtained harmonious image is to the color in the target sky background image, that is, the less the harmonious image is affected by the color in the target sky background image. It should be noted that the "*" appearing in all the formulas of this application means multiplication.

[0064] In one specific embodiment, step S104 includes the following steps:

[0065] S40. Superimpose the harmonic image and the mask image using the following formula to obtain the target image img_dst:

[0066] img_dst=img_recolor*(1-mask)+skybg*mask.

[0067] In this embodiment, based on the mask image, the harmonized image obtained after color harmonization is superimposed on the mask image to achieve a natural effect after sky replacement, as shown in the image. Figure 3 As shown.

[0068] This invention also provides a color harmonization apparatus for image fusion, which is used to perform any of the aforementioned color harmonization methods for image fusion. Specifically, please refer to... Figure 4 , Figure 4 This is a schematic block diagram of a color harmonization device for image fusion provided in an embodiment of the present invention.

[0069] like Figure 4 As shown, the color harmonization device 500 for image fusion includes:

[0070] The original image processing unit 501 is used to perform image processing on the input original image to obtain the corresponding mask image;

[0071] The difference acquisition unit 502 is used to acquire the sample mean values ​​of the target sky background image and the original image respectively, and calculate the sample difference between the two.

[0072] The color harmonization processing unit 503 is used to assign corresponding weight coefficients to the sampling difference between the target sky background image and the original image based on a preset weight assignment rule, and to perform color harmonization processing on the original image according to the sampling difference and the weight coefficients to obtain the corresponding harmonious image.

[0073] The overlay unit 504 is used to overlay the harmony image and the mask image to obtain the target image.

[0074] The target image generated by this device is more realistic and natural. Users can choose the tendency of the generated result according to the specific application scenario, so that the colors are more in line with the colors of the original image or the colors of the new background (target sky background image). This application can achieve the harmonious result of image fusion through simple calculation, which is convenient, fast and has strong practicality.

[0075] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the above-described apparatus and unit can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0076] The aforementioned color harmonization device for image fusion can be implemented as a computer program, which can be used in, for example... Figure 5 It runs on the computer device shown.

[0077] Please see Figure 5 , Figure 5 This is a schematic block diagram of a computer device provided in an embodiment of the present invention. The computer device 1100 is a server, which can be a standalone server or a server cluster composed of multiple servers.

[0078] See Figure 5 The computer device 1100 includes a processor 1102, a memory, and a network interface 1105 connected via a system bus 1101. The memory may include a non-volatile storage medium 1103 and internal memory 1104.

[0079] The non-volatile storage medium 1103 may store an operating system 11031 and a computer program 11032. When the computer program 11032 is executed, it causes the processor 1102 to perform a color harmonization method for image fusion.

[0080] The processor 1102 provides computing and control capabilities to support the operation of the entire computer device 1100.

[0081] The internal memory 1104 provides an environment for the operation of the computer program 11032 in the non-volatile storage medium 1103. When the computer program 11032 is executed by the processor 1102, the processor 1102 can perform a color harmonization method for image fusion.

[0082] The network interface 1105 is used for network communication, such as providing data transmission. Those skilled in the art will understand that... Figure 5 The structure shown is merely a block diagram of a portion of the structure related to the present invention and does not constitute a limitation on the computer device 1100 to which the present invention is applied. The specific computer device 1100 may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements.

[0083] Those skilled in the art will understand that Figure 5 The embodiments of the computer device shown do not constitute a limitation on the specific configuration of the computer device. In other embodiments, the computer device may include more or fewer components than illustrated, or combine certain components, or have different component arrangements. For example, in some embodiments, the computer device may include only memory and a processor. In such embodiments, the structure and function of the memory and processor are different from those shown. Figure 5 The embodiments shown are consistent and will not be repeated here.

[0084] It should be understood that, in this embodiment of the invention, the processor 1102 may be a Central Processing Unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0085] In another embodiment of the invention, a computer-readable storage medium is provided. This computer-readable storage medium may be a non-volatile computer-readable storage medium. The computer-readable storage medium stores a computer program, wherein the computer program, when executed by a processor, implements the color harmonization method for image fusion according to embodiments of the invention.

[0086] The storage medium is a physical, non-transient storage medium, such as a USB flash drive, portable hard drive, read-only memory (ROM), magnetic disk, or optical disk, or any other physical storage medium capable of storing program code.

[0087] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the devices, apparatuses, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0088] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A color harmonization method for image fusion, characterized in that, include: The original input image is processed to obtain the corresponding mask image; Obtain the sample mean values ​​of the target sky background image and the original image respectively, and calculate the sample difference between the two; Based on preset weighting rules, the sampling difference between the target sky background image and the original image is assigned a corresponding weighting coefficient, and the original image is color harmonized according to the sampling difference and the weighting coefficient to obtain the corresponding harmonized image. The target image is obtained by overlaying the harmony image and the mask image. The step of obtaining the sampled mean values ​​between the target sky background image and the original image, and calculating the sampled difference between the two, includes: According to a preset step size, pixel sampling is performed on the color space of the original image src, the target sky background image skybg, and the mask image mask, respectively, and the corresponding sampling results are as follows: src_samp, skybg_samp, and mask_samp; The sample mean of the target sky background image, skybg_mean, is obtained using the following formula: skybg _mean = mean(skybg _ samp); Where mean() represents the average value; The sample mean src_mean of the original image is obtained by the following formula: src _mean = sum(src _ samp * (1- mask _ samp)) / (sum (1- mask _ samp) +1e -10 ); Where sum() represents the summation operation; The sampling difference, diff, between the target sky background image and the original image is calculated using the following formula: diff = skybg_mean - src_mean; The method, based on a preset weighting rule, assigns corresponding weighting coefficients to the sampling difference between the target sky background image and the original image, and performs color harmonization processing on the original image to obtain a corresponding harmonized image, including: The original image is color-harmonized using the following formula to obtain the corresponding harmonized image img_recolor: img_recolor = src + w*diff Where w represents the weighting coefficient of the harmonious image being affected by the color in the target sky background image. The larger the weighting coefficient w, the closer the obtained harmonious image is to the color in the target sky background image; the smaller the weighting coefficient w, the less the obtained harmonious image is to the color in the target sky background image, that is, the less the harmonious image is affected by the color in the target sky background image.

2. The color harmonization method for image fusion according to claim 1, characterized in that, The step of processing the input original image to obtain the corresponding mask image includes: The original image is divided into a sky region and a non-sky region. The pixel values ​​of all pixels in the sky region are set to 0, and the pixel values ​​of all pixels in the non-sky region are set to 1, thus obtaining the corresponding mask image.

3. The color harmonization method for image fusion according to claim 1, characterized in that, The step of overlaying the harmonic image and the mask image to obtain the target image includes: The target image img_dst is obtained by superimposing the harmonic image and the mask image using the following formula: img_dst = img_recolor * (1 - mask) + skybg * mask.

4. The color harmonization method for image fusion according to claim 2, characterized in that, The process of dividing the original image into sky regions and non-sky regions includes: The original image is divided into sky and non-sky regions using the UET segmentation algorithm.

5. The color harmonization method for image fusion according to claim 3, characterized in that, The color space includes either RGB three-channel or LAB three-channel.

6. A color harmonization device for image fusion, characterized in that, include: The original image processing unit is used to process the input original image to obtain the corresponding mask image; The difference acquisition unit is used to acquire the sample mean of the target sky background image and the original image respectively, and calculate the sample difference between the two; The color harmonization processing unit is used to assign corresponding weight coefficients to the sampling difference between the target sky background image and the original image based on a preset weight assignment rule, and to perform color harmonization processing on the original image according to the sampling difference and the weight coefficients to obtain the corresponding harmonious image. The overlay unit is used to overlay the harmonic image and the mask image to obtain the target image; The step of obtaining the sampled mean values ​​between the target sky background image and the original image, and calculating the sampled difference between the two, includes: According to a preset step size, pixel sampling is performed on the color space of the original image src, the target sky background image skybg, and the mask image mask, respectively, and the corresponding sampling results are as follows: src_samp, skybg_samp, and mask_samp; The sample mean of the target sky background image, skybg_mean, is obtained using the following formula: skybg _mean = mean(skybg _ samp); Where mean() represents the average value; The sample mean src_mean of the original image is obtained by the following formula: src _mean = sum(src _ samp * (1- mask _ samp)) / (sum (1- mask _ samp) +1e -10 ); Where sum() represents the summation operation; The sampling difference, diff, between the target sky background image and the original image is calculated using the following formula: diff = skybg_mean - src_mean; The method, based on a preset weighting rule, assigns corresponding weighting coefficients to the sampling difference between the target sky background image and the original image, and performs color harmonization processing on the original image to obtain a corresponding harmonized image, including: The original image is color-harmonized using the following formula to obtain the corresponding harmonized image img_recolor: img_recolor = src + w*diff Where w represents the weighting coefficient of the harmonious image being affected by the color in the target sky background image. The larger the weighting coefficient w, the closer the obtained harmonious image is to the color in the target sky background image; the smaller the weighting coefficient w, the less the obtained harmonious image is to the color in the target sky background image, that is, the less the harmonious image is affected by the color in the target sky background image.

7. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the color harmonization method for image fusion as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, causes the processor to perform the color harmonization method for image fusion as described in any one of claims 1 to 5.

Images