Camera color calibration method and device based on remote sensing radiation reference and storage medium

Abstract

The invention provides a camera color calibration method and device based on a remote sensing radiation benchmark and a storage medium. The method comprises the steps of obtaining the remote sensing radiation benchmark as a reference image; extracting homonymy points between the to-be-corrected images and the reference image, and homonymy points of overlapping areas between the to-be-corrected images; establishing an error equation containing an imaging nucleus under the condition that the color values of the homonymy points on different images are consistent; wherein the imaging core is constructed based on a sun angle and an observation angle and is used for representing non-uniform illumination distribution in a view field of the camera; based on the error equation containing the imaging kernel, solving a color correction coefficient of each image through a least square principle; and performing pixel-by-pixel correction on each image according to the color correction coefficient and outputting a result image. According to the method, an absolute color calibration and relative color calibration model of a camera with limited calibration capability is established, an imaging kernel related to a solar angle and an observation angle is constructed, and the problem of global or local non-uniformity correction can be solved, so that a color calibration result is effectively improved.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of photogrammetry and remote sensing, and relates to a camera color calibration method based on remote sensing radiation reference, wherein the construction of an imaging kernel related to the solar angle and the observation angle is the key technology of the method. BACKGROUND

[0002] In photogrammetry, the essential features of ground objects need to be reflected by various colors of images. However, when the light changes due to changes in environmental factors (such as season, sunlight, atmospheric conditions, etc.) and imaging conditions, the camera color distortion will occur due to the inability of digital cameras to automatically correct, which will affect the visual effect of the image. In addition, according to the illumination theory, the illumination distribution in the camera field of view is not uniform, and the illumination in the edge field of view decreases sharply, thereby causing color difference within the same image and affecting the accuracy of measurement. Therefore, the camera color needs to be calibrated. The traditional correction method usually relies on white balance and integrating sphere. Among them, the white balance adjusts the color of other colors by distinguishing the white color through the camera, thereby correcting the color deviation and making the picture color closer to the true effect seen by the human eye. The white balance mainly has automatic white balance, preset white balance (such as sunlight, overcast, tungsten lamp, fluorescent lamp, etc.), and manual white balance. However, due to the great differences in actual light sources, the white balance is easy to misjudge and cause color deviation. The integrating sphere adjusts the red, green, and blue channel gains of the camera by providing a standard light source with known color temperature, thereby accurately restoring the color, especially suitable for absolute color calibration of scientific and industrial cameras. At present, the commonly used multi-spectral cameras (such as OLI carried by Landsat and MSI carried by Sentinel) in the international market have high spectral response consistency, strong on-orbit calibration ability, and strict color calibration, and are easy to realize camera color restoration. However, there are still a large number of low-calibration-ability remote sensing cameras or medium-low-consistency multi-spectral cameras (such as the cameras of the Gao Fen series) on the market due to process limitations, which have problems such as spectral channel crosstalk and non-linear detector response, resulting in camera color distortion. In addition, such cameras rely on ground calibration fields and have insufficient on-orbit calibration ability, making it difficult to realize camera color restoration. Therefore, people usually use remote sensing radiation references to calibrate the color of cameras with limited calibration ability (some domestic cameras) all the time, and the commonly used methods include linear regression, histogram matching, Wallis transformation, and block adjustment. Among them, linear regression and histogram matching rely on global statistical relationships and cannot solve the color distortion in non-uniform illumination areas; Wallis transformation improves the visual effect through local contrast normalization, but may destroy the radiation consistency and has poor effect on non-linear illumination decay; block adjustment is suitable for global consistency adjustment when multiple images are spliced, but has limited effect on color difference within a single image. In summary, the existing color calibration methods for such limited-calibration-ability cameras cannot completely solve the global or local non-uniformity correction problem, resulting in color difference between multiple images or within a single image. SUMMARY

[0003] The present application aims at the color difference problem of the corresponding color calibration method of the existing limited calibration capability camera in global and local non-uniformity color correction, and proposes a camera color calibration scheme based on remote sensing radiation reference, fully considers the global consistency and local non-uniformity problem, and can effectively improve the color calibration result of the limited calibration capability camera.

[0004] The above technical problems of the present application are mainly solved by the following technical scheme: A camera color calibration method based on remote sensing radiation reference, comprising the following processes, Acquiring a remote sensing radiation reference as a reference image; Extracting homonymic points between the to-be-corrected images and the reference image, and homonymic points in the overlapping areas between the to-be-corrected images; Establishing an error equation containing an imaging kernel based on the color value consistency of the homonymic points on different images; wherein the imaging kernel is constructed based on the solar angle and the observation angle, and is used to represent the non-uniform illumination distribution in the camera field of view; Solving the color correction coefficient of each image based on the error equation containing the imaging kernel through the least square principle; Correcting each image pixel by pixel according to the color correction coefficient and outputting a result image.

[0005] Moreover, the imaging kernel is constructed through the trigonometric function relationship of the solar zenith angle, the solar azimuth angle, the observation zenith angle and the observation azimuth angle.

[0006] Moreover, the imaging kernel includes a kernel function component composed of the cosine of the difference between the observation zenith angle and the observation azimuth angle, and a kernel function component composed of the solar zenith angle.

[0007] Moreover, the error equation includes an absolute color calibration error equation based on the homonymic points of the to-be-corrected images and the reference image, and a relative color calibration error equation based on the homonymic points between the to-be-corrected images.

[0008] Moreover, when extracting the homonymic points in the overlapping areas between the to-be-corrected images, the to-be-corrected images are sorted and screened according to the degree of overlap, and the window color average value of the homonymic points is taken as the color value, and the homonymic points with color values less than or equal to zero are removed.

[0009] Moreover, when extracting the homonymic points between the to-be-corrected images and the reference image, if the resolutions of the to-be-corrected images and the reference image are inconsistent, the high-resolution image is down-sampled to the level of the low-resolution image.

[0010] Moreover, the absolute color calibration error equation established for each image to be calibrated and the relative color calibration error equation established for each pair of images to be calibrated are combined to form an overall adjustment system.

[0011] In another aspect, the present application provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the camera color calibration method based on remote sensing radiation reference as described above when executing the program.

[0012] In another aspect, the present application provides a non-transitory computer readable storage medium having a computer program stored thereon, wherein the computer program implements the camera color calibration method based on remote sensing radiation reference as described above when executed by a processor.

[0013] In another aspect, the present application provides a computer program product comprising a computer program, wherein the computer program implements the camera color calibration method based on remote sensing radiation reference as described above when executed by a processor.

[0014] Compared with the prior art, the present application has the following advantages: the present application establishes absolute color calibration and relative color calibration models for cameras with limited calibration capability, and further constructs an imaging kernel related to the sun angle and the observation angle, which not only ensures the global consistency of camera color, but also effectively solves the problem of local non-uniformity of illumination. Compared with the traditional method, the present application can solve the problem of global or local non-uniformity correction, thereby effectively improving the color calibration result of cameras with limited calibration capability. BRIEF DESCRIPTION OF DRAWINGS

[0015] Figure 1 The flowchart of the embodiment of the present application is shown.

[0016] Figure 2 The comparison chart of the color calibration result of the first group of images of the embodiment of the present application is shown, wherein (a) is before color calibration of the entire survey area, (b) is after color calibration of the entire survey area, (c) is before color calibration of the local stitching position, and (d) is after color calibration of the local stitching position.

[0017] Figure 3 The comparison chart of the color calibration result of the first group of images of the embodiment of the present application is shown, wherein (a) is before color calibration of the entire survey area, (b) is after color calibration of the entire survey area, (c) is before color calibration of a single image, and (d) is after color calibration of a single image. DETAILED DESCRIPTION

[0018] In order to make the objects, technical solutions and advantages of the present application clearer, the technical solutions in the present application will be described clearly and completely below in combination with the drawings in the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of protection of the present application.

[0019] The embodiment of the present application provides a camera color calibration method based on remote sensing radiation reference. The method mainly comprises the following processes: firstly, obtaining remote sensing radiation reference as reference image; secondly, obtaining homonymic points between the to-be-corrected images and the reference image and homonymic points of the overlapping areas between the to-be-corrected images; then, establishing error equation with the color values of the homonymic points on different images being consistent as the condition; next, normalizing the error equation according to the least square principle, and solving the color correction coefficient of each image; finally, correcting each image pixel by pixel according to the color correction coefficient, and outputting the result image.

[0020] As shown in Figure 1 The embodiment of the present application provides a camera color calibration method based on remote sensing radiation reference, and the implementation process comprises the following steps: Step 1, obtaining remote sensing radiation reference as reference image.

[0021] In specific implementation, the remote sensing radiation reference is preferably recommended to select satellite image products (such as Landsat L2 level products and Sentinel L2A level products) collected synchronously or in the same weather and time period within one week before and after, which can complete color calibration through the parameters provided in the official document and have good color consistency.

[0022] Step 2, obtaining homonymic points of the overlapping areas between the to-be-corrected images and homonymic points between the to-be-corrected images and the reference image, and obtaining color values of the homonymic points The specific implementation steps of step 2 recommended by the embodiment are as follows: Step 2.1, selecting image matching algorithm. The homonymic points can be extracted by image matching, and in specific implementation, existing matching algorithms such as traditional methods Harris, SIFT, SURF, ORB, and deep learning methods SuperPoint, D2-Net, SuperGlue and LoFTR can be used.

[0023] Step 2.2, obtaining homonymic points of the overlapping areas between the to-be-corrected images, which is to ensure the global consistency of color correction for subsequent relative correction between images, and the specific steps are as follows: Step 2.2.1, sorting and screening the overlap degree of the images to be corrected: for each image to be corrected, the overlap degree between it and the remaining images to be corrected is calculated, and the images are arranged in order from large to small according to the overlap degree, obtaining a preliminary image sequence, and removing the images with an overlap degree less than a preset threshold (preferably recommended to be set to 10%).

[0024] Step 2.2.2, matching the current image to be corrected with the remaining images in the sequence obtained in step 2.2.1, and recording the positions of the same named points.

[0025] Step 2.2.3, for each same named point, taking a window (recommended size is 3*3 pixels) centered on its pixel coordinates, and calculating the average value of the color in the window as its color value. This step can effectively avoid the interference of noise in the color calibration process. In actual implementation, the channel of the color value can be selected according to the actual situation, for example, RGB or RGBIR (with infrared waveband). Taking RGB as an example, the color average value is the value of three wavebands obtained by averaging in three channels respectively.

[0026] Step 2.2.4, removing the same named points with color values less than or equal to 0. This step is to remove dark pixels.

[0027] Step 2.3, obtaining the same named points between the images to be corrected and the reference image, which is for the absolute correction of the image color, and the specific implementation steps are as follows: Step 2.3.1, for each image to be corrected, if its spatial resolution is higher than that of the reference image, it is down-sampled to the resolution of the reference image; otherwise, the reference image is down-sampled to the resolution of the image to be corrected. This step is mainly to avoid the color correction deviation caused by the inconsistency of the resolution. In actual implementation, existing down-sampling methods can be used, for example, Gaussian blur down-sampling or bilinear interpolation down-sampling can be selected.

[0028] Step 2.3.2, matching the current image to be corrected with the reference image, and recording the positions of the same named points.

[0029] Step 2.3.3, for each same named point, taking a window (recommended size is 3*3 pixels) centered on its pixel coordinates, and calculating the average value of the color in the window as its color value.

[0030] Step 2.3.4, removing the same named points with color values less than or equal to 0.

[0031] Step 3, establishing an error equation with the color values of the same named points on different images being consistent. The present application proposes to establish an error equation containing an imaging kernel, wherein the imaging kernel is constructed based on the sun angle and the observation angle, and is used to represent the non-uniform illumination distribution in the camera field of view.

[0032] Further, the imaging kernel is constructed by trigonometric functions of the solar zenith angle, the solar azimuth angle, the observation zenith angle and the observation azimuth angle, and specifically includes a kernel function component composed of the cosine of the difference between the observation zenith angle and the observation azimuth angle, and a kernel function component composed of the solar zenith angle.

[0033] The error equation includes an absolute color calibration error equation based on the corresponding points of the image to be corrected and the reference image, and a relative color calibration error equation based on the corresponding points between the images to be corrected.

[0034] The embodiment preferably suggests that the step 3 is specifically implemented as follows: Step 3.1, establishing the error equation of absolute color calibration. For any point on the image to be corrected ( , the number of images to be corrected , there is a relationship between the gray value and the color value as follows: (1) In the formula, C represents the color value; G represents the gray value; K is a proportional coefficient, which can be given by the image header file or obtained by solving the error equation in specific implementation; is an imaging kernel related to the solar angle and the observation angle, which mainly solves the problem of uneven illumination distribution in the camera field of view; is the coefficient of the imaging kernel.

[0035] The calculation formula of the imaging kernel is as follows: (2) In the formula, is the solar zenith angle; is the observation zenith angle; is the solar azimuth angle; is the observation azimuth angle.

[0036] For each pair of corresponding points of the image to be corrected and the reference image, the imaging kernel is calculated respectively, and the color value is substituted into formula (1) to obtain the error equation of absolute color calibration of the image to be corrected : (3) In the formula, is the observation error, is the number of corresponding points between the image to be corrected and the reference image, represents the corresponding point label, and the corresponding imaging kernel is denoted as , and the corresponding color value and gray value are denoted as , .

[0037] Simplifying equation (3) gives the following equation: (4) where, is the observation error vector, ; is the coefficient matrix, ; is the unknown vector, ; is the observation vector, .

[0038] Step 3.2, establish the error equation of relative color calibration. For the to-be-corrected image and any co-named point , it is consistent in color value on the two images, and satisfies equation (1) on the two images, that is: (5) where, is the number of to-be-corrected images.

[0039] Calculate the imaging kernel of each pair of co-named points respectively, and substitute it into equation (4) to obtain the error equation of relative color calibration between the to-be-corrected image and the to-be-corrected image : (6) where, is the observation error, is the number of co-named points between the to-be-corrected image and the to-be-corrected image ; is the imaging kernel of the co-named point w in the to-be-corrected image ; is the imaging kernel of the co-named point w in the to-be-corrected image ; and are the gray values of the co-named point w in the to-be-corrected image and respectively; is the imaging kernel coefficient of the to-be-corrected image ; is the proportion coefficient of the to-be-corrected image .

[0040] Simplifying equation (6) gives the following equation: ​(7) wherein, is the observation error vector, ; and is the coefficient matrix, , ; is the unknown vector of the image to be corrected, .

[0041] Step 3.3, for each image to be corrected, an absolute color calibration error equation is established, and a relative color calibration error equation is established between each other, and a total error equation is obtained by combining all error equations: (8) wherein, is the observation error vector; is the coefficient matrix of all unknowns; is the unknown vector; is the observation value vector.

[0042] Step 4, according to the least square principle, the error equation is normalized, and the color correction coefficient of each image is solved.

[0043] Since the absolute color calibration error equation established for each image to be corrected and the relative color calibration error equation established for all pairs of images to be corrected with overlapping areas are combined to form an overall adjustment system in step 3, the normalization process in this step is performed using an equal weight matrix to reflect the equal accuracy characteristics of the color observation values of the same points.

[0044] The specific implementation steps of the preferred embodiment of step 4 are as follows: Step 4.1, construct the normal equation of the error equation. According to the least square indirect adjustment principle , the normal equation of the error equation can be listed as: (9) wherein, is the weight matrix of the observation value, which reflects the measurement accuracy of the observation value; is the transpose matrix of the matrix . For all point color observation values, it is generally considered that the measurement is equal in accuracy, so is the unit matrix.

[0045] Step 4.2, solve the color correction coefficient of each image. According to formula (9), the solution of the equation can be obtained as: (10) Step 5, according to the color correction coefficient, each image is corrected pixel by pixel, and the result image is output.​

[0046] To facilitate understanding of the technical effects of this invention, color calibration was performed on two sets of images using the method of this invention, and the results are as follows: Figure 2 and Figure 3 As shown. For the first set of images, the original image quality is relatively high, and the colors are already quite close to natural colors ( Figure 2 (a)). Therefore, the result obtained after processing by the method of the present invention is Figure 2 (b) The overall improvement is relatively small, mainly manifested in the suppression of slightly overexposed areas, resulting in a more natural overall color. Furthermore, from... Figure 2 (c) and Figure 2 As can be seen from the comparison in (d), this method can effectively eliminate color differences between images. For the second group of images, the original quality is poor, with obvious color cast and uneven color within the same image. Figure 3 (c)), and there is a significant color difference between different images ( Figure 3 (a)). After color calibration using the method of this invention, the image quality is significantly improved: the colors are closer to natural, and the color distribution within the image is more uniform. Figure 3 (d) The color difference between images is effectively eliminated. Figure 3 (b) achieves global color consistency.

[0047] This invention provides an electronic device that may include a processor, a communication interface, a memory, and a communication bus. The processor, communication interface, and memory communicate with each other via the communication bus. The processor can invoke logical instructions from the memory to execute a camera color calibration method based on a remote sensing radiometric reference.

[0048] Furthermore, the logical instructions in the aforementioned memory can be implemented as software functional units and sold or used as independent products, and can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0049] In another aspect, the embodiments of the present application also provide a computer program product, which comprises a computer program, the computer program can be stored on a non-transitory computer readable storage medium, and the computer program can be executed by a processor to enable a computer to perform the camera color calibration method based on remote sensing radiation reference provided by the above method.

[0050] In yet another aspect, the embodiments of the present application also provide a non-transitory computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the camera color calibration method based on remote sensing radiation reference provided by the above method.

[0051] The device embodiments described above are merely illustrative, wherein the units described as separate components can or can not be physically separate, and the components displayed as units can or can not be physical units, i.e., they can be located in one place, or distributed on multiple network units. Part or all of the modules can be selected to achieve the purpose of the present embodiment according to actual needs. Those skilled in the art can understand and implement it without creative labor.

[0052] From the above description of the embodiments, those skilled in the art can clearly understand that the embodiments can be realized by means of software plus necessary universal hardware platforms, and of course, can also be realized by hardware. Based on such understanding, the above technical solutions, essentially or in other words, the part that contributes to the prior art, can be embodied in the form of a software product, which can be stored in a computer readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes a number of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute the methods described in each embodiment or some parts of the embodiments.

[0053] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement to some technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A camera color calibration method based on remote sensing radiation reference, characterized in that, The method comprises the following steps: acquiring remote sensing radiation reference as a reference image; extracting homonymic points between the image to be corrected and the reference image, and homonymic points in the overlapping area between the images to be corrected; establishing an error equation containing an imaging kernel based on the color value consistency of the homonymic points on different images; wherein the imaging kernel is constructed based on the solar angle and the observation angle, and is used to represent the non-uniform illumination distribution in the camera field of view; solving the color correction coefficient of each image by the least square principle based on the error equation containing the imaging kernel; correcting each image pixel by pixel according to the color correction coefficient and outputting the result image.

2. The camera color calibration method based on remote sensing radiation reference according to claim 1, characterized in that: The imaging kernel is constructed by the trigonometric function relationship of the solar zenith angle, the solar azimuth angle, the observation zenith angle and the observation azimuth angle.

3. The camera color calibration method based on remote sensing radiation reference according to claim 2, characterized in that: The imaging kernel includes a kernel function component composed of the cosine of the difference between the observation zenith angle and the azimuth angle, and a kernel function component composed of the solar zenith angle.

4. The camera color calibration method based on remote sensing radiation reference according to claim 1, characterized in that: The error equation includes an absolute color calibration error equation based on the homonymic points between the image to be corrected and the reference image, and a relative color calibration error equation based on the homonymic points between the images to be corrected.

5. The camera color calibration method based on remote sensing radiation reference according to claim 1, characterized in that: When extracting the homonymic points in the overlapping area between the images to be corrected, the images to be corrected are sorted and screened according to the degree of overlap, and the window color average value of the homonymic points is taken as the color value, and the homonymic points with color value less than or equal to zero are removed.

6. The camera color calibration method based on remote sensing radiation reference according to claim 1, characterized in that: When extracting the homonymic points between the image to be corrected and the reference image, if the resolutions of the image to be corrected and the reference image are inconsistent, the high-resolution image is down-sampled to the level of the low-resolution image.

7. The camera color calibration method based on remote sensing radiation reference according to claim 1, characterized in that: When solving the color correction coefficient of each image by the least square principle, the absolute color calibration error equation established for each image to be corrected and the relative color calibration error equation established for all pairs of images to be corrected with overlapping areas are combined to form an overall adjustment system.

8. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that: The processor executes the program to realize the camera color calibration method based on remote sensing radiation reference according to any one of claims 1 to 7. 9.A non-transitory computer-readable storage medium having stored thereon a computer program. The computer program is executed by the processor to realize the camera color calibration method based on remote sensing radiation reference according to any one of claims 1 to 7.

10. A computer program product comprising a computer program, characterized in that: The computer program is executed by the processor to realize the camera color calibration method based on remote sensing radiation reference according to any one of claims 1 to 7.

Images