An adaptive shadow removal method suitable for remote tower panorama and its application

Abstract

The present invention relates to an adaptive shadow elimination method suitable for a panoramic view of a remote tower and its application, wherein the following formula is used to correct the shadow of a single image: wherein f(r) adopts an even-order polynomial with a constant term of 1, and the column gain function g(r) of the image is fitted by f(r), and the polynomial coefficient of f(r) is estimated. When the panoramic image is a spliced ​​image of several single images, the shadows of the single images are first corrected before splicing, and the panoramic image is spliced ​​with the corrected single images. The present invention uses image projection characteristics to fit the shadow correction function, and the calculation is simple and fast. It is suitable for shadow correction of a single-channel video and image preprocessing when splicing multiple channels of video, and can better eliminate the splicing stripes caused by image shadows while meeting the real-time requirements.

Description

Technical Field

[0001] The invention relates to an adaptive shadow elimination method suitable for a remote tower panorama and an application thereof in panoramic image stitching. Background Art

[0002] With the increasing number of small and medium-sized airports in my country, the application demand for remote tower technology is increasing. It uses technical means such as video splicing, network transmission, intelligent analysis, and visual presentation to achieve remote command in different places, provide a unified, centralized, and standardized air traffic control support model, and help solve the problem of insufficient air traffic control support capabilities at small and medium-sized airports. Among them, the video splicing and fusion unit is a key module of the remote tower optical system. It will splice multiple videos covering the airport area into a panoramic video in real time to intuitively present the full picture of the scene.

[0003] Image shadow refers to the phenomenon that the image brightness gradually darkens as the distance from the pixel position to the image center / symmetry axis increases, also known as halo. This phenomenon is common in photographic imaging systems and is related to the optical properties of the lens, sensor performance, and lighting conditions. Image shadows in single-point cameras will cause alternating light and dark stripes to appear in the stitching results, greatly affecting the visual effect. In addition, the image shadow characteristics will change dynamically with factors such as weather and lighting. For example, shadows on cloudy days are usually more obvious than on sunny days.

[0004] Existing image shadow correction methods mainly include image calibration methods and adaptive methods. The image calibration method is to let the camera shoot a white calibration plate, and then statistically calculate the pixel value distribution of the obtained image to obtain the correction parameters; the more mature adaptive method is the iterative method based on the image logarithmic entropy criterion. The basic technical concept of this type of method is to use a certain evaluation index to optimize the correction function parameters to obtain the optimal correction effect. These methods have their own characteristics and are suitable for different occasions, but they still have certain limitations. For example, the image calibration method is generally only suitable for laboratory environments, and has high requirements for lighting and calibration plates. For imaging systems that have been put into use, they usually do not have calibration conditions and are difficult to be practical; the iterative method based on image logarithmic entropy has a large amount of calculation and is difficult to complete in real time. In addition, the image logarithmic entropy may not be able to evaluate the shadow correction effect well, which may cause the optimization process to not converge and unpredictable results. Summary of the invention

[0005] The purpose of the present invention is to effectively reduce or eliminate shadows in images or image fusion while meeting real-time requirements.

[0006] The technical solution of the present invention is: an adaptive shadow elimination method suitable for the panoramic view of a remote tower, which uses the following method to correct the shadow of a single image:

[0007]

[0008] Use f(r) to fit the column gain function g(r) of the image and estimate the polynomial coefficients of f(r), where

[0009] and I(x,y) are the image after correction and the image before correction respectively;

[0010] f(r) is the correction function, which can be an even-order polynomial with a constant term of 1;

[0011] r is the normalized distance from the pixel (or pixel point) (x, y) to the central axis of the image (the central axis parallel to the y-axis).

[0012] The above-mentioned f(r) and the corresponding shadow correction method are applicable to the situation where the shadow characteristics are symmetric about (relative to) the central axis of the image (including the situation that can be regarded as symmetric about the central axis of the image, or various situations that can be processed according to the situation where the shadow characteristics are symmetric about the central axis of the image).

[0013] Furthermore, the column gain function of the image can be calculated in the following way:

[0014] g(r)=p s (x0) / p s (x),

[0015] in,

[0016] is the smoothed column mean function of the image, p s (x0) is p when x is x0 s (x);

[0017] x r is the preset smoothing window radius (or half of the width);

[0018] is the column mean function of the image, p(x′) is p(x) when x is x′;

[0019] h is the image height;

[0020] I(x,y) uses a grayscale image.

[0021] The normalized distance from a pixel (x,y) to the image center axis can be calculated as follows:

[0022] r=(x-x0) / x0,

[0023] in,

[0024] x0 is the horizontal coordinate (x coordinate) of the image center (x0, y0), and the origin of the coordinate system is in the lower left corner of the image.

[0025] As a preferred implementation, the polynomial coefficients (or parameters) of f(r) may be estimated in the following manner:

[0026] (a * ,b * ,c * ) = argmin a,b,c ∑ r (f(r)-g(r)) 2 ,

[0027] in,

[0028] f(r)=1+ar 2 +br 4 +cr 6 ,

[0029] (a * ,b * ,c * ) are the estimated polynomial coefficients (a, b, c).

[0030] Any one of the adaptive shadow removal methods suitable for remote tower panorama disclosed in the present invention is applied in panoramic image stitching, wherein the panoramic image is a stitched image of several single images (for example, a panoramic video stitched from multiple video channels), and shadow correction is performed on each single image before stitching, and the corrected single images are used to stitch the panoramic image.

[0031] For example, the panoramic image may be a video panoramic image (or panoramic video) spliced ​​from multiple video outputs, and correspondingly, the single images are video images of each channel of the multiple video outputs.

[0032] The beneficial effects of the present invention are as follows: the present invention utilizes the image projection characteristics to fit the shadow correction function, which is simple and fast to calculate, and is suitable for shadow correction of single-channel video and image preprocessing during multi-channel video splicing. It can better eliminate the splicing stripes caused by image shadows while meeting real-time requirements, thereby improving the quality of panoramic video. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 is a flow chart of the present invention;

[0034] Figure 2 2 is a comparative example of the application of the present invention in a single image, wherein the upper figure is the image before shadow correction, and the lower figure is the image after shadow correction using the method of the present invention;

[0035] Figure 3 : is the column gain function of the input image and the shadow correction function obtained by fitting according to the present invention, wherein the solid line is the column gain function (curve) of the input image, and the dotted line is the shadow correction function (curve) obtained by fitting;

[0036] Figure 4 The following is a comparative example of the application of the present invention in stitching multiple images, wherein the upper figure is a stitched image without shadow correction, and the lower figure is a stitched image after shadow correction is performed on each image using the method of the present invention before stitching. DETAILED DESCRIPTION

[0037] The processing flow of the present invention is shown in the attached Figure 1 As shown, the following operations are performed on the input image (single image):

[0038] 1) Parameter estimation: Estimate the parameters of the shadow correction function based on the grayscale characteristics of the input image;

[0039] 2) Shadow correction: Use the estimated shadow correction function to perform shadow correction on each channel of the input image to obtain the output image.

[0040] When applied to video splicing, for each channel of input images, the above shadow correction is used as an image preprocessing step, and the shadow correction of each channel of input images is first performed using the present invention, and then the splicing is performed.

[0041] The shadow correction function and parameter estimation are further described below.

[0042] 1) Shadow correction

[0043] Shadow correction can be described by the following formula:

[0044]

[0045] Among them, I(x,y), They represent the image before and after correction respectively, f(r) represents the correction function, which increases with the distance from the pixel position (x, y) to the image center (x0, y0) or the symmetry axis passing through (x0, y0).

[0046] Taking into account certain symmetry, f(r) can be defined as an even-order polynomial function of r, as follows:

[0047] f(r)=1+ar 2 +br 4 +cr 6 Formula 2

[0048] Among them, a, b, c are the correction parameters (polynomial coefficients) to be estimated, and the value range of r is in the interval [-1,1], which can be called the normalized distance. For example, the distance from the pixel position (x,y) to the image center (x0,y0) or the vertical axis passing through (x0,y0) can be linearly projected to the interval [-1,1] to form the normalized distance r.

[0049] Formula 2 can be derived from the Taylor expansion of f(r) at point 0:

[0050]

[0051] Since there is no shadow at the center of the image, f(0)=1. Due to the symmetry of the shadow, the odd-order terms are ignored, making f(r) an even function. The sixth-order terms are retained to obtain Formula 2. Other terms can also be retained according to actual needs.

[0052] Usually, the lower left corner of the image can be used as the coordinate origin to facilitate calculations.

[0053] Since the shadow characteristics are usually symmetrical about the image axis (the vertical axis through the center of the image), in this case, we can let

[0054] r=(x-x0) / x0 Formula 3

[0055] 2) Parameter Estimation

[0056] This step is to estimate the parameters a, b, c in formula 2, as follows:

[0057] (i) Convert the input image into a grayscale image and calculate the column mean function p(x) of the image as follows:

[0058]

[0059] Among them, w and h represent the image width and height respectively.

[0060] (ii) Smooth the column mean function of the image to form the smoothed column mean function p of the image s (x), as follows:

[0061]

[0062] Among them, x r is the preset smoothing window radius (such as x r =0.01w, 0.01 times the image width).

[0063] The smoothed column mean function / column mean function reflects the projection characteristics of the image value on the horizontal axis.

[0064] (iii) Calculate the column gain function g(r) of the image as follows:

[0065]

[0066] Among them, x0 is the horizontal coordinate of the image center (x0, y0), and this formula means that the gain coefficient of the column is equal to the ratio of the mean of the image center column to the mean of the column.

[0067] For shadow images, the column means on both sides of the image will be lower than the middle value, which can be compensated by the gain function to eliminate the image shadow.

[0068] (iv) Using the least squares method, use f(r) to fit the column gain function g(r) so that the mean square error between the two is minimized. In this case, the values ​​of the parameters (a, b, c) (a * ,b * ,c * ) is the estimated parameter of the shadow correction function, as shown below:

[0069] (a * ,b * ,c * ) = argmin a,b,c ∑ r (f(r)-g(r)) 2 Formula 7

[0070] Figure 3 Given Figure 2 The column gain function of the input image and the fitted shadow correction function in the example.

[0071] In order to further reduce the time cost during implementation, the input image can be downsampled before estimating the parameters. In addition, the above two steps (i) and (ii) can be quickly calculated using the integral image of the input image to obtain p s (x). In addition, since the image shadow characteristics will not change significantly in the short term, it is not necessary to estimate the parameters of the shadow correction function for each frame, and it is only necessary to update them regularly (eg, once a minute).

[0072] The present invention can be used for image stitching of multiple video channels. Image (including video image) stitching includes two basic steps: (1) image transformation, establishing and completing the geometric transformation of each single-point image to a panoramic image; (2) image fusion, eliminating the color difference between single-point images to achieve a seamless stitching effect.

[0073] Before image transformation, each image is subjected to the shadow correction of the present invention, and the modified image is used as a video image for splicing.

[0074] from Figure 2 and Figure 4 It can be seen that the present invention can be used for single image correction and modification before splicing multiple images to obtain a good shadow correction effect.

[0075] Unless otherwise specified or when one preferred or optional technical means is a further limitation of another technical means, the preferred and optional technical means disclosed in the present invention can be arbitrarily combined to form several different specific implementation methods.

Claims

1. An adaptive shadow removal method suitable for the panoramic view of a remote tower, using the following formula to correct the shadow of a single image: It is characterized in that Use f(r) to fit the column gain function g(r) of the image and estimate the polynomial coefficients of f(r), where and I(x,y) are the image after correction and the image before correction respectively; f(r) is the correction function, which uses an even-order polynomial with a constant term of 1; r is the normalized distance from the pixel (x, y) to the image center axis, The column gain function of the image is calculated in the following way: g(r)=p s (x0)*p s (x) in, is the smoothed column mean function of the image; x r is the preset smoothing window radius; is the column mean function of the image; h is the image height; I(x,y) uses a grayscale image. The polynomial coefficients of f(r) are estimated as follows: (a * ,b * ,c * )=argmin a,b,c ∑ r (f(r)-g(r)) 2 , in, f(r)=1+ar 2 +br 4 +cr 6 , (a * ,b * ,c * ) are the estimated polynomial coefficients (a, b, c).

2. The adaptive shadow removal method according to claim 1, wherein: The normalized distance from a pixel (x,y) to the image center axis is calculated as follows: r=(x-x0) / x0, in, x0 is the horizontal coordinate of the image center (x0, y0), and the origin of the coordinate system is in the lower left corner of the image.

3. Application of the adaptive shadow removal method for remote tower panorama according to any one of claims 1 to 2 in panoramic image stitching, wherein the panoramic image is a stitched image of several single images, characterized in that Before stitching, each single image is first subjected to shadow correction, and the corrected single images are used to stitch the panoramic image.

4. The use according to claim 3, characterized in that The panoramic image is a video panoramic image spliced ​​from multiple video outputs, and the single images are video images of each channel of the multiple video outputs.

Images