A method for recovering color and texture details of scenes in shadow areas of unmanned aerial vehicle remote sensing images

Abstract

The application discloses a method for recovering color and texture details of ground objects under shadow in unmanned aerial vehicle (UAV) remote sensing images. The method first divides the UAV image into blocks as the minimum processing unit; secondly, a new shadow detection index (SDI) is constructed to enhance the shadow, and a threshold segmentation is used to obtain a shadow mask; then, the homogenous regions are extracted using laser radar intensity and elevation information. Finally, the homogenous regions of shadow / non-shadow areas are matched, and the missing information in the shadow area is recovered by using the method combining illumination compensation and gray difference. The application overcomes the problem that the existing method causes significant color heterogeneity between the shadow area scene and the real ground object due to the inaccuracy of matching between homogenous shadow / non-shadow areas. Under the premise of not interfering with the feature information of the non-shadow area of the image, the color and texture information of the shadow area scene is recovered, and the recovered image has no obvious shadow boundary. The method is mainly used for (but not limited to) recovering the color and texture information of the shadow area scene in the UAV remote sensing image.

Description

Technical Field

[0001] This invention relates to shadow area detection and restoration technology for UAV remote sensing images, specifically a method for restoring the color and texture details of objects in shadow areas of UAV remote sensing images. Background Technology

[0002] Shadows in remote sensing images are a degradation phenomenon caused by the complete or partial obstruction of light by objects, resulting in the loss of image features or low clarity. UAV remote sensing images, with their advantages of high flexibility, low acquisition cost, and high resolution, have become an important data source for ground monitoring and mapping. However, due to the combined effects of sunlight, ground reflection, and atmospheric disturbance, UAV remote sensing images widely suffer from low color and texture feature recognition in shadow areas, severely reducing image quality and significantly interfering with subsequent image processing such as image interpretation, image matching, and pattern recognition. Although current UAV cameras have automatic balancing mechanisms to avoid darkness or overexposure, these methods can only optimize areas with uneven illumination and cannot completely eliminate shadows. Accurately detecting shadows and effectively restoring color and texture details in shadow areas is therefore crucial. Currently, shadow detection and restoration methods remain challenging in handling shadows in UAV images with complex terrain features and diverse object types, especially as existing methods struggle to address the significant heterogeneity between shadowed and unshadowed areas after shadow restoration.

[0003] To effectively reduce the impact of shadows on remote sensing images, many researchers have dedicated themselves to developing shadow detection and restoration techniques. Accurate shadow detection is a crucial prerequisite for shadow restoration, and existing shadow detection methods based on image color features are relatively effective. However, due to spectral similarity, some dark objects remain indistinguishable from shadows, necessitating further research into accurate shadow detection in UAV RGB remote sensing images. Shadow compensation is a restoration process that improves image quality and enhances visual appeal. Although information in shadow areas is blurred, shadowed images still contain some valuable information. This provides the possibility of restoring ground feature information in shadowed areas. Therefore, many shadow restoration methods restore ground feature information in shadowed areas to a certain extent, rather than completely. The key to shadow restoration is to maximize the recovery of color and texture information in shadowed areas without interfering with the feature information of non-shadowed areas of the image.

[0004] Existing shadow restoration methods can be broadly categorized into linear correlation correction methods, histogram matching methods, illumination intensity-based methods, and machine learning-based methods. Linear correlation correction methods utilize the relationship between shadows and non-shadows to construct a linear correlation function for shadow compensation, restoring image information by correcting shadow pixels. However, this method is highly sensitive to the size of the histogram matching window. Illumination intensity-based methods utilize the ratio of direct light to ambient light to restore information in shadow areas, effectively improving the clarity of shadow areas in the original image, but resulting in significant color information loss and boundary effects. Machine learning-based restoration methods require manual sample selection to train model parameters in practical applications. Based on analyzing the relationship between shadow and non-shadow samples, a matching relationship between shadow and non-shadow intervals can be constructed using Markov random field theory. In recent years, deep learning has shown significant advantages in image shadow processing; the SynShadow model is an effective data augmentation technique, but currently it is only applicable to close-up images. In summary, existing shadow restoration methods can locally compensate for color information in shadow areas, but the inconsistencies in color and texture between the restored shadow boundary area and the non-shadow area remain significant. Summary of the Invention

[0005] The core idea of ​​this invention is to define a shadow detection index for extracting shadow masks, and then combine the shadow masks with information from non-shadowed features in homogeneous regions to match and restore the missing information of the shadowed features. By analyzing the spectral and detail characteristics of UAV remote sensing images, a method for restoring the color and texture of objects in shadowed areas of UAV remote sensing images is proposed, ensuring consistency between the true colors of objects in shadowed areas and the actual colors of ground features.

[0006] The method of the present invention includes the following specific steps:

[0007] S1: UAV remote sensing image segmentation; The original remote sensing image is segmented using a sliding window of size S×S. S should be greater than the length (L) or width (W) of the largest target on the research site, i.e., S=max(L,W); During the segmentation process, an overlapping region N=k×S is set between adjacent images, where k is the overlap rate of adjacent images, k∈[0,1], and it is recommended that the value of k be 0.2.

[0008] S2: Shadow detection is performed on the segmented image of the UAV remote sensing image from S1. First, a Shadow Detection Index (SDI) is constructed based on the R, G, and B band information of the UAV image: SDI = ω × |2G - BR| + ε × G, where G represents the green band of the original remote sensing image; B represents the blue band of the original remote sensing image; R represents the red band of the original remote sensing image; ω and ε are parameters, and ω + ε = 1. It is recommended that ω range from 0.1 to 0.3. Second, the Otsu method (maximum inter-class variance method) is used to threshold the SDI-enhanced image. The optimal threshold is used to segment the SDI-enhanced image to obtain a binary mask image. Pixels smaller than the threshold are black, representing shadow areas, while those larger than the threshold are white, representing non-shadow areas. Finally, mathematical morphology opening operations are used to fill holes, and closing operations are used to remove small burrs and fragments. After morphological optimization, the final shadow detection result is obtained.

[0009] S3: Perform homogeneous region segmentation processing on the block image from step S1; First, remove various types of noise from the LiDAR intensity and use histogram equalization to enhance the contrast of the intensity data; Second, statistically analyze the LiDAR intensity (IN) and elevation features (EL) of each image region, and use the LiDAR intensity and elevation features to define the intensity range and elevation range representing typical land features; Finally, based on the intensity range and elevation range, the region composed of pixels that satisfy the condition IN∩EL is divided into homogeneous regions.

[0010]

[0011] S4: The results obtained from S1, S2, and S3 are used for color and texture restoration of objects in the shadow area. This is achieved by matching homogeneous regions between the shadow and non-shadow areas and using a combination of illumination compensation and grayscale difference methods to restore missing information in the shadow area. Specific steps include:

[0012] S41: Input the UAV remote sensing image divided into m blocks in S1;

[0013] S42: Take a block image I containing n homogeneous regions. j (j∈{1,2,…,m}), all shaded regions are represented as S, and all unshaded regions are represented as U;

[0014] S43: Find homogeneous shaded regions S i The corresponding homogeneous non-shaded region U i For the i-th homogeneous shaded region S, i∈{1,2,…,n}, i Its mean value in the q-band is For q∈{R,G,B}, the mean value of its corresponding homogeneous unshaded region in the q band is

[0015] S44: Statistically analyze the texture feature entropy of the homogeneous non-shadow area. If the entropy value ent ≥ t, proceed to step S45; if the entropy value ent < t, go to step S46. It is recommended that the value of t be taken as 5.3 - 5.8. The entropy value is defined as follows:

[0016]

[0017] In the formula, l represents all possible values of the gray level difference values in the homogeneous non-shadow area, and p(i) represents the probability of each gray level difference value.

[0018] S45: and [[ID=It is used to calculate the ratio of the direct light component to the ambient light component, and the obtained ratio is expressed as follows:

[0019] [[ID=Thendo Perform shadow compensation in the q band to obtain

[0021]

[0022] Then the homogeneous shadow area S i After compensation, it is expressed as

[0023] <00遂S46: and It is used to calculate the difference between the non-shadow area and the shadow area, and the obtained difference is expressed as follows:

[0025] [[ID=Ithen Perform shadow compensation in the q band to obtain

[0027]

[0028] Then the homogeneous shadow area S i After compensation, it is expressed as

[0029]

[0030] S47: Loop from step S43 to step S46 until i = n, and the shadow compensation of one image patch ends;

[0031] S48: Loop from step S42 to step S47 until j = m, and the shadow compensation of all image patches ends;​​​​​​

[0032] S49: Stitch together the segmented images and output the shadow compensation result.

[0033] Compared with the prior art, the present invention has the following advantages:

[0034] 1. This invention defines a new shadow detection index based on the R, G, and B band information of UAV images, which can effectively extract shadows and is beneficial for the recovery of shadow area information in UAV remote sensing images.

[0035] 2. This invention performs homogeneous region segmentation on the segmented remote sensing image and uses homogeneous region matching between shadow and non-shadow areas to recover the color and texture information of objects in the shadow area without interfering with the feature information of the non-shadow area of ​​the image.

[0036] 3. This invention employs a shadow restoration method that combines illumination compensation and grayscale difference. After restoration, the shadow boundary transition is natural, and the color and texture information of ground features are consistent with the true colors of the ground features. Attached Figure Description

[0037] Figure 1 This is a flowchart of the present invention;

[0038] Figure 2 RGB images of the drone: (a) region 1, (b) region 2, (c) region 3, (d) region 4;

[0039] Figure 3 The following are the shadow restoration results obtained by the method of this invention and three other methods in regions 1, 2, 3, and 4: (a) Illumination correction method, (b) Color transfer method, (c) SynShadow, (d) Method of this invention.

[0040] Figure 4 for Figure 5 Enlarged view of the restored result;

[0041] Figure 5 Here are the graphs showing the Kappa coefficient as a function of parameters: (a) Kappa coefficient as a function of parameter k, (b) Kappa coefficient as a function of parameter ω. Detailed Implementation

[0042] A method for restoring the color and texture of objects in the shadow area of ​​a UAV remote sensing image includes the following steps:

[0043] S1: UAV image segmentation; The original remote sensing image is segmented using a sliding window of size S×S. S should be greater than the length (L) or width (W) of the largest target on the research site, i.e., S = max(L,W); During the segmentation process, an overlapping region N = k×S is set between adjacent images, where k is the overlap rate of adjacent images, k∈[0,1]. In this invention, k = 0.2 is selected.

[0044] S2: Shadow Detection in Blocked Images; First, a new Shadow Detection Index (SDI) is constructed based on the R, G, and B band information of the UAV image to enhance the shadows of the blocked image in S1. SDI = ω × |2G - BR| + ε × G, where: G represents the green band of the original remote sensing image; B represents the blue band of the original remote sensing image; R represents the red band of the original remote sensing image; ω and ε are parameters and ω + ε = 1. In this invention, ω = 0.2 and ε = 0.8 are selected. Second, the Otsu method, i.e., the maximum inter-class variance method, is used to threshold and segment the SDI-enhanced image to obtain a binary mask image. Pixels smaller than the threshold are black, representing shadow areas, and pixels larger than the threshold are white, representing non-shadow areas. Finally, mathematical morphology opening operations are used to fill holes, and closing operations are used to remove small burrs and fragments. After morphological optimization, the final shadow detection result can be obtained.

[0045] S3: Homogeneous Region Segmentation Processing; First, various types of noise are removed from the LiDAR intensity, and histogram equalization is used to enhance the contrast of the intensity data; Second, the LiDAR intensity (IN) and elevation (EL) features of each image region are statistically analyzed, and the intensity range and elevation range representing the homogeneous region are defined using the LiDAR intensity and elevation features:

[0046]

[0047] Finally, based on the above intensity and elevation ranges, the region composed of pixels that satisfy the condition IN∩EL is classified as a homogeneous region.

[0048] S4: Color and texture restoration of objects in shadow areas; This involves restoring missing information in shadow areas by matching homogeneous regions between shadow and non-shadow areas and combining illumination compensation with grayscale difference. First, input the UAV remote sensing image divided into m blocks from S1, and then extract the block image I containing n homogeneous regions. j (j∈{1,2,…,m}), denot all shaded regions as S, and all non-shaded regions as U. Next, find homogeneous shaded regions S. i The corresponding homogeneous non-shaded region U i , i∈{1,2,…,n}. For the i-th homogeneous shaded region S i Its mean value in the q-band is q ∈ {R, G, B}, and the mean value of the corresponding homogeneous non-shadow region in the q band is Statistically calculate the texture feature entropy of the homogeneous non-shadow region, where l is all possible values of the gray-level difference value, and p(i) is the probability of each gray-level difference value. If the entropy value ent ≥ t, calculate the ratio of the direct light component to the ambient light component Use r q for S i Perform shadow compensation in the q band to obtain If the entropy value ent < t, calculate the difference between the non-shadow region and the shadow region, and the obtained difference is: Use d q for S i Perform shadow compensation in the q band to obtain In this invention, t = 5.5 is selected. Secondly, use the shadow compensation results of the R, G, and B bands to synthesize the RGB image to obtain S i Shadow compensation result The shadow compensation of one homogeneous region is completed. Repeat the above steps until i = n, that is, the shadow compensation of one image patch ends; until j = m, that is, the shadow compensation of all image patches ends. Finally, splice the block images and output the shadow compensation result.

[0049] Specifically in implementation, taking 4 UAV remote sensing images containing different ground objects as examples, the longitude and latitude of the aerial photography location are 112°43′ east longitude and 38°27′ north latitude, and the aerial photography height is 85m. The selected device is the DJI "Phantom 4 RTK" UAV equipped with a camera with a 1-inch CMOS and 20 million effective pixels, and the maximum resolution of the camera photos is 5472 pixel × 3648 pixel (3:2).

[0050] This invention uses color difference, shadow standard deviation index (SSDI), and gradient similarity as evaluation indicators.

[0051] 1) Color difference

[0052] △E lab = [(△L) 2 + (△a) 2 + (△b) 2 1 / 2

[0053] The relationship between the color difference value and the visual perception is: when ΔE < 1, the color difference is hardly noticeable; when 1 < ΔE ≤ 2, the color difference is hardly felt; when 2 < ΔE ≤ 3.5, the color difference is felt moderately; when 3.5 < ΔE ≤ 6, the color difference is felt明显; when ΔE > 6, the color difference is felt strongly.

[0054] ​2) Shading Standard Deviation Index

[0055]

[0056] Where b is the current band of the image, B is the total number of bands in the image, i is the current sample pixel in the shadow region, and N is the total number of samples in the shadow region. Fs is the corrected shadow sample set. This represents the mean of the corresponding non-shaded sample set. SSDI reflects the change in shaded areas relative to non-shaded areas after shadow compensation. A low SSDI value indicates that the texture details of the shaded and non-shaded areas are consistent after shadow compensation; a high SSDI value indicates that there is a significant difference in texture details between the shaded and non-shaded areas after shadow compensation.

[0057] 3) Gradient similarity

[0058]

[0059] In the formula g x and g y These represent the center gradient values ​​of image patches x and y, respectively. C is a small positive constant to prevent the algorithm from becoming unstable due to an excessively small denominator. g(x, y) represents the gradient similarity values ​​of x and y, ranging from [0, 1].

[0060] The method of this invention, along with illumination correction, color transfer, and SynShadow methods, was used to simultaneously restore shadows in four test areas. The restoration results are shown in the attached figure. Figure 3 The images show the results of shadow restoration in regions 1, 2, 3, and 4 obtained by the shadow restoration method of the present invention and the illumination correction, color transfer, and SynShadow methods. Figure 4 for Figure 3 A magnified view of the restored result.

[0061] The table below compares the experimental results of this invention with those of the other three methods:

[0062]

[0063]

[0064]

[0065] The present invention achieved an average color difference of 1.891 across four experimental regions, which is 3.125, 2.415, and 1.916 lower than the average color difference values ​​of illumination correction, color conversion, and SynShadow methods, respectively. The average SSDI value was 15.419, which is 4.477 and 4.826 lower than the average SSDI values ​​of illumination correction and color conversion methods, respectively. The average shadow boundary gradient similarity value was 0.726, which is 0.234, 0.202, and 0.101 higher than the average shadow boundary gradient similarity values ​​of illumination correction, color conversion, and SynShadow methods, respectively. This invention restores the color and texture information of objects in shadow areas without interfering with the feature information of non-shadow areas of the image, resulting in a more visually natural shadowless image.

[0066] Due to the sensitivity settings of the parameters, this invention analyzes the influence of parameters k and ω on the performance of the method. The Kappa coefficient is selected as the consistency test and detection accuracy index. The relationship between the Kappa coefficient and parameters k and ω is as follows: Figure 5 As shown, the range of k is 0–0.7, the range of ω is 0.1–0.9, and the step size is 0.1. It is evident that the magnitudes of k and ω have a significant impact on the Kappa coefficient. Analysis of four test images from the experiment reveals that the trends in the kappa coefficient are basically consistent across the four regions. The optimal value is k = 0.2, and the optimal range for ω is 0.1–0.3.

[0067] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any modifications or substitutions that can be conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for recovering color and texture details of a scene in a shadow region of a remote sensing image of a UAV, characterized in that: The method comprises the following steps: S1: UAV remote sensing image blocking; a sliding window with a size of MxM is used to segment the original remote sensing image, M is equal to the maximum value of the length (L) and width (W) of the target on the research site, i.e. M = max(L, W); during the segmentation, an overlapping area N = kxM is set between adjacent images, k is the overlap rate of adjacent images, k∈[0, 1]; S2: shadow detection on the image after S1 unmanned aerial vehicle remote sensing image blocking; first, based on the R, G, B band information of unmanned aerial vehicle image, the shadow detection index SDI is constructed to carry out shadow enhancement on the blocked image of S1, Wherein G represents the green band of the original remote sensing image; B represents the blue band of the original remote sensing image; R represents the red band of the original remote sensing image; , epsilon is a parameter and + epsilon = 1, second, threshold value is obtained on the SDI enhanced image, the binary mask image is obtained by using the best threshold value to segment the SDI enhanced image, the pixel point less than the threshold value is black, indicating the shadow area, the part greater than the threshold value is white, indicating the non-shadow area, finally, the mathematical morphological opening operation is adopted to fill the holes, the close operation is used to remove the small burrs and fragments, after the morphological optimization processing, the final result of shadow detection can be obtained; S3: homogeneous region segmentation processing is performed on the blocked image of step S1; first, various types of noise are removed from the laser radar intensity, and a histogram equalization method is used to enhance the contrast of the intensity data; second, the laser radar intensity IN and the elevation feature EL of each image region are counted, and the intensity range and the elevation range representing the typical ground objects are defined by using the laser radar intensity and the elevation feature; finally, based on the intensity range and the elevation range, the region composed of the pixels satisfying the condition IN∩EL is divided into a homogeneous region; S4: the results obtained in S1, S2 and S3 are used for shadow area scene color and texture recovery, the missing information in the shadow area is recovered by matching the homogeneous regions in the shadow and non-shadow areas and using the method combining the light compensation and the gray difference value. 2.The method of claim 1, wherein: Step S4 specifically comprises the following steps: S41: input the UAV remote sensing image segmented into m blocks in S1; S42: taking the segmented image I comprising n homogeneous regions j , all shadow regions are denoted S and all non-shadow regions are denoted U; S43: find homogeneous shadow region S i corresponding homogeneous non-shadow region U i , , for the ith homogeneous shadow region S i , whose mean value in the q band is , , whose corresponding homogeneous non-shadow region has a mean value in the q band of ; S44: count the texture feature entropy of the homogeneous non-shadow region, if the entropy value ent≥ t, calculate step S45; if the entropy value ent< t, go to step S46, S45: and The ratio of the direct light component and the ambient light component is calculated, and the obtained ratio is expressed as follows: ， Then, for Shadow compensation is performed in the q-band , ， S is a homogeneous shadow region i is represented as : ， S46: and to calculate the difference between the non-shadowed area and the shadowed area, the difference obtained is represented as follows: ， Then, for Shadow compensation is performed in the q-band , ， Then the homogeneous shadow region S i After compensation, it is expressed as : ， S47: repeat steps S43 to S46 until i = n, and the shadow compensation of one image block is completed; S48: repeat steps S42 to S47 until j = m, and the shadow compensation of all image blocks is completed; S49: block image splicing, and output the shadow compensation result.

3. The method of claim 1 or 2, wherein the method further comprises: In step S1, the value of k is 0.2, and in step S2, the value of k is in the range of 0.1 to 0.

3.

4. The method of claim 2, wherein the method further comprises: In step S44, t is 5.3~5.8.

Images