A method and system for monitoring drones under low-light haze conditions

Abstract

The application provides a low-illumination haze monitoring unmanned aerial vehicle method and system, and relates to the field of image processing; the method is used for pre-processing low-illumination haze images through a mixed method of edge window filtering and fast edge-preserving filtering, so as to obtain more accurate ambient light estimation values; then, combined with the images and the obtained ambient light estimation values, a light source threshold is set, the transmittance of the light source area and the non-light source area is separately solved through an adaptive light source matrix mechanism, the light source area is further refined and solved, and the brightness value of the light source area is better preserved; then, the transmittance is fused and solved, and the transmittance image is compensated through a gamma correction method; finally, the images under low-illumination haze are defogged combined with an atmospheric scattering model. The processed images are overall bright, the light source area has no halo effect, the color difference of the sky area is small, the image texture details are clear, and the colors are bright, so that the unmanned aerial vehicle can be efficiently monitored and the dangerous goods carried by the unmanned aerial vehicle can be observed.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, specifically to a method and system for monitoring drones under low-light haze conditions. Background Technology

[0002] With the development of technology, people have paid increasing attention to drones in recent years. Besides military applications, the application scenarios for drones in the civilian sector are constantly expanding. For example, drones have shown remarkable performance in areas such as recreational aerial photography, agricultural plant protection, police security, power line inspection, drone mapping, logistics transportation, and performance choreography. However, while drone technology brings benefits, it also has some drawbacks. For aviation departments, there are problems such as difficulty in detection, supervision, and enforcement. During the takeoff and landing of aircraft, if the actions of drones are not identified in time, and any potentially dangerous items carried by the drones are not observed, it can easily lead to aircraft crashes, threatening human lives and causing financial losses. Therefore, the prerequisite for controlling drones is the ability to effectively monitor them.

[0003] Regarding the operating environment of drones, in daytime scenarios, image acquisition devices can easily capture drones, monitor their movements, and identify any hazardous materials they carry. However, when drones are in low-light environments accompanied by fog or haze, the difficulty of monitoring increases significantly. Current low-light dehazing methods still have many shortcomings, including poor color saturation, blurred texture details, and high noise. Existing technologies address the problem of uneven nighttime illumination; some researchers have proposed dehazing algorithms that first compensate for illumination and then correct for color. While this may seem to improve color, inaccurate illumination estimation during compensation results in inadequate handling of shimmering areas, leading to noticeable halos and high noise in the restored image. Other researchers have proposed using illumination compensation for dehazing followed by color correction, but this fails to adequately estimate transmission, resulting in distorted colors and poor dehazing performance in the final restored image. In addition, some researchers believe that artificial light sources at night have phenomena such as glare and uneven illumination. Therefore, they added the glare layer to the standard daytime dehazing model, removed the glare layer to obtain the layer separation result, and then re-estimated the nighttime atmospheric light by dividing it into blocks. They estimated the transmittance through dark channel theory and then obtained the restored image. Although the above method has a good dehazing effect, the restored image is generally dark and the texture details are not clear because there is no light compensation or brightness enhancement. Furthermore, the image is too dark after dehazing, resulting in a loss of detail.

[0004] Some published patent applications also propose solutions for monitoring drones, such as patent applications CN108734670A and CN115170404A. While these disclosed solutions improve low-light defogging methods, their applications are too broad and cannot specifically address drone images in low-light, foggy conditions. For example, they do not consider the actual operating conditions of the drone; when drones operate in low light, their lights can affect the defogging image, easily causing a halo effect. Furthermore, when there are large areas of sky, after defogging, the sky area will show severe effects such as color cast and artifacts, resulting in poor image defogging and making it difficult to observe the drone effectively. Patent application CN115616479A discloses a special device and system for monitoring drones, but it cannot directly identify any dangerous items carried by the drone, which could potentially lead to serious accidents.

[0005] Therefore, given the current technical challenges in monitoring drones in special scenarios, there is a need for a low-light defogging method applicable to these scenarios, in order to achieve better monitoring of drones and identify the dangerous items they carry. Summary of the Invention

[0006] The purpose of this invention is to provide a method and system for monitoring drones under low-light fog and haze conditions. Based on the dark channel prior theory, the invention improves the defogging method, enhances the quality of the output image, solves the problem of monitoring drones under low-light fog and haze conditions, and achieves the goal of identifying dangerous items carried by drones under complex conditions.

[0007] To achieve the above objectives, the present invention proposes the following technical solution:

[0008] Firstly, a method for monitoring drones under low-light haze conditions is disclosed, including:

[0009] The pixels of the input image are obtained, and the input image is preprocessed using a hybrid method of side window filtering and fast edge-preserving filtering to obtain the ambient light estimate of the input image.

[0010] Based on the ambient light estimate, set the light source threshold for the input image;

[0011] The input image is divided into light source region and non-light source region based on the light source threshold. The transmittance of the light source region and non-light source region is optimized respectively by an adaptive light source matrix mechanism and then fused to solve for the initial transmittance.

[0012] Light source compensation is applied to the initial transmittance to obtain and output the final transmittance of the input image;

[0013] Based on the estimated ambient light value and the final transmittance, the input image is dehazed according to the atmospheric scattering model to obtain the dehazed output image.

[0014] Furthermore, the process of processing the input image using a hybrid method of side-window filtering and fast edge-preserving filtering includes:

[0015] The pixels of the input image are obtained, and a side window filter is used to treat each pixel as a potential edge, and several edge windows are generated around each pixel.

[0016] The input image is processed, including adjusting the brightness of the input image, obtaining and outputting the edge window with the smallest Euclidean distance between each pixel and the input image, so as to preserve the edge information of the input image;

[0017] The filtered image is obtained based on the edge window with the smallest distance in the output, and the filtered image is processed by fast edge-preserving filtering to obtain a preprocessed image;

[0018] Based on the preprocessed image, the ambient light estimate is calculated using the dark channel prior method.

[0019] Furthermore, the process of setting the light source threshold of the input image based on the ambient light estimate includes:

[0020] Calculate the luminance value of each pixel in the input image;

[0021] The difference between the luminance value and the estimated ambient light value is calculated, and the absolute value of the largest difference is taken as the light source threshold.

[0022] Furthermore, the process of dividing the input image into light source regions and non-light source regions based on the light source threshold, optimizing the transmittance of the light source regions and non-light source regions respectively using an adaptive light source matrix mechanism, and then fusing them to solve for the initial transmittance includes:

[0023] For any pixel in the input image, the pixel is classified as belonging to a light source region or a non-light source region; wherein, when the luminance value of the pixel is greater than the light source threshold, the pixel is classified as belonging to a light source region, otherwise the pixel is classified as belonging to a non-light source region.

[0024] When the pixel belongs to the light source region, an adaptive light source matrix mechanism is used to calculate and optimize the transmittance of the light source region; including:

[0025] Based on the light source matrix mechanism, each pixel in the input image is numbered in descending order of luminance value, and the first light source influence matrix, the first light source influence matrix, and the pixel light source influence matrix are calculated for each pixel. Specifically, the size of the input image is defined as w*h, x∈[1, w[, y∈[1, h[], and each pixel is numbered in descending order of luminance value as {m0, m1, ..., m2}. T-1}, where T is the total number of pixels; then,

[0026] For the value of x, the influence matrix k of the first light source x The calculation is as follows:

[0027]

[0028] For the value of y, the influence matrix k of the second light source y The calculation is as follows:

[0029]

[0030] The pixel light source influence matrix K x The calculation is as follows:

[0031]

[0032] Among them, C x C y This represents the luminance value of the pixel corresponding to the x and y values, where M is the light source threshold and d is the luminance value. x，m Represents the distance from other pixels to the selected pixel;

[0033] Based on the pixel light source influence matrix, the adjustment correction coefficient w for optimizing the transmittance of the light source region is obtained. x ,

[0034]

[0035] Where α represents the adjustment and correction factor of the input image, t x Initial transmittance;

[0036] When the pixel belongs to a non-light source region, the transmittance of the non-light source region is calculated using dark channel prior theory.

[0037] Based on the optimized transmittance of the light source region and the transmittance of the non-light source region, the initial transmittance t is calculated. M ,

[0038]

[0039] Where Ω represents the light source region, w represents the transmittance of the non-light source region. xt x∈Ω This indicates the transmittance of the light source region.

[0040] Furthermore, the process of performing light source compensation on the initial transmittance to obtain and output the final transmittance of the input image is as follows:

[0041] The gamma correction method is used to compensate for the light source by adjusting the compensation coefficient to obtain the final transmittance.

[0042] Furthermore, the process of obtaining the dehazed output image includes:

[0043] Based on the estimated ambient light value and the final transmittance, dehazing is performed on any pixel in the input image according to the atmospheric scattering model, using the following formula:

[0044]

[0045] Among them, I x t represents the initial image of any pixel in the input image. F It is the final transmittance, J x This indicates the image after dehazing of this pixel;

[0046] By combining the images after dehazing of each pixel, a dehazed output image is obtained.

[0047] Secondly, a drone system for monitoring low-light haze is disclosed, including:

[0048] The acquisition and processing module is used to acquire the pixels of the input image and preprocess the input image using a hybrid method of side window filtering and fast edge-preserving filtering to obtain the ambient light estimate of the input image.

[0049] The setting module is used to set the light source threshold of the input image based on the ambient light estimation value;

[0050] The partitioning and solving module is used to partition the input image into light source regions and non-light source regions according to the light source threshold, and then optimize the transmittance of the light source regions and non-light source regions respectively by using an adaptive light source matrix mechanism before fusing them to solve for the initial transmittance.

[0051] The compensation module is used to perform light source compensation on the initial transmittance to obtain and output the final transmittance of the input image;

[0052] The calculation module is used to perform dehazing calculations on the input image based on the estimated ambient light value and the final transmittance, according to the atmospheric scattering model, to obtain the dehazed output image.

[0053] Furthermore, the execution unit of the acquisition and processing module for obtaining the ambient light estimation value of the input image includes:

[0054] The acquisition unit is used to acquire pixels of the input image, and uses side window filtering to treat each pixel as a potential edge and generate several edge windows around each pixel;

[0055] An adjustment unit is used to process the input image, including adjusting the brightness of the input image, obtaining and outputting the edge window with the smallest Euclidean distance between each pixel and the input image, so as to preserve the edge information of the input image;

[0056] The processing unit is configured to obtain a filtered image based on the output edge window with the smallest distance, and process the filtered image using fast edge-preserving filtering to obtain a preprocessed image;

[0057] The calculation unit is used to calculate the ambient light estimate based on the preprocessed image using a dark channel prior method.

[0058] Thirdly, a computer device is disclosed, comprising at least one processor coupled to a memory, the memory storing a program or instructions running on the processor, the program or instructions being executed by the processor to implement the steps of the method for monitoring drones under low-light haze as described above.

[0059] Fourthly, a readable storage medium is disclosed, on which a program or instructions are stored, characterized in that, when the program or instructions are executed by a processor, they implement the steps of the above-described method for monitoring drones under low-light haze.

[0060] As can be seen from the above technical solutions, the technical solutions of the present invention have achieved the following beneficial effects:

[0061] The present invention discloses a method and system for monitoring drones under low-light haze conditions, which aims to solve the problem of drone monitoring in low-light haze conditions. Its advantages in specific applications are as follows:

[0062] (1) The scheme preprocesses low-light haze images by using a combination of side window filtering and fast edge-preserving filtering, eliminating the influence of high light source areas and pseudo light source areas on the ambient light estimation value of the image, thus improving the accuracy of the ambient light estimation value; it solves the problem that low-light haze images are too dark after defogging, and can monitor drones more clearly.

[0063] (2) The scheme sets the light source threshold based on the brightness value of the image and the estimated ambient light value after processing. It uses the light source matrix mechanism to solve the transmittance fusion of different light source areas and then performs light source compensation through gamma correction to obtain the final transmittance image. This solves the halo effect in the light source area after defogging of low-light images, improves the texture details of the image, and can better monitor drones in low-light haze conditions and identify dangerous items carried by drones.

[0064] (3) The solution improves the display effect of the sky area by modifying the ambient light estimation and optimizing the solution process of transmittance, reduces color difference, solves the problem of easy artifacts and noise, and prevents the monitoring effect of the UAV from being affected and the dangerous items carried cannot be clearly identified.

[0065] It should be understood that all combinations of the foregoing concepts and the additional concepts described in more detail below can be considered part of the inventive subject matter of this disclosure, provided that such concepts do not contradict each other.

[0066] The foregoing and other aspects, embodiments, and features of the teachings of the present invention will be more fully understood from the following description in conjunction with the accompanying drawings. Other additional aspects of the invention, such as features and / or beneficial effects of exemplary embodiments, will become apparent from the following description or may be learned through practice of specific embodiments according to the teachings of the present invention. Attached Figure Description

[0067] The accompanying drawings are not drawn to scale. In the drawings, each identical or nearly identical component shown in the various figures may be denoted by the same reference numeral. For clarity, not every component is labeled in each figure. Embodiments of various aspects of the invention will now be described by way of example and with reference to the accompanying drawings, wherein:

[0068] Figure 1 This is a flowchart of the method for monitoring drones under low-light haze disclosed in this invention;

[0069] Figure 2 This is a flowchart illustrating how the present invention obtains the ambient light estimate of an input image;

[0070] Figure 3 A flowchart for setting the light source threshold of the input image for this invention;

[0071] Figure 4 This is a framework diagram of the unmanned aerial vehicle (UAV) system for monitoring low-light haze disclosed in this invention;

[0072] Figure 5 This is a schematic diagram of an electronic device according to an embodiment of this application;

[0073] Figure 6 Images (a), (b), and (c) in the examples are all taken by drones under low-light haze conditions.

[0074] Figure 7 (a), (b), and (c) are the corresponding ones. Figure 6 Image showing the effect of hybrid filtering processing in drone monitoring;

[0075] Figure 8 (a), (b), and (c) are the corresponding ones. Figure 7 Transmittance image obtained after fusion solution using adaptive light source matrix mechanism;

[0076] Figure 9 (a), (b), and (c) are the corresponding ones. Figure 8 Image showing the monitoring effect of a drone after fog removal. Detailed Implementation

[0077] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the described embodiments of the present invention without creative effort are within the scope of protection of the present invention. Unless otherwise defined, the technical or scientific terms used herein should have the ordinary meaning understood by those skilled in the art to which this invention pertains.

[0078] The terms "first," "second," and similar words used in the specification and claims of this patent application do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Similarly, unless the context clearly indicates otherwise, the singular forms of "an," "a," or "the," etc., do not indicate a quantity limitation, but rather indicate the presence of at least one. Terms such as "comprising" or "including" mean that the element or object preceding "comprising" encompasses the features, integrals, steps, operations, elements, and / or components listed following "comprising" or "including," and do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or collections thereof. Terms such as "upper," "lower," "left," and "right" are used only to indicate relative positional relationships; these relative positional relationships may change accordingly when the absolute position of the described object changes.

[0079] Existing technologies for monitoring drones in low-light haze conditions using image processing have the following problems: 1) Existing dark channel prior theories, when processing daytime haze images, rely on a single ambient light estimate. At night, the ambient light value calculation is affected by various lights, leading to deviations and a darker defogging effect; 2) While guided filtering can alleviate the block effect in transmittance images caused by image segmentation, residual block effects remain significant when the coarse transmittance values ​​of two adjacent blocks differ greatly. This results in noticeable halo effects near strong lights in the final output image; furthermore, while existing algorithms have made improvements, they neglect the influence of the sky region, causing color differences and artifacts in the sky area of ​​the image. Therefore, this invention proposes a low-light defogging method applicable to special scenarios such as low-light haze, solving the above problems and enabling drone monitoring and identification of dangerous items.

[0080] The following detailed description, in conjunction with the embodiments shown in the accompanying drawings, further illustrates the method and system for monitoring drones under low-light haze conditions disclosed in this invention.

[0081] Combination Figure 1 As shown in the embodiment, the method for monitoring drones under low-light haze includes the following steps:

[0082] Step S102: Obtain the pixels of the input image, and preprocess the input image using a hybrid method of side window filtering and fast edge-preserving filtering to obtain the ambient light estimation value of the input image; the purpose of this step is to process the high light source points and pseudo light source points of the image to eliminate their influence on the ambient light estimation.

[0083] Step S104: Set the light source threshold of the input image based on the ambient light estimation value;

[0084] Step S106: Divide the input image into light source region and non-light source region according to the light source threshold, and then use an adaptive light source matrix mechanism to optimize the transmittance of the light source region and non-light source region respectively, and then fuse them to solve for the initial transmittance.

[0085] Step S108: Perform light source compensation on the initial transmittance to obtain and output the final transmittance of the input image;

[0086] Step S110: Based on the estimated ambient light value and the final transmittance, perform dehazing calculations on the input image according to the atmospheric scattering model to obtain the dehazed output image.

[0087] Specific examples Figure 2As shown, the process of processing the input image using a hybrid method of side window filtering and fast edge-preserving filtering includes the following steps: Step S1022, acquiring the pixels of the input image, using side window filtering to treat each pixel as a potential edge, and generating several edge windows around each pixel; Step S1024, processing the input image, including adjusting the brightness of the input image, obtaining and outputting the edge window with the smallest Euclidean distance between each pixel and the input image, so as to preserve the edge information of the input image; Step S1026, obtaining a filtered image based on the output edge window with the smallest distance, and processing the filtered image using fast edge-preserving filtering to obtain a preprocessed image; Step S1028, calculating the ambient light estimate using the dark channel prior method based on the preprocessed image.

[0088] The calculation process for obtaining the edge window with the smallest distance using side window filtering in steps S1022 to S1024 is as follows:

[0089]

[0090]

[0091]

[0092] Where S = {U, D, L, R, NM, NE, SW, SE}, represents the set of eight edge windows generated around a pixel; n ∈ S, represents any edge window in set S; I n q represents the filter value of the edge window; i w represents the weight of pixels j near the target pixel i based on the kernel function F; ij N is the pixel value of a pixel; n Represents the sum of pixel values ​​in the eight edge windows; I sw This represents the edge window that has the smallest Euclidean distance to the pixel input image.

[0093] Step S104 is the process of setting the light source threshold of the input image based on the ambient light estimation value, as follows: Figure 3 The process includes the following steps: Step S1042, calculating the luminance value of each pixel in the input image; Step S1044, calculating the difference between the luminance value and the estimated ambient light value, and using the absolute value of the largest difference as the light source threshold.

[0094] That is, to obtain the luminance value I of each pixel in the image. x The difference between the estimated ambient light value A and |I is calculated. x -A| max As a light source threshold, it is used to distinguish between light source areas and non-light source areas.

[0095] In a specific embodiment, step S106 above, which divides the input image into light source regions and non-light source regions based on the light source threshold, optimizes the transmittance of the light source regions and non-light source regions respectively using an adaptive light source matrix mechanism, and then fuses and solves the initial transmittance, includes the following calculation process:

[0096] For any pixel in the input image, the pixel is classified as belonging to a light source region or a non-light source region; wherein, when the luminance value of the pixel is greater than the light source threshold, the pixel is classified as belonging to a light source region, otherwise the pixel is classified as belonging to a non-light source region.

[0097] When the pixel belongs to the light source area, i.e., I x >|I x -A| max The transmittance of the light source region is calculated and optimized using an adaptive light source matrix mechanism, including:

[0098] Based on the light source matrix mechanism, each pixel in the input image is numbered in descending order of luminance value, and the first light source influence matrix, the first light source influence matrix, and the pixel light source influence matrix are calculated for each pixel. Specifically, the size of the input image is defined as w*h, x∈[1, w], y∈[1, h], and each pixel is numbered in descending order of luminance value as {m0, m1, ..., m T-1}, where T is the total number of pixels; then,

[0099] For the value of x, the influence matrix k of the first light source x The calculation is as follows:

[0100]

[0101] For the value of y, the influence matrix k of the second light source y The calculation is as follows:

[0102]

[0103] The pixel light source influence matrix K x The calculation is as follows:

[0104]

[0105] Among them, C x C y This represents the luminance value of the pixel corresponding to the x and y values, where M is the light source threshold and d is the luminance value. x，m Represents the distance from other pixels to the selected pixel;

[0106] Based on the pixel light source influence matrix, the adjustment correction coefficient w for optimizing the transmittance of the light source region is obtained. x ,

[0107]

[0108] Where α represents the adjustment and correction factor of the input image, which is set to 0.5 in this embodiment; t x Indicates the initial transmittance;

[0109] When the pixel belongs to a non-light source area, i.e., I x ≤|I x -A| max The transmittance of the non-light source region is calculated using dark channel prior theory.

[0110] Based on the optimized transmittance of the light source region and the transmittance of the non-light source region, the initial transmittance t is calculated. M ,

[0111]

[0112] Where Ω represents the light source region, w represents the transmittance of the non-light source region. x t x∈Ω This indicates the transmittance of the light source region.

[0113] In a further embodiment, the present invention employs gamma correction, which adjusts the compensation coefficient to compensate for the light source of the initial transmittance, and finally outputs the final transmittance of the input image; in practice, the compensation coefficient is set to 0.8.

[0114] The process of obtaining the dehazed output image in step S110 of the present invention includes the following calculation process:

[0115] First, based on the estimated ambient light value and the final transmittance, dehazing is performed on any pixel in the input image according to the atmospheric scattering model, using the following formula:

[0116]

[0117] Among them, I x t represents the initial image of any pixel in the input image. F It is the final transmittance, J x This indicates the image after dehazing of this pixel;

[0118] Then, by combining the images after dehazing of each pixel, the dehazed output image is obtained.

[0119] This invention discloses a method for monitoring drones under low-light haze conditions. First, it preprocesses the low-light haze image using a hybrid approach of edge-window filtering and fast edge-preserving filtering to eliminate the influence of high-light and pseudo-light-source areas on the ambient light estimation value. Then, it sets a light source threshold based on the image and ambient light estimation value, employs a light source matrix mechanism to fuse and solve the transmittance of different light source regions, and performs light source compensation through gamma correction to obtain the final transmittance image, thus solving the halo effect problem in light source areas after defogging in low-light images. Finally, by modifying the ambient light estimation and optimizing the transmittance solution process, it improves the display effect of the sky area, reduces color difference, and solves the problems of artifacts and noise easily generated during image processing. This method can be effectively applied to drone identification problems in special scenarios such as low-light haze, improving the safe management and control of drones.

[0120] In embodiments of the present invention, a computer device is also provided, such as... Figure 5 As shown, the device includes at least one processor coupled to a memory, the memory storing a program or instructions that run on the processor, the program or instructions which, when executed by the processor, implement the steps of the method for monitoring drones under low-light haze as disclosed in the above embodiments.

[0121] The aforementioned program can run in a processor or be stored in memory, i.e., in a computer-readable medium. Computer-readable media includes both permanent and non-permanent, removable and non-removable media, and information storage can be achieved by any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include temporary computer-readable media, such as modulated data signals and carrier waves.

[0122] These computer programs may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1The steps for the functions specified in one or more boxes can be implemented through different modules, corresponding to different method steps.

[0123] This embodiment provides such a device or system, which can be called a drone system for monitoring low-light fog and haze. The system is as follows: Figure 4 As shown, the system includes: an acquisition and processing module for acquiring pixels of the input image, preprocessing the input image using a hybrid method of side-window filtering and fast edge-preserving filtering to obtain an ambient light estimate of the input image; a setting module for setting a light source threshold for the input image based on the ambient light estimate; a partitioning and solving module for partitioning the input image into light source regions and non-light source regions based on the light source threshold, optimizing the transmittance of the light source regions and non-light source regions respectively using an adaptive light source matrix mechanism, and then fusing them to solve for the initial transmittance; a compensation module for performing light source compensation on the initial transmittance to obtain and output the final transmittance of the input image; and a calculation module for performing dehazing calculations on the input image based on the ambient light estimate and the final transmittance, according to an atmospheric scattering model, to obtain a dehazed output image.

[0124] The steps of the system for implementing the low-light fog and haze monitoring drone system disclosed in the above embodiments have already been described and will not be repeated here.

[0125] For example, the execution unit of the acquisition and processing module for obtaining the ambient light estimation value of the input image includes: an acquisition unit for acquiring pixels of the input image, treating each pixel as a potential edge using side window filtering, and generating several edge windows around each pixel; an adjustment unit for processing the input image, including adjusting the brightness of the input image, obtaining and outputting the edge window with the smallest Euclidean distance between each pixel and the input image, so as to preserve the edge information of the input image; a processing unit for obtaining a filtered image based on the output edge window with the smallest distance, processing the filtered image using fast edge-preserving filtering, and obtaining a preprocessed image; and a calculation unit for calculating the ambient light estimation value based on the preprocessed image using a dark channel prior method.

[0126] The process by which the adjustment unit obtains the edge window with the minimum Euclidean distance between each pixel and its input image is as follows:

[0127]

[0128]

[0129]

[0130] Where S = {U, D, L, R, NM, NE, SW, SE}, represents the set of eight edge windows generated around a pixel; n ∈ S, represents any edge window in set S; I n q represents the filter value of the edge window; i w represents the weight of pixels j near the target pixel i based on the kernel function F; ij N is the pixel value of a pixel; n Represents the sum of pixel values ​​in the eight edge windows; I sw This represents the edge window that has the smallest Euclidean distance to the pixel input image.

[0131] For example, the process of setting the light source threshold of the input image based on the ambient light estimation value by the setting module includes the following execution units: a second calculation unit for calculating the luminance value of each pixel in the input image; and a third calculation unit for calculating the difference between the luminance value and the ambient light estimation value, and using the absolute value of the largest difference as the light source threshold.

[0132] For example, the partitioning and solving module divides the input image into light source regions and non-light source regions based on the light source threshold, and then uses an adaptive light source matrix mechanism to optimize the transmittance of the light source regions and non-light source regions respectively before fusing them to solve for the initial transmittance. This process includes the following calculation steps:

[0133] For any pixel in the input image, the pixel is classified as belonging to a light source region or a non-light source region; wherein, when the luminance value of the pixel is greater than the light source threshold, the pixel is classified as belonging to a light source region, otherwise the pixel is classified as belonging to a non-light source region.

[0134] When the pixel belongs to the light source area, i.e., I x >|I x -A| max The transmittance of the light source region is calculated and optimized using an adaptive light source matrix mechanism, including:

[0135] Based on the light source matrix mechanism, each pixel in the input image is numbered in descending order of luminance value, and the first light source influence matrix, the first light source influence matrix, and the pixel light source influence matrix are calculated for each pixel. Specifically, the size of the input image is defined as w*h, x∈[1, w], y∈[1, h], and each pixel is numbered in descending order of luminance value as {m0, m1, ..., m T-1}, where T is the total number of pixels; then,

[0136] For the value of x, the influence matrix k of the first light source x The calculation is as follows:

[0137]

[0138] For the value of y, the influence matrix k of the second light source y The calculation is as follows:

[0139]

[0140] The pixel light source influence matrix K x The calculation is as follows:

[0141]

[0142] Among them, C x C y This represents the luminance value of the pixel corresponding to the x and y values, where M is the light source threshold and d is the luminance value. x，m Represents the distance from other pixels to the selected pixel;

[0143] Based on the pixel light source influence matrix, the adjustment correction coefficient w for optimizing the transmittance of the light source region is obtained. x ,

[0144]

[0145] Where α represents the adjustment and correction factor of the input image, which is set to 0.5 in this embodiment; t x Indicates the initial transmittance;

[0146] When the pixel belongs to a non-light source area, i.e., I x ≤|I x -A| max The transmittance of the non-light source region is calculated using dark channel prior theory.

[0147] Based on the optimized transmittance of the light source region and the transmittance of the non-light source region, the initial transmittance t is calculated. M ,

[0148]

[0149] Where Ω represents the light source region, w represents the transmittance of the non-light source region. x t x∈Ω This indicates the transmittance of the light source region.

[0150] For example, the process by which the calculation module obtains the dehazed output image includes the following execution units:

[0151] The fourth calculation unit is used to perform dehazing calculations on any pixel in the input image based on the estimated ambient light value and the final transmittance, according to the atmospheric scattering model. The formula is as follows:

[0152]

[0153] Among them, I x t represents the initial image of any pixel in the input image. F It is the final transmittance, J x This indicates the image after dehazing of this pixel;

[0154] The output unit is used to synthesize the images after dehazing of each pixel to obtain the dehazed output image.

[0155] The specific implementation process of this invention will be fully explained below with reference to the drone flight experiment in the low-light haze scenario shown in the attached figure.

[0156] First, a hybrid approach of side-window filtering and fast edge-preserving filtering is used to preprocess the light source points, eliminating the influence of high-brightness and pseudo-light source brightness values ​​on the ambient light estimation in low-light hazy scenes; specifically, as follows... Figure 6 and Figure 7 ,in, Figure 6 This is a direct shot taken under low-light smog conditions. Figure 7 The processed image improves the accuracy of ambient light values.

[0157] Secondly, the difference between the brightness value of each pixel in the image and the estimated ambient light value is calculated, and the maximum value is used as the light source threshold to distinguish between light source regions and non-light source regions. For non-light source regions, the transmittance is solved using the traditional dark channel prior theory. For light source regions, the transmittance image is optimized using a light source matrix mechanism and then fused together. Finally, light source compensation is performed on the transmittance image using gamma correction to obtain the final transmittance image, as shown below. Figure 8 As shown in the figure, the overall brightness and detail have been greatly improved, which has preserved the light source area well and made it less prone to halo effect when defogging.

[0158] Finally, the ambient light estimate and transmittance image processed in the above steps are used to perform dehazing calculations on the low-light haze image to obtain the dehazed output image; for example... Figure 9 As shown, the image is bright and clear overall, with good light retention in the light source area, no halo effect, reduced color difference in the sky area, and no artifacts or noise. It can effectively monitor drones in low-light and hazy conditions, identify the movements of nearby drones during takeoff and landing, and prevent major accidents.

[0159] While the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the invention. Those skilled in the art can make various modifications and refinements without departing from the spirit and scope of the invention. Therefore, the scope of protection of the present invention shall be determined by the claims.

Claims

1. A method for monitoring unmanned aerial vehicles (UAVs) under low-illumination haze, the method comprising: The method comprises the following steps: obtaining pixels of an input image, and pre-processing the input image by using a mixed method of edge window filtering and fast edge-preserving filtering to obtain an ambient light estimation value of the input image; setting a light source threshold value of the input image according to the ambient light estimation value; dividing a light source region and a non-light source region of the input image according to the light source threshold value, and respectively optimizing transmittances of the light source region and the non-light source region by using an adaptive light source matrix mechanism to solve an initial transmittance, comprising: dividing any pixel point in the input image into the light source region or the non-light source region; wherein when a luminosity value of the pixel point is greater than the light source threshold value, the pixel point is divided into the light source region, otherwise the pixel point is divided into the non-light source region; when the pixel point belongs to the light source region, the transmittance of the light source region is calculated and optimized by using the adaptive light source matrix mechanism; comprising: According to the light source matrix mechanism, each pixel point in the input image is numbered in descending order of luminosity value, a first light source influence matrix of each pixel point, a first light source influence matrix and a pixel light source influence matrix are calculated; specifically, the size of the input image is defined as , , each pixel point is numbered in descending order of luminosity value , wherein T is the total number of pixel points; then, In In the case where the value of the first light source influence matrix is calculated as follows: ； In The second light source influence matrix Is calculated as follows: ； The pixel light source influence matrix The calculation is as follows: ； wherein, C x , C y represents the luminosity value of the pixel point corresponding to the x, y values, M is a light source threshold value, d x，m represents the distance of other pixels to the selected pixel point; According to the pixel light source influence matrix, an adjustment correction coefficient of an optimized light source region transmittance is obtained w x , ， wherein, denotes an adjustment correction factor for the input image, t x denotes an initial transmittance; when the pixel point belongs to the non-light source region, the transmittance of the non-light source region is calculated by using a dark channel prior theory; According to the transmittance of the light source region and the transmittance of the non-light source region after optimization, the initial transmittance is fused and calculated t M , ， wherein represents the transmittance of the light source region, represents the transmittance of the non-light source region, represents the transmittance of the light source region; performing light source compensation on the initial transmittance to obtain and output a final transmittance of the input image; performing defogging calculation on the input image according to an atmospheric scattering model based on the ambient light estimation value and the final transmittance to obtain an output image after defogging. 2.The method of claim 1, wherein, The process of pre-processing the input image by using the mixed method of edge window filtering and fast edge-preserving filtering comprises the following steps: obtaining pixels of an input image, and regarding each pixel as a potential edge by using edge window filtering to generate a plurality of edge windows around each pixel; processing the input image, comprising adjusting brightness of the input image, obtaining and outputting an edge window with the smallest Euclidean distance between each pixel and the input image, so as to retain edge information of the input image; obtaining a filtered image according to the output distance-minimized edge window, and processing the filtered image by using fast edge-preserving filtering to obtain a pre-processed image; calculating an ambient light estimation value by using a dark channel prior method based on the pre-processed image. 3.The method of claim 1, wherein, The process of setting a light source threshold value of the input image according to the ambient light estimation value comprises the following steps: calculating luminosity values of each pixel point in the input image; calculating a difference value between the luminosity value and the ambient light estimation value, and taking an absolute value of the maximum difference value as the light source threshold value. 4.The method for monitoring unmanned aerial vehicle under low-illumination haze according to claim 1, wherein, The process of performing light source compensation on the initial transmittance to obtain and output a final transmittance of the input image comprises the following steps: performing light source compensation on the initial transmittance by adjusting a compensation coefficient by using a gamma correction method to obtain the final transmittance. 5.The method for monitoring unmanned aerial vehicle under low-illumination haze based on claim 1, characterized in that, The process of obtaining an output image after defogging comprises the following steps: performing defogging calculation on any pixel point in the input image according to an atmospheric scattering model based on the ambient light estimation value and the final transmittance, and the formula is: ； wherein, I x represents an initial image of any pixel point in the input image, t F is the final transmittance, J x represents the image after the pixel point defogging processing; combining images after defogging processing of each pixel point to obtain an output image after defogging. 6.A low-illumination haze-based unmanned aerial vehicle monitoring system for performing the low-illumination haze-based unmanned aerial vehicle monitoring method according to claim 1. The method comprises the following steps: an obtaining module is configured to obtain pixels of an input image, and pre-process the input image by using a mixed method of edge window filtering and fast edge-preserving filtering to obtain an ambient light estimation value of the input image; a setting module is configured to set a light source threshold value of the input image according to the ambient light estimation value; The division solving module is configured to divide a light source region and a non-light source region of the input image according to the light source threshold, and to optimize transmittance of the light source region and the non-light source region respectively by using an adaptive light source matrix mechanism and then to fuse to solve an initial transmittance. The compensation module is configured to perform light source compensation on the initial transmittance to obtain and output a final transmittance of the input image. The calculation module is configured to perform defogging calculation on the input image according to the ambient light estimation value and the final transmittance based on an atmospheric scattering model to obtain an output image after defogging.

7. The low-illumination haze-based monitoring unmanned aerial system of claim 6, wherein, The execution unit of the acquisition processing module for obtaining an ambient light estimation value of an input image comprises: The acquisition unit is configured to acquire pixels of the input image, to regard each pixel as a potential edge by using edge window filtering, and to generate a plurality of edge windows around each pixel. The adjustment unit is configured to perform processing on the input image, including adjusting brightness of the input image, to obtain and output an edge window with minimum Euclidean distance of each pixel to its input image, so as to retain edge information of the input image. The processing unit is configured to obtain a filtered image according to the output distance-minimum edge window, to perform processing on the filtered image by using fast edge-preserving filtering, and to obtain a preprocessed image. The calculation unit is configured to calculate an ambient light estimation value by using a dark channel prior method according to the preprocessed image.

8. A computer device, comprising: The program or instruction is executed by the processor to implement the steps of the method for monitoring a UAV under low-illumination haze according to any one of claims 1 to 5.

9. A readable storage medium, on which a program or instructions are stored, characterized in that, The program or instruction is executed by the processor to implement the steps of the method for monitoring a UAV under low-illumination haze according to any one of claims 1 to 5.

Images