Image defogging method and image defogging apparatus using the same

Abstract

The present invention relates to an image defogging method and an image defogging apparatus using the image defogging method. In an aspect, the present invention relates to an image defogging method, the method including, but not limited to: receiving an input image; defogging the image by a defogging module to output a defogged red green blue image; recovering image luminance of the defogged red green blue image by a high dynamic range (HDR) module to output a high dynamic range image; and removing reflections of the high dynamic range image by a ReflectNet inference model, wherein the ReflectNet inference model uses a deep learning architecture.

Description

Technical Field

[0001] The present invention relates to an image defogging method and an image defogging device using the image defogging method. Background Art

[0002] Since the advent of silver chloride-coated paper technology, imaging devices can pass light through a slit to expose the silver chloride-coated paper, which can spatially darken the image according to the intensity of the traveling light, and as more and more end users become available, indirectly supporting the advancement from static dynamic image capture to dynamic image capture (i.e., cameras), the utilization of imaging devices has increased year by year. From professional and scientific to consumer electronics, the applicability of imaging devices has been unprecedentedly broad. With the support of large-scale production systems, the cost per pixel density of imaging devices has been correspondingly reduced. These conditions have led to the widespread application of imaging devices (i.e., cameras) in related engineering fields.

[0003] The advancement of Internet-of-Thing (IoT) capabilities in consumer electronics has made the development of autonomous driving vehicles (ADVs) even more important. Figure 1 The following figure shows an ADV that uses different imaging sensors to sense the surrounding environment of obstacles. In the application scenario of ADV, the red, green and blue (RGB) camera is the most prominent imaging device. The RGB camera can detect the surrounding environment of obstacles in the limited field of view of the corresponding camera. o The image data provides dense semantic or instance information about the surrounding environment within the field of view (FOV). This specialized information provides the actuators with the necessary data to achieve the primary goal of ADV: autonomously moving from one location to another while avoiding objects or obstacles within the predicted range of the surrounding trajectory. The image data can then be processed in various ways, such as for object detection, image segmentation, object tracking, depth estimation, and other related applications.

[0004] In addition to the capabilities of subsequent processing modules, the perceived quality from the corresponding RGB camera can impact their performance. Table 1 compares the properties of RGB cameras under various environmental constraints. As shown in Table 1, RGB cameras are susceptible to interference from light and weather conditions. Because RGB cameras are passive sensors, their image data is highly dependent on the traveling light signal perceived by the sensor array. Any airborne particles or intervening matter that could attenuate or disperse the traveling light signal is considered a decimating factor for RGB cameras.

[0005] Limiting condition Attribute Limiting condition Attribute Sensor type Passive Weather influence Light sensitivity Lumens (Lux) interference High sensitivity Field of view Limited Exposure effect High sensitivity Resolution Dense

[0006] Table 1

[0007] In ADV development, varying weather conditions are uncontrollable variables that will affect RGB camera image quality to some extent. Adverse weather conditions commonly encountered in ADV development can be categorized as: (a) sun glare / direct light exposure; (b) rain / snow conditions; and (c) foggy / hazy conditions. For scenarios involving sun glare / direct light exposure, RGB camera image quality degrades as direct exposure saturates the light sensor array, potentially affecting not only the exposed image area but also its neighboring image areas. Due to the overexposed pixels and their neighboring areas, this particular scenario reduces the perceived objectness by lowering image entropy. Consequently, image features of prominent objects are suppressed, making the task of generating appropriate detections more challenging for most object detection algorithms.

[0008] For scenarios involving rain / snow conditions, the presence of intervening matter (e.g., water droplets or snowflakes) will attenuate the amount of light entering the camera lens, with the degree of attenuation directly corresponding to the intensity of the rain / snow conditions. These matter may also cause image distortion and / or FOV obstruction depending on how the RGB camera is mounted and / or designed. Similarly, scenarios involving foggy conditions will also reduce the visibility of the RGB camera because the presence of humid air / pollutants will absorb a certain degree of light intensity, depending on the severity of the corresponding haze / fog condition. Therefore, similar to rain / snow conditions, foggy conditions will also make the execution of most object detection algorithms, such as feature extraction analysis, more challenging due to the suppression of most high-frequency information.

[0009] As a practical example, Figure 2A Under the clear afternoon conditions depicted in Figure 2, the perceived object is clear, with considerable contrast and image visibility. However, Figure 2B Unfavorable conditions, such as the unbalanced intensity in low-light environments and the undesirable conditions created by heavy rain, can pose reliability issues for RGB cameras used for ambient object sensing on autonomous vehicles. During periods of unbalanced intensity in low-light environments, contrast ratios are suppressed due to overexposure of intensities of other pixels that shift into the higher range of histogram values. Heavy rain conditions increase atmospheric channeling due to the presence of airborne matter, which attenuates the amount of traveling light entering the sensor array. These conditions highlight the limitations of RGB cameras as passive sensing devices.

[0010] In fact, a software solution as an image pre-processing function to enhance the quality of RGB images can become a patch element to the RGB camera whose image quality is usually affected by adverse weather conditions. However, to significantly address the reliability of the RGB camera during any type of weather condition (adverse or normal), a hardware improvement of the RGB camera is necessary as a complementary patch to any software solution. In the image pre-processing function, the RGB images are processed at pixel or region level, with the common purpose of enhancing the image contrast and improving the perceived objectness through selective filtering (i.e., haze removal).

[0011] Figure 2C The performance of an object detection algorithm in the related art (e.g., YOLOv3 taught by J. Redmon et al., “YOLOv3: An Incremental Improvement,” arXiv: 1804.02767, April 2018) is compared with input images from raw and processed RGB versions. The raw RGB images are obtained directly from the corresponding cameras under adverse weather conditions, such as (from top-left to bottom-right) overexposure, underexposure, rain condition, and fog condition. On the other hand, the processed RGB images are the products of the image pre-processing function to the corresponding raw RGB images; thus, the processed RGB images can be called enhanced versions of the raw RGB images. In the present technology, image dehazing and high dynamic range (HDR) enhancement are used as the image pre-processing function.

[0012] Reference Figure 2C In the related art, the detection rate of the object detection algorithm implemented using the processed RGB images is lower compared to the detection rate of the object detection algorithm implemented using the raw RGB images. This indicates that the image pre-processing is able to enhance the perceived objectness of the raw RGB images; thus, the classifier implementing the algorithm can detect any previously considered object with higher image visibility. Additionally, both the raw and processed RGB images are equally low in false positive rate. Although this can be attributed to the stability of the object detection algorithm implemented in the related art; this can demonstrate the capability of the image pre-processing in avoiding over-enhancement that would produce undesirable artifacts causing false positives.

[0013] In light of the foregoing, image pre-processing techniques that adjust and improve the perceptual quality of RGB images are critical for subsequent computational photography. Applications of the processed RGB images can be extended in the range of computer and machine vision. The most common application is autonomous vehicles that need RGB cameras as sensing devices, including ground, air, or sea vehicles. However, since those RGB cameras have proven to have a problem of quality degradation under various adverse weather conditions, an image pre-processing function is needed as a solution patch to improve image contrast and perceived objects under various weather conditions (adverse or normal weather conditions) without causing any error or over-enhancement effect. SUMMARY

[0014] In view of the foregoing, embodiments of the present disclosure provide an image defogging method and an image defogging apparatus using the image defogging method.

[0015] In an aspect, embodiments of the present disclosure provide an image defogging method, and the method includes, but is not limited to, receiving an input image; defogging the image by a defogging module to output a defogged RGB image; recovering image brightness of the defogged RGB image by a high dynamic range (HDR) module to output an HDR image; and removing reflections of the HDR image by a ReflectNet inference model, wherein the ReflectNet inference model uses a deep learning architecture.

[0016] In an aspect, embodiments of the present disclosure provide an image defogging apparatus, including, but not limited to, a sensor and a processor coupled to the sensor. The processor is configured to at least perform the following operations: receiving an input image; defogging the image by a defogging module to output a defogged RGB image; recovering image brightness of the defogged RGB image by an HDR module to output an HDR image; and removing reflections of the HDR image by a ReflectNet inference model, wherein the ReflectNet inference model uses a deep learning architecture.

[0017] In order that the manner in which the above-recited and other features and advantages of certain embodiments of the present disclosure can be understood in detail, a few brief descriptions of implementations with reference to accompanying drawings are summarized here. It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only, and are not restrictive of the application, as claimed.

[0018] It should be understood, however, that the Summary can not contain all aspects and embodiments of the present application and is therefore not intended to limit or restrict the application in any way. Further, the application will cover any improvements and modifications. BRIEF DESCRIPTION OF DRAWINGS

[0019] The accompanying drawings are included to further understand the present application, and are incorporated in and constitute a part of this specification. The drawings illustrate exemplary embodiments of the present application and, together with the description, serve to explain the principles of the present application.

[0020] Figure 1 An autonomous driving vehicle (ADV) is illustrated that utilizes different imaging sensors to sense the surrounding environment of obstacles.

[0021] Figure 2A An input RGB image is shown that is acquired by an RGB camera under sunny afternoon conditions.

[0022] Figure 2B An RGB camera is shown to be susceptible to adverse conditions such as night exposure, heavy rain, sun glare, thick fog, and dim night.

[0023] Figure 2C The effect of image pre-processing on an RGB image for an object detection algorithm is shown.

[0024] Figure 3 is a flowchart illustrating the steps of an image dehazing method according to an exemplary embodiment of the present application.

[0025] Figure 4 is a block diagram illustrating an image dehazing apparatus according to an exemplary embodiment of the present application.

[0026] Figure 5 is a flowchart illustrating the steps of an image dehazing method according to an exemplary embodiment of the present application.

[0027] Figure 6 are implementation examples of an input RGB image and an output RGB image processed by an image dehazing method according to an exemplary embodiment of the present application.

[0028] Figure 7 is a detailed flowchart illustrating the steps of an image dehazing method according to an exemplary embodiment of the present application.

[0029] Figure 8 is a detailed flowchart illustrating the algorithm of a dehazing module according to an exemplary embodiment of the present application.

[0030] Figure 9 are implementation examples of an input RGB image and an output dehazed image processed by the algorithm of a dehazing module.

[0031] Figure 10 are implementation examples of an input RGB image and a minimum channel map processed by a minimum channel estimation algorithm in a dehazing module according to an exemplary embodiment of the present application.

[0032] Figure 11Implementation example of the inverse minimum channel map and the haze transmission map processed by the guided filtering algorithm in the defogging module according to the exemplary embodiment of the present invention.

[0033] Figure 12 Implementation example of the atmospheric detection algorithm in the defogging module to obtain intensity values from the haze transmission map according to the exemplary embodiment of the present invention.

[0034] Figure 13 Implementation example of the linear color reconstruction algorithm in the defogging module to recover the defogged RGB image according to the exemplary embodiment of the present invention.

[0035] Figure 14 is a detailed flowchart illustrating the algorithm of the HDR module according to the exemplary embodiment of the present invention.

[0036] Figure 15 Implementation example of the input defogged image to the HDR module and the output HDR image processed by the HDR module according to the exemplary embodiment of the present invention.

[0037] Figure 16 is a flowchart illustrating that the defogged RGB image is enhanced in image brightness by the Naka-Rushton Compensator according to the exemplary embodiment of the present invention.

[0038] Figure 17 Implementation example of enhancing the image contrast of the defogged RGB image by the Local Histogram Operation according to the exemplary embodiment of the present invention.

[0039] Figure 18 is a flowchart illustrating that the reflection of the HDR image is removed by the ReflectNet inference model according to the exemplary embodiment of the present invention.

[0040] Figure 19 Implementation example of the input defogged image processed by the ReflectNet inference model and the output HDR image according to the exemplary embodiment of the present invention.

[0041] Figure 20 Implementation example of the encoder-decoder architecture of the ReflectNet inference model according to the exemplary embodiment of the present invention.

[0042] Figure 21 Implementation example of the original RGB image and the reflected version of the RGB image in the dataset for training the ReflectNet inference model according to the exemplary embodiment of the present invention.

[0043] Figure 22Table of comparison of performance evaluation of the embodiment of the image defogging method with the benchmark method in the prior art.

[0044] Figure 23 Table of comparison of performance evaluation of the embodiment of the image defogging method with the benchmark method in the prior art.

[0045] Figure 24 Examples of the input original image and the output image processed by the image defogging method according to the exemplary embodiment of the present application are shown compared with the output image processed by the benchmark method in the prior art.

[0046] Figure 25 Table showing performance evaluation of object detection of the RGB image processed by the image defogging method according to the exemplary embodiment of the present application compared with performance evaluation of object detection of the original RGB image and the RGB image processed by the benchmark method in the prior art.

[0047] Figure 26 Examples of the object detection algorithm for the original image, the image processed by the benchmark method in the prior art, and the image processed by the image defogging method according to the exemplary embodiment of the present application are shown.

[0048] BRIEF DESCRIPTION OF DRAWINGS

[0049] 40: image defogging device;

[0050] 401: sensor;

[0051] 402: processor;

[0052] 601, 602, 901, 902: image;

[0053] 1001: input RGB image;

[0054] 1002: minimum channel map;

[0055] 1101: inverted minimum channel map;

[0056] 1102, 1201: haze transmission map;

[0057] 1202: defogged image;

[0058] 1203: intensity value;

[0059] 1301: input image;

[0060] 1302: defogged RGB image;

[0061] 1501, 1901: input defogged image;

[0062] 1502, 1902: output high dynamic range image;

[0063] 1601: original RGB image;

[0064] 1602: luminance map;

[0065] 1603, 1604, 1605: channels;

[0066] 1606: bright pixels;

[0067] 1607: dark pixels;

[0068] S1611, S1612, S301, S302, S303, S304, S501, S502, S503, S504, S505, S701, S702, S703, S704, S801, S802, S803, S804, S1401, S1402, S1801: step;

[0069] 1701: dehazed RGB image I D () ;

[0070] 1702: output HDR image I H () ;

[0071] 2301: entropy-based measure enhancement;

[0072] 2302: contrast improvement index;

[0073] 2303: intensity measure;

[0074] 2501: accuracy;

[0075] 2502: false positive rate. DETAILED DESCRIPTION

[0076] Reference will now be made in detail to the presently exemplary embodiments of the application, examples of which are illustrated in the accompanying drawings. Wherever possible, the same reference numbers are used in the drawings and the description to refer to the same or like parts.

[0077] For the above challenges, the present invention provides a hybrid image processing framework to address various challenges that can be encountered in the deployment of digital vision technology. The image processing framework will be explained from the perspective of an image dehazing device that uses an image dehazing method. The image dehazing device can be housed within an automated or any robotic system in, for example, an outdoor environment, such as an autonomous vehicle, aerial drone, and marine drone, among others. The image dehazing device will include, but not limited to, a sensor and a processor coupled to the sensor, and the processor is configured to at least: receive an input image; dehaze the image by a dehazing module to output a dehazed RGB image; recover image brightness of the dehazed RGB image by an HDR module to output an HDR image; and remove reflections of the HDR image by a ReflectNet inference model, wherein the ReflectNet inference model uses a deep learning architecture. Through the provided framework, image quality can be enhanced and performance of subsequent processing (e.g., object detection and image segmentation) can be improved.

[0078] Figure 3 is a flowchart illustrating an image dehazing method according to an example embodiment of the present invention. In step S301, the image dehazing device will receive an input image by a sensor. In step S302, the image dehazing device will dehaze the image by a dehazing module to output a dehazed RGB image. In step S303, the image dehazing device will recover image brightness of the dehazed RGB image by an HDR module to output an HDR image. In step S304, the image dehazing device will remove reflections of the HDR image by a ReflectNet inference model, wherein the ReflectNet inference model uses a deep learning architecture.

[0079] Figure 4 is a block diagram illustrating an image dehazing device according to an example embodiment of the present invention. The image dehazing device 40 will include, but not limited to, a sensor 401 and a processor 402 coupled to the sensor 401. The processor 402 will be configured to at least: receive an input image; dehaze the image by a dehazing module to output a dehazed RGB image; recover image brightness of the dehazed RGB image by an HDR module to output an HDR image; and remove reflections of the HDR image by a ReflectNet inference model, wherein the ReflectNet inference model uses a deep learning architecture.

[0080] Figure 5is a flowchart illustrating steps of an image dehazing method according to an exemplary embodiment of the present application. In step S501, the image dehazing device receives an input RGB image through a sensor. In step S502, in an embodiment of the present application, the image dehazing device dehazes the RGB image and outputs a dehazed RGB image through an image dehazing module. More specifically, the image dehazing device can detect and recover the attenuation effect due to the presence of fog / rain or any other intervening matter between the camera sensor and the object. In one embodiment, the image dehazing module can be designed using conventional computer vision to detect the amount of fog based on depth perception and then linearly compensate or recover the affected pixels while preserving the depth sensing in the 2D RGB image. Specifically, the image dehazing device can recover the pixels from the fog or rain, thereby truly increasing the image brightness. In step S503, in an embodiment of the present application, the image dehazing device recovers the image brightness of the dehazed RGB image through an HDR module to output an HDR image. More specifically, the HDR module can be designed to improve the dynamic range of the corresponding RGB image. In one embodiment, the image dehazing device can compensate for the image darkening that can be caused by the image dehazing module or naturally from a low-light environment. Specifically, the image dehazing device can produce a brightness enhancement that finds a balance between the low-light pixels and any bright-light pixels. In step S504, the image dehazing device removes the reflections of the HDR image. In an exemplary embodiment, the reflection removal can employ a machine learning method with built-in functionality of image segmentation to separate the reflection map from the input image to obtain a reflection-free output image. Specifically, in one embodiment, the reflection removal can be implemented by a ReflectNet inference model that can use a deep learning architecture such as an encoder-decoder network.

[0081] Figure 6 are implementation examples of an input RGB image and an output RGB image processed by the image dehazing method according to an exemplary embodiment of the present application. Referring to Figure 6 , image 601 is an example of an input RGB image that can be received by a sensor as illustrated in step S501 of Figure 5 , and image 602 is an example of an output RGB image that can be produced in step S505 of Figure 5 after image 601 has undergone the processing of steps S502 to S504 as illustrated in Figure 5 .

[0082] Figure 7is a detailed flowchart illustrating steps of an image defogging method according to an exemplary embodiment of the present application. In step S701, the image defogging device will receive an input RGB image through a sensor. The input RGB image can be denoted as I(). In an exemplary embodiment, in step S702, the input RGB image I() is first processed by an image defogging module to remove the adverse effects from fog, rain, or any other medium that will attenuate the amount of light received by the camera sensor. Specifically, in one embodiment, the image defogging module in step S702 can be implemented by four sub-modules: (a) minimum channel estimation; (b) guided filtering; (c) atmospheric channel detection; (d) linear color reconstruction. After the input RGB image I() is processed by the image defogging module, an output defogged image denoted as I D () can be produced.

[0083] In step S703, in an exemplary embodiment, an HDR module can recover the decimation of image brightness that can be attributed to the image defogging module that darkens the image and / or exposes the image from light or low light environment. More specifically, the HDR module includes two sub-modules: a Nakagami-Rashid compensator and a local histogram equalization (LHE) operation. The Nakagami-Rashid compensator is a compensation formula to recover the brightness of any underexposed or overexposed image. This compensation can be obtained from the statistics of the defogged image I D () computed during initialization. The LHE operation is a local histogram equalization technique that can enhance the image contrast of each color channel (r, g, b) for any contrast loss that can be induced by the Nakagami-Rashid compensator. After the defogged image I D () is processed by the HDR module, an HDR image denoted as I H () can be produced.

[0084] In step S704, in an exemplary embodiment, a reflectance removal module can use a machine learning method, namely a ReflectNet inference model that can be based on a deep learning architecture. In one embodiment, the ReflectNet inference model can be implemented by an encoder-decoder network with a pre-trained model from a segmentation architecture. The encoder side and the pre-trained model (from a segmentation architecture) will create hypercolumn features that are decoded by the decoder side for separating the reflectance map with respect to the output RGB image denoted as O()

[0085] Figure 8 is a detailed flowchart illustrating the algorithm of the defogging module according to an exemplary embodiment of the present application. The defogging module can be formulated as follows:

[0086] I(x) = I D(x) T(x) + A(l - T(x)), (1)

[0087] where I() is the input RGB image, I D () is the output dehazed image, T() is the dehazing transmission map, A is the atmospheric coefficient, and x is the pixel position.

[0088] At step S801, in an exemplary embodiment, the minimum channel estimation in the haze / rain detection is designed based on the fog / rain or any intervening medium represented in equation (1). In the absence of fog or haze image, the color saturation of most of the pixels will reflect the actual color of the object, while the presence of fog will introduce an attenuation effect that will increase the image brightness and reduce the color saturation. In the case of a haze-free image, at least one color channel of the input RGB image I() will fall in the low end spectrum of intensity. In other words, by observing the color channel with the minimum pixel value will reflect the presence of the fog medium as illustrated in equation (1).

[0089] Specifically, in one embodiment, the minimum channel estimation implements a block-wise operation that partitions the input RGB image I() into n(Ω) number of block segments, where Ω(x) is the block segment with the center coordinate at x to estimate the minimum channel pixel value of the corresponding Ω(x) as represented in the following equation:

[0090]

[0091] where r, g, b are the color channels in the input RGB image I(). The dimension of Ω(x) is a user constraint that can be fine-tuned to achieve the desired dehazing performance. Higher dimension of Ω(x) will constitute a lower resolution of the minimum channel map while lower dimension of Ω(x) can cause overestimation of the minimum channel.

[0092] At step S802, in an exemplary embodiment, the guided filtering is implemented as taught by A. Levin, D. Lischinski, and Y. Weiss (A closed-form solution to natural image matting, Computer Vision and Pattern Recognition, Vol. 1, pp. 61-68, 2006), i.e., by refining the shape of the inverted minimum channel map into the haze transmission map T() based on the color and shape information in the input RGB image I(). Specifically, in one embodiment, the guided filtering can be represented as follows:

[0093]

[0094] where E is an identity matrix of size I() and λ is a parameter of the soft constraint. The value of λ will determine the degree of influence of I() on the control of the shape refinement in (3). Note that in (3), the minimum channel map is inverted, because T() will have to reflect the transmittance of the haze. This means that low values in T() indicate nearby objects or low haze inducing effects, while low values in I

[0095] More specifically, in one embodiment, L in (3) is the Laplacian matrix of the following formula:

[0096]

[0097] where I i , I j are the colors of I() at i-pixel, j-pixel; p ij is the Kronecker delta; w k is a window kernel of size n(); m k , s k are the mean and covariance in w k ; e is a normalization parameter; and E3 is an identity matrix with size 3x3 (similar to E in (3)).

[0098] At step S803, in an exemplary embodiment, the atmospheric coefficient can be estimated using the atmospheric channel detection. The atmospheric channel detection module gets the intensity value of I() from the lowest pixel in T(), as represented by the following formula:

[0099]

[0100] It should be noted that the atmospheric channel detection module does not estimate the atmospheric coefficient based on the brightest pixel in the image, because the atmospheric channel detection module can not be a correct representation of the airlight intensity. T utilizes the assumption that a larger transmittance value is a far object / environment (in other words, airlight intensity) to ensure the position of the pixel that reflects the airlight intensity.

[0101] At step S804, in an exemplary embodiment, after the haze transmittance map T() and the atmospheric coefficient A can be determined, linear color reconstruction can recover the color channels in I() for any haze effect using the haze transmittance map T() and the atmospheric coefficient A. The operation is a pixel-wise computation for each color channel, as represented by the following formula:

[0102]

[0103] Figure 9This is an implementation example of the dehazing module's algorithm processing the input RGB image and outputting the dehazed image. Figure 9 , linear color reconstruction can remove the haze-induced image 901 / suppress it into a haze-free image 902. Although the haze-free image 902 introduces better visibility, since the essence of image dehazing is to reduce the brightness of the image caused by haze, the overall image brightness is reduced.

[0104] Figure 10 1 is an implementation example of an input RGB image 1001 and a minimum channel map 1002 processed by the minimum channel estimation algorithm in the dehazing module. The minimum channel estimation algorithm calculates the minimum channel map 1002 of the input RGB image 1001 based on a block operation. In the presence of haze, the pixel value of the object will be affected by its position relative to the camera accordingly. Figure 10 In

[15] , the minimum channel map 1002 constitutes a depth-like map that indirectly perceives the hazy medium as a depth perception. Since the minimum channel estimation uses the channel with the minimum pixel value, the presence of bright or highly saturated objects will affect the depth-like effect in the minimum channel map 1002, such as Figure 10 The traffic lights are the same as those shown in the figure.

[0105] Figure 11 This is an implementation example of the inverted minimum channel map 1101 and the haze transmission map 1102 processed by the guided filtering algorithm in the defogging module. Figure 11 As shown in FIG, the guided filtering algorithm smoothly inverts the rough shape of the minimum channel map 1101 and uses Laplace transform to correct the erroneous estimate of the minimum channel based on the color and shape information provided in FIG. The outline of the object can be perceived at a higher resolution in the haze transmission map 1102.

[0106] Figure 12 1202 is an implementation example of an atmospheric detection algorithm in a defogging module, wherein the atmospheric detection algorithm obtains an intensity value 1203 from a haze transmission map 1201 of a defogging image 1202. Figure 12 , the atmosphere detection algorithm can estimate an intensity value 1203 representing the atmospheric coefficient A. The intensity value 1203 is obtained from the lowest pixel in the haze transmission map 1201. The use of the haze transmission map 1201 ensures the location of pixels that reflect air light intensity by assuming the largest transmittance value to be the farthest object / environment (in other words, air light intensity).

[0107] Figure 13 is an implementation example of a linear color reconstruction algorithm in a defogging module, which recovers a defogging RGB image 1302 from an input image 1301. Figure 13As illustrated in the middle, the linear color reconstruction algorithm can remove or suppress the haze-induced image in the input image 1301 and restore the haze-free image version in the dehazed RGB image 1302. Although the haze-free image 1302 introduces better visibility, the overall image brightness is reduced due to the nature of image dehazing which is to reduce the brightness of the input image 1301 induced by haze.

[0108] Figure 14 is a detailed flowchart illustrating the algorithm of the HDR module according to an exemplary embodiment of the application. At step S1401, in an exemplary embodiment, the Naka-Rashid compensator is designed using pixel-wise segmentation and pixel transformation.

[0109] Figure 15 are implementation examples of the input dehazed image 1501 to the HDR module and the output HDR image 1502 processed by the HDR module according to an exemplary embodiment of the application. As Figure 15 As illustrated in the middle, the HDR module restores the image brightness in the output HDR image 1502 which can be located in the input dehazed image 1501 due to the dehazing module which darkens the image or exposes from a light or low light environment.

[0110] Figure 16 is a flowchart illustrating that the dehazed RGB image is enhanced in image brightness by the Naka-Rashid compensator according to an exemplary embodiment of the application. Specifically, Figure 16 illustrates the details of the Naka-Rashid compensator. The Naka-Rashid compensator module starts with color correction which transforms the color channels of the original RGB image 1601 into a luminance map 1602, denoted as I D，y . The original RGB image 1601 I D is decomposed into r, g, b channels (1603 to 1605), i.e., I D，r , I D，b , I D，g .

[0111] At step S1611, these maps 1603 to 1605 are used to calculate the threshold of pixel clusters to segment the bright pixels 1606 from the dark pixels 1607 with respect to I D .

[0112]

[0113] where μ D，c represents the average image pixel at c th channel. There are two coefficients, i.e., a and b, in (7) which nonlinearly and linearly scale the brightness in I D，y . A bilinear filter kernel, denoted as B, is used to eliminate the abrupt changes from ID，y noise.

[0114] At step 1612, according to the pixel-based segmentation, a threshold value ξ can be used to segment the original RGB image 1601 I D into different color channels 1603 to channel 1605. D，c is segmented based on ξ, and the pixels are transformed using the following formula:

[0115]

[0116] wherein and are compensation factors calculated as follows:

[0117]

[0118] and

[0119]

[0120] Referring to Figure 14 , at step S1402, in one embodiment, the LHE operation is a local histogram equalization technique that enhances the image contrast of each color channel (r, g, b) against any contrast loss that can be induced by the Heitz-Rashid compensator.

[0121] Figure 17 is an implementation example to enhance the image contrast of the defogged RGB image I D () 1701 and the output HDR image I H () 1702 by the local histogram operation. As Figure 17 illustrated in FIG. 17, the contrast in the HDR image I H () 1702 can be enhanced by the local histogram operation. The contrast can be reduced due to the equalization of the luminance but can lose the contrast in the defogged RGB image I D () 1701 by the Heitz-Rashid compensator.

[0122] Figure 18 is a flowchart illustrating the removal of reflections of an HDR image by a ReflectNet inference model according to an exemplary embodiment of the present application. At step S1801, in one embodiment, the ReflectNet inference model can be designed using a combination of normal convolution and dilated convolution that is commonly used for segmentation networks. Figure 19 illustrates an implementation example of an input defogged image 1901 and an output HDR image 1902 processed by the ReflectNet inference model. As Figure 19As illustrated in the middle, the ReflectNet inference model suppresses any reflection-induced artifacts in the output HDR image 1902 from the input dehazed image 1901 that can be due to the enhancement of slight reflections by the HDR enhancement or from the reflection conditions themselves.

[0123] Figure 20 is the architecture of the ReflectNet inference model in one exemplary embodiment of the present invention. The ReflectNet inference model can use an encoder-decoder network with a pre-trained model from a segmentation architecture. The encoder side and the pre-trained model (from a segmentation architecture) will create hyper-column features that will be decoded by the decoder side for separating the reflection map from the output RGB image. The encoder-decoder architecture is trained for learning features relevant to reflection removal; while the pre-trained model provides learned features to perform spatial segmentation.

[0124] Reference Figure 21 In one embodiment, the ReflectNet inference model can be trained using a pair of original RGB image and its reflection image. In this embodiment, the dataset is produced using the ImageMix function that blends the two images into a blended image. The original RGB image will be used as ground truth, while the blended image as input.

[0125] In an exemplary embodiment, GT R represents the original RGB image, and I() represents the blended image or any reflection-induced RGB image; O() and O R () are the output reflection-removed RGB image and the estimated reflection map, respectively. The loss function is computed as:

[0126] L(θ) = wlL f (θ) + w2L a (θ) + w3L g (θ), (11)

[0127] where w are the weights to balance the losses, and θ are the network weights. There are three loss functions in (11), including the feature loss, the adversarial loss, and the gradient loss. The feature loss is computed as follows:

[0128]

[0129] This loss compares the selected features at the l-layer difference when the model runs ground truth GT R with the result from the input RGB image I(). The adversarial loss measures the negative log probability of O() to have the natural ground truth as follows:

[0130]

[0131] Finally, the gradient loss measures the contrast comparison between the ground truth and the resulting image, as expressed in the following equation:

[0132]

[0133] The hyperbolic tangent in this loss function is used to minimize the edge bias between O() and O R The edges in I() should come from either O() or O R (), not both.

[0134] Figure 22 is a table comparing the implementation of the image dehazing method with the benchmark methods in the prior art. With reference to Figure 22 , the performance evaluation consists of two methods: (a) the proposed method and (b) the benchmark method. Both methods are image pre-processing algorithms using a similar framework. Based on the flowchart in Figure 5 , the proposed method contains: image dehazing, HDR enhancement, and reflection removal. To ensure fairness of the evaluation, the benchmark method is made to comply with the general framework, where each function is constructed from the prior state of the art. For example, image dehazing uses color attenuation to represent, as taught by D. Ngo et al., “Improved Color Attenuation Prior for Single-Image Haze Removal,” Applied Sciences, [Online], September 2019; HDR enhancement is represented by histogram equalization, as taught by OpenCV, “Histogram Equalization,” Open Source Computer Vision, [Online], July 2020; reflection removal is represented by CEILNet, as taught by Q. Fan et al., “Generic Architecture for Single Image Reflection Removal and Image Smoothing,” ICCV, 2017. Both the proposed method and the benchmark method are implemented using the same hardware and software as illustrated in Figure 22 .

[0135] Figure 23is a table comparing the performance evaluation of embodiments of the image dehazing method with benchmark methods in the prior art. In exemplary embodiments, various metrics including Entropy-based Measurement Enhancement (EME) 2301, Contrast Improvement Index (CII) 2302, Intensity Measure (IM) 2303 can be implemented to perform quantitative evaluation of the perceived quality between benchmark methods in the prior art and the proposed image dehazing method.

[0136] Entropy-based Measurement Enhancement (EME) 2301 is a metric for measuring the improvement index of the output compared to the input based on entropy. A larger fraction of EME indicates a higher index of entropy information. The maximum value of EME 2301 is ∞. In one embodiment, EME 2301 is calculated as:

[0137]

[0138] where N p is the total number of pixels in the compared images, N b is the size of the block size, and I y and O y are the reference and input images.

[0139] Contrast Improvement Index (CII) 2302 is a metric for measuring the improvement index of the output compared to the input based on contrast. A larger fraction of CII 2302 indicates a higher index of contrast improvement. The maximum value of CII 2302 is ∞. In one embodiment, CII 2302 is calculated as:

[0140]

[0141] where N p is the total number of pixels in the compared images, N b is the size of the block size, and I y and O y are the reference and input images.

[0142] Intensity Measure (IM) 2303 is a metric for measuring the ratio of luminance robustness based on the original image. A larger fraction of IM 2303 indicates a better robustness of image luminance. The maximum value of IM 2303 is 1. In one embodiment, IM 2303 is defined as:

[0143]

[0144] where N pis the total pixels in the compared image, and I y and O y are the reference image and the input image.

[0145] In one embodiment, all three stages are evaluated sequentially based on a common framework. According to Figure 23 the evaluation in Table 1, the proposed method outperforms the baseline method in all the metrics in all stages. This method contains image defogging, HDR sub-module, and reflection removal functionalities. Based on the final result score (see reflection removal results), the proposed method achieves CII = 2.695 and EME = 2.336 or +39.9% and +17.5% higher than the baseline method. These results show that the proposed method can enhance the contrast and entropy information in the RGB image.

[0146] In one embodiment, in terms of intensity measurement ratio, the proposed method is able to preserve an average of 74.8% of the luminance; while the baseline method is able to preserve an average of 55.2% of the luminance. This result can demonstrate that the proposed method preserves the luminance as the original image would provide; while removing the ability of haze, reflections, and contrast ratio. In addition to the perceptual index, the proposed method and the baseline method are evaluated according to the time complexity. Figure 23 It is shown that the time complexity of the proposed method is lower than the baseline method in all functional categories. This demonstrates the efficiency of the proposed method over the state of the art.

[0147] Figure 24 An example of an input original image and an output image processed by the image defogging method according to exemplary embodiments of the present application is shown compared to an output image processed by the baseline method in the state of the art. Referring to Figure 24 qualitatively, the baseline method is not able to remove the haze effect stably, as the baseline method introduces several over-smoothing artifacts and over-darkening effects that reduce the image visibility. On the other hand, the proposed method is able to remove the haze with a slight luminance loss and proper color reproduction compared to the original RGB.

[0148] Figure 25 is a table showing a performance evaluation of object detection on the RGB image processed by the image defogging method according to exemplary embodiments of the present application compared to a performance evaluation of object detection on the original RGB image and the RGB image processed by the baseline method in the state of the art. The evaluation uses IoU-based computation to fine-tune the percentage of overlap between the prediction and the ground truth to:

[0149]

[0150] where B D and B R are the bounding boxes and ground truth data, respectively.

[0151] Reference Figure 25 In one embodiment, using the IoU information, true positive (TP), false positive (FP), and false negative (FN) are measured to obtain the accuracy (ACC) 2501 and false positive rate (FPR) 2502, as follows:

[0152] and

[0153] In one embodiment, according to the experiment in Figure 25 , the resulting images from the proposed method achieve better accuracy and false positive rate. The proposed method is able to increase the accuracy by 8.2% and decrease the false positive rate by 10.9%, while the baseline method increases the accuracy by 1.9% and increases the false positive rate by 4.2%. This phenomenon shows that the baseline method might over-enhance its images so much that it produces unwanted false appearance to the YOLOv3 to infer as objects. The visualization of the object detection results is illustrated in Figure 26

[0154] It will be apparent to those skilled in the art that various modifications and variations can be made to the disclosed embodiments without departing from the scope or spirit of the application. In light of the foregoing disclosure, it is therefore intended that the application encompass all such modifications and variations as fall within the scope of the appended claims and their equivalents.​

Claims

1. An image defogging method, characterized by, Comprising: receiving an input image; dehazing the image by a dehazing module to output a dehazed red green blue image; restoring image luminance of the dehazed red green blue image by a high dynamic range module to output a high dynamic range image; removing reflections of the high dynamic range image by a ReflectNet inference model, wherein the ReflectNet inference model uses a deep learning architecture; wherein the step of dehazing the image by a dehazing module to output a dehazed red green blue image comprises: estimating an inverse minimum channel map by a minimum channel estimation algorithm; refining the inverse minimum channel map into a haze transmission map by a guided filtering algorithm; obtaining intensity values of the haze transmission map by an atmospheric detection algorithm; and restoring a dehazed red green blue image by a linear color reconstruction algorithm.

2. The image defogging method of claim 1, wherein, wherein, the step of estimating an inverse minimum channel map by a minimum channel estimation algorithm comprises: segmenting the input image into a plurality of patch segments, wherein each of the patch segments comprises a center pixel; and obtaining pixels of patch segments having lowest intensity values for all color channels.

3. The image defogging method of claim 1, wherein, wherein, the step of refining the inverse minimum channel map into a haze transmission map by a guided filtering algorithm comprises: smoothing the inverse minimum channel map using a soft constraint parameter to control shape refinement; and correcting the inverse minimum channel map using a Laplacian matrix.

4. The image defogging method of claim 1, wherein, wherein, the step of obtaining intensity values of the haze transmission map by an atmospheric detection algorithm comprises: obtaining pixels having lowest intensity values in the haze transmission map.

5. The image defogging method of claim 1, wherein, wherein, the step of restoring a dehazed red green blue image by a linear color reconstruction algorithm comprises: performing a pixel-wise operation using the haze transmission map and an atmospheric coefficient to calculate intensity values of the dehazed red green blue image, wherein the atmospheric coefficient is obtained by the atmospheric detection algorithm.

6. The image defogging method of claim 1, wherein, wherein, the step of restoring image luminance of the dehazed red green blue image by a high dynamic range module to output a high dynamic range image comprises: enhancing image luminance of the dehazed red green blue image by a Nakagami-Laguerre compensator; and enhancing image contrast of the dehazed red green blue image by a local histogram operation.

7. The image defogging method of claim 6, wherein, wherein, the step of enhancing image luminance of the dehazed red green blue image by a Nakagami-Laguerre compensator comprises: performing color correction that transforms color channels of the dehazed red green blue image into a luminance map; calculating a threshold value for segmenting bright and dark pixels in the dehazed red green blue image using the luminance map; and transforming pixels of the dehazed red green blue image using a compensation coefficient calculated from the luminance map.

8. The image defogging method of claim 6, wherein, wherein, the step of enhancing image contrast of the dehazed red green blue image by a local histogram operation comprises: performing a histogram equalization of intensity for the dehazed red green blue image.

9. The image defogging method of claim 1, wherein, wherein, the ReflectNet inference model comprises an encoder-decoder network with features of a pre-trained image segmentation model to separate a reflection map from the image.

10. The image defogging method of claim 1, wherein, wherein, the step of removing reflections of the high dynamic range image by a ReflectNet inference model comprises: encoding features from the high dynamic range image; and decoding the features to output a red-green-blue image.

11. The image defogging method of claim 1, wherein, wherein, the ReflectNet inference model is trained using a loss function comprising: feature loss; adversarial loss; and gradient loss.

12. An image defogging device, comprising: comprising: a sensor; and a processor coupled to the sensor and configured to at least: receive an input image; dehaze the image through a dehaze module to output a dehazed red-green-blue image; restore image luminance of the dehazed red-green-blue image through a high dynamic range module to output a high dynamic range image; remove reflections of the high dynamic range image through a ReflectNet inference model, wherein the ReflectNet inference model uses a deep learning architecture; wherein the processor is configured to dehaze the image through a dehaze module to output a dehazed red-green-blue image, the processor is further configured to: estimate an inverted minimum channel map through a minimum channel estimation algorithm; refine the inverted minimum channel map into a haze transmission map through a guided filtering algorithm; obtain intensity values of the haze transmission map through an atmospheric detection algorithm; and restore a dehazed red-green-blue image through a linear color reconstruction algorithm.

13. The image defogging device according to claim 12, characterized in that, wherein, the processor is configured to estimate an inverted minimum channel map through a minimum channel estimation algorithm, the processor is further configured to: segment the input image into a plurality of patch segments, wherein each of the patch segments comprises a center pixel; and obtain pixels of patch segments having lowest intensity values for all color channels.

14. The image defogging device according to claim 12, characterized in that, wherein, the processor is configured to refine the inverted minimum channel map into a haze transmission map through a guided filtering algorithm, the processor is further configured to: smooth the inverted minimum channel map using a soft constraint parameter to control shape refinement; and correct the inverted minimum channel map using a Laplacian matrix.

15. The image defogging device according to claim 12, characterized in that, wherein, the processor is configured to obtain intensity values of the haze transmission map through an atmospheric detection algorithm, the processor is further configured to: obtain pixels having lowest intensity values in the haze transmission map.

16. The image defogging device according to claim 12, characterized in that, wherein, the processor is configured to restore a dehazed red-green-blue image through a linear color reconstruction algorithm, the processor is further configured to: perform a pixel-wise operation using the haze transmission map and an atmospheric coefficient to calculate intensity values of the dehazed red-green-blue image, wherein the atmospheric coefficient is obtained by the atmospheric detection algorithm.

17. The image defogging device according to claim 12, characterized in that, wherein, the processor is configured to restore image luminance of the dehazed red-green-blue image through a high dynamic range module to output a high dynamic range image, the processor is further configured to: enhance image luminance of the dehazed red-green-blue image through a Nakagami-Lashkar compensator; and enhance image contrast of the dehazed red-green-blue image through a local histogram operation.

18. The image defogging device according to claim 17, characterized in that, wherein, the processor is configured to enhance image luminance of the dehazed red-green-blue image through a Nakagami-Lashkar compensator, the processor is further configured to: performing color correction that translates color channels of the dehazed red green blue image into a luminance map; calculating a threshold for segmenting bright and dark pixels in the dehazed red green blue image using the luminance map; and transforming pixels of the dehazed red green blue image using a compensation factor calculated from the luminance map.

19. The image defogging device according to claim 17, characterized in that, wherein, the processor is configured to enhance image contrast of the dehazed red green blue image by a local histogram operation, the processor being further configured to: perform a histogram equalization of intensity for the dehazed red green blue image.

20. The image defogging device of claim 12, wherein, wherein, the ReflectNet inference model is further configured as an encoder-decoder network with features of a pre-trained image segmentation model to separate a reflectance map from the image.

21. The image defogging device according to claim 12, characterized in that, wherein, the processor is configured to remove reflectance of the high dynamic range image by a ReflectNet inference model, the processor being further configured to: encode features from the high dynamic range image; and decode the features to output a red green blue image.

22. The image defogging device according to claim 12, characterized in that, wherein, the ReflectNet inference model is trained using a loss function that includes: a feature loss; an adversarial loss; and a gradient loss.

Images