Defogging method and device based on infrared light and visible light image fusion

Abstract

The invention belongs to the technical field of traffic safety monitoring technologies, and discloses a defogging method and equipment based on infrared light and visible light image fusion. The method comprises the following steps: image preprocessing: denoising and enhancing a foggy image; inputting the preprocessed picture into a defogging model based on infrared light and visible light image fusion to obtain a defogged image; the defogging model based on infrared light and visible light image fusion carries out the following processing on an input picture: processing an infrared light image, enhancing the penetration effect of the infrared light image in a fog environment, and highlighting the contour information of a target; and processing the visible light image, and recovering the color and texture details of the visible light image in the fog environment. According to the method, the definition and the visual effect of the foggy day image can be remarkably improved, the target is prominent, the color is natural, the contrast ratio is effectively improved, the detail information of the image is reserved, rapid defogging processing is realized, and the accuracy of fog concentration estimation is enhanced.

Description

Technical Field

[0001] The present invention belongs to the technical field of traffic safety monitoring, and in particular relates to a defogging method and device based on the fusion of infrared light and visible light images. Background Art

[0002] Traditional single-sensor defogging methods, such as those based on deep learning or prior information, often perform poorly in dense fog. This is because these methods often rely on scene depth information, which is often unavailable in dense fog. Therefore, researchers have begun exploring methods for defogging by fusing visible and near-infrared images. This fusion method combines the detailed information of visible light images with the radiometric information of infrared images, resulting in clearer images in foggy conditions.

[0003] Current dehazing algorithms based on near-infrared and visible light images have several drawbacks, primarily color distortion and information loss. These issues may be due to the ineffective handling of the differences in the characteristics of the two types of images during the fusion process, resulting in deviations in color and brightness in the fused image. Summary of the Invention

[0004] The purpose of the present invention is to address the shortcomings of the existing technology and provide a defogging method and device based on the fusion of infrared light and visible light images, which can be used to remove fog from images in real time, enhance the accuracy of fog concentration estimation, and ensure the defogging effect in complex environments.

[0005] Specifically, the present invention is implemented by adopting the following technical solutions.

[0006] In one aspect, the present invention provides a defogging method based on infrared and visible light image fusion, comprising:

[0007] Image preprocessing: denoising and enhancing foggy images;

[0008] The preprocessed image is input into a defogging model based on the fusion of infrared and visible light images to obtain a defogged image. The defogging model based on the fusion of infrared and visible light images performs the following processing on the input image:

[0009] Process infrared images to enhance their penetration in foggy environments and highlight the target's contour information;

[0010] The visible light image is processed to restore its color and texture details in the foggy environment.

[0011] Furthermore, the infrared image is processed to enhance its penetration effect in a foggy environment and highlight the contour information of the target, including:

[0012] A near-infrared correction lens is used to eliminate the focal plane offset between infrared and visible light, and a narrow-band filter is used to allow only the near-infrared band to pass, suppressing visible light interference.

[0013] Furthermore, the processing of the visible light image to restore its color and texture details in the foggy environment includes:

[0014] Color restoration algorithms and image restoration techniques are used to enhance texture details in visible light images. High-dimensional convolutional neural networks are used to extract image color and texture features. The extraction of image color and texture features using high-dimensional convolutional neural networks includes:

[0015] First, the visible light image is input into the multi-layer CNN network. The first convolutional layer captures the texture details and color features in the image through convolution kernels of different sizes; the middle layer reduces the feature dimension through pooling operations while retaining key information, and eliminates the brightness deviation caused by haze through normalization processing; the deep network establishes a mapping relationship between haze concentration and feature damage through fully connected layers, and outputs adaptive dehazing parameters.

[0016] Furthermore, before processing the pre-processed image, the defogging model based on infrared and visible light image fusion further includes calculating the atmospheric light intensity and transmittance and estimating the effect of fog on the image, including the following steps:

[0017] (1) Dark channel acquisition

[0018] For the input image I, calculate its dark channel value , see the following formula:

[0019]

[0020] Where,

[0021] Stands for "dark channel", which is a function used to describe the dark features of local areas in an image, indicating the local area where each pixel x of the color image is located Internally, RGB three channels Corresponding to the minimum value in the red, green and blue channels, Represents a window centered on pixel x; y represents the local area where pixel x is located Other pixels within

[0022] It is the pixel value of the image at pixel y and channel c, where r, g, and b correspond to the pixel brightness of the red, green, and blue channels respectively;

[0023] (2) Calculation of atmospheric light intensity

[0024] The average value of the pixels in the sky area of ​​the image is used as the atmospheric light intensity A, and the proportion of the pixels in the sky area to the pixels in the entire image is determined. If the proportion is less than a threshold, the average value of the first K pixels with the largest brightness values ​​in the dark channel image (for example, the first 0.1% of pixels in terms of brightness) is used as the atmospheric light intensity, where K is an integer.

[0025] (3) Obtaining image transmittance

[0026] The transmittance of the defogging image in the bright area is corrected, and the corrected fog concentration estimate is as follows:

[0027]

[0028] in, Saturation The correction value is calculated as follows: is the normalized dark channel value, and the calculation formula is as follows; is the saturation weight index of pixel x;

[0029]

[0030] in, The minimum value of pixel x in the RGB three channels in the input image I With the maximum value The ratio of

[0031]

[0032] Where A is the atmospheric light intensity.

[0033] Furthermore, the defogging method based on infrared and visible light image fusion further includes performing image optimization processing on the fused image; the image optimization processing on the fused image includes:

[0034] The fused image is input into a bilateral filter or a Gaussian filter to obtain a denoised and smoothed image; the original histogram is stretched to a uniform distribution by statistically analyzing the grayscale distribution of image pixels; the image is divided into blocks and the mean and variance of each block are calculated, the brightness gain of overly bright areas is reduced, and the brightness offset of overly dark areas is increased. At the same time, combined with multi-scale Gaussian pyramid decomposition, local brightness is adjusted at different levels.

[0035] Furthermore, the image optimization processing on the fused image further includes:

[0036] The dehazed image is used as input, and the second-order derivative of the image is calculated using the Laplacian operator to obtain a Laplacian image containing edge and detail information; the Laplacian image is superimposed on the original image by controlling the sharpening strength coefficient k, where k is 0.5-1.5.

[0037] Furthermore, the defogging method based on infrared and visible light image fusion also includes using a multi-scale pyramid filtering method to enhance details at different scales. The specific steps are as follows:

[0038] First, a multi-scale pyramid of the image is constructed. By performing multiple Gaussian filtering and downsampling on the original image, pyramid-level images of different resolution levels are generated. The large-scale levels highlight the overall contour, while the small-scale levels retain the fine texture. Then, for each pyramid level, an adaptive filter kernel is designed to perform filtering enhancement on the images at each level to highlight the detailed features at that scale. After that, the enhanced images at each level are restored to the original image size at the corresponding scale through upsampling and fusion operations, and then weights are assigned according to the importance of details at each scale for fusion.

[0039] Furthermore, the image optimization processing of the fused image also includes adjusting the white balance, which specifically includes:

[0040] First, the automatic white balance algorithm analyzes the color deviation of the image and determines the color offset caused by ambient light and other factors. Then, the pixel values ​​of the red, green, and blue channels are compensated to restore the white or gray areas to their natural colors and eliminate color casts.

[0041] After completing the white balance adjustment, the saturation channel of the image is extracted based on the HSV color space. The color vividness is enhanced through linear or nonlinear transformation. At the same time, the saturation adjustment of the highlight and shadow areas is constrained in combination with the visual characteristics of the human eye.

[0042] On the other hand, the present invention provides a computer-readable storage medium having a computer program stored thereon, characterized in that when the computer program is executed by a processor, the steps of the above-mentioned defogging method based on the fusion of infrared light and visible light images are implemented.

[0043] On the other hand, the present invention provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the above-mentioned defogging method based on the fusion of infrared light and visible light images.

[0044] The beneficial effects of the defogging method and device based on infrared and visible light image fusion of the present invention are as follows:

[0045] The defogging method and device based on the fusion of infrared and visible light images of the present invention utilize the difference between infrared and visible light in penetrating fog, and by fusing the two spectral information, improve the clarity and contrast of the image, thereby achieving a defogging effect. Infrared light has a strong ability to penetrate fog and can obtain clear contour information of foggy scenes. Visible light images have low contrast in foggy environments, but contain rich color and texture information. By fusing infrared and visible light images, the strengths of the two are complemented to achieve a defogging effect, which can solve the problems of low clarity of foggy images, color distortion, loss of details, registration errors, high computational complexity, inaccurate fog concentration estimation, poor defogging effect in low light conditions at night, and equipment cost and complexity in the prior art.

[0046] Preprocessing of infrared and visible light images, including denoising and enhancement, improves subsequent fusion effects;

[0047] By calculating the air light intensity and transmittance and estimating the effect of fog on the image, the foggy and non-fog areas in the image can be quickly distinguished.

[0048] By processing infrared light images, its penetration effect in foggy environments is enhanced, and the contour information of the target is highlighted;

[0049] By processing visible light images, the color and texture details in foggy environments are restored;

[0050] By performing color balancing and contrast enhancement on the fused image, the dehazed image is made more natural and realistic.

[0051] A multi-scale feature extraction and weight distribution method is adopted to intelligently allocate fusion weights according to fog concentration and image features, effectively preserving image details and improving contrast. By optimizing the algorithm structure, the computational complexity is reduced, and fast defogging processing is achieved to meet the needs of real-time monitoring. By combining infrared and visible light information, the fog concentration is estimated more accurately, improving the stability of the defogging effect. At night or in low light conditions, the night vision capability of infrared light is utilized, combined with limited lighting information, to achieve an all-weather defogging effect.

[0052] The defogging method and device based on the fusion of infrared light and visible light images of the present invention not only significantly improves the clarity and visual effect of foggy images, making the target more prominent, the color more natural, and the contrast effectively improved, but also retains the detailed information of the image, reduces the registration error, improves the consistency of the fused image, realizes rapid defogging processing, enhances the accuracy of fog concentration estimation, ensures the defogging effect in complex environments, and at the same time reduces the user's cost investment, improves the practicality and market competitiveness of the system, provides more reliable technical support for security monitoring, traffic management, remote sensing detection and other fields, can realize real-time defogging of images, always ensure road traffic safety, and has good development and application prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 4 is a schematic diagram of the image enhancement effect of a visible light image according to an embodiment of the present invention.

[0054] Figure 2 3 is a schematic diagram of infrared image median filtering denoising according to an embodiment of the present invention.

[0055] Figure 3 3 is a schematic diagram of Gaussian filtering and denoising of infrared images according to an embodiment of the present invention.

[0056] Figure 4 2 is a schematic diagram of multi-scale filtering according to an embodiment of the present invention.

[0057] Figure 5 It is a schematic diagram of geometric transformation of images in the training set according to an embodiment of the present invention.

[0058] Figure 6 3 is a schematic diagram of histogram equalization and local brightness adjustment according to an embodiment of the present invention.

[0059] Figure 7 2 is a schematic diagram of image sharpening according to an embodiment of the present invention.

[0060] Figure 8 4 is a schematic diagram of multi-scale pyramid filtering according to an embodiment of the present invention.

[0061] Figure 9 Schematic diagram of the defogging effect under low light conditions according to an embodiment of the present invention.

[0062] Figure 10 This is a schematic diagram of the defogging effect in a complex environment according to an embodiment of the present invention.

[0063] Figure 11 Schematic diagram of indoor defogging effect according to an embodiment of the present invention. DETAILED DESCRIPTION

[0064] The present invention will be described in further detail below with reference to the embodiments and accompanying drawings.

[0065] One embodiment of the present invention is a defogging method based on infrared light and visible light image fusion, which includes the following steps.

[0066] 1. Image preprocessing: Preprocess foggy images, including denoising and enhancement, to improve subsequent fusion effects.

[0067] In view of the characteristics of infrared images, which are low sensitivity to fog and can better capture the shape of objects but lack texture details and color information, and visible light images, which contain complete color information and good detail expression but have reduced contrast and blurred edges due to fog, data enhancement and denoising methods are used to improve the processing ability of the defogging model based on the fusion of infrared and visible light images for foggy images under different conditions, so that the defogging model based on the fusion of infrared and visible light images has better performance and generalization effect.

[0068] The specific steps are as follows:

[0069] 1-1) Visible light images usually suffer from reduced contrast due to fog, resulting in loss of details and texture information. Image enhancement methods such as contrast stretching and gamma transformation can effectively enhance the contrast of visible light images to restore important details in the image. Brightness enhancement and saturation enhancement can also increase the richness of colors to restore the color information lost due to fog. Figure 1 shown.

[0070] 1-2) For infrared images, in foggy images with low noise intensity, median filtering is used to remove noise; in foggy images with high noise intensity, Gaussian filtering is used to smooth the image.

[0071] In foggy scenes, infrared images are often affected by various noise factors. For low-quality infrared images, foggy environments significantly increase the noise in the image, such as atmospheric scattering noise, sensor noise, and low contrast. Denoising is particularly important for recovering effective image information in low visibility (i.e., dense fog) and low light conditions.

[0072] According to the noise characteristics of foggy images, the present invention adopts the median filter to remove noise in images with low noise intensity, which can effectively remove non-Gaussian noise while better preserving the image edge. Figure 2 shown.

[0073] Median filtering is an image processing technique based on statistical analysis. Its basic idea is to use grayscale mean instead of grayscale variance for image denoising. Its basic principle is to divide the image into several windows of equal size in the grayscale value space, and calculate the ratio of the grayscale value of each pixel in each window to the grayscale value of the pixel in its neighborhood window. This ratio is used as a grayscale feature vector of the pixel. This feature vector is then median filtered and the weighted average of the ratio of the grayscale value of all pixels in the neighborhood window is taken to obtain a smoothed image.

[0074] In images with high noise intensity, Gaussian filtering is used to smooth the image, removing noise while retaining the image's structural information, such as Figure 3 As shown in Figure 2. Gaussian filtering is a linear smoothing filter suitable for eliminating Gaussian noise and is widely used in image processing noise reduction. In layman's terms, Gaussian filtering is a weighted averaging process for the entire image. The value of each pixel is the weighted average of its own value and the values ​​of other pixels in its neighborhood. The specific operation of Gaussian filtering is to scan each pixel in the image using a template (also called convolution or mask), and then replace the value of the central pixel of the template with the weighted average grayscale value of the pixels in the neighborhood determined by the template.

[0075] For example, to generate a 3×3 Gaussian filter template, sample with the center of the template as the coordinate origin. The coordinates of the Gaussian filter template at various positions are shown in the following table:

[0076] Table 1 Gaussian filter template

[0077] （-1，1） （0，1） （1，1） （-1，0） （0，0） （1，0） （-1，-1） （0，-1） （1，-1）

[0078] Preferably, in another embodiment, applying denoising to the image in a multi-scale hierarchical structure can restore image details at different scales, help remove low-frequency noise caused by fog, and significantly restore effective information of the image. For example, the original image is first scaled at 0.5, 1.0, 1.5, and 2.0 magnifications respectively; then, the scaled images are filtered to obtain multi-scale filtering denoising results. The filtering denoising results at multi-scale process details at different levels, and have richer and more distinct texture details and feature information compared to single-scale filtering denoising; finally, the multi-scale filtering denoising results are reprojected to the original image size, and the final fused image is obtained after equal weighted summation, such as Figure 4 shown.

[0079] When training a dehazing model based on infrared and visible light image fusion, the training set uses symmetrical data. Using a camera capable of both infrared and visible light imaging, collect images of the same scene on foggy and non-foggy days to create a set of symmetrical data. The specific steps are as follows:

[0080] First, choose a fixed position and angle, ensuring consistent camera settings for both foggy and non-foggy conditions. Under foggy conditions, capture the scene with a camera when the fog is thick, documenting the visual effects of the foggy day. Then, use the same camera to capture the same scene at the same position and angle under clear, non-foggy conditions. This creates a symmetrical data set for comparative analysis of the visual differences between the same scene in foggy and non-foggy conditions.

[0081] Preferably, when training the defogging model based on the fusion of infrared and visible light images, the training set images are subjected to geometric transformations such as rotation, scaling and cropping, such as Figure 5 As shown in Figure 2, enriching image shapes and perspectives can enhance the model's ability to learn spatial features.

[0082] Second, the pre-processed image is input into the dehazing model based on the fusion of infrared and visible light images to obtain the dehazed image. This includes:

[0083] Process infrared images to enhance their penetration in foggy environments and highlight the target's contour information;

[0084] The visible light image is processed to restore its color and texture details in the foggy environment.

[0085] The infrared image is processed to enhance its penetration in foggy environments and highlight the target's outline. This includes the use of a near-infrared correction lens (a special lens eliminates focal plane offset between infrared and visible light) and a narrowband filter (which allows only near-infrared wavelengths to pass, suppressing visible light interference). The lens combines a large aperture with a long focal length to enhance light transmission and compress the fog layer. Finally, the sensor captures the near-infrared image, or fuses it with visible light to directly output a defogging, high-contrast image.

[0086] Natural light is composed of a combination of light waves with different wavelengths. The visible range for the human eye is roughly 390nm-780nm. The wavelengths, from longest to shortest, correspond to the seven colors red, orange, yellow, green, cyan, blue, and violet. Wavelengths less than 390nm are called ultraviolet light, and wavelengths greater than 780nm are called infrared light. Small particles in the air, such as fog and smoke, block light, reflecting it and preventing it from passing through. Therefore, the human eye, which can only receive visible light, cannot see objects behind fog and smoke. Longer wavelengths, however, have a greater diffraction capacity, meaning they are more capable of bypassing obstructions. Infrared light, with its longer wavelength, is less affected by aerosols during propagation, allowing it to penetrate certain concentrations of haze and smoke, achieving precise focus. Optical fog penetration exploits the principle that near-infrared light can diffract tiny particles, achieving precise and rapid focusing. The key to this technology lies in the lens and filter. This technology utilizes physical methods and the principles of optical imaging to improve image clarity. Optical fog-penetrating technology mainly achieves clear imaging through the physical properties of near-infrared light (700nm~2500nm): Since the wavelength of near-infrared light is longer than that of visible light, it can bypass tiny particles in haze (such as PM2.5), significantly reduce Rayleigh scattering, and thus penetrate the fog.

[0087] Since haze affects the contrast and color authenticity of the image, color restoration algorithm and image restoration technology are used to enhance the texture details in the visible light image.

[0088] The processing of the visible light image to restore its color and texture details in the foggy environment includes: using a color restoration algorithm and image restoration technology to enhance the texture details in the visible light image; using a high-dimensional convolutional neural network (CNN) to extract image color and texture features to achieve adaptive dehazing processing; and optimizing color mapping and detail restoration to make the colors in the image more natural and the texture clearer, thereby restoring the visual effect of the real scene as much as possible.

[0089] Color restoration algorithms and image restoration techniques are used to enhance texture details in visible light images, specifically through a high-dimensional convolutional neural network (CNN). Taking a photo of leaves captured under haze as an example, the CNN is first used to extract the leaf's color features (such as the green spectral distribution of chlorophyll) and texture features (such as the direction of veins and the jagged structure of leaf edges). Adaptive dehazing is then used to address haze-induced color shifts (such as grayish-white leaves) by optimizing the color mapping relationship and restoring lighter greens to true dark greens. Simultaneously, a restoration algorithm enhances the clarity of leaf veins and sharpens blurred leaf edges.

[0090] A high-dimensional convolutional neural network (CNN) is used to extract image color and texture features for adaptive dehazing. The specific process is as follows: Taking a street scene captured on a foggy day as an example, the visible light image is first fed into a multi-layer CNN network. The first convolutional layer uses kernels of varying sizes (e.g., 3×3, 5×5) to capture texture details of the street tiles (e.g., gaps between bricks, surface roughness) and color features of vehicles (e.g., the RGB value distribution of red vehicle bodies). Intermediate layers use pooling to reduce feature dimensionality while retaining key information (e.g., vehicle outlines and road lines). Normalization is also used to eliminate brightness deviations caused by haze. Deeper layers use fully connected layers to map haze concentration to feature impairments and output adaptive dehazing parameters (e.g., color correction coefficients and texture enhancement weights). For example, to address the issue of a lighter red color in vehicles due to haze, the network automatically increases the red channel gain by 25% and applies a 0.8 enhancement weight to the edge features of the tile gaps. This restores the color saturation of the processed image to 90% of that of a non-foggy day and improves texture edge clarity by 50%, effectively restoring the details of the real scene.

[0091] To optimize color mapping and detail restoration, the color deviation characteristics of foggy visible light images (e.g., overall grayish-white shift and reduced saturation) were analyzed to establish a color mapping model based on symmetrical data from non-foggy conditions. Taking a yellow traffic sign captured in foggy conditions as an example, the yellow spectral range of the sign in the same scene in non-foggy conditions (e.g., RGB values ​​of R=255, G=200, B=0) was first extracted as a reference. The actual color values ​​of the sign in the foggy image (e.g., R=200, G=180, B=50) were then compared to calculate the color offset for each channel (ΔR=+55, ΔG=+20, ΔB=-50). Pixel color values ​​in the foggy image were corrected point by point using a linear interpolation algorithm. A bilateral filter was then used to enhance the contrast between the sign's edge and background (e.g., by increasing edge gradients by 40%), thereby restoring font texture details.

[0092] Preferably, in another embodiment, before processing the pre-processed image, the defogging model based on the fusion of infrared and visible light images further includes calculating the atmospheric light intensity and transmittance to estimate the effect of fog on the image, thereby enabling rapid identification of foggy and non-fog areas in the pre-processed image. This mainly includes the following steps:

[0093] (1) Dark channel acquisition

[0094] For the input image I, calculate its dark channel value , see the following formula:

[0095]

[0096] Where,

[0097] Stands for "dark channel", which is a function used to describe the dark features of local areas in an image, indicating the local area where each pixel x of the color image is located Internally, RGB three channels Corresponding to the minimum value in the red, green and blue channels, Represents a window centered at pixel x.

[0098] y represents the local area where pixel x is located Other pixels within.

[0099] It is the pixel value of the image at pixel y and channel c, where r, g, and b correspond to the pixel brightness and other information of the red, green, and blue channels respectively.

[0100] The above formula means that for each pixel x in the image, In the example, find the minimum of the three RGB channel values ​​of all pixels y, and then take the minimum value from the minimum values ​​of these three channels to get the dark channel value of the pixel x. .

[0101] (2) Calculation of atmospheric light intensity

[0102] The average value of the pixels in the sky area of ​​the image is used as the atmospheric light intensity A, and the proportion of the pixels in the sky area to the pixels in the entire image is determined. If the proportion is less than a threshold, the average value of the first K pixels with the largest brightness values ​​in the dark channel image (for example, the first 0.1% of pixels based on brightness) is used as the atmospheric light intensity, where K is an integer.

[0103] In the dark channel image, the pixels in the sky area usually have higher brightness values, that is, they are close to the area with the thickest fog. Therefore, the first K pixels with the largest brightness values ​​in the dark channel image can be taken as the sky area.

[0104] (3) Obtaining image transmittance

[0105] Bright areas, such as the sky, are usually the brightest in foggy images and appear as high values ​​in dark channels (because there are no obstructions). The reason why the defogging image has color distortion in bright areas is that the transmittance in this area is too low. Therefore, the color distortion can be solved by correcting the transmittance in this area. The corrected fog concentration estimate is as follows:

[0106]

[0107] in, Saturation The corrected value of is the normalized dark channel value; is the saturation weight index of pixel x, which can describe the degree to which the pixel belongs to the bright area to a certain extent.

[0108]

[0109] in, The minimum value of pixel x in the RGB three channels in the input image I With the maximum value The ratio.

[0110]

[0111] Where A is the atmospheric light intensity.

[0112] Dark channel value The normalized value D(x) can represent the relative distance between the pixel and the atmospheric light point to a certain extent, which directly reflects the fog concentration. In the dark area (non-foggy area, such as the road appears black), the dark channel value The normalized value D(x) approaches 0, while in the bright area (the foggy area appears white), the dark channel value The larger the value, the normalized value D(x) approaches 1.

[0113] 3. (Optional) Perform image optimization processing such as color balancing and contrast enhancement on the fused image to make the dehazed image more natural and realistic.

[0114] The fused dehazed image output by the dehazing model based on infrared and visible light image fusion is very close to the effect of the real scene, but it may still contain some unnatural or obvious artifacts, such as noise, blur, and unrealistic details. The purpose of this step of image optimization processing is to enhance the texture details of the image, making the image more natural and realistic, so as to better reflect the characteristics of real-world objects. It includes:

[0115] 4-1) Use bilateral filtering or Gaussian filtering to remove image noise and smooth the image while preserving image details as much as possible. Use histogram equalization and local brightness adjustment to adjust the brightness distribution of the image, making the dark and bright parts of the image more natural, avoiding overexposure or overdarkness in certain areas, and increasing the texture information of the image to make the image features more obvious, such as Figure 6 shown.

[0116] 4-1-1) Inputting the fused image into a bilateral filter or a Gaussian filter to obtain a denoised and smoothed image;

[0117] By using bilateral filtering or Gaussian filtering, the image noise of the fused noisy image can be removed, the image can be smoothed, and the image details can be retained as much as possible.

[0118] Bilateral filtering constructs a weight matrix by simultaneously considering pixel spatial distance and color difference, and performs a weighted average of similar color pixels in the neighborhood of each pixel, retaining edge details while removing Gaussian noise or salt and pepper noise; Gaussian filtering performs a weighted summation of neighborhood pixels based on a Gaussian kernel function, which is suitable for eliminating Gaussian noise and smoothing the image.

[0119] 4-1-2) Histogram Equalization

[0120] The brightness distribution of the image is adjusted by histogram equalization and local brightness adjustment, making the dark and bright areas in the image with uneven brightness distribution more natural, avoiding overexposure or excessive dimming of certain areas, and increasing the texture information of the image to make the image features more obvious.

[0121] By statistically analyzing the grayscale distribution of image pixels, the original histogram is stretched to a uniform distribution, thereby improving the overall contrast. For example, the dark road surface in a foggy image (where the input grayscale is concentrated in the low range) is expanded to the full grayscale range, making dark details (such as road cracks) more clearly visible.

[0122] 4-1-3) Adjust the brightness of overexposed or dark areas in the image.

[0123] By dividing the image into blocks and calculating the mean and variance of each block, the brightness gain is reduced for overly bright areas (such as the bright sky caused by fog reflection) and the brightness offset is increased for overly dark areas (such as vehicles in the shadows). At the same time, combined with multi-scale Gaussian pyramid decomposition, local brightness is adjusted at different levels to avoid detail loss caused by global adjustment. Ultimately, the bright parts of the image are not overexposed, the dark parts have details, and the texture features (such as the vehicle outline) are more prominent, resulting in an image with natural brightness.

[0124] 4-2) Using image sharpening technology to enhance image edges and details, the blurred parts of the image can be given a better sense of reality, and the edges and details of the image can be enhanced to make the image look clearer. Figure 7 shown.

[0125] In areas with heavy fog, defogging often results in loss of image details. To restore these details, this paper uses a Laplacian filter to enhance image details. The specific process is as follows:

[0126] First, the dehazed image is taken as input. The Laplacian operator is used to calculate the second-order derivative of the image, highlighting the edge areas where grayscale changes suddenly. This yields a Laplacian image containing both edge and detail information. The Laplacian image is then superimposed on the original image by controlling the sharpening coefficient k (usually set to 0.5-1.5), where I is the original image and I' is the sharpened image. See the following formula:

[0127] Let the original image be I, the Laplacian image be L, and the final sharpened image I' be calculated as follows:

[0128]

[0129]

[0130] in, is the second-order partial derivative of the original image I in the x direction, is the second-order partial derivative of the original image I in the y direction. k is the coefficient that controls the sharpening strength.

[0131] Taking a building photographed on a foggy day as an example, the wall texture in the original image is blurred (input). After calculation using the Laplace operator, edge features such as wall brick joints and window frames are extracted. After superposition, the edge contrast is enhanced, making the wall texture clearer and the reflective details of the window glass more realistic. This effectively restores the blurred details of the image caused by fog and improves visual clarity.

[0132] Preferably, in another embodiment, a multi-scale pyramid filtering method is used to enhance details at different scales, so that while restoring large-scale details, local details can also be maintained. Figure 8 shown.

[0133] First, a multi-scale pyramid of images is constructed. By performing multiple Gaussian filtering and downsampling on the original image (such as the image to be enhanced after defogging), pyramid-level images of different resolution levels (large to small) are generated. The large-scale level highlights the overall outline, while the small-scale level retains fine textures. Then, for each pyramid level, an adaptive filter kernel is designed (such as using a larger convolution kernel to enhance structural details at large scales and a small convolution kernel to capture tiny textures at small scales). The images at each level are filtered and enhanced to highlight the detailed features at that scale. After that, the enhanced images at each level are restored to the original image size at the corresponding scale through upsampling and fusion operations. Then, weights are assigned according to the importance of details at each scale (such as high weight for small-scale texture details and slightly lower weight for large-scale structural details) and fused. Finally, an image with enhanced details at different scales, overall clarity, and rich layers is obtained. For example, when processing a distant building image on a foggy day, the building outline is enhanced at the large scale, while the wall texture and window details are enhanced at the small scale. After fusion, the building details are fully highlighted, and the visual effect is more realistic.

[0134] 4-3) Dehazing may reduce the contrast of an image, making it appear flat and prone to color deviations. For example, the image may be too cold or too warm. To restore the natural color of the image, white balance is adjusted to eliminate unnatural color differences and the saturation of the image is moderately increased to make the colors more vivid and maintain a natural feel. This includes:

[0135] First, use an automatic white balance algorithm (such as the gray world method, which assumes that the mean value of the RGB components in the image tends to neutral gray and calculates the gain of each channel for correction; or the perfect reflection method, which finds the brightest area in the image that should be white and adjusts the color balance accordingly) to analyze the image color deviation, determine the color offset caused by ambient light, etc., and make targeted compensation for the pixel values ​​of the red, green, and blue channels to restore the white or gray areas to natural colors and eliminate color casts (for example, foggy images often tend to be bluish-gray, so the gains of the red and green channels are increased).

[0136] After completing the white balance adjustment, the saturation channel of the image is extracted based on the HSV color space. Through linear or nonlinear transformations (such as setting an appropriate gain coefficient to stretch the saturation value to avoid oversaturation), the color vividness is enhanced. At the same time, combined with the visual characteristics of the human eye, the saturation adjustment of the highlight and shadow areas is constrained to ensure that the adjusted image colors are natural and realistic, eliminating color casts and making colors more vivid. For example, when processing foggy landscape photos, after correcting the cold white balance, the saturation of flowers, plants, and sky is appropriately increased to restore the colors of the picture and make it more visually attractive.

[0137] The defogging method and device based on the fusion of infrared and visible light images of the present invention utilize the difference between infrared and visible light in penetrating fog, and by fusing the two spectral information, improve the clarity and contrast of the image, thereby achieving a defogging effect. Infrared light has a strong ability to penetrate fog and can obtain clear contour information of foggy scenes. Visible light images have low contrast in foggy environments, but contain rich color and texture information. By fusing infrared and visible light images, the strengths of the two are complemented to achieve a defogging effect, which can solve the problems of low clarity of foggy images, color distortion, loss of details, registration errors, high computational complexity, inaccurate fog concentration estimation, poor defogging effect in low light conditions at night, and equipment cost and complexity in the prior art.

[0138] Preprocessing of infrared and visible light images, including denoising and enhancement, improves subsequent fusion effects;

[0139] By calculating the air light intensity and transmittance and estimating the effect of fog on the image, the foggy and non-fog areas in the image can be quickly distinguished.

[0140] By processing infrared light images, its penetration effect in foggy environments is enhanced, and the contour information of the target is highlighted;

[0141] By processing visible light images, the color and texture details in foggy environments are restored;

[0142] By performing color balancing and contrast enhancement on the fused image, the dehazed image is made more natural and realistic.

[0143] Adopting multi-scale feature extraction and weight distribution method, intelligently distribute fusion weights according to fog concentration and image features, effectively retain image details and improve contrast; by optimizing algorithm structure, reduce computational complexity, achieve fast defogging processing, meet the needs of real-time monitoring; combine infrared light and visible light information to more accurately estimate fog concentration and improve the stability of defogging effect; at night or in low light conditions, use infrared light's night vision capability, combined with limited light information, to achieve all-weather defogging effect. The defogging method based on infrared light and visible light image fusion of the present invention is used to achieve defogging effect such as Figures 9-11 shown.

[0144] The defogging method and device based on the fusion of infrared light and visible light images of the present invention not only significantly improves the clarity and visual effect of foggy images, making the target more prominent, the color more natural, and the contrast effectively improved, but also retains the detailed information of the image, reduces the registration error, improves the consistency of the fused image, realizes rapid defogging processing, enhances the accuracy of fog concentration estimation, ensures the defogging effect in complex environments, and at the same time reduces the user's cost investment, improves the practicality and market competitiveness of the system, provides more reliable technical support for security monitoring, traffic management, remote sensing detection and other fields, can realize real-time defogging of images, always ensure road traffic safety, and has good development and application prospects.

[0145] In some embodiments, certain aspects of the above-described technology can be implemented by one or more processors of a processing system that executes software. The software includes one or more sets of executable instructions stored or otherwise tangibly implemented on a non-transitory computer-readable storage medium. The software may include instructions and certain data that, when executed by one or more processors, manipulate the one or more processors to perform one or more aspects of the above-described technology. The non-transitory computer-readable storage medium may include, for example, magnetic or optical disk storage devices, solid-state storage devices such as flash memory, cache, random access memory (RAM), or other non-volatile memory devices. The executable instructions stored on the non-transitory computer-readable storage medium may be source code, assembly language code, object code, or other instruction formats that are interpreted or otherwise executed by one or more processors.

[0146] Computer-readable storage media may include any storage medium or combination of storage media that can be accessed by a computer system during use to provide instructions and / or data to the computer system. Such storage media may include, but are not limited to, optical media (e.g., compact discs (CDs), digital versatile discs (DVDs), Blu-ray discs), magnetic media (e.g., floppy disks, magnetic tapes, or magnetic hard drives), volatile memory (e.g., random access memory (RAM) or cache), non-volatile memory (e.g., read-only memory (ROM) or flash memory), or microelectromechanical systems (MEMS)-based storage media. Computer-readable storage media may be embedded in a computing system (e.g., system RAM or ROM), fixedly attached to a computing system (e.g., a magnetic hard drive), removably attached to a computing system (e.g., an optical disc or universal serial bus (USB)-based flash memory), or coupled to a computer system via a wired or wireless network (e.g., network accessible storage (NAS)).

[0147] Please note that not all activities or elements in the above general description are required, that a portion of a particular activity or device may not be required, and that one or more further activities or included elements may be performed in addition to those described. Furthermore, the order in which the activities are listed is not necessarily the order in which they are performed. Moreover, the concepts have been described with reference to specific embodiments. However, those skilled in the art recognize that various modifications and changes may be made without departing from the scope of the present disclosure as set forth in the claims below. Accordingly, the specification and drawings are to be regarded as illustrative rather than restrictive, and all such modifications are to be included within the scope of the present disclosure.

[0148] Benefits, other advantages, and solutions to problems have been described above with respect to specific embodiments. However, none of the benefits, advantages, solutions to problems, nor any feature that may cause any benefit, advantage, or solution to occur or become more apparent should be construed as key, essential, or an essential feature of any or all of the claims in any or other respects. Furthermore, the specific embodiments disclosed above are illustrative only, as the disclosed subject matter may be modified and practiced in different but equivalent manners apparent to those skilled in the art having the benefit of the teachings herein. No limitation is intended to the details of construction or design shown herein, except as described in the claims. It is therefore apparent that the specific embodiments disclosed above may be altered or modified, and all such variations are considered within the scope of the disclosed subject matter.

Claims

1. A defogging method based on infrared and visible light image fusion, characterized in that: include: Image preprocessing: denoising and enhancing foggy images; The preprocessed image is input into a defogging model based on the fusion of infrared and visible light images to obtain a defogged image. The defogging model based on the fusion of infrared and visible light images performs the following processing on the input image: Process infrared images to enhance their penetration in foggy environments and highlight the target's contour information; The visible light image is processed to restore its color and texture details in the foggy environment.

2. The defogging method based on infrared and visible light image fusion according to claim 1, characterized in that: The infrared image is processed to enhance its penetration effect in a foggy environment and highlight the contour information of the target, including: A near-infrared correction lens is used to eliminate the focal plane offset between infrared and visible light, and a narrow-band filter is used to allow only the near-infrared band to pass, suppressing visible light interference.

3. The defogging method based on infrared and visible light image fusion according to claim 1, characterized in that: The processing of the visible light image to restore its color and texture details in the foggy environment includes: Color restoration algorithms and image restoration techniques are used to enhance texture details in visible light images. High-dimensional convolutional neural networks are used to extract image color and texture features. The extraction of image color and texture features using high-dimensional convolutional neural networks includes: First, the visible light image is input into the multi-layer CNN network. The first convolutional layer captures the texture details and color features in the image through convolution kernels of different sizes; the middle layer reduces the feature dimension through pooling operations while retaining key information, and eliminates the brightness deviation caused by haze through normalization processing; the deep network establishes a mapping relationship between haze concentration and feature damage through fully connected layers, and outputs adaptive dehazing parameters.

4. The defogging method based on infrared and visible light image fusion according to claim 3, characterized in that: Before processing the pre-processed image, the defogging model based on infrared and visible light image fusion also includes calculating the atmospheric light intensity and transmittance to estimate the impact of fog on the image, including the following steps: (1) Dark channel acquisition For the input image I, calculate its dark channel value , see the following formula: Where, Stands for "dark channel", which is a function used to describe the dark features of local areas in an image, indicating the local area where each pixel x of the color image is located Internally, RGB three channels Corresponding to the minimum value in the red, green and blue channels, Represents a window centered on pixel x; y represents the local area where pixel x is located Other pixels within It is the pixel value of the image at pixel y and channel c, where r, g, and b correspond to the pixel brightness of the red, green, and blue channels respectively; (2) Calculation of atmospheric light intensity The average value of the pixels in the sky area of ​​the image is used as the atmospheric light intensity A, and the proportion of the pixels in the sky area to the pixels in the entire image is determined. If the proportion is less than a threshold, the average value of the first K pixels with the largest brightness values ​​in the dark channel image (for example, the first 0.1% of pixels in terms of brightness) is used as the atmospheric light intensity, where K is an integer. (3) Obtaining image transmittance The transmittance of the defogging image in the bright area is corrected, and the corrected fog concentration estimate is as follows: in, Saturation The correction value is calculated as follows: is the normalized dark channel value, and the calculation formula is as follows; is the saturation weight index of pixel x; in, The minimum value of pixel x in the RGB three channels in the input image I With the maximum value The ratio of Where A is the atmospheric light intensity.

5. The defogging method based on infrared and visible light image fusion according to any one of claims 1 to 4, characterized in that: The method further includes performing image optimization processing on the fused image; the image optimization processing on the fused image includes: The fused image is input into a bilateral filter or a Gaussian filter to obtain a denoised and smoothed image; the original histogram is stretched to a uniform distribution by statistically analyzing the grayscale distribution of image pixels; the image is divided into blocks and the mean and variance of each block are calculated, the brightness gain of overly bright areas is reduced, and the brightness offset of overly dark areas is increased. At the same time, combined with multi-scale Gaussian pyramid decomposition, local brightness is adjusted at different levels.

6. The defogging method based on infrared and visible light image fusion according to claim 5, characterized in that: The image optimization processing on the fused image further includes: The dehazed image is used as input, and the second-order derivative of the image is calculated using the Laplacian operator to obtain a Laplacian image containing edge and detail information; the Laplacian image is superimposed on the original image by controlling the sharpening strength coefficient k, where k is 0.5-1.

5.

7. The defogging method based on infrared and visible light image fusion according to claim 6, characterized in that: It also includes the use of multi-scale pyramid filtering to enhance details at different scales. The specific steps are: First, a multi-scale pyramid of images is constructed. By performing multiple Gaussian filtering and downsampling on the original image, pyramid-level images of different resolution levels are generated. Large-scale levels highlight the overall contour, while small-scale levels retain fine textures. Then, for each pyramid level, an adaptive filter kernel is designed to perform filtering enhancement on the images at each level, highlighting the detailed features at that scale. Afterwards, the enhanced images at each level are restored to the original image size at the corresponding scale through upsampling and fusion operations, and then fused based on the weights assigned according to the importance of details at each scale.

8. The defogging method based on infrared and visible light image fusion according to claim 5, characterized in that: The image optimization processing of the fused image further includes adjusting the white balance, specifically including: First, the automatic white balance algorithm analyzes the color deviation of the image and determines the color offset caused by ambient light and other factors. Then, the pixel values ​​of the red, green, and blue channels are compensated to restore the white or gray areas to their natural colors and eliminate color casts. After completing the white balance adjustment, the saturation channel of the image is extracted based on the HSV color space. The color vividness is enhanced through linear or nonlinear transformation. At the same time, the saturation adjustment of the highlight and shadow areas is constrained in combination with the visual characteristics of the human eye.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the defogging method based on infrared light and visible light image fusion according to any one of claims 1 to 8 are implemented.

10. A computer program product comprising a computer program, characterized in that: When the computer program is executed by a processor, the steps of the defogging method based on infrared light and visible light image fusion according to any one of claims 1 to 8 are implemented.

Images