Differential defogging method based on near infrared and visible light
Based on the difference between the difference between the near -infrared and visible light, the problem of poor fog effects in existing technologies on the condition of dense fog, and achieved the effect of efficient removal of smog, restoring scene information and target information.
Patent Information
- Application Number
- CN202411444341.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-16
- Publication Date
- 2025-01-28
AI Technical Summary
The prior art is difficult to effectively remove smog under the conditions of treating dense fog, and fog residues, artifacts and noise often appear, making it difficult to achieve the ideal recovery effect.
Using a differential method based on near-infrared and visible light, through the clear correction of near-infrared images, establish the thickness of the haze first, and use the near-infrared-visible light difference strategy to obtain the fog concentration of the RGB three bands. Finally, the dark fog is based on dark fog. Reduce the model to achieve fog.
Simply and effectively remove different degrees of smog in real outdoor field scenarios, avoid introducing pseudo -shadows and noise, significantly restore scene information and target information, and improve image quality and resolution.
Smart Images
Figure CN119379559A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical fields of computer vision, image processing, etc., and specifically relates to a differential defogging method based on near-infrared and visible light. Background Art
[0002] As common atmospheric suspended scattering particles in real scenes, haze often leads to loss of image details and reduced contrast, which greatly affects scene information acquisition and target recognition. Therefore, developing effective defogging methods is of great significance for restoring image quality. In recent years, defogging methods for single images have become a research hotspot, which can be mainly divided into three types: image enhancement-based, prior information-based, and deep learning-based. Among them, the image enhancement-based method only improves the image contrast through image processing and does not really remove the haze, so the effect is very limited; the method based on prior information and deep learning is the current mainstream research direction, and has achieved good defogging effects. However, the difficult-to-breakthrough haze thickness limit limits its performance under dense fog conditions.
[0003] In order to overcome the limitations of a single image in strong haze conditions, the strategy of fusing near-infrared and visible light images has attracted much attention. Visible light (400-700nm) images have high resolution and rich color information, while near-infrared (900-1700nm) images show excellent penetration in haze. Through a variety of fusion strategies, such as space-frequency domain conversion, color space transformation, bilateral filtering, guided filtering and multi-scale analysis, the advantages of visible light and near-infrared images are combined to obtain high-contrast dehazed images. By utilizing the prior information of the near-infrared, it has a higher upper limit of dehazing than a single image. However, most of the existing near-infrared-visible light fusion dehazing methods focus on image processing and lack the support of physical models. Therefore, fog residues, artifacts and noise often occur, making it difficult to achieve ideal restoration effects. Summary of the invention
[0004] In view of the defects and shortcomings of the above-mentioned prior art, the present invention proposes a differential defogging method based on near-infrared and visible light. By correcting the clarity of the near-infrared image, the problem of detail loss in the processing process is solved. Then, the haze thickness prior is established and the haze concentration of each of the three RGB bands is obtained using the near-infrared-visible light differential strategy. Finally, defogging is achieved based on the dark haze reduction model. This method can easily and effectively remove different degrees of haze in real outdoor large-field-of-view scenes without introducing additional artifacts and noise.
[0005] It synchronously collects near-infrared and visible light images, performs image registration and near-infrared clarity correction, and then uses near-infrared-visible light difference technology to obtain the initial fog concentration map. On this basis, a haze thickness prior is established based on a large amount of statistical information, and a defogged image is generated in combination with a dark fog reduction model. This method effectively solves the problem that the existing image defogging technology has obvious fog residue and is easy to introduce artifacts and noise in a dense fog environment, and realizes the fine extraction of image details in foggy scenes. This method not only significantly restores the scene information, but also accurately obtains high-resolution and high-definition target information. The method of the present invention has low complexity, few parameters to adjust, and is easy to implement, and has broad application potential in the field of computer vision.
[0006] The technical solution specifically adopted by the present invention to solve the technical problem is: A dehazing method based on near-infrared and visible light differential: image registration is performed on the synchronously collected near-infrared and visible light images, and near-infrared clarity correction is achieved by establishing feature point mapping relationship and using multi-scale fusion; the fog concentration of each RGB band is obtained by establishing a haze thickness prior and using the near-infrared-visible light differential strategy, and finally dehazing is achieved based on the dark fog reduction model.
[0007] Furthermore, near-infrared and visible light images are collected synchronously, and the field of view overlap of the dual-channel images is ensured to meet the temporal and spatial consistency.
[0008] Furthermore, the image registration utilizes a feature point detection and description method to extract feature information in the image and find matching point pairs between the near-infrared and visible light images; extracts and records the coordinates of the matching feature points from the dual-channel image; then screens the matching feature points to remove mismatched points and obtain inliers; extracts corresponding coordinate pairs based on the inliers, and completes the image registration using the estimated transformation matrix.
[0009] Furthermore, the near-infrared clarity correction achieved by multi-scale fusion is specifically as follows: first, a linear mapping relationship is established between the feature points after the dual-channel images are correctly matched, so as to reconstruct a near-infrared image whose clarity and haze residue meet the set standards; then, a multi-scale fusion method is used to simultaneously retain the strong fog-penetrating details of the original near-infrared and the high clarity of the reconstructed near-infrared, so as to obtain a corrected near-infrared image.
[0010] Furthermore, the fog thickness prior is established and the fog concentration of each of the three bands of RGB is obtained by using the near-infrared-visible light difference strategy, and finally the defogging is realized based on the dark fog reduction model. Specifically, the corrected near-infrared image with strong fog penetration is approximately regarded as a fog-free image, and the average fog concentration is separated by the difference between the grayscale visible light image and the fog-free image, and the product of three times the average fog concentration is used as the initial fog concentration map. ; Then, a large number of (e.g., 2000) real outdoor foggy-fog-free image pairs are intercepted, and the fog concentration of each of the three RGB bands is separated by differential operation. The distribution patterns between them are statistically analyzed and fitted to obtain linear expressions; as an experience-based prior for haze thickness, after obtaining the fog concentration map of the three RGB bands, the dark fog subtraction model is used to remove fog and obtain the final differential dehazed image.
[0011] Furthermore, the empirically based priori estimate of haze thickness is specifically: in, The fog density maps of the three bands of RGB are combined Separate the haze component of visible light images.
[0012] Furthermore, the method of using the dark fog subtraction model to remove fog and obtain the final difference defogging image is specifically as follows: in, x is a specific pixel in the image, I and J These are foggy and non-fog images respectively.
[0013] And, a differential defogging system based on near-infrared and visible light, including: a registration module, used to perform image registration on synchronously collected near-infrared and visible light images; a clarity correction module, used to achieve near-infrared clarity correction by establishing a feature point mapping relationship and utilizing multi-scale fusion; and a defogging module, used to obtain the fog concentration of each of the three RGB bands by establishing a haze thickness prior and utilizing a near-infrared-visible light differential strategy, and finally achieve defogging based on a dark fog reduction model.
[0014] An electronic device comprises a memory, a processor and a computer program stored in the memory and executable on the processor. When the processor executes the program, the steps of a differential defogging method based on near infrared and visible light as described above are implemented.
[0015] A non-transitory computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of a near-infrared and visible light differential defogging method as described above.
[0016] Compared with the prior art, the present invention and its preferred solution are designed to simply and effectively remove different degrees of haze in real outdoor large-field scenes while avoiding the introduction of artifacts and noise. Through reliable haze component extraction and removal strategies, the scene details hidden in the thick fog can be significantly restored while maintaining the naturalness of the image, ensuring the clarity and information integrity of the image. This not only greatly improves the image quality, but also provides a more reliable data basis for subsequent image analysis and processing. BRIEF DESCRIPTION OF THE DRAWINGS
[0017] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments: Figure 1 A flowchart for implementing the method in an embodiment of the present invention; Figure 2 This is a near infrared-visible light pixel value mapping relationship diagram after feature point matching in an embodiment of the present invention; Figure 3 is a schematic diagram of near-infrared clarity correction results in an embodiment of the present invention; Figure 4 A schematic diagram of a priori haze thickness in an embodiment of the present invention; Figure 5 Schematic diagram of near infrared-visible light differential defogging results in an embodiment of the present invention. DETAILED DESCRIPTION
[0018] In order to make the features and advantages of this patent more obvious and easy to understand, the following embodiments are specifically described in detail as follows: It should be noted that the following detailed descriptions are illustrative and are intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used in this specification have the same meanings as those commonly understood by those skilled in the art to which the present application belongs.
[0019] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.
[0020] like Figure 1 As shown, this embodiment provides a method for defogging based on near infrared and visible light differences, and its typical implementation process may include the following steps: Step S1: collecting near-infrared and visible light images; Step S2: feature point matching and image registration; Step S3: Establishing feature point mapping relationship and realizing near infrared clarity correction by using multi-scale fusion; Step S4: obtaining an initial fog concentration map by near infrared-visible light difference; Step S5: Establish haze thickness prior to separate the RGB three-band haze concentration, and use the dark haze subtraction model to remove haze.
[0021] In this embodiment, in step S1, near-infrared and visible light images are collected synchronously, and the field of view overlap of the dual-channel images is ensured to meet the temporal and spatial consistency. The acquisition band of the near-infrared image is 900-1700nm, and the resolution is 640*512 pixels; the resolution of the visible light image is 3072*2048 pixels.
[0022] Consider that the present invention needs to perform image registration on near-infrared-visible light image pairs. In the embodiment, as a preferred solution, image feature detection and matching refers to the SIFT algorithm (version 0.9.21) in the VLFeat (Vision Lab Features Library) library, which is used to extract local feature points of the image and their descriptors. In addition, in the near-infrared clarity correction process, this embodiment refers to "Multi-scale Guided Image and Video Fusion: A Fast and Efficient Approach" (MGF for short) published by Bavirisetti et al. in the journal "Circuits, Systems, and Signal Processing" (2019, Vol. 38, pp. 5576-5605). Through MGF, the present invention retains the fog-penetrating details of the original image and the visual clarity of the visible light image in the corrected near-infrared image.
[0023] Therefore, in this embodiment, step S2 specifically includes the following steps: Step S21: Use the SIFT algorithm in the VLFeat library to complete feature point detection and extract the corresponding feature descriptors, and use the matchFeatures function to find the matching point pairs between the near-infrared and visible light images based on the feature descriptors, where the matching threshold is set to 50, the maximum ratio is 0.6, and each match is unique. Then extract the feature point coordinates of the corresponding matching pairs from the dual-channel image feature points, and record the initially obtained matching feature points; Step S22: Use the RANSAC algorithm to filter the matched feature points to remove the mismatched points and obtain the inliers (i.e., correctly matched points). Then, according to the inlier index, the coordinates of the inliers are extracted from the original matching point set. Finally, based on the inlier coordinate pairs, the estimateGeometricTransform function is used to estimate the similarity transformation matrix, and the image is transformed by the imwarp function, combined with the nearest neighbor upsampling, to complete the image registration (see Figure 3 (a)).
[0024] Since near-infrared images have a larger field of view but smaller resolution than visible light images, the visual clarity of the registered near-infrared images is significantly weaker than that of the visible light images. Therefore, the present invention will introduce a near-infrared clarity correction method in combination with the existing multi-scale fusion.
[0025] In this embodiment, step S3 specifically includes the following steps: Step S31: Figure 2 As shown in Figure 2, a linear mapping relationship is established between the feature points after the dual-channel images are correctly matched, and the linear mapping relationship is used to reconstruct a near-infrared image with high definition but more haze residue (see Figure 3 (b)). This is because, on the one hand, simple pixel value stretching is not enough to separate the haze, and on the other hand, the feature points are a small part of all pixels with clear contours and are not enough to represent the entire scene; Step S32: The present invention uses MGF (Multi-scale Guided Image and Video Fusion, a type of multi-scale fusion) to simultaneously retain the strong fog-penetrating details of the original near-infrared image and the high definition of the reconstructed near-infrared image to obtain a corrected near-infrared image (see Figure 3 (c)). In MGF, the relevant formula for guided filtering is: in, For the k A local window, g To guide the image, q For the output image, a and b is the linear coefficient within the window, represents the gradient. Therefore q Will have g Similar gradient changes.
[0026] In this embodiment, in step S4, the corrected near-infrared image with strong fog penetration is approximately regarded as a fog-free image, and the average fog concentration is separated by the difference between the grayscale visible light image and the image, and the product of three times the average fog concentration is used as the initial fog concentration map. .
[0027] In this embodiment, step S5 specifically includes the following steps: Step S51: intercept image blocks from a real outdoor foggy-foggy image pair, and use the differential operation mentioned in step S4 to separate the fog concentration of each of the three RGB bands, such as Figure 4 The distribution law between them is statistically analyzed and fitted to obtain a linear expression. By statistically analyzing 2000 image blocks, the final empirically based haze thickness prior is obtained: The empirically based prior of haze thickness is specifically: in, The fog density maps are RGB bands. The haze component of the visible light image can be separated; In the foggy-fog-free image pair, the additional information contained in the foggy image compared to the foggy image can be considered as the haze component, which can be extracted by pixel value difference. Different from the previous method of only studying foggy images, the present invention focuses on analyzing the relationship between the pure haze components in the RGB three bands; Step S52: After obtaining the fog density map of the RGB three-band, use the dark fog subtraction model to remove the fog and obtain the final Figure 5 The difference dehazed image shown: in, x is a specific pixel in the image, I and J These are foggy and non-fog images respectively.
[0028] The embodiment of the present invention takes the near-infrared and visible light images captured in foggy scenes as the research objects, and proposes a differential defogging method based on near-infrared and visible light, which realizes the effective removal of dense fog without introducing additional artifacts and noise. Considering the low resolution of the near-infrared image and the low clarity after registration, the present invention introduces a near-infrared clarity correction method, and retains the original near-infrared fog-penetrating details. Then, through a large number of statistics on foggy-fog-free image pairs, the haze thickness prior is established, and the fog concentration map of the RGB three-band is extracted and separated. Finally, the dark fog subtraction model is used to achieve the defogging task.
[0029] like Figure 5The figure shows a schematic diagram of the near-infrared-visible light differential defogging result obtained by using an embodiment of the present invention in a real haze scene with a large field of view outdoors. It can be clearly seen that after the differential defogging process, the image successfully removes the interference of haze of different concentrations, especially the details of the mountains in the thick fog and the distant buildings are significantly enhanced, and the scene information is greatly restored. In addition, the defogging image has no obvious noise and presents high definition, which ensures the effective recognition of the scene and improves the practical value of the image in complex environments.
[0030] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.
[0031] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0032] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0033] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0034] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in the present invention should be understood by people with ordinary skills in the field to which the present invention belongs. The "first", "second" and similar words used in the present invention do not indicate any order, quantity or importance, but are only used to distinguish different components. "Include" or "comprise" and similar words mean that the elements or objects appearing before the word include the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.
[0035] The above is only a preferred embodiment of the present invention, and does not limit the present invention in other forms. Any technician familiar with the profession may use the above disclosed technical content to change or modify it into an equivalent embodiment with equivalent changes. However, any simple modification, equivalent change and modification made to the above embodiment according to the technical essence of the present invention without departing from the technical solution of the present invention still belongs to the protection scope of the technical solution of the present invention.
[0036] This patent is not limited to the above-mentioned optimal implementation mode. Anyone can derive other various forms of a differential defogging method based on near-infrared and visible light under the inspiration of this patent. All equal changes and modifications made according to the scope of the patent application of the present invention should be covered by this patent.
Claims
1. A differential defogging method based on near infrared and visible light, characterized in that: Image registration is performed on the synchronously collected near-infrared and visible light images, and near-infrared clarity correction is achieved by using multi-scale fusion by establishing a feature point mapping relationship. The fog concentration of each RGB band is obtained by establishing a haze thickness prior and using the near-infrared-visible light difference strategy, and finally defogging is achieved based on the dark fog reduction model.
2. The method for defogging based on difference between near infrared and visible light according to claim 1, characterized in that: Synchronously collect near-infrared and visible light images, and ensure the field of view overlap of dual-channel images to meet temporal and spatial consistency.
3. The method for defogging based on near infrared and visible light differential according to claim 1, characterized in that: The image registration utilizes a feature point detection and description method to extract feature information from an image and find matching point pairs between near-infrared and visible light images; extracts and records the coordinates of matching feature points from dual-channel images; then screens the matching feature points to remove mismatched points and obtain inliers; extracts corresponding coordinate pairs based on the inliers, and completes the image registration using an estimated transformation matrix.
4. The method for defogging based on near infrared and visible light differential according to claim 1, characterized in that: The method of using multi-scale fusion to realize near-infrared clarity correction is specifically as follows: first, a linear mapping relationship between feature points after correct matching of dual-channel images is established to reconstruct a near-infrared image whose clarity and haze residue meet the set standards; then, a multi-scale fusion method is used to simultaneously retain the strong fog-penetrating details of the original near-infrared and the high definition of the reconstructed near-infrared to obtain a corrected near-infrared image.
5. The method for defogging based on near infrared and visible light differential according to claim 1, characterized in that: The method establishes a priori of haze thickness and uses a near-infrared-visible light difference strategy to obtain the fog concentration of each of the three RGB bands, and finally realizes defogging based on a dark fog reduction model. Specifically, the corrected near-infrared image with strong fog penetration is approximately regarded as a fog-free image, and the average fog concentration is separated by the difference between the grayscale visible light image and the fog-free image, and the product of three times the average fog concentration is used as the initial fog concentration map. ; Then, image blocks are intercepted from multiple real outdoor foggy-fog-free image pairs, and the fog concentration of each of the three RGB bands is separated by differential operation. The distribution patterns between them are statistically analyzed and fitted to obtain linear expressions; as an experience-based prior for haze thickness, after obtaining the fog concentration map of the three RGB bands, the dark fog subtraction model is used to remove fog and obtain the final differential dehazed image.
6. The method for defogging based on near infrared and visible light differential according to claim 5, characterized in that: The empirically based prior of haze thickness is specifically: in, The fog density maps of the three bands of RGB are combined Separate the haze component of visible light images.
7. The method for defogging based on difference between near infrared and visible light according to claim 5, characterized in that: The method of using the dark fog subtraction model to remove fog and obtain the final difference defogging image is specifically as follows: in, x is a specific pixel in the image, I and J These are foggy and non-fog images respectively.
8. A differential defogging system based on near infrared and visible light, characterized in that: include: A registration module, used for performing image registration on the synchronously collected near-infrared and visible light images; A clarity correction module is used to achieve near-infrared clarity correction by establishing a feature point mapping relationship and utilizing multi-scale fusion; And the dehazing module is used to obtain the fog concentration of each of the three RGB bands by establishing the haze thickness prior and using the near-infrared-visible light difference strategy, and finally realize dehazing based on the dark haze reduction model.
9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of a differential defogging method based on near-infrared and visible light as described in any one of claims 1 to 7 are implemented.
10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of a near-infrared and visible light differential defogging method as described in any one of claims 1 to 7 are implemented.