Infrared-guided image restoration method under interference of non-uniform scattering medium

Abstract

The invention discloses an infrared-guided image restoration method under interference of a non-uniform scattering medium. The infrared-guided image restoration method comprises the following steps: (1) constructing a progressive guide aggregation module; (2) designing a differential amplification attention module; (3) constructing a structure guide enhancement module; and (4) designing physical coupling loss. Aiming at the problems of non-uniform distribution of interference areas and image feature redundancy, infrared image structure information and visible light texture expression capability are effectively combined, and multi-modal complementary information and physical priori knowledge are combined, so that self-adaptive enhancement and fine structure restoration of the visible light image under the interference of the non-uniform scattering medium are realized. The method provided by the invention is excellent in performance on a non-uniform interference data set, effectively relieves the problems of structure loss, color cast, modal redundancy and the like, provides a high-robustness solution for image enhancement in complex environments such as dust, sand dust, water mist and the like, and provides a new direction for research of an infrared guide image restoration method.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of computer vision and image processing, and particularly relates to an image restoration method under infrared guided non-uniform scattering medium interference. BACKGROUND

[0002] In complex weather perception, road monitoring and metallurgical production, the visual data collected by image sensing devices is often disturbed by non-uniform scattering media such as fog, haze, dust and dust, and the perception quality directly affects the subsequent image understanding, target detection and production safety. In practical applications, due to the combined effects of dynamic distribution of non-uniform scattering media, changes in environmental conditions and differences in device imaging, the image degradation problem shows strong spatial variation, complex blur characteristics and other stationary characteristics, resulting in a significant decrease in image clarity and contrast, thereby affecting the reliability of the automated identification and monitoring system.

[0003] Considering the wide range and typicality of dust in non-uniform scattering media, this paper takes the image dust removal task as a representative task to study the image degradation problem caused by such media. Although non-uniform scattering media differ in composition and distribution, they all exhibit significant scattering effects in the optical imaging process, which have similar effects on image quality, so it is feasible to develop a general image restoration method around the dust removal problem.

[0004] Currently, image dust removal methods mainly include two categories: single-modal image information-based dust removal methods and multi-modal data-assisted dust removal methods.

[0005] Single-modal image information-based dust removal methods use the statistical properties of images themselves or learn priors to model the degradation process, and use deep learning networks or physical models to invert and improve image quality, with the advantages of end-to-end and high efficiency, but they are difficult to adapt to severely degraded image scenarios, and color restoration is prone to distortion, so there are still challenges in generalization. For example, the patent CN114881875B entitled Single Image Defogging Network and Defogging Method Based on U-Net Structure and Residual Network constructs an end-to-end defogging network based on U-Net and residual modules, enhances the feature extraction capability through smooth hollow convolution residual blocks and feature weighted summation modules, and achieves good defogging effect on synthetic foggy dataset. However, this method only relies on visible light image information and does not introduce infrared and other auxiliary sensing data as guiding information, lacking multi-modal perception ability, making it difficult to effectively restore texture details and spatial structure information in complex fog and haze scenarios, limiting the application effect of the model in real industrial environments.

[0006] Dust removal methods assisted by multimodal data introduce additional modalities, such as infrared images, as an aid, enhancing feature restoration through information guidance or fusion. While achieving some success in some complex scenarios, these methods still face shortcomings in the fusion strategy and guidance accuracy of multimodal information, making it difficult to accurately locate and effectively enhance key structures. This leads to problems such as limited modeling capabilities for non-uniform interference and image artifacts. For example, patent CN118781018B, titled "A Deep Learning-Based Infrared Image-Assisted Image Dehazing Method," introduces infrared images as a haze density prior, combines a Transformer with an improved CNN network, and designs a channel-spatial dual attention mechanism, demonstrating good dehazing performance under uniform haze conditions. However, this model is primarily designed for idealized uniform dust scenarios and does not fully consider complex factors such as uneven dust concentration distribution and dynamic particle interference in actual industrial environments. It also lacks robust modeling for non-uniform haze, limiting its fusion performance in complex industrial scenarios such as real blast furnaces.

[0007] From the perspective of modal characteristics, visible light images and infrared images have certain complementarity in perceiving interference areas: the former retains rich texture and color information, which is conducive to detail restoration and enhanced visual consistency, but is sensitive to low visibility areas; the latter has stable advantages in penetrating interference and maintaining structure, and can stably perceive key targets under strong interference conditions.

[0008] Therefore, it is of great significance to improve the quality of restored images by constructing an image restoration strategy with reasonable structure, clear guidance and complementary information, giving full play to the regional guidance ability of infrared images on visible light images, and realizing dynamic perception and adaptive enhancement of non-uniform interference. Summary of the Invention

[0009] In view of the above problems in the prior art, the present application aims to provide an image restoration method under infrared guided non-uniform scattering medium interference. In order to effectively model the scattering degradation characteristics and improve the structure restoration capability of the key area of the image, a progressive guided aggregation module is constructed, a physical-mask coupler composed of a multi-layer double-branch physical perception module and a soft mask modulation unit is embedded step by step, the physical-mask coupler is used to guide the network to perceive the scattering distribution characteristics at different semantic levels, the double-branch physical perception module generates an atmospheric light map, a transmittance map and an intermediate restoration image, the soft mask modulation unit replaces the traditional skip connection, realizes the display adjustment of the non-uniform scattering area, relieves the problem of excessive repair in the weak interference area, and effectively preserves the image details. In view of the differences in the perception of structure and texture representation of infrared and visible light images in different intensity interference areas, a difference amplification attention module and a structure guiding enhancement module are designed, the former constructs a channel and spatial double-path modulation mechanism, dynamically excavates the differences between modalities, and enhances the cross-modal guidance effect; the latter adaptively amplifies or suppresses the difference features based on the interference intensity estimated by the transmittance map, ensures that the high-concentration interference area preferentially fuses the infrared structure information, and the low-concentration area does not excessively depend on the guidance, and improves the accuracy and pertinence of the overall restoration effect. In order to ensure the consistency of the network output and the physical model, a physical coupling loss is constructed, the atmospheric scattering model is embedded into the end-to-end training process, the transmittance map, the atmospheric light map and the restoration image are optimized cooperatively, the physical consistency among the three is maintained under weak supervision, and the network generalization ability and explainability are enhanced. The image restoration method provided by the present application combines infrared guidance and physical consistency constraints, considers structure restoration and information integrity, and provides a new path for physical modeling and cross-modal cooperation for image restoration tasks in non-uniform interference environments such as dust, sand and water mist.

[0010] In order to achieve the above purpose, the solution of the present application is to provide an image restoration method under infrared guided non-uniform scattering medium interference, comprising the following steps:

[0011] (1) Construct a progressive guided aggregation module, embed a physical-mask coupler composed of a double-branch physical perception module and a soft mask modulation unit step by step, the double-branch physical perception module guides the network to perceive the scattering characteristics at different semantic levels, and the soft mask modulation unit realizes adaptive adjustment of the interference area.

[0012] (2) Design a difference amplification attention module, extract difference information between infrared and visible light modalities from two dimensions of channel and space, obtain enhanced infrared features after difference, realize fine modeling of modal difference, and improve the quality of infrared guidance.

[0013] (3) Construct a structure guiding enhancement module, use the multi-scale features of the transmittance map to control the amplification or suppression of the modal difference information, finely adjust the dependence of different interference areas on infrared information, and complete the structure enhancement and information selective fusion under modal guidance.

[0014] (4) Design the physical coupling loss to achieve the coordinated optimization of the transmittance map, atmospheric light intensity map and restored image, and ensure the overall consistency of the final output image and the scattering physical process.

[0015] As an embodiment of the present invention, the specific implementation scheme of the present invention is as follows:

[0016] (1) Constructing a progressive boot aggregation module:

[0017] In the presence of non-uniform scattering media, traditional dust removal methods can easily lead to loss of detail in thin interference areas. This paper proposes a progressive guided aggregation module that achieves progressive perception and suppression through a stacked hierarchy. Multiple physical-mask couplers are designed for each level to adaptively enhance the local non-uniform areas. The specific steps are as follows:

[0018] Step 1: The original interference graph is passed through the first level of the progressive guided aggregation module , input convolution to obtain the initial visible light image features:

[0019]

[0020] in, represents the original interference graph; Represents the first-order visible light characteristics.

[0021] Step 2: Input the visible light features into the next level module step by step. Each level module contains multiple physical-mask couplers:

[0022]

[0023]

[0024] in, Indicates the progressive boot aggregation module class, =2,3,4; Indicates the Level visible light characteristics; Indicates module cascade; Indicates the Level A physical-mask coupler.

[0025] Step 3: A dual-branch physical perception module is designed inside each physical-mask coupler based on the atmospheric scattering model to estimate the atmospheric light , transmittance graph and intermediate restoration image , and by Generate a soft mask through the soft mask modulation unit to alleviate the over-repair caused by global consistency processing:

[0026]

[0027]

[0028]

[0029] in, represents the dual-branch physical perception module; Indicates the Level The visible light characteristics of the output of a physical coupler; represents a soft mask modulation unit; Indicates soft mask.

[0030] (2) Design the differential amplification attention module:

[0031] Step 1: Extract infrared features from the infrared image through a single layer of convolution:

[0032]

[0033] in, represents the original infrared image; Indicates infrared characteristics; Represents a single convolution layer.

[0034] Step 2: Design the differential amplifier module based on the differential amplifier principle:

[0035]

[0036] in, Indicates differential mode signal; Indicates common mode signal; Indicates differential mode gain, which controls the degree of amplification of the difference; It represents the common mode gain and controls the residual ratio of redundant information. The present invention focuses on the amplification of the differential mode signal and only retains the first term.

[0037] Step 3: Use infrared characteristics and first-level visible light characteristics to construct differential mode signals:

[0038]

[0039] right Focus on the channel direction and guide channel selective amplification:

[0040]

[0041]

[0042] in, Indicates the channel amplification factor; Indicates channel difference enhancement feature; represents channel-wise global average pooling; represents a fully connected layer; Represents the activation function.

[0043] right Attention to spatial orientation guides responses to texture details:

[0044]

[0045]

[0046] in: represents the spatial magnification factor; Indicates spatial difference enhancement features; represents pixel-wise average pooling; represents convolution; Represents the activation function.

[0047] Step 4: Enhance the channel difference features and spatial enhancement results Coupling, simulate the nonlinear fusion output after dual-path modulation in the differential amplifier circuit:

[0048]

[0049] in, Represents the output infrared feature after differential enhancement.

[0050] The (3) construction structure guide enhancement module:

[0051] Step 1: Concatenate the multi-level visible light features output by the progressively guided aggregation module and use multi-level attention:

[0052]

[0053]

[0054] in, represents channel attention; represents pixel attention; Indicates enhanced spatial attention.

[0055] Step 2: The dual-branch physical perception module outputs the final atmospheric light map , transmittance graph and restored image ,based on Design structure guidance enhancement module to obtain infrared information guidance weight :

[0056]

[0057]

[0058] in, represents the structure-guided enhancement module; Represents a dual-branch physical perception module.

[0059] Step 3: Use the output infrared features after differential enhancement guide Further restore the enhanced structural features and output the final restored image :

[0060]

[0061] Said (4) designs the loss function:

[0062] Step 1: Construct physical coupling loss based on atmospheric scattering model, and use atmospheric light and transmittance graph Restored image of network output Perform reconstruction and obtain the reconstructed dust map:

[0063]

[0064]

[0065] in: Indicates the presence of dust images; Represents the reconstructed dust map.

[0066] The physical consistency is achieved by minimizing the L1 gap between the reconstructed dust map and the original interference map, achieving no ground truth. Indirect supervision :

[0067]

[0068] in: represents the L1 gap, Represents physical coupling loss.

[0069] Step 2: Construct an enhanced color loss based on the color consistency loss, divide the restored image into patches, calculate the color consistency loss separately, and enhance local color consistency:

[0070]

[0071] in, Indicates a true and clear image; Indicates the Patch The average value of the channel; Indicates shared a patch; Indicates shared channels; Indicates enhanced color loss.

[0072] Step 3: Use L1 loss to calculate the pixel-level difference between the restored image and the true clear image to ensure the overall restoration effect of the final output:

[0073]

[0074] in, Represents pixel intensity loss.

[0075] Step 4: Construct TV regularization loss to measure the difference between adjacent pixels in the image, that is, the total variation of the image, to promote the smoothness of the transmittance map and reduce noise and artifacts:

[0076]

[0077] in, represents the TV regularization loss of the transmittance map.

[0078] Step 5: Total loss for:

[0079]

[0080] in, and denote the weights of physical coupling loss, enhanced color consistency loss, pixel intensity loss, and TV regularization loss, respectively.

[0081] Compared with the prior art, the present invention has the following beneficial effects:

[0082] (1) In order to improve the structural restoration capability and detail preservation of visible light images in complex interference areas, the present invention constructs a progressive guided aggregation module, and gradually embeds a physical-mask coupler consisting of a dual-branch physical perception module and a soft mask modulation unit. This fully captures the non-uniform scattering interference characteristics at different semantic levels, realizes the effective perception and modeling of scattering characteristics at multiple semantic levels, and realizes physical prior guidance and regional adaptive adjustment.

[0083] (2) To address the problems of structural blur and insufficient infrared guidance response in visible light images, a differential amplification attention module was designed to jointly model multimodal differences from the channel and spatial dimensions, effectively amplifying the key structural information of infrared images; a structure-guided enhancement module was constructed to dynamically perceive the non-uniform scattering intensity in combination with the predicted transmittance map, thereby achieving adaptive enhancement or suppression of infrared features in different regions and strengthening the structural restoration capability of images in complex environments; through differential attention and structural adjustment, structural completion in high-interference areas and detail preservation in low-interference areas were achieved.

[0084] (3) In order to improve the physical consistency and interpretability of the network under weak supervision conditions, a physical coupling loss based on the atmospheric scattering model was designed to achieve the coordinated optimization of the transmittance map, atmospheric light intensity map and restored image, ensuring the overall consistency and interpretability of the output restored image and the scattering physical process.

[0085] (5) The present invention comprehensively considers the characteristics of image degradation under the interference of non-uniform scattering media and the structural prior advantages of infrared guidance, and proposes an infrared-guided image restoration method under the interference of non-uniform scattering media. This method effectively combines the structural information of infrared images with the expression ability of visible light texture, combines multimodal complementary information and physical prior knowledge, and addresses the problems of uneven distribution of interference areas and redundancy of image features. It realizes adaptive enhancement and fine structural restoration of visible light images under the interference of non-uniform scattering media; it breaks through the bottleneck of weak adaptability of traditional single-modal dust removal methods to complex environments, and overcomes the problem of rough control of existing guidance methods, effectively improving the image restoration quality under the interference of non-uniform scattering media; it performs well on non-uniform interference data sets, effectively alleviating problems such as structural loss, color deviation and modal redundancy, providing a highly robust solution for image enhancement in complex environments such as dust, sand, and water mist, and providing a new direction for the research of infrared-guided image restoration methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0086] Figure 1Figure 2 shows the image restoration method under the interference of infrared-guided non-uniform scattering media.

[0087] Figure 2 is the original interference graph.

[0088] Figure 3 For a real clear image.

[0089] Figure 4 To restore the image.

[0090] Figure 5 Transmittance diagram.

[0091] Figure 6 Atmospheric light map. DETAILED DESCRIPTION

[0092] The technical solutions in the embodiments of the present invention will be described clearly and completely below with reference to the accompanying drawings.

[0093] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

[0094] Example 1:

[0095] Figure 1 This is a flowchart for the implementation of the method proposed in this invention (Figure 1: Method for Image Restoration Under Interference from Infrared-Guided Non-Uniform Scattering Media). This example is constructed based on a public dataset and uses artificial synthesis to generate representative images of non-uniform scattering medium interference. The interference pattern closely resembles real industrial dust and natural sand dust environments. The constructed dataset contains 4200 pairs of visible light and infrared image samples, with a unified image spatial size of (1024, 768), fully covering non-uniform scattering scenarios with low, medium, and high concentration distributions, providing a good foundation for generalization evaluation. The aforementioned multimodal images were input into the infrared-guided image restoration method proposed in this invention under interference from non-uniform scattering media, and experiments were conducted on the structural restoration and information enhancement of visible light images. Figure 2-Figure 6 The restoration results of this method on typical samples are shown. Figure 2 is the original interference graph, Figure 3 For a true and clear image, Figure 4 To restore the image, Figure 5 and Figure 6The transmittance map and the atmospheric light intensity map are respectively. From the image contrast result, it can be seen that the model can accurately restore the high structure area of road and building edge, and still maintain the texture continuity in the high concentration interference area, the predicted atmospheric light and the transmittance map have good physical consistency. In summary, the experimental results fully verify the non-uniform interference removal capability of the application in the typical scattering environment, and show good physical consistency and perceptual quality, and have the potential for popularization and application in the actual environment perception and industrial intelligent perception scene.

[0096] The above shows and describes the basic principles and main features of the present application and the advantages of the present application. Those skilled in the art should understand that the present application is not limited to the above examples, and the above examples and descriptions in the specification are only to illustrate the principles of the present application. Without departing from the spirit and scope of the present application, various changes and improvements can be made to the present application, and these changes and improvements all fall within the scope of the claimed present application. The scope of protection of the present application is defined by the appended claims and their equivalents.

Claims

1. An infrared-guided image restoration method under interference from non-uniform scattering media, characterized in that: The following steps are involved: (1) Construct a progressive guided aggregation module and gradually embed a physical-mask coupler consisting of a dual-branch physical perception module and a soft mask modulation unit. The dual-branch physical perception module guides the network to perceive the scattering characteristics at different semantic levels, and the soft mask modulation unit realizes adaptive adjustment of the interference area. (2) Design a differential amplification attention module to extract the difference information between infrared and visible light modalities from the two dimensions of channel and space, obtain differentially enhanced infrared features, achieve fine modeling of modal differences, and improve the quality of infrared guidance; (3) Constructing a structure-guided enhancement module, using the multi-scale characteristics of the transmittance map to control the amplification or suppression of modal difference information, fine-tuning the degree of dependence of different interference areas on infrared information, and completing the structure enhancement and information selective fusion under modal guidance; (4) Design the physical coupling loss to achieve the coordinated optimization of the transmittance map, atmospheric light intensity map and restored image, and ensure the overall consistency of the final output image and the scattering physical process.

2. The infrared-guided image restoration method under interference from non-uniform scattering media according to claim 1, characterized in that: The (1) step of constructing a progressive guidance aggregation module is as follows: Step 1: The original interference graph is passed through the first level of the progressive guided aggregation module , input convolution to obtain the initial visible light image features: ； in, represents the original interference graph; Indicates the first-order visible light characteristics; Step 2: Input the visible light features into the next level module step by step. Each level module contains multiple physical-mask couplers: ； ； in, Indicates the progressive boot aggregation module class, =2,3,4; Indicates the Level visible light characteristics; Indicates module cascade; Indicates the Level a physical-mask coupler; Step 3: A dual-branch physical perception module is designed inside each physical-mask coupler based on the atmospheric scattering model to estimate the atmospheric light , transmittance graph and intermediate restoration image , and by Generate a soft mask through the soft mask modulation unit to alleviate the over-repair caused by global consistency processing: ； ； ； in, represents the dual-branch physical perception module; Indicates the Level The visible light characteristics of the output of a physical coupler; represents a soft mask modulation unit; Indicates soft mask.

3. The infrared-guided image restoration method under interference from non-uniform scattering media according to claim 1, characterized in that: The (2) design of the differential amplification attention module is as follows: Step 1: Extract infrared features from the infrared image through a single layer of convolution: ； in, represents the original infrared image; Indicates infrared characteristics; Represents a single layer of convolution; Step 2: Design the differential amplifier module based on the differential amplifier principle: ； in, Indicates differential mode signal; Indicates common mode signal; Indicates differential mode gain, which controls the degree of amplification of the difference; Represents the common mode gain, which controls the residual ratio of redundant information; Step 3: Use infrared characteristics and first-level visible light characteristics to construct differential mode signals: ； right Focus on the channel direction and guide channel selective amplification: ； ； in, Indicates the channel amplification factor; Indicates channel difference enhancement feature; represents channel-wise global average pooling; represents a fully connected layer; represents the activation function; right Attention to spatial orientation guides responses to texture details: ； ； in: represents the spatial magnification factor; Indicates spatial difference enhancement features; represents pixel-wise average pooling; represents convolution; represents the activation function; Step 4: Enhance the channel difference feature and spatial enhancement results Coupling, simulate the nonlinear fusion output after dual-path modulation in the differential amplifier circuit: ； in, Represents the output infrared feature after differential enhancement.

4. The infrared-guided image restoration method under interference from non-uniform scattering media according to claim 1, characterized in that: The (3) construction of the structure-guided enhancement module is as follows: Step 1: Concatenate the multi-level visible light features output by the progressively guided aggregation module and use multi-level attention: ； ； in, represents channel attention; represents pixel attention; indicates enhanced spatial attention; Step 2: The dual-branch physical perception module outputs the final atmospheric light map. , transmittance graph and restored image ,based on Design structure guidance enhancement module to obtain infrared information guidance weight : ； ； in, represents the structure-guided enhancement module; represents the dual-branch physical perception module; Step 3: Use the output infrared features after differential enhancement guide Further restore the enhanced structural features and output the final restored image : 。 5. The infrared-guided image restoration method under interference from non-uniform scattering media according to claim 1, characterized in that: The (4) loss function is designed, and the specific steps are as follows: Step 1: Construct physical coupling loss based on atmospheric scattering model, and use atmospheric light and transmittance graph Restored image of network output Perform reconstruction and obtain the reconstructed dust map: ； ； in: Indicates the presence of dust images; represents the reconstructed dust map; The physical consistency is achieved by minimizing the L1 gap between the reconstructed dust map and the original interference map, achieving no ground truth. Indirect supervision : ； in: represents the L1 gap, represents the physical coupling loss; Step 2: Construct an enhanced color loss based on the color consistency loss, divide the restored image into patches, calculate the color consistency loss separately, and enhance local color consistency: ； in, Indicates a true and clear image; Indicates the Patch The average value of the channel; Indicates shared a patch; Indicates shared channels; Indicates enhanced color loss; Step 3: Use L1 loss to calculate the pixel-level difference between the restored image and the true clear image to ensure the overall restoration effect of the final output: ； in, represents pixel intensity loss; Step 4: Construct TV regularization loss to measure the difference between adjacent pixels in the image, that is, the total variation of the image, to promote the smoothness of the transmittance map and reduce noise and artifacts: ； in, represents the TV regularization loss of the transmittance map; Step 5: Total loss for: ； in, and denote the weights of physical coupling loss, enhanced color consistency loss, pixel intensity loss, and TV regularization loss, respectively.

Images