Atmospheric disturbance removal method based on narrow-band image guidance
Through the dual perturbation encoder network, the perturbation clues were extracted from color and narrowband images, and the perturbation intensity was reduced by narrowband imaging, which solved the lack of performance and robustness of the existing atmospheric perturbation removal method in complex scenarios, achieving efficient and robust Atmospheric disturbances are removed.
Patent Information
- Application Number
- CN202410983626.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-22
- Publication Date
- 2024-11-19
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
The existing single-frame and multi-frame atmospheric disturbance removal methods are insufficient in processing complex disturbance scenarios, poor robustness, and multi-frame methods require inconvenient multi-frame image capture and low computational efficiency.
The perturbation removal network is adopted with a dual perturbation encoder, and the perturbation cues of multi-scale spatial features are extracted from color images and narrowband images, and the feature is enhanced by multi-head self-attention blocks, and the features are merged through the perturbation clue connection mechanism to gradually restore the perturbation-free image. , narrowband imaging is used to reduce the perturbation intensity.
Improve the performance and robustness of atmospheric disturbance removal. Only one color image and one narrowband image can achieve the performance and robustness of the multi-image method, maintain the convenience and efficiency of the single-image method, and reduce data acquisition Difficulty.
Smart Images

Figure CN118982479A_ABST
Abstract
Description
Technical Field
[0001] The invention belongs to the technical field of atmospheric disturbance removal, and in particular relates to an atmospheric disturbance removal method based on narrow-band image guidance. Background Art
[0002] Atmospheric disturbance is a common and annoying physical phenomenon in long-range imaging, which can degrade image quality. Recently, learning-based single-frame disturbance removal (SITR) methods have been proposed to deal with this problem, providing visually plausible results and high efficiency by extracting features from a large amount of training data. However, due to the highly pathological nature of atmospheric disturbances, accurately distinguishing image content from complex disturbance fields using only a single observation remains a very challenging task, and these methods encounter limitations in handling challenging disturbance scenes.
[0003] To alleviate the pathology and improve the overall performance of atmospheric disturbance removal, some methods propose to recover the unperturbed image from multiple consecutive degraded images, exploiting the time-varying physical properties of atmospheric disturbances (e.g., the "lucky frame phenomenon" where clearer frames occasionally appear in the perturbed sequence, and the "zero-mean statistics" of the zero-mean Gaussian distribution that indicates the temporal dependence of the perturbation tilt). By exploiting the temporal diversity from multiple consecutive images, these methods generally show better detail recovery and higher robustness. However, multi-image methods require capturing a series of frames (usually 10, 12, or 16 frames) to obtain enough information about the atmospheric disturbance or encounter the so-called lucky frame. In addition, the motion that may occur in the actual scene will further increase this inconvenience. Therefore, it is very interesting to explore a compromise solution that balances the robustness of the enhancement results and the convenience of the capture effort (hopefully using one or two images).
[0004] The prior art "TurbNet: Single Frame Atmospheric Turbulence Mitigation: A Benchmark Study and A New Physics-Inspired Transformer Model" proposes a new transformer-based single-frame atmospheric disturbance removal model TurbNet, which uses the transformer module to jointly extract the perturbation distortion map and restore the unperturbed image. A self-supervised consistency loss function is also introduced in the network to better constrain the training. Specifically, the TurbNet model is based on the transformer structure and mainly consists of two parts: an encoder and a decoder. The encoder receives the degraded image I d , through a series of converter blocks T E Extract the perturbation distortion map D and the latent feature F; the decoder receives F and D and passes through a series of Transformer blocks T D And up-sample to reconstruct the unperturbed image I rTo better guide the training, the authors of this paper also proposed a self-supervised consistency loss: d and I r All are input into the encoder to obtain the potential feature F d and F r , minimize the L2 distance between the two in the feature space, and the final loss function is the reconstruction loss L rec and consistency loss L con The weighted sum of .
[0005] The main shortcomings of existing single-frame atmospheric disturbance removal (SITR) techniques include: (1) Inadequate performance. Due to the lack of sufficient information in a single-frame image to accurately estimate complex disturbance patterns, the performance of existing SITR methods often drops significantly when dealing with challenging real-world disturbance scenes. Distinguishing image content from complex disturbance fields from only a single observation is an ill-posed problem, which makes it difficult for existing methods to recover clear details. (2) Insufficient robustness. Existing SITR methods are usually trained on specific synthetic datasets, so their performance depends largely on the distribution of training data. When the test scene is different from the distribution of training data, the performance of these methods is often not robust enough and difficult to generalize to various practical applications. (3) Lack of physical priors. Most existing SITR methods do not make good use of the physical properties of atmospheric disturbances, such as wavelength correlation. They often ignore wavelength information and fail to realize that narrowband imaging can significantly reduce the disturbance intensity, a physical property that is beneficial to disturbance removal.
[0006] The existing technology "Physics-Driven Turbulence Image Restoration with Stochastic Refinement" proposes a two-stage atmospheric turbulence image restoration method PiRN-SR, which aims to simultaneously improve the fidelity and perceptual quality of the restored image. The first stage, Physics-integrated Restoration Network (PiRN), integrates a physics-based turbulence simulator into network training to enhance the consistency between image formation and restoration through the re-degradation process. The second stage uses a conditional diffusion model to stochastically refine the output of PiRN to generate natural images with high perceptual quality.
[0007] Although the existing multi-frame atmospheric disturbance removal methods have good performance and robustness, they also have the following shortcomings:
[0008] (1) It is necessary to capture multiple frames of images. The multi-frame atmospheric disturbance removal method requires capturing a series of frames (usually 10 to 16 frames) to obtain sufficient disturbance information, which brings inconvenience to shooting.
[0009] (2) Sensitive to motion. The motion of objects and cameras in real scenes brings additional difficulties to the registration and fusion of multi-frame methods.
[0010] (3) Low computational efficiency. Processing multiple frames of images requires more computing resources and time, and lacks real-time performance. Summary of the invention
[0011] The purpose of the present invention is to provide an atmospheric disturbance removal method based on narrowband image guidance. The present invention overcomes the shortcomings of existing single-frame and multi-frame atmospheric disturbance removal methods by introducing simultaneously captured narrowband images as additional information. While maintaining the convenience of the single-frame atmospheric disturbance removal method, the present invention utilizes the physical properties of narrowband imaging to suppress disturbances to provide reliable guidance information, thereby achieving higher performance and robustness, while also avoiding the inconvenience and inefficiency of the multi-frame atmospheric disturbance removal method. Therefore, the atmospheric disturbance removal method based on narrowband image guidance of the present invention provides a novel and effective solution for practical, efficient, and high-quality atmospheric disturbance removal.
[0012] The technical solution adopted by the present invention to solve the technical problem is as follows:
[0013] The present invention provides a method for removing atmospheric disturbances based on narrowband image guidance, which mainly includes the following steps:
[0014] A disturbance removal network with a dual perturbation encoder is designed to extract perturbation cues of multi-scale spatial features from color image and narrowband image pairs. With the assistance of the perturbation cues, the real color image and narrowband image pairs are reversed into perturbation inputs respectively, the features of the color image and narrowband image are spliced together, and a multi-head self-attention block is used to enhance the feature representation while compressing the spatial dimension. The perturbation cues from the dual perturbation encoders are then merged with the skip connections at each perturbation encoder level through the connection of the perturbation cues to enhance the feature mixing performance of the upsampling stage, and the image is gradually restored to its original resolution. The details of the specific wavelength are combined with the spectral data by processing the features output by the upsampling stage to produce a non-perturbation image.
[0015] Furthermore, the disturbance removal network includes: a disturbance clue extraction module, a re-disturbance module and a guided disturbance removal module, and the guided disturbance removal module includes: two multi-scale transformer modules and a disturbance clue connection module; the multi-scale transformer module is used to reconstruct a clear image under the guidance of the extracted disturbance clues, and the multi-scale transformer module uses a multi-head self-attention block to enhance feature representation while compressing spatial dimensions.
[0016] Furthermore, the weights of the two perturbation encoders for the color image and the narrowband image are shared.
[0017] Furthermore, the core of the guided disturbance removal module is the disturbance cue connection mechanism, which provides effective feature communication between the multi-scale transformer module and the disturbance cues. The disturbance cue connection mechanism ensures that the guided disturbance removal module effectively integrates the disturbance information from two different types of image sources, providing a more robust and comprehensive disturbance cue representation.
[0018] Furthermore, the guided perturbation removal module uses a convolutional block attention module to jointly process the differences between the extracted perturbation cues and perform perturbation inference.
[0019] Furthermore, the perturbation inference is performed at multiple scales and the perturbation cues are gradually upsampled to match the latent space of skip connections.
[0020] Furthermore, the guided perturbation removal module uses convolutional layers to obtain better feature alignment, and then adopts double convolutional layers to optimize the merged perturbation cues to provide better feature alignment before integrating them with the skip connections of the guided perturbation removal module.
[0021] Furthermore, synthetic training data is used to train the disturbance removal network. The specific training process is as follows:
[0022] 20,000 images are selected from the Places dataset for training and 10,000 images for testing. Each image is first cropped to 512×512 pixels. The Zernike-based fast atmospheric disturbance simulator P2S is used to generate atmospheric disturbance image pairs. The network loss function includes consistency loss and reconstruction loss, expressed as L = L consist +L recon , L consist is the consistency loss, ensuring that meaningful perturbation clues are extracted, L recon is the reconstruction loss, responsible for generating clear and undisturbed images; the guided disturbance removal module adopts MHSA blocks, each with three attention heads; all parameters in the network are jointly trained for 300K iterations with a batch size of 16; the training is performed on images of size 224×224, and horizontal flipping and vertical cropping are used for data augmentation; the Adam optimizer is used, with hyperparameters β1=0.9, β2=0.999; a cosine annealing learning rate scheduler is used to reduce the learning rate from the initial value of 5e-5 to 1e-6 during the entire training process.
[0023] The beneficial effects of the present invention are:
[0024] (a) By introducing a narrowband image taken at the same time as a guide, the present invention can significantly improve the performance of atmospheric disturbance removal, making the restored image details richer and closer to the real scene. Compared with the disturbance removal method of a single degraded color image, the method of the present invention has higher quality.
[0025] (b) The present invention utilizes the characteristic that narrowband imaging can significantly reduce the disturbance intensity of the image plane, and designs a two-step fusion strategy for color images and narrowband images to better perceive the disturbance field and guide the removal of disturbances. This is better than the strategy of directly splicing the color image and the narrowband image into the existing method.
[0026] (c) The present invention only requires one degraded color image and one narrow-band guide image to achieve or even exceed the performance and robustness of the multi-image disturbance removal method, while maintaining the convenience and efficiency of the single-image method and reducing the difficulty of data acquisition.
[0027] (d) The present invention specifically designs the NB-GTR network and jointly optimizes the disturbance extraction and guided disturbance removal modules to make the entire process more coordinated and efficient. BRIEF DESCRIPTION OF THE DRAWINGS
[0028] Figure 1 The present invention is a flowchart of a method for removing atmospheric disturbances based on narrow-band image guidance.
[0029] Figure 2 This is a structural diagram of a disturbance removal network (NB-GTR network) specifically for color images (RGB images) and narrowband image pairs of the present invention.
[0030] Figure 3 The present invention is a specific embodiment of an atmospheric disturbance removal method based on narrow-band image guidance. DETAILED DESCRIPTION
[0031] Although the disturbance intensity is weaker at long wavelengths, existing atmospheric disturbance removal techniques often ignore the influence of wavelength. Due to the cumulative effect on the camera filter passband, the difference in disturbance intensity observed on the RGB channels is negligible. However, the present invention, through theoretical deduction and experimental verification, found that narrowband imaging can significantly suppress the observed disturbance intensity, thereby effectively assisting disturbance removal. Since the color image and the narrowband image are captured simultaneously, the pattern of random disturbances differs only in scale. Therefore, the present invention further proposes an RGB and narrowband hybrid strategy to learn intrinsic disturbance cues and assist in disturbance removal accordingly. Figure 1 As shown, a color image is formed by incident light, a narrow-band image is obtained by a narrow-band filter, and further a method for removing atmospheric disturbances based on narrow-band image guidance of the present invention is used to remove single-image disturbances from the color image, and the color image and the narrow-band image are used to perform joint disturbance removal under guidance.
[0032] The present invention discloses an atmospheric disturbance removal method based on narrowband image guidance, which simultaneously shoots a color image and a narrowband image, utilizes the characteristic of narrowband imaging to reduce disturbance intensity, introduces the narrowband image into the disturbance removal process through a two-step fusion strategy, and guides the restoration of a high-quality disturbance-free image.
[0033] The present invention provides a method for removing atmospheric disturbances based on narrowband image guidance, including the steps of registration of narrowband images and color images, feature extraction and fusion, etc., using color and narrowband images to jointly extract disturbance clues, and effectively removing disturbances based on inherent relationships to highlight the effect of narrowband imaging in reducing disturbance intensity, and reducing the cumulative effect through narrowband filtering, providing advantages for disturbance removal, so as to better perceive the disturbance field and guide disturbance removal. The specific operation process is as follows:
[0034] (1) Network structure design;
[0035] The present invention proposes a disturbance removal network (NB-GTR network) specifically for color images (RGB images) and narrowband image pairs, such as Figure 2 As shown in the figure, the NB-GTR network mainly includes: a disturbance clue extraction module, a re-disturbance module and a guided disturbance removal module. The disturbance clue extraction module, the re-disturbance module and the guided disturbance removal module are jointly optimized and work together. Among them, the guided disturbance removal module includes: two multi-scale transformer modules and a disturbance clue connection module. In addition, the disturbance clue extraction module and the re-disturbance module are both implemented using convolutional neural networks. The disturbance clue extraction module reliably extracts disturbance clues from the color image and the narrowband image by analyzing the intrinsic connection between them; the guided disturbance removal module uses the extracted disturbance information to guide the removal of disturbances in the color image. By optimizing the extraction and removal modules, the disturbance removal performance and robustness are improved.
[0036] The NB-GTR network adopts a dual perturbation encoder, which can overcome the modal differences in color and narrowband image pairs and extract unique but interrelated perturbation cues at the same time. The perturbation cues extracted by the perturbation clue extraction module interact with the re-perturbation module and the guided perturbation removal module through a specially designed perturbation clue connection mechanism, which enhances the effect of atmospheric perturbation extraction and achieves more accurate atmospheric perturbation removal.
[0037] (2) Perturbation cue extraction and re-perturbation module;
[0038] Although the scale relation is conceptually simple, it is challenging to apply it directly in image space due to spatially varying perturbations. Therefore, this paper adopts an improved lightweight U-Net structure as a perturbation clue extraction module to describe the implementation of perturbation inference. Figure 2As shown in the figure, the perturbation clue extraction module is used to extract the perturbation clue ε of the multi-scale spatial features from the input color image and narrowband image pair (I1, I2). For the perturbation clue ε of the extracted multi-scale spatial features, the weights of the two perturbation encoders of the color image and the narrowband image are shared. Then the re-perturbation module is introduced. With the assistance of the previously extracted perturbation clues, the re-perturbation module reverses the real color image and narrowband image pair (I1, I2) into perturbation input This ensures that the disturbance clues Robust extraction of . For parameter efficiency, the parameters in the re-perturbation module are shared when processing color images or narrowband images. The re-perturbation module integrates the early separated perturbation cues through spatial adaptive modulation to provide information and guidance for the image reconstruction process. The method of integrating the early separated perturbation cues through spatial adaptive modulation by the re-perturbation module plays an important role in ensuring that the dual perturbation encoder fully captures the perturbation cues.
[0039] (3) guided disturbance removal;
[0040] By integrating the extracted perturbation cues with the input image pair (I1, I2), more robust image reconstruction results can be achieved. Considering the complexity of atmospheric perturbations, a multi-scale transformer module is introduced in the guided disturbance removal module to reconstruct a clear image under the guidance of the previously extracted perturbation cues, and the ability of the multi-scale transformer module to handle spatial variations in atmospheric perturbations is utilized. First, the features of the color image and the narrowband image are spliced together and integrated using overlapping blocks, and then the multi-scale transformer module is used to enhance the feature representation using multi-head self-attention blocks (MHSA) at different encoder levels, while compressing the spatial dimension, which is crucial for effective perturbation removal. In addition, a key issue in the guided perturbation removal process is the perturbation cues. The connection through the perturbation clues can effectively interact and guide the removal of atmospheric perturbations. The connection through the perturbation clues merges the perturbation clues from the dual perturbation encoders with the skip connection of each perturbation encoder level, which enhances the features of the upsampling stage. Mixed performance. The NB-GTR network gradually restores the input image to its original resolution. The multi-scale transformer module processes the features output by the upsampling stage. The final convolutional layer combines the wavelength-specific details with the spectral data to produce an unperturbed image.
[0041] (4) Connection of perturbation cues
[0042] The core of the guided perturbation removal module is the perturbation cue connection mechanism, which provides efficient feature communication between the multi-scale transformer module and the perturbation cues. The features of the perturbation cues can be inferred by comparing the differences between two different observations. Although analytically infeasible in the image field, the perturbation inference can be performed efficiently by jointly processing the differences between the previously extracted perturbation cues using the convolutional block attention module (CBAM) mechanism. This perturbation cue connection mechanism ensures that the guided perturbation removal module can effectively integrate the perturbation information from two different types of image sources, providing a more robust and comprehensive perturbation cue representation. It is important to note that the perturbation inference can be done at multiple scales and the perturbation cues are gradually upsampled to match the latent space of the skip connection. Convolutional layers are then applied to obtain better feature alignment, and double convolutional layers are used to optimize the merged perturbation cues, providing better feature alignment before integrating them with the skip connection of the guided perturbation removal module. The final output of this guided perturbation removal module is a mixture of the rescaled encoder features and the latent perturbation features extracted by the transformer encoder, optimizing the input to the subsequent processing stages in the network.
[0043] Overall, the NB-GTR network aims to fully exploit the spectral properties of the dual image capture method to enhance the perturbation removal process in terms of accuracy and efficiency. This is achieved by optimizing the extraction of perturbation cues and the removal of perturbations, enabling the NB-GTR network to effectively identify and use these features to improve perturbation removal.
[0044] The present invention uses a combination of synthetic training data and real data to train the NB-GTR network, so that the network can cope with various disturbance scenarios and improve the generalization ability of the model. The specific training process is as follows:
[0045] (1) Synthetic training data;
[0046] The present invention selects 20,000 images from the Places dataset for training and 10,000 images for testing. Each image is first cropped to a size of 512×512 pixels. The present invention uses a fast atmospheric disturbance simulator P2S (phase-to-space transform) based on Zernike to generate atmospheric disturbance image pairs. In order to simulate the atmospheric disturbance image pairs captured simultaneously by RGB and narrowband filters, the present invention needs to fix the random field of the generated image and then rescale it using different Zernike coefficients in P2S.
[0047] Specifically, the present invention sets the wavelength of the color image to λ RGB = 550nm (the center wavelength of the visible spectrum), set the wavelength of the narrowband filter to λ N=680nm (a smaller total variance can be obtained at this value if sufficient incident light is ensured). The scaling factor is calculated from the selected wavelength based on the experimentally determined relationship between the Zernike coefficient and the wavelength. After the perturbation simulation, the present invention reweights the RGB channels of the narrowband image by assigning coefficients of 0.95, 0.04, and 0.01. This reweighting process is intended to make the simulation results consistent with the average distribution determined by the present invention through actual narrowband photography. Finally, the present invention introduces noise into the image by first applying a Gaussian noise randomly selected with a standard deviation between 0.015 and 0.025 to the narrowband image and a standard deviation between 0.005 and 0.015 to the color image to reflect the different noise intensities observed in actual measurements, and then applying a similarly scaled quantization noise.
[0048] (2) Training of NB-GTR network;
[0049] The loss function of the NB-GTR network includes consistency loss and reconstruction loss, which can be expressed as L =
[0050] L consist +L recon Among them, L consist is the consistency loss, ensuring that meaningful perturbation clues are extracted; L recon is the reconstruction loss, which is responsible for generating clear and undisturbed images. Specifically, the consistency loss is calculated by comparing the re-perturbation results With disturbance input I i To calculate, where the subscript i specifies a color image or a narrowband image, the formula is Where L1 represents L1 loss, α i is the weight coefficient, which is set to 0.4 in the experiments of the present invention. In addition, the reconstruction loss L recon Defined as the L1 loss between the output J of the guided disturbance removal module and the true value J, it can be written as
[0051] The present invention implements the method of the present invention on AMD EPYC 7532CPU and 4 NVIDIA3090 GPUs using PyTorch. The guided disturbance removal module uses MHSA blocks, each with three attention heads. From top to bottom, the number of blocks in each cascade layer is arranged as 2, 3, 3, and 4, respectively. All parameters in the network are jointly trained for 300K iterations with a batch size of 16. Training is performed on images of size 224×224, and data augmentation is performed using horizontal flipping and vertical cropping. For optimization, the Adam optimizer is used with hyperparameters β1=0.9, β2=0.999. A cosine annealing learning rate scheduler is used to reduce the learning rate from an initial value of 5e-5 to 1e-6 throughout the training process.
[0052] A specific embodiment of the atmospheric disturbance removal method based on narrow-band image guidance of the present invention has the following specific operation process:
[0053] like Figure 3 As shown in the figure, a dual-camera coaxial imaging system is first built. The system consists of two Sony α7RIII cameras, both of which are Sony FE 70-200mm f / 2.8GMII lenses. A narrowband filter with a central wavelength of 680nm and a bandwidth of 40nm is installed on the lens of one of the cameras. The two cameras are placed in parallel with a small distance, which is much smaller than the distance of the photographed object, and can be approximately considered as coaxial imaging. This narrowband imaging method can significantly reduce the impact of atmospheric disturbances. Then data acquisition is carried out. The built dual-camera coaxial imaging system is used to shoot different scenes in a simulated disturbance environment, and color images (RGB images) and narrowband images (grayscale images) in the 680nm band are obtained simultaneously. Color images contain rich color and texture detail information, but are easily affected by atmospheric disturbances; narrowband images are less affected by disturbances, but lack color information. Finally, a deep learning algorithm is used for image fusion, and the color image and the corresponding narrowband image are input into the pre-trained NB-GTR network to obtain a de-perturbed image. The de-perturbed image is a fused image that inherits the low perturbation intensity of the narrowband image and has the color information of the color image.
[0054] In terms of network structure design, the present invention can replace the existing convolutional neural network with other similar network structures, such as using residual network (ResNet), densely connected convolutional network (DenseNet), etc. as the backbone network to extract features.
[0055] When fusing narrowband image information, the present invention is not limited to a two-step fusion strategy, and other feature fusion methods such as an attention mechanism and a gating mechanism can also be tried.
[0056] In the aspect of narrowband imaging, in addition to using narrowband filters of specific wavelengths, the present invention may also consider using other imaging methods that can reduce the influence of disturbances, such as polarization imaging, multispectral imaging, etc. These imaging methods can also provide additional guidance information to assist in disturbance removal.
[0057] The present invention not only enjoys the convenience and efficiency of single-image disturbance removal methods, but also achieves comparable or higher performance and robustness compared to multi-image methods. Experimental results show that the present invention achieves state-of-the-art performance in both synthetic and real-world scenarios.
[0058] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.
Claims
1. A method for removing atmospheric disturbances based on narrowband image guidance, characterized in that: The following steps are involved: A disturbance removal network with a dual perturbation encoder is designed to extract perturbation cues of multi-scale spatial features from color image and narrowband image pairs. With the assistance of the perturbation cues, the real color image and narrowband image pairs are reversed into perturbation inputs respectively, the features of the color image and narrowband image are spliced together, and a multi-head self-attention block is used to enhance the feature representation while compressing the spatial dimension. The perturbation cues from the dual perturbation encoders are then merged with the skip connections at each perturbation encoder level through the connection of the perturbation cues to enhance the feature mixing performance of the upsampling stage, and the image is gradually restored to its original resolution. The details of the specific wavelength are combined with the spectral data by processing the features output by the upsampling stage to produce a non-perturbation image.
2. The method for removing atmospheric disturbances based on narrowband image guidance according to claim 1, characterized in that: The disturbance removal network includes: a disturbance clue extraction module, a re-disturbance module and a guided disturbance removal module, and the guided disturbance removal module includes: two multi-scale transformer modules and a disturbance clue connection module; the multi-scale transformer module is used to reconstruct a clear image under the guidance of the extracted disturbance clues, and the multi-scale transformer module uses a multi-head self-attention block to enhance feature representation while compressing spatial dimensions.
3. The method for removing atmospheric disturbances based on narrowband image guidance according to claim 1, characterized in that: The weights of the two perturbation encoders for the color image and the narrowband image are shared.
4. The method for removing atmospheric disturbances based on narrowband image guidance according to claim 1, characterized in that: The core of the guided disturbance removal module is the disturbance cue connection mechanism, which provides effective feature communication between the multi-scale transformer module and the disturbance cues. The disturbance cue connection mechanism ensures that the guided disturbance removal module effectively integrates the disturbance information from two different types of image sources and provides a more robust and comprehensive disturbance cue representation.
5. The method for removing atmospheric disturbances based on narrowband image guidance according to claim 1, characterized in that: The guided perturbation removal module uses a convolutional block attention module to jointly process the differences between the extracted perturbation cues and perform perturbation inference.
6. The method for removing atmospheric disturbances based on narrowband image guidance according to claim 5, characterized in that: The perturbation inference is done at multiple scales, and the perturbation cues are gradually upsampled to match the latent space of skip connections.
7. The method for removing atmospheric disturbances based on narrowband image guidance according to claim 1, characterized in that: The guided perturbation removal module uses convolutional layers to obtain better feature alignment and then employs dual convolutional layers to optimize the merged perturbation cues to provide better feature alignment before integrating them with the skip connections of the guided perturbation removal module.
8. The method for removing atmospheric disturbances based on narrowband image guidance according to claim 1, characterized in that: The disturbance removal network is trained using synthetic training data. The specific training process is as follows: 20,000 images are selected from the Places dataset for training and 10,000 images for testing. Each image is first cropped to 512×512 pixels. The Zernike-based fast atmospheric disturbance simulator P2S is used to generate atmospheric disturbance image pairs. The network loss function includes consistency loss and reconstruction loss, expressed as L = L consist +L recon , L consist is the consistency loss, ensuring that meaningful perturbation clues are extracted, L recon is the reconstruction loss, responsible for generating clear and undisturbed images; the guided disturbance removal module adopts MHSA blocks, each with three attention heads; all parameters in the network are jointly trained for 300K iterations with a batch size of 16; the training is performed on images of size 224×224, and horizontal flipping and vertical cropping are used for data augmentation; the Adam optimizer is used, with hyperparameters β1=0.9, β2=0.999; a cosine annealing learning rate scheduler is used to reduce the learning rate from the initial value of 5e-5 to 1e-6 during the entire training process.
Citation Information
Patent Citations
Multi-stage progressive underwater image enhancement method
CN114445292A
Cited By
Polarization restoration imaging method based on conditional diffusion model
CN119722518A
