Image deraining method and device

Abstract

The present invention discloses an image deraining method and apparatus, comprising: obtaining a rainy image and a corresponding infrared image; inputting the rainy image and the infrared image into a preset deraining network, extracting image features through the deraining network, encoding and decoding the image features, predicting rain streak negative residual information based on the decoded image features, and generating a derained image and a derained infrared image based on the rain streak negative residual information; inputting the derained image and the derained infrared image into a preset fusion network, generating a fused image based on the derained image and the derained infrared image through the fusion network, estimating brightness information on the fused image, generating an illumination-enhanced derained image based on the brightness estimation result, calibrating the derained infrared image, and iteratively fusing the brightness estimation result of the fused image with the calibrated derained infrared image to generate an illumination-enhanced derained image that maximizes pixel information. This method can simultaneously remove rain lines and improve the effects of low light.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to an image rain removal method and device. Background Art

[0002] In images captured in rainy conditions, rain streaks obscure important information, reducing image clarity and quality. This reduces visibility and impacts the performance of many computer vision tasks, including object detection, tracking, autonomous driving, and video surveillance. Image deraining aims to analyze and process rainy images, remove interfering information, restore a clean, clear background image, and achieve high-quality image reconstruction.

[0003] Similarly, image quality and the information contained in low-light scenes are often affected by the darkness of the scene. Image information in rainy and low-light scenes is even more severely damaged, with clarity and visibility drastically reduced. In these scenarios, the acquired images and information are insufficient to meet the requirements of some intelligent application systems, significantly limiting their application. In the field of autonomous driving, particularly, harsh driving scenarios in rainy and low-light conditions present a major challenge. Rain obstruction and the reduced visual perception capabilities of RGB sensors in low-light scenarios make it difficult to capture and discern the vehicle's surroundings. Therefore, removing rain streaks, restoring image texture details, and enhancing nighttime visual perception are urgent issues that need to be addressed.

[0004] Methods that utilize multi-source image fusion can provide a more comprehensive representation of the environment, improving nighttime visual perception. However, existing image fusion methods, limited by lighting conditions and generalization capabilities, cannot adequately address the contrast and spectral impairments of the fused image. Therefore, there is an urgent need for a rain removal method tailored to multi-source data and low-light scenarios. This method can improve and enhance image quality in rainy and low-light conditions, enhancing clarity, contrast, and visibility, and thus broadening its application.

[0005] The problem of rain removal from single images has become a research hotspot in recent years, garnering widespread attention. Early methods primarily relied on prior knowledge, imposing certain priors based on the statistical characteristics of rain streaks and the clear image background to achieve rain removal. For example, Kang et al.'s 2011 study, "Automatic single-image-based rain streaks removal via image decomposition," proposed the first automatic image decomposition framework based on MCA, transforming the rain removal problem into an image decomposition problem based on morphological component analysis (MCA). However, due to the complexity and variability of rainy scenes, these handcrafted priors based on empirical statistical results were unable to effectively model the clean background and rain streaks in rain images, significantly limiting their rain removal performance.

[0006] In recent years, with the continuous development of deep learning technology, deep learning-based methods have been widely applied to image deraining tasks, significantly improving deraining performance. For example, Fu et al.'s 2017 work, "Removing rain from single images via a deep detail network," and Qin et al.'s 2021 work, "An efficient transformer deraining model," both utilize deep learning methods to remove rain streaks and reconstruct clean images. However, both the introduction of convolutional neural networks and the development of transformers in image processing have their own limitations and bottlenecks in image deraining. Convolutional neural networks (CNNs), due to their limited local receptive field, are hampered in processing long-range information dependencies, encountering a performance bottleneck in deraining. While Transformer-based methods can better model non-local information and achieve higher performance than CNN-based methods when applied to image deraining, they typically require large amounts of training data and additional supervision. They also lack the local invariance characteristic of CNNs, failing to model local feature information and thus easily confusing background details and rain streaks located in local regions.

[0007] Therefore, recent research has begun to explore combining CNNs and Transformers for image deraining tasks to better model local feature information and non-local long-range dependencies, thereby improving the performance of single-image deraining. Examples include Yuan et al.'s 2021 work "Incorporating convolution designs into visual transformers" and Jiang et al.'s 2022 work "Image deraining meets association learning and transformers."

[0008] While current methods for single-image rain removal are constantly being updated and their performance is improving, rain streak removal and illumination enhancement in specific scenarios such as rainy and low-light nighttime driving remain extremely challenging. In rainy and low-light scenarios, the obstruction of rain in images and the reduced visual perception of RGB sensors in low light dramatically degrade the quality of images captured in these conditions, impacting the performance of outdoor imaging systems and related computer vision systems. In the field of autonomous driving, these images struggle to meet the minimum requirements of intelligent systems for capturing, identifying, and determining information surrounding the vehicle. Existing image rain removal methods fail to simultaneously remove rain streaks and mitigate blur and information loss in low-light images. Therefore, there is an urgent need for an image rain removal method and system for multi-source data and low-light scenarios that can effectively remove rain while simultaneously integrating information across viewpoints, further enhancing image illumination intensity, and supplementing image detail information to produce higher-quality, clean, rain-free images. Summary of the Invention

[0009] In order to overcome the problems existing in the related art, the present invention provides an image deraining method and apparatus to solve the defects in the related art.

[0010] According to a first aspect of the present invention, a method for removing rain from an image is provided, the method comprising:

[0011] Obtain rainy images and corresponding infrared images;

[0012] Inputting the rainy image and the infrared image into a preset rain removal network respectively to obtain a rain-removed image and a rain-removed infrared image; the rain removal network is used to extract image features, encode and decode the image features, predict rain streak negative residual information based on the decoded image features, and generate the rain-removed image and the rain-removed infrared image based on the rain streak negative residual information;

[0013] The derained image and the derained infrared image are input into a preset fusion network to obtain an illumination-enhanced derained image; the fusion network is used to generate a fused image based on the derained image and the derained infrared image, perform brightness information estimation on the fused image, generate an illumination-enhanced derained image based on the brightness information estimation result, calibrate the derained infrared image, and iteratively fuse the brightness information estimation result of the fused image with the calibrated derained infrared image to generate an illumination-enhanced derained image that maximizes pixel information.

[0014] Preferably, the rain removal network includes an attention layer group module and an encoding and decoding module;

[0015] The attention layer group module includes several attention layers and several operation layers, which are used to extract image features and predict rain stripe negative residual information based on the decoded image features;

[0016] The encoding and decoding module includes several encoders and several decoders, which are used to encode and decode image features.

[0017] Preferably, the attention layer group module is specifically used to:

[0018] Calculating an attention weight by the attention layer according to a first feature of the data input into the attention layer group module;

[0019] Normalizing the attention weights;

[0020] The operation layer changes and operates the image input to the attention layer group module according to the attention weight, thereby generating a second feature representation of the data input to the attention layer group module.

[0021] Preferably, the encoding and decoding module is specifically used for:

[0022] The feature map input to the encoding and decoding module is encoded by a multi-layer encoder with progressive downsampling, wherein the encoder is composed of a plurality of Transformer Blocks; wherein, after each Transformer Block, the feature map is downsampled by a corresponding downsampling module to reduce the spatial size of the feature map;

[0023] The encoded feature map is decoded by a multi-layer decoder with progressive upsampling that is symmetrical to the encoder structure; wherein the decoder is composed of a number of Transformer Blocks equal to the number of the encoder; wherein after each Transformer Block, the feature map is upsampled by a corresponding upsampling module to increase the spatial size of the feature map and reduce the number of channels of the feature map;

[0024] The encoded feature map output by each Transformer Block of the encoder and the decoded and upsampled feature map output by the next level Transformer Block of the Transformer Block in the decoder corresponding to the Transformer Block of the encoder are connected to transmit low-dimensional feature information.

[0025] Preferably, the step of inputting the rainy image and the infrared image into a preset rain removal network to obtain a rain-removed image and a rain-removed infrared image specifically includes:

[0026] Converting the rain image and the infrared image into rain image and infrared image represented by low-dimensional features through a convolutional layer;

[0027] Inputting the rainy image and the infrared image represented by the low-dimensional features into the first attention layer group module to extract shallow features of the rainy image and the infrared image;

[0028] Encoding and decoding the shallow features of the rain image and the infrared image through the encoding and decoding module;

[0029] Inputting the decoded rain image and infrared image into the second attention layer group module to further extract deep features of the rain image and infrared image to obtain predicted rain streak negative residual information;

[0030] The rain stripe negative residual information is mapped and output as a feature image through a convolution layer, and the rain stripe negative residual information in the feature image is summed with the rain image and the infrared image to obtain the rain-removed image and the rain-removed infrared image.

[0031] Preferably, the fusion network includes a feature extraction module, an information measurement module, an image fusion module, an illumination estimation module and a content calibration module;

[0032] The feature extraction module is used to extract features of the derained image and the derained infrared image;

[0033] The information measurement module is used to generate information retention of the derained image and the derained infrared image according to the features of the derained image and the derained infrared image;

[0034] The image fusion module is used to fuse the derained image and the derained infrared image according to their information retention, so as to generate a fused image;

[0035] The illumination estimation module is used to estimate brightness information of the fused image;

[0036] The content calibration module is used to calibrate the derained infrared image.

[0037] Preferably, inputting the derained image and the derained infrared image into a preset fusion network to obtain the illumination-enhanced derained image specifically includes:

[0038] Converting the derained image from RGB space to YCrCb space to obtain three component images of the derained image in YCrCb space;

[0039] Inputting the Y component image of the derained image and the derained infrared image into a pre-trained feature extraction module to extract corresponding feature maps; wherein the feature maps include shallow feature maps and deep feature maps;

[0040] Inputting the feature map into an information measurement module, calculating information measurement values ​​of the derained image and the derained infrared image, and calculating information retention of the derained image and the derained infrared image based on the information measurement values;

[0041] inputting the derained image and the derained infrared image into an image fusion module, fusing the derained image and the derained infrared image according to their information retention, to generate a fused image;

[0042] Convert the fused image from YCrCb space to RGB space;

[0043] The fusion image is estimated by using an illumination estimation module for pixel-level parameter brightness information, and the estimated pixel-by-pixel curve parameters are applied to the derained image input to the fusion network to obtain an illumination-enhanced derained image; at the same time, the derained infrared image input to the fusion network is calibrated by using a content calibration module;

[0044] The illumination-enhanced derained image and the calibrated derained infrared image are re-input into the fusion network, and are continuously iteratively fused to finally generate an illumination-enhanced derained image that maximizes pixel information.

[0045] Preferably, the training method of the rain removal network and the fusion network includes:

[0046] Acquire a first image dataset, where the image dataset includes rain-free images without rain, and the rain-free images include visible light images and infrared images;

[0047] Using a raindrop synthesis algorithm on the rain-free image to generate a rainy image;

[0048] constructing a rain removal target image pair based on the rainy image and the rain-free image to generate a second image dataset;

[0049] Constructing a training set and a test set based on the second image dataset;

[0050] The deraining network and the fusion network are trained using the training set, and the trained deraining network and the fusion network are tested using the test set.

[0051] Preferably, the training method of the rain removal network and the fusion network includes:

[0052] AdamW is used as the optimizer for the deraining network, the initial learning rate is set to 3e-4, and the cyclic cosine annealing learning rate adjustment method is used for training, and the learning rate is adjusted to the final ;

[0053] Adam is used as the optimizer for the fusion network, and the initial learning rate is set to , and trained using an exponentially decaying learning rate adjustment method.

[0054] According to a second aspect of the present invention, there is provided an image deraining apparatus, the apparatus comprising:

[0055] An input module is used to obtain rain images and corresponding infrared images;

[0056] a rain removal module, configured to input the rainy image and the infrared image into a preset rain removal network, respectively, to obtain a rain-removed image and a rain-removed infrared image; the rain removal network is configured to extract image features, encode and decode the image features, predict rain streak negative residual information based on the decoded image features, and generate the rain-removed image and the rain-removed infrared image based on the rain streak negative residual information;

[0057] a fusion module configured to input the derained image and the derained infrared image into a preset fusion network to obtain a light-enhanced derained image; the fusion network configured to generate a fused image based on the derained image and the derained infrared image, perform brightness information estimation on the fused image, generate a light-enhanced derained image based on the brightness information estimation result, calibrate the derained infrared image, and iteratively fuse the brightness information estimation result of the fused image with the calibrated derained infrared image to generate a light-enhanced derained image that maximizes pixel information.

[0058] The present invention discloses an image deraining method and device. Through a preset deraining network, it is capable of removing interference information such as rain and fog from data of different modalities, particularly RGB visible light images and their corresponding infrared spectrum images, i.e., rainy images and corresponding infrared images. This method then performs cross-perspective information fusion through a preset fusion network. Furthermore, to enable broader and more practical applications in low-light scenarios, the preset fusion network is capable of fusing information from different perspectives while enhancing the illumination intensity of the fused image, thereby improving information blur and limited visual perception caused by insufficient illumination in low-light scenarios. The image deraining method proposed in the present invention can simultaneously remove rain streaks and improve low-light image blur and information loss. It is suitable for specific scenarios such as nighttime driving in rainy and low-light conditions, enhancing its effectiveness in practical applications and broadening its scope of application.

[0059] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 The figure is a flowchart of an image rain removal method according to an embodiment of the present invention.

[0061] Figure 2 FIG. 4 is a structural diagram of an image rain removal system according to an embodiment of the present invention.

[0062] Figure 3 FIG. 4 is a structural diagram of a rain removal network according to an embodiment of the present invention.

[0063] Figure 4 It is a structural diagram of an attention layer group module according to one embodiment of the present invention.

[0064] Figure 5 FIG. 4 is a structural diagram of a Transformer Block according to an embodiment of the present invention.

[0065] Figure 6 This is a structural diagram of a converged network according to an embodiment of the present invention.

[0066] Figure 7 FIG. 4 is a schematic diagram showing test results of an image rain removal method according to an embodiment of the present invention.

[0067] Figure 8 FIG. 4 is a schematic diagram illustrating test results of another image rain removal method according to an embodiment of the present invention.

[0068] Figure 9 FIG. 4 is a schematic structural diagram of an image rain removal device according to an embodiment of the present invention.

[0069] Figure 10 The figure is a schematic diagram showing the structure of a computing device hardware according to an embodiment of the present invention. DETAILED DESCRIPTION

[0070] Exemplary embodiments will be described in detail herein, examples of which are illustrated in the accompanying drawings. In the following description, when referring to the drawings, like numbers in different figures represent like or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all possible embodiments consistent with the present invention. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present invention, as detailed in the appended claims.

[0071] The terms used in this invention are for the purpose of describing specific embodiments only and are not intended to limit the invention. The singular forms "a," "the," and "the" used in this invention and the appended claims are also intended to include plural forms unless the context clearly indicates otherwise. It should also be understood that the term "and / or" as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0072] It should be understood that although the terms "first," "second," "third," etc. may be used in the present invention to describe various information, such information should not be limited to these terms. These terms are merely used to distinguish information of the same type from one another. For example, first information may also be referred to as second information, and similarly, second information may also be referred to as first information, without departing from the scope of the present invention. Depending on the context, the term "if" as used herein may be interpreted as "when," "when," or "in response to determining."

[0073] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0074] like Figure 1 As shown, Figure 1 FIG. 1 is a flowchart of an image deraining method according to an embodiment of the present invention, comprising the following steps:

[0075] Step S101, obtaining a rain image and a corresponding infrared image;

[0076] Step S102: Inputting the rain image and the infrared image into a preset rain removal network to obtain a rain-removed image and a rain-removed infrared image, respectively. The rain removal network is configured to extract image features, encode and decode the image features, predict rain streak negative residual information based on the decoded image features, and generate the rain-removed image and the rain-removed infrared image based on the rain streak negative residual information.

[0077] Step S103: Input the derained image and the derained infrared image into a preset fusion network to obtain a light-enhanced derained image; the fusion network is used to generate a fused image based on the derained image and the derained infrared image, estimate brightness information on the fused image, generate a light-enhanced derained image based on the brightness information estimation result, calibrate the derained infrared image, and iteratively fuse the brightness information estimation result of the fused image with the calibrated derained infrared image to generate a light-enhanced derained image that maximizes pixel information.

[0078] like Figure 2 As shown, Figure 2 This is a structural diagram of an image deraining system according to one embodiment of the present invention. This image deraining system generates a derained image and a derained infrared image by inputting a rainy image and a corresponding infrared image into a deraining network. These images are then iteratively fused through a fusion network, ultimately producing a derained image with enhanced illumination. Specifically, the deraining network utilizes a layered encoder-decoder architecture and a CNN-Transformer framework to construct a module for removing interference information such as rain. Interference information is removed from both the RGB visible light image data and the corresponding infrared spectral image data, resulting in interference-free multi-source image data for cross-view information fusion. The fusion network employs a zero-reference depth curve estimation method to estimate pixel-level parameters of brightness information for the input low-light image. These pixel-by-pixel curve parameters are iteratively applied to the input image for illumination enhancement. A no-reference loss function is then used to implicitly evaluate the enhancement quality, driving network learning. The method of the present invention simultaneously enhances illumination while fusing information, resulting in higher-quality derained images with greater practical applications. The combination of the deraining network and the fusion network makes the image deraining method more practical and widely used.

[0079] In step S101, the acquired rain image refers to RGB visible light image data captured in foggy and rainy weather, containing interference information such as rain streaks. This image can be a single image or a single frame from a continuous video, and the present invention is not limited thereto. The infrared image, on the other hand, is infrared spectral image data corresponding to the rain image, acquired by a corresponding sensor. The infrared image and its corresponding rain image are image data captured from the same scene and angle.

[0080] In step S102, the rain removal network is a pre-trained network model that can be used to extract image features, encode and decode the image features, predict rain streak negative residual information based on the decoded image features, and generate an image with rain lines removed based on the rain streak negative residual information. By inputting the rainy image and the corresponding infrared image obtained in step S101 into the rain removal network, a series of operations, such as extracting image features and encoding and decoding the image features, can be used to obtain rain streak negative residual information in the rainy image and the infrared image. This rain streak negative residual information is then summed with the rainy image and the infrared image, respectively, to obtain a rain-removed image corresponding to the rainy image and a rain-removed infrared image corresponding to the infrared image.

[0081] Specifically, in some embodiments, the deraining network may include an attention layer group module and a coding and decoding module; wherein the attention layer group module includes several attention layers and several operation layers for extracting image features and predicting the negative residual information of rain stripes based on the decoded image features; and the coding and decoding module includes several encoders and several decoders for encoding and decoding image features.

[0082] Specifically, in some embodiments, the attention layer group module can calculate the attention weight according to the first feature of the data input to the attention layer group module through the attention layer; and normalize the attention weight; and then change and operate the image of the input attention layer group module according to the attention weight through the operation layer to generate a second feature representation of the data input to the attention layer group module.

[0083] Specifically, in some embodiments, the encoding and decoding module can encode the feature map of the input encoding and decoding module through a multi-layer encoder with step-by-step downsampling; wherein the encoder is composed of several Transformer Blocks; wherein, after each Transformer Block, the feature map is downsampled through a corresponding downsampling module to reduce the spatial size of the feature map; and then the encoded feature map is decoded through a multi-layer decoder with step-by-step upsampling that is symmetrical to the encoder structure; wherein the decoder is composed of several Transformer Blocks with the same number as the encoder; wherein, after each Transformer Block, the feature map is upsampled through a corresponding upsampling module to enhance the spatial size of the feature map and reduce the number of channels of the feature map; at the same time, the encoded feature map output by each Transformer Block of the encoder and the decoded and upsampled feature map output by the next level Transformer Block of the Transformer Block in the decoder corresponding to the Transformer Block of the encoder are connected to transmit low-dimensional feature information.

[0084] Specifically, in some embodiments, in step S102, respectively inputting the rain image and the infrared image into a preset deraining network to obtain a derained image and a derained infrared image may include:

[0085] First, a convolutional layer is used to convert the rain image and infrared image into rain images and infrared images represented by low-dimensional features. The rain image and infrared image represented by low-dimensional features are then input into the first attention layer group module to extract the shallow features of the rain image and infrared image. Then, the encoding and decoding module is used to encode and decode the shallow features of the rain image and infrared image. The decoded rain image and infrared image are then input into the second attention layer group module to further extract the deep features of the rain image and infrared image to obtain the predicted rain streak negative residual information. Finally, a convolutional layer is used to map the rain streak negative residual information into a feature image, and the rain-free image and rain-free infrared image are obtained by summing the rain streak negative residual information in the feature map with the rain image and infrared image.

[0086] Specifically, if Figure 3 As shown, Figure 3 This is a structural diagram of a rain removal network according to one embodiment of the present invention, wherein the rain removal network includes two 3x3 convolutional layers, two attention layer group modules (Grouped Attention Layer Module, GALM) and an encoding and decoding module composed of multiple Transformer Blocks and corresponding downsampling modules (DownSampling) and upsampling modules (UpSampling).

[0087] Specifically, each rainy image and the corresponding infrared image obtained in step S101 can be input into the GALM convolution operation layer module and the layered encoder in the deraining network to extract and encode shallow features to obtain rich spatially varying rain distribution features.

[0088] Among them, before the input source image enters the feature extraction backbone, a 3x3 convolutional layer is used to embed overlapping image blocks and convert them into low-dimensional feature representation.

[0089] The low-dimensional features are passed through the GALM module, which is a stack of multiple attention layers and operation layers, to perform initial low-level feature extraction. Figure 4 As shown, Figure 4This is a structural diagram of an attention layer group module according to an embodiment of the present invention, which includes several attention layers and several operation layers. The attention layer calculates attention weights based on the characteristics of the input data to guide the processing of the input data and select important information. The attention weights are then normalized and used in the operation layer. The operation layer transforms and operates the input data according to the attention weights and generates a new feature representation through a 1×1 convolution layer, i.e., the shallow feature. Through the GALM module, we can learn richer data representation and feature expression capabilities.

[0090] Shallow features extracted by the GALM module Input is a multi-layer encoder-decoder architecture based on Transformer Block, that is, the encoding and decoding modules perform feature encoding and decoding in sequence. Among them, the encoder and decoder use jump connections to transfer low-dimensional feature information. Shallow features First, it enters the multi-layer encoder with step-by-step downsampling to encode the features. The encoder consists of multiple Transformer Blocks. Figure 5 As shown, Figure 5 This is a structural diagram of a Transformer Block according to an embodiment of the present invention. Each Transformer Block includes a sparse self-attention mechanism (Top-k Sparse Attention, TKSA) and a mutually fused feedforward neural network (Mutual Deformable Feedforward Network, MDFN) to capture the contextual relationship between features and model nonlinear relationships. After each encoding level, the feature map is downsampled by the corresponding DownSampling module to reduce the spatial size of the feature map. The encoded shallow feature map is subjected to a multi-layer decoder that is symmetrical with the encoder structure and performs feature decoding operations. The feature map is upsampled by the corresponding UpSampling module to enhance the spatial size of the feature map and reduce the number of feature map channels. After the decoder decodes and reconstructs the features, the decoder structure of the Transformer Block decodes the low-level features and restores the details.

[0091] The decoder output then enters a refinement module (GALM) at the end of the model for further refinement. This module predicts the negative residual information of rain streaks in the rainy image and enhances the quality of the reconstructed image. The final feature map is mapped to the final output image through the convolutional layer output. The final reconstruction results are generated by summing the rain streak negative residual information with the rainy image and the infrared image to generate the final de-rained image corresponding to the rainy image and the de-rained infrared image corresponding to the infrared image.

[0092] In step S103, the fusion network is a pre-trained network model that can be used to generate a fused image, estimate brightness information for the fused image, generate a light-enhanced image based on the brightness estimation result, calibrate the infrared spectral image, and iteratively fuse the brightness estimation result of the fused image with the calibrated infrared spectral image to generate a light-enhanced image that maximizes pixel information. By inputting the derained image obtained in step S102 and the corresponding derained infrared image into the deraining network, the light information of the derained image can be enhanced through a series of operations such as image fusion and brightness estimation. Through multiple iterative fusion operations, a light-enhanced derained image that maximizes pixel information is ultimately generated.

[0093] Specifically, in some embodiments, the fusion network may include a feature extraction module, an information measurement module, an image fusion module, an illumination estimation module and a content calibration module; wherein the feature extraction module is used to extract features of the derained image and the derained infrared image; the information measurement module is used to generate information retention of the derained image and the derained infrared image based on the features of the derained image and the derained infrared image; the image fusion module is used to fuse the derained image and the derained infrared image based on the information retention of the derained image and the derained infrared image to generate a fused image; the illumination estimation module is used to estimate brightness information of the fused image; and the content calibration module is used to calibrate the derained infrared image.

[0094] Specifically, in some embodiments, in step S103, the derained image and the derained infrared image are input into a preset fusion network to obtain a derained image with illumination enhancement, which may specifically include:

[0095] First, the derained image is converted from RGB space to YCrCb space to obtain the three component images of the derained image in YCrCb space; the Y component image of the derained image and the derained infrared image are input into the pre-trained feature extraction module to extract the corresponding feature maps; wherein the feature maps include shallow feature maps and deep feature maps; the feature maps are then input into the information measurement module to calculate the information measurement values ​​of the derained image and the derained infrared image, and the information retention of the derained image and the derained infrared image is calculated based on the information measurement values; then the derained image and the derained infrared image are input into the image fusion module, and the information retention of the derained image and the derained infrared image is calculated based on the information retention of the derained image and the derained infrared image. The derained image and the derained infrared image are fused to generate a fused image. The fused image is then converted from YCrCb space to RGB space. The illumination estimation module is then used to estimate the brightness information of the fused image at the pixel level. The estimated pixel-by-pixel curve parameters are applied to the derained image input to the fusion network to obtain an illumination-enhanced derained image. Simultaneously, the derained infrared image input to the fusion network is calibrated using the content calibration module. The illumination-enhanced derained image and the calibrated derained infrared image are then re-input into the fusion network and iteratively fused to generate an illumination-enhanced derained image that maximizes pixel information.

[0096] Specifically, if Figure 6 As shown, Figure 6This is a structural diagram of a fusion network according to an embodiment of the present invention, wherein the fusion network includes a feature extraction module (Feature Extraction), an information measurement module (Information Measurement), an image fusion module (Dense Block), a spatial conversion module (YCbCr2RGB), an illumination estimation module (Illumination Estimation), a content calibration module (Content Calibration), and two loss functions (Loss Function). The feature extraction module can be a VGG16 network structure model, including five layers. The first two layers are composed of two convolutional layers and a maximum pooling layer, and each convolutional layer is followed by a ReLU activation function. The last three layers are composed of three convolutional layers and a maximum pooling layer. Similarly, each convolutional layer is followed by a ReLU activation function. The image fusion module is composed of nine convolutional layers, wherein the first eight convolutional layers are followed by a LeakyReLU activation function, and the last convolutional layer is followed by a tanh activation function. The illumination estimation module includes a convolutional layer, a structure consisting of a convolutional layer, BatchNorm and ReLU activation functions, and a structure consisting of a convolutional layer and a Sigmoid activation function. The content calibration module includes seven structures consisting of convolutional layers, BatchNorm and ReLU activation functions, and a structure consisting of a convolutional layer and a Sigmoid activation function.

[0097] Specifically, the RGB image under weak light and its corresponding IR image after removing the rain line obtained in step S102, namely the rain removal image and the rain removal infrared image, are marked as and .

[0098] First, the RGB image Convert to YCrCb space and get 、 、 . Then [ The corresponding feature map is extracted from the pre-trained VGG16 feature extraction module in the input fusion network: ,…, and ,…, ,in 、 The features in are based on shallow features such as texture and shape details, while higher-level feature maps such as 、 The main feature information of the deep layer, such as content or spatial structure, is retained in the image. The combination of shallow and deep features forms a comprehensive representation of essential information that may be difficult for the human visual perception system to perceive.

[0099] Then, the information measurement of these extracted feature maps is performed to obtain two information measurement values 、 , and further processing is performed to obtain the information retention of multi-source image information in the fusion process: , In order to preserve the information in the source image, two adaptive weights are assigned as information retention, which define the weight of the similarity between the fused image and the source image. The higher the weight, the higher the expected similarity, and the higher the information preservation of the corresponding source image. The calculation formula is:

[0100]

[0101] in represents the feature map extracted by the convolution layer before the kth maximum pooling layer, and n represents The feature map of the nth channel in the channels, ‖·‖ represents the Frobenius norm, is the Laplacian operator. And the information preservation degree is: , .

[0102] [ After information measurement, the two are combined in series and input into the Dense Block of the fusion network to generate a fused image . Convert the fused image from YCrCb space to RGB space to obtain the fused RGB image. Then, the Illumination Estimation module estimates the pixel-level parameters of the brightness information of the fused low-light image, and iteratively applies the pixel-by-pixel curve parameters to the input image to enhance the illumination. Combined with the structural information relative to the brightness domain in the infrared spectrum, the fused image is continuously iteratively applied to approximate the pixel level to maximize the pixel information, thereby obtaining an enhanced image with clearer and richer detail textures. At the same time, the Content Calibration module is continuously used to calibrate the input infrared spectrum image to ensure the contrast and color balance of the image.

[0103] The enhanced image is converted again from RGB to YCrCb space, becoming the visible light input for the next level of iterative fusion. It then interacts with the calibrated infrared spectrum image to generate the next level of illumination-enhanced image. A zero-reference spatial consistency loss function and an illumination smoothness loss function are used to implicitly measure the quality of the enhancement and drive further network learning. Multi-source image data is iteratively fused across viewpoints, allowing the enhanced image from the previous iteration to serve as the visible light input for the next level of fusion. This progressive enhancement further improves detail, contrast, and color balance, continuously enhancing image quality to a clearer, more realistic, and more natural appearance.

[0104] Compared with the existing research on the application of deep learning methods in rain removal tasks, the image rain removal method described in the present invention can take into account the harsh scenes of rainy night driving, where rain and fog interference information and insufficient illumination seriously obstruct the driver's vision, and even seriously affect the role and performance of intelligent working systems in the field of autonomous driving. In order to make the rain removal method more widely and practically applied in low-light scenes, a rain removal method for special scenes is proposed, and a dataset for special scenes is constructed. Specifically, the rain removal network is first trained using the divided training dataset to obtain images of different perspectives with rain strip interference information removed. The images are then input into the fusion network to fuse multi-source data, complement and enhance information, so that the brightness, contrast and quality of the rain-removed images under low-light conditions are improved, thereby improving the effect of the image rain removal method and broadening its application range.

[0105] Furthermore, compared to existing multi-source image fusion methods, this paper considers that existing methods struggle to adequately address contrast and spectral impairments in the fused image due to limitations such as lighting conditions and generalization capabilities. Therefore, a new fusion network is proposed. During the information fusion process, it learns the mapping between the input low-light image and its corresponding enhanced image. This low-light image is then iteratively applied to maximize luminance pixel information, thereby enhancing image brightness and improving structural information and contrast. The resulting RGB image is cleared of interference such as rain streaks and enhanced in illumination, further facilitating the application of rain removal methods.

[0106] Specifically, in some embodiments, the training method of the above-mentioned deraining network and fusion network may include: obtaining a first image data set, which includes rain-free images without rain, and the rain-free images include visible light images and infrared images; then using a raindrop synthesis algorithm on the rain-free images to generate rainy images; then constructing a deraining target image pair based on the rainy images and the rain-free images to generate a second image data set; then constructing a training set and a test set based on the second image data set; then training the deraining network and the fusion network through the training set, and testing the trained deraining network and fusion network through the test set.

[0107] Specifically, the RoadScene dataset, derived from FLIR video, accurately aligns RGB and infrared images and contains a rich variety of scenes. A raindrop synthesis algorithm is then used to artificially add rain to the RoadScene dataset, simulating driving scenes in rainy, low-light environments. This creates the RoadScene-rain dataset, which contains 221 RGB_rain-IR_rain pairs and 221 RGB-IR pairs before raindrops are added, serving as target image pairs for deraining. 80% of the total image pairs are randomly selected as training data, and the remaining 20% ​​as test data, thus constructing the corresponding RoadScene-rain training and test sets.

[0108] Specifically, in some embodiments, the selected first image dataset can be a dataset from another source or a dataset captured independently, as long as it includes RGB images and infrared images. The present invention is not limited to this. Specifically, in some embodiments, the method for obtaining the training set and the test set is not limited to the above method. For example, rather than artificially generating rain images through a raindrop synthesis algorithm, images with rain lines can be directly obtained. The present invention is not limited to this.

[0109] Specifically, in some embodiments, the training method of the deraining network and the fusion network may further include: using AdamW as the optimizer for the deraining network, setting the initial learning rate to 3e-4, and using the cyclic cosine annealing learning rate adjustment method for training, adjusting the learning rate to the final ; Adam is used as the optimizer for the fusion network, and the initial learning rate is set to , and trained using an exponentially decaying learning rate adjustment method.

[0110] Specifically, the detailed parameter model training and testing process of the deraining network and fusion network used in the present invention may include the initialization data set stage, the training network stage and the test set test stage. In the initialization data set stage, 80% of the constructed 221 image pairs are randomly selected as the training set, and the remaining 20% ​​are used as the test set. In the training network stage, the training set is input into the deraining network framework for training. In the present invention, the deraining network uses AdamW as the optimizer, the initial learning rate is set to 3e-4, and the cyclic cosine annealing learning rate adjustment method is used to adjust the learning rate to the final ; The fusion network uses Adam as the optimizer, and the initial learning rate is set to , using an exponentially decaying learning rate adjustment method. After training, the model parameters for each stage are saved. Then, during the test set testing phase, the trained model obtained during the network training phase is used to test the images in the test set. During testing, the images in the test set are processed by the deraining network to produce clear, rain-free images. These images are then fed into the fusion network for cross-view information fusion prediction, outputting a clear fused image that improves texture detail, structural contrast, and illumination intensity. The images generated from the test set after processing using the above model exhibit high quality and superior visual effects.

[0111] Specifically, if Figure 7 and Figure 8 As shown, Figure 7 and Figure 8 Figure 7-1 and Figure 8-1 are test results of two image deraining methods according to an embodiment of the present invention. Figures 7-1 and 8-1 are rainy images, while Figures 7-2 and 8-2 are illumination-enhanced derained images corresponding to the rainy images, generated using the image deraining method described in the embodiment of the present invention. Compared to rainy images, the derained images generated using the image deraining method described in the embodiment of the present invention not only remove interfering information such as rain lines, but also improve image texture detail and structural contrast, mitigate the effects of low light, and are clearer, higher quality, and offer better visual effects.

[0112] Corresponding to the above-mentioned embodiment of the image rain removal method, the present invention also provides an image rain removal device.

[0113] like Figure 9 As shown, Figure 9 An image rain removal device according to an embodiment of the present invention includes the following modules:

[0114] Input module 910, used to obtain rain image and corresponding infrared image;

[0115] a rain removal module 920 configured to input the rainy image and the infrared image into a preset rain removal network to obtain a rain-removed image and a rain-removed infrared image, respectively; the rain removal network is configured to extract image features, encode and decode the image features, predict rain streak negative residual information based on the decoded image features, and generate the rain-removed image and the rain-removed infrared image based on the rain streak negative residual information;

[0116] Fusion module 930 is configured to input the derained image and the derained infrared image into a preset fusion network to obtain a light-enhanced derained image; the fusion network is configured to generate a fused image based on the derained image and the derained infrared image, perform brightness information estimation on the fused image, generate a light-enhanced derained image based on the brightness information estimation result, calibrate the derained infrared image, and iteratively fuse the brightness information estimation result of the fused image with the calibrated derained infrared image to generate a light-enhanced derained image that maximizes pixel information.

[0117] Preferably, the rain removal network includes an attention layer group module and an encoding and decoding module;

[0118] The attention layer group module includes several attention layers and several operation layers, which are used to extract image features and predict rain stripe negative residual information based on the decoded image features;

[0119] The encoding and decoding module includes several encoders and several decoders, which are used to encode and decode image features.

[0120] Preferably, the attention layer group module is specifically used to:

[0121] Calculating an attention weight by the attention layer according to a first feature of the data input into the attention layer group module;

[0122] Normalizing the attention weights;

[0123] The operation layer changes and operates the image input to the attention layer group module according to the attention weight, thereby generating a second feature representation of the data input to the attention layer group module.

[0124] Preferably, the encoding and decoding module is specifically used for:

[0125] The feature map input to the encoding and decoding module is encoded by a multi-layer encoder with progressive downsampling, wherein the encoder is composed of a plurality of Transformer Blocks; wherein, after each Transformer Block, the feature map is downsampled by a corresponding downsampling module to reduce the spatial size of the feature map;

[0126] The encoded feature map is decoded by a multi-layer decoder with progressive upsampling that is symmetrical to the encoder structure; wherein the decoder is composed of a number of Transformer Blocks equal to the number of the encoder; wherein after each Transformer Block, the feature map is upsampled by a corresponding upsampling module to increase the spatial size of the feature map and reduce the number of channels of the feature map;

[0127] The encoded feature map output by each Transformer Block of the encoder and the decoded and upsampled feature map output by the next level Transformer Block of the Transformer Block in the decoder corresponding to the Transformer Block of the encoder are connected to transmit low-dimensional feature information.

[0128] Preferably, the step of inputting the rainy image and the infrared image into a preset rain removal network to obtain a rain-removed image and a rain-removed infrared image specifically includes:

[0129] Converting the rain image and the infrared image into rain image and infrared image represented by low-dimensional features through a convolutional layer;

[0130] Inputting the rainy image and the infrared image represented by the low-dimensional features into the first attention layer group module to extract shallow features of the rainy image and the infrared image;

[0131] Encoding and decoding the shallow features of the rain image and the infrared image through the encoding and decoding module;

[0132] Inputting the decoded rain image and infrared image into the second attention layer group module to further extract deep features of the rain image and infrared image to obtain predicted rain streak negative residual information;

[0133] The rain stripe negative residual information is mapped and output as a feature image through a convolution layer, and the rain stripe negative residual information in the feature image is summed with the rain image and the infrared image to obtain the rain-removed image and the rain-removed infrared image.

[0134] Preferably, the fusion network includes a feature extraction module, an information measurement module, an image fusion module, an illumination estimation module and a content calibration module;

[0135] The feature extraction module is used to extract features of the derained image and the derained infrared image;

[0136] The information measurement module is used to generate information retention of the derained image and the derained infrared image according to the features of the derained image and the derained infrared image;

[0137] The image fusion module is used to fuse the derained image and the derained infrared image according to their information retention, so as to generate a fused image;

[0138] The illumination estimation module is used to estimate brightness information of the fused image;

[0139] The content calibration module is used to calibrate the derained infrared image.

[0140] Preferably, inputting the derained image and the derained infrared image into a preset fusion network to obtain the illumination-enhanced derained image specifically includes:

[0141] Converting the derained image from RGB space to YCrCb space to obtain three component images of the derained image in YCrCb space;

[0142] Inputting the Y component image of the derained image and the derained infrared image into a pre-trained feature extraction module to extract corresponding feature maps; wherein the feature maps include shallow feature maps and deep feature maps;

[0143] Inputting the feature map into an information measurement module, calculating information measurement values ​​of the derained image and the derained infrared image, and calculating information retention of the derained image and the derained infrared image based on the information measurement values;

[0144] inputting the derained image and the derained infrared image into an image fusion module, fusing the derained image and the derained infrared image according to their information retention, to generate a fused image;

[0145] Convert the fused image from YCrCb space to RGB space;

[0146] The fusion image is estimated by using an illumination estimation module for pixel-level parameter brightness information, and the estimated pixel-by-pixel curve parameters are applied to the derained image input to the fusion network to obtain an illumination-enhanced derained image; at the same time, the derained infrared image input to the fusion network is calibrated by using a content calibration module;

[0147] The illumination-enhanced derained image and the calibrated derained infrared image are re-input into the fusion network, and are continuously iteratively fused to finally generate an illumination-enhanced derained image that maximizes pixel information.

[0148] Preferably, the training method of the rain removal network and the fusion network includes:

[0149] Acquire a first image dataset, where the image dataset includes rain-free images without rain, and the rain-free images include visible light images and infrared images;

[0150] Using a raindrop synthesis algorithm on the rain-free image to generate a rainy image;

[0151] constructing a rain removal target image pair based on the rainy image and the rain-free image to generate a second image dataset;

[0152] Constructing a training set and a test set based on the second image dataset;

[0153] The deraining network and the fusion network are trained using the training set, and the trained deraining network and the fusion network are tested using the test set.

[0154] Preferably, the training method of the rain removal network and the fusion network includes:

[0155] AdamW is used as the optimizer for the deraining network, the initial learning rate is set to 3e-4, and the cyclic cosine annealing learning rate adjustment method is used for training, and the learning rate is adjusted to the final ;

[0156] Adam is used as the optimizer for the fusion network, and the initial learning rate is set to , and trained using an exponentially decaying learning rate adjustment method.

[0157] The implementation process of the functions and effects of each module in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.

[0158] For the device embodiment, since it basically corresponds to the method embodiment, the relevant parts can be referred to the partial description of the method embodiment. The device embodiment described above is merely illustrative, wherein the modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules, that is, they may be located in one place, or they may be distributed on multiple network modules. Some or all of the modules can be selected according to actual needs to achieve the purpose of the present invention. Those of ordinary skill in the art can understand and implement it without paying any creative work.

[0159] The present invention also provides a computer device, which at least includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method described in any one of the aforementioned embodiments is implemented.

[0160] Figure 101004 , a more specific hardware structure diagram of a computing device provided by the present invention is shown. The device may include: a processor 1001, a memory 1002, an input / output interface 1003, a communication interface 1004, and a bus 1005. The processor 1001, the memory 1002, the input / output interface 1003, and the communication interface 1004 are communicatively connected to each other within the device via the bus 1005.

[0161] Processor 1001 can be implemented using a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is configured to execute relevant programs to implement the technical solutions provided by the present invention. Processor 1001 may also include a graphics card, such as an Nvidia Titan X graphics card or an 1080Ti graphics card.

[0162] The memory 1002 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage devices, dynamic storage devices, etc. The memory 1002 can store an operating system and other application programs. When the technical solution provided by the present invention is implemented through software or firmware, the relevant program code is stored in the memory 1002 and is called and executed by the processor 1001.

[0163] The input / output interface 1003 is used to connect to an input / output module to implement information input and output. The input / output module can be configured as a component within the device (not shown) or can be externally connected to the device to provide corresponding functions. Input devices may include a keyboard, mouse, touch screen, microphone, various sensors, etc. Output devices may include a display, speaker, vibrator, indicator light, etc.

[0164] The communication interface 1004 is used to connect to a communication module (not shown) to enable communication between the device and other devices. The communication module can communicate via wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, Wi-Fi, Bluetooth, etc.).

[0165] The bus 1005 comprises a path for transmitting information between various components of the device (eg, the processor 1001 , the memory 1002 , the input / output interface 1003 , and the communication interface 1004 ).

[0166] It should be noted that although the above device only shows the processor 1001, the memory 1002, the input / output interface 1003, the communication interface 1004, and the bus 1005, in a specific implementation, the device may also include other components necessary for normal operation. In addition, those skilled in the art will understand that the above device may only include the components necessary to implement the solution of the present invention, and does not necessarily include all the components shown in the figure.

[0167] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which implements the method described in any of the aforementioned embodiments when the program is executed by a processor.

[0168] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can be implemented using any method or technology for information storage. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change RAM (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory computer-readable media such as modulated data signals and carrier waves.

[0169] From the above description of the embodiments, it is clear that those skilled in the art will clearly understand that the present invention can be implemented using software and a necessary general-purpose hardware platform. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium such as ROM / RAM, a magnetic disk, or an optical disk, and includes instructions for enabling a computer device (such as a personal computer, server, or network device) to execute the methods described in various embodiments of the present invention, or portions thereof.

[0170] The systems, devices, modules, or units described in the above embodiments may be implemented by computer chips or entities, or by products having certain functions. A typical implementation device is a computer, which may be in the form of a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email transceiver, game console, tablet computer, wearable device, or any combination of these devices.

[0171] Each embodiment of the present invention is described in a progressive manner. The same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. The device embodiment described above is merely illustrative, wherein the modules described as separate components may or may not be physically separated. When implementing the scheme of the present invention, the functions of each module can be implemented in the same one or more software and / or hardware. It is also possible to select some or all of the modules according to actual needs to achieve the purpose of the scheme of this embodiment. A person of ordinary skill in the art can understand and implement it without paying any creative work.

[0172] Obviously, the above embodiments of the present invention are merely examples for the purpose of illustrating the present invention, and are not intended to limit the embodiments of the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention shall be included within the scope of protection of the claims of the present invention.

Claims

1. An image deraining method, characterized in that: The method comprises: Obtain rainy images and corresponding infrared images; Inputting the rainy image and the infrared image into a preset deraining network to obtain a derained image and a derained infrared image respectively; Specifically include: Converting the rain image and the infrared image into rain image and infrared image represented by low-dimensional features through a convolutional layer; Inputting the rainy image and the infrared image represented by the low-dimensional features into a first attention layer group module to extract shallow features of the rainy image and the infrared image; Encoding and decoding the shallow features of the rain image and the infrared image through the encoding and decoding module; Inputting the decoded rain image and infrared image into the second attention layer group module to further extract deep features of the rain image and infrared image to obtain predicted rain streak negative residual information; Mapping the rain streak negative residual information to a feature image through a convolutional layer, and obtaining the rain-removed image and the rain-removed infrared image by summing the rain streak negative residual information in the feature image with the rain image and the infrared image; The deraining network is used to extract image features, encode and decode the image features, predict rain stripe negative residual information based on the decoded image features, and generate the derained image and the derained infrared image based on the rain stripe negative residual information; Inputting the derained image and the derained infrared image into a preset fusion network to obtain a light-enhanced derained image; Specifically include: Converting the derained image from RGB space to YCrCb space to obtain three component images of the derained image in YCrCb space; Inputting the Y component image of the derained image and the derained infrared image into a pre-trained feature extraction module to extract corresponding feature maps; wherein the feature maps include shallow feature maps and deep feature maps; Inputting the feature map into an information measurement module, calculating information measurement values ​​of the derained image and the derained infrared image, and calculating information retention of the derained image and the derained infrared image based on the information measurement values; inputting the derained image and the derained infrared image into an image fusion module, fusing the derained image and the derained infrared image according to their information retention, to generate a fused image; Convert the fused image from YCrCb space to RGB space; The fusion image is estimated by using an illumination estimation module for pixel-level parameter brightness information, and the estimated pixel-by-pixel curve parameters are applied to the derained image input to the fusion network to obtain an illumination-enhanced derained image; at the same time, the derained infrared image input to the fusion network is calibrated by using a content calibration module; Re-inputting the illumination-enhanced derained image and the calibrated derained infrared image into the fusion network, and continuously performing iterative fusion to ultimately generate an illumination-enhanced derained image that maximizes pixel information; The fusion network is used to generate a fused image based on the derained image and the derained infrared image, perform brightness information estimation on the fused image, generate an illumination-enhanced derained image based on the brightness information estimation result, calibrate the derained infrared image, and iteratively fuse the brightness information estimation result of the fused image with the calibrated derained infrared image to generate an illumination-enhanced derained image that maximizes pixel information.

2. The method according to claim 1, characterized in that The deraining network includes an attention layer group module and an encoding and decoding module; The attention layer group module includes several attention layers and several operation layers, which are used to extract image features and predict rain stripe negative residual information based on the decoded image features; The encoding and decoding module includes several encoders and several decoders, which are used to encode and decode image features.

3. The method according to claim 2, characterized in that The attention layer group module is specifically used to: Calculating an attention weight by the attention layer according to a first feature of the data input into the attention layer group module; Normalizing the attention weights; The operation layer changes and operates the image input to the attention layer group module according to the attention weight, thereby generating a second feature representation of the data input to the attention layer group module.

4. The method according to claim 2, characterized in that The encoding and decoding module is specifically used for: The feature map input to the encoding and decoding module is encoded by a multi-layer encoder with progressive downsampling, wherein the encoder is composed of a plurality of Transformer Blocks; wherein, after each Transformer Block, the feature map is downsampled by a corresponding downsampling module to reduce the spatial size of the feature map; The encoded feature map is decoded by a multi-layer decoder with progressive upsampling that is symmetrical to the encoder structure; wherein the decoder is composed of a number of Transformer Blocks equal to the number of the encoder; wherein after each Transformer Block, the feature map is upsampled by a corresponding upsampling module to increase the spatial size of the feature map and reduce the number of channels of the feature map; The encoded feature map output by each Transformer Block of the encoder and the decoded and upsampled feature map output by the next level Transformer Block of the Transformer Block in the decoder corresponding to the Transformer Block of the encoder are connected to transmit low-dimensional feature information.

5. The method according to claim 1, wherein The fusion network includes a feature extraction module, an information measurement module, an image fusion module, an illumination estimation module and a content calibration module; The feature extraction module is used to extract features of the derained image and the derained infrared image; The information measurement module is used to generate information retention of the derained image and the derained infrared image according to the features of the derained image and the derained infrared image; The image fusion module is used to fuse the derained image and the derained infrared image according to their information retention, so as to generate a fused image; The illumination estimation module is used to estimate brightness information of the fused image; The content calibration module is used to calibrate the derained infrared image.

6. The method according to claim 1, characterized in that The training method of the rain removal network and the fusion network includes: Acquire a first image dataset, where the image dataset includes rain-free images without rain, and the rain-free images include visible light images and infrared images; Using a raindrop synthesis algorithm on the rain-free image to generate a rainy image; constructing a rain removal target image pair based on the rainy image and the rain-free image to generate a second image dataset; Constructing a training set and a test set based on the second image dataset; The deraining network and the fusion network are trained using the training set, and the trained deraining network and the fusion network are tested using the test set.

7. The method according to claim 1, characterized in that The training method of the rain removal network and the fusion network includes: AdamW is used as the optimizer for the deraining network, the initial learning rate is set to 3e-4, and the cyclic cosine annealing learning rate adjustment method is used for training, and the learning rate is adjusted to the final ; Adam is used as the optimizer for the fusion network, and the initial learning rate is set to , and trained using an exponentially decaying learning rate adjustment method.

8. An image deraining device, characterized in that: The device comprises: An input module is used to obtain rain images and corresponding infrared images; a rain removal module, configured to input the rainy image and the infrared image into a preset rain removal network to obtain a rain-removed image and a rain-removed infrared image respectively; Specifically include: Converting the rain image and the infrared image into rain image and infrared image represented by low-dimensional features through a convolutional layer; Inputting the rainy image and the infrared image represented by the low-dimensional features into a first attention layer group module to extract shallow features of the rainy image and the infrared image; Encoding and decoding the shallow features of the rain image and the infrared image through the encoding and decoding module; Inputting the decoded rain image and infrared image into the second attention layer group module to further extract deep features of the rain image and infrared image to obtain predicted rain streak negative residual information; Mapping the rain streak negative residual information to a feature image through a convolutional layer, and obtaining the rain-removed image and the rain-removed infrared image by summing the rain streak negative residual information in the feature image with the rain image and the infrared image; The deraining network is used to extract image features, encode and decode the image features, predict rain stripe negative residual information based on the decoded image features, and generate the derained image and the derained infrared image based on the rain stripe negative residual information; a fusion module, configured to input the derained image and the derained infrared image into a preset fusion network to obtain a light-enhanced derained image; Specifically include: Converting the derained image from RGB space to YCrCb space to obtain three component images of the derained image in YCrCb space; Inputting the Y component image of the derained image and the derained infrared image into a pre-trained feature extraction module to extract corresponding feature maps; wherein the feature maps include shallow feature maps and deep feature maps; Inputting the feature map into an information measurement module, calculating information measurement values ​​of the derained image and the derained infrared image, and calculating information retention of the derained image and the derained infrared image based on the information measurement values; inputting the derained image and the derained infrared image into an image fusion module, fusing the derained image and the derained infrared image according to their information retention, to generate a fused image; Convert the fused image from YCrCb space to RGB space; The fusion image is estimated by using an illumination estimation module for pixel-level parameter brightness information, and the estimated pixel-by-pixel curve parameters are applied to the derained image input to the fusion network to obtain an illumination-enhanced derained image; at the same time, the derained infrared image input to the fusion network is calibrated by using a content calibration module; Re-inputting the illumination-enhanced derained image and the calibrated derained infrared image into the fusion network, and continuously performing iterative fusion to ultimately generate an illumination-enhanced derained image that maximizes pixel information; The fusion network is used to generate a fused image based on the derained image and the derained infrared image, perform brightness information estimation on the fused image, generate an illumination-enhanced derained image based on the brightness information estimation result, calibrate the derained infrared image, and iteratively fuse the brightness information estimation result of the fused image with the calibrated derained infrared image to generate an illumination-enhanced derained image that maximizes pixel information.

Images