Night image rain removal method based on global semantic guidance

Abstract

The invention provides a nighttime image rain removal method based on global semantic guidance, and the method comprises the steps: 1, generating an RGB-infrared dual-mode nighttime rain image data set # imgabs0 #; 2, constructing an RGB image rain removal model based on infrared guidance, and inputting the RGB image # imgabs1 # with rain and the corresponding infrared image # imgabs2 # into an RGB image rain removal model based on infrared guidance for convolution, feature fusion, coding, decoding, feature compensation and selection, and outputting an image after preliminary rain removal. And 3, introducing the global semantic guidance module into the RGB image rain removal model, and outputting a final rain-removed image. According to the constructed IRDeNet method, different types of convolutional neural networks and attention mechanisms including a space attention mechanism, a channel attention mechanism and a self-attention mechanism are combined, and the global and local feature modeling and extraction capability of the model is ensured; the RGB image and the infrared image are fused to be used for a night image rain removal task, and the rain removal effect is remarkably improved.

Description

Technical Field

[0001] The invention relates to a night image rain removal method based on global semantic guidance, and belongs to the technical field of image processing. Background Art

[0002] Image deraining is a hotly debated topic in the field of computer vision. Its core task is to restore rainy images to clear rain-free images and retain as much background information as possible. However, the challenge of this task lies in the fact that even in the same image, the positions, shapes, and directions of raindrops and rain streaks vary. In addition, the superposition of rain may produce the effect of rain and fog, resulting in more severe occlusion. These factors will lead to image quality degradation and make the texture and detail information in the image difficult to recognize.

[0003] The image deraining task mainly includes two parts: construction of rain map dataset and design of image deraining algorithm.

[0004] There are two main sources of paired single-image rain removal datasets: one is to generate realistic rain streak images by analyzing the physical mechanisms behind the formation of rain streaks and raindrops, and then use different methods to merge the rain streak images with the clean background images. The datasets using this method mainly include: Rain100L, Rain200L, DDC-Data, RainCityscapes, etc. These datasets based on physical mechanisms are relatively easy to produce, but they cannot naturally fit the changes in speed, angle, shape, and scale of rain during the falling process; in addition, the artificially set rain-related parameters are limited by prior knowledge and cannot fully reflect the complexity and diversity of the real world.

[0005] The other is to use a semi-automatic method to collect real rain data sets. The representative SPA-data data set is to record rainy day videos by manually adjusting the camera parameters, and use manual operations to obtain rain images and rain-free background images. In addition, there are also data sets such as Raindrop and RainDS. Although this method can obtain real rainy day images, it is difficult to capture various edge cases and rare events in real scenes. The lack of training sets for these cases will have a great impact on the robustness and performance evaluation of subsequent rain removal models. In addition, although obtaining rain images through video cropping and other methods has reduced the workload of collection to a certain extent, it will inevitably lead to problems such as high repetition rate of rain images, single scenes, and low quality.

[0006] A common problem with the above-mentioned commonly used datasets is that they are more targeted at daytime and well-lit conditions. However, as known from the present embodiment, important application fields of image deraining, such as autonomous driving and security monitoring, are also widely used at night, and the night scene itself is plagued by problems such as insufficient lighting, color distortion, limited dynamic range, and reduced contrast. In addition, the degradation, blurring, and distortion caused by rain make it difficult to directly apply the model trained on daytime rain images to nighttime deraining. The above defects result in the generalization effect of the deraining models trained with these datasets on real rainy images to be significantly insufficient, and these problems are of great significance to the real image deraining task. Therefore, in response to these problems, the present invention constructs a new RGB-infrared dual-modal rainy image dataset. Summary of the invention

[0007] The present invention aims to solve the technical problem in the prior art that the generalization effect of the model trained by daytime rain images directly applied to nighttime rain removal on real rain images is obviously insufficient, and further proposes a nighttime image rain removal method based on global semantic guidance.

[0008] The technical solution adopted by the present invention to solve the above problems is: the present invention proposes a night image rain removal method based on global semantic guidance. It includes:

[0009] Step 1: Generate RGB-infrared dual-modal nighttime rainy image dataset

[0010] Step 2: Build an infrared-guided RGB image deraining model;

[0011] The infrared-guided RGB image deraining model includes convolution module, hybrid structure block, residual-convolution block, parallel attention module, optimal selection sparse attention module, mixed scale feedforward network, attention enhancement fusion network, attention guided expert network, which can derain RGB rainy images. And the corresponding infrared image The infrared-guided RGB image deraining model is input for convolution, feature fusion, encoding, decoding, feature compensation and selection, and the image after preliminary deraining is output.

[0012] Step 3: Construct a global semantic guidance module and introduce it into the RGB image deraining model to supervise the RGB image deraining model to generate more realistic and complete image textures, so that the derained image can maintain the semantic consistency of the background texture while removing rain traces, and output the final derained image.

[0013] Optionally, in step 1, obtain the RGB-infrared dual-modality nighttime rainy image dataset The steps include:

[0014] Step 1.1: Build a rain streak synthesis network based on the variational inference framework, use the rain image collected based on the semi-automatic method as training data to train the rain streak synthesis network, and obtain the real rain-free background image b and rain streak layer r;

[0015] Step 1.2: Construct an atmospheric scattering model, obtain a foggy image based on the atmospheric scattering model, synthesize the foggy image with the rain streak layer r to obtain a rain streak map, and synthesize the rain streak map with the real rain-free background map b to obtain an RGB-infrared dual-modal rainy image.

[0016] Optional,

[0017] The rain streak synthesis network based on the variational inference framework in step 1.1 includes a background extraction module, a rain streak inference module, a generator, and a discriminator;

[0018] The background extraction module embeds the rain-free background image x in the training data into the Gaussian prior distribution of the real rain-free background image b, uses the PReNet model to complete the posterior inference of the real rain-free background image b, and infers the posterior parameters σ and μ from the rain image in the training data to obtain the real rain-free background image b;

[0019] The rain streak inference module is composed of five convolution-ReLU blocks and linear layers in sequence. Rain modeling is performed through the generator method to obtain the latent variable z, and the latent variable z is inferred to obtain the posterior parameters α and β;

[0020] The generator consists of transposed convolution and activation function ReLU, and obtains rain streak information based on the latent variable z;

[0021] The discriminator uses the SAGAN network, which consists of a convolutional layer and an activation function LeakyReLU, and adds an attention mechanism to capture the global correlation in the image to distinguish the real rain image o from the generated rain image.

[0022] The expression of the latent variable z is:

[0023] r=G θ (z)(1)

[0024] z~N(z|0,I)(2)

[0025] In formulas (1) and (2), r is the rain streak layer, z is the latent variable, θ is the generator parameter, and z~N(z|0,I) is the prior distribution of the latent variable z;

[0026] The Gaussian prior distribution of the real rain-free background image b is:

[0027]

[0028] In formula (3), is a hyperparameter that measures the similarity between the rain-free background image x and the true rain-free background image b.

[0029] Optionally, the steps of obtaining the foggy image in step 1.2 include:

[0030] Step 1.2.1: Use DIFFNet model to estimate the rain image I in the training data RGB The depth map d(x) of (x);

[0031] Step 1.2.2: Use the hyperparameter β to adjust the scattering intensity of the depth map and obtain the transmission rate t(x);

[0032] Step 1.2.3: Use hyperparameter A to adjust I RGB (x) color and intensity, to obtain a foggy image;

[0033] The calculation formula of the transmission rate t(x) is:

[0034] t(x)=e -β×d(x) (4)

[0035] The calculation formula for foggy images is:

[0036] I haze (x) = I RGB (x)×t(x)+A×(1-t(x))(5)

[0037] In formula (5), I haze (x) is a foggy image.

[0038] Optionally, the step of outputting the image after preliminary rain removal in step 2 includes:

[0039] Step 2.1: Based on the RGB-infrared dual-modal rainy image dataset Get RGB rain image And the corresponding infrared image Where H is the image height and W is the image width;

[0040] Step 2.2: RGB rainy image based on convolution module And the corresponding infrared image Perform 3×3 convolution to expand the channel dimension;

[0041] Step 2.3: Get the convolutional RGB rain image And the corresponding infrared image The shallow features of RGB rainy images And the corresponding infrared image The shallow features of the convolutional RGB rainy image are encoded and decoded in four layers based on the hybrid structure block. Encoding, based on residual-convolutional blocks for infrared images Encode and encode the infrared image through the hybrid structure block Decode;

[0042] Step 2.4: Convert RGB to rain image And the corresponding infrared image Upsampling and downsampling are performed during the encoding-decoding process to obtain the RGB image feature F RGB and infrared image feature F IR , and use the skip connection to complete each encoding-decoding to obtain the RGB image feature F RGB and infrared image feature F IR Input the attention enhancement fusion network for feature fusion;

[0043] Step 2.5: Input the fusion features of RGB image and infrared image into the parallel attention module for rough feature extraction;

[0044] Step 2.6: Input the fused features after rough feature extraction into the optimal selection sparse attention module for feature selection;

[0045] Step 2.7: Input the selected features into the mixed-scale feedforward network, use 1×1 convolution after layer normalization to expand the number of channels, and send the input feature map to two parallel multi-scale paths, using 3×3 and 5×5 deep convolutions to enhance multi-scale local information extraction respectively;

[0046] Step 2.8: Input the multi-scale features of the RGB image and the infrared image into the attention-guided expert network for feature screening and extraction to obtain a rain-free image;

[0047] Step 2.9: Based on the pixel-level relationship between the rain-free image and the true rain-free image, a Charbonnier Loss function is constructed, and the output is a preliminary rain-free image I after deraining with a loss less than a first preset value. Derain ;

[0048] The calculation formula for deep convolution enhanced multi-scale local information extraction is:

[0049]

[0050] X=f 1×1 (concat(X′1,X′2))+X. (6)

[0051] In formula (6), f 1×1 is a 1*1 convolution, f 3×3 and f 5×5They are 3*3 depth convolution and 5*5 depth convolution respectively, and concat(·) is cross-channel connection;

[0052] The expression of Charbonnier Loss loss function is:

[0053]

[0054] Optionally, the RGB image feature F obtained after each encoding-decoding in step 2.4 RGB and infrared image feature F IR The steps of inputting attention enhancement fusion network for feature fusion include:

[0055] Step 2.4.1: Use the linear layer to transform the RGB image feature F RGB and infrared image feature F IR Mapping is performed and activated using the GELU activation function to obtain the RGB image feature weight W RGB and infrared image feature weight W IR ;

[0056] Step 2.4.2: Based on RGB image feature weight W RGB and infrared image feature weight W IR Get the relevant attention feature matrix A;

[0057] Step 2.4.3: Combine the relevant attention feature matrix A with the RGB image feature F RGB and infrared image feature F IR Multiply by channel to obtain initial fusion features, and perform cross-channel fusion processing on the initial fusion features to obtain enhanced RGB image features and infrared image features;

[0058] Step 2.4.4: Perform secondary fusion of the enhanced RGB image features and infrared image features;

[0059] Step 2.4.5: Differentiate the features obtained by the secondary fusion to obtain the fused RGB image features and infrared image features The RGB image features and infrared image features Add together to get the fusion feature F fuse ;

[0060] RGB image feature weight W RGB and infrared image feature weight W IR The expression is:

[0061] W RGB =f MLP (f GAP (FRGB )),

[0062] W IR =f MLP (f GAP (F IR )). (8)

[0063] In formula (8), f GAP is the global mean pooling, f MLP is a multi-layer perceptron;

[0064] The expression of the relevant attention feature matrix A is:

[0065]

[0066] In formula (9), σ(·) is the Sigmoid activation function; the coefficient c is the number of channels, which acts as a proportional factor for attention enhancement. is element-wise multiplication;

[0067] The calculation formula of enhanced RGB image features and infrared image features is:

[0068]

[0069] In formula (10), ⊙ represents the channel-by-channel product, concat(·) represents the cross-channel connection, and f 7×7 Represents a depthwise convolution with a convolution kernel size of 7×7;

[0070] The expression of the feature after secondary fusion is:

[0071]

[0072] In formula (11), f 1×1 It is a convolution with a convolution kernel size of 1*1;

[0073] Fusion feature F fuse The calculation formula is:

[0074]

[0075] In formula (12), split(·) is splitting by channel.

[0076] Optionally, the parallel attention module in step 2.5 includes a gated attention layer, a channel attention layer, and a pixel attention layer;

[0077] The gated attention layer includes a feature extraction branch and a gated branch. The feature extraction branch includes convolution kernels of sizes 1*1 and 3*3 respectively, and the gated branch layer includes a convolution kernel of size 1*1 and a Sigmoid activation function.

[0078] The steps of feature rough extraction include:

[0079] Step 2.5.1: Fusion feature F of the input fuse Standardize the process;

[0080] Step 2.5.2: Preliminary extraction of fusion features F based on the gated attention layer fuse Pixel-level feature information;

[0081] Step 2.5.3: Based on the channel attention layer, a global average pooling operation is used to extract the spatial information of each channel. Point-by-point convolution is used to learn the dependencies between channels, and the GELU activation function is used to enhance the nonlinear relationship between channels. The Sigmoid activation function is used to generate an attention score in the range of [0,1] for each channel. The attention score of each channel is multiplied by the corresponding channel of the original feature map to obtain the weighted feature map.

[0082] Step 2.5.4: Based on the pixel attention layer, point-wise convolution and GELU activation function are used to learn and generate pixel-level attention maps from the input feature map, and a Sigmoid activation function is used to generate an attention score between [0,1] for each channel, and feature weighting is performed on the original feature map;

[0083] Step 2.5.5: Connect the attention results of the gated attention layer, channel attention layer, and pixel attention layer along the channel dimension, and use point-by-point convolution-GELU-point-by-point convolution to reduce the dimension of the connected feature channel to the same dimension as the input, and use skip connection to sum them up to get the output feature map.

[0084] Optionally, in step 2.6, the step of inputting the fused features after the rough feature extraction into the optimal selection sparse attention module for feature selection includes:

[0085] Step 2.6.1: Aggregate pixel-level cross-channel features using point-by-point convolution with a kernel size of 1*1;

[0086] Step 2.6.2: Use deep convolutional encoding with a kernel size of 3*3 to encode channel-level spatial information and obtain the query, key, and value mapped according to the input feature map;

[0087] Step 2.6.3: Perform a dot product operation on the query and key to calculate the similarity between all queries and keys and obtain the attention map Keep the largest k self-attention score values ​​and set the other self-attention score values ​​to negative infinity;

[0088] Step 2.6.4: Use the softmax function to normalize the attention score to get the attention weight, multiply the attention weight and value to calculate the weighted sum of self-attention, and get the self-attention output for each position;

[0089] The query, key and value are calculated as:

[0090]

[0091] In formula (13), Q is the query, K is the key, and V is the value. is the depthwise convolution, It is point-wise convolution;

[0092] The calculation formula for the self-attention output at each position is:

[0093]

[0094] Optionally, the attention-guided expert network in step 2.8 includes an average pooling layer with a convolution kernel size of 3*3, a separable convolution layer with convolution kernel sizes of 1*1, 3*3, 5*5, and 7*7, and an expanded convolution layer with convolution kernel sizes of 3*3, 5*5, and 7*7;

[0095] The steps to obtain a rain-free map include:

[0096] Step 2.8.1: For the input feature map, use the average pooling layer to get the descriptor corresponding to each channel

[0097] Step 2.8.2: Obtain the coefficients T of each expert network using the learnable weight matrices W1 and W2;

[0098] Step 2.8.3: Combine the output of each expert network according to the coefficient T to obtain a rain-free map.

[0099] The descriptor z corresponding to each channel k The calculation formula is:

[0100]

[0101] In formula (15), X(i,j,k) is the feature Figure X The pixel value of the kth channel at (i, j);

[0102] The calculation formula of the coefficient T of each expert network is:

[0103] T=W2σ(W1z),

[0104]

[0105] In formula (16), σ(·) is the ReLU activation function, f 1×1 is a 1×1 convolution, is the expert network operation, and [·] is the cross-channel connection.

[0106] Optionally, the step of outputting the final rain-free image in step 3 includes:

[0107] For the real rain-free image I gt And the initial image after rain removal I Derain Use the pre-trained SAM model to segment the image and obtain the corresponding mask map;

[0108] Based on focal loss and dice loss, a seg loss function is constructed to obtain the real rain-free image I gt And the rain-free image I after preliminary rain removal Derain and outputting a preliminary de-rained image whose difference is less than a second preset value;

[0109] The overall loss function of the RGB image deraining model is constructed based on the seg loss function and the Charbonnier Loss function, and the images after preliminary deraining whose loss value is less than the third preset value are retained;

[0110] Outputting the preliminary de-rained image with a loss value less than a third preset value as the final de-rained image;

[0111] The calculation formula of seg loss function is:

[0112]

[0113] In formula (17), α and β are both adjustable hyperparameters;

[0114] The calculation formula of focal loss is:

[0115]

[0116] The calculation formula of dice loss is:

[0117]

[0118] In formula (20), y i is the predicted value of the RGB image deraining model between [0,1], t i is the target value;

[0119] The calculation formula of the overall loss function of the RGB image deraining model is:

[0120]

[0121] The beneficial effects of the present invention are:

[0122] 1. The IRDeNet method constructed in the present invention combines different types of convolutional neural networks and attention mechanisms including spatial attention mechanism, channel attention mechanism and self-attention mechanism, ensuring the model's ability to model and extract global and local features.

[0123] 2. The present invention constructs a data set by learning rain streak information, and the model trained with daytime rain images can be directly applied to nighttime rain removal.

[0124] 3. The present invention combines RGB images and infrared images for image deraining tasks, and the deraining effect is significantly improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0125] Figure 1 It is a flowchart of a night image deraining method based on global semantic guidance provided by the present invention;

[0126] Figure 2 A structural diagram of a rain streak synthesis network based on a variational inference framework provided by the present invention;

[0127] Figure 3 The RGB-infrared dual-mode rain image provided by the present invention;

[0128] Figure 4 A structural diagram of the RGB image deraining model provided by the present invention;

[0129] Figure 5 A structural diagram of the residual-convolution block provided by the present invention;

[0130] Figure 6 A structural diagram of the parallel attention module provided by the present invention;

[0131] Figure 7 A structural diagram of the gated attention layer, channel attention layer and pixel attention layer in the parallel attention module provided by the present invention;

[0132] Figure 8 A structural diagram of the optimal selection sparse attention module provided by the present invention;

[0133] Fig. 9 A structural diagram of a mixed-scale feedforward network provided by the present invention;

[0134] Fig.10 A structural diagram of the attention enhancement fusion network provided by the present invention;

[0135] Fig.11 A structural diagram of the attention-guided expert network provided by the present invention;

[0136] Fig.12 This is a visualization of the image deraining result provided by the present invention. DETAILED DESCRIPTION

[0137] Example 1

[0138] like Figure 1 As shown, the steps of a night image deraining method based on global semantic guidance described in this embodiment include:

[0139] S1: Generate RGB-infrared dual-modality nighttime rainy image dataset

[0140] S2: Construct an infrared-guided RGB image deraining model;

[0141] The infrared-guided RGB image deraining model includes a convolution module, a hybrid structure block, a residual-convolution block, a parallel attention module, an optimal selection sparse attention module, a mixed-scale feedforward network, an attention enhancement fusion network and an attention-guided expert network. The RGB rainy image and the corresponding infrared image are input into the infrared-guided RGB image deraining model for convolution, feature fusion, encoding, decoding, feature compensation and selection, and the derained image is output.

[0142] S3: Introducing global semantic guidance for model-assisted enhancement;

[0143] Based on the constructed infrared-guided RGB image deraining model, global semantic guidance is introduced to supervise the model to generate more realistic and complete image textures, while removing rain traces and maintaining the semantic consistency of background texture.

[0144] Example 2

[0145] Combination Figure 2 and Figure 3 This embodiment is described to generate an RGB-infrared dual-modal rainy image dataset. The steps include:

[0146] S101: Constructing a rain streak synthesis network based on a variational inference framework;

[0147] The rain-free background image of the RGB-infrared dual-modal rainy image dataset comes from the KAIST multispectral pedestrian detection dataset and the M 3 FD dataset. The KAIST multispectral pedestrian detection dataset has 95,000 pairs of aligned RGB-IR image pairs, which are collected by a color camera, an infrared camera and a beam splitter. 3The FD dataset contains 4,177 pairs of aligned RGB-infrared images, which are collected by a binocular optical camera and a binocular infrared sensor. They contain a wealth of night images and images of wet roads. In addition, they all provide aligned infrared images for subsequent processing in this embodiment. This embodiment selects 16,000 pairs of RGB-infrared image pairs with large differences from the KAIST multispectral pedestrian detection dataset as training sets, 3,000 pairs as test sets, and 16,000 pairs as test sets from the M dataset. 3 In the FD dataset, 4,000 pairs of RGB-infrared images were selected as training sets and 200 pairs were selected as test sets. Finally, a dataset containing 20,000 pairs of RGB-infrared images for subsequent image deraining training sets and two test sets of 3,000 pairs and 200 pairs were constructed. These two test sets are used to measure the deraining effect of the model on RGB images of different qualities.

[0148] The method for generating an RGB-infrared dual-modal rainy image dataset is to fit the real distribution of rain images in a data-driven manner based on a variational inference framework, thereby obtaining an image deraining dataset with better diversity and realism. This embodiment builds an adaptive rain image synthesis model to construct an image deraining dataset that is as realistic as possible to reflect the complex conditions of rainy days in the real world. It includes two parts. First, rain streaks are generated through a variational inference framework, and second, rain images are synthesized in combination with various complex conditions existing on rainy days.

[0149] The core of the rain streak synthesis network (VRGNet) based on the variational inference framework is to implicitly model the rain layer as a generator and build a generative model for the rain image under the variational inference framework. Through the designed inference algorithm, the model can extract the general statistical distribution of rainy images and rain streaks based on the training data set in a data-driven manner, which enables the network to generate rain streak morphologies with different shapes. In addition, in order to enable the rain streak synthesis network to learn from the rain conditions in the real world, this embodiment uses SPA-data collected based on a semi-automatic method as a training set to train the rain streak synthesis network. However, since the SPA-data itself has problems such as too much data but a high repetition rate of rain images and unclear rain streaks in some images, this embodiment selects a certain amount of high-quality rain images from it as a training set.

[0150] The basic principle of the rain streak synthesis network structure based on the variational inference framework is as follows:

[0151] First, we can construct the following generative expression to represent the rain map:

[0152] o=r+b.(1)

[0153] In formula (1), o is a single rainy image, r is a rain pattern image, and b is a clean background image without rain.

[0154] According to formula (1), the rain image is decomposed into the rain part and the background part, and then the two parts are modeled separately.

[0155] S102: Stormwater Modeling;

[0156] It can be observed that rain streak images have characteristics such as direction, scale, and density, so they can be implicitly modeled through the generator. The hidden variable z is used to represent the physical structural factors behind the rain. The appearance of the rain (direction, scale, thickness, etc.) can be modified by transforming z. In this way, the rain streak layer r can be modeled as the following generator form:

[0157] r=Gθ(z)·(2)

[0158] In formula (2), z is a latent variable and θ represents the parameters of the generator.

[0159] In order to ensure the isotropy of rainfall factors, the network assumes that the prior distribution of the latent variable z follows a normal distribution with a mean of 0 and a covariance of the unit matrix, that is:

[0160] z~N(z|0,I).(3)

[0161] Such a priori assumption helps to unravel the physical rainfall factors in z, and different rain patterns can be generated by adjusting specific dimensions of z.

[0162] S103: Background modeling;

[0163] In the process of supervised rain streak generation, it is necessary to provide a training pair including a rain image and a rain-free background image. Since the training data set SPA-data used in this embodiment includes rain images from the real world and rain-free background images estimated based on multiple real rain images taken under the same conditions. In other words, the rain-free background image of SPA-data is not accurate enough. Since the rain-free background image x in the training set is not exactly the same as the real background image b, x can be embedded into the Gaussian prior distribution of b:

[0164]

[0165] In formula (4), is a hyperparameter that measures the similarity between x and b.

[0166] According to the above formula, the probability distribution of rain map o can be obtained as follows:

[0167] o~p θ (o|z,b).(5)

[0168] In formula (5), p θ(·) indicates that the distribution also depends on the parameters θ of the generator.

[0169] In this way, the task of generating rain images can be turned into the task of learning the general distribution p(o), which can be expressed as follows:

[0170] p(o)=∫∫p θ (o|z,b)p(z)p(b)dzdb.(6)

[0171] In formula (6), p(z) and p(b) represent the prior distribution of z and b respectively.

[0172] S104: Implicit distribution learning based on variational inference framework;

[0173] Since the implicit distribution p in p(o) θ Due to the existence of (o|z, b), the distribution of o cannot be learned directly, so the variational inference framework can be used for learning.

[0174] like Figure 2 As shown in Figure 2, the rain streak synthesis network based on the variational inference framework is the VRGNet network, which consists of four main substructures: background extraction module, rain streak inference module, generator, and discriminator (see the specific structure for details). Figure 2 ). The background extraction module uses a simple and fast-training PReNet model to implement the posterior inference of the background layer, that is, to infer the posterior parameters μ and σ from the rain image, and use them to restore the hidden clean rain-free background image b. The rain streak inference module is composed of 5 convolution-ReLU blocks and linear layers in sequence, which is used to infer the posterior parameters α and β of the latent variable z. The generator generates a rain streak image based on the latent variable z, which is mainly composed of transposed convolution and activation function ReLU. The discriminator uses the discriminator of the SAGAN network, which is mainly composed of convolution layers and activation functions LeakyReLU. An attention mechanism is also added to capture the global correlation in the image. The discriminator structure is used to distinguish between the real rain image o and the generated rain image. This helps the generator learn the rain streak information.

[0175] Based on the synthesis of rain images using the VRGNet model, through the analysis of real rainy day images in this embodiment, under cloudy or overcast weather conditions, rain suspended in the air produces a scattering effect on light, which in turn leads to the formation of rain fog, resulting in a shallower sense of depth of field for rainy day images, that is, distant objects appear more blurred than nearby objects. Therefore, the synthesis of rain fog is an important issue that needs to be considered in the construction of rain map datasets. In order to better simulate the rain fog in the rain map, this embodiment uses the atmospheric scattering model (ASM) model as the basis for synthesizing rain fog.

[0176] From daily observations, we can find that rain and fog usually appear in the distance of a single image, so we first need to obtain the depth map of the image. A common method is to use a monocular depth estimator E(x) to estimate the depth d(x) of the input image I(x). This embodiment uses the DIFFNet model to estimate the depth of the RGB image I(x). RGB After obtaining d(x), a hyperparameter β is used to adjust the required scattering intensity, and the transmission rate t(x) is obtained according to the following formula:

[0177] t(x)=e -β×d(x) .(7)

[0178] Then, the hyperparameter A is used to adjust the color and intensity of the foggy image, and the foggy image I is obtained by the following formula: haze (x):

[0179] I haze (x) = I RGB (x)×t(x)+A×(1-t(x)).(8)

[0180] After completing the rain streak generation and rain fog synthesis, the rain streak image is added to the selected rain-free background image. Considering that the rain streak and the background image are not completely unrelated, but affect each other, the screen blending model (SMB) is used to synthesize the rain streak and the background image. The formula is as follows:

[0181] R=B+SB⊙S.(9)

[0182] In formula (9), R represents the synthesized rain image, B represents the rain-free background image, and S represents the rain streak image.

[0183] Finally, this embodiment constructed a dataset containing 20,000 pairs of RGB-infrared image training sets and two test sets of 3,000 pairs and 200 pairs respectively. In order to facilitate subsequent processing, this embodiment cuts the dataset into 256×256 images and uses the VRGNet model to synthesize rain images with different directions, scales, and thicknesses. Finally, the above-mentioned ASM model is used to synthesize rain and fog, and a (rainy image, rainless image, infrared image) triple is constructed for subsequent deraining work. Part of the rain image of the RGB-infrared dual-modal rainy image dataset in this embodiment is shown in Figure 3.

[0184] Example 3

[0185] Combination Figure 4-11 This embodiment is described as follows. Figure 4 As shown, the infrared-guided RGB image deraining model described in this embodiment includes: a convolution module, a hybrid structure block, a residual-convolution block (such as Figure 5As shown), parallel attention module, optimal selection sparse attention module, mixed scale feedforward network, attention enhancement fusion network, attention guidance expert network and global semantic guidance module;

[0186] Overall, the infrared-guided RGB image deraining model (IRDeNet) proposed in this embodiment is a U-shaped hierarchical image deraining network. The model accepts an RGB image with rain and its corresponding infrared image as input, and outputs the image after deraining. Specifically, for an RGB rain image And its corresponding infrared image The model first uses 3×3 convolution operations to expand their channel dimensions. Subsequently, the shallow features of the RGB rain image and the infrared image are subjected to a 4-layer encoding-decoding process for feature fusion and conversion into deep features. When processing RGB images, a mixed structure block is used for encoding, while for infrared images, a residual-convolution block is used for encoding and a mixed structure block is used for decoding. Finally, the model compensates and selects features through an expert network to generate a restored rain-free image. In the encoding-decoding process, downsampling and upsampling operations are used to capture feature information of different scales in the image, and the skip connection technology is used to reduce the loss of spatial feature information that may be caused by downsampling, further enhancing the performance of the model.

[0187] S201: Parallel Attention Module (PAM);

[0188] like Figure 6 As shown in the figure, the parallel attention module includes a gated attention layer, a channel attention layer, and a pixel attention layer; the parallel attention module (PAM) mixes different types of attention mechanisms. It contains a gated attention, a channel attention, and a pixel attention. The input features are processed by batch normalization (BN) before parallel attention-based feature extraction.

[0189] S20101: Figure 7 As shown in Figure 1, the gated attention layer can initially extract pixel-level feature information, which consists of a feature extraction branch and a gating branch. The feature extraction branch includes convolutions with kernel sizes of 1×1 and 3×3, and the gating branch includes 1×1 convolution and Sigmoid activation function.

[0190] S20102: Figure 7As shown in the figure, the channel attention layer can effectively extract global information across channels. By assigning different importance to the features of different channels, the model can focus more on features that are more useful for the current deraining task. It first uses the global average pooling operation (GlobalAverage Pooling, GAP) to extract the global spatial information of each channel. Then point-by-point convolution is used to learn the complex dependencies between channels, and the GELU activation function is used to increase nonlinearity. An attention score is generated for each channel through the Sigmoid activation function to ensure that the attention score is between 0 and 1. Finally, the attention score of each channel is multiplied by the corresponding channel of the original feature map to obtain a weighted feature map. This strengthens important feature channels and suppresses unimportant channels.

[0191] S20103: Figure 7 As shown in the figure, the pixel attention layer can extract global pixel gating features. It focuses more on the importance of each pixel in the feature map, allowing the model to weight features at a finer level, thereby more accurately focusing on local features that are more important to the current deraining task. It first uses point-by-point convolution and GELU activation functions to learn from the input feature map and generate the corresponding pixel-level attention map. The value of each position in the map represents the importance weight of the pixel at that position. Then, the Sigmoid activation function is used to ensure that the attention score is between 0 and 1, and finally the original feature map is subjected to feature weighting operations.

[0192] Finally, the three different attention results are connected along the channel dimension, and the dimension of the connected feature channel is reduced to the same dimension as the input using point-by-point convolution-GELU-point-by-point convolution, and the output feature map is summed up using skip connection.

[0193] S202: Top-selection Sparse Attention (TSSA);

[0194] like Figure 8 As shown, after using the parallel attention module (PAM) for feature coarse extraction, this embodiment introduces an optimal selection sparse attention module (TSSA) based on the self-attention mechanism and the optimal selection mechanism. The time complexity required for the calculation of the traditional self-attention mechanism grows quadratically with the increase of the spatial resolution of the input image, which makes it difficult to directly apply it to the field of image processing. Although the window-based attention mechanism can reduce the amount of calculation, it is limited by the window size and is difficult to capture more extensive global information. Therefore, this embodiment adopts a cross-channel self-attention mechanism to make it have a linear computational complexity. This not only avoids the problem of computational efficiency, but also can effectively capture cross-channel global features.

[0195] For a feature map that has been layer normalized (LayerNormalization, LN) First, we use point-by-point convolution with a kernel size of 1×1 to aggregate pixel-level cross-channel features, and use depthwise convolution with a kernel size of 3×3 to encode channel-level spatial information, thereby obtaining Figure X The query (Q), key (K), and value (V) of the map;

[0196] The query, key and value are calculated as:

[0197]

[0198] In formula (10), Q is the query, K is the key, and V is the value. is the depthwise convolution, It is point-wise convolution;

[0199] Next, the original self-attention mechanism calculates the similarity between each token and other tokens to obtain an attention map. However, in the field of image deraining, since the distribution of rain streaks is usually irregular and has different effects at different positions and scales, the tokens from the query and key are not always correlated, which may affect the subsequent clear image restoration work. In order to solve this problem, this embodiment introduces an optimal selection mechanism for feature selection, which adaptively selects the top k self-attention scores in a sparse connection manner, thereby avoiding the participation of irrelevant information in the feature interaction process. Specifically, after obtaining the query (Q), key (K), and value (V), the similarity between all queries and keys is calculated by performing a dot product operation on Q and K to obtain an attention map. Only the largest k self-attention scores are retained, and the other self-attention scores are set to negative infinity. Finally, the softmax function is used to normalize the attention scores to obtain the attention weights, so that more important information is adaptively retained and useless information is removed.

[0200] Finally, the attention weight is multiplied by the value (V) to calculate the weighted sum of the self-attention and obtain the self-attention output of each position. It can be expressed as follows:

[0201] The calculation formula for the self-attention output at each position is:

[0202]

[0203] S203: Mixed-scale Feed-forward Network (MSFF);

[0204] Since the density and size of rain streaks may vary depending on the distance and other factors, there are rain streak information of different scales, such as Fig. 9 As shown in Figure 1, the mixed-scale feedforward network (MSFF) uses multi-scale convolution to capture features of different scales, so that the model can adaptively process information of different scales.

[0205] For an input feature map After layer normalization, 1×1 convolution is used to expand the number of channels, and then they are sent to two parallel multi-scale depth convolution paths, and 3×3 and 5×5 depth convolutions are used to enhance multi-scale local information extraction, which can be expressed as:

[0206] The calculation formula for deep convolution enhanced multi-scale local information extraction is:

[0207]

[0208] X=f 1×1 (concat(X′1,X′2))+X.(12)

[0209] In formula (12), f 1×1 is a 1*1 convolution, f 3×3 and f 5×5 They are 3*3 depth convolution and 5*5 depth convolution respectively, and concat(·) is a cross-channel connection.

[0210] S204: Attention-enhanced Fusion Network (AEFN)

[0211] Inspired by the research in the field of image fusion, this embodiment finds that the fusion of RGB images and infrared images can produce information-rich composite fused images that can both retain important object information and provide clear scene details. Therefore, after extracting features of different scales from RGB images and infrared images, such as Fig.10 As shown, this embodiment constructs an attention enhancement fusion network (AEFN) to better achieve feature complementarity between the two. Different from the traditional manually specified fusion strategy, this embodiment adopts a more flexible method of integrating channel and spatial interaction for fusion.

[0212] The Attention Enhanced Fusion Network (AEFN) mainly consists of two parts: a shared feature enhancement branch and a complementary feature fusion branch. The shared feature enhancement branch is responsible for processing the common features of RGB and infrared images, such as texture details and edge information, to ensure consistent representation across modalities; while the complementary feature fusion branch focuses on integrating unique attributes that only exist in RGB or infrared images, such as thermal features in infrared images or color information in RGB images. Through the synergistic combination of these two modules, feature information can be effectively aggregated and enriched at the current scale.

[0213] For the features F from RGB images and infrared images RGB and F IR , firstly, the global spatial information is integrated into the corresponding channel descriptor through an adaptive mean pooling operation. In order to thoroughly capture the dependencies between different channels, a linear layer is used for mapping, followed by a GELU activation. This process produces the RGB image feature weight W RGB and infrared image feature weight W IR , expressed as:

[0214] W RGB =f MLP (f GAP (F RGB )),

[0215] W IR =f MLP (f GAP (F IR )).(13)

[0216] In formula (13), f GAP represents global mean pooling, f MLP Represents a multi-layer perceptron. Next, this embodiment uses a correlation attention matrix A to measure the features F from the RGB image and the infrared image. VIS and F IR The relationship between:

[0217]

[0218] In formula (14), σ(·) represents the Sigmoid activation function; the coefficient c represents the number of channels and acts as a proportional factor for attention enhancement; Represents element-wise multiplication.

[0219] In the shared feature enhancement branch, we first combine the relevant attention matrix A with the RGB feature F RGB and infrared signature F IRMultiply by channel to obtain the initial fused features. In order to further enhance the multimodal interaction capabilities of RGB features and infrared features and capture their inter-channel features, this embodiment processes the fused features through a cross-channel fusion enhancement module:

[0220]

[0221] In formula (15), ⊙ represents channel-wise multiplication, concat(·) represents cross-channel concatenation, and f 7×7 Represents a depthwise convolution with a kernel size of 7×7.

[0222] In the complementary feature fusion branch, that is, in the scenario where only one modality provides useful features, the complementary feature attention enhancement is simply performed, which is recorded as:

[0223]

[0224] Finally, the spatial attention mechanism is used to aggregate the enhanced RGB and infrared features as output, and the fused features are obtained in the following way:

[0225]

[0226] In formula (17), f 1×1 Indicates that the convolution kernel size is 1×1 convolution.

[0227] Will get Split to obtain the fused RGB image features and infrared image features Add them together to get the fusion feature F fuse :

[0228]

[0229] In formula (18), split(·) means splitting by channel.

[0230] S205: Attention-guided Experts Network (AGEN)

[0231] like Fig.11As shown in the figure, the core idea of ​​the attention-guided expert network (AGEN) is to use multiple parallel convolutional neural networks (CNNs) as a hybrid expert model, where each expert is responsible for processing different aspects of the image, and then their outputs are combined through dynamically calculated weights to produce the final image restoration result. Such a hybrid expert network can effectively screen and extract features from RGB images and infrared images, on the one hand to avoid feature redundancy, and on the other hand to compensate for the information loss caused by sparsity. AGEN contains a total of 8 expert networks, namely: an average pooling layer with a receptive field of 3×3, a separable convolution layer with convolution kernel sizes of 1×1, 3×3, 5×5, and 7×7, and an expanded convolution layer with convolution kernels of 3×3, 5×5, and 7×7.

[0232] For the input feature map First, use the adaptive pooling layer to get the descriptor corresponding to each channel

[0233]

[0234] In formula (19), X(i, j, k) represents the feature Figure X The pixel value at (i, j) of the kth channel of . Then the coefficients T of each expert network are obtained using the learnable weight matrices W1 and W2:

[0235] T=W2σ(W1z),

[0236]

[0237] In formula (20), σ(·) represents the ReLU activation function, f 1×1 represents 1×1 convolution, represents the expert network operation and [·] represents the cross-channel connection.

[0238] The Charbonnier Loss function is used to train the IRDeNet network, which can effectively avoid problems such as images being too smooth and lacking in sensory realism. The loss function formula is as follows:

[0239]

[0240] In formula (21), I gt Indicates a true rainless image, I derain Represents a rain-free image generated by IRDeNet.

[0241] S301: global semantic guidance module;

[0242] In common low-level visual tasks, in order to obtain semantic clues, the semantic segmentation network and the image processing network are usually jointly trained. Although this can obtain more accurate semantic clues, the joint training method will make the network parameters too large, especially for paired deraining data sets. There is still a scarcity problem, which makes the network more difficult to train. Therefore, the global semantic guidance of this embodiment considers directly using the pre-trained semantic segmentation network to perform image deraining work. This embodiment adopts the SegmentAnything Model (SAM) proposed by Meta, which has two main advantages over general segmentation models: First, it is designed and trained in a promptable manner, and only a simple prompt (such as a point on the image, a bounding box, and a region mask) is given to achieve object segmentation. Secondly, SAM training is implemented on the current largest segmentation data set, so the segmentation effect is better. Therefore, this embodiment implements global semantic guidance based on the pre-trained SAM model.

[0243] In this embodiment, the real rain-free image I gt And the rain-free image after rain removal I Derain Use the pre-trained SAM model to segment the image, obtain their corresponding mask images, and then construct a loss function to measure the difference between the two. The loss function (seg loss) constructed in this embodiment consists of two parts: focal loss and dice loss:

[0244]

[0245] In formula (22), α and β are adjustable hyperparameters.

[0246] Since the number of samples in the background category is usually much larger than the number of samples in the foreground category, it will lead to category imbalance. Therefore, focal loss is used to solve the problem of sample category imbalance, which is defined as:

[0247]

[0248] In formula (23), p t It is expressed as:

[0249]

[0250] Dice loss can be used to evaluate the similarity between two samples. It measures the performance of the segmentation model by comparing the similarity between the predicted results and the true labels. It is defined as:

[0251]

[0252] In formula (25), y iRepresents the value predicted by the model (ranging from 0 to 1), t i Indicates the target value.

[0253] In summary, the overall loss function of the IRDeNet network can be expressed as:

[0254]

[0255] In summary, the present invention uses the constructed RGB-infrared dual-modal rainy image dataset to retrain the MPRNet, DGUNet, ELFormer, Uformer, and NAFNet models, and tests the model deraining effect on two test sets. The test indicators used are Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity Index (SSIM). According to the test results, the IRDeNet method proposed in this embodiment has been improved in multiple quantitative indicators (PSNR and SSIM), verifying its effectiveness in image deraining tasks.

[0256] In order to preliminarily prove the value of the fusion model, this embodiment simply connects the RGB image and the infrared image using a cross-channel connection method and trains them with MPRNet. It is found that the deraining effect is deteriorated. This is because the simple cross-channel connection can neither make good use of the infrared image features nor lose the information of the RGB image itself.

[0257] In this embodiment, IRDeNet is trained for 300,000 iterations, with an initial learning rate of 0.0002 and a cosine annealing strategy. In addition, vertical and horizontal flipping are randomly applied for data enhancement.

[0258] like Fig.12 As shown, it can be seen from the visualized rain removal results generated by this embodiment that the rain removal effect of the present invention is significantly higher than that of other mainstream methods.

[0259] The above is only a preferred embodiment of the present invention and does not limit the present invention in any form. Although the present invention has been disclosed as a preferred embodiment as above, it is not used to limit the present invention. Any technician familiar with this profession can make some changes or modify the technical contents disclosed above into equivalent embodiments without departing from the scope of the technical solution of the present invention. However, any simple modification, equivalent replacement and improvement made to the above embodiments without departing from the content of the technical solution of the present invention, based on the technical essence of the present invention, within the spirit and principles of the present invention, still fall within the protection scope of the technical solution of the present invention.

Claims

1. A nighttime image deraining method based on global semantic guidance, characterized in that: The steps of the night image deraining method based on global semantic guidance include: Step 1: Generate RGB-infrared dual-modal nighttime rainy image dataset Step 2: Build an infrared-guided RGB image deraining model; The infrared-guided RGB image deraining model includes a convolution module, a hybrid structure block, a residual-convolution block, a parallel attention module, an optimal selection sparse attention module, a mixed scale feedforward network, an attention enhancement fusion network, and an attention-guided expert network. And the corresponding infrared image Input the infrared-guided RGB image deraining model for convolution, feature fusion, encoding, decoding, feature compensation and selection, and output the image after preliminary deraining. Step 3: Construct a global semantic guidance module and introduce it into the RGB image deraining model to supervise the RGB image deraining model to generate more realistic and complete image textures, so that the derained image can maintain the semantic consistency of the background texture while removing rain traces, and output the final derained image.

2. The method for removing rain from nighttime images based on global semantics guidance according to claim 1, characterized in that: In step 1, we obtain the RGB-infrared dual-modality nighttime rainy image dataset. The steps include: Step 1.1: Build a rain streak synthesis network based on the variational inference framework, use the rain image collected based on the semi-automatic method as training data to train the rain streak synthesis network, and obtain the real rain-free background image b and rain streak layer r; Step 1.2: Construct an atmospheric scattering model, obtain a foggy image based on the atmospheric scattering model, synthesize the foggy image with the rain streak layer r to obtain a rain streak map, and synthesize the rain streak map with the real rain-free background map b to obtain an RGB-infrared dual-modal rainy image.

3. The method for removing rain from nighttime images based on global semantics guidance according to claim 2, characterized in that: The rain streak synthesis network based on the variational inference framework in step 1.1 includes a background extraction module, a rain streak inference module, a generator, and a discriminator; The background extraction module embeds the rain-free background image x in the training data into the Gaussian prior distribution of the real rain-free background image b, uses the PReNet model to complete the posterior inference of the real rain-free background image b, and infers the posterior parameters σ and μ from the rain image in the training data to obtain the real rain-free background image b; The rain streak inference module is composed of 5 convolution-ReLU blocks and linear layers in sequence. Rain modeling is performed in a generator manner to obtain latent variables z, and the latent variables z are inferred to obtain posterior parameters α and β; The generator consists of a transposed convolution and an activation function ReLU, and obtains rain streak information based on the latent variable z; The discriminator adopts the SAGAN network, which consists of a convolutional layer and an activation function LeakyReLU, and adds an attention mechanism to capture the global correlation in the image to distinguish the real rain image o from the generated rain image The expression of the latent variable z is: r=G θ (z)(1) z~N(z0,I)(2) In formulas (1) and (2), r is the rain streak layer, z is the latent variable, θ is the generator parameter, and z~N(z0,I) is the prior distribution of the latent variable z; The Gaussian prior distribution of the real rain-free background image b is: In formula (3), is a hyperparameter that measures the similarity between the rain-free background image x and the true rain-free background image b.

4. The method for removing rain from nighttime images based on global semantics guidance according to claim 2, characterized in that: The steps for obtaining the foggy image in step 1.2 include: Step 1.2.1: Use DIFFNet model to estimate the rain image I in the training data RGB The depth map d(x) of (x); Step 1.2.2: Use the hyperparameter β to adjust the scattering intensity of the depth map and obtain the transmission rate t(x); Step 1.2.3: Use hyperparameter A to adjust I RGB (x) color and intensity, to obtain a foggy image; The calculation formula of the transmission rate t(x) is: t(x)=e -β×d(x) (4) The calculation formula for foggy images is: I haze (x)=I RGB (x)×t(x)+A×(1-t(x))(5) In formula (5), I haze (x) is a foggy image.

5. The method for removing rain from nighttime images based on global semantics guidance according to claim 1, characterized in that: The steps of outputting the image after preliminary rain removal in step 2 include: Step 2.1: Based on the RGB-infrared dual-modal rainy image dataset Get RGB rain image And the corresponding infrared image Where H is the image height and W is the image width; Step 2.2: RGB rainy image based on convolution module And the corresponding infrared image Perform 3×3 convolution to expand the channel dimension; Step 2.3: Get the convolutional RGB rain image And the corresponding infrared image The shallow features of RGB rainy images And the corresponding infrared image The shallow features of the convolutional RGB rainy image are encoded and decoded in four layers based on the hybrid structure block. Encoding, based on residual-convolutional blocks for infrared images Encode and encode the infrared image through the hybrid structure block Decode; Step 2.4: Convert RGB to rain image And the corresponding infrared image Upsampling and downsampling are performed during the encoding-decoding process to obtain the RGB image feature F RGB and infrared image feature F IR , and use the skip connection to complete each encoding-decoding to obtain the RGB image feature F RGB and infrared image feature F IR Input the attention enhancement fusion network for feature fusion; Step 2.5: Input the fusion features of RGB image and infrared image into the parallel attention module for rough feature extraction; Step 2.6: Input the fused features after rough feature extraction into the optimal selection sparse attention module for feature selection; Step 2.7: Input the selected features into the mixed-scale feedforward network, use 1×1 convolution after layer normalization to expand the number of channels, and send the input feature map to two parallel multi-scale paths, using 3×3 and 5×5 deep convolutions to enhance multi-scale local information extraction respectively; Step 2.8: Input the multi-scale features of the RGB image and the infrared image into the attention-guided expert network for feature screening and extraction to obtain a rain-free image; Step 2.9: Based on the pixel-level relationship between the rain-free image and the true rain-free image, a Charbonnier Loss function is constructed, and a preliminary rain-free image I after deraining with a loss less than a first preset value is output. Derain ; The calculation formula for deep convolution enhanced multi-scale local information extraction is: In formula (6), f 1×1 is a 1*1 convolution, f 3×3 and f 5×5 They are 3*3 depth convolution and 5*5 depth convolution respectively, and concat(·) is cross-channel connection; The expression of Charbonnier Loss loss function is: In formula (7), I gt For a real rainless picture, I derain Rain-free image generated by the deraining model for RGB images.

6. The method for removing rain from nighttime images based on global semantics guidance according to claim 5, characterized in that: The RGB image feature F obtained after each encoding-decoding in step 2.4 RGB and infrared image feature F IR The steps of inputting attention enhancement fusion network for feature fusion include: Step 2.4.1: Use the linear layer to transform the RGB image feature F RGB and infrared image feature F IR Mapping is performed and activated using the GELU activation function to obtain the RGB image feature weight W RGB and infrared image feature weight W IR ; Step 2.4.2: Based on RGB image feature weight W RGB and infrared image feature weight W IR Get the relevant attention feature matrix A; Step 2.4.3: Combine the relevant attention feature matrix A with the RGB image feature F RGB and infrared image feature F IR Multiply by channel to obtain initial fusion features, and perform cross-channel fusion processing on the initial fusion features to obtain enhanced RGB image features and infrared image features; Step 2.4.4: Perform secondary fusion of the enhanced RGB image features and infrared image features; Step 2.4.5: Differentiate the features obtained by the secondary fusion to obtain the fused RGB image features and infrared image features The RGB image features and infrared image features Add together to get the fusion feature F fuse ; RGB image feature weight W RGB and infrared image feature weight W IR The expression is: W RGB =f MLP (f GAP (F RCB )), W IR =f MLP (f GAP (F IR )). (8) In formula (8), f GAP is the global mean pooling, f MLP is a multi-layer perceptron; The expression of the relevant attention feature matrix A is: In formula (9), σ(·) is the Sigmoid activation function; the coefficient c is the number of channels, which acts as a proportional factor for attention enhancement. is element-wise multiplication; The calculation formula of enhanced RGB image features and infrared image features is: In formula (10), ⊙ represents the channel-by-channel product, concat(·) represents the cross-channel connection, and f 7×7 Represents a depthwise convolution with a convolution kernel size of 7×7; The expression of the feature after secondary fusion is: In formula (11), f 1×1 It is a convolution with a convolution kernel size of 1*1; Fusion feature F fuse The calculation formula is: In formula (12), split(·) is splitting by channel.

7. The method for removing rain from nighttime images based on global semantics guidance according to claim 5, characterized in that: The parallel attention module described in step 2.5 includes a gated attention layer, a channel attention layer, and a pixel attention layer; The gated attention layer includes a feature extraction branch and a gated branch, the feature extraction branch includes convolution kernels of sizes 1*1 and 3*3 respectively, and the gated branch layer includes a convolution kernel of size 1*1 and a Sigmoid activation function; The steps of feature rough extraction include: Step 2.5.1: Fusion feature F of the input fuse Standardize the process; Step 2.5.2: Preliminary extraction of fusion features F based on the gated attention layer fuse Pixel-level feature information; Step 2.5.3: Based on the channel attention layer, a global average pooling operation is used to extract the spatial information of each channel. Point-by-point convolution is used to learn the dependencies between channels, and the GELU activation function is used to enhance the nonlinear relationship between channels. The Sigmoid activation function is used to generate an attention score in the range of [0,1] for each channel. The attention score of each channel is multiplied by the corresponding channel of the original feature map to obtain the weighted feature map. Step 2.5.4: Based on the pixel attention layer, point-wise convolution and GELU activation function are used to learn and generate pixel-level attention maps from the input feature map, and a Sigmoid activation function is used to generate an attention score between [0,1] for each channel, and feature weighting is performed on the original feature map; Step 2.5.5: Connect the attention results of the gated attention layer, channel attention layer, and pixel attention layer along the channel dimension, and use point-by-point convolution-GELU-point-by-point convolution to reduce the dimension of the connected feature channel to the same dimension as the input, and use skip connection to sum them up to get the output feature map.

8. The method for removing rain from nighttime images based on global semantics guidance according to claim 5, characterized in that: In step 2.6, the steps of inputting the fused features after rough feature extraction into the optimal selection sparse attention module for feature selection include: Step 2.6.1: Aggregate pixel-level cross-channel features using point-by-point convolution with a kernel size of 1*1; Step 2.6.2: Use deep convolutional encoding with a kernel size of 3*3 to encode channel-level spatial information and obtain the query, key, and value mapped according to the input feature map; Step 2.6.3: Perform a dot product operation on the query and key to calculate the similarity between all queries and keys and obtain the attention map Keep the largest k self-attention score values ​​and set the other self-attention score values ​​to negative infinity; Step 2.6.4: Use the softmax function to normalize the attention score to obtain the attention weight, multiply the attention weight and the value to calculate the weighted sum of the self-attention, and obtain the self-attention output of each position; The query, key and value are calculated as: In formula (13), Q is the query, K is the key, and V is the value. is the depthwise convolution, It is point-wise convolution; The calculation formula for the self-attention output at each position is:

9. The method for removing rain from nighttime images based on global semantics guidance according to claim 5, characterized in that: In step 2.8, the attention-guided expert network includes an average pooling layer with a convolution kernel size of 3*3, a separable convolution layer with convolution kernel sizes of 1*1, 3*3, 5*5, and 7*7, and an expanded convolution layer with convolution kernel sizes of 3*3, 5*5, and 7*7; The steps to obtain a rain-free map include: Step 2.8.1: For the input feature map, use the average pooling layer to get the descriptor corresponding to each channel Step 2.8.2: Obtain the coefficients T of each expert network using the learnable weight matrices W1 and W2; Step 2.8.3: Combine the output of each expert network according to the coefficient T to obtain a rain-free map. The descriptor z corresponding to each channel k The calculation formula is: In formula (15), X(i,j,k) is the pixel value of the kth channel of the feature map X at (i,j); The coefficient T of each expert network is calculated as: T=W2σ(W1z), In formula (16), σ(·) is the ReLU activation function, f 1×1 is a 1×1 convolution, is the expert network operation, and [·] is the cross-channel connection.

10. The method for removing rain from nighttime images based on global semantics guidance according to claim 5, characterized in that: The steps of outputting the final de-rained image in step 3 include: For the real rain-free image I gt And the initial image after rain removal I Derain Use the pre-trained SAM model to segment the image and obtain the corresponding mask map; Based on focal loss and dice loss, a seg loss function is constructed to obtain the real rain-free image I gt And the rain-free image I after preliminary rain removal Derain and outputting a preliminary de-rained image whose difference is less than a second preset value; Constructing the overall loss function of the RGB image deraining model based on the seg loss function and the Charbonnier Loss function, and retaining the images after preliminary deraining whose loss value is less than the third preset value; Outputting the preliminary de-rained image whose loss value is less than the third preset value as the final de-rained image; The calculation formula of seg loss function is: In formula (17), α and β are both adjustable hyperparameters; The calculation formula of focal loss is: The calculation formula of dice loss is: In formula (20), y i is the predicted value of the RGB image deraining model between [0,1], t i is the target value, ε is the smoothing term; The calculation formula of the overall loss function of the RGB image deraining model is:

Images