Low-illumination rain image semantic segmentation method based on efficient frequency domain decomposition and bar grouping residual attention EFD-SGRA

Abstract

The invention provides a low-illumination rain image semantic segmentation method based on efficient frequency domain decomposition and bar grouping residual attention EFD-SGRA. According to the method, firstly, a Vision Transform is introduced into an unsupervised pre-training framework, teacher and student architectures are adopted, then, frequency domain information separation and denoising processing are performed on features extracted by the Vision Transform by using an efficient frequency domain decomposition module, then, a self-attention feature map is calculated through a strip-shaped grouping residual attention module, and finally, the weight of a teacher model is updated through an exponential moving average mechanism. According to the method, the distribution difference between the source domain and the target domain can be effectively reduced, the interference caused by the rain imprint is inhibited, the model feature extraction capability is enhanced, and the semantic segmentation performance of the model in the low-illumination rain scene is further remarkably improved.

Description

Technical Field

[0001] This invention relates to deep learning, unsupervised pre-training frameworks, and image semantic segmentation techniques, specifically a semantic segmentation method for low-light rainy images that combines efficient frequency domain decomposition and bar-grouped residual attention EFD-SGRA. Background Technology

[0002] Image segmentation is one of the core technologies in computer vision, with wide applications in fields such as autonomous driving, medical image analysis, and security monitoring. In recent years, with the breakthrough development of deep learning technology, image segmentation methods based on deep neural networks have achieved significant improvements in both accuracy and efficiency. However, these methods typically assume that the input image is processed under ideal conditions of ample lighting and good weather, and their performance often drops sharply in real-world complex environments, especially under low-light and rainy conditions.

[0003] Low-light rainy images generally suffer from low overall brightness, poor contrast, low signal-to-noise ratio, and significant loss of detail due to factors such as insufficient ambient light, sensor noise, and rain streaks. Traditional image segmentation methods (such as threshold-based, edge-based, or region-based segmentation algorithms) heavily rely on the image's grayscale or color features. Under low-light rainy conditions, these features become extremely unreliable, resulting in poor segmentation performance that fails to meet the needs of practical applications. To address the challenges of difficult and costly image annotation in harsh environments, researchers have recently proposed Unsupervised Domain Adaptation (UDA) methods to effectively transfer models trained under normal lighting conditions to severe weather scenes without requiring real-world labeled data for the target domain. Nevertheless, existing deep learning-based semantic segmentation schemes for low-light rainy images still face a series of significant technical bottlenecks and challenges, necessitating an effective semantic segmentation method for these images. Summary of the Invention

[0004] This invention discloses a semantic segmentation method for low-light rainy images based on efficient frequency domain decomposition and bar-grouped residual attention (EFD-SGRA). This method first introduces the Vision Transformer into an unsupervised pre-training framework, and then separates the frequency domain information of the features extracted by the Vision Transformer through an efficient frequency domain decomposition module and performs denoising. Subsequently, a self-attention feature map is calculated using a bar-grouped residual attention module, and finally, the teacher model weights are updated using an exponential moving average mechanism. This method can guide the model to focus on areas severely affected by rain streaks, thereby reducing the distribution difference between the source and target domains, suppressing interference from rain streaks, enhancing the model's feature extraction capability, and ultimately achieving fast and accurate segmentation of low-light rainy images.

[0005] The technical solution to achieve the objective of this invention is:

[0006] A semantic segmentation method for low-light rainy images based on efficient frequency domain decomposition and strip grouped residual attention EFD-SGRA includes the following steps:

[0007] (1) Construct a low-light rain dataset NightCity-rain;

[0008] Cityscapes was used as the training set for normal illumination images in the source domain. A low illumination rainy dataset NightCity-rain was constructed based on the public dataset NightCity. The self-made dataset NightCity-rain was used as the training dataset for low illumination rainy images in the target domain.

[0009] (2) Construct teacher network a and student network b;

[0010] (2.1) Construct a teacher network a and a student network b. Network a samples the target domain data for rare categories, and network b samples the source domain data for rare categories. By analyzing the category distribution, rare category sampling can identify and oversample categories that appear less frequently in the source domain, thereby increasing the probability of rare category samples appearing in training and enhancing the model's learning ability and recognition effect for rare categories. Both networks contain an encoder consisting of a Vision Transformer (ViTblock) block, an efficient frequency domain decomposition module (EFD Module), and a bar grouping residual attention module (SGRA Module) connected in series, as well as a decoder for outputting the final segmentation prediction result.

[0011] (3) Input characteristics of teacher network and student network;

[0012] Input the target domain sample Xt of network a into the teacher feature extraction network, and input the source domain sample Xs of network b into the student feature extraction network;

[0013] (4) Blended images;

[0014] The target domain sample Xt is mixed with the source domain sample Xs extracted by the b network to obtain a mixed sample Xm, which is then input into the student feature extraction network.

[0015] (5) Construct an efficient frequency domain decomposition module EFD;

[0016] The high-efficiency frequency domain decomposition module (EFD) includes the frequency domain decomposition and denoising module (FFDM), the denoising module (DeNoise), and the hybrid differential convolution module (MDCM). The specific steps are as follows:

[0017] (5.1) Input the source domain features Xs and the target domain features Xt into the four stages of the Vision Transformer, ViTblock;

[0018] (5.2) The features extracted by the four stages of the ViT block are sequentially input into the EFD to extract frequency domain information. In the EFD, the normalization layer and the decomposition and denoising module FFDM are sequentially input and added to the original input through a skip connection. The resulting features are then added to the output of the SGRA after passing through the normalization layer.

[0019] (5.3) Construct the Frequency Domain Decomposition Denoising Module (FFDM). The construction method is as follows: The input feature map first passes through a combined layer consisting of a 1×1 convolution and a GELU activation function for feature projection and nonlinear transformation. Then, the obtained features are uniformly divided in the channel dimension to form two parallel processing branches. One branch uses a 3×3 depthwise separable convolution to capture high-frequency components that represent image details, and the other branch uses a 5×5 depthwise separable convolution to extract low-frequency components that represent the overall contour, thus simulating the frequency domain decomposition process in the spatial domain. After that, the features output by the two branches are spliced ​​and fused, and input into a dedicated denoising module DeNoise to suppress noise interference. The denoised features are then passed through a lightweight combined layer consisting of a 1×1 convolution and a 3×3 depthwise separable convolution for local feature enhancement and integration. Finally, the feature channels are adaptively reweighted through a channel attention mechanism to highlight important information and suppress redundancy, and the final output is the denoised features with frequency domain decomposition and enhancement.

[0020] (5.4) Construct a denoising module DeNoise. In this module, the input data is first transformed to the frequency domain by a 2D fast Fourier transform to obtain the frequency domain features in complex form. Then, the real and imaginary parts of the features are separated and input into the hybrid differential convolution module MDCM. Finally, the dimensions are rearranged and reshaped. The results of the dimension rearrangement and reshaping are subjected to a 2D inverse Fourier transform to obtain the feature tensor output suitable for subsequent convolution processing.

[0021] (5.5) Construct a hybrid differential convolution module MDCM. The module uses vertical differential convolution kernels, horizontal differential convolution kernels and ordinary convolution kernels in parallel to extract the gradient and structural information of frequency domain features in the vertical direction, horizontal direction and global direction respectively, and fuses the output results of the three paths. The fused features are nonlinearly enhanced and integrated by a CBR block consisting of a convolutional layer, a batch normalization layer and a ReLU activation function connected in sequence, and restored to the original feature size. This module achieves efficient filtering of image noise and preservation of details through frequency domain and spatial domain collaboration and directional differential convolution.

[0022] (6) Construct a strip-grouped residual attention SGRA module;

[0023] (6.1) Construct a strip-grouped residual attention (SGRA) module. Its structure is designed to efficiently capture long-range contextual dependencies across image regions while maintaining sensitivity to local details. The specific construction steps of this module are as follows: First, calculate the maximum and minimum channel values ​​of the input feature map and perform range and fusion operations. Then, input the output to three parallel processing branches to calculate the values ​​of self-attention Q, K, and V respectively.

[0024] (6.2) The obtained Q is reshaped and transposed, and the resulting feature map is multiplied pixel by pixel with K. The result is then reshaped and transposed again and processed by the activation function. Finally, it is multiplied pixel by pixel with V to obtain the output of SGRA. The features are further enhanced and contextualized by the bar-grouped residual attention module, and the output of the bar-grouped residual attention module is input into the gate unit of EFD to fuse the features of the SGRA module with the features of the EFD module.

[0025] (7) Update the teacher network synchronously;

[0026] (7.1) Input Xs directly into the student network and update the weights of the student network to the teacher network synchronously through the exponential moving average (EMA) mechanism.

[0027] (8) Calculate the consistency losses Lt and Ls;

[0028] (8.1) Mix the target domain sample Xs with the target domain sample Xt to generate a mixed image Xm. Input Xm into the student network and calculate the consistency loss Lt between the segmentation result obtained and the segmentation result yt obtained from Xt. Calculate the supervision loss Ls between the segmentation result obtained from Xs and the label ys. Attached Figure Description

[0029] Figure 1 EFD-SGRA network structure diagram;

[0030] Figure 2 High-efficiency frequency domain decomposition module (EFD) structure diagram;

[0031] Figure 3 DeNoise module structure diagram;

[0032] Figure 4 SGRA structure diagram of bar-grouped residual attention. Detailed Implementation

[0033] (1) Construct a low-light rain dataset NightCity-rain;

[0034] Cityscapes was used as the training set for normal illumination images in the source domain. A low illumination rainy dataset NightCity-rain was constructed based on the public dataset NightCity. The self-made dataset NightCity-rain was used as the training set for low illumination rainy images in the target domain. This dataset contains 2998 training images with a resolution of 1024×512 and 1299 test images. All images have pixel-level precise annotations.

[0035] (2) Construct teacher network a and student network b;

[0036] (2.1) As Figure 1 As shown, teacher network (a) and student network (b) with identical structures are constructed. Teacher network (a) is dedicated to processing unlabeled target domain data, while student network (b) is dedicated to processing finely labeled source domain data. Both networks consist of an encoder composed of a Vision Transformer, an Efficient Frequency Decomposition Module (EFD Module), and a Bar Grouping Residual Attention Module (SGRA Module) connected in series, and a decoder for outputting the final segmentation prediction result. Network a samples the target domain data using rare categories, while network b samples the source domain data using rare categories. By analyzing the category distribution, rare category sampling can identify and oversample categories with low frequency of occurrence in the source domain, thereby increasing the probability of rare category samples appearing during training and enhancing the model's learning ability and recognition performance for rare categories. In the source and target domains, the proportion of a certain category c pixels obtained by rare category sampling (RCS) is:

[0037]

[0038] Where N c N represents the number of pixels in category c. k Let M be the number of pixels in category k, and M be the total number of categories.

[0039] (3) Input characteristics of teacher network and student network;

[0040] After completing the construction and data preparation of networks a and b, the core process moves to the feature extraction stage. This step involves inputting data from different domains into the teacher network a and the student network b, respectively, and allowing them to flow through their internal core modules for deep feature learning and enhancement. Samples Xt from network a are input into the teacher feature extraction network, and samples Xs from network b are input into the student feature extraction network.

[0041] (4) Blended images;

[0042] The target domain sample Xt is mixed with the source domain sample Xs extracted by the b network to obtain a mixed sample Xm, which is then input into the student feature extraction network. The target domain sample Xt and the source domain sample Xs are mixed to generate:

[0043]

[0044] in Adjust dynamically according to training, in the early stages of training. Larger, later Smaller;

[0045] (5) Construct an efficient frequency domain decomposition module EFD;

[0046] The high-efficiency frequency domain decomposition module (EFD) includes the frequency domain decomposition and denoising module (FFDM), the denoising module (DeNoise), and the hybrid differential convolution module (MDCM). The specific steps are as follows:

[0047] (5.1) Input the source domain features Xs and the target domain features Xt into the four stages of the Vision Transformer, ViTblock;

[0048] (5.2) As Figure 2 As shown, the features extracted by the four-stage ViT block are sequentially input into the EFD to extract frequency domain information. In the EFD, they are sequentially input into the normalization layer and the decomposition and denoising module FFDM, and then added to the original input through a skip connection. The resulting features are then added to the output of SGRA after passing through the normalization layer.

[0049] (5.3) such as Figure 2As shown, the construction process of the Frequency Domain Decomposition (FFDM) module aims to simulate frequency domain decomposition through spatial domain operations to enhance feature representation and suppress noise interference. The input feature map of this module first undergoes feature projection and nonlinear transformation through a combined layer consisting of a 1×1 convolutional layer and the GELU activation function. This step effectively compresses the channel dimension and introduces a smooth nonlinear response, laying the foundation for subsequent frequency domain decomposition. Subsequently, the resulting features are uniformly divided into two parallel processing branches along the channel dimension: one branch uses a 3×3 depthwise separable convolution to focus on capturing high-frequency components representing image details, while the other branch uses a 5×5 depthwise separable convolution to extract low-frequency components representing the overall contour. This design simulates the core idea of ​​frequency domain decomposition in the spatial domain, namely, separating multi-band information through convolutional kernels of different scales, thereby avoiding the computationally expensive Fourier transform. Subsequently, the features output from the two branches are concatenated and fused along the channel dimension to form a multi-scale frequency domain feature representation, which is then input into a dedicated denoising module, DeNoise. This denoising module, based on the principle of frequency domain filtering, improves the robustness of the features by suppressing high-frequency noise components. The denoised features are then enhanced and integrated through a lightweight compositing layer, which includes a 1×1 convolution and a 3×3 depthwise separable convolution, further optimizing feature representation while ensuring efficiency. Finally, the feature channels are adaptively reweighted using a channel attention mechanism to highlight important information and suppress redundancy, ultimately outputting denoised features that have undergone frequency domain decomposition and enhancement, thereby significantly improving the semantic segmentation performance of the model in low-light, rainy scenes.

[0050] (5.4) such as Figure 3 As shown, a denoising module DeNoise is constructed. In this module, the input data is first transformed to the frequency domain by a 2D Fast Fourier Transform to obtain the frequency domain features in complex form. Then, the real and imaginary parts of the features are separated and input into the Hybrid Difference Convolution Module MDCM. Finally, the dimensions are rearranged and reshaped. The results of the dimension rearrangement and reshaping are subjected to a 2D Inverse Fourier Transform to obtain the feature tensor output suitable for subsequent convolution processing.

[0051] (5.5) such as Figure 3As shown, a Hybrid Differential Convolutional Module (MDCM) is constructed. The MDCM consists of two parallel branches: a standard convolutional branch to capture basic semantic information in the frequency domain, and a differential convolutional branch that enhances the perception of local details such as edges and textures by fusing horizontal and vertical differential convolutions. In the differential convolutional branch, the linear additivity principle of convolution operations is utilized: when multiple convolutional layers with the same kernel size, stride, and padding are applied in parallel to the same input, their output is equivalent to a new convolutional operation, whose weights and biases can be obtained by element-wise weighted summation of the original convolutional kernels. Two parallel differential convolutional layers are designed in the differential branch, with their weights and biases trained independently. During the inference phase, their parameters are weighted and fused into an equivalent convolutional kernel. Finally, the features output by the two branches are concatenated along the channel dimension and restored to the original feature size using a CRB block. Through the synergistic effect of standard convolution and differential convolution, MDCM can effectively suppress high-frequency noise in the frequency domain, especially showing a significant suppression effect on fine-grained interference such as rain streaks, thereby further enhancing the structural expressiveness of the image. Its overall operation can be formally represented as:

[0052]

[0053] in, Represents the frequency domain features of the input. This is a standard convolution branch operation. For differential convolution branch operations, For channel concatenation operations, CRB is a standard convolution, batch normalization, and ReLU operation. This is the final feature output of MDCM. This module achieves efficient noise removal and detail preservation in images through frequency and spatial domain collaboration and directional differential convolution.

[0054] (6) Construct a strip-grouped residual attention SGRA module;

[0055] (6.1) As Figure 4As shown, a strip-grouped residual attention (SGRA) module is constructed. Its structure is designed to efficiently capture long-range contextual dependencies across image regions while maintaining sensitivity to local details. The specific construction steps of this module are as follows: First, the maximum and minimum channel values ​​of the input feature map are calculated, and range and fusion operations are performed. Then, the output is simultaneously input to three parallel processing branches to calculate the self-attention values ​​Q, K, and V, respectively. The same structure is used when calculating self-attention Q, K, and V. First, the input is copied, and then input into a two-branch structure. The first branch is a linear layer, with a skip connection between the linear layer and the input. The second branch contains a strip pooling module and a linear layer, used to calculate the vertical strip-shaped downsampling of the input features. Then, the two branches are multiplied pixel-by-pixel to obtain Q, K, and V, respectively.

[0056] (6.2) The obtained Q is reshaped and transposed, and the resulting feature map is multiplied pixel by pixel with K. The result is then reshaped and transposed again and processed by the activation function. Finally, it is multiplied pixel by pixel with V to obtain the output of SGRA. The features are further enhanced and contextualized by the bar-grouped residual attention module, and the output of the bar-grouped residual attention module is input into the gate unit of EFD to fuse the features of the SGRA module with the features of the EFD module.

[0057] (7) Update the teacher network synchronously;

[0058] (7.1) Input Xs directly into the student network, and update the weights of the student network to the teacher network synchronously using the exponential moving average (EMA) mechanism. For example... Figure 2 As shown, the overall network adopts a teacher-student architecture, where the student network processes the source domain data, while the teacher network dynamically synchronizes weights through an exponential moving average (EMA) mechanism to ensure training stability. The formula for updating the teacher network using EMA is:

[0059]

[0060] Among them, teacher network weight Student network weight Updated via exponential moving average. For the teacher network weights in phase t-1, For the teacher network weight in phase t, For student network weights in phase t, For student network weight adjustment coefficient;

[0061] (8) Calculate the consistency losses Lt and Ls;

[0062] (8.1) Mix Xs with Xt obtained from sampling the target domain to generate a mixed image Xm. Input Xm into the student network, calculate the consistency loss Lt between the segmentation results obtained and yt obtained from Xt, and calculate the supervision loss Ls between Xm and the segmentation result label ys obtained from Xs. The prediction result of the mixed feature Xm in the student network is: The prediction of the target domain sampling feature Xt on the teacher network is: The prediction of source domain sampling features Xs on the student network is: Consistency loss is defined as:

[0063]

[0064]

[0065] Finally, Lt and Ls are used for model training.

Claims

1. A semantic segmentation method for low-light rainy images based on efficient frequency domain decomposition and strip grouped residual attention EFD-SGRA, characterized in that, Includes the following steps: (1) Construct a low-light rain dataset NightCity-rain; We constructed a low-light rain dataset, NightCity-rain, based on the public dataset NightCity, and used Cityscapes as the source domain training dataset and NightCity-rain as the target domain training dataset. (2) Construct teacher network a and student network b; Construct a teacher network a and a student network b. Network a samples the target domain data using rare categories, and network b samples the source domain data using rare categories. (3) Input characteristics of teacher network and student network; Input the target domain sample Xt of network a into the teacher feature extraction network, and input the source domain sample Xs of network b into the student feature extraction network; (4) Blended images; The target domain sample Xt is mixed with the source domain sample Xs extracted by the b network to obtain a mixed sample Xm, which is then input into the student feature extraction network. (5) Construct an efficient frequency domain decomposition module EFD; The high-efficiency frequency domain decomposition module (EFD) includes the frequency domain decomposition and denoising module (FFDM), the denoising module (DeNoise), and the hybrid differential convolution module (MDCM). (6) Construct a strip-grouped residual attention SGRA module; A strip-grouped residual attention (SGRA) module is constructed. Its structure is designed to efficiently capture long-range contextual dependencies across image regions while maintaining sensitivity to local details. The output of the SGRA module is input into the gated unit of the EFD module, and the features of the SGRA module are fused with the features of the EFD module. (7) Update the teacher network synchronously; Xs is directly input into the student network, and the weights of the student network are synchronously updated to the teacher network through the exponential moving average (EMA) mechanism. (8) Calculate the consistency losses Lt and Ls; The target domain samples Xs and Xt are mixed to generate a mixed image Xm. Xm is input into the student network, and the consistency loss Lt between the segmentation results obtained and the segmentation results yt obtained from Xt is calculated. The supervision loss Ls between the segmentation results obtained from Xs and the label ys is calculated.

2. According to the semantic segmentation method for low-light rainy images based on efficient frequency domain decomposition and strip grouping residual attention EFD-SGRA described in claim 1, the specific process of step (2) is as follows: (2.1) Construct a teacher network (a) and a student network (b) with identical structures. The teacher network (a) is dedicated to processing unlabeled target domain data, while the student network (b) is dedicated to processing finely labeled source domain data. Network a samples the target domain data using rare class sampling, and network b samples the source domain data using rare class sampling. Both networks contain an encoder consisting of a Vision Transformer (ViT) block, an Efficient Frequency Decomposition Module (EFD Module), and a Strip Grouped Residual Attention Module (SGRA Module) connected in series, and a decoder for outputting the final segmentation prediction result.

3. According to the semantic segmentation method for low-light rainy images based on efficient frequency domain decomposition and strip grouping residual attention EFD-SGRA described in claim 1, the specific process of step (5) is as follows: (5.1) Input the source domain features Xs and the target domain features Xt into the four stages of the Vision Transformer ViT block; (5.2) The features extracted by the ViT block in the four stages are sequentially input into the EFD to extract frequency domain information; (5.3) Construct a frequency domain decomposition and denoising module (FFDM); (5.4) Construct the DeNoise module; (5.5) Construct the Hybrid Differential Convolution Module (MDCM).

4. According to the semantic segmentation method for low-light rainy images based on efficient frequency domain decomposition and strip grouping residual attention EFD-SGRA described in claim 1, the specific process of step (6) is as follows: (6.1) Construct a strip-grouped residual attention SGRA module. Its structure is designed to efficiently capture long-range contextual dependencies across image regions while maintaining sensitivity to local details. The specific construction steps of this module are as follows: First, calculate the maximum and minimum channel values ​​of the input feature map and perform range and fusion operations. Then, input the output to three parallel processing branches to calculate the values ​​of self-attention Q, K, and V respectively. (6.2) The obtained Q is reshaped and transposed, and the resulting feature map is multiplied pixel by pixel with K. The result is then reshaped and transposed again and processed by the activation function. Finally, it is multiplied pixel by pixel with V to obtain the output of SGRA. The features are further enhanced and contextualized by the bar-grouped residual attention module, and the output of the bar-grouped residual attention module is input into the gate unit of EFD to fuse the features of the SGRA module with the features of the EFD module.

5. According to the semantic segmentation method for low-light rainy images based on efficient frequency domain decomposition and strip grouping residual attention EFD-SGRA described in claim 1, the specific process of step (7) is as follows: (7.1) Input Xs directly into the student network and update the weights of the student network to the teacher network synchronously through the exponential moving average (EMA) mechanism.

6. According to claim 1, a semantic segmentation method for low-light rainy images based on efficient frequency domain decomposition and strip grouping residual attention EFD-SGRA, the specific process of step 8 is as follows: (8.1) Mix the target domain sample Xs with the target domain sample Xt to generate a mixed image Xm. Input Xm into the student network and calculate the consistency loss Lt between the segmentation result obtained and the segmentation result yt obtained from Xt. Calculate the supervision loss Ls between the segmentation result obtained from Xs and the label ys.

Images