Multi-scale pulse gated image rain removal method and system with biological inspiration

Abstract

The invention discloses a multi-scale pulse gated image rain removal method and system with biological inspiration, and the method comprises the steps: firstly obtaining a synthetic data set of a rain removal task of a single image, and carrying out the preprocessing; secondly, a bionic driving color antagonism feature extraction mechanism is introduced, separation and enhanced expression of color degradation information in a rainy day image are achieved, and color antagonism features are obtained; then performing structure enhancement and multi-scale abstraction on the color antagonism features through bionic visual cortex multi-scale coding, and constructing visual cortex feature representation; and finally, for visual cortex feature representation, proposing a bionic visual cortex dual-channel prediction structure and a hybrid coding reconstruction structure, and outputting an image rain removal result graph. And constructing a multi-target mixed loss function, and carrying out reverse training and testing. According to the invention, the color information and the brightness information are accurately separated, the interference of background noise on the target color is reduced, and image rain removal is accurately and efficiently realized.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and image restoration technology, and specifically relates to a bio-inspired multi-scale pulse-gated image deraining method and system. Background Technology

[0002] Image deraining is an important low-level image restoration task in computer vision, aiming to recover a clear, rain-free background scene from a single image disturbed by rain. This technology has key value in practical applications such as autonomous driving, video surveillance, remote sensing imaging, and intelligent security, because raindrops, rain streaks, or fog can severely reduce image contrast, blur edge details, and introduce unstructured noise, thus significantly affecting the performance of high-level vision tasks such as object detection, semantic segmentation, and scene understanding.

[0003] Because rain streaks are multi-scale, multi-directional, and non-uniformly distributed, and traditional networks lack the ability to jointly perceive local structure and global context, they often misclassify high-frequency image details (such as textures, edges, and small-scale object contours) as noise and filter them out while removing dense or intersecting rain streaks, resulting in structural distortion in the derained image. Furthermore, most methods fail to adequately consider the restoration mechanisms of color constancy and perceptual consistency when dealing with raindrop contamination coupled with brightness and color, leading to color casts or abnormal local contrast in the deraining results, severely affecting visual realism and the reliability of subsequent visual tasks. Although deep learning models continue to achieve performance breakthroughs, their "black box" nature makes network behavior difficult to understand and attribute, limiting the model's credibility and guidance for further optimization. Biological visual systems, through long-term evolution, possess efficient multi-channel information separation, antagonistic processing, and hierarchical prediction mechanisms. For example, color antagonistic cells in the retina can effectively distinguish color contrasts, and orientation-selective neurons in the primary visual cortex (V1 area) can accurately respond to edges and stripes in specific directions—these mechanisms precisely correspond to the need to remove rain streaks as directional interference. Therefore, accurately preserving and restoring high-frequency details and natural textures of images while removing complex rain patterns has become a core challenge in improving the practicality of deraining. Simultaneously, incorporating the computational principles of biological visual systems to construct deraining models with clear neural mechanism explanations not only helps improve the robustness and generalization ability of algorithms but also provides an important path for the development of explainable artificial intelligence. There is an urgent need for a novel image deraining framework that can achieve high-quality detail reconstruction and has clear biological mechanism support to overcome the dual bottlenecks of current technology in terms of restoration accuracy and cognitive rationality. Summary of the Invention

[0004] To address the problems of existing image deraining techniques, such as weak local modeling capabilities, lack of dynamic adaptive mechanisms, and insufficient multi-scale feature fusion, this invention proposes an image deraining method and system based on visual pathway color antagonism and primary visual cortex mechanisms. This aims to solve the problems of incomplete deraining, severe detail loss, and lack of biological interpretability in complex rainy scenes using existing techniques. This method simulates the information processing mechanism from the retina to the primary visual cortex (V1 area) in biological visual systems, constructing a deep learning network architecture with clear neuroscientific basis: First, a color antagonistic encoding mechanism is used to convert the input image to red-green, yellow-blue, and brightness channels, effectively separating color pollution caused by raindrops from background structural information and enhancing the perceptual sensitivity to rain patterns. Second, multi-scale Gabor convolution and direction-selective filtering are used to simulate the directional response characteristics of V1 area neurons to edges and stripes, accurately capturing rain pattern structures at different directions and scales. This invention introduces a multi-pathway fusion prediction structure to simulate the functional division of the ventral and dorsal pathways, enabling collaborative modeling of image semantic content and spatial structural information. This efficiently removes rain streaks while preserving the texture details and structural integrity of the original image to the maximum extent. Finally, by combining impulse activation, positional encoding, and a hybrid loss function, end-to-end fine-grained restoration is achieved. This invention not only outperforms existing methods in rain removal performance but also possesses good biological rationality and interpretability, providing a new biomimetic computational model for image restoration tasks.

[0005] In one aspect, this invention provides a bio-inspired multi-scale pulse-gated image deraining method, comprising the following steps:

[0006] Step 1: Obtain the synthetic dataset for the single-image deraining task.

[0007] Step 2: This invention introduces standardized data preprocessing and robustness enhancement mechanisms at the input end to eliminate distributional differences in image data and expand the diversity of training samples. Specifically, the input rainy images and their corresponding ground truth labels without rain are normalized and standardized at the pixel level. By uniformly allocating the numerical range of pixel values ​​and aligning the distribution based on channel statistical characteristics, data offsets caused by differences in acquisition equipment and lighting conditions are eliminated, ensuring the stability of model training.

[0008] Furthermore, during the training phase, spatial geometric transformation and random cropping strategies are introduced. By synchronously flipping, rotating, and sampling local regions of image pairs, the varying spatial distribution and imaging perspective of rain streaks in real-world scenes are simulated, significantly improving the model's adaptability and generalization performance to different rain streak morphologies. In the inference and evaluation phases, a deterministic processing procedure is adopted to ensure the objectivity and consistency of the rain removal results.

[0009] Step 3: Based on image preprocessing, this invention introduces a biomimetic-driven color antagonistic feature extraction mechanism to simulate the color processing mechanism of the retina-cortex pathway in the human visual system, thereby achieving efficient separation and enhanced expression of color degradation information in rainy day images.

[0010] Specifically, the input RGB image is mapped to a biologically plausible feature space through a learnable color antagonistic transformation, generating three biomimetic response channels: a red-green antagonistic channel, a blue-yellow antagonistic channel, and a brightness channel. This decouples the color contrast and brightness information in the image, highlighting the color difference between the rain streaks and the background.

[0011] In the feature extraction process, a local receptive field mechanism is employed. By applying learnable spatial convolution kernels to the RGB three channels, the color responses within the neighborhood of each pixel are weighted and aggregated to generate an initial antagonistic feature map. Subsequently, a nonlinear activation and bidirectional response modeling strategy is introduced: on the one hand, the ReLU function is used to extract the positive activation response from the feature map, preserving the enhanced portion of color difference; on the other hand, the negative response portion is subjected to absolute value transformation and an adjustable suppression coefficient is introduced to generate an inhibitory response component. By differentially fusing the positive and inhibitory responses, a direction-selective color antagonistic response is constructed, enhancing the model's ability to perceive raindrop edges and texture structures.

[0012] To further capture color contrast features at different scales, this invention employs multi-scale convolutional kernel groups to process color antagonism responses in parallel, extracting color antagonism features at small, medium, and large scales respectively. Each feature map at each scale is assigned a learnable weight parameter to dynamically adjust its contribution to the overall feature representation. Through weighted fusion, the collaborative representation of local details and global structural information is achieved, generating a multi-scale color antagonism response map rich in hierarchy.

[0013] To further improve the stability and contrast adaptability of the features, local contrast normalization is performed on the multi-scale color antagonism response map to obtain color antagonism features: with each pixel as the center, the sum of the absolute values ​​of the response intensity in its neighborhood is calculated as a local normalization factor. The original response value is divided by this factor to achieve adaptive gain control of the feature map, thereby effectively suppressing the over-enhancement phenomenon in strong contrast areas and preserving the visibility of weak contrast details.

[0014] The color antagonistic feature extraction module constructed in this step not only simulates the color processing mechanism of biological vision and has good interpretability, but also significantly enhances the network's ability to model chromatic degradation in complex rainy scenes through multi-scale fusion and adaptive normalization, providing a highly discriminative input representation for the subsequent rain streak removal module based on the primary visual cortex mechanism.

[0015] Step 4: This invention provides a biomimetic multi-scale encoding method for the visual cortex, aiming to enhance the structure and abstract color antagonistic features at multiple scales, simulating the hierarchical processing mechanism of visual information in the primary visual cortex (V1 area). This method gradually constructs a high-level feature representation with direction selectivity and structural robustness through multi-directional structural perception, multi-scale context modeling, neural sparse response, and dynamic feature fusion, effectively addressing the stripe characteristics and complex spatial distribution of rain patterns.

[0016] First, multi-directional selective filtering is performed. Multiple sets of direction-sensitive learnable convolutional kernels are used to extract the structural responses of input features in parallel along multiple typical directions, simulating the direction selectivity of V1 neurons for edges and stripes, capturing the dominant orientation information of rain patterns. The outputs from each direction are fused along the channel dimension and then integrated via 1×1 convolution to form a structural feature map with direction awareness. This map is then element-wise added to color antagonistic features through residual connections. Next, multi-scale receptive field modulation is implemented. The directional feature map is processed through multiple parallel branches: one path uses spatial convolution to extract local details and multiplies them element-wise with the input features through residual connections; the other two paths use dilated convolutions with different dilation rates to expand the receptive field and capture mid- to long-range contextual information; the features output by the two dilated convolutions are upsampled by pooling and then interactively weighted and fused, and the two weighted and fused features are then processed by a self-attention module (SE) and convolution operations to obtain the output features corresponding to the two paths; the outputs of the three paths are concatenated and fused into a multi-scale attention feature map, and then a spiking neural mechanism is introduced to achieve a sparse response. The multi-scale attention features are converted into a multi-time-step sequence through a pulse coding layer and input into a linear layer containing a spiking activation function to simulate the neuron integration-firing process and generate a sparse response map; this process uses a differentiable approximate gradient to participate in backpropagation, retaining high-confidence feature responses and suppressing low-significance redundant activations.

[0017] Finally, dynamic fusion of multi-source features is performed. The results of multi-scale attention features, sparse response features, and the original input features after random forgetting are weighted and fused point-by-point to generate the final robust visual cortex feature representation. This encoding module consists of five stacked levels. The outputs of the first four levels are downsampled by average pooling and input to the next level, while also serving as input to the visual cortex dual-path prediction module at the same level. The output of the fifth level is only used as input to the upper-level prediction module and does not participate in feature transfer at the same level, thus constructing a hierarchical, multi-scale feature representation system.

[0018] Step 5: This invention proposes a biomimetic visual cortex dual-pathway prediction structure, comprising a visual cortex dual-pathway prediction structure, a multi-pathway fusion prediction structure, and a hybrid coding reconstruction structure, to enhance the joint modeling capability of structure recognition and texture restoration in image deraining tasks. This structure simulates the functional division of labor between the ventral and dorsal pathways in the biological visual cortex, achieving accurate prediction and high-quality reconstruction of image content under rain pattern interference through parallel processing of spatial structural information and semantic detail information. The overall module consists of three parts: dual-pathway feature perception, multi-pathway fusion prediction, and hybrid coding reconstruction.

[0019] First, a dual-pathway prediction structure for the visual cortex is constructed. The multi-scale encoded output of the visual cortex at the same level and the input image from the upper layer are upsampled separately. The multi-scale encoded output of the visual cortex is restored to resolution through transposed convolution. The input image is aligned to spatial scale through bilinear interpolation, and then concatenated along the channel dimension. After a 1×1 convolution, a fused initial feature map is formed. This design achieves cross-level alignment between low-level details and high-level semantics, providing a unified input foundation for the subsequent multi-pathway fusion prediction structure. In the dorsal pathway, multi-directional structure perception is performed on the fused initial feature map. After converting it to grayscale representation, directional response modeling is performed using Gabor filters in multiple classic directions to simulate the selective response of neurons in region V1 to stripe directions. After weighted fusion of the responses in each direction, a multi-directional attention map is generated through 1×1 convolution and nonlinear activation. This attention map is then multiplied with the original feature map to enhance structural saliency. Finally, a linear layer is used to map and output the dorsal pathway perception map, strengthening the modeling ability of the spatial structure and directional distribution of rain streaks. In the ventral pathway, the focus is on the joint perception of local details and long-range dependencies. The first branch introduces a windowed self-attention mechanism, calculating feature responses within an 8×8 local window for the initial fused feature map. This is combined with convolutional operations using a cyclic shift strategy to expand the receptive field diversity and enhance the ability to capture local texture structures. The second branch constructs multi-scale convolutional branches, expanding the receptive field through dilated convolutions to capture contextual information. The output of this branch is then differentially analyzed with the output of the regular convolutional branch to highlight regions of feature change and enhance the discriminative power for subtle structures. The weighted fusion of the output features from both branches forms the ventral pathway perception map.

[0020] Subsequently, the perceptual maps output from the dorsal and ventral pathways are concatenated and their features are integrated using 1×1 convolutions to generate a dual-pathway fusion prediction result. This structure consists of four levels, receiving multi-scale encoded outputs from the visual cortex and upper-layer information inputs at each level to achieve multi-scale prediction. The output of the last level serves as the final perceptual prediction map, providing high-order semantic guidance for subsequent reconstruction.

[0021] Furthermore, a hybrid encoding reconstruction structure is designed to achieve high-fidelity image reconstruction. This module includes two differentiated processing paths: First, after processing the perceptual prediction map through linear layers and convolution, positional and coordinate encodings are introduced, and the two encoding results are concatenated to construct enhanced features containing spatial priors; Second, the input features are directly fused with positional and coordinate encodings, and the two encoding results are concatenated to form a baseline feature path. The output features of the two paths are differentially analyzed point by point to generate relative positional coordinate features, effectively enhancing the model's sensitivity to changes in spatial structure. After the relative positional coordinate features are added to the perceptual prediction map, a 1×1 convolution is performed for preliminary transformation, and a biologically interpretable nonlinear response is introduced through a pulse coding mechanism. The features are copied to multiple time steps and random noise is superimposed to simulate the temporal fluctuations of neural input. Subsequently, temporal processing is performed through a differentiable pulse activation function to retain highly significant responses and suppress redundant activations. Finally, the pulse response results are fused with the perceptual prediction map, and the final de-rained image is output through spatial convolution.

[0022] Step 6: This invention employs an end-to-end differentiable training framework and designs a quaternary hybrid loss function that integrates time-domain and frequency-domain awareness to jointly optimize the parameters of the entire network, achieving a synergistic improvement in performance, convergence, and stability of the rain removal model. This optimization strategy effectively guides the network to learn discriminative feature representations of rain patterns and background through an adaptive parameter update mechanism, dynamic learning rate scheduling, and multi-dimensional loss supervision.

[0023] Specifically, to construct a multi-objective hybrid loss function and achieve multi-dimensional supervision, the loss is composed of four complementary terms weighted together: First, Charbonnier loss is used as the basic pixel-level reconstruction term, which has good numerical robustness and effectively alleviates the gradient explosion problem; second, structure-aware loss is introduced to enhance the model's ability to model the high-level structure and semantic consistency of the image; third, Laplacian pyramid edge loss is designed to focus on the recovery of multi-scale edge structures and improve detail clarity; fourth, frequency domain Fourier loss is introduced to preserve the natural statistical characteristics of the image in the frequency space through spectral alignment constraints, avoiding artifact generation. These losses are weighted and fused using learnable or empirical weights to form a comprehensive optimization objective that fully guides the network parameter updates.

[0024] In another aspect, the present invention provides a bio-inspired multi-scale pulse-gated image deraining system, comprising the following modules:

[0025] The rain-inducing image acquisition module is used to acquire the synthetic dataset for the rain removal task of a single image and perform preprocessing.

[0026] The color antagonism feature extraction module is used to extract color antagonism features based on the preprocessed image by introducing a biomimetic-driven color antagonism feature extraction mechanism, thereby separating and enhancing the expression of color degradation information in rainy day images to obtain color antagonism features.

[0027] The visual cortex feature extraction module is used to enhance the structure and abstract the color antagonistic features through biomimetic multi-scale encoding of the visual cortex, and to construct a visual cortex feature representation with orientation selectivity and structural robustness.

[0028] The image deraining result output module is designed to enhance the ability of structure recognition and texture restoration by proposing a biomimetic visual cortex dual-pathway prediction structure and hybrid coding output mechanism based on visual cortex feature representation, and outputting the image deraining result map.

[0029] The training module is used to construct a multi-objective hybrid loss function, perform reverse training, and test it.

[0030] Beneficial effects of this invention:

[0031] 1. To address the issues of unstable color feature extraction and susceptibility to interference in traditional machine vision systems under complex lighting conditions, this invention draws upon the color antagonism mechanism of the biological retina and visual cortex. By simulating the processing of opposing color channels such as red-green and yellow-blue, it achieves efficient decoupling and encoding of color information. This mechanism effectively simulates the human eye's color perception characteristics, enhancing the system's ability to detect subtle color differences and significantly improving color constancy under varying lighting conditions or low-light environments. Compared to traditional direct processing of the RGB color space, this method can more accurately separate color and brightness information, reducing background noise interference with the target color, thereby improving the robustness and accuracy of color feature extraction and achieving accurate and efficient image deraining.

[0032] 2. To address the limitation of single-scale feature extraction in handling multi-scale targets or complex textures due to its inability to simultaneously capture both global image structure and local details, this invention introduces a multi-scale coding strategy based on the visual cortex. This method simulates the hierarchical processing mechanism of the brain's visual cortex for different spatial frequencies, constructing a coding network capable of simultaneously capturing broad contextual information and fine local features. Specifically, low spatial frequency information is integrated broadly through specific pathways to identify the object as a whole, while high spatial frequency information is enhanced for detail through local interaction mechanisms. This multi-scale parallel processing not only bridges the perceptual gap between macroscopic scenes and microscopic pixels but also enables the system to maintain excellent recognition and reconstruction capabilities when facing blurred, occluded, or distant targets.

[0033] 3. To address the problems of high computational redundancy and lack of dynamic adjustment of low-level features by high-level semantics in traditional feedforward neural networks, leading to low coding efficiency, this invention constructs a dual-pathway prediction structure and hybrid coding mechanism based on the characteristics of the visual cortex. This structure simulates the ventral and dorsal pathways, achieving decoupling and fusion of semantic and spatial information through collaborative computation of the two pathways. Finally, through hybrid coding output, sparse coding and predictive coding are combined, significantly reducing the energy consumption of data transmission and computation while ensuring information integrity, achieving efficient and low-power visual information processing similar to that of a biological brain. Attached Figure Description

[0034] Figure 1 This is a schematic diagram of the implementation process provided in the embodiments of the present invention;

[0035] Figure 2 This is a color antagonism feature extraction module provided in an embodiment of the present invention;

[0036] Figure 3 This is a visual cortex multi-scale coding module provided in an embodiment of the present invention;

[0037] Figure 4 This is a visual cortex dual-pathway prediction module provided in an embodiment of the present invention;

[0038] Figure 5 The hybrid encoding output module provided in this embodiment of the invention;

[0039] Figure 6 This is a visualization of the ablation experiments conducted on three different datasets based on the present invention. Detailed Implementation

[0040] To make the objectives, technical solutions, and advantages of this invention clearer, the specific embodiments of this invention will be described in detail below with reference to the accompanying drawings. It should be noted that the specific embodiments described herein are only for explaining this invention and are not intended to limit it. A schematic diagram of the implementation process of this invention is shown below. Figure 1 As shown below, the steps are explained in detail.

[0041] In one aspect, this invention provides a bio-inspired multi-scale pulse-gated image deraining method, comprising the following steps:

[0042] Step 1: Building the dataset

[0043] Obtain the synthetic datasets Rain200L, Rain200H, and DDN datasets for the single-image deraining task, and divide the datasets into training, validation, and test sets.

[0044] Preferably, the datasets Rain200L and Rain200H represent datasets synthesized from publicly available images and artificially generated sparse and dense rain, respectively. Rain200L and Rain200H each contain 1800 pairs of image data (rain-free and rainy) with a resolution of 481×321 for training and 200 pairs for testing.

[0045] Preferably, the dataset DDN represents a dataset synthesized from publicly available images and artificially generated multi-directional rain patterns. DDN contains 12,600 pairs of image data (rain-free and rainy) with a resolution of 384×512 for training and 1,400 pairs of image data (rain-free and rainy) for testing.

[0046] Furthermore, the method for dividing the training set into training set, validation set, and three other sets is as follows: the original dataset is randomly arranged and divided into training set and validation set in an 8:1 ratio.

[0047] Step 2: Image Preprocessing Stage

[0048] Before training and inference, all input rainy images and their corresponding rainless labeled images undergo uniform preprocessing. First, the image pixel values ​​are linearly normalized from the original integer range of 0 to 255 to floating-point numbers between 0 and 1. Then, for each dataset, the global mean and standard deviation of its red, green, and blue channels are calculated separately, and the pixel values ​​of each channel of each image are standardized by subtracting the corresponding channel mean and dividing by its standard deviation. This eliminates differences in illumination and contrast between different datasets and improves the model's generalization ability.

[0049] During the training phase, to further enhance the model's robustness, data augmentation strategies were introduced: image pairs were randomly horizontally flipped with a 50% probability, and synchronous rotations of 0°, 90°, 180°, or 270° were randomly selected with a 25% probability; additionally, all training images were randomly cropped into 128×128 sub-blocks as network input. During the validation and testing phases, only normalization and standardization operations were performed, without any geometric transformations or cropping, to ensure the objectivity and comparability of the evaluation results. This preprocessing mechanism provides a high-quality, robust input representation foundation for subsequent feature extraction and rain removal reconstruction based on biological visual mechanisms.

[0050] Step 3: Color Antagonism Feature Extraction Stage

[0051] like Figure 2 As shown, the preprocessed RGB image is transformed to a color antagonistic space, outputting feature maps for three biomimetic channels: red and blue channels. Blue-Yellow Channel and brightness channel .

[0052]

[0053] in This represents a 5×5 local receptive field; For the first The first confrontation channel Spatial convolution kernel weights on each color input, It is the first The output on a biomimetic channel It is the index of the selected channel; These are image coordinates; These are the coordinates of the convolution kernel. (The result is...) , , The merged result is the calculated feature map.

[0054] The ReLU activation function is used to retain all values ​​greater than zero in the feature map (i.e., the positive color difference portion), while setting the less than zero portion to zero to obtain the positive response of the feature map. Then, the less than zero portion in the original feature map is inversely absolute and multiplied by a suppression coefficient (0.3 in this paper) to form a suppressed response to the inverse color.

[0055] Subtracting the positive response of the feature map from the suppression response of the inverse color yields a color antagonism response with stronger directional selectivity.

[0056] Three antagonistic convolutional kernels of different sizes (3×3, 5×5 and 7×7) were used to process the color antagonistic response in parallel, and color antagonistic features at three different scales were extracted respectively.

[0057] A learnable weight coefficient (which changes as the model is trained) is assigned to the feature map at each scale to represent the importance of the current scale in overall perception. Then, the feature maps at all scales are weighted and fused to obtain a multi-scale color antagonism response map that contains both local details and global structural information.

[0058] The local contrast of the generated multi-scale color antagonism response map, which contains both local details and global structural information, is normalized, and the feature map is adaptively adjusted.

[0059] Specifically, the absolute values ​​of the response values ​​within a 3×3 neighborhood of each pixel are summed to obtain the overall response intensity of that region. Then, the original response value of the current pixel is divided by this local total response intensity, resulting in a normalized color antagonism feature.

[0060] For example, this paper uses 128×128 sub-blocks with 3 channels and a batch size of 4 as training inputs (i.e., the input tensor size of the training model is [4,3,128,128]). The tensor size remains unchanged after the pixels undergo antagonistic spatial transformation and suppression response. After spatial convolutions with edge padding of 1 and a size of 3×3, edge padding of 2 and a size of 5×5, and edge padding of 3 and a size of 7×7, the tensor size of the input image remains unchanged. In this paper, the edge padding for local contrast normalization is 1, therefore the tensor size of the processed color antagonistic features is also [4,3,128,128].

[0061] Step 4: Visual Cortex Multiscale Coding Module

[0062] In this embodiment, to effectively model the stripe characteristics of rain streaks in different directions and enhance the model's ability to jointly perceive local structure and global context, the encoder section introduces a visual cortex multi-scale coding module inspired by the multifunctionality of the primary visual cortex (V1 area), such as... Figure 3 As shown, this module progressively constructs a feature representation with strong directional selectivity and structural robustness through four stages: multi-directional selection, multi-scale receptive field modulation, neural sparse response, and cortical inhibition.

[0063] Step 41: The input image first undergoes multi-directional selection processing. Four orientation-sensitive 2D Gabor convolution kernels are used in parallel, each corresponding to a typical orientation angle. (level), , (Vertical) and Each Gabor convolution kernel is composed of a sine function modulated by a Gaussian envelope in the spatial domain, which can produce significant responses to stripes, edges, and repetitive texture structures in the image in specific directions. Through the parallel operation of these four sets of convolution kernels, the model obtains the structural response maps of the input image in four main directions, thereby accurately capturing the dominant directional trend of raindrop texture.

[0064] Specifically, a 7×7 learnable convolutional kernel with 3-padded edges is designed for each direction, and the number of output channels is set to one-quarter of the total feature dimension. Unlike general convolution, the Gabor convolutional kernel is generated by Gabor functions in different directions. The output feature maps of the four directions are concatenated along the channel dimension to form a fused feature tensor. This tensor integrates multi-directional structural information to form a structure-aware feature map with direction selectivity. To integrate information from multiple directions, the 4-channel input is channel-integrated through a 1×1 convolutional layer, and then element-wise added to the color antagonistic features through residual connections to form a structure-aware feature map with direction awareness. Finally, the direction feature extraction map is obtained by element-wise adding to the color antagonistic features through residual connections.

[0065] Step 42 aims to improve the model's contextual understanding capabilities over a larger spatial range. In this embodiment, the directional feature extraction map output in step 41 is modulated with multi-scale receptive fields to enhance its features at different scales across three parallel channels.

[0066] In the first channel, a 3×3 spatial convolution is used to extract local details from the input tensor, and the result of the first channel is obtained by element-wise multiplication with the input features through residual connections.

[0067] The features output from the two dilated convolutions are upsampled via pooling and then interactively weighted and fused. Specifically, in the second channel, the input tensor undergoes the first type of dilated convolution, namely a 3×3 dilated convolution with a dilation rate of 2 and an edge padding of 2. The resulting receptive field modulated image is called... After pooling, the bilinear interpolation performed on the image is called image interpolation. In the third channel, a second type of dilated convolution is performed on the input tensor, namely a 3×3 dilated convolution with a dilation rate of 4 and an edge padding of 4. The resulting receptive field modulation image is called... The image after pooling and bilinear interpolation is called the bilinear interpolated image. .

[0068] Subsequently, the two weighted and fused features are processed by the self-attention module (SE) and convolution operations to obtain the output features corresponding to the two paths. Specifically, and The fused graph is obtained by performing point-by-point weighted summation. ,Will and The fused graph is obtained by performing point-by-point weighted summation. The weights are learnable weights. For example, in this step, the weights are initialized so that all elements of the weight tensor are 0.5, and then continuously changed according to backpropagation. The obtained... and Each channel map is processed through an SE module and a 1×1 convolution, resulting in the following channel maps: and

[0069] Will , and The outputs of the three paths are concatenated and fused into a multi-scale attention feature map.

[0070] Step 43: Although the features processed by multi-scale attention already possess good structural sensitivity, the problem of redundant activations and excessive participation of low-confidence responses in the fusion still needs to be addressed. The obtained multi-scale attention feature map is processed through a pulsed linear layer to obtain a sparse response map. The pulsed linear layer is essentially a pulse coding layer and a linear layer using a pulsed activation function, with an input layer size of 3, an output layer size of 3, and a hidden layer size of 256. The pulsed activation function chosen is the "Leak-Integrate-Release" (LIF) model. The alternative gradient of the pulsed activation function is as follows:

[0071]

[0072] in Represents the input quantity of the next level in the computation graph. For impulse activation function gradient, Threshold voltage

[0073] The pulse coding layer flattens the input image to two dimensions (including the batch size), replicates it for T steps (for example, the step size of the spiking neural network is set to 4 in this paper), adds a random perturbation generated by the Poisson function at each step, and then merges them into a multi-time-step image sequence input to a linear layer using the pulse activation function. After the linear layer operation, the image is restored to its original dimension and a sparse response map is output.

[0074] Step 44 involves weighting the multi-scale attention feature map output in step 42, the sparse response map output in step 43, and the original image after random forgetting point by point to obtain the final output multi-scale coding map of the visual cortex in step 4.

[0075] Furthermore, starting with the feature map output from step three, the process goes through a total of five levels of visual cortical multi-scale coding modules. After the outputs from levels 1-4 of the visual cortical multi-scale coding modules, the output results are subjected to 2×2 average pooling with a stride of 2 to obtain a downsampled result, which is then input into the next level of the visual cortical multi-scale coding module. Simultaneously, the output results are used as the input image for the same level of the visual cortical dual-path prediction module. After inputting an image of size [4,3,128,128] in step 3, the tensor size remains unchanged after processing in step 41. After the first channel processing in step 42, the input size is [4,1,128,128]. After the first type of dilated convolution processing, the tensor size remains unchanged, but the pooled result... and The image will become [4,3,64,64] (with pooling size set to 2×2 and stride to 2). Therefore, bilinear interpolation upsampling is needed to scale the image to [4,3,128,128]. After convolution, the output tensors of the second and third channels should be [4,1,128,128]. The result after merging the three channels is [4,3,128,128]. Subsequent operations and weighting obviously do not change the tensor size, so no special explanation is given.

[0076] Unlike the outputs of levels 1-4, the output of the level 5 visual cortex multiscale coding module is only used as the upper-level image input of the level 4 visual cortex dual-path prediction module (step 5), and no longer becomes the same-level input image of the visual cortex dual-path prediction module of any level.

[0077] Step 5: Bionic visual cortex dual-pathway prediction module and hybrid coding output

[0078] To further enhance the model's capabilities in structural recognition and texture reconstruction, this step proposes a biomimetic visual cortex dual-pathway prediction module, such as... Figure 4 As shown, this structure includes a visual cortex dual-pathway prediction structure, a multi-pathway fusion prediction structure, and a hybrid coding reconstruction structure.

[0079] Step 51: The visual cortex dual-path prediction structure processes spatial structure and semantic detail information in the image using two complementary paths. By parallel extraction and fusion of these two types of features, more accurate image content prediction and rain pattern removal are achieved. The visual cortex multi-scale coding output from Step 4 serves as the input image for this sub-step, denoted as visual cortex multi-scale coding output A, while simultaneously receiving the upper-layer input image, denoted as input image X. This step first upsamples both the visual cortex multi-scale coding output A and the input image X. The visual cortex multi-scale coding output A is then subjected to a transposed convolution with a kernel size of 4×4, a stride of 2, and edge padding of 1 to obtain the image. Simultaneously, the input image X is bilinearly interpolated in parallel to obtain the image. Then the image and images Channel merging is performed, and the resulting 1×1 convolution forms the initial fused feature map. .

[0080] Step 52: Multi-path fusion prediction structure: fuse the initial feature maps Converted to grayscale, and passed in parallel through four directions corresponding to typical orientation angles. (level), , (Vertical) and The Gabor function is used. It's important to distinguish this step from step 41; in this step, Gabor filtering is not Gabor convolution, but rather Gabor filtering is applied to the entire image in different directions. After generating the filtered images for each direction, they are weighted and their channels are merged. The merged image is then convolved with a 1×1 convolution and activated using the ReLU function to obtain a multi-directional merged attention map. The merged attention map is then compared with the input feature map. After multiplication, input a linear layer with 128 hidden layers.

[0081] Specifically, the linear layers mentioned in this step and other subsequent steps all involve first flattening the input vector to two dimensions (the first dimension being the batch size), feeding it into the linear layer, and then restoring the original vector to its original dimensions. This operation is somewhat repetitive and will not be mentioned again later.

[0082] The linear layer output yields the back-side pathway sensing map. .

[0083] Step 53: In parallel, in the ventral pathway, the first branch will fuse the initial feature maps. The input is an 8×8 window attention module, where feature maps are fed in parallel into 8×8 window self-attention units. Each unit employs a cyclic shift mechanism to ensure diversity of local receptive fields and calculates its own feature response. The calculation method for the window sub-attention branches is as follows:

[0084]

[0085] in It outputs an image. This refers to the window size. In this article, the window size is 8. Let the output size of this module in this article be . . It's a merge operation. It's a loop translation operation, a window attention module operation. This refers to adding the output of a depthwise convolution (DWConv) to the Value (V) of a classic attention module and combining it with the output of the classic attention module. The calculation method is as follows:

[0086]

[0087] in It is a linear layer operation. It is the softmax activation function. , , These are three methods that utilize the input tensor Vectors generated by linear projection Input quantity Location encoding, It is a depthwise convolution. The output tensor is input into a 3×3 convolution module to obtain channel one output. .

[0088] In parallel, within the ventral pathway, the second branch constructs multi-scale convolutional branches, including convolutional branches and dilated convolutional branches. The dilated convolutional branch converts the feature map... Perform a dilated convolution with a kernel size of 5, edge padding of 4, and dilation of 2. Then, perform a 3×3 convolution to obtain the result. The convolutional branch directly converts the feature map... The result is obtained by performing a 3×3 convolution. .Will and Perform a differential operation to obtain the output of channel two. The output features of the two branches are weighted and fused to obtain the ventral pathway perception map. .

[0089] Step 53: Obtain the dorsal pathway sensing map And ventral pathway perception map The images are stitched together and then subjected to a 1×1 convolution operation to obtain the dual-path fusion prediction result.

[0090] Hybrid Coding Reconstruction Structure: This structure comprises four levels, receiving multi-scale coded outputs from the visual cortex and upper-level information inputs at each level to achieve multi-scale prediction. For example, at level four, the coded image input at level four is the output of the level four visual cortex multi-scale coding module. The upper-level input images from levels three to one are the output of the level five visual cortex multi-scale coding module. Specifically, the input to the level four visual cortex dual-path prediction module is the output of the level five visual cortex multi-scale coding module. Furthermore, the output of level one, the final level, is taken as the final perceptual prediction map and denoted as the perceptual prediction map. Continue with the input image set in step 2.

[0091] Step 54 Hybrid Encoding Output Module: To achieve high-fidelity reconstruction of the derained image and effectively fuse the perceptual prediction map input from the biomimetic visual cortex dual-pathway prediction module. To address the multi-level feature information carried, this step designs a hybrid encoding output module, such as... Figure 5 As shown, this module uses two differentiated feature processing channels to perform depth transformation and positional information fusion on the input features, respectively. Finally, through a pulse activation mechanism and linear mapping, it generates the final de-raining image output. The core of this module lies in enhancing the discriminative ability of features through differential channel design and improving spatial perception accuracy by merging positional encoding and coordinate encoding.

[0092] Differential Channel 1: The perceptual prediction map output from step 5 The input is fed into the compressed convolution module. Essentially, the compressed convolution module processes the feature map through linear layers and convolutions. It begins with a linear layer that performs preliminary feature space mapping to enhance its expressive power. A 3×3 convolutional layer with 1 padding is then applied to the feature map after the linear layer transformation. This operation primarily adjusts the channel dimensions of the feature map and further fuses information between channels to obtain a channel-integrated feature map. Subsequently, positional encoding and coordinate encoding are performed separately, and the two encoding results are concatenated channel-wise to obtain the final feature map.

[0093] Differential Channel 2: Directly perform position encoding and coordinate encoding fusion on the perceptual prediction map, and concatenate the two encoding results to form the baseline feature path. Feature map obtained from differential channel 1 Feature map obtained from differential channel 2 The relative position coordinate features are obtained by performing point-by-point difference.

[0094] The relative position coordinate features obtained after step 55 differencing With perception prediction graph The input is added together, and after a 1×1 convolution, pulse coding is performed. A biologically interpretable nonlinear response is introduced through the pulse coding mechanism. The features are copied to multiple time steps and random noise is superimposed to simulate the temporal fluctuations of neural input. Then, temporal processing is performed through a differentiable pulse activation function.

[0095] Specifically, the input graph is flattened and copied to four time steps, and random Poisson noise is added to obtain a time series. The time series is then pulse-activated using LIF across four time steps.

[0096] Finally, the impulse response results are compared with the sensing prediction map. The inputs are summed, and the final derained image is output after a 3×3 spatial convolution.

[0097] Step 6: Optimize network parameters.

[0098] This invention employs an end-to-end approach, constructing a quaternary hybrid loss that fuses the time and frequency domains to train the entire network. The specific training parameters are set as follows:

[0099] The optimizer used was Adam, with an initial learning rate of 0.0002 and a weight decay coefficient of 0.05. The learning rate was scheduled using a cosine annealing strategy, which smoothly decayed from the initial value over 600 training epochs. The training batch size was set to 16, and the input image was randomly cropped into 128×128 sub-blocks during training.

[0100] The loss function consists of four weighted components: Charbonnier loss with a weight of 1.0, structure-aware loss with a weight of 0.1, Laplacian pyramid edge loss with a weight of 0.05, and frequency domain Fourier loss with a weight of 0.01. After each training batch, the data-augmented de-rained image and the final de-rained image are used as inputs to each loss function to calculate the four loss terms, which are then summed according to their weights.

[0101] During training, the peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) are evaluated on the validation set every 50 rounds, and the current best model is saved.

[0102] All experiments were conducted on a single NVIDIA RTX 4090D GPU, with an average training time of approximately 130 seconds per round. The above parameter combination has been verified through extensive experiments to achieve the best balance between rain removal performance, training stability, and convergence speed.

[0103] Step 7: Train the model on the training set obtained in Step 1. Evaluate the model training performance on the validation set after each round, and optimize the model parameters using the loss function from Step 6. Evaluate the final training results using the test set, and output the visualized polyp segmentation results on the test sets of the Rain200H, Rain200L, and DDN datasets. The results of the ablation experiments are shown in Table 1, and the visualized results are as follows: Figure 6 As shown, the ablation experiments were compared by eliminating steps four and five.

[0104] Table 1 Ablation on different datasets

[0105] The table above and experimental results show that the visual cortex multi-scale coding module, the biomimetic visual cortex dual-path prediction module, the hybrid coding output and the quadruple composite loss function proposed in this invention are not simply stacked isolated components, but achieve deep coupling and dynamic complementarity through a bio-inspired feedforward-feedback collaborative mechanism, thereby producing a synergistic enhancement effect in multiple dimensions such as rain removal accuracy, detail preservation and computational efficiency.

[0106] In another aspect, the present invention provides a bio-inspired multi-scale pulse-gated image deraining system, comprising the following modules:

[0107] The rain-inducing image acquisition module is used to acquire the synthetic dataset for the rain removal task of a single image and perform preprocessing.

[0108] The color antagonism feature extraction module is used to extract color antagonism features based on the preprocessed image by introducing a biomimetic-driven color antagonism feature extraction mechanism, thereby separating and enhancing the expression of color degradation information in rainy day images to obtain color antagonism features.

[0109] The visual cortex feature extraction module is used to enhance the structure and abstract the color antagonistic features through biomimetic multi-scale encoding of the visual cortex, and to construct a visual cortex feature representation with orientation selectivity and structural robustness.

[0110] The image deraining result output module is designed to enhance the ability of structure recognition and texture restoration by proposing a biomimetic visual cortex dual-pathway prediction structure and hybrid coding output mechanism based on visual cortex feature representation, and outputting the image deraining result map.

[0111] The training module is used to construct a multi-objective hybrid loss function, perform reverse training, and test it.

[0112] The above description is only a partial embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent modifications, structural substitutions, or parameter adjustments that can be easily conceived by those skilled in the art within the scope of the technical ideas and framework disclosed in the present invention should be considered to fall within the scope of protection of the present invention.

Claims

1. A biologically inspired multi-scale pulse-gated image deraining method, characterized in that, The method comprises the following steps: Step 1: obtain a synthetic dataset of a single image rain removal task, and perform preprocessing; Step 2: based on the preprocessed image, introduce a biomimetic driving color antagonism feature extraction mechanism to separate and enhance the expression of color degradation information in the rainy image, and obtain color antagonism features; Step 3: through biomimetic visual cortex multi-scale coding, the color antagonism features are structurally enhanced and multi-scale abstracted to construct a visual cortex feature representation with direction selectivity and structural robustness; Step 4: for the visual cortex feature representation, a biomimetic visual cortex double-channel prediction structure and a hybrid coding reconstruction structure are proposed to enhance the structure recognition and texture restoration capability, and output the image rain removal result image; Step 5: a multi-objective hybrid loss function is constructed for back training and testing.

2. The bio-inspired multi-scale pulse-gated image deraining method according to claim 1, wherein, The preprocessing includes pixel-level normalization and standardization processing of the rainy image and its corresponding rain-free true value label; The spatial geometric transformation and random cropping strategy are introduced to simulate the variable rain streak spatial distribution and imaging viewing angle in the real scene by synchronously flipping, multi-angle rotating and local region sampling of the image pair.

3. The bio-inspired multi-scale pulse-gated image deraining method of claim 1, wherein, The specific implementation process of step 2 is as follows: The input RGB image is mapped to a biological feature space through a learnable color antagonism transformation to generate three biomimetic response channels: red-green antagonism channel, blue-yellow antagonism channel and brightness channel, decoupling color contrast and brightness information in the image and highlighting the chrominance difference between rain streaks and background; In the feature extraction process of color antagonism transformation, a local receptive field mechanism is adopted to generate a preliminary antagonistic feature map by applying a learnable spatial convolution kernel to the RGB three channels to weight and aggregate the color response in each pixel neighborhood; Subsequently, a nonlinear activation and bidirectional response modeling strategy is introduced: on the one hand, the ReLU function is used to extract the positive activation response in the feature map to retain the enhanced part of the color difference; on the other hand, the negative response part is subjected to absolute value transformation and an adjustable suppression coefficient is introduced to generate a suppressive response component; by differentiating and fusing the positive response and the suppressive response, a color antagonistic response with direction selectivity is constructed to enhance the perception ability of the model to raindrop edges and texture structures; A multi-scale convolution kernel group is used to process the color antagonistic response in parallel to extract three color antagonistic features of different scales; each scale feature map is assigned a learnable weight parameter for dynamically adjusting the contribution degree in the overall feature expression, and the local details and global structure information are cooperatively expressed through weighted fusion to generate a multi-scale color antagonistic response map; The multi-scale color antagonistic response map is subjected to local contrast normalization: the absolute value sum of the response intensity in the neighborhood of each pixel is calculated as a local normalization factor, and the original response value is divided by the factor to realize adaptive gain control of the feature map, and color antagonistic features are obtained.

4. The method of claim 1, wherein, The specific implementation process of step 3 is as follows: First, multi-directional selective filtering is performed: a plurality of sets of direction-sensitive learnable convolution kernels are used to extract structural responses of input color antagonistic features in multiple directions in parallel, to simulate the direction selectivity of V1 neurons to edges and stripes, and to capture dominant orientation information of rain streaks; after the outputs of each direction are fused in the channel dimension, 1x1 convolution integration is performed to form a structural feature map with direction perception capability, and the structural feature map is added element by element with the color antagonistic feature through a residual connection; then, multi-scale receptive field modulation is implemented, and the directional feature map is processed through multiple parallel branches: one path uses spatial convolution to extract local details and performs element-wise multiplication with the input feature through a residual connection; the other two paths use a dilated convolution with different expansion rates to expand the receptive field and capture the context information in the middle and long distances; the features output by the two dilated convolutions are fused after being pooled and upsampled; the two features after weighted fusion are input into a self-attention module SE and a convolution operation to obtain the output features corresponding to the two paths; The outputs of the three paths are spliced and fused into a multi-scale attention feature map, and then a pulse neural mechanism is introduced to realize sparse response; the multi-scale attention feature is converted into a multi-time step sequence through a pulse coding layer, and is input into a linear layer with a pulse activation function to simulate the integration-firing process of neurons and generate a sparse response map; Finally, multi-source feature dynamic fusion is performed; The multi-scale attention feature, the sparse response feature, and the result of the random forgetting processing of the original input feature are weighted and fused point by point to generate the final visual cortex multi-scale encoding output.

5. The method of claim 1, wherein, The visual cortex multi-scale encoding is co-stacked for five levels, the outputs of the first four levels are input into the next level after being downsampled by average pooling, and are also input into the same level visual cortex dual-path prediction module; the output of the fifth level is only input into the upper prediction module and does not participate in the feature transmission of the same level, thereby constructing a hierarchical and multi-scale feature expression system.

6. The method of claim 1, wherein, The bionic visual cortex dual-path prediction structure and the hybrid coding reconstruction structure are implemented as follows: A visual cortex dual-path prediction structure is constructed, the same level visual cortex multi-scale encoding output and the input image are respectively upsampled, the visual cortex multi-scale encoding output is restored in resolution through transposed convolution, and the input image is aligned in spatial scale through bilinear interpolation, then the two are spliced in the channel dimension to form a fused initial feature map after 1x1 convolution; Multi-path fusion prediction structure: after converting the fusion initial feature map into a gray-scale representation, direction response modeling is performed through multiple direction Gabor filters to simulate the selective response of V1 neurons to stripe direction; after weighted fusion of each direction response, a 1x1 convolution and a nonlinear activation are performed to generate a multi-direction attention map, which is multiplied by the original feature map, and then mapped by a linear layer to output a dorsal pathway perception map; in the ventral pathway, the first branch introduces a windowed self-attention mechanism to calculate feature responses within a local window for the fusion initial feature map, and combines a convolution operation with a cyclic shift strategy to expand the receptive field diversity; the second branch constructs a multi-scale convolution branch including a convolution branch and a dilated convolution branch, which expands the receptive field through dilated convolution to capture context information, and performs a difference operation with the output of the convolution branch to highlight the feature change area; after weighted fusion of the output features of the two branches, a ventral pathway perception map is formed; the perception maps output by the dorsal and ventral pathways are spliced in the channel, and the features are integrated through 1x1 convolution to generate a dual-path fusion prediction result; Hybrid coding reconstruction structure: this structure constructs four levels, which gradually receive the visual cortex multi-scale coding output and the upper layer information input from the corresponding level to realize multi-scale prediction; the output of the last level is the final perception prediction map; a hybrid coding output module is designed, which includes two differentiated processing paths: one is to introduce position encoding and coordinate encoding after the perception prediction map is processed by a linear layer and a convolution, and to splice the two encoding results in the channel to construct an enhanced feature containing spatial priori; the other is to directly fuse the position and coordinate encoding of the perception prediction map, and to splice the two encoding results in the channel to form a reference feature path; the output features of the two paths are point-by-point differentiated to generate a relative position coordinate feature; after adding the relative position coordinate feature and the perception prediction map, a 1x1 convolution is performed for preliminary transformation, and a biologically interpretable nonlinear response is introduced through a pulse coding mechanism to copy the feature to multiple time steps and superimpose random noise to simulate the timing fluctuations of neural input, then a differentiable pulse activation function is used for timing processing, finally, the pulse response result is added and fused with the perception prediction map, and a spatial convolution is performed to output the final rain removal image.

7. The method of claim 1, wherein, The specific process of constructing the multi-target hybrid loss function is as follows: The multi-target hybrid loss function is composed of four complementary terms: one is to use the Charbonnier loss as the basic pixel-level reconstruction term; the second is to introduce a structure perception loss to enhance the model's ability to model the consistency of image high-level structure and semantics; the third is to design a Laplacian pyramid edge loss to focus on the recovery of multi-scale edge structure; the fourth is to introduce a frequency domain Fourier loss to constrain the spectrum alignment and preserve the natural statistical properties of the image in the frequency space; After each training batch, the four loss terms are calculated based on the data augmentation and the final rain removal image as the input of each loss function; then the four loss terms are weighted by learnable or empirical weights to form a comprehensive optimization target to guide the parameter update.

8. A biologically inspired multi-scale pulse gating image deraining system for implementing the multi-scale pulse gating image deraining method of any one of claims 1 to 7, characterized in that, The modules include: The rain-containing image acquisition module is configured to acquire a synthetic data set for a single image rain removal task and perform preprocessing. The color antagonism feature extraction module is configured to introduce a color antagonism feature extraction mechanism driven by a biomimetic method based on the preprocessed image, separate and enhance the expression of color degradation information in the rainy day image, and obtain color antagonism features. The visual cortex feature extraction module is configured to perform structural enhancement and multi-scale abstraction on the color antagonism features through a biomimetic visual cortex multi-scale coding, and construct a visual cortex feature representation with direction selectivity and structural robustness. The image rain removal result output module is configured to propose a biomimetic visual cortex double-path prediction structure and hybrid coding output mechanism for the visual cortex feature representation, enhance the structure recognition and texture restoration capability, and output an image rain removal result image. The training module is configured to construct a multi-objective hybrid loss function, perform back training and testing.

Images