Image deraining method based on time sequence LIF cortex

Abstract

This invention relates to the field of image processing technology, specifically to an image deraining method based on the temporal LIF cortex. The method includes preprocessing the rainy image, including size verification and padding, normalization, and data augmentation; extracting multi-resolution feature maps and embedding preliminary features; performing depth extraction of multi-resolution features using a TLC block, preserving skip connection features; downsampling to a low resolution followed by progressive upsampling, and performing multi-scale feature fusion using the SK-Fusion module; finally, outputting a restored clear image and removing padding areas. This invention leverages the low-power characteristics of the temporal LIF cortex to achieve efficient operation on resource-constrained devices. By employing multi-resolution feature extraction and skip connection fusion mechanisms, it enhances adaptability to different rainfall amounts and complex rain streaks. The SK-Fusion module dynamically adjusts the receptive field, fuses multi-scale features, reduces artifacts, and preserves details.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, specifically to an image deraining method based on temporal LIF cortex. Background Technology

[0002] With the development of imaging technology, acquiring information through visual means has become an important tool in modern society. However, image quality can be severely degraded by rain. Rain not only blurs images but also creates noticeable rain streaks and water droplets, reducing image contrast and sharpness. Therefore, image deraining and restoration are particularly important in practical applications.

[0003] Existing technologies mainly fall into the following three categories:

[0004] Image enhancement methods directly improve the contrast and detail clarity of an image through image processing techniques;

[0005] The physical model approach starts with the rain imaging model, establishes a mathematical model of the degradation of images caused by rainfall, and performs inversion and restoration of clear images;

[0006] Deep learning methods, such as convolutional neural networks and generative adversarial networks, achieve rain ripple modeling through end-to-end training, perform excellently in rain removal tasks, and can automatically learn effective feature representations to achieve high rain removal accuracy.

[0007] However, in practical applications, the above technologies are:

[0008] Image enhancement methods work well only in light rain, and have limited effectiveness in heavy rain.

[0009] When dealing with complex rain effects in dynamic scenes, the physical model method has a large computational load and poor real-time performance.

[0010] While deep learning methods have improved rain removal accuracy, they rely on a large amount of labeled data and their performance is unstable under different rainfall amounts and scene variations. Summary of the Invention

[0011] The purpose of this invention is to provide an image deraining method based on temporal LIF cortex, in order to solve the problems of computational complexity and poor robustness of image enhancement-based algorithms and physical models, as well as the dependence of deep learning methods on massive amounts of samples.

[0012] To achieve the above objectives, the present invention provides the following technical solution: an image deraining method based on temporal LIF cortex, comprising the following steps:

[0013] S1. Input the rainy image and perform preprocessing: Perform size verification and padding on the rainy image to ensure that the image size is divisible by the block size, normalize the pixel values ​​to the range of [-1,1], and perform data augmentation operations;

[0014] S2. Multi-resolution feature map generation and preliminary embedding: Generate multi-resolution feature maps from the image in step S1, including original resolution features, medium resolution features and low resolution features, and embed preliminary features through two-dimensional convolution.

[0015] S3. Temporal LIF cortical feature extraction: The original resolution features are extracted using the TLC module and retained as skip connection feature skip1;

[0016] The original resolution features are downsampled to medium resolution, the downsampled features are concatenated with the low-resolution preliminary features, the number of channels is compressed, and the medium-resolution depth features are extracted and retained as the skip connection feature skip2.

[0017] The original resolution features are downsampled to a low resolution, the downsampled features are spliced ​​together with the low-resolution preliminary features, and the low-resolution depth features are extracted after compressing the number of channels.

[0018] S4. Multi-scale feature fusion: Upsample the low-resolution features to medium resolution, use the SK-Fusion module to fuse the upsampled features with the skip2 features, add the fusion result to the current medium-resolution features, perform depth extraction on the fused medium-resolution features and then upsample them to the original resolution.

[0019] The upsampled features and skip1 features are fused using the SK-Fusion module, and the fused result is added to the current original resolution features.

[0020] S5. Image restoration and cropping: Map the features fused in step S4 back to the image space, remove the filled areas, and obtain a clear image after rain removal.

[0021] Preferably, the data augmentation operations in step S1 include image size change, random flipping and occlusion, and edge reduction.

[0022] Preferably, the medium resolution feature is reduced to half of the original resolution by bilinear interpolation;

[0023] The low-resolution feature is achieved by reducing the resolution to 1 / 4 of the original resolution through bilinear interpolation.

[0024] Preferably, the TLC module in step S3 mainly includes a TLC (Temporal LIF Cortex) Spike structure, whose mathematical model is expressed as:

[0025]

[0026] in These are the number of channels, height, and width; the initial membrane potential is set. Set initial threshold For each iteration For each pixel position Perform the following calculations:

[0027] (1) Input signal:

[0028]

[0029] It is an external stimulus, i.e., the input feature map. In position The value; It is the link strength weight; It is a weight matrix, usually with a Gaussian kernel, representing the degree of influence of neighboring pixels on the center pixel; It is a location ;

[0030] (2) Membrane potential renewal:

[0031]

[0032] It is the membrane potential decay coefficient, which controls the degree of forgetting of the membrane potential. The value ranges from 0.2 to 0.8;

[0033] (3) Pulse firing judgment:

[0034]

[0035] When the membrane potential exceeds the dynamic threshold, the neuron at that location fires a pulse signal and propagates it to the neighborhood. Combined with the weight matrix, the high-frequency noise signal of the raindrops is suppressed across channels.

[0036] (4) Dynamic threshold update:

[0037]

[0038] It is the threshold decay constant; It is the threshold increment, representing the amount by which the threshold increases after the neuron fires a pulse;

[0039] (5) Output results:

[0040]

[0041] The significant difference between the resting state and the impulse state of neurons lies in the generation of impulse bursts. When the input bit of the neural simulation receives an input higher than the impulse threshold, it enters the impulse mode, simulating the firing of biological neurons to transmit information.

[0042] The TLC module selectively suppresses the spatial frequency corresponding to rain noise in the feature map by dynamically adjusting the neuron membrane potential and coupling the link strength weights of neighboring pixels in the image.

[0043] Preferably, the SK-Fusion fusion module in step S4 includes: using Selective Kernel to calculate the weight parameters of each convolutional kernel through an attention mechanism, and allocating channel attention coefficients to achieve dynamic feature fusion, so as to automatically select and fuse convolutional kernels of the same or different sizes, thereby enabling the network to dynamically adjust the receptive field size according to the input features.

[0044] Compared with the prior art, the beneficial effects of the present invention are:

[0045] This invention leverages the low-power characteristics of the temporal LIF cortex to achieve efficient operation on resource-constrained devices, solving the problem of traditional deep learning methods relying on high computational resources. It employs a multi-resolution feature extraction and skip connection fusion mechanism to enhance adaptability to different rainfall intensities and complex rain morphologies, overcoming the performance limitations of image enhancement methods in heavy rainfall scenarios. A dynamic threshold update mechanism accurately simulates neuron response characteristics, effectively distinguishing rain noise from background details, improving rain removal accuracy, and avoiding the computational complexity and poor adaptability of physical model methods in dynamic scenes. The SK-Fusion module dynamically adjusts the receptive field through selective convolution kernels, fusing multi-scale features to ensure detail preservation and overall restoration, solving the problem of traditional methods easily introducing artifacts or losing details. It can stably remove raindrop interference in complex environments, balancing real-time performance and low resource consumption, providing an efficient and reliable solution for practical applications. Attached Figure Description

[0046] Figure 1 This is a schematic diagram of the overall process of an image deraining method based on temporal LIF cortex according to the present invention;

[0047] Figure 2 This is a schematic diagram of the overall structure of the rain removal model of the present invention;

[0048] Figure 3 This is a schematic diagram illustrating the principle of Temporal LIF Cortex Spike in this invention.

[0049] Figure 4 This is a schematic diagram of the SK-Fusion module in the rain model of this invention;

[0050] Figure 5 This is a schematic diagram illustrating the rain removal effect of a high-intensity raindrop dataset image according to an embodiment of the present invention;

[0051] Figure 6 This is a schematic diagram illustrating the rain removal effect on a low-intensity raindrop dataset image according to an embodiment of the present invention. Detailed Implementation

[0052] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0053] In recent years, many researchers have devoted themselves to the research of image deraining algorithms and have achieved certain results. These algorithms are mainly divided into three categories: image enhancement-based methods, physical model-based methods, and deep learning-based methods.

[0054] Image enhancement methods primarily improve image contrast and detail clarity directly through image processing techniques. Examples include fast guided filtering and adaptive contrast enhancement. These methods utilize the structural features of the image to remove rain streaks or enhance edge information, thereby improving the visibility of rainy images. However, these methods only work well in light rain and have limited effectiveness in heavy rain.

[0055] Physical model-based methods start with rain imaging models, establish mathematical models of how rainfall degrades images, and then invert and restore clear images. These rain streak feature models model the spatial and temporal characteristics of rain streaks by observing the characteristics of rain streaks under different rainfall intensities. Based on this model, clear information in images can be recovered to some extent. However, when processing complex rain effects in dynamic scenes, the computational load is large and the real-time performance is poor.

[0056] With the rapid development of deep learning technology, deep learning-based rain removal algorithms have gradually become a research hotspot. These methods utilize deep learning models such as Convolutional Neural Networks (CNNs) and Generative Adversarial Networks (GANs) to directly learn how to separate rain streaks from the background image from a large amount of rainy day image data. Models like RainNet and RESCAN perform excellently in rain removal tasks and can automatically learn effective feature representations, achieving high rain removal accuracy. However, these methods rely on a large amount of labeled data, and their performance may be unstable under different rainfall amounts and scene variations.

[0057] In recent years, with the introduction of TLC (Temporal LIF Cortex), its low power consumption and high efficiency have shown potential in the field of image processing. Unlike traditional artificial neural networks, it transmits and processes information through discrete pulse signals, which is closer to the working mechanism of biological neurons. In image deraining tasks, TLC can maintain high processing accuracy and efficiency while reducing energy consumption.

[0058] Therefore, by combining the advantages of temporal LIF cortex, it is expected to achieve more efficient image deraining and restoration, thus providing a more promising solution for rainy day image processing.

[0059] Please see Figure 1-6 This invention provides a technical solution: an image deraining method based on temporal LIF cortex, comprising the following steps:

[0060] S1. Input a rainy image and perform preprocessing: Perform size verification and filling on the rainy image to ensure that the height and width of the input image can be divided by the default block size. If not, perform reflection filling, normalize the pixel values, that is, map the pixel values ​​of the image from [0,255] to the range of [−1,1], and perform data augmentation operations, including image size change, random flipping and occlusion, and edge weakening.

[0061] S2. Multi-resolution feature map generation and preliminary embedding: Generate multi-resolution feature maps from the image in step S1, including original resolution features, medium resolution features and low resolution features. Perform multi-scale input and embedding operation through two-dimensional convolution to extract preliminary features. The medium resolution features are reduced to 1 / 2 of the original resolution through bilinear interpolation, and the low resolution features are reduced to 1 / 4 of the original resolution through bilinear interpolation.

[0062] S3. Temporal LIF Cortical Feature Extraction: A base layer composed of TLC Blocks (Temporal LIF Cortical Modules for Multi-Level Feature Extraction) is constructed to perform depth extraction on the original resolution features, and the resolution is gradually adjusted. Simultaneously, the extracted features are saved as skip1 for subsequent multi-scale feature fusion. The core computational unit TLC (Temporal LIF Cortex) Spike structure within the TLC Block is as follows: Figure 3 As shown, its mathematical model is expressed as:

[0063]

[0064] in These are the number of channels, height, and width; the initial membrane potential is set. Set initial threshold For each iteration For each pixel position Perform the following calculations:

[0065] (1) Input signal:

[0066]

[0067] It is an external stimulus, i.e., the input feature map. In position The value; It is the link strength weight; It is a weight matrix, usually with a Gaussian kernel, representing the degree of influence of neighboring pixels on the center pixel; It is a location By linking the neighboring pixel intensity of the Gaussian kernel with the weights, the high-frequency features of rain noise are suppressed, while the low-frequency features of the background edge are preserved, thus improving the signal-to-noise ratio.

[0068] (2) Membrane potential renewal:

[0069]

[0070] It is the membrane potential decay coefficient, which controls the degree of forgetting of the membrane potential. The value ranges from 0.2 to 0.8. In this embodiment... The value is 0.5;

[0071] (3) Pulse firing judgment:

[0072]

[0073] When the membrane potential exceeds the dynamic threshold, the neuron at that location fires a pulse signal and propagates it to the neighborhood. Combined with the weight matrix, the high-frequency noise signal of the raindrops is suppressed across channels.

[0074] (4) Dynamic threshold update:

[0075]

[0076] It is the threshold decay constant; It is the threshold increment, representing the amount by which the threshold increases after a neuron fires a pulse.

[0077] (5) Output results:

[0078]

[0079] The significant difference between the resting and pulsating states of neurons lies in the generation of impulse bursts. When the input bit of a neural simulation receives a sufficiently high input, above the impulse threshold, it enters impulse mode, simulating the firing of biological neurons to transmit information.

[0080] The TLC module selectively suppresses the spatial frequency corresponding to rain noise in the feature map by dynamically adjusting the neuron membrane potential and coupling the link strength weights of neighboring pixels in the image.

[0081] The original resolution features are downsampled to medium resolution, the downsampled features are concatenated with the low-resolution preliminary features, the number of channels is compressed, and the medium-resolution depth features are extracted and retained as the skip connection feature skip2.

[0082] The original resolution features are downsampled to a low resolution, the downsampled features are spliced ​​together with the low-resolution preliminary features, and the low-resolution depth features are extracted after compressing the number of channels.

[0083] S4. Multi-scale feature fusion: The low-resolution features are upsampled to medium resolution. SK-Fusion (Selective Kernel Fusion) is used to calculate the weight parameters of each convolutional kernel through an attention mechanism. Channel attention coefficients are assigned to achieve dynamic feature fusion. The upsampled features are fused with the skip2 features, and the fusion result is added to the current medium-resolution features to enhance feature expression. Afterwards, the fused medium-resolution features are depth-extracted and upsampled to the original resolution.

[0084] The SK-Fusion module is used to fuse upsampled features with skip1 features, and the fusion result is added to the current original resolution features to further enhance the expressive power of multi-scale features.

[0085] S5. Image restoration and cropping: Map the features fused in step S4 back to the image space. If the input image was filled in step S1, remove the excess filling and output the processed image.

[0086] Example 1

[0087] Please provide explanations for some of the terms used in the application:

[0088] The Rain200 dataset is a commonly used dataset for research on raindrop removal algorithms, primarily used to evaluate and compare the performance of image raindrop removal algorithms. The Rain200H and Rain200L datasets contain synthetic images generated under different raindrop intensities to help researchers test the effectiveness of raindrop removal algorithms under various environmental conditions. ① Rain200H is the high-intensity raindrop dataset within the Rain200 dataset. This dataset was generated by applying raindrop effects to 1000 clear images to simulate a scene of heavy rainfall. Each image contains raindrops of different intensities and shapes, aiming to help researchers optimize the performance of raindrop removal algorithms under heavy rain conditions. ② Rain200L is the low-intensity raindrop dataset within the Rain200 dataset. Unlike Rain200H, Rain200L simulates a more moderate rainfall environment with a lower raindrop density, generally closer to real-world light rainfall scenarios. By providing raindrop effects of different intensities, Rain200L helps researchers optimize the performance of raindrop removal algorithms under weak rainfall conditions.

[0089] Rainy conditions significantly impact image acquisition. Raindrops and streaks act as foreground obstructions, physically blocking camera equipment and affecting light propagation paths through refraction and scattering. This interference often manifests as image blurring, loss of detail, and sharp contrast changes due to light spots and streaks formed by raindrops. Rainy environments can also cause overall color distortion, further reducing image sharpness and visibility, thus hindering effective target identification and information extraction.

[0090] Currently available related technologies can be broadly categorized into the following types: image enhancement-based methods, physical model-based methods, and deep learning-based methods. However, each of these methods has its limitations: image enhancement methods are only suitable for light rain, performing poorly in heavy rain and easily introducing artifacts; physical model methods are computationally complex and time-consuming, with poor applicability in dynamic scenes; deep learning methods rely on large amounts of labeled data, have poor scene adaptability, and require high computational resources, making them unsuitable for real-time processing.

[0091] Based on the current situation, this application provides a novel image deraining method, which aims to improve the quality of derained images.

[0092] First, the input rainy day image is normalized, mapping the pixel value range to the [−1,1] interval to ensure numerical stability in subsequent processing. In some cases, data augmentation techniques are used to improve the robustness and generalization ability of the model, including but not limited to random cropping, mirroring, and rotation, to expand the diversity of training samples.

[0093] Model Construction: The model of this invention mainly adopts the temporal LIF cortex as its core architecture. It extracts multi-scale features from the input image through a multi-resolution convolutional module, combining a dynamic threshold update mechanism and Gaussian kernel-based convolution operations to simulate the spiking behavior of biological neurons. The TLC module is designed to handle sparse features by controlling spiking activity, improving computational efficiency in low-power scenarios. Simultaneously, an SK-Fusion (Selective Kernel Fusion) module is introduced, which dynamically adjusts the receptive field size to fuse feature information from different resolutions. Its fusion process incorporates residual learning to ensure the robustness and consistency of multi-scale feature representation. The model contains multiple basic layer modules, combining spiking neural networks and MLP (Multilayer Perceptron) to achieve deep feature extraction and learning. Then, low-resolution features are recovered through upsampling and fused with high-resolution features from skip connections. Finally, a de-embedding module is used to map the features back to the image space, generating a clear image after rain removal. The model as a whole uses L1 and LPIPS loss functions, adjusting parameters through backpropagation. To improve training efficiency and performance, the AdamW optimizer was employed, combined with a cosine annealing strategy to gradually reduce the learning rate. This reduces the risk of getting trapped in local optima and avoids instability caused by large learning rate updates, promoting faster model convergence and thus improving prediction accuracy. Furthermore, multiple rounds of evaluation and tuning ensured that the model performed as expected on both the training and validation sets.

[0094] Model Training: The model employs supervised learning, using paired data of clearly labeled images and corresponding rainy day images as the training set. To improve rain removal performance, a combination of L1 loss and LPIPS loss is chosen as the loss function, balancing the need for overall image quality restoration with the preservation of local details. Furthermore, the number of training epochs and batch size are flexibly adjusted according to different dataset sizes and complexities to ensure the model can adapt to diverse scenarios.

[0095] Model Optimization: To improve training efficiency and performance, this invention employs Automatic Mixed Precision (AMP) technology, which dynamically switches between half-precision and full-precision to reduce memory requirements and computational costs. The main model parameters and spiking neuron parameters are optimized by the AdamW optimizer and its weight decay version, respectively, to enhance generalization ability. Simultaneously, the learning rate is gradually adjusted using cosine decay to reduce the risk of getting trapped in local optima, avoid unstable large adjustments, promote more efficient model convergence, and improve prediction accuracy.

[0096] Model Validation and Testing: After model training, its performance was evaluated using a defined validation set, and the optimal parameters were selected accordingly. Finally, the model was quantitatively and qualitatively evaluated on a test set, comparing it with existing mainstream rain removal methods in terms of image quality, processing efficiency, and algorithm robustness. Experimental results show that this method significantly improves the clarity and detail restoration of rain-removed images while maintaining low computational power consumption.

[0097] Model Deployment: Finally, this invention can be applied and deployed in real-world applications, such as autonomous driving systems, indoor and outdoor security monitoring, and satellite image processing, to achieve excellent image deraining effects through its lightweight model structure.

[0098] Performance Evaluation and Optimization: To objectively and accurately evaluate the experimental results, this study uses Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity Index (SSIM) to measure the rain removal effect, and optimizes the rain removal network parameters based on the results. PSNR is used to quantify the degree of distortion in an image or video. It is calculated by comparing the error between the original signal (e.g., a rain-containing image or video) and the processed rain-removed signal, using the following formula:

[0099] ,

[0100] in, The maximum value of the image pixels. This represents the mean squared error. SSIM is used to evaluate the similarity between two images, one being the original image without distortion and the other a distorted image. The formula is:

[0101]

[0102] in, : Structural similarity index, representing the structural similarity index of images and images The degree of structural similarity.

[0103] and : Represents images respectively and The average brightness. and : Represents images respectively and The variance of the image and The degree of dispersion in brightness distribution. :image and images The covariance represents the correlation between the brightness distributions of the two images. c1 and c2 are two constants introduced to avoid the denominator being zero. They are usually taken as c1=(K1L)2 and c2=(K2L)2, where L is the dynamic range of pixel values ​​(e.g., for an 8-bit image, L=255), and K1 and K2 are constants less than 1.

[0104] Experimental results

[0105] We tested the proposed image rain removal and restoration method based on the temporal LIF cortex on the Rain200H and Rains200L datasets, respectively. The proposed method achieved excellent results with high PSNR and SSIM values. The specific experimental results are shown in Table 1 below.

[0106] Table 1: Test results of Rain200H and Rains200L datasets

[0107]

[0108] Figure 5 , Figure 6 This shows a comparison of the model's performance metrics on two different datasets, where Figure 5 , Figure 6 In the image, (a) represents the original image, (b) represents the raindrop image, and (c) MS_TLC (PSNR / SSIM) represents the image after rain removal using the method of this invention.

[0109] Table 2 shows the comparison data of PSNR / SSIM between the method described in this invention and traditional CNN and physical models.

[0110] Table 2: Comparison of PSNR / SSIM between this method and traditional CNN and physical models

[0111]

[0112] This invention combines the high efficiency and accuracy of temporal LIF cortex, significantly improving the clarity and detail recovery of derained images, and is suitable for various image processing scenarios.

[0113] Compared with existing technologies, it has the following beneficial effects:

[0114] High efficiency: The low power consumption of the temporal LIF cortex enables the algorithm to operate efficiently on resource-constrained devices.

[0115] Multi-scale feature extraction and fusion: The model enhances its adaptability to different rainfall amounts and rain streak patterns by using multi-resolution feature extraction and skip connections to ensure the preservation of image details and overall restoration effect.

[0116] Dynamic threshold mechanism: The method uses a dynamic threshold to update the model, which can more accurately simulate the characteristics of neuron response to rain noise and effectively improve the rain removal accuracy.

[0117] Capability to withstand diverse scenarios: Compared to traditional methods based on physical models or deep learning, this method can remove rain streaks more stably in dynamic scenarios and complex rainfall changes, and has strong adaptability.

[0118] Low resource consumption: Compared with traditional deep learning models, this algorithm combines the characteristics of the temporal LIF cortex and can achieve high-quality rain removal effect with less labeled data, thus reducing the cost of data acquisition.

[0119] Specific applications

[0120] The method of this invention can be widely applied to various scenarios in daily life and work that require image deraining and restoration, including but not limited to:

[0121] 1. Autonomous Driving: In autonomous driving systems, rainy weather images are often affected by raindrops, impacting the vehicle's perception capabilities. By using image de-raining technology, the influence of rainwater on images can be removed, improving the vehicle's accuracy in recognizing obstacles, pedestrians, and road conditions, thereby enhancing the safety of autonomous driving.

[0122] 2. Drone Vision: In rainy weather, images captured by drones are often interfered with by raindrops, affecting the detection of ground targets. De-rain processing can clearly restore images, helping drones complete tasks such as agricultural monitoring and disaster relief under complex weather conditions.

[0123] 3. Security Monitoring: In inclement weather, images captured by surveillance cameras in rainy conditions are often blurry, affecting the effectiveness of video surveillance. Image de-raining models can effectively remove the effects of rain, improve the quality of surveillance video, and enhance the accuracy of tasks such as facial recognition and vehicle recognition.

[0124] 4. Satellite and remote sensing image processing: Rainy weather can affect the quality of satellite and remote sensing images. Rain removal processing can remove atmospheric pollution such as raindrops, helping to improve the accuracy of ground object identification in images. It is applicable to fields such as post-disaster monitoring and urban planning.

[0125] 5. Smartphone Applications: Smartphones are often affected by rain when shooting in the rain. Rain removal technology can be integrated into mobile applications to automatically remove raindrops and rainwater pollution, helping users take clear photos and videos.

[0126] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0127] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. An image deraining method based on temporal LIF cortex, characterized in that: Includes the following steps: S1. Input the rainy image and perform preprocessing: Perform size verification and padding on the rainy image to ensure that the image size is divisible by the block size, normalize the pixel values ​​to the range of [-1,1], and perform data augmentation operations; S2. Multi-resolution feature map generation and preliminary embedding: Generate multi-resolution feature maps from the image in step S1, including original resolution features, medium resolution features and low resolution features, and embed preliminary features through two-dimensional convolution. S3. Temporal LIF cortical feature extraction: The original resolution features are extracted using the TLC module and retained as skip connection feature skip1; The original resolution features are downsampled to medium resolution, the downsampled features are concatenated with the low-resolution preliminary features, the number of channels is compressed, and the medium-resolution depth features are extracted and retained as the skip connection feature skip2. The original resolution features are downsampled to a low resolution, the downsampled features are spliced ​​together with the low-resolution preliminary features, and the low-resolution depth features are extracted after compressing the number of channels. S4. Multi-scale feature fusion: Upsample the low-resolution features to medium resolution, use the SK-Fusion module to fuse the upsampled features with the skip2 features, add the fusion result to the current medium-resolution features, perform depth extraction on the fused medium-resolution features and then upsample them to the original resolution. The upsampled features and skip1 features are fused using the SK-Fusion module, and the fused result is added to the current original resolution features. S5. Image restoration and cropping: Map the features fused in step S4 back to the image space, remove the filled areas, and obtain a clear image after rain removal.

2. The image deraining method based on temporal LIF cortex according to claim 1, characterized in that: The data augmentation operations in step S1 include image resizing, random flipping and occlusion, and edge reduction.

3. The image deraining method based on temporal LIF cortex according to claim 1, characterized in that: The medium resolution feature is achieved by reducing the resolution to half of the original resolution through bilinear interpolation; The low-resolution feature is achieved by reducing the resolution to 1 / 4 of the original resolution through bilinear interpolation.

4. The image deraining method based on temporal LIF cortex according to claim 1, characterized in that: The TLC module in step S3 mainly includes a TLC (Temporal LIF Cortex) Spike structure, whose mathematical model is expressed as: in These are the number of channels, height, and width; the initial membrane potential is set. Set initial threshold For each iteration For each pixel position Perform the following calculations: (1) Input signal: It is an external stimulus, i.e., the input feature map. In position The value; It is the link strength weight; It is a weight matrix, usually with a Gaussian kernel, representing the degree of influence of neighboring pixels on the center pixel; It is a location ; (2) Membrane potential renewal: It is the membrane potential decay coefficient, which controls the degree of forgetting of the membrane potential. The value ranges from 0.2 to 0.8; (3) Pulse firing judgment: When the membrane potential exceeds the dynamic threshold, the neuron at that location fires a pulse signal and propagates it to the neighborhood. Combined with the weight matrix, the high-frequency noise signal of the raindrops is suppressed across channels. (4) Dynamic threshold update: It is the threshold decay constant; It is the threshold increment, representing the amount by which the threshold increases after the neuron fires a pulse; (5) Output results: The significant difference between the resting state and the impulse state of neurons lies in the generation of impulse bursts. When the input bit of the neural simulation receives an input higher than the impulse threshold, it enters the impulse mode, simulating the firing of biological neurons to transmit information. The TLC module selectively suppresses the spatial frequency corresponding to rain noise in the feature map by dynamically adjusting the neuron membrane potential and coupling the link strength weights of neighboring pixels in the image.

5. The image deraining method based on temporal LIF cortex according to claim 1, characterized in that: The SK-Fusion fusion module in step S4 includes: using selective convolution (Selective Kernel) to calculate the weight parameters of each convolution kernel through an attention mechanism, and allocating channel attention coefficients to achieve dynamic feature fusion, so as to automatically select and fuse convolution kernels of the same or different sizes, thereby enabling the network to dynamically adjust the receptive field size according to the input features.

Images