Image defogging algorithm based on multi-scale pulse neural network

Abstract

The application provides an image defogging algorithm based on a multi-scale pulse neural network. The method comprises the following steps: preprocessing an input fog image, and extracting preliminary features through two-dimensional convolution; constructing a basic layer composed of multi-scale LIF (Leaky Integrate and Fire) modules, using convolution kernels of different scales to obtain multi-scale features, simulating the pulse emission mechanism of biological neurons to extract features, and performing feature fusion through SK-fusion after full connection layer mapping; and finally inputting two-dimensional convolution to restore the embedded feature map to obtain a clear defogging image. The method combines the high efficiency of the pulse neural network and the accuracy of the full connection neural network, significantly improves the clarity and detail recovery effect of the defogging image, and is suitable for various image processing scenes.

Description

Technical Field

[0001] This application relates to the field of image processing, and more specifically, to an image dehazing algorithm based on a multi-scale spiking neural network. Background Technology

[0002] Due to factors such as fog, haze, and air pollution, image details and contrast may be weakened, and colors may be distorted. Therefore, image dehazing and restoration are extremely important.

[0003] Traditional image processing methods include image enhancement-based methods, physical model-based methods, and deep learning-based methods. Physical model-based methods start with atmospheric scattering models, establishing a mathematical model of the image degradation process, and then reconstructing a clear image. The most famous method in this category is the Dark Channel Prior (DCP) algorithm. DCP is based on an important observation: in natural scenes, most non-sky areas contain at least one pixel with an extremely low value in one or more color channels. Using this prior knowledge, DCP can effectively estimate hazy images, thus reconstructing a clear image. While DCP performs well in many cases, its high complexity leads to poor performance in certain scenarios.

[0004] Therefore, to address the shortcomings of traditional image processing methods in achieving poor combined results, a new image dehazing algorithm based on multi-scale spiking neural networks is proposed. Summary of the Invention

[0005] The summary section of this application is intended to provide a brief overview of the concepts, which will be described in detail in the detailed description section below. This summary section is not intended to identify key or essential features of the claimed technical solutions, nor is it intended to limit the scope of the claimed technical solutions.

[0006] To address the technical problems mentioned in the background section, some embodiments of this application provide an image dehazing algorithm based on a multi-scale spiking neural network, comprising:

[0007] The input fog image is preprocessed; the preprocessing includes image normalization and data augmentation.

[0008] An embedding procedure is performed on the preprocessed image to extract preliminary features;

[0009] Construct a base layer composed of multi-scale LIF; the base layer includes multi-scale convolution, LIF pulse firing, and fully connected layers; the fully connected layers are composed of two-dimensional convolutions connected by LeakyReLU.

[0010] The image is input into the pulse firing structure of the LIF to form a pulse firing mechanism that simulates biological neurons;

[0011] Perform drop path regularization on the image output by the LIF Module;

[0012] The input image is fed into a fully connected layer for regularization.

[0013] Use SK-fusion to fuse features in the image;

[0014] Return to the preprocessed image and execute the embedding procedure to extract preliminary features until the number of iterations is N; N is a positive integer;

[0015] Unembed the output image;

[0016] The embedded feature map is restored to achieve image dehazing.

[0017] Furthermore, the data is scaled proportionally so that its pixel values ​​fall within the range of [-1, 1].

[0018] Furthermore, image features are extracted based on convolutional layers;

[0019] Channel blending is performed using image features;

[0020] The pulse firing mechanism of biological neurons is simulated based on LIF neuron layers;

[0021] Perform token mixing operations;

[0022] Based on the activation function layer, the output of the LIF neuron layer is converted into a pulse signal.

[0023] Furthermore, based on the linear transformation layer, the features of the input feature map and the original image are fused;

[0024] Based on the activation function layer, the output port of the neural network is formed;

[0025] Incorporate linear transformation layers and activation function layers into fully connected neural network layers;

[0026] Connect each fully connected neural network layer to form a fully connected network.

[0027] Furthermore, based on LIF, an MLP neural network layer is formed;

[0028] The input features are converted into embedding vectors using MLP neural network layers;

[0029] Based on the embedding vector, the input parameters of the spiking neuron module in the spiking firing structure are formed.

[0030] Furthermore, add a dwconv layer to the first MLP layer;

[0031] AxialShift, which replaces the MLP layer with LIF;

[0032] Horizontal LIF based on MLP layer accumulates and propagates information in the horizontal direction of image features;

[0033] Vertical LIF based on MLP layers accumulates and propagates information in the vertical direction of image features.

[0034] Furthermore, by utilizing the accumulation and propagation of information in the horizontal direction and the vertical direction of image features, the pulse firing mechanism of biological neurons is simulated to perform feature mixing.

[0035] Furthermore, when u <V th At that time, the membrane potential u decays over time and receives external input I:

[0036]

[0037] When the membrane potential u reaches the threshold V th At this time, the neuron fires a pulse, and the membrane potential is reset to u. reset :

[0038] o = 1, u = u reset ,u≥V th

[0039] Where u is the membrane potential, I is the input from the upper layer, τ is the time coefficient, o is the output, and V is the time coefficient. th This is the firing threshold of this neuron. When a pulse is triggered, the membrane potential u is reset to u. reset In the formula, t represents the group index.

[0040] Furthermore,

[0041]

[0042] o=uu=u reset ,u≥V th

[0043] We replace the '1' in the output with 'u', transforming the original binary output into a linear output, thus preserving full-precision information. Applying the full-precision LIF model to iterative LIF yields:

[0044]

[0045] W T The transpose of the weight matrix, where x is the input feature and r is the weight matrix. t+1 nIt is the final full-precision output at step t+1, o t+1 n It is a temporary variable that records the output state at step t+1, with coefficients τ and V. th These are the learning parameters for the neural network.

[0046] Furthermore,

[0047]

[0048] In summary, the method of this invention can be widely applied to various real-world scenarios requiring image dehazing and restoration, including but not limited to: dehazing images captured by traffic cameras to improve image clarity and recognizability, thereby enhancing traffic management and accident prevention capabilities; processing satellite and drone images to remove blurring effects caused by atmospheric conditions and improve the accuracy of ground object recognition; dehazing surveillance videos in security monitoring systems under adverse weather conditions to improve monitoring effectiveness and ensure public safety; and processing medical images to remove noise and blur, thereby improving image clarity and diagnostic accuracy. Attached Figure Description

[0049] The accompanying drawings, which form part of this application, are used to provide a further understanding of the application and to make other features, objects, and advantages of the application more apparent. The illustrative embodiments and descriptions of this application are used to explain the application and do not constitute an undue limitation of the application.

[0050] Furthermore, throughout the accompanying drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the elements are not necessarily drawn to scale.

[0051] In the attached diagram:

[0052] Figure 1 This is an overall flowchart of the method of the present invention.

[0053] Figure 2 This is a schematic diagram of the overall structure of the hybrid defogging model.

[0054] Figure 3 This is a schematic diagram of the SK-Fusion module in the fog model. Detailed Implementation

[0055] Embodiments of this application will now be described in more detail with reference to the accompanying drawings. While some embodiments of this application are shown in the drawings, it should be understood that this application can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this application. It should be understood that the drawings and embodiments of this application are for illustrative purposes only and are not intended to limit the scope of protection of this application.

[0056] It should also be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings. Unless otherwise specified, the embodiments and features described herein can be combined with each other.

[0057] The present application will now be described in detail with reference to the accompanying drawings and embodiments.

[0058] S1, preprocesses the input fog image, including image normalization (to make the pixel values ​​of the image within a uniform range) and data augmentation (including image cropping, flipping, etc.);

[0059] S2, performs an embedding operation on the preprocessed image (achieved through two-dimensional convolution) to extract preliminary features;

[0060] S3 constructs a base layer composed of multi-scale LIF (Leaky Integrate and Fire), including multi-scale convolution, LIF burst firing, and fully connected (Multi-Layer Perceptron, MLP);

[0061] S4, the feature map is normalized by the multi-scale LIF module, and convolution operations are performed using three different scale convolution kernels (3x3, 5x5, 7x7) to extract features at different scales, and then processed separately.

[0062] S5, input the processed feature map into the LIF Module (LIF pulse firing structure) to simulate the pulse firing mechanism of biological neurons;

[0063] S6. Drop path regularization is performed on the output of the LIF Module to introduce randomness and prevent overfitting.

[0064] S7, input two fully connected layers consisting of two-dimensional convolutions connected by LeakyReLU and perform regularization processing;

[0065] S8, used as Figure 3 The SK-fusion (Selective Kernel fusion) shown performs feature fusion, integrating multi-scale feature information;

[0066] S9, repeat the above basic layer operations five times to extract higher-level features layer by layer;

[0067] S10 performs a de-embedding operation on the final output of the base layer (implemented through 2D convolution), restoring the embedded feature map and achieving dehazing.

[0068] This invention employs a novel hybrid model architecture that combines spiking neural networks with fully connected neural networks. Specifically, the model uses a Leaky Integrate-and-Fire (LIF) module, processed by convolutional kernels of three different scales to simulate biological neurons, as the feature extraction layer. This allows the neural network to transmit and process information through discrete pulse signals, improving computational efficiency and energy efficiency. The output features are further processed by fully connected neural networks to enhance the accuracy of image detail restoration. Each LIF module contains a convolutional layer, a LIF neuron layer, and an activation function layer. The convolutional layer extracts image features and performs channel mixing; the LIF neuron layer simulates the pulse firing mechanism of biological neurons and performs token mixing; the activation function layer converts the output of the LIF neuron layer into a pulse signal. The LIF module includes an MLP layer that converts input features into embedding vectors and provides suitable input for subsequent spiking neuron modules. In this section, a dwconv (Depthwise Convolution) layer is added after the first MLP layer, and the shift operation used in AxialShift (an axial displacement-based convolution mechanism that more effectively captures local and global information in the image) is replaced with a LIF neuron. Since the iterative LIF neuron is essentially an activation function, it is necessary to add a dwconv layer before iterative LIF neurons. Next, we use Horizontal LIF (HLIF) and Vertical LIF (VLIF) to accumulate and propagate information in the horizontal and vertical directions of image features, respectively. These two modules perform feature mixing by simulating the pulse firing mechanism of biological neurons, preserving local and global information in the feature map. Our introduced full-precision LIF neuron differs from the traditional LIF neuron; the behavior of the classic LIF model can be modeled as follows:

[0069] (1) When u <V th At that time, the membrane potential u decays over time and receives external input I:

[0070]

[0071] (2) When the membrane potential u reaches the threshold V thAt this time, the neuron fires a pulse, and the membrane potential is reset to u. reset :

[0072] o = 1, u = u reset ,u≥V th

[0073] Where u is the membrane potential, I is the input from the upper layer, τ is the time coefficient, o is the output, and V th This is the firing threshold of this neuron. When a pulse is triggered, the membrane potential u is reset to u. reset Unlike traditional LIF neurons that accumulate in the time domain, the neurons used here accumulate and fire in the spatial domain. In this design, the LIF module first segments the image into several groups, where 't' in the formula represents the group index, not the time step. Since the input features are full-precision, we prefer to obtain full-precision outputs to preserve the information within the groups. To meet our needs, we used the following full-precision LIF function:

[0074]

[0075] o=uu=u reset ,u≥V th

[0076] We replace the output 1 with u, transforming the original binary output into a linear output, thus preserving full-precision information. Applying the full-precision LIF model to iterative LIF yields:

[0077]

[0078] Here, W T The transpose of the weight matrix, where x is the input feature and r is the weight matrix. t+1 n It is the final full-precision output at step t+1, o t+1 n It's just a temporary variable recording the output state at step t+1. The coefficients τ and V... th It is learnable. Using this explicit iterative LIF neuron, the backpropagation process can be completed using the chain rule:

[0079]

[0080]

[0081] The output signal is further processed through an activation function layer to generate a pulse signal. Finally, SK-fusion is used for fusion. Overall, five consecutive base layers are connected in series to extract higher-level features layer by layer, resulting in superior output.

[0082] The image dehazing and restoration method based on multi-scale spiking neural networks proposed in this invention is highly efficient. The introduction of spiking neural networks significantly reduces the computational complexity and energy consumption of the model, thereby improving processing efficiency. It boasts high performance, with the model possessing multi-scale feature extraction and restoration capabilities, enabling high-quality image dehazing and restoration in various hazy environments. It also offers broad applicability, being applicable to a variety of image processing scenarios and possessing wide application prospects. Furthermore, it reduces resource consumption compared to traditional deep learning models, ensuring high performance while minimizing computational resource consumption.

[0083] The Realistic Single Image Dehazing (RESIDE) dataset is a commonly used dataset for image dehazing algorithm research. It provides real-world images in various environments and scenes to help evaluate and compare the performance of dehazing algorithms. The RESIDE dataset contains several subsets, each with its own characteristics and uses. Our experiments used three subsets: RESIDE-IN (Indoor Training Set), RESIDE-OUT (Outdoor Training Set), and RESIDE-6K (Real-world 6K Dehazing Dataset). RESIDE-IN is a training set specifically for indoor scenes, primarily containing synthetic fog images of indoor scenes. This dataset was generated using indoor images and simulated fog using a physical model. Indoor scenes typically have more stable lighting, which helps the algorithm optimize performance in specific environments. RESIDE-OUT is a training set for outdoor scenes, primarily containing synthetic fog images of outdoor scenes, simulating fog effects in natural environments. Due to the greater variation and complexity of lighting in outdoor scenes, its scene diversity is higher, better simulating complex situations in real life. RESIDE-6K is a dataset of 6,000 real-world haze images from the RESIDE dataset. It mainly contains real haze images from real life, which were obtained under natural conditions without the use of any synthetic fog generation techniques.

[0084] In foggy or hazy weather, a large number of tiny suspended particles in the air refract and scatter light. The light then mixes with the light reflected from the target to be observed, resulting in a foggy image. In a foggy image, the image clarity and contrast of the target to be observed are reduced, and there may even be color shifts and loss of a lot of details, which makes it impossible to obtain the true image information of the target to be observed.

[0085] In existing related technologies, foggy images are converted into defogging images by methods such as enhancing contrast and restoring color. However, such implementation methods require that the foggy image to be processed is obtained under sufficient light source. In complex lighting scenarios, such as in nighttime environments, the defogging methods used in related technologies are prone to causing the loss of a lot of valuable information, ultimately resulting in poor quality of the target defogging image.

[0086] Based on the current situation, this application provides a novel image dehazing method aimed at improving the quality of dehazed images.

[0087] Next, the image dehazing method in this application will be described through the following specific implementation.

[0088] Data preprocessing: First, the algorithm normalizes the input fog image so that its pixel values ​​are within the range of [-1, 1]. Then, in some cases, data augmentation techniques, such as random cropping, flipping, and rotation, are used to generate more training samples.

[0089] Model Construction: The hybrid model is primarily built upon multi-scale spiking neural network layers. The spiking neural network employs horizontal and vertical LIF neuron models to simulate the spiking activity of biological neural networks, while multi-scale convolutions are used for feature processing and fusion. This part of the network effectively captures and processes sparse data, significantly improving processing efficiency. The model uses L1 loss as the loss function and optimizes model parameters through backpropagation. To improve training efficiency and model performance, the AdamW optimizer is used, employing cosine annealing to gradually reduce the learning rate during training, minimizing the chance of the model getting trapped in local minima and allowing for large learning rate updates, thus helping the model converge better and improving prediction accuracy. Furthermore, the model undergoes multiple evaluations and optimizations to ensure excellent performance on both the training and validation sets.

[0090] Model Training: Supervised learning was employed for model training, utilizing both labeled clear images and hazy images. The L1 loss function was used to balance global and local detail recovery. The model was trained an varying number of times based on different datasets.

[0091] Model Optimization: To improve training efficiency and performance, this invention introduces mixed-precision training technology, using Automatic Mixed Precision (AMP) to dynamically switch between half-precision and full-precision, reducing memory usage and computational overhead. Simultaneously, the AdamW optimizer and the AdamW optimizer with weight decay are used to optimize the main model parameters and spiking neuron parameters respectively, further enhancing the model's generalization ability. Cosine decay is used to gradually reduce the learning rate during training, minimizing the chance of the model getting trapped in local minima and allowing for large learning rate updates, helping the model converge better and thus improving prediction accuracy.

[0092] Model Validation and Testing: After training, the model is evaluated using a pre-defined validation set to select the optimal model parameters. Finally, the model is tested on a test set and compared with existing dehazing methods to verify the superiority of the proposed method. Experimental results show that the proposed method significantly improves the clarity and detail recovery of dehazed images while maintaining low power consumption and high efficiency.

[0093] Model Deployment: Finally, the method of this invention can be deployed in practical applications, such as video surveillance and autonomous driving, where real-time dehazing is required. Through model lightweighting and acceleration technology, efficient image dehazing and restoration can be achieved on edge devices.

[0094] Performance Evaluation and Optimization: To objectively evaluate the experimental results, we used Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity Index (SSIM) to assess the dehazing effect, and optimized the parameters of the dehazing network based on the evaluation results. PSNR is used to evaluate the degree of image or video distortion. It is calculated by comparing the error between the original signal (usually the original hazy image or video) and the reconstructed signal (usually the dehazed image or video), as shown in the following formula:

[0095]

[0096] Among them, MAX I The maximum pixel value is denoted by , and MSE is the mean squared error. SSIM, on the other hand, is a metric for measuring the similarity between two images. It uses two images: one uncompressed and undistorted, and the other distorted. The calculation formula is as follows:

[0097]

[0098] Where SSIM(x,y): structural similarity index, representing the degree of structural similarity between image x and image y.

[0099] μx : Average brightness of image x. μ y : Average brightness of image y. The variance of image x represents the degree of dispersion in the brightness distribution of image x. The variance of image y represents the degree of dispersion in the brightness distribution of image y. σ xy : The covariance of image x and image y, representing the correlation of the brightness distribution of the two images. c1 and c2: Two constants introduced to avoid the denominator being zero, usually taken as c1 = (K1L)2, c2 = (K2L)2, where L is the dynamic range of pixel values ​​(e.g., for an 8-bit image, L = 255), and K1 and K2 are constants less than 1.

[0100] The above description is merely a selection of preferred embodiments of this application and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in the embodiments of this application is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described inventive concept. For example, technical solutions formed by substituting the above-described features with (but not limited to) technical features with similar functions disclosed in the embodiments of this application.

Claims

1. An image dehazing method based on a multi-scale spiking neural network, comprising: Preprocess the input fog image; The preprocessing includes image normalization and data augmentation; An embedding procedure is performed on the preprocessed image to extract preliminary features; A base layer composed of multi-scale LIFs is constructed. This base layer includes multi-scale convolutions, LIF pulse firing, and fully connected layers. The fully connected layers consist of two-dimensional convolutions connected by LeakyReLU. The construction of this multi-scale LIF base layer includes: extracting image features based on convolutional layers; performing channel blending using these image features; simulating the pulse firing mechanism of biological neurons based on LIF neuron layers; performing token blending; and converting the output of the LIF neuron layers into pulse signals based on activation function layers. Based on the linear transformation layer, the features of the input feature map and the original image are fused; Based on the activation function layer, the output port of the neural network is formed; the linear transformation layer and the activation function layer are incorporated into the fully connected neural network layer. Connect each fully connected neural network layer to form a fully connected network; The image is input into the pulse firing structure of the LIF to form a pulse firing mechanism that simulates biological neurons; Perform drop path regularization on the image output by the LIF Module; The input image is fed into a fully connected layer for regularization. Use SK-fusion to fuse features in the image; Return to the preprocessed image and execute the embedding procedure to extract preliminary features until the number of iterations is N; N is a positive integer; Unembed the output image; The embedded feature map is restored to achieve image dehazing.

2. The image dehazing method based on a multi-scale spiking neural network according to claim 1, characterized in that: The preprocessing of the input fog image includes: The data is scaled proportionally so that its pixel values ​​fall within the range of [-1, 1].

3. The image dehazing method based on a multi-scale spiking neural network according to claim 2, characterized in that: The method of inputting an image into the pulse firing structure of the LIF to form a pulse firing mechanism that simulates a biological neuron includes: Based on LIF, an MLP neural network layer is formed; The input features are converted into embedding vectors using MLP neural network layers; Based on the embedding vector, the input parameters of the spiking neuron module in the spiking firing structure are formed.

4. The image dehazing method based on a multi-scale spiking neural network according to claim 3, characterized in that: The method of forming an MLP neural network layer based on LIF includes: Add a dwconv to the first MLP layer; AxialShift, which replaces the MLP layer with LIF; Horizontal LIF based on MLP layer accumulates and propagates information in the horizontal direction of image features; Vertical LIF based on MLP layers accumulates and propagates information in the vertical direction of image features.

5. The image dehazing method based on a multi-scale spiking neural network according to claim 4, characterized in that: The method of inputting an image into the LIF pulse firing structure to form a pulse firing mechanism that simulates biological neurons also includes: By accumulating and propagating information in the horizontal direction and the vertical direction of image features, the pulse firing mechanism of biological neurons is simulated to perform feature mixing.

6. The image dehazing method based on a multi-scale spiking neural network according to claim 3, characterized in that: The image dehazing algorithm based on multi-scale spiking neural networks also includes behavioral modeling of the LIF model: when At this time, membrane potential Decays over time and receives external input : When membrane potential Reaching the threshold At this time, the neuron fires a pulse, and the membrane potential is reset to [value missing]. : ,  in It is the membrane potential. It is input from the upper layer. It is a time coefficient. It is the output. This is the firing threshold of this neuron; when a pulse is triggered, the membrane potential u is reset to... In the formula, t represents the group index.

7. The image dehazing method based on a multi-scale spiking neural network according to claim 6, characterized in that: The image dehazing algorithm based on multi-scale spiking neural networks also includes full-precision LIF modeling: We replace the 1 in the output with u, turning the original binary output into a linear output, thus preserving the full-precision information. Applying the full-precision LIF model to iterative LIF yields: Transpose of the weight matrix The input feature is r. t+1 n It is the final full-precision output at step t+1, o t+1 n It is a temporary variable that records the output state at step t+1, and its coefficient is... and These are the learning parameters for the neural network.

8. The image dehazing method based on a multi-scale spiking neural network according to claim 7, characterized in that: The image dehazing algorithm based on multi-scale spiking neural networks also includes backpropagation modeling using the chain rule of explicit iterative LIF neurons: Where L is the dynamic range of pixel values.

Images