An image restoration method and system based on pulse neural network

Abstract

This invention discloses an image restoration method and system based on a spiking neural network, comprising: processing the original degraded image to obtain multiple degraded sub-images of different resolutions; constructing an image restoration model based on a spiking neural network and a multi-scale feature fusion mechanism, which includes a pulse coding layer, a feature fusion network, and a residual pulse decoding layer; multiple pulse coding layers are used to pulse code the degraded sub-images of different resolutions to obtain multi-scale pulse features; the feature fusion network adopts a U-Net architecture containing an encoder and a decoder, which includes multiple stages, each stage processing the pulse features input from multiple pulse coding layers at each stage to achieve multi-scale feature extraction and fusion to gradually restore high-resolution image details; the residual pulse decoding layer decodes the pulse features to form the image restoration result; and the image restoration model is trained and used for image restoration, which significantly reduces the time steps and energy consumption while ensuring the image restoration quality.

Description

Technical Field

[0001] This invention relates to the field of computer vision and neuromorphic computing, and specifically to an image restoration method and system based on spiking neural networks. Background Technology

[0002] Images, as a high-information-density data carrier, play a crucial role in computer vision applications such as autonomous driving, medical imaging, and satellite remote sensing. However, during actual acquisition and transmission, images are often affected by noise interference, weather, and other factors, leading to a decrease in clarity and consequently impacting the accuracy and stability of subsequent visual tasks. Therefore, image restoration has become a key technology for ensuring the reliability of vision systems.

[0003] Most current image restoration methods are based on artificial neural network (ANN) algorithms, such as the image restoration method based on a restoration neural network model disclosed in patent application CN120355624A, and the image restoration method based on a backpropagation neural network disclosed in patent application CN108460742A. The neural network models of these methods have a considerable number of parameters, and training and inference often require enormous computational and storage resources, making them difficult to deploy on mobile devices or edge devices.

[0004] Against this backdrop, spiking neural networks (SNNs) and multi-scale feature fusion offer a new direction for overcoming existing bottlenecks: as a third-generation neural network, spiking neural networks simulate the spiking mechanism of biological neurons, employ an event-driven sparse computing model, possess low power consumption advantages, and are naturally adapted to neuromorphic hardware. Existing research shows that SNNs perform well in tasks such as image classification, object detection, and speech recognition. However, in the field of image restoration, the application of SNNs still faces challenges: firstly, traditional ANN-SNN conversion methods result in numerous time steps and low training efficiency; secondly, how to achieve multi-scale feature fusion while maintaining low power consumption advantages remains a bottleneck that urgently needs to be overcome.

[0005] Therefore, developing an efficient image restoration scheme based on spiking neural networks can not only improve the restoration quality in specific degradation scenarios, but also significantly reduce energy consumption and enable low-power deployment on mobile and edge devices, which is of great significance for promoting the practical application of image restoration technology. Summary of the Invention

[0006] Given the technical problems of high computational complexity and high energy consumption in existing image restoration methods, the purpose of this invention is to provide an image restoration method and system based on spiking neural networks. By utilizing the temporal dynamic characteristics and event-driven efficient computing mechanism of spiking neural networks, combined with a multi-scale feature fusion mechanism, the time steps are significantly reduced and energy consumption is lowered while ensuring the quality of image restoration, thereby achieving low-power and high-efficiency image restoration, which is particularly suitable for resource-constrained mobile or edge devices.

[0007] To achieve the above-mentioned objectives, an embodiment provides an image restoration method based on a spiking neural network, comprising the following steps:

[0008] The original degraded image is acquired and preprocessed to obtain multiple degraded sub-images with different resolutions;

[0009] An image restoration model is constructed based on a spiking neural network and a multi-scale feature fusion mechanism. The model includes a pulse coding layer, an encoder-decoder feature fusion network, and a residual pulse decoding layer. Multiple pulse coding layers are used to pulse-code the original degraded image and degraded sub-images of different resolutions to obtain multi-scale pulse feature sequences. The encoder-decoder feature fusion network adopts a U-Net architecture consisting of an encoder and a decoder. Each encoder and decoder includes multiple stages, each processing pulse feature sequences input from multiple pulse coding layers at each stage, achieving multi-scale feature extraction and fusion to gradually restore high-resolution image details. The residual pulse decoding layer decodes the pulse features output from the encoder-decoder feature fusion network into continuous pixel values ​​and fuses them with the input original degraded image to obtain the final image restoration result.

[0010] The image restoration model is trained and then used for image restoration.

[0011] Preferably, each pulse coding layer includes several cascaded pulse convolutional units. After processing by the pulse coding layer, the number of output channels of the pulse feature sequence is consistent with the number of input channels of the connected encoder, which is used to maintain the structural matching of cross-scale feature mapping.

[0012] Preferably, each stage of the encoder includes a pulse feature extraction module and a pulse feature fusion module connected in series. During the encoding process, the input of each stage is the output pulse feature of the previous encoding stage.

[0013] Among them, the pulse feature extraction module is used to extract deep features of input pulse features and introduces a time-channel joint attention mechanism to jointly model pulse features in the time dimension and channel dimension. Through time convolution and channel convolution, the correlation features across time steps and across channels are extracted to achieve multi-dimensional feature fusion, thereby enhancing the model's adaptability to images with different degradation levels during training.

[0014] The pulse feature fusion module is used to fuse the deep features extracted by the pulse feature extraction module with the pulse features output by the pulse coding layer of the corresponding degraded sub-image, thereby integrating and enhancing multi-scale spatial information and improving the robustness of the coded features.

[0015] The decoder also consists of multiple symmetrical stages, each corresponding one-to-one with the stages of the encoder in spatial scale in reverse order. Each stage includes a pulse feature fusion module and a pulse feature extraction module connected in series. During decoding, the input of each stage includes the output of the previous decoding stage and the output of the encoding stage corresponding to its spatial scale. These two are fused by the pulse feature fusion module to supplement spatial detail information, and then the high-resolution features are gradually recovered by the pulse feature extraction module. Through the construction of the encoder-decoder feature fusion network described above, the spiking neural network can achieve adaptive fusion of multi-scale features.

[0016] Preferably, the pulse feature extraction module includes at least two pulse convolution units and one multi-attention unit, wherein the first pulse convolution unit takes the overall input of the pulse feature extraction module as input, each subsequent pulse convolution unit takes the output of the previous pulse convolution unit as input, the multi-attention unit takes the output of the last pulse convolution unit as input, and the output of the multi-attention unit is added to the overall input of the pulse feature extraction module to obtain the overall output of the pulse feature extraction module.

[0017] Preferably, the multi-attention unit performs average pooling on the output of the corresponding pulse convolution unit with different downsampling ratios. Different downsampling ratios refer to applying different scaling factors to the output features from the corresponding pulse convolution unit in the spatial dimension for average pooling calculation, resulting in multi-scale spatial features. The low-scale spatial features are then processed by the time and channel fusion submodule and fused into the high-scale spatial features, gradually completing the feature fusion process, thereby achieving multi-scale information aggregation, and calculated according to the following formula:

[0018] , =0

[0019]

[0020] in, Let {1, 2, 3} be the index of the computational branch obtained from the corresponding downsampling. For the first The calculation results of each branch, The value is 0. The output features are from the corresponding pulse convolutional units. For features Downsampling average pooling and index Indicates the downsampling factor. For the corresponding upsampling operation and index Indicates the upsampling factor. This is handled by the time and channel fusion submodule. This is a pulse convolution operation. The time and channel fusion submodule represents the computation results of the multi-attention unit. Calculate according to the following formula:

[0021]

[0022]

[0023] in, The input features for the temporal and channel fusion submodule are of shape B×T×C×H×W, where B represents the training batch size, T represents the number of time steps, C represents the number of channels, and H and W represent the height and width of the image. The mean of the input features calculated along the spatial dimension (H×W). And its shape is B×T×C. for The transpose of and its shape is B×C×T. This represents a one-dimensional convolution computation in the time dimension, used to obtain temporal attention features. This represents a one-dimensional convolution computation along the channel dimension, used to obtain channel attention features. This indicates adding a new axis in the spatial dimension and broadcasting it along the spatial dimension to make the parameter shape consistent with... Consistent This indicates element-wise multiplication. This represents the sigmoid activation function. This is the output of the time and channel fusion submodule.

[0024] The time and channel fusion submodule jointly models the pulse features in the time and channel dimensions, realizing the dynamic correlation of pulse features between multiple time steps and multiple channels, thereby enhancing the model's adaptability to images with different degrees of degradation during training.

[0025] Preferably, the pulse feature fusion module includes a concatenation operation and a pulse convolution unit. It concatenates the input features along the channel dimension, and the resulting tensor is output after pulse convolution by the pulse convolution unit and passed to the next stage for adaptive fusion of multi-scale pulse features at each stage. In the encoder, each pulse feature fusion module fuses the deep features output by the pulse feature extraction module of the current stage with the shallow features obtained from the corresponding scale degraded sub-image through a pulse coding layer and pulse convolution, thereby forming a multi-scale comprehensive feature representation and enhancing the model's adaptability and robustness to images with different degrees of degradation. In the decoder, each pulse feature fusion module fuses the features from the output of the previous decoding stage and the output of the encoder at the corresponding scale to supplement spatial detail information during the decoding process and gradually restore the high-resolution features of the image.

[0026] Preferably, each spiking convolutional unit includes a spiking neuron, a convolution calculation layer, and a normalization layer connected in sequence, and is calculated according to the following formula:

[0027]

[0028] in, This represents the input to the pulse convolution unit. This represents neuron impulse computation, and its output is a impulse tensor containing only 0s and 1s. Convolution operation Refers to the normalization method used. It is the output of the pulse convolution unit.

[0029] Preferably, the spiking neuron adopts the leak-integral-fire (LIF) model, which is adaptable to supported application scenarios and hardware platforms, and calculates the pulse firing according to the following formula:

[0030]

[0031] Where τ is the membrane potential time constant, To reset the potential of the neuron, The pulse firing threshold, Let be the membrane potential of the nth layer neuron at time t. and These represent the potentials of the nth layer neuron after pulse firing reset at times t and t-1, respectively. For the current input, Let θ be the pulse fired by the nth neuron at time t, and let Θ be the unit step function, which has the following specific form:

[0032]

[0033] in, For the input of the unit step function, The threshold of the step function.

[0034] Preferably, the residual pulse decoding layer performs a convolution operation on the pulse sequence output by the last stage of the decoder in the time dimension after averaging it, decodes it into continuous pixel values, and adds them to the input original degraded image, so as to transform the features output by the encoder-decoder feature fusion network into the final image restoration result.

[0035] Preferably, when training the image restoration model, a weighted combination of structural similarity (SSIM) loss, spatial L1 loss, and perceptual loss is used as the comprehensive loss function, with minimizing the comprehensive loss function as the optimization objective to improve the perceptual quality of the restored image. An end-to-end alternative gradient backpropagation method is used to update the network parameters. For the non-differentiable firing function in spiking neurons, a surrogate gradient function is used for approximate differentiation to ensure effective gradient propagation across time and layers, thereby achieving stable model training. The surrogate gradient function is specifically expressed as follows:

[0036]

[0037] in, As the input to the surrogate gradient function, These are control parameters used to adjust the function curve at the threshold point. The steepness of the nearby changes.

[0038] To achieve the above-mentioned objectives, an embodiment also provides an image restoration system based on a spiking neural network, comprising:

[0039] The degraded image preprocessing module is used to acquire the original degraded image and preprocess it to obtain multiple degraded sub-images with different resolutions;

[0040] The model building module is used to construct an image restoration model based on a spiking neural network and a multi-scale feature fusion mechanism. The image restoration model includes a pulse coding layer, an encoder-decoder feature fusion network, and a residual pulse decoding layer. Multiple pulse coding layers are used to pulse-code the original degraded image and degraded sub-images of different resolutions to obtain multi-scale pulse feature sequences. The encoder-decoder feature fusion network adopts a U-Net architecture consisting of an encoder and a decoder. Each encoder and decoder includes multiple stages, each processing pulse feature sequences input from multiple pulse coding layers at each stage to achieve multi-scale feature extraction and fusion to gradually restore high-resolution image details. The residual pulse decoding layer decodes the pulse features output from the encoder-decoder feature fusion network into continuous pixel values ​​and fuses them with the input original degraded image to obtain the final image restoration result.

[0041] The model training and inference module is used to train the image restoration model for image restoration.

[0042] Compared with the prior art, the beneficial effects of the present invention include at least the following:

[0043] Leveraging the event-driven sparse computation characteristics of spiking neural networks, this method significantly reduces computational energy consumption and latency during model training and inference in image restoration applications. Combining spiking neural networks with the U-Net structure enables multi-scale, multi-stage feature fusion, improving the precision and robustness of image restoration. Employing an end-to-end direct training approach avoids the efficiency loss associated with ANN-SNN conversion, thus enhancing training and inference speed. While ensuring high-quality restoration, it also supports low-power operation on edge devices and mobile terminals, demonstrating strong practical value and application prospects. Attached Figure Description

[0044] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0045] Figure 1 This is an application environment diagram of an image restoration method based on a spiking neural network according to an embodiment of this application;

[0046] Figure 2 A flowchart illustrating an image restoration method based on a spiking neural network, provided in one embodiment of this application;

[0047] Figure 3 , Figure 4 ,as well as Figure 5 This is a schematic diagram of the structure of each module in an image restoration method based on a spiking neural network provided in an embodiment of this application;

[0048] Figure 6 A training flowchart of an image restoration model based on a spiking neural network is provided for one embodiment of this application;

[0049] Figure 7 This is a schematic diagram of the functional modules of an image restoration system based on a spiking neural network, provided in an embodiment of this application. Detailed Implementation

[0050] To facilitate understanding of the present invention, the following detailed description will be provided in conjunction with embodiments. Obviously, the described embodiments are merely some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0051] To make the objectives, features, and advantages of the present invention more apparent and understandable, the present invention will be further described below in conjunction with the accompanying drawings and specific embodiments.

[0052] The image restoration method based on a spiking neural network provided in this invention can be applied to, for example... Figure 1 In the application environment shown, the terminal device communicates with the server via a network. The data storage system stores the original degraded image and its restoration result. The data storage service can be deployed independently, integrated into the server, or implemented as cloud storage. The terminal can upload the degraded image to be restored to the server. The server constructs an image restoration model based on a spiking neural network and a multi-scale feature fusion mechanism. Based on an end-to-end alternative gradient backpropagation algorithm, the model is trained on a specific degraded image dataset. A weighted combination of SSIM loss, spatial L1 loss, and perceptual loss is used as the comprehensive loss function to obtain the trained image restoration model. The degraded image to be restored is input into the trained image restoration model, and the restored image result is output. The server can feed back the obtained restoration result to the terminal so that the user can view or further process it.

[0053] Furthermore, in some embodiments, the image restoration method based on spiking neural networks can also be implemented independently by a server or a terminal. For example, the terminal can directly process the degraded image using a locally deployed image restoration model, or the server can obtain the degraded image from a data storage service, complete the processing based on a trained image restoration model, and return the restoration result. The terminal can be a desktop computer, laptop, smartphone, smart car, drone, or other embedded device. The server can be a single server, a cluster of multiple servers, or a cloud server.

[0054] In one exemplary embodiment, such as Figure 2 The flowchart shown illustrates an image restoration method based on a spiking neural network. This method is executed by a computer device, specifically a terminal or server, or both. In this embodiment, the method is applied to... Figure 1 Taking the server in the example, the explanation includes the following steps S1 to S3:

[0055] S1: Obtain the original degraded image and perform preprocessing to obtain multiple degraded sub-images with different resolutions.

[0056] In this embodiment, the original degraded image is acquired and preprocessed by cropping, flipping, normalizing, etc. Then, bilinear interpolation is used to downsample the original degraded image with several different scaling ratios to obtain multiple degraded sub-images with different resolutions, such as reducing the original image to 1 / 2, 1 / 4, etc. of the original size. Finally, each image is extended to a time dimension and copied along that dimension to simulate the pulse input form of multiple time steps, forming an image sequence that conforms to the multi-time step input form of a spiking neural network.

[0057] Specifically, the original degraded image is first downsampled using bilinear interpolation with scaling factors of 1 / 2 and 1 / 4, respectively, to obtain the corresponding degraded sub-images. Considering the spatiotemporal modeling characteristics of spiking neural networks, the original degraded image and its degraded sub-images with an input shape of 3×H×W are copied over T time steps in the time dimension, resulting in a model input with a shape of T×3×H×W. In subsequent steps, the model performs pulse coding on the above input image, converting it into a pulse sequence for subsequent feature extraction and fusion.

[0058] S2, based on spiking neural networks and multi-scale feature fusion mechanisms, constructs an image restoration model.

[0059] like Figures 3-5 The schematic diagram of the model structure shown illustrates that the image restoration model comprises a pulse coding layer (SEB), an encoder-decoder feature fusion network, and a residual pulse decoding layer, connected sequentially along the data flow direction. The pulse coding layer performs pulse coding on the original degraded image and its degraded sub-images at different resolutions to transform static image information into a multi-scale pulse feature sequence. The encoder-decoder feature fusion network performs layer-by-layer encoding, decoding, and feature fusion on the input multi-scale pulse feature sequence to extract multi-scale spatial and temporal features and achieve a comprehensive representation of the degraded image features. The residual pulse decoding layer (SDB) decodes the pulse features output by the encoder-decoder feature fusion network, converting them into continuous pixel values, and performs residual fusion with the input original degraded image to obtain the final image restoration result.

[0060] In this embodiment, there are multiple pulse coding layers (SEBs) in the model, each corresponding to a degraded sub-image with a different sampling rate. Each SEB performs feature encoding on the degraded sub-images with different scaling ratios obtained by bilinear interpolation downsampling. The output of each SEB is connected to the pulse feature fusion module in the encoder at the corresponding spatial scale in the encoder-decoder feature fusion network, which is used to provide shallow feature representation at the corresponding scale, thereby realizing multi-scale feature extraction and fusion.

[0061] In this embodiment, each pulse coding layer (SEB) includes several cascaded pulse convolutional units (SCBs). Taking a sub-image of the corresponding resolution as input, the input image is mapped into a pulse feature sequence through these cascaded SCBs to provide shallow multi-scale pulse features. Each SCB consists of a spiking neuron (LIF), a convolution calculation (Conv), and a normalization layer (tdBN) connected in sequence. It is used for local pulse feature extraction. The input is processed by the spiking neuron to generate a binary pulse tensor, which is then convolved and normalized to obtain the unit output. To ensure cross-scale matching, the number of output channels of each SEB is consistent with the number of input channels of the pulse feature fusion module at the corresponding stage of the connected encoder. The pulse convolutional unit (SCB) is calculated according to the following formula:

[0062]

[0063] in, This represents the input to the pulse convolution unit. This represents neuron impulse computation, and its output is a impulse tensor containing only 0s and 1s. Convolution operation Refers to the normalization method used. It is the output of the pulse convolution unit.

[0064] In this embodiment, the encoder-decoder feature fusion network adopts the U-Net architecture, which includes an encoder and a decoder. The encoder and decoder each include multiple stages. Each stage is composed of a pulse feature extraction module (SFEB) and a pulse feature fusion module (MFFB) connected in series. It processes the input from multiple pulse coding layers (SEB) at each stage to achieve multi-scale feature extraction and fusion to gradually restore high-resolution image details.

[0065] In this embodiment, the pulse feature extraction module SFEB includes one or more pulse convolutional units SCB and a multi-attention unit SMAB. The first pulse convolutional unit SCB takes the overall input of the pulse feature extraction module as input, and each subsequent pulse convolutional unit SCB takes the output of the previous pulse convolutional unit as input. The multi-attention unit SMAB takes the output of the last pulse convolutional unit SCB as input, and the output of the multi-attention unit SMAB is added to the overall input of the pulse feature extraction module SFEB to obtain the overall output of the pulse feature extraction module SFEB.

[0066] The Multi Attention Unit (SMAB) first transforms the input into feature spaces at different scales through average pooling (AP) with different downsampling ratios. Different downsampling ratios refer to using different scaling factors for average pooling calculations on the output features from the corresponding pulse convolution units in the spatial dimension. The low-scale spatial features are processed by the Temporal and Channel Fusion (TCA) submodule and then incorporated into the next higher-scale spatial features through an addition operation, finally aggregating into the total features. This achieves the elimination of degenerate features from coarse to fine within each scale.

[0067] like Figure 3 As shown, the Multi-Attention Unit (SMAB) is calculated according to the following formula:

[0068] , =0

[0069]

[0070] in, Let {1, 2, 3} be the index of the computational branch obtained from the corresponding downsampling. For the first The calculation results of each branch, The value is 0. This is the output from the corresponding pulse convolution unit. For features Downsampling average pooling and index Indicates the downsampling factor. For the corresponding upsampling operation and index Indicates the upsampling factor. This is a time and channel fusion submodule. This is a pulse convolution operation. This is the computation result for the multi-attention unit. Temporal and channel fusion submodule. Calculate according to the following formula:

[0071]

[0072]

[0073] in, The input features of the time and channel fusion submodule are B×T×C×H×W (B is the training batch size, T is the number of time steps, C is the number of channels, and H and W are the height and width of the image). The mean of the input features is calculated in the spatial dimension (H×W) and has a shape of B×T×C. for The transpose of and its shape is B×C×T. This represents a one-dimensional convolution computation in the time dimension, used to obtain temporal attention features. This represents a one-dimensional convolution computation along the channel dimension, used to obtain channel attention features. This means adding a new axis in the spatial dimension and broadcasting it along the spatial dimension to make the shape of the variable consistent with... Consistent This indicates element-wise multiplication. This represents the sigmoid activation function. This is the output of the time and channel fusion submodule.

[0074] In this embodiment, the pulse feature fusion module MFFB consists of a channel dimension concatenation operation Concat and a pulse convolution unit SCB. In the encoder, the output of each stage will fuse the pulse feature sequence extracted from the corresponding degraded sub-image, enabling the model to extract pulse feature information from coarse to fine granular. In the decoder, the output of each stage will fuse the output of the encoder corresponding to each stage, taking into account both structural restoration and detail reconstruction.

[0075] In this embodiment, the residual pulse decoding layer SDB is used to perform convolution operation on the pulse feature sequence output by the last module of the decoder after averaging it in the time dimension, decode it into continuous pixel values ​​and fuse it with the original degraded image information by addition to obtain the final image restoration result.

[0076] This invention utilizes spiking neural networks to restore degraded images, fully leveraging the temporal dynamic characteristics and event-driven efficient computation mechanism of spiking neural networks. It significantly reduces computational overhead and energy consumption while maintaining image restoration quality, making it particularly suitable for deployment on neuromorphic computing chips and edge devices, with broad application prospects. Furthermore, this invention expands the application boundaries of spiking neural networks in computer vision, and is of great significance for research on neuromorphic computing and spiking neural networks.

[0077] Specifically, in this embodiment, the pulse coding layer SEB, the pulse feature extraction module SFEB, and the pulse feature fusion module MFFB all use leaky integral-and-fire (LIF) neurons, with a convolutional topology. The basic parameters of the LIF neuron include the membrane potential time constant τ, the membrane potential V, and the neuron reset potential. and membrane potential threshold At each time step when an input pulse is received, the input current I is first calculated based on the synaptic weights; then the membrane potential is updated according to the time constant τ and the input current; when the membrane potential exceeds a threshold... At time t, the neuron fires a pulse and performs a reset operation. Let the layer number of the neuron be n, and the time step be t. The formula for calculating the discretized LIF neuron pulse is as follows:

[0078]

[0079] in, Let be the membrane potential of the nth layer neuron at time t. Let be the pulse fired by the nth layer neuron at time t. For the current input, Let θ be the potential of the nth layer neuron after the impulse firing reset at time t, and let Θ be the unit step function, with the following specific form:

[0080]

[0081] in, For the input of the unit step function, The threshold of the step function.

[0082] S3, The image restoration model is trained and then used for image restoration.

[0083] In this embodiment, a specific degraded image dataset including several original degraded images and corresponding non-degraded images is used. Parameters are initialized based on the principle of maintaining the stability of the membrane potential of spiking neurons. An end-to-end alternative gradient backpropagation strategy is used to train the model based on spiking neural networks and multi-scale feature fusion mechanism to obtain the trained image restoration model.

[0084] The specific training process is as follows: Figure 6 As shown, the dataset contains original high-quality images and their corresponding degraded versions. A preprocessed training example includes: the original degraded image, degraded sub-images generated at different sampling rates, and the corresponding clear image. During training, the spatial resolution of the input images is uniformly adjusted to 256×256, and data augmentation is performed using preprocessing such as random cropping, random horizontal or vertical flipping, and random brightness perturbation to improve the model's generalization ability. Then, iterative training is initiated, traversing the clear images, degraded images, and degraded sub-images in the task set. The degraded images and degraded sub-images are input into the image restoration model based on a spiking neural network to obtain restored images. The restored images are compared with the clear images for loss calculation and backpropagation of gradient proxy error to update the spiking neural network parameters. If the number of iterations is not reached, the task set is retraced. If the number of iterations is reached, the image restoration model parameters are saved, and the model training is completed.

[0085] This invention employs supervised learning to train the image restoration model end-to-end. The training loss function is a weighted combination of structural similarity (SSIM) loss, L1 loss, and perceptual loss to balance structural fidelity and perceptual quality. During training, the network parameters are updated using Spatio-Temporal Backpropagation (STBP). For non-differentiable step functions in spiking neurons, a surrogate gradient function is used for approximation, ensuring effective gradient propagation and stable training. The surrogate gradient function can be specifically expressed as:

[0086]

[0087] in, As the input to the surrogate gradient function, These are control parameters used to adjust the function curve at the threshold point. The steepness of the nearby changes.

[0088] When a trained image restoration processing model is used for image restoration, the degraded image to be restored is input into the trained image restoration processing model, and the image restoration processing result is obtained through inference calculation.

[0089] In the embodiments, based on the above-described spiking neural network structure and training method, the performance and energy consumption of the image restoration model were tested on a public dataset. The spatial resolution of the test data was uniformly set to 256×256, and the number of time steps for the spiking neural network was 4.

[0090] Taking the image dehazing task in the subdivided application scenarios of image restoration as an example, Tables 1 and 2 show the comparison of the performance metrics of the proposed image restoration method based on spiking neural network (GSRNet) and other artificial neural networks based on SG-Net, SDA-GAN, and DehazeDDPM for the publicly available dataset SOTS-indoor.

[0091] Table 1

[0092]

[0093] Table 2

[0094]

[0095] The metrics in Table 1 above are the peak signal to noise ratio (PSNR), structural similarity (SSIM), and learned perceptual image patch similarity (LPIPS).

[0096] The number of operations in Table 2 above represents the number of multiply-accumulate (MAC) operations performed by the neural network during the entire process of restoring a degraded image. However, the network proposed in this invention utilizes multi-timestep binary pulse signals to encode and transmit information, allowing weight accumulation (AC) operations to replace the cumbersome multiplication process in traditional convolution calculations. Each AC operation requires only one floating-point addition. According to existing research, in a 45nm CMOS process, the energy consumption of a single 32-bit MAC operation is 4.6pJ (3.7 + 0.9pJ), approximately 5.1 times the energy consumption of a 32-bit AC operation (0.9pJ). This invention employs event-driven implementation, generating no computation or energy consumption when no pulse signal is present.

[0097] The experimental results above demonstrate that the image restoration method based on spiking neural networks proposed in this invention significantly reduces computational complexity and energy consumption while maintaining the restoration quality indicators achievable by existing image restoration methods, achieving low-power, high-quality image restoration. This method utilizes the temporal dynamic characteristics and event-driven efficient computation mechanism of spiking neural networks, combined with a multi-scale feature fusion mechanism, to not only improve the restoration capability of complex degraded images but also demonstrates feasibility for deployment on resource-constrained edge devices or mobile terminals. When paired with neuromorphic computing hardware, this invention has promising practical application prospects and widespread application value.

[0098] like Figure 7 As shown, the embodiment also provides an image restoration system 70 based on a spiking neural network, including a degraded image preprocessing module 71, a model building module 72, and a model training and inference module 73. The image preprocessing module 71 is used to acquire the original degraded image and perform preprocessing to obtain multiple degraded sub-images of different resolutions; the model building module 72 is used to construct an image restoration model based on a spiking neural network and a multi-scale feature fusion mechanism; and the model training and inference module 73 is used to train the image restoration model for image restoration.

[0099] It should be noted that the image restoration system based on spiking neural networks provided in the above embodiments should be illustrated using the above-described division of functional modules. The functions can be assigned to different functional modules as needed, i.e., the internal structure of the terminal or server can be divided into different functional modules to complete all or part of the functions described above. Furthermore, the image restoration system based on spiking neural networks provided in the above embodiments and the image restoration method embodiments based on spiking neural networks belong to the same concept. For details of their implementation, please refer to the image restoration method embodiments based on spiking neural networks, which will not be repeated here.

[0100] The specific embodiments described above illustrate the technical solution and beneficial effects of the present invention in detail. It should be understood that the above description is only the most preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, additions, and equivalent substitutions made within the scope of the principles of the present invention should be included within the protection scope of the present invention.

Claims

1. An image restoration method based on a spiking neural network, characterized in that, Includes the following steps: The original degraded image is acquired and preprocessed to obtain multiple degraded sub-images with different resolutions; An image restoration model is constructed based on a spiking neural network and a multi-scale feature fusion mechanism. The image restoration model includes a pulse coding layer, an encoder-decoder feature fusion network, and a residual pulse decoding layer. The pulse coding layer consists of multiple layers, which pulse-code the original degraded image and degraded sub-images of different resolutions to obtain multi-scale pulse feature sequences. The encoder-decoder feature fusion network adopts a U-Net architecture consisting of an encoder and a decoder. The encoder and decoder each include multiple stages. Each stage processes pulse feature sequences input from multiple pulse coding layers at each stage, realizing multi-scale feature extraction and fusion to gradually restore high-resolution image details. The residual pulse decoding layer is used to decode the pulse features output by the encoder-decoder feature fusion network into continuous pixel values, and fuse them with the input original degraded image to obtain the final image restoration result; The image restoration model is trained and then used for image restoration. Each stage of the encoder includes a cascaded pulse feature extraction module and a pulse feature fusion module. During the encoding process, the input of each stage is the output pulse feature of the previous encoding stage. The pulse feature extraction module is used to extract deep features from the input pulse features and introduces a time-channel joint attention mechanism to jointly model the pulse features in the time and channel dimensions. It extracts cross-time step and cross-channel correlation features through temporal convolution and channel convolution to achieve multi-dimensional feature fusion. The pulse feature fusion module is used to fuse the deep features extracted by the pulse feature extraction module with the pulse features output by the pulse coding layer of the corresponding degraded sub-image. Each stage of the decoder corresponds one-to-one with each stage of the encoder in the spatial scale in reverse order. Each stage includes a pulse feature fusion module and a pulse feature extraction module connected in series. During the decoding process, the input of each stage includes the output of the previous decoding stage and the output of the encoding stage corresponding to its spatial scale. The two are fused by the pulse feature fusion module to supplement spatial detail information, and then the high-resolution features are gradually recovered by the pulse feature extraction module. The pulse feature fusion module includes a concatenation operation and a pulse convolution unit. It concatenates the input features along the channel dimension, and the resulting tensor is output after pulse convolution by the pulse convolution unit and passed to the next stage for adaptive fusion of multi-scale pulse features at each stage. In the encoder, each pulse feature fusion module fuses the deep features output by the pulse feature extraction module of the current stage with the shallow features obtained by pulse coding and pulse convolution of the corresponding scale degraded sub-image, thereby forming a comprehensive feature representation of multiple scales. In the decoder, each pulse feature fusion module fuses the features from the output of the previous decoding stage and the output of the encoder at the corresponding scale to supplement spatial detail information during the decoding process and gradually restore the high-resolution features of the image.

2. The image restoration method based on a spiking neural network according to claim 1, characterized in that, Each pulse coding layer includes several cascaded pulse convolutional units. After processing by the pulse coding layer, the number of output channels of the pulse feature sequence is consistent with the number of input channels of the connected encoder, which is used to maintain the structural matching of cross-scale feature mapping.

3. The image restoration method based on a spiking neural network according to claim 1, characterized in that, The pulse feature extraction module includes at least two pulse convolution units and one multi-attention unit. The first pulse convolution unit takes the overall input of the pulse feature extraction module as its input, and each subsequent pulse convolution unit takes the output of the previous pulse convolution unit as its input. The multi-attention unit takes the output of the last pulse convolution unit as its input, and the output of the multi-attention unit is added to the overall input of the pulse feature extraction module to obtain the overall output of the pulse feature extraction module.

4. The image restoration method based on a spiking neural network according to claim 3, characterized in that, The multi-attention unit performs average pooling on the output of the corresponding pulse convolution unit with different downsampling ratios to obtain multi-scale spatial features. The low-scale spatial features are then processed by the time and channel fusion submodule and fused into the high-scale spatial features, thus gradually completing the feature fusion process. The calculation is performed according to the following formula: ， =0 in, Let {1, 2, 3} be the index of the computational branch obtained from the corresponding downsampling. For the first The calculation results of each branch, The value is 0. The output features are from the corresponding pulse convolutional units. For features The downsampling average pooling is used, and the subscript indicates the downsampling factor. This corresponds to the upsampling operation, and the subscript indicates the upsampling factor. This is handled by the time and channel fusion submodule. This is a pulse convolution operation. The time and channel fusion submodule represents the computation results of the multi-attention unit. Calculate according to the following formula: in, The input features for the temporal and channel fusion submodule are of shape B×T×C×H×W, where B represents the training batch size, T represents the number of time steps, C represents the number of channels, and H and W represent the height and width of the image. The mean of the input features calculated along the spatial dimension (H×W). And its shape is B×T×C. for The transpose of and its shape is B×C×T. This represents a one-dimensional convolution computation in the time dimension, used to obtain temporal attention features. This represents a one-dimensional convolution computation along the channel dimension, used to obtain channel attention features. This indicates adding a new axis in the spatial dimension and broadcasting it along the spatial dimension to make the parameter shape consistent with... Consistent, This indicates element-wise multiplication. This represents the sigmoid activation function. This is the output of the time and channel fusion submodule.

5. The image restoration method based on a spiking neural network according to claim 2 or 3, characterized in that, Each spiking convolutional unit consists of sequentially connected spiking neurons, convolution calculation, and a normalization layer, and is calculated according to the following formula: in, This represents the input to the pulse convolution unit. This represents neuron impulse computation, and its output is a impulse tensor containing only 0s and 1s. Convolution operation Refers to the normalization method used. It is the output of the pulse convolution unit.

6. The image restoration method based on a spiking neural network according to claim 5, characterized in that, The spiking neuron calculates pulse firing according to the following formula: Where τ is the membrane potential time constant, To reset the potential of the neuron, The pulse firing threshold, Let be the membrane potential of the nth layer neuron at time t. and These represent the potentials of the nth layer neuron after pulse firing reset at times t and t-1, respectively. For the current input, Let θ be the pulse fired by the nth neuron at time t, and let Θ be the unit step function, which has the following specific form: in, For the input of the unit step function, The threshold of the step function.

7. The image restoration method based on a spiking neural network according to claim 1, characterized in that, When training the image restoration model, a weighted combination of structural similarity loss, spatial L1 loss, and perceptual loss is used as the comprehensive loss function, and the minimization of the comprehensive loss function is the optimization objective. The network parameters are updated using an end-to-end alternative gradient backpropagation method. For the non-differentiable firing function in spiking neurons, a surrogate gradient function is used for approximate differentiation to ensure effective gradient propagation across time and layers. The surrogate gradient function is specifically expressed as follows: in, As the input to the surrogate gradient function, These are control parameters used to adjust the function curve at the threshold point. The steepness of the nearby changes.

8. An image restoration system based on a spiking neural network, characterized in that, include: The degraded image preprocessing module is used to acquire the original degraded image and preprocess it to obtain multiple degraded sub-images with different resolutions; The model building module is used to construct an image restoration model based on a spiking neural network and a multi-scale feature fusion mechanism. The image restoration model includes a pulse coding layer, an encoder-decoder feature fusion network, and a residual pulse decoding layer. There are multiple pulse coding layers, which pulse code the original degraded image and degraded sub-images of different resolutions to obtain multi-scale pulse feature sequences. The encoder-decoder feature fusion network adopts a U-Net architecture consisting of an encoder and a decoder. The encoder and decoder each include multiple stages. Each stage processes pulse feature sequences input from multiple pulse coding layers at each stage, realizing multi-scale feature extraction and fusion to gradually restore high-resolution image details. The residual pulse decoding layer is used to decode the pulse features output by the encoder-decoder feature fusion network into continuous pixel values, and fuse them with the input original degraded image to obtain the final image restoration result; The model training and inference module is used to train the image restoration model for image restoration. Each stage of the encoder includes a cascaded pulse feature extraction module and a pulse feature fusion module. During the encoding process, the input of each stage is the output pulse feature of the previous encoding stage. The pulse feature extraction module is used to extract deep features from the input pulse features and introduces a time-channel joint attention mechanism to jointly model the pulse features in the time and channel dimensions. It extracts cross-time step and cross-channel correlation features through temporal convolution and channel convolution to achieve multi-dimensional feature fusion. The pulse feature fusion module is used to fuse the deep features extracted by the pulse feature extraction module with the pulse features output by the pulse coding layer of the corresponding degraded sub-image. Each stage of the decoder corresponds one-to-one with each stage of the encoder in the spatial scale in reverse order. Each stage includes a pulse feature fusion module and a pulse feature extraction module connected in series. During the decoding process, the input of each stage includes the output of the previous decoding stage and the output of the encoding stage corresponding to its spatial scale. The two are fused by the pulse feature fusion module to supplement spatial detail information, and then the high-resolution features are gradually recovered by the pulse feature extraction module. The pulse feature fusion module includes a concatenation operation and a pulse convolution unit. It concatenates the input features along the channel dimension, and the resulting tensor is output after pulse convolution by the pulse convolution unit and passed to the next stage for adaptive fusion of multi-scale pulse features at each stage. In the encoder, each pulse feature fusion module fuses the deep features output by the pulse feature extraction module of the current stage with the shallow features obtained by pulse coding and pulse convolution of the corresponding scale degraded sub-image, thereby forming a comprehensive feature representation of multiple scales. In the decoder, each pulse feature fusion module fuses the features from the output of the previous decoding stage and the output of the encoder at the corresponding scale to supplement spatial detail information during the decoding process and gradually restore the high-resolution features of the image.

Images