A multi-degradation scene binocular image adaptive enhancement method based on pulse neural network

Abstract

The present application relates to a kind of based on multi-degradation scene binocular image adaptive enhancement method of pulse neural network, belong to image processing field.The method includes: constructing the pulse neural network model for the binocular image enhancement of multi-degradation scene, the network model includes first branch and second branch, all using encoder-decoder structure, in the encoder block and decoder block of first branch and second branch, feature interaction is carried out through pulse stereoscopic intersection attention module;Prepare data set to train the network model constructed, and the network model trained is evaluated, whether the image recovery effect of network model reaches performance requirement is judged, if not meet the requirement, then retraining is carried out;Finally, using the network model trained and evaluated to the binocular image of multi-degradation scene is recovered.The present application can greatly reduce the calculation energy consumption, improve the calculation efficiency of pulse network, has good removal effect to rain line and raindrop in image, with universality.

Description

Technical Field

[0001] This invention belongs to the field of image processing and relates to an adaptive enhancement method for binocular images in multiple degradation scenes based on a spiking neural network. Background Technology

[0002] Images captured by real-world devices often exhibit various degradation factors, including low light, low resolution, rain, and blur. Restoring and enhancing these images is crucial for applications such as everyday photography, autonomous driving, and outdoor surveillance. While single-task processing methods can address specific degradation types, they have significant limitations. Real-world images often suffer from multiple degradation effects simultaneously, making it difficult for single-task models to effectively handle complex situations. Furthermore, building separate models for each degradation type results in a large number of models, high resource consumption, and poor versatility. Rain, a common degradation factor, exhibits extremely complex morphology and distribution due to its density, length, type, and angle of impact. Moreover, rain is often not isolated, frequently occurring alongside other degradation factors such as fog and noise. By focusing on rain removal and constructing a general model capable of comprehensively handling multiple degradation types, it is hoped that the limitations of single-task approaches can be overcome.

[0003] Traditional monocular image deraining methods rely on image priors or deep convolutional networks to predict rain maps and restore clear images. However, due to raindrops and rain streaks obscuring the background scene, it is difficult to fully recover details in the obscured areas, resulting in information loss and blurring. Binocular image deraining utilizes complementary information from the left and right views to mine correlations between images, reducing information uncertainty and providing richer evidence for restoring clear images. Its core lies in effectively realizing the interaction of information between the left and right views. However, traditional ANN-based binocular raindrop and rain streak removal methods face challenges such as insufficient cross-view information mining, high model complexity, and low computational efficiency when dealing with complex scenes.

[0004] Spike Neural Networks (SNNs), as a new generation of neural networks, possess unique computational advantages and low-energy consumption characteristics. Unlike Artificial Neural Networks (ANNs), information in SNNs is transmitted in the form of binary pulse sequences, and computation is performed only when some neurons are activated. This pulse-driven computation method significantly reduces the energy consumption of SNNs during operation. When processing image tasks, ANNs typically require a large number of multiplication and accumulation (MAC) operations, resulting in high computational complexity. In contrast, the spiking neurons in SNNs generate pulses only when the membrane potential reaches a threshold, completing the computation through sparse synaptic accumulation operations. This enables SNNs to achieve low-power computation.

[0005] To address the problems of traditional binocular rain removal methods, researchers have proposed a variety of innovative approaches. Shi et al. [1] proposed a binocular raindrop removal method based on the row expansion attention module (RDA), which expands the attention receptive field to achieve efficient information transmission between the left and right images and introduces attention consistency loss to enhance the left-right consistency of stereo images. It performed well in both quantitative and qualitative evaluations. CDINet [2] utilizes the global context interaction module and the local detail interaction module to explore cross-view information, effectively improving the performance of binocular rain line and raindrop removal. The techniques used in these two papers emphasize the interaction of information between the left and right images. Through innovative module design and loss function, they have achieved good results in the binocular rain removal task. Although the stereo cross attention module (SCAM) proposed by NAFSSR [3] is applied to image super-resolution tasks, it has important reference value for binocular rain removal tasks. SCAM is based on the scaling dot product attention mechanism, calculates the bidirectional cross attention between the features of the left and right views and fuses cross-view and single-view features, which can effectively utilize the complementary information of binocular images. Introducing this SCAM module into the binocular rain removal task can better capture the feature differences of raindrops and rain lines in the left and right views and improve the rain removal effect.

[0006] In the development of SNN, Yao et al. [4] first proposed the pulse-driven self-attention (SDSA) mechanism, which combines the traditional Transformer with the pulse-driven paradigm of SNN. This mechanism replaces the complex matrix multiplication and softmax operation in traditional self-attention with mask and sparse addition operations to achieve low-energy self-attention computation. It has linear complexity in time and channel dimensions, providing a new direction for designing efficient attention mechanisms in binocular rain removal tasks, which helps to reduce computational costs and improve processing efficiency. Fang et al. [5] proposed SEW ResNet, which is an extension of ResNet in the SNN field. It improves the residual connection method and solves the problems of traditional Spiking ResNet in achieving identity mapping and gradient vanishing and exploding. The Membrane Shortcut (MS) [6] proposed in subsequent studies adds skip connections between the membrane potentials of spiking neurons, improves SEW Shortcut [5], and realizes pulse-driven and identity mapping, which provides the possibility for building deeper and more efficient SNNs.

[0007] At present, most of the image deraining methods adopt the ANN method. ESDNet[7] innovatively applies SNN to the single image deraining task. This network effectively alleviates the information loss problem caused by discrete binary activation by designing impulse residual blocks (SRB). At the same time, it uses hybrid attention units (MAU) to adjust the impulse response and learn rain pattern features better. It shows obvious advantages in terms of restoration effect and energy consumption, providing a new solution for single image deraining and providing strong practical support for applying SNN to the binocular deraining task.

[0008] Based on the above research, combining the interaction of left and right views with spiking neural networks provides a novel approach for the reconstruction of binocular raindrop and rain line images. On the one hand, the interaction of left and right views can fully utilize the complementary information of binocular images, reducing uncertainties in the rain removal process and improving the accuracy and completeness of rain removal. On the other hand, the introduction of spiking neural networks can leverage their advantages in energy consumption and computational efficiency, reducing the computational complexity and energy consumption of the model.

[0009] [1]Shi Z,Fan N,Yeung DY,et al.Stereo waterdrop removal with row-wisedilated attention[C] / / 2021IEEE / RSJ International Conference on IntelligentRobots and Systems(IROS).IEEE,2021:3829-3836.

[0010] [2]Nie J,Xie J,Cao J,et al.Context and detail interaction network forstereo rain streak and raindrop removal[J].Neural Networks, 2023,166:215-224.

[0011] [3]Chu

[0012] [4]Yao M, Hu J, Zhou Z, et al. Spike-driven transformer[J]. Advances inneural information processing systems, 2023,36:64043-64058.

[0013] [5]Fang W, Yu Z, Chen Y, et al. Deep residual learning in spiking neural networks [J]. Advances in Neural Information Processing Systems, 2021, 34: 21056-21069.

[0014] [6] Yao M, Hu J, Hu T, et al. Spike-driven transformer v2: Meta spiking neural network architecture inspiring the design of next-generation neuromorphic chips [J]. arXiv preprint arXiv:2404.03663, 2024.

[0015] [7]Song T, Jin G, Li P, et al. Learning a spiking neural network for efficient image deraining[J]. arXiv preprint arXiv:2405.06277,2024. Summary of the Invention

[0016] In view of this, the purpose of this invention is to provide a binocular image adaptive enhancement method for multi-degradation scenes based on spiking neural networks. By applying SNN to general image enhancement tasks and combining spiking residual blocks and spiking stereo cross attention blocks, the network can have sufficient information representation and left-right view interaction to meet the requirements of binocular general image adaptive enhancement tasks.

[0017] To achieve the above objectives, the present invention provides the following technical solution:

[0018] An adaptive enhancement method for binocular images in multi-degradation scenes based on spiking neural networks, the method comprising:

[0019] First, a spiking neural network model for binocular image enhancement in multi-degradation scenes is constructed. The network model includes a first branch and a second branch. Both the first branch and the second branch adopt an encoder-decoder structure. Between the encoder block and the decoder block of the first branch and the second branch, the outputs of the encoder blocks of the first branch and the second branch are combined by a spiking stereo cross attention module and then input into the decoder blocks of the first branch and the second branch respectively.

[0020] Secondly, prepare a dataset containing rain line images or a dataset containing raindrop images, perform data augmentation, and use it to train the constructed network model. Optimize the model parameters by calculating the loss value between the network model's output image and the real image.

[0021] Then, the trained network model is evaluated to determine whether the image restoration effect of the network model meets the performance requirements. If it does not meet the requirements, it is retrained.

[0022] Finally, a trained and evaluated spiking neural network model was used to restore stereo images of multiple degradation scenes.

[0023] Furthermore, the first branch and the second branch have the same encoder-decoder structure; wherein, the encoder block includes first to fourth encoder layers connected in sequence, the first encoder layer includes a convolution module and a feature extraction block, and the third to fourth encoder layers each include a downsampling module and a feature extraction block; the decoder block includes first to fourth decoder layers connected in sequence, the first to third decoder layers each include an upsampling module and a feature extraction block, and the fourth decoder layer includes a convolution module and a feature extraction block.

[0024] The feature extraction block comprises two cascaded pulse residual blocks. The output of the latter pulse residual block is added to the input of the feature extraction block to obtain the output of the feature extraction block. The pulse residual block also comprises two cascaded pulse convolution units II. The output of the latter pulse convolution unit II is added to the input of the pulse residual block to obtain the output of the pulse residual block.

[0025] Furthermore, the pulsed stereo cross-attention module includes first to fourth pulse-separable convolution modules and first to second pulse convolution units I; the first and second pulse-separable convolution modules receive the output F of the first branch encoder block. L The third and fourth pulse separable convolution modules receive the output F of the second branch encoder block. R The output of the third pulse separable convolution module is first transposed and then multiplied with the output of the second pulse separable convolution module to obtain the feature F1.

[0026] Feature F1 is obtained by multiplying the matrix transpose of feature F1 with the output of the first pulse separable convolution module. L→R Feature F L→R After entering the second pulse convolution unit I, the output of the second pulse convolution unit I is first compared with F. R The sums are then reshaped into matrix dimensions to obtain the feature F. R ′, the feature F R ′ is used as the input to the second branch decoder block;

[0027] Feature F1 is obtained by multiplying feature F1 by the output of the fourth pulse separable convolution module. R→L Feature F R→L After entering the first pulse convolution unit I, the output of the first pulse convolution unit I is first compared with F. L The sums are then reshaped into matrix dimensions to obtain the feature F. L ′, the feature F L ′ is used as the input to the first branch decoder block.

[0028] Among them, the first to fourth pulse separable convolutional modules have the same structure, including a pulse convolutional unit I, a spiking neuron, a 3×3 depthwise convolutional layer, a 1×1 convolutional layer and a batch normalization layer connected in sequence.

[0029] Furthermore, pulse convolution unit I includes a spiking neuron, a 1×1 convolutional layer, and a batch normalization layer connected in sequence; pulse convolution unit II includes a spiking neuron, a 3×3 convolutional layer, and a batch normalization layer connected in sequence.

[0030] Furthermore, the data augmentation includes random cropping and horizontal flipping; the random cropping involves randomly selecting a sub-region of a specific size from any original image in the dataset as a new image; the horizontal flipping involves flipping the image along the vertical central axis to generate a new image.

[0031] Furthermore, the evaluation of the trained spiking neural network model includes: using peak signal-to-noise ratio and structural similarity index as evaluation indicators to evaluate the image restoration effect of the spiking neural network model; and calculating the energy consumption of the spiking neural network model through synaptic operands to evaluate the energy consumption advantage of the spiking neural network model.

[0032] The beneficial effects of this invention are as follows: This invention proposes an adaptive enhancement method for binocular images in multiple degradation scenes based on spiking neural networks. It innovatively integrates the sparse computational characteristics of SNNs with the binocular information complementarity mechanism through a spiking stereo cross-attention module, and replaces the dense matrix operations of traditional cross-attention with spiking convolutional units. While maintaining the binocular feature difference perception capability, it can significantly reduce computational energy consumption and improve the computational efficiency of the spiking network. Simultaneously, this invention constructs feature extraction blocks through a hierarchical spiking residual architecture, introducing membrane potential jump connections and spiking driven residual blocks (SRBs), effectively alleviating the information attenuation problem caused by binary pulses, resulting in a significant improvement in peak signal-to-noise ratio (PSNR) and a substantial reduction in the number of synaptic operations. Furthermore, this invention combines spiking residual blocks and spiking stereo cross-attention, enabling the spiking neural network model to acquire sufficient information representation and left-right view interaction. It has a good removal effect on two completely different degradation forms in images: rain lines and raindrops, and can visually eliminate rain line and raindrop traces in images, possessing versatility.

[0033] Other advantages, objectives, and features of the invention will be set forth in part in the description which follows, and in part will be apparent to those skilled in the art from the following examination, or may be learned from practice of the invention. The objectives and other advantages of the invention can be realized and obtained through the following description. Attached Figure Description

[0034] To make the objectives, technical solutions, and advantages of the present invention clearer, the preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings, wherein:

[0035] Figure 1 The flowchart of training and evaluation of the spiking neural network proposed in this invention.

[0036] Figure 2 The overall structure diagram of the spiking neural network proposed in this invention.

[0037] Figure 3 The evaluation results of the spiking neural network proposed in this invention on the RainKITTI 2015 dataset are shown.

[0038] Figure 4 The evaluation results of the spiking neural network proposed in this invention on the Stereo Waterdrop dataset are presented.

[0039] Figure 5 This is a schematic diagram of the process of the binocular image adaptive enhancement method based on spiking neural network for multi-degradation scenes proposed in this invention. Detailed Implementation

[0040] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0041] The accompanying drawings are for illustrative purposes only and are schematic diagrams, not actual pictures. They should not be construed as limiting the invention. To better illustrate the embodiments of the invention, some parts in the drawings may be omitted, enlarged, or reduced, and do not represent the actual product dimensions. It is understandable to those skilled in the art that some well-known structures and their descriptions may be omitted in the drawings.

[0042] In the accompanying drawings of the embodiments of the present invention, the same or similar reference numerals correspond to the same or similar components. In the description of the present invention, it should be understood that if terms such as "upper," "lower," "left," "right," "front," and "rear" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, they are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, the terms used to describe positional relationships in the drawings are only for illustrative purposes and should not be construed as limiting the present invention. For those skilled in the art, the specific meaning of the above terms can be understood according to the specific circumstances.

[0043] This embodiment provides an adaptive enhancement method for binocular images in multi-degradation scenes based on a spiking neural network, such as... Figure 5 As shown, the method includes:

[0044] 1. Construct a spiking neural network model for binocular image enhancement in multi-degradation scenes, such as... Figure 2 As shown, the network model includes a first branch and a second branch. Both the first branch and the second branch adopt an encoder-decoder structure (for example, a 4-layer U-Net architecture can be used). Between the encoder blocks and decoder blocks of the first branch and the second branch, the outputs of the encoder blocks of the first branch and the second branch are combined by a pulse stereo cross attention module and then input into the decoder blocks of the first branch and the second branch respectively.

[0045] (1) The first branch and the second branch have the same encoder-decoder structure.

[0046] The encoder block comprises four encoder layers connected sequentially. The first encoder layer includes a convolution module and a feature extraction block, while the third and fourth encoder layers each include a downsampling module and a feature extraction block. During the encoding stage, the first encoder layer first obtains the features of the input image, and the subsequent three encoder layers perform three downsampling operations to obtain features at different scales.

[0047] The decoder block consists of four sequentially connected decoder layers: the first to the third layer, each containing an upsampling module and a feature extraction block; and the fourth layer, containing a convolution module and a feature extraction block. During the decoding stage, the feature map is first upsampled three times by the first to third layer decoders to restore it to the original image size. Then, the fourth layer decoder performs image reconstruction to obtain the final restored result.

[0048] The feature extraction block comprises two cascaded pulse residual blocks. The output of the latter pulse residual block is added to the input of the feature extraction block to obtain the output of the feature extraction block. The pulse residual block also comprises two cascaded pulse convolutional units II. The output of the latter pulse convolutional unit II is added to the input of the pulse residual block to obtain the output of the pulse residual block.

[0049] (2) The pulsed stereo cross attention module includes the first to fourth pulse separable convolution modules and the first to second pulse convolution units I.

[0050] The first and second pulse separable convolution modules receive the output F of the first branch encoder block. L The third and fourth pulse separable convolution modules receive the output F from the second branch encoder block. R The output of the third pulse separable convolution module is first matrix transposed and then multiplied with the output of the second pulse separable convolution module to obtain feature F1. Feature F1 is then matrix transposed and multiplied with the output of the first pulse separable convolution module to obtain feature F. L→R Feature F L→R After entering the second pulse convolution unit I, the output of the second pulse convolution unit I is first compared with F. R The sums are then reshaped into matrix dimensions to obtain the feature F. R ′, the feature F R ′ is used as the input to the second branch decoder block. Feature F1 is multiplied by the output of the fourth pulse-separable convolution module to obtain feature F. R→L Feature F R→L After entering the first pulse convolution unit I, the output of the first pulse convolution unit I is first processed and then combined with F. L The sums are then reshaped into matrix dimensions to obtain the feature F. L ′, the feature F L ′ serves as the input to the first branch decoder block.

[0051] Among them, the first to fourth pulse separable convolution modules have the same structure, including a pulse convolution unit I, a spiking neuron, a 3×3 depthwise convolution layer, a 1×1 convolution layer, and a threshold-dependent batch normalization layer (tdBN) connected in sequence.

[0052] (3) Spike Convolutional Unit (SCU)

[0053] The spiking convolutional unit is composed of spiking neurons, a convolutional layer, and threshold-dependent batch normalization (tdBN). There are two types of spiking neurons. In this embodiment, the LIF (Leaky Integrate-and-Fire) neuron is used. This neuron simulates the spiking signal transmission process of biological neurons in the human brain, which is divided into three steps: charging, firing, and resetting, as shown in equations (1), (2), and (3), respectively.

[0054] H(t)=f(V[t-1],X[t])(1)

[0055] S[t]=θ(H[t]-V threshold (2)

[0056]

[0057] V[t]=H[t]·(1-S[t])+V reset ·S[t] (4)

[0058] Where X[t] represents the current input feature, V[t-1] and H[t] represent the previous state and the current membrane potential, respectively, and the function f is the neuron state update equation. The charging process is the process of updating the neuron membrane potential based on the membrane potential of the previous state and the current input feature; if the updated membrane potential exceeds the threshold potential V... threshold If a pulse is emitted, S[t] represents the current neuron pulse output. The step function θ(x) is used to determine the magnitude of the current membrane potential relative to the threshold potential, outputting 0 or 1. After the discharge ends, the membrane potential is reset, V reset To reset the potential.

[0059] Based on the kernel size of the convolutional layers in the spiking convolutional unit, this invention employs spiking convolutional unit I and spiking convolutional unit II. Structurally, both include sequentially connected spiking neurons, convolutional layers, and tdBNs. The tdBNs, during normalization, consider both temporal and spatial dimensions, better aligning with the input data in spiking neural networks, which contains both spatial and temporal information. Specifically, the convolutional kernel of the convolutional layer in spiking convolutional unit I is 1×1, and the convolutional kernel of the convolutional layer in spiking convolutional unit II is 3×3.

[0060] (4) Spike Separable Convolution (SSC) module

[0061] In the pulse separable convolution module, the signal first passes through pulse convolution unit I, and then is sequentially connected to LIF neurons, 3×3 depthwise convolutional layers, 1×1 convolutional layers, and tdBN. This is the application of separable convolution in pulse networks. Compared with ordinary convolution, separable convolution can reduce the number of parameters and computation, and speed up the training process.

[0062] (5) Spike Residual Block (SRB) and Feature Extract Block (FEB)

[0063] The pulsed residual block contains two pulsed convolutional units II and one shortcut connection. This structure is very similar to the residual basic block, except that the ReLU activation function is removed due to the discreteness of the data in the pulsed network. The feature extraction block contains two pulsed residual blocks and one shortcut connection. The feature extraction block can effectively obtain local information of the image through multiple layers of pulsed convolutional units.

[0064] (6) Spike Stereo Cross Attention Module (SSCAM)

[0065] Spurious Stereo Cross Attention (SCAM) is an implementation of the Stereo Cross Attention Module (SCAM) on a spiking neural network. SCAM is a variant of Self-Attention (SA). Self-Attention performs matrix operations on the query vector and the key vector, and then uses softmax to obtain the correlation weights between positions in the input matrix. Based on these weights, it performs a weighted summation of the value vector to obtain contextual information closely related to the query. The calculation principle is shown in Equation (5).

[0066]

[0067] Where Q, K, and V are all derived from the input matrix through a weight matrix transformation, and their dimensions are all H×W×C, representing the image height, width, and number of channels, respectively. It has the effect of accelerating convergence. In stereo vision tasks, Q can be attributed to one type of view information (e.g., the left image), while K and V come from another view (e.g., the right image). This leads to the principle of stereo cross-attention, as shown in the following equation:

[0068]

[0069] Among them, W1 L W1 R , Both are weight matrices, F L Input matrix F for the left figure R Input matrix F for the right figure R→L and F L→R This is the result of cross-attention between the left and right images. Due to the characteristics of stereo vision, horizontal parallax occurs, while it does not occur in the vertical direction. Therefore, attention calculation is more efficient in the horizontal direction. Finally, cross-view information and intra-view information are fused, i.e., F... R→L and F L→R After performing one transformation, respectively with F L and F R By performing matrix addition, the 3D cross-attention module can effectively enable interaction between the left and right images.

[0070] Replacing the convolutional layer with a pulsed convolutional unit and removing the softmax activation function yields the pulsed stereo cross-attention module, such as... Figure 2 As shown. The formula for the pulse stereo cross-attention module is expressed as follows:

[0071]

[0072]

[0073] W1 L W1 R , W3 L W3 R This refers to the second, third, first, and fourth pulse separable convolutional blocks and two 1×1 pulse convolutional units in the pulsed stereo cross-attention module. "*" indicates matrix multiplication, and T indicates matrix transpose. After the two input matrices are input, the attention weight map, W1, is calculated. L F L *(W1 R F R ) T and This is essentially the result of operations on the query vector and the key vector, and respectively compared with... and That is, the corresponding value vectors are operated on to obtain their respective cross-attention results, and after a transformation, a jump connection is performed to obtain the output matrices of the left and right images.

[0074] 2. Regarding the network model constructed above, such as Figure 1 As shown, a dataset containing rain line images or a dataset containing raindrop images is prepared, and after data augmentation, it is used to train the constructed network model. The model parameters are optimized by calculating the loss value between the network model output image and the real image.

[0075] This embodiment uses two publicly available stereo synthetic datasets: the RainKITTI 2015 dataset and the Stereo Waterdrop dataset. The RainKITTI 2015 dataset was created by artificially adding rain lines to the KITTI2015 dataset using Photoshop. It contains two subsets: a training set and a test set. The training set contains 4200 stereo image pairs, and the test set contains 4189 stereo image pairs. The Stereo Waterdrop dataset was created by adding a transparent glass plate in front of the lens of a ZED 2 stereo camera (a camera capable of producing stereo images), and then spraying water onto the glass plate to simulate raindrops. It contains a training set, a validation set, and a test set, with 642, 89, and 106 stereo image pairs in each subset, respectively. An image pair refers to (a real image without rain, and an image with rain). A stereo image pair is equivalent to (a left image pair, a right image pair), and a stereo image pair is equivalent to four images.

[0076] After each dataset is read, data augmentation is performed through random pruning and horizontal flipping.

[0077] Random cropping involves randomly selecting a sub-region of a specific size from the original image to create a new image. The core idea is to change the spatial location and extent of the image, allowing the model to learn the features of different local regions and increasing data diversity. The RainKITTI 2015 and Stereo Waterdrop datasets use sizes of 64×64 pixels and 128×128 pixels, respectively, with initial sizes and locations randomized using the Random algorithm.

[0078] Horizontal flipping involves flipping an image along its vertical central axis. This reverses the left-right relationships in the original image, generating a new image sample. This expands the amount of data, enriches the data format, and allows the model to learn more diverse features. It also helps prevent overfitting and improves generalization.

[0079] After data augmentation, we obtain (gt_left, lq_left, gt_right, lq_right), where (lq_left, lq_right) serve as the model's input data, representing the degraded left and right images, and (gt_left, gt_right) represent the ground truth left and right images, used in conjunction with the restored image (rst_left, rst_right) to calculate the loss value. In this embodiment, the L1 Loss function is used. The loss values ​​of the left and right images are calculated separately, and the average is taken as the loss value for each iteration of the model for backpropagation.

[0080] 3. Evaluate the trained network model to determine whether its image restoration performance meets the requirements. If it does not meet the requirements, retrain the network model.

[0081] The image restoration evaluation metrics used are Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity Index Measure (SSIM). The PSNR is measured in decibels (dB). The higher the PSNR value, the fewer the interference factors in the image, which means the better the image restoration effect of the model. The SSIM value ranges from 0 to 1. The closer the value is to 1, the higher the image quality.

[0082] After the network model was trained on the RainKITTI 2015 dataset and the Stereo Waterdrop dataset, two pre-trained models were obtained, and then evaluated on the test sets of the two datasets respectively.

[0083] For calculating the energy consumption of the network model, synaptic operations (SOPs) are the computational cost representation in SNNs, typically referring to accumulation operations (ACs), each consuming 0.9 pJ of energy. Floating-point operations (FLOPs) are the computational cost representation in ANNs, typically referring to multiply-accumulate operations (MACs), each consuming 4.6 pJ of energy. In this embodiment, both synaptic operations and FLOPs are calculated on a 256×256 RGB image. The energy consumption calculation formula for SNN is:

[0084] E ANN = 4.6pJ × FLOPs (12)

[0085] E SNN=0.9pJ×SOPs (13)

[0086] To compare the energy efficiency advantage of the present invention using SNN compared to the traditional ANN, an ANN-based network model with the same structure was designed in the experimental stage.

[0087] 4. A trained and evaluated spiking neural network model is used to restore binocular images of multiple degradation scenes.

[0088] In the RainKITTI 2015 dataset, the test set contains 4189 image pairs. The proposed network model achieves an average PSNR of 31.26, an average SSIM of 0.9187, and an energy consumption of 61.33 mJ, while the ANN-based network model consumes 510.20 mJ. In the Stereo Waterdrop dataset, the test set contains 106 image pairs. The proposed network model achieves an average PSNR of 25.91, an average SSIM of 0.8923, and an energy consumption of 59.53 mJ, while the ANN-based network model consumes 510.20 mJ. It can be seen that in the energy consumption comparison of the two datasets, the SNN structure used in this invention exhibits a significant reduction in energy consumption.

[0089] Taking one example from all recovery graphs in the test set, the qualitative evaluation results of the network model proposed in this invention on the two datasets are as follows: Figure 3 and Figure 4 As shown, in the RainKITTI 2015 dataset, compared to the degraded image, the images restored by the model proposed in this invention show improved PSNR and SSIM values ​​of 17.05 and 0.4051 respectively in the left image, and improved PSNR and SSIM values ​​of 16.57 and 0.417 respectively in the right image. The traces of rain lines are almost invisible in the restored image. In the Stereo Waterdrop dataset, compared to the degraded image, the PSNR and SSIM values ​​of the left image are improved by 6.88 and 0.0929 respectively, and improved PSNR and SSIM values ​​of the right image by 3.01 and 0.0747 respectively. The raindrops are also basically removed visually.

[0090] The final recovery results show that the method proposed in this invention has good results in removing rain streaks and raindrops, proving the effectiveness of spiking neural networks in binocular rain removal tasks. The removal effect of rain streaks and raindrops, two completely different forms of degradation, proves the versatility of this method.

[0091] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A method for adaptive enhancement of binocular images in multi-degradation scenes based on spiking neural networks, characterized in that, The method includes: First, a spiking neural network model for binocular image enhancement in multi-degradation scenes is constructed. The network model includes a first branch and a second branch. Both the first branch and the second branch adopt an encoder-decoder structure. Between the encoder block and the decoder block of the first branch and the second branch, the outputs of the encoder blocks of the first branch and the second branch are combined by a spiking stereo cross attention module and then input into the decoder blocks of the first branch and the second branch respectively. The first branch and the second branch have the same encoder-decoder structure; wherein, the encoder block includes first to fourth encoder layers connected in sequence, the first encoder layer including a convolution module and a feature extraction block, and the third to fourth encoder layers each including a downsampling module and a feature extraction block; the decoder block includes first to fourth decoder layers connected in sequence, the first to third decoder layers each including an upsampling module and a feature extraction block, and the fourth decoder layer including a convolution module and a feature extraction block; the feature extraction block includes two cascaded pulse residual blocks, the output of the latter pulse residual block being added to the input of the feature extraction block to obtain the output of the feature extraction block; the pulse residual block includes two cascaded pulse convolution units II, the output of the latter pulse convolution unit II being added to the input of the pulse residual block to obtain the output of the pulse residual block; The pulsed stereo cross-attention module includes first to fourth pulse separable convolution modules and first to second pulse convolution units I; the first and second pulse separable convolution modules receive the output of the first branch encoder block. The third and fourth pulse separable convolution modules receive the output of the second branch encoder block. The output of the third pulse separable convolution module is first transposed and then multiplied with the output of the second pulse separable convolution module to obtain the feature. ;feature After matrix transposition, the feature is obtained by multiplying it with the output of the first pulse separable convolution module. ,feature The output of the second pulse convolution unit I, after processing, is first combined with... The sums are then reshaped into matrix dimensions to obtain the features. The features As input to the second branch decoder block; features The feature is obtained by multiplying it with the output of the fourth pulse separable convolution module. ,feature After entering the first pulse convolution unit I, the output of the first pulse convolution unit I is first combined with... The sums are then reshaped into matrix dimensions to obtain the features. The features As input to the first branch decoder block; Secondly, prepare a dataset containing rain line images or a dataset containing raindrop images, perform data augmentation, and use it to train the constructed spiking neural network model. Optimize the model parameters by calculating the loss value between the output image of the spiking neural network model and the real image. Then, the trained spiking neural network model is evaluated to determine whether the image restoration effect of the spiking neural network model meets the performance requirements. If it does not meet the requirements, it is retrained. Finally, a trained and evaluated spiking neural network model was used to restore stereo images of multiple degradation scenes.

2. The method according to claim 1, characterized in that, The first to fourth pulse-separable convolutional modules have the same structure, including a pulse convolutional unit I, a spiking neuron, a 3×3 depthwise convolutional layer, a 1×1 convolutional layer and a batch normalization layer connected in sequence.

3. The method according to claim 1, characterized in that, The data augmentation includes random cropping and horizontal flipping; the random cropping involves randomly selecting a sub-region of a specific size from any original image in the dataset as a new image; the horizontal flipping involves flipping the image along the vertical central axis to generate a new image.

4. The method according to claim 1, characterized in that, The evaluation of the trained spiking neural network model includes: using peak signal-to-noise ratio and structural similarity index as evaluation metrics to assess the image restoration performance of the spiking neural network model; and calculating the energy consumption of the spiking neural network model using synaptic operands to evaluate the energy efficiency advantage of the spiking neural network model.

Images