An image restoration method, system and storage medium

Abstract

This application relates to an image restoration method, system, and storage medium. The method first constructs a multi-time-step input sequence from a static degraded image. Second, a retina / LGN module performs center-periphery antagonism processing, separating the low-frequency structural features of the first path and the high-frequency detail features of the second path. Then, V1 / V2 dual-path encoders are used to model the image separately, and a cross-gating mechanism is employed to achieve collaborative interaction. Next, a V4 module performs multi-scale fusion, and an IT module performs large receptive field semantic modulation. Finally, a decoder combines skip connections with stepwise upsampling to recover the residual image. The results from each time step are integrated, averaged, and added to the input to obtain the final restored image. This invention achieves high-quality image de-raining and de-hazing restoration while maintaining low power consumption.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and in particular to an image restoration method, system and storage medium. Background Technology

[0002] With the development of computer vision technology, single-image restoration technology is playing an increasingly important role in fields such as intelligent sensing, unmanned systems, traffic monitoring, and remote sensing imaging. During actual image acquisition, images are often affected by environmental factors such as raindrop occlusion, fog scattering, and light attenuation, leading to reduced image contrast, blurred details, and loss of structural information. Since the degradation process is usually nonlinear and irreversible, the single-image restoration problem is theoretically an ill-conditioned inverse problem, posing a significant challenge to modeling.

[0003] In existing technologies, most image restoration tasks, such as rain removal and dehazing, employ continuous-valued deep learning methods based on convolutional neural networks or Transformers. These methods enhance feature representation capabilities through multi-scale convolution, attention mechanisms, or frequency domain transformations, achieving certain results on specific datasets. However, these methods generally rely on intensive floating-point operations, resulting in high computational complexity and energy consumption, making them unsuitable for resource-constrained or low-power applications. Furthermore, their network architecture design largely depends on engineering experience, limiting the interpretability and generalization ability of the models.

[0004] In existing technologies, for example, CN114648467B discloses an image dehazing method. This method first uses a dark channel prior dehazing algorithm to process the image to be processed to obtain a coarse transmittance map. Then, it determines the scattered atmospheric light component corresponding to the image to be processed. Next, it uses a transmittance map filtering network model trained based on a convolutional neural network to filter the coarse transmittance map to obtain a refined transmittance map. Finally, it substitutes the image to be processed, the scattered atmospheric light component, and the refined transmittance map into an atmospheric scattering model to obtain a dehazed image. However, this method relies on the dark channel prior algorithm to obtain the coarse transmittance map. The basic assumptions of the dark channel prior do not hold in sky areas, white objects, or high-brightness scenes, which can easily lead to incorrect transmittance estimation, resulting in color distortion or halo artifacts, and severely limiting its generalization ability. Secondly, this method is based on a traditional convolutional neural network, which uses dense floating-point multiply-accumulate operations inside the network. This results in high computational complexity and high energy consumption, making it difficult to deploy on resource-constrained edge computing platforms such as drones, vehicle systems, and mobile terminals, thus limiting its practical application in low-power scenarios. For example, CN117437132A discloses an image restoration method. This method uses multiple filter kernels to filter the target data to obtain auxiliary data, then concatenates the auxiliary data with the input data of each layer of a neural network model to generate new input data, and finally executes the neural network model to obtain the restored image. However, this method has limited restoration performance in complex scenes such as non-uniform rain and fog, and mixed noise. Summary of the Invention

[0005] The purpose of this invention is to provide an image restoration method, system, and storage medium that partially solves or alleviates the above-mentioned shortcomings in the prior art, and achieves high-quality image de-raining and de-hazing restoration while maintaining the advantage of low power consumption.

[0006] To solve the aforementioned technical problems, the present invention specifically adopts the following technical solution: A first aspect of the present invention is to provide an image restoration method, comprising the following steps: S100: Acquire the degraded image; S200: Input the degraded image into an image restoration model for restoration processing to obtain the final restored image, including: S201: Perform biomimetic early visual processing on the degraded image, including center-periphery antagonistic enhancement, and use learnable threshold LIF neurons for pulse coding to obtain pulse features; S202: Feature extraction of the pulse features is performed using a dual-path spiking neural network, including: The first path of the dual-path spiking neural network is used to extract global structural features from the pulse features, and the second path of the dual-path spiking neural network is used to extract local detail features from the pulse features. The first pathway uses a first time constant to simulate fast response characteristics and utilizes a large receptive field modeling mechanism to extract brightness-dominated global structural features; the second pathway uses a second time constant to simulate a fine integration process and utilizes a high resolution preservation mechanism to extract color-dominated local detail features. S203: The global structural features and the local detail features are fused to obtain mid-level fused features, including: By performing convolution operations with different receptive fields, global structural features and local detail features are fused to construct a mid-level representation feature with semantic meaning, and multi-scale contextual information is extracted using the mid-level representation feature. By utilizing multi-scale contextual information and attention mechanisms, salient structural regions and detailed textures in the mid-level representation features are selectively enhanced to obtain mid-level fused features; S204: Perform semantic modeling on the mid-level fusion features to obtain high-level semantic features, including: Large convolutional kernels are used to perform convolution operations to capture global contextual information in the mid-level fusion features, so as to extract high-level representation features with semantic meaning; and compact global context vectors are extracted from the high-level representation features through global average pooling. The high-level representation features are then adaptively modulated in terms of channel dimension using the global context vectors to obtain high-level semantic features. S205: The high-level semantic features are upsampled and fused step by step using a multi-level decoding and recovery structure to obtain a preliminary recovered image; S206: Construct a multi-time-step input sequence for the degraded image, and perform temporal integration on the preliminary restored image corresponding to the multi-time-step input sequence to obtain the final restored image; The image restoration model is obtained in the following way: Obtain a training dataset, which includes degraded images and corresponding real clean images; The image restoration model is trained using the training dataset to obtain the trained image restoration model; S300: Obtain the degraded image to be processed, input the degraded image to be processed into the trained image restoration model, and output the restored image.

[0007] Furthermore, in step S201, the center-periphery antagonistic processing of the degraded image is implemented using the difference of Gaussian operator and encoded as pulse features using LIF neurons with learnable thresholds, including: The input degraded image is subjected to center-periphery antagonism processing to suppress smooth regions and enhance local contrast and edge information, resulting in enhanced features; The enhanced features are converted into input currents and input into LIF neurons with learnable thresholds for pulse encoding to obtain pulse features.

[0008] Furthermore, step S201 also includes: An adaptive weight is generated using a gating mechanism, and the enhanced features are then weighted and fused with the degraded image using the adaptive weight to obtain adaptive contrast enhancement features. The adaptive contrast enhancement feature is used as an input current to the spiking neuron model for pulse coding.

[0009] Furthermore, in step S202, extracting global structural features from the pulse features using the first path of the dual-path spiking neural network specifically includes: The brightness component is extracted from the input pulse features, and the multi-channel pulse features are weighted and fused to obtain the single-channel brightness pulse features. The brightness pulse features are input into a LIF neuron with a first time constant for processing. The fast response characteristics of the first pathway are simulated through fast integration and leakage mechanisms to obtain the pulse sequence of the first pathway. The first path pulse sequence is subjected to dilated convolution operation to expand the receptive field to capture a wide range of brightness change information, thereby obtaining the enhanced first path pulse features; The enhanced first pathway pulse features are aggregated in the time dimension, and the average firing rate of each neuron in all time steps is calculated to obtain the brightness-dominated global structural features.

[0010] Furthermore, in step S202, extracting local detail features from the pulse features using the second path of the dual-path spiking neural network specifically includes: The color information of the input pulse features is preserved, maintaining the independence of the red, green, and blue channels, to obtain the color pulse features; The color pulse features are input into LIF neurons with a second time constant for processing. The fine integration process of the second pathway is simulated through a slow integration and accumulation mechanism to obtain the pulse sequence of the second pathway. The second path pulse sequence is subjected to a standard convolution operation. A small-sized convolution kernel is used to densely sample in a local region to maintain spatial resolution. The enhanced second path pulse features are obtained through residual connections. The enhanced second pathway pulse features are aggregated in the time dimension, and the average firing rate of each neuron in all time steps is calculated to obtain color-dominated local detail features. Wherein, the second time constant is greater than the first time constant.

[0011] Furthermore, S204 specifically includes: The mid-layer fusion features are processed in parallel using convolutions with receptive fields of various sizes to extract global contextual information at different receptive field scales, thereby obtaining semantic features at multiple scales. The semantic features at multiple scales are fused and residually connected with the input mid-level fused features to obtain high-level representation features with rich global semantic information. The high-level representation features are spatially compressed using global average pooling to extract a compact global context vector. The global context vector is input into a lightweight multilayer perceptron for nonlinear transformation to generate channel-dimensional modulation weights. The high-level representation features are subjected to adaptive weighted modulation on a channel-by-channel basis using the modulation weights to obtain the modulated features; The modulated features are residually concatenated with the high-level representation features to obtain the final high-level semantic features.

[0012] Furthermore, S205 specifically includes: S2051. For the first decoding stage, the high-level semantic features are used as input coarse-scale features. For subsequent decoding stages, the output features of the previous decoding stage are used as input coarse-scale features for the next decoding stage. Upsample the coarse-scale features of the current decoding stage to restore them to the target resolution, and obtain the upsampled features. S2052. Perform channel alignment on the upsampled features by adjusting the number of channels to the same dimension as the corresponding level skip connection features through convolution operations to obtain channel-aligned features. S2053. The channel-aligned features are spliced ​​and fused with the corresponding level of jump connection features to obtain stage fusion features; S2054. Perform multi-scale enhancement on the stage fusion features by convolutional operations with different dilation rates, and extract details and structural information of different receptive fields to obtain multi-scale enhanced features. S2055. Perform residual fusion on the multi-scale enhanced features and add them to the stage fusion features to obtain residual enhanced features; S2056. The residual enhancement features are refined by introducing an attention mechanism to highlight key structural regions and texture details, and the output features of the current decoding stage are obtained as the input coarse-scale features of the next decoding stage. S2057. Repeat steps S2051 to S2056 to perform stepwise upsampling and feature fusion, and pass the decoded features stepwise until the image is restored to the same resolution as the input degraded image, thus obtaining a preliminary restored image.

[0013] Furthermore, S206 specifically includes: A multi-time-step input sequence of length T is constructed for the degraded image, wherein the input for each time step is obtained by applying a small multiplicative perturbation sampled from a uniform distribution to the degraded image; The input at each time step is restored using the methods in steps S201 to S205 to obtain preliminary restored images for all time steps. The preliminary restored images at all time steps are time-averaged to obtain the final restored image.

[0014] Furthermore, in step S206, the loss function for training the image restoration model includes Charbonnier loss, structural similarity loss, and perceptual loss, and different weights are assigned to different losses. The perceptual loss is achieved by extracting high-level features of the final restored image and the real clean image at a specified layer through a perceptual feature extraction network to obtain the first high-level feature and the second high-level feature, and calculating the Euclidean distance between the first high-level feature and the second high-level feature as the loss term. And / or, A dynamic weighting strategy based on Warmup is adopted to gradually increase the weights of structural similarity loss and perceptual loss during training.

[0015] Furthermore, the learnable threshold-capable LIF neuron includes: For each LIF neuron, set unconstrained learnable parameters, convert the learnable parameters into positive threshold adjustment values ​​through nonlinear mapping, and add them to a preset minimum threshold constant to obtain the actual discharge threshold, ensuring that the actual discharge threshold is always positive and far from zero. The learnable parameters are initialized based on the desired initial threshold, and the difference between the desired initial threshold and the minimum threshold constant is converted into the initial value of the learnable parameters using the inverse transformation of the nonlinear mapping.

[0016] Secondly, this application also discloses an image restoration system, the system comprising: Degraded image acquisition module, configured to acquire degraded images; Image restoration model building modules include: The pulse feature extraction submodule is configured to perform biomimetic early visual processing on the degraded image, including center-periphery antagonistic enhancement, and pulse coding using LIF neurons with learnable thresholds to obtain pulse features; A pathway feature extraction module, configured to extract features from the pulse features using a dual-path spiking neural network, includes: The first path of the dual-path spiking neural network is used to extract global structural features from the pulse features, and the second path of the dual-path spiking neural network is used to extract local detail features from the pulse features. The first pathway uses a first time constant to simulate fast response characteristics and utilizes a large receptive field modeling mechanism to extract brightness-dominated global structural features; the second pathway uses a second time constant to simulate a fine integration process and utilizes a high resolution preservation mechanism to extract color-dominated local detail features. The fusion processing submodule performs fusion processing on the global structural features and the local detail features to obtain mid-level fused features, including: By performing convolution operations with different receptive fields, global structural features and local detail features are fused to construct a mid-level representation feature with semantic meaning, and multi-scale contextual information is extracted using the mid-level representation feature. By utilizing multi-scale contextual information and attention mechanisms, salient structural regions and detailed textures in the mid-level representation features are selectively enhanced to obtain mid-level fused features; The semantic modeling submodule performs semantic modeling on the mid-level fusion features to obtain high-level semantic features, including: Large convolutional kernels are used to perform convolution operations to capture global contextual information in the mid-level fusion features, so as to extract high-level representation features with semantic meaning; and compact global context vectors are extracted from the high-level representation features through global average pooling. The high-level representation features are then adaptively modulated in terms of channel dimension using the global context vectors to obtain high-level semantic features. The decoding submodule utilizes a multi-level decoding and recovery structure to perform step-by-step upsampling and feature fusion on the high-level semantic features to obtain a preliminary recovered image. The temporal integration submodule constructs a multi-time-step input sequence for the degraded image and performs temporal integration on the preliminary restored images corresponding to the multi-time-step input sequence to obtain the final restored image; The image restoration model is obtained in the following way: Obtain a training dataset, which includes degraded images and corresponding real clean images; The image restoration model is trained using the training dataset to obtain the trained image restoration model; The image restoration submodule is configured to acquire the degraded image to be processed, input the degraded image to be processed into the trained image restoration model, and output the restored image.

[0017] Thirdly, this application also discloses a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the image restoration method as described in the first aspect.

[0018] Beneficial technical effects: This invention provides an image restoration method based on the ventral visual pathway using a dual-stream spiking neural network. Starting from the information processing mechanism of the ventral visual pathway, this method constructs a hierarchical dual-stream spiking neural network framework from the lateral geniculate body to higher visual cortex. Dual-pathway separation is achieved through the center-periphery antagonism mechanism of the retina / LGN module. V1 / V2 dual-path encoders model the global low-frequency structure and local high-frequency details respectively. Cross-pathway dynamic gating mechanisms enable feature collaboration and interaction. Then, V4 multi-scale fusion and IT semantic modulation are applied. Finally, the decoder combines skip connections with stepwise upsampling to reconstruct the image. Temporal construction and multi-timestep integration strategies are employed to improve the stability of the restoration.

[0019] In the LGN preprocessing stage, this invention employs a center-periphery antagonistic DoG operator and a learnable gating fusion mechanism, which adaptively enhances local image contrast and suppresses noise in smooth regions, while simultaneously separating the visual signal into a first path and a second path. This step effectively reduces the learning burden on subsequent networks, lays the feature foundation for the separate modeling of low-frequency structures and high-frequency details, and thus improves the ability to process different frequency information in degraded images.

[0020] In the V1 / V2 dual-path encoding stage, this invention introduces dilated convolution in the first path to expand the receptive field and capture global structural information, while employing standard convolution in the second path to preserve fine texture details. A cross-gating mechanism is also designed to achieve adaptive collaborative interaction between the two paths. This step enables global structural information to effectively constrain and guide the detail restoration process, avoiding the problems of independent branch processing and insufficient collaboration in existing dual-branch networks. It significantly improves the structural consistency and detail fidelity of image restoration under complex degradation conditions.

[0021] In the V4 and IT module processing stage, this invention fuses contextual features at different scales through multi-scale convolution and introduces high-level semantic modulation using deep separable convolution with a large receptive field and global average pooling. This step enhances the model's robustness to various complex scenes such as non-uniform rain and fog, and multi-scale degradation, while improving the performance of the restored image in terms of overall brightness, contrast, and structural consistency.

[0022] In the decoding and reconstruction stage, this invention employs a step-by-step upsampling strategy combining multi-layer skip connections and introduces multi-dilation rate convolutions in each decoding layer for detail enhancement. This step effectively preserves fine-grained structural information from each level of the encoder, avoiding the problem of detail loss during downsampling in traditional encoder-decoder structures, and improving the restoration quality of high-frequency areas such as edges and textures.

[0023] In terms of temporal construction and integration, this invention designs a method for constructing multi-time-step input sequences for static images and integrates and averages the reconstruction results of each time step. This step fully leverages the computational advantages of spiking neural networks in the temporal dimension, effectively suppresses random noise and unstable responses during the reconstruction process, and significantly improves the visual consistency and overall stability of the final reconstructed image.

[0024] While maintaining the low-power advantage of spiking neural networks, this invention achieves targeted optimizations in multiple stages, including LGN preprocessing, dual-path coding, cross-gated interaction, multi-scale fusion, semantic modulation, step-by-step decoding, and temporal integration, by simulating the hierarchical processing and dual-path collaboration mechanism of the visual ventral pathway. This comprehensively improves the detail preservation capability, structural consistency, training stability, and computational efficiency in image restoration tasks, providing an engineering-feasible technical solution for the application of spiking neural networks in edge computing and embedded vision systems. Attached Figure Description

[0025] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. In all the drawings, similar elements or parts are generally identified by similar reference numerals. The elements or parts in the drawings are not necessarily drawn to scale. Obviously, the drawings described below are some embodiments of the present invention, and those skilled in the art can obtain other drawings based on these drawings without any creative effort.

[0026] Figure 1 This is a flowchart of an image restoration method according to this application.

[0027] Figure 2 This is a schematic diagram of an image restoration model architecture that can be applied to the image restoration method of this application.

[0028] Figure 3 This is a schematic diagram of an image restoration model data processing flow that can be applied to the image restoration method of this application.

[0029] Figure 4 This is a schematic diagram of the overall training process of an image restoration method in one embodiment of this application.

[0030] Figure 5 This is a schematic diagram of the module structure of an image restoration system in one embodiment of this application. Detailed Implementation

[0031] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0032] In this document, suffixes such as "module," "part," or "unit" used to denote elements are used only for the purpose of illustrative purposes and have no specific meaning in themselves. Therefore, "module," "part," or "unit" may be used interchangeably.

[0033] In this document, the terms "upper," "lower," "inner," "outer," "front," "rear," "one end," and "the other end," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing the present invention and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the present invention. Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0034] In this document, unless otherwise explicitly specified and limited, the terms "installed," "equipped with," "connected," etc., should be interpreted broadly. For example, "connection" can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection, a direct connection, or an indirect connection through an intermediate medium; it can be a connection within two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0035] In this document, "and / or" includes any and all combinations of one or more of the listed related items.

[0036] In this article, "multiple" means two or more, that is, it includes two, three, four, five, etc.

[0037] Figure 1 A flowchart of an image restoration method according to this application is shown, with reference to... Figure 1 The method specifically includes the following steps: S100: Obtain the degraded image.

[0038] S200: Input the degraded image into an image restoration model for restoration processing to obtain the final restored image, including: S201: Perform biomimetic early visual processing on the degraded image, including center-periphery antagonistic enhancement, and use learnable threshold LIF neurons for pulse coding to obtain pulse features.

[0039] For example, in step S201, the center-periphery antagonistic processing of the degraded image is implemented using the difference of Gaussian operator and encoded as impulse features using LIF neurons with learnable thresholds, including: The input degraded image is subjected to center-periphery antagonism processing to suppress smooth regions and enhance local contrast and edge information, resulting in enhanced features; The enhanced features are converted into input currents and input into LIF neurons with learnable thresholds for pulse encoding to obtain pulse features.

[0040] In some embodiments, the learnable threshold-based LIF neuron includes: For each LIF neuron, set unconstrained learnable parameters, convert the learnable parameters into positive threshold adjustment values ​​through nonlinear mapping, and add them to a preset minimum threshold constant to obtain the actual discharge threshold, ensuring that the actual discharge threshold is always positive and far from zero. The learnable parameters are initialized based on the desired initial threshold, and the difference between the desired initial threshold and the minimum threshold constant is converted into the initial value of the learnable parameters using the inverse transformation of the nonlinear mapping.

[0041] S202: Feature extraction of the pulse features is performed using a dual-path spiking neural network, including: The first path of the dual-path spiking neural network is used to extract global structural features from the pulse features, and the second path of the dual-path spiking neural network is used to extract local detail features from the pulse features. The first pathway uses a first time constant to simulate fast response characteristics and utilizes a large receptive field modeling mechanism to extract brightness-dominated global structural features; the second pathway uses a second time constant to simulate a fine integration process and utilizes a high resolution preservation mechanism to extract color-dominated local detail features. In step S202, extracting global structural features from the pulse features using the first path of the dual-path spiking neural network may specifically include: The brightness component is extracted from the input pulse features, and the multi-channel pulse features are weighted and fused to obtain the single-channel brightness pulse features. The brightness pulse features are input into a LIF neuron with a first time constant for processing. The fast response characteristics of the first pathway are simulated through fast integration and leakage mechanisms to obtain the pulse sequence of the first pathway. The first path pulse sequence is subjected to dilated convolution operation to expand the receptive field to capture a wide range of brightness change information, thereby obtaining the enhanced first path pulse features; The enhanced first pathway pulse features are aggregated in the time dimension, and the average firing rate of each neuron in all time steps is calculated to obtain the brightness-dominated global structural features.

[0042] In step S202, extracting local detail features from the pulse features using the second path of the dual-path spiking neural network may specifically include: The color information of the input pulse features is preserved, maintaining the independence of the red, green, and blue channels, to obtain the color pulse features; The color pulse features are input into LIF neurons with a second time constant for processing. The fine integration process of the second pathway is simulated through a slow integration and accumulation mechanism to obtain the pulse sequence of the second pathway. The second path pulse sequence is subjected to a standard convolution operation. A small-sized convolution kernel is used to densely sample in a local region to maintain spatial resolution. The enhanced second path pulse features are obtained through residual connections. The enhanced second pathway pulse features are aggregated in the time dimension, and the average firing rate of each neuron in all time steps is calculated to obtain color-dominated local detail features. Wherein, the second time constant is greater than the first time constant.

[0043] S203: The global structural features and the local detail features are fused to obtain mid-level fused features, which may include: By performing convolution operations with different receptive fields, global structural features and local detail features are fused to construct a mid-level representation feature with semantic meaning, and multi-scale contextual information is extracted using the mid-level representation feature. By utilizing multi-scale contextual information and attention mechanisms, salient structural regions and detailed textures in the mid-level representation features are selectively enhanced to obtain mid-level fused features; S204: Perform semantic modeling on the mid-level fusion features to obtain high-level semantic features, including: Large convolutional kernels are used to perform convolution operations to capture global contextual information in the mid-level fusion features, so as to extract high-level representation features with semantic meaning; and compact global context vectors are extracted from the high-level representation features through global average pooling. The high-level representation features are then adaptively modulated in terms of channel dimension using the global context vectors to obtain high-level semantic features. In some embodiments, S204 specifically includes: The mid-layer fusion features are processed in parallel using convolutions with receptive fields of various sizes to extract global contextual information at different receptive field scales, thereby obtaining semantic features at multiple scales. The semantic features at multiple scales are fused and residually connected with the input mid-level fused features to obtain high-level representation features with rich global semantic information. The high-level representation features are spatially compressed using global average pooling to extract a compact global context vector. The global context vector is input into a lightweight multilayer perceptron for nonlinear transformation to generate channel-dimensional modulation weights. The high-level representation features are subjected to adaptive weighted modulation on a channel-by-channel basis using the modulation weights to obtain the modulated features; The modulated features are residually concatenated with the high-level representation features to obtain the final high-level semantic features.

[0044] S205: The high-level semantic features are upsampled and fused step by step using a multi-level decoding and recovery structure to obtain a preliminary recovered image; In some embodiments, S205 specifically includes: S2051. For the first decoding stage, the high-level semantic features are used as input coarse-scale features. For subsequent decoding stages, the output features of the previous decoding stage are used as input coarse-scale features for the next decoding stage. Upsample the coarse-scale features of the current decoding stage to restore them to the target resolution, and obtain the upsampled features. S2052. Perform channel alignment on the upsampled features by adjusting the number of channels to the same dimension as the corresponding level skip connection features through convolution operations to obtain channel-aligned features. S2053. The channel-aligned features are spliced ​​and fused with the corresponding level of jump connection features to obtain stage fusion features; S2054. Perform multi-scale enhancement on the stage fusion features by convolutional operations with different dilation rates, and extract details and structural information of different receptive fields to obtain multi-scale enhanced features. S2055. Perform residual fusion on the multi-scale enhanced features and add them to the stage fusion features to obtain residual enhanced features; S2056. The residual enhancement features are refined by introducing an attention mechanism to highlight key structural regions and texture details, and the output features of the current decoding stage are obtained as the input coarse-scale features of the next decoding stage. S2057. Repeat steps S2051 to S2056 to perform stepwise upsampling and feature fusion, and pass the decoded features stepwise until the image is restored to the same resolution as the input degraded image, thus obtaining a preliminary restored image.

[0045] S206: Construct a multi-time-step input sequence for the degraded image, and perform temporal integration on the preliminary restored image corresponding to the multi-time-step input sequence to obtain the final restored image; The image restoration model is obtained in the following way: Obtain a training dataset, which includes degraded images and corresponding real clean images; The image restoration model is trained using the training dataset to obtain the trained image restoration model.

[0046] In some embodiments, S206 specifically includes: A multi-time-step input sequence of length T is constructed for the degraded image, wherein the input for each time step is obtained by applying a small multiplicative perturbation sampled from a uniform distribution to the degraded image; The input at each time step is restored using the methods in steps S201 to S205 to obtain preliminary restored images for all time steps. The preliminary restored images at all time steps are time-averaged to obtain the final restored image.

[0047] In step S206, the loss function for training the image restoration model includes Charbonnier loss, structural similarity loss, and perceptual loss, and different weights are assigned to different losses.

[0048] The perceptual loss is achieved by extracting high-level features of the final restored image and the real clean image at a specified layer through a perceptual feature extraction network to obtain the first high-level feature and the second high-level feature, and calculating the Euclidean distance between the first high-level feature and the second high-level feature as the loss term. In some embodiments, a Warmup-based dynamic weighting strategy is adopted to gradually increase the weights of structural similarity loss and perceptual loss during training.

[0049] S300: Obtain the degraded image to be processed, input the degraded image to be processed into the trained image restoration model, and output the restored image.

[0050] It should be noted that in this application, the dual-path design exhibits clear functional heterogeneity and physical interpretability: the M-path (i.e., the first path) takes the luminance component as input, employs a large receptive field dilated convolution and a small time constant, and is specifically responsible for the recovery of global structure, low-frequency contours, and illumination variations; the P-path takes the complete RGB three channels as input, employs a small receptive field standard convolution and a larger time constant, and is specifically responsible for the recovery of high-frequency textures, edge sharpness, and color fidelity. This asymmetric heterogeneous design is based on prior knowledge in visual neuroscience.

[0051] In this embodiment, a cross-gating collaborative mechanism is further employed, which has the following substantial differences: gating weights are generated using the global structural features of the M-path (i.e., the first path) to selectively enhance or suppress the local detail features of the P-path; simultaneously, gating weights are generated using the local detail features of the P-path to adaptively modulate the global structural features of the M-path. This bidirectional, selective, feature-level cross-modulation mechanism achieves a complementary fusion effect where the global structure guides the local details, and the local details enrich the global structure.

[0052] Figure 2 This diagram illustrates an image restoration model architecture of a specific embodiment that can be applied to the image restoration method of this application. Figure 3 This is a schematic diagram of the data processing flow of the image restoration model applicable to this embodiment, see reference. Figure 2 and Figure 3 This embodiment specifically includes the following steps: The image restoration problem involves mapping an observed degraded RGB image to its corresponding clean image. Let... This represents the input degraded image (R represents the matrix symbol). Represents the corresponding real clean image, where and These represent the spatial height and width of the image, respectively. The goal is to learn a parameter... The characterization of the restoring function To satisfy: in, This indicates the restored result output by the network.

[0053] To address the aforementioned issues, the network proposed in this application sequentially performs the following processing steps: 1. Early center-periphery preprocessing and pathway separation (retinal / LGN module).

[0054] 2. Dual-path feature extraction (V1 / V2).

[0055] 3. Mid-level multi-scale feature fusion (V4).

[0056] 4. High-level semantic modulation (IT).

[0057] 5. Combine step-by-step decoding and reconstruction with skip connections (Decoder).

[0058] Unlike directly predicting the target image Y, this application employs a residual learning strategy, which predicts a residual correction term superimposed on the input image. This strategy helps stabilize the optimization process and effectively preserves low-frequency structural information in the input. Formally, the entire network can be represented as: in , , , and These represent the retina / LGN module, V1 / V2 dual-path encoder, V4 fusion module, IT module, and decoder, respectively. In the above formula, the residual branch is mainly used to learn and compensate for degradation factors in the input image (such as blur, noise, and contrast loss), while the identity mapping branch is used to preserve the overall structural information of the input image.

[0059] In some embodiments, the retina / LGN module is designed to perform a biomimetic early visual processing procedure: enhancing local contrast through a center-periphery antagonistic mechanism and separating the visual signal into a magnocellular (M) pathway (first pathway) and a parvocellular (P) pathway (second pathway). Given an input image X in The retina / LGN module outputs three types of features: first pathway features. Second pathway characteristics And skip connection features for subsequent decoding stages .

[0060] In some embodiments, to characterize the center-periphery receptive field structure, this application employs the Difference-of-Gaussian (DoG) operator. For a scalar image I(x,y), its DoG response is defined as: in, and Let represent the amplitude coefficients of the central and surrounding components, respectively, and σ be the standard deviation of the Gaussian kernel. This ensures that the surrounding area of ​​inhibition has a larger range of sensation. A two-dimensional Gaussian kernel representing the center amplitude coefficient. A two-dimensional Gaussian kernel representing the amplitude coefficients of the surrounding components. x This represents the horizontal coordinate of a pixel relative to the center of the Gaussian kernel.y This represents the vertical coordinate of a pixel relative to the center of the Gaussian kernel. The two-dimensional Gaussian kernel is defined as: DoG (DoG Enhancement) is essentially a bandpass filter that suppresses smooth regions and highlights local contrast and edge information, thus facilitating the restoration of detailed structures. To avoid over-enhancement and achieve data-driven adaptive adjustment, this application performs a weighted fusion of the original features and DoG enhancement features. The fused features are as follows: in, pass Learn and gain This represents the Sigmoid function. These are trainable scalar parameters, initialized using a logit transform. This learnable gating mechanism adaptively controls the contrast enhancement intensity, thereby improving the model's robustness under different levels of degradation. These are the original features.

[0061] In some embodiments, convolution operations combined with threshold batch normalization (tdBN) are employed in both the M and P paths to stabilize the amplitude of the pulse characteristics under different discharge threshold conditions. Given an input tensor... The output of the convolutional layer is: in and These represent the convolution weights and bias terms, respectively. Subsequently, BN is applied to... Normalize: in and These represent the batch mean and variance, respectively. To prevent small constants from being numerically unstable, and These are learnable affine parameters. Threshold-dependent scaling factor. Defined as: in, This represents the firing threshold of the spiking neuron. Because... This will affect the discharge rate and activation amplitude. It can compensate for amplitude differences caused by threshold changes, thereby improving the stability of the training process and enhancing the consistency of cross-layer signals.

[0062] In some embodiments, the pulsatile neurodynamics are modeled by Leaky Integrate-and-Fire (LIF) neurons. Compared to LIF neurons with a fixed threshold, this application introduces a learnable firing threshold to enhance the adaptability between different network layers and functional modules.

[0063] Membrane potential dynamics: Let Represents discrete time steps The membrane potential under the current, This represents the synaptic input at the corresponding time step. The dynamic equation of a LIF neuron is defined as: in Controlling the leakage and integral intensity of the membrane potential, and These are learnable parameters. To ensure... The values ​​are within a reasonable range, and this application parameterizes them as follows: in For learnable scalar parameters, This represents the Sigmoid function. This parameterization method is used in constraints. While maintaining numerical stability, the model is still allowed to adaptively learn time integral properties in a data-driven manner.

[0064] Pulse firing mechanism based on alternative gradients: When membrane potential Exceeding the discharge threshold At this time, the neuron generates a pulse output. : in Controlling the slope of the substitution function, Let represent a differentiable alternative form of the Heaviside step function. This paper uses the arctangent function as the alternative gradient: The introduction of alternative gradients makes the discrete impulse process differentiable during the backpropagation phase, thereby supporting end-to-end training of the network.

[0065] Parameterization of learnable threshold: To ensure the discharge threshold Always positive and far from zero, this application adopts the following parameterization form: in The preset minimum threshold constant, These are unconstrained, learnable parameters. The threshold is initialized to the desired value. This application employs the inverse softplus transform: in is the shape control constant. By introducing a learnable firing threshold, neurons at different levels can adaptively adjust firing sparsity and feature selectivity, which is particularly crucial when the difficulty of image restoration varies with scale and modules.

[0066] In some embodiments, refer to Figure 2 and Figure 3 The first pathway primarily focuses on luminance information and low spatial frequency cues, which are particularly important for global structure modeling and perception of illumination changes. First, the RGB image is converted into luminance components: in Represents the input image X in The three color channels are then used. Subsequently, a Gaussian blur (to suppress high-frequency noise) is applied to the luminance image, followed by DoG enhancement. in, This is the blurred brightness feature map of the M-path obtained after applying Haus blur to the brightness component. To The output feature map of the M-path after DoG enhancement. For the luminance component, This is a Gaussian blur operation. To simulate faster response characteristics, this application employs a smaller effective membrane potential time constant for neurons in the first pathway. Smaller It enables faster time integration and enhances sensitivity to rapid changes in brightness.

[0067] DoG is then performed in the first path, and its output characteristic is represented as F. out : Where, mix is ​​an adaptive mixing coefficient with a value of (0, 1), used to control the fusion ratio of the original blurred features and the DoG enhanced features; Fin is the input feature for the DoG operation, i.e., the brightness feature of the first path obtained after Gaussian blurring; DoG(·) represents the Gaussian difference enhancement operation; Fout is the feature output of the first path after DoG enhancement and adaptive mixing.

[0068] After that, F outThe input is fed into a threshold-correlated batch normalization (SCB) spiking convolution module for processing. The SCB module contains convolutional layers, batch normalization layers, and Leaky Integrate-and-Fire (LIF) spiking neuron activation functions, which perform sparse encoding and nonlinear transformation of the input features in the spatiotemporal domain to generate spiking feature responses with event-driven characteristics.

[0069] Subsequently, the output of the aforementioned SCB module is fed into a convolutional module CB that incorporates threshold-related batch normalization. Its main function is to stabilize the gradient flow during network training and to further refine and integrate the spatial information of the impulse features output by the SCB module, thereby obtaining the deep feature representation of the backbone path, denoted as F. main .

[0070] Meanwhile, to effectively preserve and utilize the global brightness structure information in the original image, this invention introduces a parallel brightness-guided residual path. In this path, the brightness component... Y rgb It is independently input into another dedicated CB module. This module performs channel-dimensional adaptation and lightweight feature transformation on the luminance component, generating a core feature F. main Luminosity guiding features F that are perfectly aligned in both spatial and channel dimensions lum .

[0071] Subsequently, this invention will use the depth features F of the main path main Features F of the brightness guiding path lum Element-wise addition is performed to complete the residual connection fusion, and the result F is... M,refined This refers to the refined output features of the M-path. Through this residual connection mechanism, the network can explicitly inject low-frequency structural priors from the original brightness information while performing complex degradation modeling, effectively preventing image distortion or artifacts caused by overprocessing, and significantly improving global illumination consistency and local detail fidelity in image restoration tasks such as dehazing and deraining.

[0072] In some embodiments, the second path emphasizes high spatial frequency details and color information. This path first applies a DoG operation to the RGB input: After that, F p After processing by the Spiking Conv Block (SCB), visual features in the form of pulses are extracted and encoded stepwise.

[0073] Subsequently, the output of the SCB module is fed into a CB module for smoothing and channel dimension adaptation, and further integrates spatial context information to obtain the deep feature representation of the backbone path, denoted as F. P,main .

[0074] Meanwhile, to effectively preserve and utilize the global structure and low-frequency content in the original RGB image, this invention introduces a parallel residual guidance path. In this path, the original RGB input image is directly input into another dedicated CB module. This module performs lightweight linear transformations and feature alignment operations on the original image, generating a feature vector F. P,main The residual guided feature that is completely consistent in spatial resolution and channel dimension is denoted as F. P,res .

[0075] Subsequently, this invention will use the depth features F of the main path P,main The characteristic F of the residual guiding path P,res Element-wise addition is performed to complete the residual connection fusion. The result F P,refined This refers to the refined output features of the P-path. Through this residual connection mechanism, the network enhances the representation of high-frequency details and color contrast while explicitly injecting low-frequency structural priors from the original image. This effectively alleviates the semantic drift or loss of details that may be caused by multi-level impulse processing, and significantly improves the perceptual quality and robustness of the model in texture-sensitive tasks such as image super-resolution, denoising, or fine-grained classification.

[0076] To simulate a slower but more refined integration process, the second path uses a larger time constant. Larger The ability to accumulate finer-grained evidence over time helps in the reconstruction of texture details.

[0077] In some embodiments, to achieve complementary coordination between the two paths, this application introduces a cross-gate mechanism: in This represents the Sigmoid function. and This is the gate mask for the corresponding path. The membrane potential after gate modulation is expressed as: in This indicates element-wise multiplication. , The membrane potentials modulated by the M and P pathways are respectively. The cross-gating mechanism adaptively modulates the other pathway by utilizing information from one pathway, effectively balancing the global structure and local details, thereby improving the robustness of image restoration under complex degradation conditions.

[0078] In some embodiments, the V1 / V2 encoder primarily performs pulse processing and cross-gating interaction. The V1 / V2 encoder uses two parallel pulse processing paths to model the features of the first and second paths respectively, and achieves collaborative interaction between the two through a cross-gating mechanism.

[0079] In one specific implementation, the first path (M-path) of the V1 / V2 encoder aims to construct a feature extraction structure with a large receptive field to effectively capture global degradation patterns and low-frequency structural information in images. Specifically, this path significantly expands the effective receptive field of the network without increasing the number of parameters or computational complexity by introducing dilated convolution operations, thereby enhancing the model's ability to model long-range dependencies. As shown in the following equation, the first two feature extraction modules of the first path employ convolutional structures with different dilation rates: in, k This indicates the kernel size (fixed at 3×3). d The expansion rate is used for the first layer. d =1 (i.e., standard convolution), the second layer is raised to... d =2, expanding the receptive field of a single convolution operation from 3×3 to 7×7, thus achieving the integration of information from a larger neighborhood without introducing pooling or downsampling. Subsequently, residual aggregation is performed on the features: This represents the input features of the encoding module in the first path, used as a residual term; when the number of input and output channels is inconsistent, channel alignment is performed through 1×1 convolution projection. This represents the fusion characteristics of the first path after gated residual aggregation; This represents the features extracted by the dilated convolution in the second layer of the M-path (d=2); ConvTDBN(·) represents the convolution operation combined with threshold batch normalization. This indicates multiplication by channel, i.e., by Channel-wise gated weights, with values ​​between 0 and 1, generated by the gated subnetwork, are used to adjust the residual terms. Adaptive scaling is used to achieve gated residual connections.

[0080] Dilated convolutions help enhance global context modeling capabilities, while residual connections effectively preserve original information and promote gradient propagation.

[0081] In obtaining the fusion characteristics H of the M channel M Then, a channel attention mechanism is applied to enhance key channels and suppress redundant information to obtain recalibrated features. This operation enables the network to dynamically highlight the semantic channels that are most discriminative for the recovery process, such as defogging and deraining, based on task requirements, thereby improving the focus and effectiveness of feature representation.

[0082] In one specific implementation, the second path (P-path) of the V1 / V2 encoder aims to simulate the high spatial resolution, color sensitivity, and detail-preserving feature processing mechanism dominated by the P-path of the visual cortex. Its core objective is to accurately model high-frequency texture, edge structure, and color fidelity information in the image, providing fine local representation support for image restoration tasks. To this end, this path adopts a standard convolutional structure (i.e., dilation rate d=1) to avoid detail blurring caused by excessive expansion of the receptive field and ensure spatial localization accuracy.

[0083] As shown in the following equation, the P-path contains two cascaded ConvTDBNLIF modules: Feature aggregation is performed using a residual method, resulting in the following fused features for the P-channel: The second pathway primarily emphasizes high-frequency texture information and color fidelity, which are crucial for detail reconstruction in image restoration tasks.

[0084] In obtaining the fusion characteristics of the P pathway H P Subsequently, this invention also applies a channel attention mechanism to adaptively enhance key channels related to high-frequency details and color fidelity, while suppressing redundant channels with low response or noise dominance, thereby obtaining finer features after recalibration. This mechanism extracts channel statistics through global average pooling, generates channel weights via a lightweight bottleneck network, and then multiplies them channel by channel with the original features to achieve recalibration. Since the P-path focuses on local structure and color details, this attention operation can significantly enhance key cues such as edge sharpness, texture continuity, and color difference consistency, enabling the model to more accurately reconstruct microstructures and natural color transitions in image restoration, thereby improving the quality and stability of detail reconstruction.

[0085] In this embodiment, step S201 includes: An adaptive weight is generated using a gating mechanism, and the enhanced features are then weighted and fused with the degraded image using the adaptive weight to obtain adaptive contrast enhancement features. The adaptive contrast enhancement feature is used as an input current to the spiking neuron model for pulse coding.

[0086] Specifically, in this embodiment, the workflow of the M-path (equivalent to the first path) and the P-path (equivalent to the second path) is as follows: The gating controller generates corresponding gating weights for the P path based on the global structural features of the M path and the local structural features of the P path (which are used to define how much information the P path needs to borrow from the M path next), and generates corresponding gating weights for the M path (which are used to define how much information the M path needs to borrow from the P path next).

[0087] Specifically, the gating generator jointly receives the global features of the M path and the local features of the P path, and generates gating weights (values ​​from 0 to 1) in the P→M and M→P directions respectively, which are used to control the intensity of information borrowed between the paths; during back propagation, the loss of each path only considers itself, while during forward propagation, the other path is also taken into account.

[0088] In other words, the gating generator simultaneously receives the global structural features of the M-path and the local detailed features of the P-path, and after joint evaluation, generates: weights for modulating the P→M information flow and weights for modulating the M→P information flow.

[0089] Furthermore, different forward and reverse propagation paths are set up as follows: in the reverse propagation process, the loss function design of each path only considers its own loss, while in the forward propagation process, each path needs to take into account the situation of the other path.

[0090] For example, the gating generator can dynamically calculate a set of weights between 0 and 1 based on the current output states of the M and P pathways through a trainable nonlinear subnetwork. These weights are used in the iterative loop to modulate the information exchange intensity between the pathways and the final output mixing ratio. The parameters of this subnetwork are jointly optimized with the main network to achieve adaptive gating scheduling of the task.

[0091] For example, detach can be used to configure the forward and backward propagation mechanisms for the M and P paths. PyTorch's computation graph mechanism works as follows: During forward computation, PyTorch automatically records each step of the operation, constructing a computation graph that strings together all tensors and their dependencies into a directed acyclic graph. During backward propagation, PyTorch uses this graph to propagate the gradient back from the loss to all tensors that require grad=True, following the chain rule.

[0092] Specifically, this approach of forward expansion (i.e., incorporating data from another channel) and reverse suppression can be implemented through the following pattern: The weighted fusion model of the adaptive contrast enhancement features of the first path is as follows: m enhanced = m membrane *(1 – λ1*gate_s) +p membrane.detach()* (λ2*gate_s) m enhanced represents the adaptive contrast enhancement feature after the first path enhancement, m membrane represents the structural feature identified by the first path at the previous time step, detach() represents the blocking function (or gradient blocking operation, the purpose of which is to block the back propagation path of the gradient), gate_s represents the corresponding gate weights, p membrane represents the structural feature identified by the second path at the previous time step, and λ1 and λ2 represent the set weights.

[0093] Understandably, the adaptive contrast enhancement feature represents the enhanced state of the M-path. This state includes the currently identified structural features, as well as the integration of structural features identified in previous stages. More specifically, it refers to the differentiable numerical representation generated by the M-path at a specific computation node (usually its output layer or a pre-defined hidden layer) within its current iteration step after performing forward computation on all received input information (including raw input data and optionally historical outputs from the P-path) on a specific input node.

[0094] m_membrane represents the state from the M pathway at the previous time step; p membrane represents the state from the P pathway at the previous time step; λ1 and λ2 are coefficients that can be defined by the engineer (both can be 0.2).

[0095] Understandably, the enhancement logic for the P pathway is the same as the above scheme, and will not be repeated here.

[0096] Therefore, this embodiment employs a constrained dual-path optimization image reconstruction mechanism using M and P pathways for complex images (especially noisy photos in foggy scenes). This mechanism allows the M and P pathways to learn from the experience of the other (e.g., through membrane potential updates) during each feature acquisition, while simultaneously imposing constraints to ensure that the parameter updates of M and P do not interfere with each other, and the gradient of each pathway comes only from its own loss. In this way, the two pathways can improve themselves by leveraging the advantages of the other, while being less likely to be penalized for the errors of the other.

[0097] From another perspective, by imposing certain reverse restrictions on the cross-optimization of the two pathways, the computational pressure of the cross-optimization of the two pathways can be reduced to a certain extent. This is especially beneficial when dealing with complex images, as it helps to improve the training efficiency of the model.

[0098] Furthermore, in some embodiments, the forward and backward propagation mechanisms can be dynamically adjusted according to the training / iteration phase of the model: Obtain the confidence level of one of the channels at the current moment; When the confidence level is greater than the set score, the gradient blocking restriction on the pathway is lifted.

[0099] For example, if the current confidence level of the M path is very high, the restriction on the M path can be lifted in the back propagation path of the P path, that is, the detach() blocking operation can be canceled.

[0100] Furthermore, in some embodiments, the current iteration step will also be identified, specifically including the steps: When the number of iterations is less than the set number, a signal to release the constraint is generated. In other words, in the early stages of model development, even if one channel has a high confidence level, it still needs to be constrained on the backpropagation path. This is to avoid creating unreliable artifacts due to model instability during early training, which could negatively impact the model's training. For example, in the early stages, the accuracy of the identified content may fluctuate (i.e., be unstable), and releasing the constraint too early could cause this fluctuation error to spread.

[0101] In this embodiment, the gradient blocking mechanism in the backpropagation path is enabled or disabled restrictively based on the stage of the model and the actual optimization state of the pathway (such as the confidence level). This allows for a proper balance between improving model efficiency and enhancing the cross-optimization capability of the dual pathways. Specifically, it avoids overly restricting backpropagation learning due to gradient blocking while also preventing increased backpropagation error caused by unstable information.

[0102] In some embodiments, to achieve complementary coordination between the two paths, a cross-gate mechanism is also introduced in the V1 / V2 encoder: in This represents the Sigmoid function. , These represent the input characteristics of the M and P paths before gated modulation, respectively. , These are the gate weights for the M and P paths, respectively, with values ​​between 0 and 1, adaptively controlling the information borrowing intensity between the paths; the membrane potential after gate modulation is expressed as: This represents the input characteristics of the M-path before gated modulation. This represents the input characteristics of the P-path before gating modulation. This represents the membrane potential after gating modulation of the first pathway (M pathway). This represents the membrane potential after gating modulation of the second pathway (P pathway). ⊙ indicates element-wise multiplication. The constants 0.6 and 0.4 represent the baseline retention ratio and the gating adjustment ratio, respectively.

[0103] By utilizing a cross-gating mechanism, information from one pathway is used to adaptively modulate another, effectively balancing global structure and local details, thereby improving the robustness of image restoration under complex degradation conditions. In some embodiments, the V4 module aims to integrate features from both pathways to generate a mid-level fusion representation. First, the features from the first and second pathways are concatenated: Then, the basic representation is extracted: To capture complementary contextual information, multi-scale branching is introduced. and convolution): And these are spliced ​​together to obtain multi-scale features: The output of the V4 module is obtained through residual refinement and attention mechanisms: CBA refers to applying both channel attention and spatial attention simultaneously to highlight more discriminative features. Multi-scale fusion can effectively improve the model's robustness to different degradation scales, while residual connectivity and attention mechanisms help preserve significant structural information and enhance the discriminative ability of key channels.

[0104] In some embodiments, the IT module is designed to provide high-level semantic cues and global contextual information for image restoration. This application employs depthwise separable convolutions with multiple large kernels to model a large receptive field context: Multi-scale semantic features are fused using residual paths: To further incorporate global semantic information, this application extracts global context through Global Average Pooling (GAP) and modulates the features using a lightweight multilayer perceptron (MLP): Among them, F IT The input feature map for the IT module is the original feature map to be subjected to global modulation, with dimensions C×H×W (C is the number of channels, and H and W are the height and width of the feature map).

[0105] GAP (Global Average Pooling): This method applies a global average pooling algorithm to the input feature map F. IT The averaging operation across the spatial dimension (H×W) compresses each channel into a single value, which is used to extract global contextual statistics.

[0106] C global The global context vector (channel descriptor) obtained after global average pooling has a dimension of C×1×1, and each element represents the global response of the corresponding channel on the entire feature map.

[0107] MLP (Multilayer Perceptron): A lightweight multilayer perceptron subnetwork that operates based on the global context C. global Learn to generate the modulation coefficients for each channel.

[0108] A global The global modulation weights (channel attention vectors) output by the MLP have a dimension of C×1×1 and are used to recalibrate each channel.

[0109] ⊙: Channel-wise (element-wise) multiplication operation (Hadamard product), that is, A global Press channel and F IT Multiply.

[0110] H IT The features output after global semantic modulation are used as the final output of the IT module.

[0111] The large convolutional kernels in the V4 module are used to capture a wider range of contextual and semantic information, while the global modulation mechanism helps to enhance the global consistency of the restoration results in terms of overall brightness, structural consistency, etc.

[0112] In some embodiments, the decoder combines high-level semantic features with multi-level skip features through progressive upsampling to gradually reconstruct the full-resolution image. In a certain decoding stage, let... Indicates coarse-scale features, This represents the skip connection features at the corresponding level. First, [the following is a list of steps / methods]... Upsampled to target resolution: Subsequently, through Convolution for channel alignment: After channel alignment is completed, the two types of features are spliced ​​and fused: To further refine structural and texture information at various scales, this application introduces two convolution operations with different dilation rates: And it is enhanced through fusion using residual methods: Finally, the features are refined and attention-weighted: By combining a stepwise decoding strategy with skip connections, the model can effectively preserve fine-grained structural information; at the same time, the multi-scale enhancement mechanism further improves the quality of detail restoration in different textures and edge regions.

[0113] In some embodiments, to fully utilize the temporal dynamic characteristics of spiking neural networks in image restoration tasks, this application constructs a static input image as a sequence of lengths... The time series. Given an input image. The time series is defined as follows: in express t The time-time perturbation term. This random temporal perturbation, as a time-dimensional data augmentation method, introduces temporal diversity, avoiding the degradation caused by completely identical inputs at each time step. This helps improve temporal credit assignment and enhance the model's generalization ability. At each time step... The residual restoration results corresponding to the network prediction The final output is obtained by averaging the results over all time steps: Time averaging can effectively reduce the variance between prediction results at different time steps, thereby improving the stability of the restoration results. Its effect is similar to integrated smoothing in the time dimension.

[0114] In some embodiments, the loss function of this application is optimized by a weighted combination of pixel fidelity, structural similarity, and perceptual consistency: in and Used to balance the relative contributions (i.e., weights) of different loss terms.

[0115] Charbonnier's loss is A differentiable and robust variant of the loss is defined as: in This indicates the number of pixels (and channels) involved in the loss calculation. and Representing the restored images The first image of the real image Y One element, For a very small constant (such as Compared to The Charbonnier loss is insensitive to outliers and provides more stable gradients in image restoration tasks.

[0116] Structural similarity index (SSIM) is used to measure the reconstructed image. With real images The degree of similarity at the structural level is defined as follows: in Represents the local mean. Represents local variance. Represents local covariance. and To ensure numerical stability, a constant is used. Based on this, this application defines the SSIM loss as: This loss term can constrain the model to maintain the structural consistency of the image during the restoration process, complementing the pixel-level loss.

[0117] Perceptual loss measures the quality of restoration by comparing high-level features extracted by a pre-trained network (such as VGG16). Indicates the network in the 1st... The feature map extracted by the layer has a dimension of Then the perceived loss is defined as: This loss term helps encourage the generation of texture details with perceived plausibility and effectively suppresses oversmoothing.

[0118] To improve stability during model training and avoid excessive interference from high-level semantic constraints on the low-level reconstruction process, this invention employs a phased weight scheduling strategy for the perceptual loss term. Specifically, in the initial training phase, the weight of the perceptual loss is appropriately reduced; as training progresses, it is gradually restored to the preset weight in subsequent phases, thereby enhancing the model's ability to model high-level semantic information. In contrast, the structural similarity loss and reconstruction loss maintain fixed weights during training to ensure the stability of basic structure reconstruction. Therefore, the total loss function can be expressed as: in, Represents the structural similarity loss weights. This represents the perceptual loss weight function that changes with the training phase.

[0119] The overall training process is as follows Figure 4 As shown, this application optimizes model parameters by backpropagating spiking neurons using surrogate gradients. To improve training stability, gradient pruning is employed during training, and cosine annealing is used to schedule the learning rate. Specifically: first, the model parameters θ and the cosine annealing learning rate scheduler are initialized; in each training epoch, the current epoch loss is zeroed and the sample order is randomly shuffled; for each mini-batch of samples, a time-series input of length T is constructed (time dimension expansion is achieved by adding small noise perturbations), and the membrane potential and pulse state of all spiking neurons are reset; then, forward propagation is performed sequentially: the retinal / LGN module extracts M / P dual-path degradation features, V1 / V2 performs dual-path encoding and cross-gating, V4 fuses multi-scale context, IT models global semantics, the decoder upsamples stepwise and fuses the LGN / V1 / V4 three-level skip connections, and finally outputs the T-step time-averaged reconstructed image. Next, the three types of loss are calculated: Charbonnier pixel loss. SSIM structural similarity loss With perceived loss The total loss L is obtained by weighted summation; the gradient is calculated using backpropagation via the surrogate gradient method. The gradients are pruned (maximum norm 1.0) to stabilize training, and then the parameters are updated using a cosine annealing strategy. At the end of each round, the loss of the current round is accumulated, and the performance is evaluated on the validation set. If the optimal result is achieved, the model is saved. The entire process uses time series modeling and threshold-aware normalization (tdBN) to synergistically improve the training stability of SNN and the performance of dehazing / deraining tasks.

[0120] From another perspective, the image restoration method provided by this invention actually adopts the following architecture: (1) Temporal construction and integration mechanism. The static degraded image is constructed into a multi-time-step input sequence at the input end to make full use of the inherent temporal dynamic characteristics of the spiking neural network. The recovery results of each time step are integrated by time averaging and output, which effectively suppresses random noise and unstable response, and improves the stability and robustness of the recovery results.

[0121] (2) Early processing and pathway separation in biomimetic vision. The front end uses a retinal / lateral geniculate nucleus (LGN) module to enhance the local contrast of the input image through a center-periphery antagonism mechanism (based on the difference of Gaussians operator) and separates the visual signal into two processing pathways with different functional focuses: the first pathway is biased towards modeling low-frequency structural information dominated by brightness, using a small time constant and a large receptive field dilated convolution; the second pathway is biased towards modeling color and high-frequency texture details, using a larger time constant and standard convolution to maintain spatial resolution.

[0122] (3) Dual-path coding and cross-gating collaboration. In the V1 / V2 dual-path coding stage, adaptive information collaboration between the two paths is achieved through a cross-gating mechanism: the features of one path are used to generate gating weights, and the other path is selectively modulated. As a result, global structural information can constrain and guide the recovery of local details, and local detail information can also supplement and enrich the global structural expression, realizing the complementary fusion of the two types of information.

[0123] (4) Multi-scale fusion and high-level semantic modulation. In the high-level processing part, the V4 module fuses context features of different scales through multi-scale branch convolution to construct a mid-level representation with semantic meaning; the IT module further introduces large receptive field depth separable convolution and global average pooling to realize the modeling of high-level semantic information and adaptive modulation of channel dimensions.

[0124] (5) Step-by-step decoding and residual learning. In the decoding stage, multi-level skip connections are combined to perform step-by-step upsampling. Multi-dilation rate convolutions are introduced in each decoding layer to enhance details and finally restore the full-resolution image. The network as a whole adopts a residual learning strategy, that is, predicting the residual correction term of the input image instead of directly predicting the target image, which is beneficial to preserve low-frequency structural information and stabilize the optimization process.

[0125] (6) Multi-loss joint optimization and training strategy. During the training phase, a weighted combination of Charbonnier loss (pixel-level fidelity), SSIM loss (structural constraints), and perceptual loss (semantic constraints, based on pre-trained VGG16 network) is adopted, along with gradient pruning, cosine annealing learning rate scheduling, and perceptual loss progressive enhancement strategies to improve the stability and convergence quality of training.

[0126] Specifically, to reduce the risk of missing information during the reconstruction process, the following mechanisms are used to supplement and integrate information: The first path (M path) converts the input image into a brightness component, combines a large receptive field dilated convolution with a fast time constant response, and extracts compensation information such as global illumination distribution, large-scale structural contours, and brightness changes to restore the overall structural continuity lost due to degradation.

[0127] The second path (P path): By preserving the complete RGB three-channel information, combined with standard convolution with a small receptive field and a slower integration process, it extracts compensation information such as local texture patterns, edge details and color fidelity, which is used to restore fine visual content that has been blurred or attenuated due to degradation.

[0128] Cross-gating mechanism: Based on the characteristic response intensity of one path, gating weights are adaptively generated to modulate another path, enabling the two paths to selectively introduce complementary information that the other is good at extracting, thereby achieving cross-compensation for missing information across paths.

[0129] In some embodiments, S206 specifically includes: A multi-time-step input sequence of length T is constructed for the degraded image, wherein the input for each time step is obtained by applying a small multiplicative perturbation sampled from a uniform distribution to the degraded image; The input at each time step is restored using the methods in steps S201 to S205 to obtain preliminary restored images for all time steps. The preliminary restored images at all time steps are time-averaged to obtain the final restored image.

[0130] In some embodiments, in step S206, the loss function for training the image restoration model includes Charbonnier loss, structural similarity loss, and perceptual loss, and different weights are assigned to different losses. The perceptual loss is achieved by extracting high-level features of the final restored image and the real clean image at a specified layer through a perceptual feature extraction network to obtain the first high-level feature and the second high-level feature, and calculating the Euclidean distance between the first high-level feature and the second high-level feature as the loss term. In some embodiments, a Warmup-based dynamic weighting strategy is adopted to gradually increase the weights of structural similarity loss and perceptual loss during training.

[0131] In some embodiments, the learnable threshold-based LIF neuron includes: For each LIF neuron, set unconstrained learnable parameters, convert the learnable parameters into positive threshold adjustment values ​​through nonlinear mapping, and add them to a preset minimum threshold constant to obtain the actual discharge threshold, ensuring that the actual discharge threshold is always positive and far from zero. The learnable parameters are initialized based on the desired initial threshold, and the difference between the desired initial threshold and the minimum threshold constant is converted into the initial value of the learnable parameters using the inverse transformation of the nonlinear mapping.

[0132] Further reference Figure 5 As an implementation of the above-described method, this application provides an embodiment of an image restoration system, which is similar to... Figure 1 Corresponding to the method embodiments shown, the system can be specifically applied to various electronic devices.

[0133] refer to Figure 5 An image restoration system, comprising: Degraded image acquisition module 300, configured to acquire degraded images; Image restoration model building module 310 includes: The pulse feature extraction submodule 3101 is configured to perform biomimetic early visual processing on the degraded image, including center-periphery antagonistic enhancement, and pulse coding using LIF neurons with learnable thresholds to obtain pulse features; The pathway feature extraction submodule 3102 is configured to extract features from the pulse features using a dual-path spiking neural network, including: The first path of the dual-path spiking neural network is used to extract global structural features from the pulse features, and the second path of the dual-path spiking neural network is used to extract local detail features from the pulse features. The first pathway uses a first time constant to simulate fast response characteristics and utilizes a large receptive field modeling mechanism to extract brightness-dominated global structural features; the second pathway uses a second time constant to simulate a fine integration process and utilizes a high resolution preservation mechanism to extract color-dominated local detail features. The fusion processing submodule 3103 performs fusion processing on the global structural features and the local detail features to obtain mid-level fused features, including: By performing convolution operations with different receptive fields, global structural features and local detail features are fused to construct a mid-level representation feature with semantic meaning, and multi-scale contextual information is extracted using the mid-level representation feature. By utilizing multi-scale contextual information and attention mechanisms, salient structural regions and detailed textures in the mid-level representation features are selectively enhanced to obtain mid-level fused features; Semantic modeling submodule 3104 performs semantic modeling on the mid-level fusion features to obtain high-level semantic features, including: Large convolutional kernels are used to perform convolution operations to capture global contextual information in the mid-level fusion features, so as to extract high-level representation features with semantic meaning; and compact global context vectors are extracted from the high-level representation features through global average pooling. The high-level representation features are then adaptively modulated in terms of channel dimension using the global context vectors to obtain high-level semantic features. The decoding submodule 3105 uses a multi-level decoding recovery structure to perform step-by-step upsampling and feature fusion on the high-level semantic features to obtain a preliminary recovered image. The temporal integration submodule 3106 constructs a multi-time-step input sequence for the degraded image and performs temporal integration on the preliminary restored image corresponding to the multi-time-step input sequence to obtain the final restored image; The image restoration model is obtained in the following way: Obtain a training dataset, which includes degraded images and corresponding real clean images; The image restoration model is trained using the training dataset to obtain the trained image restoration model; Image restoration module 320 is configured to acquire a degraded image to be processed, input the degraded image to be processed into the trained image restoration model, and output a restored image.

[0134] In another aspect, this application also provides a computer-readable storage medium, which may be included in the electronic device described in the above embodiments; or it may exist independently and not assembled into the electronic device. The aforementioned computer-readable storage medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to perform the following... Figure 1 The method shown.

[0135] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0136] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a computer terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0137] The embodiments of the present invention have been described above with reference to the accompanying drawings. However, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of the present invention without departing from the spirit and scope of the claims. All of these forms are within the protection scope of the present invention.

Claims

1. An image restoration method, characterized in that, Includes the following steps: S100: Acquire the degraded image; S200: Input the degraded image into an image restoration model for restoration processing to obtain the final restored image, including: S201: Perform biomimetic early visual processing on the degraded image, including center-periphery antagonistic enhancement, and use learnable threshold LIF neurons for pulse coding to obtain pulse features; S202: Feature extraction of the pulse features is performed using a dual-path spiking neural network, including: The first path of the dual-path spiking neural network is used to extract global structural features from the pulse features, and the second path of the dual-path spiking neural network is used to extract local detail features from the pulse features. The first pathway uses a first time constant to simulate fast response characteristics and utilizes a large receptive field modeling mechanism to extract brightness-dominated global structural features; the second pathway uses a second time constant to simulate a fine integration process and utilizes a high resolution preservation mechanism to extract color-dominated local detail features. S203: The global structural features and the local detail features are fused to obtain the mid-level fused features; S204: Perform semantic modeling on the mid-level fusion features to obtain high-level semantic features; S205: The high-level semantic features are upsampled and fused step by step using a multi-level decoding and recovery structure to obtain a preliminary recovered image; S206: Construct a multi-time-step input sequence for the degraded image, and perform temporal integration on the preliminary restored image corresponding to the multi-time-step input sequence to obtain the final restored image; S300: Obtain the degraded image to be processed, input the degraded image to be processed into the trained image restoration model, and output the restored image.

2. The image restoration method according to claim 1, characterized in that: In step S201, the center-periphery antagonistic processing of the degraded image is implemented using the difference of Gaussian operator and encoded as pulse features using LIF neurons with learnable thresholds, including: The input degraded image is subjected to center-periphery antagonism processing to suppress smooth regions and enhance local contrast and edge information, resulting in enhanced features; The enhanced features are converted into input currents and input into LIF neurons with learnable thresholds for pulse encoding to obtain pulse features.

3. The image restoration method according to claim 2, characterized in that: Step S201 also includes: An adaptive weight is generated using a gating mechanism, and the enhanced features are then weighted and fused with the degraded image using the adaptive weight to obtain adaptive contrast enhancement features. The adaptive contrast enhancement feature is used as an input current to the spiking neuron model for pulse coding.

4. The image restoration method according to claim 3, characterized in that: The weighted fusion model for the adaptive contrast enhancement features of the first path is as follows: m enhanced = m membrane *(1 – λ1*gate_s) +p membrane.detach()* (λ2*gate_s); m enhanced represents the adaptive contrast enhancement feature after the first path enhancement, m membrane represents the structural feature identified by the first path at the previous time step, gate_s represents the corresponding gate weights, p membrane represents the structural feature identified by the second path at the previous time step, detach() represents the gradient blocking operation, and λ1 and λ2 represent the set weights.

5. The image restoration method according to claim 1, characterized in that: In step S202, extracting global structural features from the pulse features using the first path of the dual-path spiking neural network specifically includes: The brightness component is extracted from the input pulse features, and the multi-channel pulse features are weighted and fused to obtain the single-channel brightness pulse features. The brightness pulse features are input into a LIF neuron with a first time constant for processing. The fast response characteristics of the first pathway are simulated through fast integration and leakage mechanisms to obtain the pulse sequence of the first pathway. The first path pulse sequence is subjected to dilated convolution operation to expand the receptive field to capture a wide range of brightness change information, thereby obtaining the enhanced first path pulse features; The enhanced first pathway pulse features are aggregated in the time dimension, and the average firing rate of each neuron in all time steps is calculated to obtain the brightness-dominated global structural features.

6. The image restoration method according to claim 5, characterized in that: In step S202, extracting local detail features from the pulse features using the second path of the dual-path spiking neural network specifically includes: The color information of the input pulse features is preserved, maintaining the independence of the red, green, and blue channels, to obtain the color pulse features; The color pulse features are input into LIF neurons with a second time constant for processing. The fine integration process of the second pathway is simulated through a slow integration and accumulation mechanism to obtain the pulse sequence of the second pathway. The second path pulse sequence is subjected to a standard convolution operation. A small-sized convolution kernel is used to densely sample in a local region to maintain spatial resolution. The enhanced second path pulse features are obtained through residual connections. The enhanced second pathway pulse features are aggregated in the time dimension, and the average firing rate of each neuron in all time steps is calculated to obtain color-dominated local detail features. Wherein, the second time constant is greater than the first time constant.

7. The image restoration method according to claim 1, characterized in that: S204 specifically includes: The mid-layer fusion features are processed in parallel using convolutions with receptive fields of various sizes to extract global contextual information at different receptive field scales, thereby obtaining semantic features at multiple scales. The semantic features at multiple scales are fused and residually connected with the input mid-level fused features to obtain high-level representation features with rich global semantic information. The high-level representation features are spatially compressed using global average pooling to extract a compact global context vector. The global context vector is input into a lightweight multilayer perceptron for nonlinear transformation to generate channel-dimensional modulation weights. The high-level representation features are subjected to adaptive weighted modulation on a channel-by-channel basis using the modulation weights to obtain the modulated features; The modulated features are residually concatenated with the high-level representation features to obtain the final high-level semantic features.

8. The image restoration method according to claim 1, characterized in that: S205 specifically includes: S2051. For the first decoding stage, the high-level semantic features are used as input coarse-scale features. For subsequent decoding stages, the output features of the previous decoding stage are used as input coarse-scale features for the next decoding stage. Upsample the coarse-scale features of the current decoding stage to restore them to the target resolution, and obtain the upsampled features. S2052. Perform channel alignment on the upsampled features by adjusting the number of channels to the same dimension as the corresponding level skip connection features through convolution operations to obtain channel-aligned features. S2053. The channel-aligned features are spliced ​​and fused with the corresponding level of jump connection features to obtain stage fusion features; S2054. Perform multi-scale enhancement on the stage fusion features by convolutional operations with different dilation rates, and extract details and structural information of different receptive fields to obtain multi-scale enhanced features. S2055. Perform residual fusion on the multi-scale enhanced features and add them to the stage fusion features to obtain residual enhanced features; S2056. The residual enhancement features are refined by introducing an attention mechanism to highlight key structural regions and texture details, and the output features of the current decoding stage are obtained as the input coarse-scale features of the next decoding stage. S2057. Repeat steps S2051 to S2056 to perform stepwise upsampling and feature fusion, and pass the decoded features stepwise until the image is restored to the same resolution as the input degraded image, thus obtaining a preliminary restored image.

9. An image restoration system, characterized in that, The system includes: Degraded image acquisition module, configured to acquire degraded images; Image restoration model building modules include: The pulse feature extraction submodule is configured to perform biomimetic early visual processing on the degraded image, including center-periphery antagonistic enhancement, and pulse coding using LIF neurons with learnable thresholds to obtain pulse features; A pathway feature extraction submodule, configured to extract features from the pulse features using a dual-path spiking neural network, includes: The first path of the dual-path spiking neural network is used to extract global structural features from the pulse features, and the second path of the dual-path spiking neural network is used to extract local detail features from the pulse features. The first pathway uses a first time constant to simulate fast response characteristics and utilizes a large receptive field modeling mechanism to extract brightness-dominated global structural features; the second pathway uses a second time constant to simulate a fine integration process and utilizes a high resolution preservation mechanism to extract color-dominated local detail features. The fusion processing submodule performs fusion processing on the global structural features and the local detail features to obtain the mid-level fused features; The semantic modeling submodule performs semantic modeling on the mid-level fusion features to obtain high-level semantic features; The decoding submodule utilizes a multi-level decoding and recovery structure to perform step-by-step upsampling and feature fusion on the high-level semantic features to obtain a preliminary recovered image. The temporal integration submodule constructs a multi-time-step input sequence for the degraded image and performs temporal integration on the preliminary restored images corresponding to the multi-time-step input sequence to obtain the final restored image; The image restoration model is obtained in the following way: Obtain a training dataset, which includes degraded images and corresponding real clean images; The image restoration model is trained using the training dataset to obtain the trained image restoration model; The image restoration module is configured to acquire the degraded image to be processed, input the degraded image to be processed into the trained image restoration model, and output the restored image.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, it implements the image restoration method as described in any one of claims 1 to 8.

Images