Image reconstruction method and system based on pulse memory autoencoder

Abstract

The application belongs to the field of pulse neural network and image reconstruction, and discloses an image reconstruction method and system based on a pulse memory autoencoder. The pulse memory autoencoder comprises an encoder, a memory network and a decoder. An input image is obtained, the input image is converted into a pulse sequence by using delay coding, and the pulse sequence is input into the encoder to obtain corresponding pulse features. The pulse features are input into the memory network to obtain corresponding memory features. In the recall stage, the pulse features are only transmitted to the perception neuron, the response neuron is activated by the synaptic connection learned by the perception neuron in the memory stage, and the memory features are output to the decoder to generate a corresponding reconstructed image. The application can enhance the maintenance capability of the potential representation and complete high-quality image reconstruction while maintaining a low pulse overhead.

Description

Technical Field

[0001] This invention belongs to the field of spiking neural networks and image reconstruction, specifically relating to an image reconstruction method and system based on a pulse memory autoencoder. Background Technology

[0002] Autoencoders are essentially artificial neural network structures used for feature learning and data reconstruction, primarily consisting of two parts: an encoder and a decoder. The encoder maps the input data to a low-dimensional latent space, extracting its main features by compressing the original input. The decoder reconstructs the input data based on the latent representation output by the encoder, generating a result that is as close as possible to the original input. By minimizing the error between the input data and the reconstructed result, autoencoders learn a compact representation of the data. Because this process relies entirely on the structural features of the data itself without external labels, autoencoders are widely used in tasks such as dimensionality reduction, denoising, and feature extraction.

[0003] However, traditional autoencoders are mostly built on artificial neural networks, and their information transmission usually takes the form of continuous numerical values. The training process mainly aims to minimize the error between the input image and the reconstructed image. Although this type of method can improve the reconstruction ability of training samples, it tends to focus too much on pixel-level error optimization and still has shortcomings in learning the feature distribution, structural information, and diversity representation in the latent space. In some image reconstruction or image restoration tasks, traditional autoencoders may suffer from insufficient latent representation stability, limited ability to preserve details of the reconstructed image, and weak model interpretability. Spiking neural networks use discrete pulses as information transmission carriers and can use the temporal relationship of neuronal pulse firing to represent complex spatiotemporal information, with characteristics such as temporal dynamic expression, sparse computation, and biological interpretability. Introducing spiking neural networks into autoencoders can help build more sparse and low-power image reconstruction models.

[0004] Most existing pulse autoencoders' latent representations only participate in the reconstruction process as an intermediate mapping between the encoder and decoder, lacking the storage, preservation, and extraction of latent features. Furthermore, they lack effective coordination between local synaptic plasticity rules and global error optimization, making it difficult to simultaneously achieve both pulse sparsity and reconstruction quality. Summary of the Invention

[0005] The purpose of this invention is to provide an image reconstruction method and system based on a pulse memory autoencoder. The pulse autoencoder incorporates a memory network consisting of a perceptual layer, a response layer, and a suppression layer. This allows the pulse features output by the encoder to be stored, associated, and extracted within the memory network before being input into the decoder for image reconstruction, rather than merely serving as an intermediate mapping between the encoder and decoder. This approach enhances the model's ability to preserve latent representations and achieves high-quality image reconstruction while maintaining low pulse overhead.

[0006] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0007] In a first aspect, an image reconstruction method based on a pulse memory autoencoder is provided. The pulse memory autoencoder includes an encoder, a memory network, and a decoder. The image reconstruction method based on the pulse memory autoencoder includes the following steps:

[0008] The input image is acquired, and delayed coding is used to convert the input image into a pulse sequence. The pulse sequence is then input into the encoder to obtain the corresponding pulse features.

[0009] The pulse features are input into the memory network to obtain the corresponding memory features. The memory network consists of two layers of excitatory neurons and one layer of inhibitory neurons. The excitatory neurons include sensory neurons and response neurons. The response neurons are unidirectionally connected to a corresponding inhibitory neuron. The inhibitory neurons are unidirectionally connected to all other response neurons except the input response neuron.

[0010] During the memory phase, the pulse features are simultaneously transmitted to the sensory neurons and the response neurons. The sensory neurons and the response neurons establish and update synaptic connections in real time through trace-based pulse temporal dependence plasticity learning rules. The pulse memory autoencoder uses spatiotemporal backpropagation training rules for error backpropagation training.

[0011] During the recall phase, the impulse features are transmitted only to the sensory neurons. The response neurons are activated by the synaptic connections learned by the sensory neurons during the memory phase and output the memory features to the decoder, which then generates the corresponding reconstructed image.

[0012] Several alternative methods are provided below, but they are not intended as additional limitations on the overall solution above. They are merely further additions or optimizations. Provided there are no technical or logical contradictions, each alternative method can be combined individually with respect to the overall solution above, or multiple alternative methods can be combined with each other.

[0013] Preferably, the step of converting the input image into a pulse sequence using delay coding includes:

[0014] For each pixel in the input image, if the pixel intensity is greater than the firing threshold, then the firing time of the pixel is set to... Otherwise, the pixels are not fired, thus generating the corresponding pulse sequence; where The membrane time constant is Indicates the intensity of a pixel. Indicates the issuance threshold. It is the natural logarithm function.

[0015] Preferably, the encoder is composed of a convolutional neural network, which contains two convolutional blocks composed of leaky integral release neurons and a dense layer. Each convolutional block contains a convolutional layer and a max pooling layer.

[0016] The decoder consists of two dense layers of neurons based on leakage integral firing.

[0017] Preferably, the spatiotemporal backpropagation training rule is used to train the weights of the encoder and decoder. Specifically, it adopts a chain rule to expand layer by layer. First, the partial derivative of the loss function with respect to the membrane voltage is obtained. Then, the partial derivative of the loss function with respect to the weights is obtained by combining the partial derivative relationship between the membrane voltage and the input current and the weights.

[0018] Preferably, during the memory stage, the inputs to the response neurons include impulse characteristics, impulse information transmitted by the sensory neurons, and inhibitory signals generated by the inhibitory neurons.

[0019] During the recall phase, the input to the responsive neurons includes impulse information transmitted by the sensory neurons and inhibitory signals generated by the inhibitory neurons.

[0020] Preferably, the trace-based pulse timing-dependent plasticity learning rule adjusts synaptic weights according to the pulse firing traces of presynaptic and postsynaptic neurons.

[0021] When a presynaptic or postsynaptic neuron fires a pulse, the corresponding pulse firing trace increases instantaneously, and then increases over time at a constant rate. It decays exponentially.

[0022] When the presynaptic neuron fires a pulse, the synaptic weight update between the presynaptic and postsynaptic neurons is expressed as follows: When the postsynaptic neuron fires a pulse, the update of the synaptic weights between the presynaptic and postsynaptic neurons is expressed as follows: ; and These represent the synaptic reinforcement learning rate and the synaptic inhibition learning rate, respectively. This is the upper limit of synaptic weight. Presynaptic neuron exist The pulse firing traces at any given moment, Postsynaptic neurons exist The pulse firing traces at any given moment, Presynaptic neuron and postsynaptic neurons between Synaptic weights at specific moments;

[0023] By performing time differentiation on the synaptic weights between presynaptic and postsynaptic neurons, the continuous time dynamic relationship of the synaptic weights between them is obtained, which can be used to adaptively adjust the synaptic weights.

[0024] Preferably, the pulse firing trace is updated as follows:

[0025] If the corresponding neuron is If no pulse is issued at a given time, then The pulse firing trace at a given moment is Pulse firing traces and coefficients at different times The product; if the corresponding neuron is in If pulses are emitted at all times, then The pulse firing trace at that moment is, Pulse firing traces and coefficients at different times The product of, and then with sum.

[0026] Secondly, an image reconstruction system based on a pulse memory autoencoder is provided, including a processor and a memory storing a plurality of computer instructions, wherein the computer instructions, when executed by the processor, implement the steps of the image reconstruction method based on the pulse memory autoencoder.

[0027] The image reconstruction method and system based on pulse memory autoencoder provided by this invention have the following advantages compared with the prior art:

[0028] (1) A memory network consisting of perceptual neurons, response neurons, and inhibitory neurons is constructed in the pulse memory autoencoder, so that the pulse features extracted by the encoder are no longer directly input into the decoder, but can be stored, associated, and retrieved in the memory network. The perceptual layer and the response layer establish and update synaptic connections in real time through the Trace-Based Spike-Timing-Dependent Plasticity (TR-STDP) learning rule, so that the latent representation can be stored inside the memory network. The response neurons and inhibitory neurons form a competitive regulatory relationship through one-to-one correspondence. The inhibitory neurons can inhibit the response neurons, thereby reducing redundant activation and making the output of the response layer sparser and more discriminative. During the recall stage, the perceptual neurons activate the response neurons according to the established synaptic connections, thereby extracting memory features related to the input image. In this way, the model can enhance the ability to store, retain, and retrieve latent representations, retain more effective information under conditions of less pulse activity, and thus better balance pulse sparsity and image reconstruction ability.

[0029] (2) A dual learning rule of TR-STDP and Spatio-Temporal Backpropagation (STBP) is adopted to balance local memory learning and global reconstruction optimization. The TR-STDP learning rule is used to update the synaptic connections in the memory network, and the STBP training rule is used for error backpropagation optimization. Specifically, the TR-STDP learning rule adjusts the synaptic weights according to the pulse firing sequence between sensory neurons and response neurons, enabling the memory network to store and associate input pulse information. Using STBP for training improves the image reconstruction quality. Through the synergistic effect of the two learning rules, this invention retains the characteristics of local plasticity learning in spiking neural networks while also possessing the ability to optimize reconstruction errors. Attached Figure Description

[0030] Figure 1 This is a flowchart of the image reconstruction method based on pulse memory autoencoder of the present invention. Detailed Implementation

[0031] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0032] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein in the description of the invention is for the purpose of describing particular embodiments only and is not intended to limit the invention.

[0033] To overcome the shortcomings of existing technologies, this embodiment provides an image reconstruction method based on a pulse memory autoencoder. A memory network is introduced into the pulse autoencoder, which enhances the ability to store, retain, and extract latent representations while maintaining the sparsity of pulse activity, and completes high-quality image reconstruction, thereby taking into account both the performance of the pulse autoencoder in image reconstruction tasks and energy consumption.

[0034] like Figure 1 As shown, the image reconstruction method based on pulse memory autoencoder in this embodiment includes the following steps:

[0035] Step 1: Acquire the input image, use delay coding to convert the input image into a pulse sequence, so that the pixel intensity of the image is expressed by the pulse firing time, and input the pulse sequence into the encoder to extract the information of the input image and obtain the corresponding pulse features.

[0036] The delay coding method determines the pulse firing time based on the pixel intensity of the input image. High-intensity pixels correspond to earlier pulse firing times, while low-intensity pixels correspond to later pulse firing times or no pulse is generated. Thus, the grayscale information of the input image is converted into pulse firing timing information. Through delay coding, the input image can enter the network as a sparse pulse sequence. The delay coding expression is:

[0037]

[0038] in, Indicates pixel intensity The corresponding pulse firing time; The threshold for neuronal impulse firing; This represents the membrane time constant, which can control the decay rate of the threshold. This represents the intensity of the input pixel; the greater the input intensity, the better the result. The smaller the value, the earlier the pulse is emitted.

[0039] The encoder in this embodiment is composed of a convolutional neural network, which contains two convolutional blocks consisting of Leaky Integrate-and-Fire (LIF) neurons and a dense layer. Each convolutional block contains a convolutional layer and a max-pooling layer. The decoding layer consists of two dense layers based on LIF neurons, whose main function is to convert the pulse sequence output by the memory network into the final output. After processing by the encoder, the encoded pulse features are obtained, which contain both spatial structural information and pulse temporal information of the input image.

[0040] In this embodiment, the encoder uses a 3×3 kernel for the convolutional layers and a stride of 1; the max-pooling layers use a 2×2 kernel and a stride of 2. The original 784-dimensional image is input into the convolutional block, and the output dimension of the convolutional block is compressed to 32 dimensions. The output of the convolutional block then passes through a dense layer with the same dimension as the memory network, outputting 100-dimensional features. These features are then passed to the memory network. The perceptual neurons, response neurons, and inhibitory neurons in the memory network all have a dimension of 100.

[0041] Step 2: The pulse memory autoencoder in this embodiment includes an encoder, a memory network, and a decoder. Therefore, the obtained pulse features are not directly input into the decoder, but are input into the memory network to obtain the corresponding memory features.

[0042] In this embodiment, the memory network consists of two layers of excitatory neurons and one layer of inhibitory neurons. The excitatory neurons include sensory neurons and response neurons. Sensory neurons receive the pulse characteristics output by the encoder, while response neurons generate corresponding pulses based on the established synaptic connections between the sensory neurons and response neurons. Inhibitory neurons primarily inhibit the membrane potential of the excitatory neurons, thereby regulating them.

[0043] During the memory phase, the pulse features output by the encoder are simultaneously transmitted to both perceptual and response neurons. The synaptic connections between these neurons are strengthened or weakened according to the pulse firing sequence, thus storing the latent features of the input image in the memory network as synaptic connections. In this embodiment, perceptual and response neurons establish and update these synaptic connections in real time using the TR-STDP learning rule. Response neurons are connected to inhibitory neurons in a one-to-one correspondence; each inhibitory neuron is connected to all response neurons except the one receiving its input. When a response neuron fires a pulse, its corresponding inhibitory neuron is activated and sends inhibitory signals to other response neurons, thereby reducing the activity level of other response neurons, enhancing competition among them, and making neurons with stronger responses more likely to be retained, while irrelevant or redundant responses are suppressed, reducing redundant activation and making the response pulses sparser and more discriminative.

[0044] After a period of pulsed activity, the synapses between sensory neurons and response neurons strengthen, weaken, or form new connections, thus establishing associations. The input current received by the sensory neuron during this process can be described as:

[0045]

[0046] in, Represents sensory neurons exist Input current at any given time Represents the neurons in the last layer of the pulsed convolutional layer. exist A sequence of pulses transmitted over time. Represents the neurons in the last layer of the pulsed convolutional layer. To sensory neurons Synaptic connection weights.

[0047] Accordingly, the response neuron receives not only the pulse sequence output by the feature extractor, but also the pulse information transmitted by the sensory neuron and the inhibitory signal generated by the inhibitory neuron. Its input current can be expressed as:

[0048]

[0049] in, In response to neurons exist The input current at any given time. It is a sensory neuron exist A pulse of time. Inhibitory neurons exist The pulse of time, This refers to the inhibitory factor used to regulate the intensity of the inhibitory effect. Represents sensory neurons With response neurons Synaptic weights between Indicating inhibitory neurons With response neurons Synaptic weights between This is the index of the corresponding neuron.

[0050] During the recall phase, the pulse features output by the encoder are only input to the perceptual neurons. The response neurons no longer directly receive the encoder output; instead, they generate corresponding pulses through the established synaptic connections between the perceptual and response neurons. The response neurons are activated by the perceptual neurons through previously learned synaptic connections. This process enables the model to extract information using stored synaptic connections while simultaneously receiving inhibitory signals generated by inhibitory neurons. At this point, the input current received by the response neurons can be expressed as:

[0051]

[0052] In this embodiment, the pulse memory autoencoder uses the STBP training rule for error backpropagation training and combines it with the TR-STDP learning rule to update the synaptic connections between excitatory neurons in the memory network, forming a dual learning rule that combines local synaptic plasticity learning with global error optimization.

[0053] The TR-STDP learning rule adjusts synaptic weights based on the pulse firing traces of presynaptic and presynaptic neurons. Each neuron's pulse generates a time-decreasing trace variable; under this mechanism, the trace value is only related to the time interval since the most recent pulse. When a presynaptic neuron fires a pulse, its corresponding trace increases instantaneously by a certain magnitude, then decays over time at a constant value. It decays exponentially. The specific update process is as follows: if the corresponding neuron is... If no pulse is issued at a given time, then The pulse firing trace at a given moment is Pulse firing traces and coefficients at different times The product; if the corresponding neuron is in If pulses are emitted at all times, then The pulse firing trace at that moment is, Pulse firing traces and coefficients at different times The product of, and then with sum.

[0054] With presynaptic neurons and postsynaptic neurons Taking connectivity as an example, each pulse from the presynaptic neuron generates a trace at the synapse, which characterizes the synapse's short-term memory or influence on the pulse event. This trace gradually decays when there is no new pulse input, and its expression is as follows:

[0055]

[0056] in, Presynaptic neuron exist The pulse firing traces at any given moment, Presynaptic neuron exist The pulse firing traces at any given moment, This indicates that the presynaptic neuron is in The value indicates whether a pulse is generated at any given time; a value of 1 indicates a pulse is generated, and a value of 0 indicates no pulse is generated. Therefore, when a postsynaptic neuron fires a pulse, the weights are correlated with the presynaptic traces. It increases proportionally. Similarly, when a presynaptic neuron fires a pulse, the synaptic weight increases proportionally to the trace left by the previous postsynaptic pulse. The proportional reduction is expressed as follows:

[0057]

[0058] in, Postsynaptic neurons exist The pulse firing traces at any given moment, Postsynaptic neurons exist The pulse firing traces at any given moment, This indicates that postsynaptic neurons are in The value indicates whether a pulse is generated at a given time. A value of 1 indicates that a pulse is generated, and a value of 0 indicates that no pulse is generated.

[0059] Based on the dynamic changes of the trace variable mentioned above, the synaptic weight update between presynaptic and postsynaptic neurons can be adjusted according to the following rule, expressed as:

[0060]

[0061] in and These represent the learning rates for synaptic enhancement and inhibition, respectively. This is the upper limit of synaptic weight. Presynaptic neuron and postsynaptic neurons between The amount of synaptic weight update at any given moment. Presynaptic neuron and postsynaptic neurons between Synaptic weights at time points.

[0062] This learning rule indicates that when a postsynaptic neuron fires a pulse, the weights will be determined according to the presynaptic trace. The value increases proportionally, i.e., the update amount is... When a presynaptic neuron fires a pulse, the weights are adjusted according to the postsynaptic trace. The value decreases proportionally, i.e., the update amount is... .

[0063] Furthermore, by performing temporal differentiation on the synaptic weights between presynaptic and postsynaptic neurons, the continuous-time dynamic relationship of the synaptic weights between them is obtained, which can be used to adaptively adjust the synaptic weights. That is, the continuous-time dynamic process of the synaptic weights can be expressed by a differential equation:

[0064]

[0065] in, Indicates synaptic weight Regarding time Differentiate, This represents the time difference between the firing of presynaptic and postsynaptic pulses. Through the above mechanism, TR-STDP can adaptively adjust synaptic weights according to the temporal relationship of neuronal activity, thereby achieving fine-grained control over neural network function. Based on this, this embodiment uses the STBP training rule to optimize image reconstruction error, while using the TR-STDP learning rule to complete the memory storage of latent features.

[0066] This embodiment uses the STBP training rule to train the weights of the encoder and decoder. It employs STBP training for error backpropagation, updating network parameters through gradient propagation in both the temporal and spatial dimensions. During training, the error between the input image and the reconstructed image is used as the optimization objective, and the encoder and decoder parameters are updated through error backpropagation in both the temporal and spatial dimensions. Its expression is as follows:

[0067]

[0068] in, The loss function is simplified as follows: , This represents the original input image, corresponding to a length of... The input pulse sequence, This represents the reconstructed image, specifically the image obtained from the reconstructed image with a length of... The output pulse sequence, Represents the first in the input image 1 pixel, Represents the first in the reconstructed image Each pixel. This embodiment uses a chain rule to expand the expression across... The layers are used to calculate the update rules for the spiking time and weights of each neuron in the network, as shown below:

[0069]

[0070] in, This represents the loss function, used to describe the overall error of the network. Indicates the first Layer weights express Time of the first The film voltage of the layer, express Time of the first The input current of the layer, This represents the change in input current. express Time of the first The layer's pulse output, The total number of time points is represented. This expression is expanded layer by layer using the chain rule. First, the partial derivative of the loss function with respect to the membrane voltage is calculated. Then, combining the partial derivatives of the membrane voltage with respect to the input current and weights, the partial derivative of the loss function with respect to the weights is obtained. This backpropagation process simultaneously utilizes information from both the time and spatial dimensions to adjust the network parameters, thereby minimizing the error between the output and the target.

[0071] In this embodiment, the TR-STDP learning rule and the STBP training rule are applied to different parts of the network. The TR-STDP learning rule applies to the synaptic connections in the memory network, while the STBP training rule applies to the trainable weights in the encoder and decoder. In this way, the model can simultaneously achieve latent feature memorization and image reconstruction optimization.

[0072] Step 3: During the recall phase, the pulse features output by the encoder are only transmitted to the perceptual neurons. The response neurons are activated by the synaptic connections learned by the perceptual neurons during the memory phase and output memory features to the decoder, which then generates the corresponding reconstructed image. In this embodiment, the decoder consists of two dense layers of neurons based on leakage integral firing neurons. It is easy to understand that both the pulse features output by the encoder and the memory features output by the memory network are pulse sequences. To facilitate the distinction between pulse sequences in different stages, they are given different names.

[0073] In one specific embodiment, to intuitively demonstrate the advantages of the method of the present invention, the following experiments were conducted. All experiments were implemented using the Python programming language, and the experimental environment was a personal computer equipped with an Intel(R) Core(TM) i5-11400 central processing unit and an NVIDIA GeForce GTX 4090 graphics processor (24GB).

[0074] (1) Experimental setup:

[0075] The parameter settings for this experiment are shown in Table 1.

[0076] Table 1 Experimental Parameter Settings

[0077]

[0078] The experiments were conducted on the MNIST and FashionMNIST datasets. The MNIST dataset contains grayscale images of handwritten digits, each 28×28 pixels in size and a single-channel grayscale image. The FashionMNIST dataset contains grayscale images of 10 categories of clothing, each 28×28 pixels in size and a single-channel grayscale image.

[0079] (2) Comparison of pulse overhead (referring to the number of pulses fired to achieve image reconstruction):

[0080] The method of the present invention is compared with four existing benchmark models, and the comparison results are shown in Table 2.

[0081] Comparative Example 1 is the paper "Training deep spiking auto-encoders without bursting ordying neurons through regularization", which discloses the Deep Regularized Spiking Auto-Encoder (DR-SAE).

[0082] Comparative Example 2 is the paper "E-STDP: a spatio-temporally local unsupervised learning rule for sparse coded spiking convolutional autoencoders", which discloses a sparse coded spiking convolutional autoencoder based on E-STDP (ESTDP-SCAE).

[0083] Comparative Example 3 is the paper "Efficient sparse spiking auto-encoder for reconstruction, denoising and classification", which discloses an efficient sparse spiking auto-encoder (ES-SAE).

[0084] Comparative Example 4 is the literature "Spiking autoencoders with temporal coding", which discloses a temporal coding spike autoencoder (TC-SAE).

[0085] Table 2 Comparison of pulse overhead (unit: pulses) on different datasets

[0086]

[0087] Combining Table 2 with the image reconstruction results of the proposed method on the MNIST and FashionMNIST datasets, it can be concluded that the proposed pulse memory autoencoder achieves a more reasonable balance between reconstruction performance and pulse overhead. It can still achieve high-quality image reconstruction with fewer pulses and maintains excellent performance in important metrics such as classification accuracy. It should be noted that the number in Table 2 represents the average number of pulses fired for all images on the datasets.

[0088] (3) Comparison of image classification tasks:

[0089] To explore the advantages of the pulse memory autoencoder of the present invention, the pulse memory autoencoder proposed in the present invention was compared with six existing methods in a downstream image classification task. The classification accuracy was compared based on the reconstructed images, and the comparison results are shown in Table 3.

[0090] Comparative Example 1 is the publicly available image classification task in the paper "E-STDP: a spatio-temporally local unsupervised learning rule for sparse coded spiking convolutional autoencoders".

[0091] Comparative Example 2 is the document "A heterogeneous spiking neural network for computationally efficient face recognition", which discloses an image classification task;

[0092] Comparative Example 3 is the paper "Efficient sparse spiking auto-encoder for reconstruction, denoising and classification", which discloses an image classification task.

[0093] Comparative Example 4 is the paper "Unsupervised learning of digit recognition using spike-timing-dependent plasticity", a publicly available image classification task;

[0094] Comparative Example 5 is the paper "Bio-inspired digit recognition using reward-modulated spike-timing-dependent plasticity in deep convolutional networks", a publicly disclosed image classification task;

[0095] Comparative Example 6 is the paper "Fspinn: An optimization framework for memory-efficient and energy-efficient spiking neural networks", which discloses an image classification task.

[0096] Table 3 Comparison of image classification accuracy (in %) on different datasets

[0097]

[0098] Based on the results in Table 3, it can be seen that the spiking memory autoencoder achieves good results in downstream classification tasks compared with other unsupervised SNN classification algorithms. However, its accuracy is slightly lower than that of benchmark shallow spiking neural networks and deep spiking convolutional neural networks. Nevertheless, the spiking memory autoencoder forms more refined and meaningful feature representations in the latent space through synaptic plasticity. These latent features are not merely simple mappings of data, but can be effectively extracted from the model when needed through memory storage, thus ensuring good representation and interpretability of the latent space. During the feature representation formation process in the latent space, the spiking memory autoencoder only establishes synaptic connections between relevant neurons. This strategy effectively reduces unnecessary connections, avoids redundant computation, and reduces storage overhead.

[0099] In another embodiment, the present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of an image reconstruction method based on a pulse memory autoencoder.

[0100] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided by this invention can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0101] In another embodiment, the present invention provides an image reconstruction system based on a pulse memory autoencoder, namely a computer device, including a processor and a memory storing a plurality of computer instructions, wherein the computer instructions are executed by the processor to implement the steps of the image reconstruction method based on the pulse memory autoencoder.

[0102] The memory and processor are electrically connected directly or indirectly to enable data transmission or interaction. For example, these components can be electrically connected to each other via one or more communication buses or signal lines. The memory stores a computer program that can run on the processor, which implements the method of the present invention by running the computer program stored in the memory.

[0103] The memory may be, but is not limited to, Random Access Memory (RAM), Read Only Memory (ROM), Programmable Read-Only Memory (PROM), Erasable Programmable Read-Only Memory (EPROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), etc. The memory stores the program, and the processor executes the program upon receiving an execution instruction.

[0104] The processor may be an integrated circuit chip with data processing capabilities. The aforementioned processor can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this invention. The general-purpose processor can be a microprocessor or any conventional processor.

[0105] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0106] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the appended claims.

Claims

1. An image reconstruction method based on a pulse memory autoencoder, characterized in that, The pulse memory autoencoder includes an encoder, a memory network, and a decoder. The image reconstruction method based on the pulse memory autoencoder includes the following steps: The input image is acquired, and delayed coding is used to convert the input image into a pulse sequence. The pulse sequence is then input into the encoder to obtain the corresponding pulse features. The pulse features are input into the memory network to obtain the corresponding memory features. The memory network consists of two layers of excitatory neurons and one layer of inhibitory neurons. The excitatory neurons include sensory neurons and response neurons. The response neurons are unidirectionally connected to a corresponding inhibitory neuron. The inhibitory neurons are unidirectionally connected to all other response neurons except the input response neuron. During the memory phase, the pulse features are simultaneously transmitted to the sensory neurons and the response neurons. The sensory neurons and the response neurons establish and update synaptic connections in real time through trace-based pulse temporal dependence plasticity learning rules. The pulse memory autoencoder uses spatiotemporal backpropagation training rules for error backpropagation training. During the recall phase, the impulse features are transmitted only to the sensory neurons. The response neurons are activated by the synaptic connections learned by the sensory neurons during the memory phase and output the memory features to the decoder, which then generates the corresponding reconstructed image.

2. The image reconstruction method based on pulse memory autoencoder according to claim 1, characterized in that, The step of converting the input image into a pulse sequence using delay coding includes: For each pixel in the input image, if the pixel intensity is greater than the firing threshold, then the firing time of the pixel is set to... Otherwise, the pixels are not fired, thus generating the corresponding pulse sequence; where The membrane time constant is Indicates the intensity of a pixel. Indicates the issuance threshold. It is the natural logarithm function.

3. The image reconstruction method based on pulse memory autoencoder according to claim 1, characterized in that, The encoder is composed of a convolutional neural network, which contains two convolutional blocks consisting of leaky integral release neurons and a dense layer. Each convolutional block contains a convolutional layer and a max pooling layer. The decoder consists of two dense layers of neurons based on leakage integral firing.

4. The image reconstruction method based on pulse memory autoencoder according to claim 3, characterized in that, The spatiotemporal backpropagation training rule is used to train the weights of the encoder and decoder. Specifically, it adopts a chain rule to expand layer by layer. First, the partial derivative of the loss function with respect to the membrane voltage is obtained. Then, combined with the partial derivative relationship between the membrane voltage and the input current and the weights, the partial derivative of the loss function with respect to the weights is obtained.

5. The image reconstruction method based on pulse memory autoencoder according to claim 1, characterized in that, During the memory phase, the inputs to the response neurons include impulse characteristics, impulse information transmitted by the sensory neurons, and inhibitory signals generated by the inhibitory neurons. During the recall phase, the input to the responsive neurons includes impulse information transmitted by the sensory neurons and inhibitory signals generated by the inhibitory neurons.

6. The image reconstruction method based on pulse memory autoencoder according to claim 1, characterized in that, The trace-based pulse timing-dependent plasticity learning rule adjusts synaptic weights according to the pulse firing traces of presynaptic and postsynaptic neurons. When a presynaptic or postsynaptic neuron fires a pulse, the corresponding pulse firing trace increases instantaneously, and then increases over time at a constant rate. It decays exponentially. When the presynaptic neuron fires a pulse, the synaptic weight update between the presynaptic and postsynaptic neurons is expressed as follows: When the postsynaptic neuron fires a pulse, the update of the synaptic weights between the presynaptic and postsynaptic neurons is expressed as follows: ; and These represent the synaptic reinforcement learning rate and the synaptic inhibition learning rate, respectively. This is the upper limit of synaptic weight. Presynaptic neuron exist The pulse firing traces at any given moment, Postsynaptic neurons exist The pulse firing traces at any given moment, Presynaptic neuron and postsynaptic neurons between Synaptic weights at specific moments; By performing time differentiation on the synaptic weights between presynaptic and postsynaptic neurons, the continuous time dynamic relationship of the synaptic weights between them is obtained, which can be used to adaptively adjust the synaptic weights.

7. The image reconstruction method based on pulse memory autoencoder according to claim 6, characterized in that, The pulse firing trace is updated as follows: If the corresponding neuron is If no pulse is issued at a given time, then The pulse firing trace at a given moment is Pulse firing traces and coefficients at different times The product; if the corresponding neuron is in If pulses are emitted at all times, then The pulse firing trace at that moment is, Pulse firing traces and coefficients at different times The product of, and then with sum.

8. An image reconstruction system based on a pulse memory autoencoder, comprising a processor and a memory storing a plurality of computer instructions, characterized in that, When the computer instructions are executed by the processor, they implement the steps of the image reconstruction method based on pulse memory autoencoder as described in any one of claims 1 to 7.

Images