A visual reconstruction method, device, storage medium and terminal based on pulse neural network

Abstract

The present invention discloses a visual reconstruction method, device, storage medium and terminal based on a pulse neural network. The method includes: receiving a continuous pulse array signal to be processed in each time period from a visual sensor; discretizing the signal to generate discrete signal data carrying time information; inputting the discrete signal data into a pre-trained pulse neural network to output a continuous visual image; the pre-trained pulse neural network is generated by cyclic training based on pulse array signals of multiple consecutive time steps, and the visual image is generated based on membrane potential information, which is generated by feature extraction of the continuous pulse array signal to be processed. Because the present application uses pulse signals based on multiple time steps and combines them with a pulse neural network for network training, the network can extract the membrane potential information of the pulse signal, and then restore the visual image based on the membrane potential information, thereby solving the visual reconstruction problem of the pulse array signal.

Description

Technical Field

[0001] The present invention relates to the field of signal processing technology, and in particular to a visual reconstruction method, device, storage medium and terminal based on a pulse neural network. Background Art

[0002] Spiking neural networks are considered the third generation of artificial neural networks. Internally, information is transmitted entirely in binary form, resulting in time-domain information processing capabilities and low power consumption. Spiking neurons and their synaptic connection mechanisms are the primary differences between spiking neural networks and traditional artificial neural networks. Spiking neurons aggregate presynaptic input and determine whether to fire a spike based on the neuron's state, thus possessing temporal memory. Information within synapses in spiking neural networks is transmitted via binary spikes, a highly efficient data processing method.

[0003] New visual sensors, such as event cameras and pulse cameras, offer advantages such as high speed, high dynamic range, and low power consumption. However, unlike traditional cameras, the signals emitted by these sensors are not intuitively understandable to humans. Therefore, recovering visual information from these neuromorphic signals is a critical task. Summary of the Invention

[0004] The embodiments of the present application provide a method, apparatus, storage medium, and terminal for visual reconstruction based on a spiking neural network. To provide a basic understanding of some aspects of the disclosed embodiments, a brief summary is provided below. This summary is not intended to be a comprehensive review, identify key / important elements, or delineate the scope of protection for these embodiments. Its sole purpose is to present some concepts in a simplified form, serving as a prelude to the detailed description that follows.

[0005] In a first aspect, an embodiment of the present application provides a visual reconstruction method based on a spiking neural network, the method comprising:

[0006] Receiving a continuous pulse array signal to be processed in each time period from a visual sensor;

[0007] Discretize the continuous pulse array signal to be processed to generate discrete signal data carrying time information;

[0008] The discrete signal data carrying time information is input into the pre-trained pulse neural network, and the visual image corresponding to the continuous pulse array signal to be processed is output; wherein,

[0009] The pre-trained spiking neural network is generated by cyclic training based on pulse array signals of multiple consecutive time steps. The visual image is generated based on membrane potential information, and the membrane potential information is generated by feature extraction of the continuous pulse array signal to be processed.

[0010] Optionally, generate a pre-trained spiking neural network by following these steps:

[0011] Create spiking neural networks;

[0012] receiving a target pulse array signal collected by a vision sensor at a first time step;

[0013] Discretize the target pulse array signal to generate target discrete signal data carrying time information;

[0014] Input the target discrete signal data carrying time information into the spiking neural network, update the internal state of the neurons in the spiking neural network, and then output the grayscale value and network loss value corresponding to the target pulse array signal;

[0015] in,

[0016] The internal state of a neuron includes at least membrane potential, firing threshold, and membrane time constant;

[0017] Generate a pre-trained spiking neural network based on the network loss value.

[0018] Optionally, generate a pre-trained spiking neural network based on the network loss value, including:

[0019] If the network loss value does not reach the preset loss value or the current training times do not reach the preset training times, the gradient value of back propagation is calculated;

[0020] Back-propagating the gradient value of back propagation, updating the neuron weight value of the pulse neural network according to the gradient value of back propagation, and receiving the target pulse array signal collected by the visual sensor at the next time step;

[0021] Continue to perform the step of discretizing the target pulse array signal until the network loss value reaches a preset loss value;

[0022] or,

[0023] If the network loss value reaches the preset loss value and the current training times reach the preset training times, a pre-trained spiking neural network is generated.

[0024] Optionally, calculate the gradient value for backpropagation, including:

[0025] Obtaining supervisory signals for the network;

[0026] Calculate the error based on the grayscale value and the supervision signal;

[0027] The error is determined as the gradient value of backpropagation.

[0028] Optionally, the target pulse array signal is discretized to generate target discrete signal data carrying time information, including:

[0029] The target pulse array signal is stacked into multiple pulse arrays according to preset stacking parameters to obtain multiple pulse planes;

[0030] Add the corresponding time domain information to each pulse plane to obtain multiple pulse tensors;

[0031] A plurality of pulse tensors are determined as target discrete signal data carrying time information.

[0032] Optionally, the target pulse array signal is discretized to generate target discrete signal data carrying time information, including:

[0033] Converting the target pulse array signal into a plurality of pulse time interval information;

[0034] A pixel matrix is constructed based on multiple pulse time interval information to obtain target discrete signal data carrying time information.

[0035] Optionally, the spiking neural network includes multiple feature extraction layers and dynamic information extraction layers; the dynamic information extraction layer is the last layer of the spiking neural network;

[0036] Input the target discrete signal data carrying time information into the pulse neural network, and output the grayscale value and network loss value corresponding to the target pulse array signal, including:

[0037] Perform multiple feature extractions on the target discrete signal data carrying time information according to multiple feature extraction layers to obtain initial features;

[0038] The initial features are input into the dynamic information extraction layer to obtain the spiking neuron membrane potential;

[0039] Normalize the membrane potential of each spiking neuron to a value between 0 and 1 to obtain the grayscale value corresponding to the target spiking array signal;

[0040] According to the grayscale value, the network loss value is calculated in combination with the loss function; among them,

[0041] The pulse neurons include at least an integral-release neuron, an integral-leakage-release neuron, a pulse neuron with learnable parameters, and an improved neuron that outputs continuous values of intermediate states of the above neurons.

[0042] In a second aspect, an embodiment of the present application provides a visual reconstruction device based on a spiking neural network, the device comprising:

[0043] A pulse array signal receiving module is used to receive the continuous pulse array signal to be processed in each time period from the visual sensor;

[0044] A discrete signal data generation module is used to discretize the continuous pulse array signal to be processed and generate discrete signal data carrying time information;

[0045] A visual image output module is used to input discrete signal data carrying time information into a pre-trained spiking neural network and output a visual image corresponding to the continuous pulse array signal to be processed;

[0046] The pre-trained spiking neural network is generated by cyclic training based on pulse array signals of multiple consecutive time steps. The visual image is generated based on membrane potential information, and the membrane potential information is generated by feature extraction of the continuous pulse array signal to be processed.

[0047] In a third aspect, an embodiment of the present application provides a computer storage medium, which stores a plurality of instructions suitable for being loaded by a processor and executing the above-mentioned method steps.

[0048] In a fourth aspect, an embodiment of the present application provides a terminal, which may include: a processor and a memory; wherein the memory stores a computer program, and the computer program is suitable for being loaded by the processor and executing the above-mentioned method steps.

[0049] The technical solutions provided by the embodiments of the present application may have the following beneficial effects:

[0050] In an embodiment of the present application, a visual reconstruction device based on a pulse neural network first receives a continuous pulse array signal to be processed in each time period from a visual sensor, then discretizes the signal to generate discrete signal data carrying time information, and finally inputs the discrete signal data into a pre-trained pulse neural network to output a continuous visual image, wherein the pre-trained pulse neural network is generated by cyclic training based on pulse array signals of multiple consecutive time steps, and the visual image is generated based on membrane potential information, which is generated by feature extraction of the continuous pulse array signal to be processed. Since the present application uses pulse signals based on multiple time steps and combines the pulse neural network for network training, the network can extract the membrane potential information of the pulse signal, and then restore the visual image based on the membrane potential information, thereby solving the problem of visual reconstruction of the pulse array signal.

[0051] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0053] Figure 1 This is a flow chart of a visual reconstruction method based on a spiking neural network provided in an embodiment of the present application;

[0054] Figure 2 is a network diagram of a visual reconstruction method based on a spiking neural network provided in an embodiment of the present application;

[0055] Figure 3 is a network diagram of a visual reconstruction method based on a spiking neural network provided in an embodiment of the present application;

[0056] Figure 4 This is a flow chart of a pulse neural network training method provided by an embodiment of the present application;

[0057] Figure 5 This is a schematic diagram of input data and image grayscale estimation results of a visual reconstruction method based on a spiking neural network provided in this application;

[0058] Figure 6 This is an example diagram of input data and optical flow estimation results of a visual reconstruction method based on a spiking neural network provided in this application;

[0059] Figure 7 This is an example diagram of input data and depth estimation results of a visual reconstruction method based on a spiking neural network provided in this application;

[0060] Figure 8 This is a schematic diagram of the structure of a visual reconstruction device based on a spiking neural network provided by the present application;

[0061] Figure 9 This is a schematic diagram of the structure of a terminal provided in an embodiment of the present application. DETAILED DESCRIPTION

[0062] The following description and the drawings sufficiently illustrate specific embodiments of the invention to enable those skilled in the art to practice them.

[0063] It should be understood that the embodiments described are only a portion of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without creative work are within the scope of protection of the present invention.

[0064] In the following description, when referring to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Instead, they are merely examples of devices and methods consistent with certain aspects of the present invention, as detailed in the appended claims.

[0065] In the description of the present invention, it should be understood that the terms "first", "second", etc. are used for descriptive purposes only and are not to be understood as indicating or implying relative importance. For those skilled in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances. In addition, in the description of the present invention, unless otherwise specified, "plurality" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the previous and subsequent associated objects are in an "or" relationship.

[0066] The present application provides a method, device, storage medium, and terminal for visual reconstruction based on a spiking neural network to solve the problems existing in the above-mentioned related technical problems. In the technical solution provided by the present application, since the present application uses a pulse signal based on multiple time steps and combines it with a spiking neural network for network training, the network can extract the membrane potential information of the pulse signal and then restore the visual image based on the membrane potential information, thereby solving the problem of visual reconstruction of pulse array signals. The following is a detailed description using an exemplary embodiment.

[0067] The following will be combined with the Figure 1 -Attached Figure 7 This article details the visual reconstruction method based on a spiking neural network provided in the embodiments of this application. This method can be implemented using a computer program and run on a spiking neural network-based visual reconstruction device based on a von Neumann architecture. This computer program can be integrated into an application or run as a standalone tool application.

[0068] See Figure 1 , provides a flow chart of a visual reconstruction method based on a pulse neural network for an embodiment of the present application. Figure 1 As shown, the method of the embodiment of the present application may include the following steps:

[0069] S101, receiving a continuous pulse array signal to be processed in each time period from a visual sensor;

[0070] The pulse array signal is a signal collected by a visual sensor. The visual sensor is a new type of neuromorphic sensor, which includes but is not limited to dynamic visual sensors, pulse cameras, and other visual sensors based on event or pulse representation.

[0071] In a possible implementation, when visually reconstructing a pulse array signal, it is first necessary to receive the pulse array signal in each time period from the visual sensor to obtain a continuous pulse array signal to be processed.

[0072] S102, discretizing the continuous pulse array signal to be processed to generate discrete signal data carrying time information;

[0073] Among them, the discretization processing of the pulse signal refers to converting the pulse array signal into several two-dimensional features. The conversion methods include but are not limited to: counting the number of pulses per unit time, counting the weighted value of each pulse based on time attenuation, counting the votes based on the pulse emission time, and counting the pulse emission interval based on the same pixel.

[0074] In one possible implementation, the continuous pulse array signal to be processed is first stacked into multiple pulse arrays according to preset stacking parameters to obtain multiple pulse planes, and then the corresponding time domain information is added to each pulse plane to obtain multiple pulse tensors. Finally, the multiple pulse tensors are determined as discrete signal data carrying time information.

[0075] For example, using several frames of pulse array signals, a tensor composed of pulse data is obtained as a discretized representation: R = [S1S2…S l ]

[0076] Among them, S l Represents the lth pulse plane in the pulse array data. By stacking the pulse planes as a pulse tensor, the time domain information can be preserved.

[0077] In another possible implementation, the continuous pulse array signal to be processed is first converted into a plurality of pulse time interval information, and then a pixel matrix is constructed based on the plurality of pulse time interval information to obtain discrete signal data carrying time information.

[0078] For example, convert the pulse array signal to be processed into pulse time interval information as a discretized representation:

[0079]

[0080] Among them, ISI is the pulse interval of the pixel at the current moment, which implies the time domain information; m and n are the sizes of the pixel plane.

[0081] S103, inputting discrete signal data carrying time information into a pre-trained spiking neural network, and outputting a visual image corresponding to the continuous pulse array signal to be processed;

[0082] The pre-trained spiking neural network is generated by cyclically training pulse array signals over multiple time steps. The visual image is generated based on membrane potential information, which is generated by extracting features from the processed continuous pulse array signal. The visual image includes but is not limited to grayscale information, optical flow information, depth information, spectral information, etc.

[0083] In an embodiment of the present application, when generating a pre-trained spiking neural network, a spiking neural network is first created, and then a target pulse array signal collected from a visual sensor at a first time step is received, and then the target pulse array signal is discretized to generate target discrete signal data carrying time information, and then the target discrete signal data carrying time information is input into the spiking neural network, and after updating the internal state of the neurons in the spiking neural network, the grayscale value and network loss value corresponding to the target pulse array signal are output; wherein the internal state of the neurons includes at least membrane potential, emission threshold, and membrane time constant, and finally a pre-trained spiking neural network is generated based on the network loss value.

[0084] In one possible implementation, discrete signal data carrying time information is input into a pre-trained spiking neural network, and features are extracted from the discrete signal data carrying time information by the pre-trained spiking neural network to obtain initial features. The initial features are then input into the last layer of the spiking neural network to extract the dynamic information inside the spiking neurons and normalize it into visual reconstruction values.

[0085] Specifically, the neurons in the last layer of the network are set to non-spiking neurons, which can aggregate the input information to obtain the membrane potential of the neuron. By extracting the membrane potential and performing normalization operations, the corresponding visual information can be restored:

[0086]

[0087] Where N refers to the size of the neuron membrane potential, N max Refers to the maximum value of the membrane potential of all neurons, N min Refers to the minimum value of the membrane potential of all neurons. The above formula can be used to normalize the membrane potential to the range of 0-1 to obtain grayscale information.

[0088] Specifically, the pulse neural network based on the encoder-decoder structure is as follows Figure 2As shown, the network is based on an encoder and a decoder. The network downsampling times in the figure is 3, and in a specific implementation, the downsampling times can be set to 2 or 4 or other. The network connects the encoder and decoder parts through a pulse skip layer connection.

[0089] Furthermore, the spiking neural network based on the Resnet structure, such as Figure 3 As shown in Figure 2, the number of residual blocks can be set to 4, 6, 8, or other numbers.

[0090] In an embodiment of the present application, a visual reconstruction device based on a pulse neural network first receives a continuous pulse array signal to be processed in each time period from a visual sensor, then discretizes the signal to generate discrete signal data carrying time information, and finally inputs the discrete signal data into a pre-trained pulse neural network to output a continuous visual image, wherein the pre-trained pulse neural network is generated by cyclic training based on pulse array signals of multiple consecutive time steps, and the visual image is generated based on membrane potential information, which is generated by feature extraction of the continuous pulse array signal to be processed. Since the present application uses pulse signals based on multiple time steps and combines the pulse neural network for network training, the network can extract the membrane potential information of the pulse signal, and then restore the visual image based on the membrane potential information, thereby solving the problem of visual reconstruction of the pulse array signal.

[0091] See Figure 4 , which is a flow chart of a pulse neural network training method provided in the embodiment of the present application. Figure 4 As shown, the method of the embodiment of the present application may include the following steps:

[0092] S201, creating a spiking neural network;

[0093] In the embodiments of the present application, the spiking neural network structure based on spiking neurons refers to supervised and unsupervised spiking neural networks, including but not limited to: U-net network based on spiking neurons, Resnet network based on spiking neurons, feedforward spiking neural network based on spiking neurons, and other artificial neural network image codecs whose activation functions are replaced with spiking neurons.

[0094] S202, receiving a target pulse array signal collected by a visual sensor at a first time step;

[0095] S203, discretizing the target pulse array signal to generate target discrete signal data carrying time information;

[0096] In one possible implementation, the target pulse array signal is first stacked into multiple pulse arrays according to preset stacking parameters to obtain multiple pulse planes, and then the corresponding time domain information is added to each pulse plane to obtain multiple pulse tensors. Finally, the multiple pulse tensors are determined as target discrete signal data carrying time information.

[0097] In another possible implementation, the target pulse array signal is first converted into a plurality of pulse time interval information, and then a pixel matrix is constructed based on the plurality of pulse time interval information to obtain target discrete signal data carrying time information.

[0098] S204, inputting the target discrete signal data carrying time information into the spiking neural network, updating the internal state of the neurons in the spiking neural network, and outputting the grayscale value and network loss value corresponding to the target pulse array signal;

[0099] Among them, the internal state of the neuron includes at least membrane potential, firing threshold, and membrane time constant; the pulse neural network includes multiple feature extraction layers and dynamic information extraction layers; the dynamic information extraction layer is the last layer of the pulse neural network.

[0100] In an embodiment of the present application, first, multiple feature extraction depths are performed on the target discrete signal data carrying time information according to multiple feature extraction layers to obtain initial features, and then the initial features are input into the dynamic information extraction layer to obtain the membrane potential of the spiking neuron. Secondly, the membrane potential of each spiking neuron is normalized to normalize the membrane potential to between 0 and 1 to obtain the grayscale value corresponding to the target pulse array signal. Finally, the network loss value is calculated based on the grayscale value and combined with the loss function; wherein, the spiking neurons include at least integral-release neurons, integral-leakage-release neurons, spiking neurons with learnable parameters, and improved neurons that output continuous values of intermediate states of the above neurons.

[0101] S205: Generate a pre-trained spiking neural network based on the network loss value.

[0102] In the embodiments of this application, a supervised spiking neural network is trained by comparing visual information extracted from neuron dynamics with supervisory signals during the training phase, performing backpropagation weight updates. During the inference phase, the dynamic information is used for visual reconstruction. Each new input interacts with the existing neuron state, completing a continuous training or inference process in the time domain.

[0103] In one possible implementation, when a pre-trained spiking neural network is generated based on a network loss value, first, if the network loss value does not reach a preset loss value or the current number of training times does not reach a preset number of training times, the back-propagation gradient value is calculated, and then the back-propagation gradient value is back-propagated, and the neuron weight value of the spiking neural network is updated according to the back-propagation gradient value, and the target pulse array signal collected by the visual sensor at the next time step is received, and finally, the step of discretizing the target pulse array signal is continued until the network loss value reaches the preset loss value.

[0104] In another possible implementation, when a pre-trained spiking neural network is generated based on a network loss value, if the network loss value reaches a preset loss value and the current number of training cycles reaches a preset number of training cycles, a pre-trained spiking neural network is generated.

[0105] Specifically, when calculating the gradient value of back propagation, the network's supervisory signal is first obtained, then the error is calculated based on the grayscale value and the supervisory signal, and finally the error is determined as the gradient value of back propagation.

[0106] In this embodiment, the loss functions used include l2 distance and LPIPS distance.

[0107] For example Figure 5 As shown, Figure 5 It is a schematic diagram of the input data and image grayscale estimation results of a visual reconstruction method based on a pulse neural network. The pulse data of each time period is input in a discrete form, and the reconstructed grayscale values at different moments are continuously output.

[0108] Furthermore, optical flow estimation can be achieved by training the network using data based on optical flow. Figure 6 shown.

[0109] Furthermore, depth estimation can be achieved by training the network using depth-based data. Figure 7 shown.

[0110] In an embodiment of the present application, a visual reconstruction device based on a pulse neural network first receives a continuous pulse array signal to be processed in each time period from a visual sensor, then discretizes the signal to generate discrete signal data carrying time information, and finally inputs the discrete signal data into a pre-trained pulse neural network to output a continuous visual image, wherein the pre-trained pulse neural network is generated by cyclic training based on pulse array signals of multiple consecutive time steps, and the visual image is generated based on membrane potential information, which is generated by feature extraction of the continuous pulse array signal to be processed. Since the present application uses pulse signals based on multiple time steps and combines the pulse neural network for network training, the network can extract the membrane potential information of the pulse signal, and then restore the visual image based on the membrane potential information, thereby solving the problem of visual reconstruction of the pulse array signal.

[0111] The following are embodiments of the apparatus of the present invention, which can be used to implement the method embodiments of the present invention. For details not disclosed in the apparatus embodiments of the present invention, please refer to the method embodiments of the present invention.

[0112] See Figure 8 , which shows a schematic diagram of the structure of a spiking neural network-based visual reconstruction device provided by an exemplary embodiment of the present invention. This spiking neural network-based visual reconstruction device can be implemented as all or part of a terminal through software, hardware, or a combination of both. The device 1 includes a pulse array signal receiving module 10, a discrete signal data generation module 20, and a visual image output module 30.

[0113] The pulse array signal receiving module 10 is used to receive the continuous pulse array signal to be processed in each time period from the visual sensor;

[0114] The discrete signal data generating module 20 is used to discretize the continuous pulse array signal to be processed and generate discrete signal data carrying time information;

[0115] The visual image output module 30 is used to input discrete signal data carrying time information into a pre-trained spiking neural network and output a visual image corresponding to the continuous pulse array signal to be processed;

[0116] The pre-trained spiking neural network is generated by cyclic training based on pulse array signals of multiple consecutive time steps. The visual image is generated based on membrane potential information, and the membrane potential information is generated by feature extraction of the continuous pulse array signal to be processed.

[0117] It should be noted that the visual reconstruction device based on a spiking neural network provided in the above embodiment only uses the division of the above functional modules as an example when executing the visual reconstruction method based on a spiking neural network. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the visual reconstruction device based on a spiking neural network provided in the above embodiment and the visual reconstruction method based on a spiking neural network embodiment belong to the same concept. The implementation process thereof is detailed in the method embodiment and will not be repeated here.

[0118] The serial numbers of the above embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0119] In an embodiment of the present application, a visual reconstruction device based on a pulse neural network first receives a continuous pulse array signal to be processed in each time period from a visual sensor, then discretizes the signal to generate discrete signal data carrying time information, and finally inputs the discrete signal data into a pre-trained pulse neural network to output a continuous visual image, wherein the pre-trained pulse neural network is generated by cyclic training based on pulse array signals of multiple consecutive time steps, and the visual image is generated based on membrane potential information, which is generated by feature extraction of the continuous pulse array signal to be processed. Since the present application uses pulse signals based on multiple time steps and combines the pulse neural network for network training, the network can extract the membrane potential information of the pulse signal, and then restore the visual image based on the membrane potential information, thereby solving the problem of visual reconstruction of the pulse array signal.

[0120] The present invention also provides a computer-readable medium having program instructions stored thereon, which, when executed by a processor, implement the visual reconstruction method based on pulse neural network provided by the above-mentioned various method embodiments.

[0121] The present invention also provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute the visual reconstruction method based on a pulse neural network according to each of the above method embodiments.

[0122] See Figure 9 , provides a schematic diagram of the structure of a terminal according to an embodiment of the present application. Figure 9 As shown, the terminal 1000 may include: at least one processor 1001 , at least one network interface 1004 , a user interface 1003 , a memory 1005 , and at least one communication bus 1002 .

[0123] The communication bus 1002 is used to implement the connection and communication between these components.

[0124] The user interface 1003 may include a display screen (Display) and a camera (Camera). Optionally, the user interface 1003 may also include a standard wired interface and a wireless interface.

[0125] The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a WI-FI interface).

[0126] The processor 1001 may include one or more processing cores. The processor 1001 utilizes various interfaces and circuits to connect various components within the terminal 1000. It executes instructions, programs, code sets, or instruction sets stored in the memory 1005, and calls data stored in the memory 1005 to perform various functions and process data for the terminal 1000. Optionally, the processor 1001 may be implemented using at least one hardware form selected from the group consisting of a digital signal processing (DSP), a field-programmable gate array (FPGA), and a programmable logic array (PLA). The processor 1001 may integrate one or a combination of a central processing unit (CPU), a graphics processing unit (GPU), and a modem. The CPU primarily processes the operating system, user interface, and application programs; the GPU is responsible for rendering and drawing the content displayed on the display; and the modem handles wireless communications. It is understood that the modem may not be integrated into the processor 1001 and may be implemented separately on a separate chip.

[0127] Among them, the memory 1005 may include a random access memory (RAM) or a read-only memory (Read-Only Memory). Optionally, the memory 1005 includes a non-transitory computer-readable storage medium. The memory 1005 can be used to store instructions, programs, codes, code sets or instruction sets. The memory 1005 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the above-mentioned various method embodiments, etc.; the data storage area may store data involved in the above-mentioned various method embodiments, etc. The memory 1005 may also be optionally at least one storage device located away from the aforementioned processor 1001. As Figure 9 As shown, the memory 1005 as a computer storage medium may include an operating system, a network communication module, a user interface module, and a visual reconstruction application based on a pulse neural network.

[0128] exist Figure 9 In the terminal 1000 shown, the user interface 1003 is mainly used to provide an input interface for the user and obtain user input data; and the processor 1001 can be used to call the visual reconstruction application based on the pulse neural network stored in the memory 1005 and specifically perform the following operations:

[0129] Receiving a continuous pulse array signal to be processed in each time period from a visual sensor;

[0130] Discretize the continuous pulse array signal to be processed to generate discrete signal data carrying time information;

[0131] The discrete signal data carrying time information is input into the pre-trained pulse neural network, and the visual image corresponding to the continuous pulse array signal to be processed is output; wherein,

[0132] The pre-trained spiking neural network is generated by cyclic training based on pulse array signals of multiple consecutive time steps. The visual image is generated based on membrane potential information, and the membrane potential information is generated by feature extraction of the continuous pulse array signal to be processed.

[0133] In one embodiment, the processor 1001 further performs the following operations:

[0134] Create spiking neural networks;

[0135] receiving a target pulse array signal collected by a vision sensor at a first time step;

[0136] Discretize the target pulse array signal to generate target discrete signal data carrying time information;

[0137] Input the target discrete signal data carrying time information into the spiking neural network, update the internal state of the neurons in the spiking neural network, and then output the grayscale value and network loss value corresponding to the target pulse array signal;

[0138] in,

[0139] The internal state of a neuron includes at least membrane potential, firing threshold, and membrane time constant;

[0140] Generate a pre-trained spiking neural network based on the network loss value.

[0141] In one embodiment, when the processor 1001 generates a pre-trained spiking neural network according to the network loss value, the processor 1001 specifically performs the following operations:

[0142] If the network loss value does not reach the preset loss value or the current training times do not reach the preset training times, the gradient value of back propagation is calculated;

[0143] Back-propagating the gradient value of back propagation, updating the neuron weight value of the pulse neural network according to the gradient value of back propagation, and receiving the target pulse array signal collected by the visual sensor at the next time step;

[0144] Continue to perform the step of discretizing the target pulse array signal until the network loss value reaches a preset loss value;

[0145] or,

[0146] If the network loss value reaches the preset loss value and the current training times reach the preset training times, a pre-trained spiking neural network is generated.

[0147] In one embodiment, when calculating the gradient value of back propagation, the processor 1001 specifically performs the following operations:

[0148] Obtaining supervisory signals for the network;

[0149] Calculate the error based on the grayscale value and the supervision signal;

[0150] The error is determined as the gradient value of backpropagation.

[0151] In one embodiment, when the processor 1001 discretizes the target pulse array signal to generate target discrete signal data carrying time information, the processor 1001 specifically performs the following operations:

[0152] The target pulse array signal is stacked into multiple pulse arrays according to preset stacking parameters to obtain multiple pulse planes;

[0153] Add the corresponding time domain information to each pulse plane to obtain multiple pulse tensors;

[0154] A plurality of pulse tensors are determined as target discrete signal data carrying time information.

[0155] In one embodiment, when the processor 1001 discretizes the target pulse array signal to generate target discrete signal data carrying time information, the processor 1001 specifically performs the following operations:

[0156] Converting the target pulse array signal into a plurality of pulse time interval information;

[0157] A pixel matrix is constructed based on multiple pulse time interval information to obtain target discrete signal data carrying time information.

[0158] In one embodiment, when the processor 1001 inputs the target discrete signal data carrying time information into the spiking neural network and outputs the grayscale value and network loss value corresponding to the target pulse array signal, the processor 1001 specifically performs the following operations:

[0159] Perform multiple feature extractions on the target discrete signal data carrying time information according to multiple feature extraction layers to obtain initial features;

[0160] The initial features are input into the dynamic information extraction layer to obtain the spiking neuron membrane potential;

[0161] Normalize the membrane potential of each spiking neuron to a value between 0 and 1 to obtain the grayscale value corresponding to the target spiking array signal;

[0162] According to the grayscale value, the network loss value is calculated in combination with the loss function; among them,

[0163] The pulse neurons include at least an integral-release neuron, an integral-leakage-release neuron, a pulse neuron with learnable parameters, and an improved neuron that outputs continuous values of intermediate states of the above neurons.

[0164] In an embodiment of the present application, a visual reconstruction device based on a pulse neural network first receives a continuous pulse array signal to be processed in each time period from a visual sensor, then discretizes the signal to generate discrete signal data carrying time information, and finally inputs the discrete signal data into a pre-trained pulse neural network to output a continuous visual image, wherein the pre-trained pulse neural network is generated by cyclic training based on pulse array signals of multiple consecutive time steps, and the visual image is generated based on membrane potential information, which is generated by feature extraction of the continuous pulse array signal to be processed. Since the present application uses pulse signals based on multiple time steps and combines the pulse neural network for network training, the network can extract the membrane potential information of the pulse signal, and then restore the visual image based on the membrane potential information, thereby solving the problem of visual reconstruction of the pulse array signal.

[0165] Those skilled in the art will appreciate that all or part of the processes in the above-described method embodiments can be implemented by instructing the relevant hardware through a computer program. The program for visual reconstruction based on a spiking neural network can be stored in a computer-readable storage medium. When executed, the program can include the processes in the above-described method embodiments. The storage medium can be a magnetic disk, an optical disk, a read-only memory, or a random access memory.

[0166] The above disclosure is only a preferred embodiment of the present application, and certainly cannot be used to limit the scope of rights of the present application. Therefore, equivalent changes made according to the claims of the present application are still within the scope covered by the present application.

Claims

1. A visual reconstruction method based on a spiking neural network, characterized in that: The method comprises: Receiving a continuous pulse array signal to be processed in each time period from a visual sensor; Discretize the continuous pulse array signal to be processed to generate discrete signal data carrying time information; Inputting discrete signal data carrying time information into a pre-trained pulse neural network, and outputting a visual image corresponding to the continuous pulse array signal to be processed; wherein, The pre-trained spiking neural network is generated by cyclic training based on a pulse array signal of multiple consecutive time steps, and the visual image is generated based on membrane potential information, which is generated by feature extraction of the continuous pulse array signal to be processed; wherein, Follow these steps to generate a pre-trained spiking neural network, including: Create spiking neural networks; receiving a target pulse array signal collected by a vision sensor at a first time step; Discretizing the target pulse array signal to generate target discrete signal data carrying time information; Inputting the target discrete signal data carrying time information into the spiking neural network, updating the internal state of the neurons in the spiking neural network, and outputting the grayscale value and network loss value corresponding to the target pulse array signal; wherein, The internal state of the neuron includes at least membrane potential, firing threshold, and membrane time constant; A pre-trained spiking neural network is generated according to the network loss value.

2. The method according to claim 1, characterized in that Generating a pre-trained spiking neural network according to the network loss value includes: If the network loss value does not reach the preset loss value or the current training number does not reach the preset training number, calculate the gradient value of back propagation; Back-propagating the back-propagated gradient value, updating the neuron weight value of the spiking neural network according to the back-propagated gradient value, and receiving the target pulse array signal collected by the visual sensor at the next time step; Continuing to perform the step of discretizing the target pulse array signal until the network loss value reaches a preset loss value; or, If the network loss value reaches a preset loss value and the current training times reaches a preset training times, a pre-trained spiking neural network is generated.

3. The method according to claim 2, characterized in that The calculation of the gradient value of the back propagation includes: Obtaining supervisory signals for the network; Calculating an error based on the grayscale value and the supervisory signal; The error is determined as the gradient value of the back propagation.

4. The method according to claim 1, wherein The discretizing the target pulse array signal to generate target discrete signal data carrying time information includes: The target pulse array signal is stacked into multiple pulse arrays according to preset stacking parameters to obtain multiple pulse planes; Add the corresponding time domain information to each pulse plane to obtain multiple pulse tensors; The plurality of pulse tensors are determined as target discrete signal data carrying time information.

5. The method according to claim 1, wherein The discretizing the target pulse array signal to generate target discrete signal data carrying time information includes: Converting the target pulse array signal into a plurality of pulse time interval information; A pixel matrix is constructed based on the multiple pulse time interval information to obtain target discrete signal data carrying time information.

6. The method according to claim 1, characterized in that The spiking neural network includes multiple feature extraction layers and a dynamic information extraction layer; the dynamic information extraction layer is the last layer of the spiking neural network; The step of inputting the target discrete signal data carrying time information into the pulse neural network and outputting the grayscale value and network loss value corresponding to the target pulse array signal comprises: Performing multiple feature depth extractions on the target discrete signal data carrying time information according to the multiple feature extraction layers to obtain initial features; Inputting the initial features into the dynamic information extraction layer to obtain the spiking neuron membrane potential; Normalizing the membrane potential of each of the spiking neurons to a value between 0 and 1 to obtain a grayscale value corresponding to the target pulse array signal; According to the grayscale value, the network loss value is calculated in combination with the loss function; wherein, The pulse neurons at least include an integral-release neuron, an integral-leakage-release neuron, a pulse neuron with learnable parameters, and an improved neuron that outputs continuous values of intermediate states of the above neurons.

7. A visual reconstruction device based on a spiking neural network implemented using the method according to any one of claims 1 to 6, characterized in that: The device comprises: A pulse array signal receiving module is used to receive the continuous pulse array signal to be processed in each time period from the visual sensor; A discrete signal data generating module, configured to discretize the continuous pulse array signal to be processed to generate discrete signal data carrying time information; A visual image output module is used to input discrete signal data carrying time information into a pre-trained spiking neural network and output a visual image corresponding to the continuous pulse array signal to be processed; The pre-trained pulse neural network is generated by cyclic training based on pulse array signals of multiple consecutive time steps, and the visual image is generated based on membrane potential information, which is generated by feature extraction of the continuous pulse array signal to be processed.

8. A computer storage medium, characterized in that The computer storage medium stores a plurality of instructions, and the instructions are suitable for being loaded by a processor and executing the method according to any one of claims 1 to 6.

9. A terminal, characterized in that: include: A processor and a memory; wherein the memory stores a computer program, and the computer program is suitable for being loaded by the processor and executing the method according to any one of claims 1 to 6.

Images