Pulse neural network training method and image segmentation method

Abstract

The invention provides a training method of a spiking neural network and an image segmentation method. The method comprises the steps of inputting a sample pulse sequence feature of a sample image into a pulse neural network model, and obtaining a pulse time difference of generating a pre-synaptic pulse and a post-synaptic pulse on the basis of a condition that a leakage integral distribution neuron in the pulse neural network model responds to the sample pulse sequence feature and generates a pulse, the pulse neural network model has a current synaptic weight, and the current synaptic weight is used for processing the pre-synaptic pulse to obtain a post-synaptic pulse; a weight change amount corresponding to the pulse time difference is calculated through a weight updating formula, and the weight updating formula is obtained by fitting a pulse time sequence dependence plasticity formula based on sample pulse time sequence dependence plasticity data; and adjusting the current synaptic weight by using the weight change amount so as to train the pulse neural network model.

Description

Technical Field

[0001] The present disclosure relates to the fields of artificial intelligence and neuromorphic computing, and more particularly, to a training method for a spiking neural network and an image segmentation method. Background Art

[0002] With the rapid development of deep learning, CNN (Convolutional Neural Networks) has performed excellently in visual tasks such as semantic segmentation and object detection. However, its high energy consumption and dependence on computing resources limit its application in embedded and mobile devices. In addition, ANN (Artificial Neural Network) has deficiencies in terms of biological interpretability and low power consumption.

[0003] As the third-generation neural network, SNN (Spiking Neuron Networks) has characteristics such as biological interpretability, event-driven, and low power consumption, simulating the behavior of biological neurons, achieving efficient computing through event-driven, and having the advantages of low power consumption and biological interpretability. However, it is difficult to calculate the gradient of SNN, resulting in a complex training process and difficult to achieve effective weight optimization. Summary of the Invention

[0004] In view of this, the present disclosure provides a training method for a spiking neural network and an image segmentation method.

[0005] One aspect of the present disclosure provides a training method for a spiking neural network, including: inputting the sample pulse sequence features of a sample image into a spiking neural network model, and when generating pulses based on the response of the leaky integrate-and-fire neurons in the spiking neural network model to the sample pulse sequence features, obtaining the pulse time difference between the generated pre-synaptic pulses and post-synaptic pulses, where the spiking neural network model has a current synaptic weight, and the current synaptic weight is used to process the pre-synaptic pulses to obtain post-synaptic pulses; calculating the weight change amount corresponding to the pulse time difference by using a weight update formula, where the weight update formula is obtained by fitting the spike timing-dependent plasticity formula based on sample spike timing-dependent plasticity data; and adjusting the current synaptic weight by using the weight change amount to train the spiking neural network model.

[0006] Another aspect of the present disclosure provides an image segmentation method, including: obtaining an image to be segmented; and inputting the pulse sequence image of the image to be segmented into a spiking neural network to generate an image segmentation result, where the spiking neural network is trained by using the training method for a spiking neural network based on the present disclosure. Description of the Drawings

[0007] Through the following description of the embodiments of the present disclosure with reference to the accompanying drawings, the above and other objects, features, and advantages of the present disclosure will become clearer. In the drawings:

[0008] Figure 1 Schematically shows an exemplary system architecture that can apply at least one of the training method and the image segmentation method of the spiking neural network according to an embodiment of the present disclosure;

[0009] Figure 2 Schematically shows a flowchart of the training method of the spiking neural network according to an embodiment of the present disclosure;

[0010] Figure 3 Schematically shows a relationship diagram between the spike firing times of the pre- and post-synaptic neurons and the weight update according to an embodiment of the present disclosure;

[0011] Figure 4 Schematically shows an STDP curve and a fitting result diagram of a memristor according to an embodiment of the present disclosure;

[0012] Figure 5 Schematically shows an embedded structure diagram of the STDP class in the SNN DeepLab network according to an embodiment of the present disclosure;

[0013] Figure 6 Schematically shows a method of applying the memristor spike-timing-dependent plasticity to the SNN DeepLab network according to an embodiment of the present disclosure;

[0014] Figure 7 Schematically shows a flowchart of the image segmentation method according to an embodiment of the present disclosure;

[0015] Figure 8 Schematically shows a block diagram of a training device for a spiking neural network according to an embodiment of the present disclosure;

[0016] Figure 9 Schematically shows a block diagram of an image segmentation device according to an embodiment of the present disclosure; and

[0017] Figure 10 Schematically shows a block diagram of an electronic device suitable for implementing at least one of the training method and the image segmentation method of the spiking neural network according to an embodiment of the present disclosure. Detailed Embodiments

[0018] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are merely exemplary and are not intended to limit the scope of the present disclosure. In the following detailed description, for the sake of explanation, numerous specific details are set forth to provide a thorough understanding of the embodiments of the present disclosure. However, obviously, one or more embodiments can also be implemented without these specific details. In addition, in the following description, descriptions of well-known structures and technologies are omitted to avoid unnecessarily confusing the concepts of the present disclosure.

[0019] The terms used herein are merely for describing specific embodiments and are not intended to limit the present disclosure. The terms "including", "comprising", etc. used herein indicate the presence of the described features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0020] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0021] In the case of using expressions such as "at least one of A, B, and C, etc.", generally, it should be interpreted according to the meaning commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include, but is not limited to, a system having only A, only B, only C, having A and B, having A and C, having B and C, and / or having A, B, and C, etc.).

[0022] In the embodiments of the present disclosure, in terms of the collection, update, analysis, processing, use, transmission, provision, disclosure, storage, etc. of the involved data (for example, including but not limited to user personal information), they all comply with the provisions of relevant laws and regulations, are used for legal purposes, and do not violate public order and good customs. In particular, necessary measures are taken for user personal information to prevent illegal access to user personal information data and to safeguard the security of user personal information, network security, and national security.

[0023] Figure 1 An exemplary system architecture 100 is schematically shown that can apply at least one of the training method and the image segmentation method of a spiking neural network according to an embodiment of the present disclosure. It should be noted that Figure 1 What is shown is only an example of the system architecture to which the embodiments of the present disclosure can be applied, to help those skilled in the art understand the technical content of the present disclosure, but it does not mean that the embodiments of the present disclosure cannot be used in other devices, systems, environments, or scenarios.

[0024] As Figure 1As shown, the system architecture 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 is used to provide a medium for communication links between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired and / or wireless communication links, etc.

[0025] Users can use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 through the network 104 to receive or send messages, etc. Various communication client applications may be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, and / or social platform software, etc. (only as examples).

[0026] The first terminal device 101, the second terminal device 102, and the third terminal device 103 may be various electronic devices with a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop portable computers, and desktop computers, etc.

[0027] The server 105 may be a server that provides various services, such as a background management server that supports the websites browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103 (only as an example). The background management server may analyze and process data such as received user requests, and feedback the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal device.

[0028] It should be noted that at least one of the training method of the spiking neural network and the image segmentation method provided by the embodiments of the present disclosure can generally be executed by the server 105. Correspondingly, at least one of the training device of the spiking neural network and the image segmentation device provided by the embodiments of the present disclosure can generally be arranged in the server 105. At least one of the training method of the spiking neural network and the image segmentation method provided by the embodiments of the present disclosure can also be executed by a server or a server cluster different from the server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or the server 105. Correspondingly, at least one of the training device of the spiking neural network and the image segmentation device provided by the embodiments of the present disclosure can also be arranged in a server or a server cluster different from the server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or the server 105. Alternatively, at least one of the training method of the spiking neural network and the image segmentation method provided by the embodiments of the present disclosure can also be executed by the first terminal device 101, the second terminal device 102, or the third terminal device 103, or can also be executed by other terminal devices different from the first terminal device 101, the second terminal device 102, or the third terminal device 103. Correspondingly, at least one of the training device of the spiking neural network and the image segmentation device provided by the embodiments of the present disclosure can also be arranged in the first terminal device 101, the second terminal device 102, or the third terminal device 103, or arranged in other terminal devices different from the first terminal device 101, the second terminal device 102, or the third terminal device 103.

[0029] For example, the sample image or the image to be segmented can be originally stored in any one of the first terminal device 101, the second terminal device 102, or the third terminal device 103 (for example, the first terminal device 101, but not limited thereto), or stored on an external storage device and can be imported into the first terminal device 101. Then, the first terminal device 101 can execute the training method of the spiking neural network or the image segmentation method provided by the embodiments of the present disclosure locally, or send the sample image or the image to be segmented to other terminal devices, servers, or server clusters, and the other terminal devices, servers, or server clusters that receive the sample image or the image to be segmented execute the training method of the spiking neural network or the image segmentation method provided by the embodiments of the present disclosure.

[0030] It should be understood that Figure 1 the numbers of terminal devices, networks, and servers in

[0031] Figure 2The flowchart of the training method of the spiking neural network according to an embodiment of the present disclosure is schematically shown.

[0032] As Figure 2 shown, the method includes operations S201 to S203.

[0033] In operation S201, the sample pulse sequence feature of the sample image is input into the spiking neural network model. When the spiking neural network model generates a pulse in response to the sample pulse sequence feature based on the leaky integrate-and-fire neurons in the model, the pulse time difference between the generated presynaptic pulse and postsynaptic pulse is obtained. Among them, the spiking neural network model has a current synaptic weight, and the current synaptic weight is used to process the presynaptic pulse to obtain the postsynaptic pulse.

[0034] According to an embodiment of the present disclosure, the spiking neural network uses LIF (Leaky Integrate and Fire) neurons to simulate the electrophysiological behavior of biological neurons. The core mechanisms of LIF neurons include: membrane potential integration, leakage (Leaky), and firing a pulse (Fire). Among them, membrane potential integration: receiving input current and gradually accumulating membrane potential. Leakage: The membrane potential decays exponentially over time, similar to the ion leakage of biological neurons. Firing a pulse: When the membrane potential exceeds the threshold, a pulse (Spike) is emitted, and then the potential is reset. The membrane potential update formula of LIF neurons can be characterized as formula (1).

[0035] Formula (1)

[0036] When the membrane potential V of the LIF neuron reaches the threshold V th , the LIF neuron will generate a pulse and reset the membrane potential to the initial value V reset . The initial condition is: the membrane potential V(0) is usually initialized to V reset . represents the membrane potential, characterizing the potential state of the neuron at time t. represents the time constant, which determines the decay rate of the membrane potential. represents the input current, which is obtained by weighting the presynaptic pulse of the presynaptic neuron by the weight w. A synapse is the connection structure between neurons, responsible for transmitting signals and weighting the input. In the code, a synapse usually appears as a weight matrix. After the input signal is weighted by the synaptic weight, it is transmitted to the next layer of neurons.

[0037] According to an embodiment of the present disclosure, by real-time monitoring and recording the pulse firing moment of the presynaptic neuron and the pulse firing moment of the postsynaptic neuron , the pulse time difference can be calculated.

[0038] In operation S202, a weight change amount corresponding to the pulse time difference is calculated using a weight update formula, where the weight update formula is obtained by fitting the spike-timing dependent plasticity formula based on sample spike timing-dependent plasticity data.

[0039] According to an embodiment of the present disclosure, the weight update formula may represent the functional relationship of the weight change amount with respect to the pulse time difference. The specific implementation manner of this functional relationship may not be limited in the present disclosure.

[0040] According to an embodiment of the present disclosure, STDP (Spike-Timing Dependent Plasticity) is a synaptic plasticity rule based on spike timing dependence for adjusting synaptic weights. Its core rule is: if the presynaptic spike is earlier than the postsynaptic spike, the synaptic weight is enhanced (long-term potentiation, LTP for short). If the presynaptic spike is later than the postsynaptic spike, the synaptic weight is weakened (long-term depression, LTD for short). The learning mechanism of STDP is for adjusting the weights of synaptic connections.

[0041] According to an embodiment of the present disclosure, a spiking neural network may include one or more LIF neurons. For the synaptic connection between every two LIF neurons in the spiking neural network, STDP can adjust the corresponding synaptic weight according to the spike time difference between the presynaptic spike and the postsynaptic spike. For the case where there is only one LIF neuron in the spiking neural network, as long as there is a self-connected synapse in the spiking neural network, that is, the input and output of the synapse both come from the same neuron, STDP can still adjust the synaptic weight of its self-connected synapse. Specifically, in the case where the single LIF neuron outputs a spike, it can be used as the presynaptic neuron, and the output spike is the presynaptic spike, and the first timestamp of the presynaptic spike can be recorded; in the case where the single LIF neuron receives the input of the self-connected synapse and generates a response, it can be used as the postsynaptic neuron, and the spike generated by the response output is the postsynaptic spike, and the second timestamp of the postsynaptic spike can be recorded, so that STDP can adjust the synaptic weight of its self-connected synapse according to the spike time difference and the fitted weight update formula.

[0042] In operation S203, the current synaptic weight is adjusted using the weight change amount to train the spiking neural network model.

[0043] According to an embodiment of the present disclosure, based on the above operations S201 to S203, an STDP rule can be determined. Based on this STDP rule, during the forward propagation process of the spiking neural network, the input sample spike train features pass through each layer of the network, and the membrane potential and spike firing state of each neuron are calculated. Then, spike firing recording and weight update are realized.

[0044] Figure 3 Schematically shows a relationship diagram of the spike firing times of pre- and post-synaptic neurons and weight update according to an embodiment of the present disclosure.

[0045] As Figure 3 shown, it shows the specific application of the STDP rule 300. In the STDP rule 300, record_pre_spike_times can be used to record the spike firing times of pre-synaptic neurons, record_post_spike_times can be used to record the spike firing times of post-synaptic neurons, calculate_delta_w can be used to calculate the weight change amount, and update_weights can be used to update the weights.

[0046] Based on the STDP rule 300, corresponding to the above operation S201 and its embodiments, a pulse detection module 310 can be constructed to record the pulse time difference between pre-synaptic and post-synaptic pulses. Corresponding to the above operation S202 and its embodiments, a weight calculation module 320 can be constructed to calculate the weight change amount according to the recorded pulse time difference and the weight update formula. Corresponding to the above operation S203, a weight update module 330 can be constructed to adjust the current synaptic weight. On this basis, based on the functions of the pulse detection module 310, the weight calculation module 320, and the weight update module 330 respectively, an STDP class can be constructed on the basis of the foregoing STDP rule 300. The STDP class can have the overall functions of the foregoing pulse detection module 310, weight calculation module 320, and weight update module 330.

[0047] According to an embodiment of the present disclosure, the above STDP class can be defined and embedded in the pulse network model for training, so that the STDP class records the spike firing times of pre- and post-synaptic neurons at each time step, and can calculate the weight change amount according to the STDP rule. Update the synaptic weight. Thus, the pulse network model can dynamically adjust the synaptic weight at the corresponding position according to the pulse time difference, which is beneficial to realizing the dynamic adjustment of the weights of complex network structures.

[0048] Through the above embodiments of the present disclosure, an alternative gradient method implemented by adjusting the synaptic weight by calculating the weight change amount based on the weight update formula and the pulse time difference can successfully alleviate the problem of difficult gradient calculation in SNN training, ensure the effective training of the deep SNN model, and promote the development of SNN in the field of deep learning. Other layers of the spiking neural network can still be trained using the gradient descent method.

[0049] The following will further illustrate the Figure 2 method shown with specific embodiments.

[0050] In the process of implementing the concept of the present disclosure, the inventors also found that most STDP simulations are limited to the software level and do not fully utilize the hardware characteristics to achieve more efficient neural computing, resulting in limitations in hardware implementation.

[0051] According to an embodiment of the present disclosure, a memristor is a novel electronic device that can simulate the plasticity characteristics of biological synapses, especially having hardware advantages in the STDP mechanism. Therefore, combining memristors with SNNs is a potential path to achieve efficient neural computing.

[0052] According to an embodiment of the present disclosure, before performing the above operation S202, sample STDP data can be first constructed based on the STDP characteristics of the memristor to fit the weight update formula. Constructing sample STDP data based on the STDP characteristics of the memristor may include: obtaining a pulse combination sequence determined based on the pulse combination of the pre-synaptic pulse and the post-synaptic pulse, where the pulse combination sequence includes multiple pulse combinations, and different pulse combinations have different positive time differences or negative time differences; when sequentially applying multiple pulse combinations to the memristor, recording the positive memristor conductance information corresponding to the positive time difference and the negative memristor conductance information corresponding to the negative time difference; constructing a positive sample based on the corresponding positive time difference and positive memristor conductance information; constructing a negative sample based on the corresponding negative time difference and negative memristor conductance information; and determining sample pulse-timing-dependent plasticity data based on the positive sample and the negative sample.

[0053] For example, the STDP characteristics of the memristor can be first experimentally measured, and a series of combinations of pre-synaptic pulses and post-synaptic pulses can be designed accordingly, covering different pulse time differences Δt, including positive time differences (pre-synaptic pulse arrives first) and negative time differences (post-synaptic pulse arrives first). Then, data recording is performed, and for each pulse combination sequence, the conductance change Δw of the memristor is measured and recorded.

[0054] Specifically, by applying a specific pulse sequence to the memristor and measuring the relationship between its conductance value and the pulse time difference (Δt) between the pre-synaptic and post-synaptic neurons, a set of STDP data of the memristor can be experimentally obtained: positive Δt, negative Δt, and the corresponding ΔG values.

[0055] Figure 4 Schematically shows the STDP curve and the fitting result graph of a memristor according to an embodiment of the present disclosure.

[0056] As Figure 4 shown, shows the relationship between the conductance change ΔG and the pulse time difference Δt.

[0057] Preferably, for example, the following experimental data can be selected as the sample STDP data obtained based on the memristor: (10, 24.49036), (-10, -25.53967), (20, 19.34483), (-20, -22.29258), (30, 18.83779), (-30, -19.7499), (50, 17.61693), (-50, -16.6894), (100, 16.19264), (-100, -14.72984), (200, 15.10674), (-200, -13.39692), (500, 13.64198), (-500, -12.22432), (1000, 12.76475), (-1000, -11.45604), (5000, 12.79409), (-5000, -11.53491), (10000, 12.58153), (-10000, -11.12614), etc., and is not limited thereto.

[0058] According to an embodiment of the present disclosure, the fitting of the pulse-timing-dependent plasticity formula based on the sample pulse-timing-dependent plasticity data in the above operation S202 may include: using the positive samples to perform a first fitting on the pulse-timing-dependent plasticity formula to determine the positive amplitude in the pulse-timing-dependent plasticity formula and the positive time constant corresponding to the positive amplitude; using the negative samples to perform a second fitting on the pulse-timing-dependent plasticity formula to determine the negative amplitude in the pulse-timing-dependent plasticity formula and the negative time constant corresponding to the negative amplitude; and determining the pulse-timing-dependent plasticity formula after the first fitting and the second fitting as the weight update formula.

[0059] For example, a double-exponential function can be used to fit the experimental data to obtain the weight update formula shown in formula (2), which accurately describes the relationship between the conductance change and the pulse time difference.

[0060] Formula (2)

[0061] Wherein, represents the weight change amount of the memristor under the action of two pulses, represents the positive time constant, represents the negative time constant, A represents the positive amplitude, and B represents the negative amplitude. Equation (2) is implemented in hardware by simulating the conductance change of the memristor to achieve pulse clock-dependent plasticity.

[0062] By fitting the above experimental data, for example, the fitting results can be obtained: A = 17.4736, = 21.44, B = 17.5946, = 15.2463.

[0063] Through the above embodiments of the present disclosure, different from simulating synaptic plasticity at the software level, by combining the device characteristics of the memristor with the pulsed neural network and utilizing the physical characteristics of the memristor and the STDP mechanism, the dynamic and physically adjustable synaptic weights at the hardware-like level are realized, breaking through the limitations of traditional methods, and constructing a semantic segmentation model with biologic interpretability, low energy consumption, and high performance.

[0064] According to the embodiments of the present disclosure, for the obtained weight update formula by fitting, the initialization of the STDP synaptic weight update parameters can be set first. For example, it can include setting: w min = -1.0, w max = 1.0, tau pre = 2.0, tau post = 2.0, N in = 4, N out = 3, T = 128, and defining the fitted experimental data. Among them, w min represents the minimum value of the synaptic weight to prevent the weight from being too small. w max represents the maximum value of the synaptic weight to prevent the weight from being too large. tau pre represents the presynaptic time constant, which is used to control the influence degree of the presynaptic pulse on the weight update. tau post represents the postsynaptic time constant, which is used to control the influence degree of the postsynaptic pulse on the weight update. N in represents the number of input neurons, which can characterize the number of neurons in the input layer of the network. N out represents the number of output neurons, which can characterize the number of neurons in the output layer of the network. T represents the time step, which can be used to define the time length of the simulation or training.

[0065] According to the embodiments of the present disclosure, based on the weight update formula obtained by the above fitting, an STDP class can be constructed to support the update and limitation of the weights. Corresponding to the weight calculation module part in the STDP class, by calculating , according to the positive or negative of Δt, the corresponding formula in the weight update formula can be selected to calculate the weight change amount 。Corresponding to the weight update module part in the STDP class, the adjustment method of the current synaptic weight can be controlled in combination with the above initialization configuration.

[0066] According to an embodiment of the present disclosure, the above operation S203 may include: calculating the sum of the weight change amount and the current synaptic weight to obtain an updated synaptic weight; performing a first normalization process on the updated synaptic weight to obtain a normalized weight, where the normalized weight is within a preset weight range; and replacing the current synaptic weight with the normalized weight to achieve the adjustment of the current synaptic weight.

[0067] For example, after calculating , the current synaptic weight can be updated ,, and then clip w jk =clip(w jk , w min , w max ), to ensure that the adjusted synaptic weight is between the preset minimum value w min and the maximum value w max .

[0068] According to an embodiment of the present disclosure, before performing the foregoing operation S201, the sample highlight may be preprocessed first to obtain a sample pulse sequence feature. The method may include: performing a second normalization process on the sample image to obtain a normalized image, where the image scale of the normalized image satisfies a preset scale; using the pixel value of the normalized image as a Poisson distribution parameter; for each pixel in the normalized image, within each time step, determining the pulse generation probability of the pixel within the time step according to the time step and the Poisson distribution parameter; and generating a sample pulse sequence feature corresponding to the sample image according to the pulse generation probability.

[0069] For example, in the case of obtaining a sample image, a second normalization, that is, image normalization, can be performed to standardize the pixel values of the sample image and eliminate the scale difference between different images. Specifically, the implementation method may include: for each pixel value I(x,y), using the formula I′(x,y)=(I(x,y)-I min ) / (I max -I min ). Where I min represents the minimum pixel value of the image, I maxRepresents the maximum pixel value of the image. For example, through the second normalization process, the pixel values of the sample image can be scaled to the interval (0, 1). Then, combined with Poisson random pulse coding, the normalized image pixel values can be converted into a Poisson random pulse sequence to generate a sample pulse sequence, and the pulse firing rate is proportional to the pixel value. Specifically, the implementation method can include: using the normalized pixel value I′(x,y) as the Poisson distribution parameter λ(x,y). For each pixel in the second-normalized sample image, at each time step h, determine whether to generate a pulse with a probability P = λ(x,y)×h. Thus, the sample pulse sequence characteristics can be determined according to the pulse generation result.

[0070] According to an embodiment of the present disclosure, generating the sample pulse sequence characteristics corresponding to the sample image according to the pulse generation probability may include: generating a Poisson random pulse sequence corresponding to the sample image according to the pulse generation probability; performing one-hot encoding on the Poisson random pulse sequence according to the class label of the sample image to obtain a sample pulse sequence vector; and determining the sample pulse sequence characteristics according to the sample pulse sequence vector.

[0071] According to an embodiment of the present disclosure, when the spiking neural network only includes two network layer structures, namely an encoding layer and a decoding layer, the Poisson random pulse sequence can be determined as the sample pulse sequence characteristics for the spiking neural network to process. It is also possible to perform label encoding on the Poisson random pulse sequence, and determine the encoding result after converting the Poisson random pulse sequence into a format suitable for network processing according to the class label of semantic segmentation as the sample pulse sequence characteristics for the spiking neural network to process, and it is not limited thereto.

[0072] For example, one-hot encoding can be performed on the class label of each pixel to generate a C-dimensional vector suitable for SNN processing. Then, the C-dimensional vector can be determined as the sample pulse sequence characteristics. Where C is the number of label categories.

[0073] According to an embodiment of the present disclosure, when the spiking neural network includes multiple network layer structures such as an encoding layer, a convolutional layer, a pooling layer, a fully connected layer, and a decoding layer, STDP classes can be embedded into at least one of the network layers such as the convolutional layer, the pooling layer, the fully connected layer, and the decoding layer.

[0074] For example, the SNN can be applied to complex semantic segmentation tasks to construct an SNN DeepLab (a semantic segmentation model) network structure based on the LIF model to fill the gap of the SNN in semantic segmentation tasks.

[0075] Figure 5 Schematically shows the embedding structure diagram of STDP classes in the SNN DeepLab network according to an embodiment of the present disclosure.

[0076] As shown Figure 5 in the figure, it is shown that the STDP class can be applied to the convolutional layer and the decoding layer of the network to realize STDP update of the weights of the corresponding layers. The specific implementation is as follows: in the forward propagation process, the pulse detection module of the STDP class records the pulse emission time, and the weight calculation module and the weight update module adjust the weights at appropriate times.

[0077] Corresponding to this scenario, the output features of each network layer can be determined as the above sample pulse sequence features.

[0078] According to an embodiment of the present disclosure, determining the sample pulse sequence features according to the sample pulse sequence vector may further include: performing first feature extraction on the sample pulse sequence vector to obtain a sample pulse feature map; when generating an intermediate pulse sequence based on a leaky integrate-and-fire neuron in response to the sample pulse feature map, obtaining the intermediate pulse sequence; and performing second feature extraction on the intermediate pulse sequence to obtain the sample pulse sequence features.

[0079] For example, in a spiking neural network such as the SNN DeepLab network structure, the encoding layer can receive a Poisson random pulse sequence as the input of the network. The pulse convolutional layer can extract the spatial features of the Poisson random pulse sequence, perform convolution operations using a convolution kernel, embed the LIF neuron model, and output a first feature map. The pooling layer can reduce the size of the first feature map, adopt pulse max pooling or average pooling methods to increase the receptive field, and obtain a second feature map. The decoding layer can restore the size of the second feature map, use deconvolution or upsampling operations to achieve pixel-level prediction, and obtain a prediction feature. Thus, the output layer can use pulse frequency or frequency representation, calculate the class probability of each pixel by accumulating the number of pulses, and complete the semantic segmentation task. In this embodiment, the first feature map, the second feature map, and the prediction feature can all be determined as the above sample pulse sequence features.

[0080] According to an embodiment of the present disclosure, due to the discontinuity of the pulse emission function, direct differentiation is not feasible. A differentiable alternative function (such as the Sigmoid function, the linear approximation function) can be used to approximate the gradient of the pulse emission function to solve the gradient calculation problem in SNN training.

[0081] During the training process, on the basis of recording the pulse emission times of the pre- and post-synaptic neurons based on the STDP class, calculating the conductance update amount according to the STDP rule, and dynamically adjusting the weights to realize the enhancement (LTP) or inhibition (LTD) of the synaptic weights. The spatial cross-entropy loss function can also be combined to calculate the error between the network output and the true label. Thus, the STDP class and the spatial cross-entropy loss function can be combined to replace the gradient method for backpropagation to update the network parameters.

[0082] According to an embodiment of the present disclosure, the training method of the above-mentioned spiking neural network model may further include: obtaining an output result predicted by the spiking neural network for a sample image, where the output result has a spiking output frequency and the sample image has a true class label; performing a third normalization process on the spiking output frequency according to the label category of the sample image to obtain a probability distribution of the output result; calculating a spatial cross-entropy loss according to the true class label and the probability distribution; and adjusting the model parameters of the spiking neural network by using the spatial cross-entropy loss.

[0083] According to an embodiment of the present disclosure, the output result may output the class probability of each pixel, represented by a spiking frequency.

[0084] For example, to obtain the output result, the spiking frequency output by the output layer of the spiking neural network may be calculated first, including: counting the number of spikes at each pixel position within the simulation time , calculating the spiking frequency , where T is the total simulation time. Then, in combination with the class probability representation of formula (3), the class probability to which each pixel belongs may be calculated according to the spiking frequency. After that, for each pixel, the spiking frequencies of each class may be normalized to obtain a probability distribution.

[0085] Formula (3)

[0086] After that, the spatial cross-entropy loss function shown in formula (4) may be used to calculate the loss for each pixel position to improve the semantic segmentation accuracy.

[0087] Formula (4)

[0088] In formulas (3) and (4), are parameters. L is the average loss of the entire batch. N represents the total number of samples in the batch. In the case of image segmentation, it can be considered as the number of images or more specifically the number of all pixel points, depending on how the samples are defined. C is the number of classes, that is, the number of classes to which the model attempts to assign each pixel. is the actual label of the th sample belonging to the th class. For a multi-classification problem, it is usually in one-hot encoded form, meaning that if the pixel truly belongs to the th class, the value is 1, otherwise 0. is the probability that the model predicts the th sample belongs to the th class.

[0089] Based on the above losses, it is possible to use the surrogate gradient function to calculate the gradient of the loss with respect to the weights, and use optimization algorithms such as Adam (Adaptive Moment Estimation) or SGD (Stochastic Gradient Descent) to update the model parameters, thereby implementing backpropagation and parameter update.

[0090] Through the above embodiments of the present disclosure, during the network training process, an STDP-like surrogate gradient method is created, which solves the problem of gradient calculation caused by the non-differentiability of the pulse function in the spiking neural network, making it impossible to directly apply the traditional backpropagation algorithm, and improves the training efficiency.

[0091] According to the embodiments of the present disclosure, after each training batch ends, the dynamic state of the leaky integrate-and-fire neurons can be reset.

[0092] According to the embodiments of the present disclosure, the dynamic state of the LIF neurons may include at least one of the following: membrane potential, firing state, internal variables, etc.

[0093] During reset, for example, the membrane potential can be reset to the resting potential v reset . It can be set when initializing the LIF neurons. For example, LIFNode(v_reset=-1.0), and the membrane potential will become -1.0 during reset. The firing state representing the pulse firing state of the neurons (i.e., the spike tensor) can be reset to False or 0, indicating no pulse has been fired. In the presence of other internal variables, for example, if the LIF neurons contain other dynamic variables (such as current I), they can be reset to the initial value (for example, 0). In the presence of pre- and post-synaptic pulse time records, the buffer storing the pre- and post-synaptic pulse times can be cleared. For intermediate cumulative variables, if the weight change amounts are accumulated during the STDP update process, these temporary variables can be reset to 0.

[0094] It should be noted that STDP time constants such as tau pre , tau post etc. are initialization parameters, and the reset operation will not modify them.

[0095] Through the above embodiments of the present disclosure, after each training batch ends, by resetting the internal states of the STDP-like and LIF neurons, the internal discharge history memory can be cleared in a timely manner, effectively preventing the excessive memory occupation caused by the accumulation of historical information.

[0096] Figure 6Schematically shows a method of applying memristor pulse timing-dependent plasticity to the SNN DeepLab network according to an embodiment of the present disclosure.

[0097] As Figure 6 shown, the method includes operations S601~S607, demonstrating the complete process from data acquisition to output results.

[0098] In operation S601, the memristor STDP parameters are measured experimentally.

[0099] In operation S602, data fitting is performed according to the weight update formula.

[0100] In operation S603, an STDP class is defined and a DeepLab network based on the LIF model is constructed.

[0101] In operation S604, data preprocessing is performed and converted into pulse coding.

[0102] In operation S605, gradient substitution is used for network training and weight update.

[0103] In operation S606, parameters are updated and the internal state is reset.

[0104] In operation S607, the output results are obtained.

[0105] Through the above embodiments of the present disclosure, by using the experimentally measured STDP data for fitting, the pulse firing times of the front and rear neurons are obtained, and an STDP class is created based on this to achieve dynamic update of synaptic weights. An SNN model capable of efficiently performing semantic segmentation is constructed. This method can utilize the physical characteristics of memristors and the advantages of pulse neural networks to achieve efficient image segmentation tasks on low-power hardware platforms, providing a new approach for the combination of neuromorphic computing and deep learning.

[0106] According to an embodiment of the present disclosure, after the pulse neural network is trained based on the above method, the trained pulse neural network can be applied to multi-class semantic segmentation tasks, enabling efficient pixel-level classification on low-power hardware platforms, and can also be extended and applied to computer vision tasks such as object detection and image classification.

[0107] Figure 7 Schematically shows a flowchart of an image segmentation method according to an embodiment of the present disclosure.

[0108] As Figure 7 shown, the method includes operations S701~S702.

[0109] In operation S701, the image to be segmented is obtained.

[0110] In operation S702, the pulse sequence image of the image to be segmented is input into the pulsed neural network to generate an image segmentation result.

[0111] According to an embodiment of the present disclosure, the pulse sequence image may have the same or similar technical characteristics as the aforementioned Poisson random pulse sequence, which will not be elaborated here.

[0112] According to an embodiment of the present disclosure, the category with the highest probability may be selected as the image segmentation result according to the probability distribution represented by the prediction result output by the pulsed neural network for the pulse sequence image.

[0113] After that, it is possible to select to perform color mapping on the image segmentation result to generate a visualized segmentation image and achieve visualized output.

[0114] Through the above embodiments of the present disclosure, a method for semantic segmentation combining the STDP performance based on memristors with SNN is provided, which solves the problems of high energy consumption, difficult training, and lack of hardware-level synaptic plasticity in the prior art, and provides a new technical approach for the combination of neuromorphic computing and deep learning.

[0115] According to an embodiment of the present disclosure, the trained pulsed neural network can also be popularized and applied to other computer vision tasks that require fine prediction, such as object detection, instance segmentation, video analysis, etc. At the same time, combined with memristor hardware, efficient computing on a low-power platform is realized, providing a reference for future neuromorphic computing and artificial intelligence chip design.

[0116] The following is the experimental verification and result analysis for the above method.

[0117] Experimental settings:

[0118] Dataset: A public semantic segmentation dataset is selected.

[0119] Evaluation metrics: Metrics such as PA (pixel accuracy) and mIoU (Mean Intersection over Union) are used to evaluate the model performance.

[0120] Comparison models: Comparison is made with traditional ANN models and SNN models without using STDP.

[0121] Experimental results:

[0122] Performance improvement: The method of the present invention is superior to the comparison models in terms of both PA and mIoU metrics, proving the effectiveness of the STDP mechanism and the surrogate gradient method.

[0123] Energy consumption reduction: Due to the sparsity of SNN and the low-power characteristics of memristors, the energy consumption is significantly reduced.

[0124] Result analysis:

[0125] Contribution of the STDP mechanism: Through the dynamic adjustment of weights, the network can better adapt to the input data, improving the learning ability of the model.

[0126] Role of the surrogate gradient method: Solved the problem that the gradient cannot be propagated, enabling the training of deep SNN models.

[0127] Advantages of memristors: Simulated biological synaptic plasticity at the hardware level, providing support for achieving low-power and high-efficiency computing.

[0128] In addition, it solved the problem of balancing energy consumption and performance, providing a new solution for high-performance computing in resource-constrained environments.

[0129] The main innovation points of the method provided by this disclosure are as follows:

[0130] 1. Utilize the spike-timing-dependent plasticity (STDP) characteristics of memristors: Combine the physical characteristics of memristors with neural networks, and use the STDP mechanism of memristors to achieve dynamic and physically adjustable synaptic weights. This hardware-level synaptic plasticity enhances the biological interpretability of the model.

[0131] 2. Create an STDP class and embed it into a spiking neural network (SNN): According to the experimentally measured STDP data of memristors, a weight update formula was obtained through fitting, and an STDP class was created. Embedding this STDP class into the convolutional layer and fully connected layer of the SNN enables the network to dynamically adjust weights according to the time difference between the spike firings of the pre- and post-synaptic neurons.

[0132] 3. Adopt the surrogate gradient method to solve the SNN training problem: In response to the problem that the gradient cannot be directly calculated in SNNs, the surrogate gradient method was introduced to solve the difficulty of gradient calculation in SNN training, ensuring the trainability of deep SNN models.

[0133] 4. Construct an SNN DeepLab network structure based on the LIF model: Convert the traditional DeepLab semantic segmentation network structure into an SNN based on the LIF model, and embed the STDP mechanism into the network, successfully applying the SNN to complex semantic segmentation tasks.

[0134] 5. Achieve low-power and high-performance semantic segmentation: Combining the low-power advantages of memristor hardware and SNNs, high-performance semantic segmentation tasks were achieved under low energy consumption conditions, providing a new solution for artificial intelligence applications on resource-constrained devices.

[0135] It has the following advantages:

[0136] Hardware-software co-design: By combining the device characteristics of memristors and the computational advantages of SNNs, a hardware-software integrated neuromorphic computing system is achieved.

[0137] Biological interpretability: By introducing the STDP mechanism to simulate the learning and memory processes of biological neural systems, the biological interpretability of the model is improved.

[0138] Low power consumption and high performance: On the premise of ensuring the model performance, the energy consumption is significantly reduced, making it suitable for applications in resource-constrained environments such as mobile devices and embedded systems.

[0139] Innovative training strategy: By adopting the surrogate gradient method, the problem of gradient calculation in SNN training is solved, providing a new idea for the training of deep SNNs.

[0140] Figure 8 A block diagram of a training device for a spiking neural network according to an embodiment of the present disclosure is schematically shown.

[0141] As Figure 8 shown, the training device 800 for a spiking neural network includes a spike time difference acquisition module 810, a weight change amount calculation module 820, and a weight adjustment module 830.

[0142] The spike time difference acquisition module 810 is configured to input the sample pulse sequence feature of a sample image into the spiking neural network model, and when generating a spike based on the response of the leaky integrate-and-fire neurons in the spiking neural network model to the sample pulse sequence feature, acquire the spike time difference between the presynaptic spike and the postsynaptic spike, wherein the spiking neural network model has a current synaptic weight, and the current synaptic weight is used to process the presynaptic spike to obtain the postsynaptic spike.

[0143] The weight change amount calculation module 820 is configured to calculate the weight change amount corresponding to the spike time difference by using a weight update formula, wherein the weight update formula is obtained by fitting the spike timing-dependent plasticity formula based on the sample spike timing-dependent plasticity data.

[0144] The weight adjustment module 830 adjusts the current synaptic weight by using the weight change amount to train the spiking neural network model.

[0145] According to an embodiment of the present disclosure, the training device for a spiking neural network further includes a pulse combination sequence acquisition module, a memristor conductance recording module, a positive sample construction module, a negative sample construction module, and a sample spike timing-dependent plasticity data determination module.

[0146] The pulse combination sequence acquisition module is configured to acquire a pulse combination sequence determined based on pulse combinations of pre-synaptic pulses and post-synaptic pulses. The pulse combination sequence includes a plurality of pulse combinations, and different pulse combinations have different positive time differences or negative time differences.

[0147] The memristor conductance recording module is configured to record positive memristor conductance information corresponding to the positive time difference and negative memristor conductance information corresponding to the negative time difference when a plurality of pulse combinations are sequentially applied to the memristor.

[0148] The positive sample construction module is configured to construct positive samples according to the corresponding positive time differences and positive memristor conductance information.

[0149] The negative sample construction module is configured to construct negative samples according to the corresponding negative time differences and negative memristor conductance information.

[0150] The sample pulse timing-dependent plasticity data determination module is configured to determine sample pulse timing-dependent plasticity data according to the positive samples and the negative samples.

[0151] According to an embodiment of the present disclosure, the training device of the spiking neural network further includes a first fitting module, a second fitting module, and a weight update formula determination module.

[0152] The first fitting module is configured to perform a first fitting on the pulse timing-dependent plasticity formula by using the positive samples, and determine the forward amplitude in the pulse timing-dependent plasticity formula and the forward time constant corresponding to the forward amplitude.

[0153] The second fitting module is configured to perform a second fitting on the pulse timing-dependent plasticity formula by using the negative samples, and determine the negative amplitude in the pulse timing-dependent plasticity formula and the negative time constant corresponding to the negative amplitude.

[0154] The weight update formula determination module is configured to determine the pulse timing-dependent plasticity formula after the first fitting and the second fitting as the weight update formula.

[0155] According to an embodiment of the present disclosure, the weight adjustment module includes an updated synaptic weight calculation unit, a first normalization processing unit, and a weight adjustment unit.

[0156] The updated synaptic weight calculation unit is configured to calculate the sum of the weight change amount and the current synaptic weight to obtain the updated synaptic weight.

[0157] The first normalization processing unit is configured to perform a first normalization processing on the updated synaptic weight to obtain a normalized weight, where the normalized weight is within a preset weight range.

[0158] The weight adjustment unit is configured to replace the current synaptic weight with the normalized weight to implement the adjustment of the current synaptic weight.

[0159] According to an embodiment of the present disclosure, the training device of the spiking neural network further includes a second normalization processing module, a Poisson distribution parameter determination module, a pulse generation probability determination module, and a sample pulse sequence feature generation module.

[0160] The second normalization processing module is configured to perform second normalization processing on the sample image to obtain a normalized image, wherein the image scale of the normalized image satisfies a preset scale.

[0161] The Poisson distribution parameter determination module is configured to use the pixel value of the normalized image as the Poisson distribution parameter.

[0162] The pulse generation probability determination module is configured to determine, for each pixel in the normalized image and within each time step, the pulse generation probability of the pixel within the time step according to the time step and the Poisson distribution parameter.

[0163] The sample pulse sequence feature generation module is configured to generate sample pulse sequence features corresponding to the sample image according to the pulse generation probability.

[0164] According to an embodiment of the present disclosure, the sample pulse sequence feature generation module includes a Poisson random pulse sequence generation unit, a one-hot encoding unit, and a sample pulse sequence feature generation unit.

[0165] The Poisson random pulse sequence generation unit is configured to generate a Poisson random pulse sequence corresponding to the sample image according to the pulse generation probability.

[0166] The one-hot encoding unit is configured to perform one-hot encoding on the Poisson random pulse sequence according to the class label of the sample image to obtain a sample pulse sequence vector.

[0167] The sample pulse sequence feature generation unit is configured to determine sample pulse sequence features according to the sample pulse sequence vector.

[0168] According to an embodiment of the present disclosure, the sample pulse sequence feature generation unit includes a first feature extraction subunit, an intermediate pulse sequence acquisition subunit, and a second feature extraction subunit.

[0169] The first feature extraction subunit is configured to perform first feature extraction on the sample pulse sequence vector to obtain a sample pulse feature map.

[0170] The intermediate pulse sequence acquisition subunit is configured to acquire an intermediate pulse sequence when generating the intermediate pulse sequence based on a leaky integrate-and-fire neuron in response to the sample pulse feature map.

[0171] The second feature extraction subunit is configured to perform second feature extraction on the intermediate pulse sequence to obtain sample pulse sequence features.

[0172] According to an embodiment of the present disclosure, the training device of the spiking neural network further includes an output result acquisition module, a third normalization processing module, a spatial cross-entropy loss calculation module, and a model parameter adjustment module.

[0173] The output result acquisition module is configured to acquire an output result predicted by the spiking neural network for a sample image, where the output result has a pulse output frequency, and the sample image has a true class label.

[0174] The third normalization processing module is configured to perform third normalization processing on the pulse output frequency according to the label category of the sample image to obtain a probability distribution of the output result.

[0175] The spatial cross-entropy loss calculation module is configured to calculate a spatial cross-entropy loss according to the true class label and the probability distribution.

[0176] The model parameter adjustment module is configured to adjust the model parameters of the spiking neural network by using the spatial cross-entropy loss.

[0177] According to an embodiment of the present disclosure, the training device of the spiking neural network further includes a reset module.

[0178] The reset module is configured to reset the dynamic state of the leaky integrate-and-fire neurons after each training batch.

[0179] Figure 9 A block diagram of an image segmentation device according to an embodiment of the present disclosure is schematically shown.

[0180] As Figure 9 shown, the image segmentation device 900 includes an image to be segmented acquisition module 910 and a spiking neural network processing module 920.

[0181] The image to be segmented acquisition module 910 is configured to acquire an image to be segmented.

[0182] The spiking neural network processing module 920 is configured to input the pulse sequence image of the image to be segmented into the spiking neural network to generate an image segmentation result, where the spiking neural network is trained by using the training method of the spiking neural network described above.

[0183] Any number of modules, units, and subunits according to embodiments of the present disclosure, or at least some functions of any of them, may be implemented in one module. Any one or more of the modules, units, and subunits according to embodiments of the present disclosure may be split into multiple modules for implementation. Any one or more of the modules, units, and subunits according to embodiments of the present disclosure may be at least partially implemented as a hardware circuit, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application-specific integrated circuit (ASIC), or may be implemented by any other reasonable way of integrating or packaging circuits, in hardware or firmware, or implemented in any one of the three implementation manners of software, hardware, and firmware, or in an appropriate combination of any several of them. Alternatively, one or more of the modules, units, and subunits according to embodiments of the present disclosure may be at least partially implemented as a computer program module, and when the computer program module is run, the corresponding functions may be executed.

[0184] For example, any number of the pulse time difference acquisition module 810, the weight change amount calculation module 820, and the weight adjustment module 830, or the to-be-segmented image acquisition module 910 and the spiking neural network processing module 920 may be combined and implemented in one module / unit / subunit, or any one of the modules / units / subunits may be split into multiple modules / units / subunits. Alternatively, at least some functions of one or more of these modules / units / subunits may be combined with at least some functions of other modules / units / subunits and implemented in one module / unit / subunit. According to embodiments of the present disclosure, at least one of the pulse time difference acquisition module 810, the weight change amount calculation module 820, and the weight adjustment module 830, or the to-be-segmented image acquisition module 910 and the spiking neural network processing module 920 may be at least partially implemented as a hardware circuit, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application-specific integrated circuit (ASIC), or may be implemented by any other reasonable way of integrating or packaging circuits, etc., in hardware or firmware, or implemented in any one of the three implementation manners of software, hardware, and firmware, or in an appropriate combination of any several of them. Alternatively, at least one of the pulse time difference acquisition module 810, the weight change amount calculation module 820, and the weight adjustment module 830, or the to-be-segmented image acquisition module 910 and the spiking neural network processing module 920 may be at least partially implemented as a computer program module, and when the computer program module is run, the corresponding functions may be executed.

[0185] Figure 10A block diagram of an electronic device suitable for implementing at least one of a training method and an image segmentation method of a spiking neural network according to an embodiment of the present disclosure is schematically shown. Figure 10 The electronic device shown is merely an example and should not impose any limitations on the functions and scope of use of the embodiments of the present disclosure.

[0186] As Figure 10 shown, the electronic device 1000 according to an embodiment of the present disclosure includes a processor 1001, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage section 1008 into a random access memory (RAM) 1003. The processor 1001 can include, for example, a general microprocessor (such as a CPU), an instruction set processor, and / or a related chipset, and / or a dedicated microprocessor (such as an application specific integrated circuit (ASIC)), and so on. The processor 1001 can also include on-board memory for caching purposes. The processor 1001 can include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0187] In the RAM 1003, various programs and data required for the operation of the electronic device 1000 are stored. The processor 1001, the ROM 1002, and the RAM 1003 are connected to each other via a bus 1004. The processor 1001 performs various operations of the method flow according to an embodiment of the present disclosure by executing the program in the ROM 1002 and / or the RAM 1003. It should be noted that the program can also be stored in one or more memories other than the ROM 1002 and the RAM 1003. The processor 1001 can also perform various operations of the method flow according to an embodiment of the present disclosure by executing the program stored in the one or more memories.

[0188] According to an embodiment of the present disclosure, the electronic device 1000 may further include an input / output (I / O) interface 1005, and the input / output (I / O) interface 1005 is also connected to the bus 1004. The system 1000 may further include one or more of the following components connected to the input / output (I / O) interface 1005: an input portion 1006 including a keyboard, a mouse, etc.; an output portion 1007 including, for example, a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage portion 1008 including a hard disk, etc.; and a communication portion 1009 including a network interface card such as a LAN card, a modem, etc. The communication portion 1009 performs communication processing via a network such as the Internet. A drive 1010 is also connected to the input / output (I / O) interface 1005 as needed. A removable medium 1011, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 1010 as needed so that a computer program read from it can be installed into the storage portion 1008 as needed.

[0189] According to an embodiment of the present disclosure, the method flow according to the embodiment of the present disclosure may be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product that includes a computer program carried on a computer-readable storage medium, and the computer program includes program codes for executing the method shown in the flowchart. In such an embodiment, the computer program may be downloaded and installed from the network through the communication portion 1009, and / or installed from the removable medium 1011. When the computer program is executed by the processor 1001, the above-described functions defined in the system of the embodiment of the present disclosure are executed. According to an embodiment of the present disclosure, the above-described system, device, apparatus, module, unit, etc. may be implemented by computer program modules.

[0190] The present disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or may exist separately without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method according to the embodiment of the present disclosure is implemented.

[0191] According to an embodiment of the present disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium. For example, it may include but is not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the above. In the present disclosure, the computer-readable storage medium may be any tangible medium that contains or stores a program, and this program can be used by or in combination with an instruction execution system, apparatus, or device.

[0192] For example, according to an embodiment of the present disclosure, the computer-readable storage medium may include one or more memories other than the above-described ROM 1002 and / or RAM 1003 and / or ROM 1002 and RAM 1003.

[0193] Embodiments of the present disclosure also include a computer program product, which includes a computer program that contains program code for executing the method provided by the embodiments of the present disclosure. When the computer program product runs on an electronic device, the program code is used to cause the electronic device to implement at least one of the training method of the pulse neural network and the image segmentation method provided by the embodiments of the present disclosure.

[0194] When the computer program is executed by the processor 1001, the above functions defined in the system / apparatus of the embodiments of the present disclosure are executed. According to an embodiment of the present disclosure, the above-described systems, apparatuses, modules, units, etc. may be implemented by computer program modules.

[0195] In one embodiment, the computer program may rely on tangible storage media such as optical storage devices and magnetic storage devices. In another embodiment, the computer program may also be transmitted and distributed in the form of a signal on a network medium, and be downloaded and installed through the communication part 1009, and / or be installed from the removable medium 1011. The program code contained in the computer program can be transmitted by any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0196] According to embodiments of the present disclosure, program code for executing the computer programs provided by the embodiments of the present disclosure may be written in any combination of one or more programming languages. Specifically, these computing programs may be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages include, but are not limited to, such as Java, C++, Python, the "C" language, or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., by connecting through the Internet using an Internet service provider).

[0197] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architectures, functions, and operations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagram may represent a module, a program segment, or a part of code that contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than marked in the accompanying drawings. For example, two consecutive blocks shown may actually be executed substantially in parallel, and they may sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagram or flowchart, and combinations of blocks in the block diagram or flowchart, may be implemented by a dedicated hardware-based system for performing the specified functions or operations, or may be implemented by a combination of dedicated hardware and computer instructions. Those skilled in the art can understand that the features recited in the various embodiments and / or claims of the present disclosure can be combined and / or combined in various ways, even if such combinations or combinations are not explicitly recited in the present disclosure. In particular, without departing from the spirit and teachings of the present disclosure, the features recited in the various embodiments and / or claims of the present disclosure can be combined and / or combined in various ways. All such combinations and / or combinations fall within the scope of the present disclosure.

[0198] The above describes the embodiments of the present disclosure. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of the present disclosure. Although the embodiments are described separately above, this does not mean that the measures in each embodiment cannot be used advantageously in combination. The scope of the present disclosure is defined by the appended claims and their equivalents. Without departing from the scope of the present disclosure, those skilled in the art can make various substitutions and modifications, and all such substitutions and modifications should fall within the scope of the present disclosure.

Claims

1. A method for training a spiking neural network, comprising: Inputting a sample pulse sequence feature of a sample image into a pulse neural network model, and obtaining a pulse time difference between a presynaptic pulse and a postsynaptic pulse when a leaky integral discharge neuron in the pulse neural network model responds to the sample pulse sequence feature and generates a pulse, wherein the pulse neural network model has a current synaptic weight, and the current synaptic weight is used to process the presynaptic pulse to obtain the postsynaptic pulse; Calculating a weight change corresponding to the pulse time difference using a weight update formula, wherein the weight update formula is obtained by fitting a pulse timing-dependent plasticity formula based on sample pulse timing-dependent plasticity data; and The current synaptic weight is adjusted using the weight change amount to train the spiking neural network model.

2. The method according to claim 1, further comprising: Before calculating the weight change corresponding to the pulse time difference using the weight update formula, Acquire a pulse combination sequence determined based on a pulse combination of a presynaptic pulse and a postsynaptic pulse, wherein the pulse combination sequence includes a plurality of the pulse combinations, and different pulse combinations have different positive time differences or negative time differences; When the plurality of pulse combinations act on the memristor in sequence, recording positive memristor conductance information corresponding to the positive time difference and negative memristor conductance information corresponding to the negative time difference; Constructing a positive sample according to the corresponding positive time difference and the positive memristor conductance information; Constructing a negative sample according to the corresponding negative time difference and the negative memristor conductance information; as well as The sample pulse timing-dependent plasticity data is determined according to the positive sample and the negative sample.

3. The method according to claim 2, wherein: The step of fitting the pulse timing dependent plasticity formula based on the sample pulse timing dependent plasticity data comprises: Using the positive sample, performing a first fitting on the pulse timing-dependent plasticity formula to determine a positive amplitude in the pulse timing-dependent plasticity formula and a positive time constant corresponding to the positive amplitude; Using the negative sample, performing a second fitting on the pulse timing dependent plasticity formula to determine a negative amplitude in the pulse timing dependent plasticity formula and a negative time constant corresponding to the negative amplitude; and The pulse timing dependent plasticity formula after the first fitting and the second fitting is determined as the weight updating formula.

4. The method according to claim 1, wherein: The adjusting the current synaptic weight by using the weight change amount comprises: Calculating the sum of the weight change amount and the current synaptic weight to obtain an updated synaptic weight; performing a first normalization process on the updated synaptic weight to obtain a normalized weight, wherein the normalized weight is within a preset weight range; and The normalized weight is used to replace the current synaptic weight, so as to adjust the current synaptic weight.

5. The method according to claim 1, further comprising: Before the sample pulse sequence features of the sample image are input into the pulse neural network model, Performing a second normalization process on the sample image to obtain a normalized image, wherein the image scale of the normalized image satisfies a preset scale; Using the pixel value of the normalized image as a Poisson distribution parameter; For each pixel in the normalized image, in each time step, determining a pulse generation probability of the pixel in the time step according to the time step and the Poisson distribution parameter; and A sample pulse sequence feature corresponding to the sample image is generated according to the pulse generation probability.

6. The method according to claim 5, wherein: Generating a sample pulse sequence feature corresponding to the sample image according to the pulse generation probability comprises: generating a Poisson random pulse sequence corresponding to the sample image according to the pulse generation probability; According to the category label of the sample image, one-hot encoding is performed on the Poisson random pulse sequence to obtain the sample pulse sequence vector; and The sample pulse sequence feature is determined according to the sample pulse sequence vector.

7. The method according to claim 6, wherein: Determining the sample pulse sequence feature according to the sample pulse sequence vector includes: Performing a first feature extraction on the sample pulse sequence vector to obtain a sample pulse feature graph; In the case where an intermediate pulse sequence is generated based on the leaky integration and firing neuron responding to the sample pulse characteristic diagram, acquiring the intermediate pulse sequence; and A second feature extraction is performed on the intermediate pulse sequence to obtain the sample pulse sequence feature.

8. The method according to any one of claims 1 to 7, further comprising: Obtaining an output result predicted by the spiking neural network for the sample image, wherein the output result has a spiking output frequency, and the sample image has a true category label; According to the label category of the sample image, performing a third normalization process on the pulse output frequency to obtain a probability distribution of the output result; Calculating a spatial cross entropy loss based on the true category label and the probability distribution; and The spatial cross entropy loss is used to adjust the model parameters of the spiking neural network.

9. The method according to claim 8, further comprising: After each training batch, the dynamic state of the leaky integration and firing neuron is reset.

10. An image segmentation method, comprising: Obtain the image to be segmented; as well as The pulse sequence image of the image to be segmented is input into a pulse neural network to generate an image segmentation result, wherein the pulse neural network is trained using the method according to any one of claims 1 to 9.

Images