A pre-attention extraction method based on convolutional spiking neural network

Abstract

The present invention discloses a pre-attention extraction method based on a convolutional spike neural network. After merging the polarity of the original pulse data, it filters it in the time dimension, then calculates the cumulative number of pulse activations of all samples at different pixel positions, and obtains pre-attention samples that can represent the frequency of pulse occurrence at different positions. While the original pulse data is normally fed forward through the convolutional spike neural network, the pre-attention samples are convoluted and fed forward together. At each layer, the membrane potential of each neuron is adjusted by converting it into a pre-attention size through a specific function to guide the pulse emission of neurons in each layer. The method of the present invention enables the convolutional spike neural network to perform operations in combination with pre-attention, so that the network can pay more attention to more important pulse events, reduce attention to unimportant pulse times, and ignore noise pulses, thereby improving the overall performance of the network.

Description

Technical Field

[0001] The present invention belongs to the technical field of image recognition, and in particular relates to a pre-attention extraction method based on a convolutional pulse neural network. Background Art

[0002] Spiking neural networks have significant hardware advantages due to their event-driven nature. In recent years, the use of convolutional filters in spiking neural networks has enhanced their ability to extract spatial features, making convolutional spiking neural networks a research hotspot in neuromorphic vision tasks.

[0003] Although convolutional spike neural networks can extract local visual features well through task learning, the local field of view characteristics of the convolution kernel will make it impossible for the network to make a global judgment on the importance of each spike event during convolution feedforward, thereby focusing on some unimportant spike events, causing it to emit a large number of redundant spikes in the time domain, which will damage the network's information representation.

[0004] To address the above problems, the pre-attention method can be used to analyze and mine the pulse data itself to obtain a global importance judgment of the pulse event, and organically combine it with the learning process of the convolutional spike neural network to help it make a more comprehensive and reasonable judgment on the importance of each pulse event, avoid redundant pulses and noise, and improve network performance. Summary of the Invention

[0005] Based on the above-mentioned shortcomings and deficiencies in the prior art, one of the objects of the present invention is to at least solve one or more of the above-mentioned problems in the prior art. In other words, one of the objects of the present invention is to provide a pre-attention extraction method based on a convolutional pulse neural network that meets one or more of the above-mentioned needs.

[0006] In order to achieve the above-mentioned object of the invention, the present invention adopts the following technical solutions:

[0007] A pre-attention extraction method based on convolutional spiking neural network, comprising the steps of:

[0008] S1. Obtain several original pulse samples with the same pixel size. The original pulse samples are captured by an event camera, and the image changes are recorded as pulses appearing at each pixel position.

[0009] S2. Set a pre-attention threshold greater than 1, count the number of pulses that appear at each pixel position of each original pulse sample, reset the pixel position where the number of pulses in each original pulse sample is greater than the pre-attention threshold to have one pulse, and reset the pixel position where the number of pulses is less than the pre-attention threshold to a silent state;

[0010] S3. After resetting all original pulse samples at each pixel position, the number of pulses at that position is accumulated to obtain a pre-attention sample. The value of the pre-attention sample at each pixel position represents the frequency of pulses at that pixel position. The higher the frequency, the less important the pixel position.

[0011] S4. Feed the original spike sample through a convolutional spike neural network, which is a normal feedforward convolutional spike neural network; and feed the pre-attention sample through a convolutional neural network, where the convolution kernel size of the convolutional neural network is the same as that of the convolutional spike neural network, so that the receptive field range of the neurons at corresponding positions in the convolutional spike neural network and the convolutional neural network are the same, the convolution kernel weight of the convolutional neural network is 1, and the activation function after convolution is replaced by a normalization transformation;

[0012] The output of the pre-attention sample in each convolution layer of the convolutional neural network is normalized and converted into the pre-attention of each pixel point in the output of each layer of the convolutional spike neural network; based on the pre-attention of each pixel point in the output of each layer, the neuronal membrane potential of the original spike sample at the corresponding position is enhanced or weakened.

[0013] As a preferred solution, step S2 specifically includes the following method:

[0014] S21, processing the original pulse sample into a size of P×T×W×H The dimensions of P, T, W, and H are respectively equal to the number of polarities, the pulse time window, the width of the original pulse sample, and the height of the original pulse sample. According to the polarity type, the time of occurrence, the width and height of the original pulse sample, the corresponding positions in the P, T, W, and H dimensions are set to 1, and the other positions are set to 0.

[0015] S22, adding the P and T dimensions of the original pulse samples to obtain a matrix representing the number of pulse emission times at each pixel position;

[0016] S23. Set a pre-attention threshold greater than 1, and reset the pixel positions whose total pulse emission times are greater than the pre-attention threshold to 1, otherwise reset to 0.

[0017] As a further preferred solution, step S3 is specifically as follows:

[0018] The matrices obtained after resetting all the original pulse samples in step S23 are accumulated so that the values at the same pixel position are accumulated together to obtain the matrix of pre-attention samples; the value of the matrix of pre-attention samples at each pixel position represents the frequency of pulse occurrence at that pixel position.

[0019] As a preferred solution, step S4 specifically includes the following method:

[0020] S41, feeding the original pulse sample forward through a convolutional spike neural network for one layer, where the convolutional spike neural network is a normal feedforward convolutional spike neural network;

[0021] S42, feed the pre-attention sample through a convolutional neural network for one layer, the convolution kernel size of the convolutional neural network is the same as that of the convolutional spiking neural network, so that the receptive fields of neurons at corresponding positions in the convolutional spiking neural network and the convolutional neural network are the same, and the convolution kernel weight of the convolutional neural network is 1;

[0022] S43, normalize each output of the current feedforward of the convolutional neural network to [-1, 1] in proportion;

[0023] S44. Performing a function conversion based on information theory on the normalized value to obtain an importance correction value, where the importance correction value indicates the degree to which the neuron membrane potential at the corresponding position in the convolutional spike neural network should be weakened or strengthened;

[0024] S45. weakening or strengthening the neuron membrane potential at the corresponding position in the convolutional spike neural network according to the importance correction value;

[0025] S46. Return to step S41 until the convolutional spike neural network completes the output of the final layer.

[0026] As a further preferred solution, the normalization of step S43 is specifically as follows:

[0027] Assume that one output of the convolutional neural network feedforward is p n , p n ∈ numMap , the normalized value is p a , p a ∈ attnMap , numMap It's each location p n The matrix composed of attnMap It's each location p a The matrix composed of

[0028] Will numMap within p n Converted to normal distribution, the converted matrix is recorded as X :

[0029] ;

[0030] in, μ express numMap middle pn The mean of σ express numMap middle p n The standard deviation of

[0031] Will X Elements in X ij Convert to importance-adjusted value p i :

[0032] ;

[0033] in, min ( X )and max ( X ) represent matrices X The minimum and maximum values in .

[0034] As a further preferred solution, the function conversion based on the information theory principle in step S44 is specifically as follows:

[0035] Let the normalized value be p a , p a ∈ attnMap , the importance correction value is p i , p i ∈ interMap , attnMap It's each location p a The matrix composed of interMap It's each location p i The matrix composed of

[0036] p a = α × arctan ( β × p i + γ );

[0037] in α Decided p a Upper and lower limits; β Determines the shape of the curve for attention allocation; γ It is a learnable variable that is slightly adjusted with the back propagation of the convolutional spike neural network error to adjust the ratio of important and unimportant parts perceived by attention.

[0038] Compared with the prior art, the present invention has the following beneficial effects:

[0039] The method of the present invention enables the convolutional spike neural network to perform operations in combination with pre-attention, so that the network can pay more attention to more important spike events, pay less attention to unimportant spike times and ignore noise spikes, thereby improving the information representation effect of the network. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 This is a flowchart of a pre-attention extraction method based on a convolutional pulse neural network in an embodiment of the present invention. DETAILED DESCRIPTION

[0041] To more clearly illustrate the embodiments of the present invention, specific embodiments of the present invention will be described below with reference to the accompanying drawings. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings and other embodiments can be obtained based on these drawings without inventive efforts.

[0042] Embodiment: A pre-attention extraction method based on a convolutional spike neural network in this embodiment includes the following steps:

[0043] S1. Obtain several raw pulse samples with the same pixel size. The raw pulse samples are captured by an event camera, and image changes are recorded as pulses appearing at each pixel position. The specific form of each raw pulse sample is a data set that includes the event pulse occurrence position (length and width pixel position), event pulse occurrence time, and event pulse polarity.

[0044] S2. Set a pre-attention threshold greater than 1, count the number of pulses that appear at each pixel position of each original pulse sample, reset the pixel position where the number of pulses in each original pulse sample is greater than the pre-attention threshold to a pulse, and reset the pixel position where the number of pulses is less than the pre-attention threshold to a silent state; this step is used to find the position where pulses frequently appear in each original pulse sample, thereby providing data for the subsequent calculation of the overall pre-attention position of the algorithm.

[0045] This embodiment provides a preferred implementation of step S2, which specifically includes the following steps:

[0046] S21, processing the original pulse sample into a size of P×T×W×H The matrix, P 、 T 、 W 、 H The sizes of are equal to the number of polarities, pulse time window, original pulse sample width, and original pulse sample height. According to the polarity type, occurrence time, width and height position of the original pulse sample,P 、 T 、 W 、 H The corresponding positions in the dimension are set to 1, and the other positions are set to 0. In addition, it should be noted that the number of polarities is generally 2, representing positive and negative polarities respectively. In this embodiment, P is also set to 2 as usual.

[0047] Specifically, if a pulse with polarity 1 occurs at pixel position (1001, 1500) in the original pulse sample at second 2, the value of (1, 2, 1001, 1500) in the four-dimensional matrix is set to 1. Similarly, if a raw pulse sample contains multiple event pulses, the value of the corresponding four-dimensional position is set to 1 based on the polarity, time, and position of each event pulse, and the values of all other positions are set to 0. This process stores the event pulse attributes in a reasonable data format, allowing for subsequent statistics on the event pulse positions.

[0048] S22, the original pulse sample P 、 T The dimensions are added together to obtain a matrix representing the number of pulse emission at each pixel position; P×T×W×H After a simple addition and merging of the positive and negative polarity dimensions, the size becomes T×W×H , and then add the size along its own time dimension to become W×H Since multiple pulses of different polarities are not activated at the same width and height position and at the same time, P The sum of dimensions can only produce 0 or 1, so the sum operation is actually to count the number of pulses that occur at different times at the same width and height position. For example, if there are two pulses (0, 2, 1001, 1500) and (1, 5, 1001, 1500) at the position (1001, 1500), then P 、 T After adding the dimensions T Adding the result, we get the value of 2 at (1001, 1500) W×H The matrix of .

[0049] S23. Set a pre-attention threshold greater than 1, and reset the pixel positions whose total pulse emission times are greater than the pre-attention threshold to 1, otherwise reset to 0.

[0050] Specifically, you can set d ij Indicates that the matrix is in ( i , j ) position, using the polynomial Perform filtering operation on , φ is the hyper parameter used for filtering, and its value range is [0, T]. Reset the value greater than or equal to φ to 1, so that the position of the value will emit pulses in subsequent applications; reset the value less than φ to 0, so that the position of the value will remain silent in subsequent applications. The above filter reset operation is equivalent to extracting the position of the more frequent pulse emission in a single sample, which can be considered important and representative in the current sample, while the position of the silence can be considered negligible even if it has emitted pulses in the past due to its low frequency. After reset d ij The value is 0 or 1.

[0051] After completing the reset in step S2, proceed to step S3, accumulating the number of pulses at each pixel position after resetting all original pulse samples at that position to obtain a pre-attention sample. The value of the pre-attention sample at each pixel position represents the frequency of pulses at that pixel position. The greater the frequency of pulses, the less important the pixel position.

[0052] The difference between S2 and S3 in the above process is that S2 counts the number of pulses at different locations within a single sample to identify the more important pixel locations within that sample, while ignoring occasional noise pulses in that sample during the threshold filtering process. S3, on the other hand, considers all samples and identifies pixel locations where pulses are common across all samples. According to information theory and common sense, when comparing two samples (such as images), pixel locations that differ between them are more helpful. If a pixel location consistently pulses across all samples, it will be impossible to identify differences between individual pulse samples. Therefore, pixel locations where pulses are common in S2 are more important, while pixel locations where pulses are common across all samples are less important in S3.

[0053] Furthermore, in step S3, the matrices obtained after all the original pulse samples are reset in step S23 are accumulated so that the values at the same pixel position are accumulated together to obtain the matrix of pre-attention samples; the values of the matrix of pre-attention samples at each pixel position represent the frequency of the pulses occurring at that pixel position.

[0054] Specifically, in the above step S2, the W×H In a matrix where each pixel position takes a value of 0 or 1, in step S3, each sample is accumulated together so that the values of each pixel position in all original pulse samples are accumulated together to obtain the number of important and representative samples at each pixel position. According to the principle of information theory, the importance of an event is inversely proportional to its frequency of occurrence in all samples. The value obtained in this way can express the importance of the pixel position in all samples. The larger the value, the smaller the pre-attention value that should be configured for the pixel position.

[0055] After obtaining the pre-attention of each pixel position in step S3, step S4 is performed, the original pulse sample is fed forward through the convolutional spike neural network, and the convolutional spike neural network is a normal feedforward convolutional spike neural network; and the pre-attention sample is fed forward through the convolutional neural network, the convolution kernel size of the convolutional neural network is the same as that of the convolutional spike neural network, so that the receptive field range of the neurons at the corresponding positions in the convolutional spike neural network and the convolutional neural network is the same, and the convolution kernel weight of the convolutional neural network is constant to 1. A weight of 1 represents that it cannot be learned during the convolution process. Therefore, the convolution operation of the pre-attention sample is actually to add the pre-attention value of the input pixel of each convolution kernel, so that the size of the sample can be consistent with the original pulse sample in the convolutional spike neural network at all times, and the subsequent normalized value is used to replace the activation function. In this way, the output of the convolution of the pre-attention sample in each layer of the convolutional neural network can be converted into the pre-attention of each pixel point in the output of the convolutional spike neural network at each layer; finally, based on the pre-attention of each pixel point in the output of each layer, the neuronal membrane potential of the original pulse sample at the corresponding position is enhanced or weakened to realize the application of pre-attention.

[0056] Furthermore, step S4 specifically includes the following method:

[0057] S41. Feed the original pulse sample through the convolutional pulse neural network for one layer. The convolutional pulse neural network is a normal feedforward convolutional pulse neural network, that is, the feedforward process of the normal sample in the convolutional pulse neural network. Specifically, assuming that the original input is 50 pulse samples, the time window size is 10, that is, the size is 50×2×10×32×32, and the time point The size of the matrix B input to the convolutional spike neural network is 50×2×32×32. After the first convolution layer with a kernel size of 3 and a stride of 1, the output size is 50×C×30×30, where C is the number of channels after convolution.

[0058] S42: Feed the pre-attention sample forward through a convolutional neural network (CNN) layer. The convolution kernel size of the CNN is the same as that of the CNN, ensuring that the receptive fields of neurons at corresponding positions in the CNN and CNN are the same. The convolution kernel weight of the CNN is 1. Based on the example of S41 above, the 32×32 pre-attention sample is no longer the same size as the original convolved spike sample. Therefore, the pre-attention sample needs to be fed forward synchronously using a convolution kernel of the same size as in S41. After convolution, the pre-attention sample size becomes 30×30, maintaining the same dimensionality as the data at the same layer in the CNN.

[0059] S43, normalize each output of the current feedforward of the convolutional neural network to [-1, 1] in proportion;

[0060] Specifically, step S43 may use the following method:

[0061] Assume that one output of the convolutional neural network feedforward is p n , p n ∈ numMap , the standardized value is p a , p a ∈ attnMap , numMap It's each location p n The matrix composed of attnMap It's each location p a The matrix composed of

[0062] Will numMap within p n Converted to normal distribution, the converted matrix is recorded as X :

[0063] ;

[0064] in, μ express numMap middle p n The mean of σ express numMap middle p n The standard deviation of

[0065] will X Elements in X ij Convert to importance-adjusted value p i :

[0066] ;

[0067] in, min ( X )and max ( X ) represent matrices X The minimum and maximum values in .

[0068] Through the above transformation, the value range of the pre-attention matrix is limited to [-1, 1]. The limited value range can ensure that the pre-attention matrix obtained after subsequent transformations has controllable boundaries.

[0069] S44. The normalized value is converted into a function based on the principle of information theory to obtain the importance correction value. The importance correction value indicates the degree to which the neuron membrane potential at the corresponding position in the convolutional spike neural network should be weakened or strengthened. According to the principle of information theory, the more frequently an event occurs, the less discriminant information it contains, and the less important it is in the recognition task. Therefore, a value greater than 0 p i (Indicates that the event occurs more frequently), and its location can be roughly judged as relatively unimportant.

[0070] Specifically, the standardized value can be set to p a , p a ∈ attnMap , the importance correction value is p i , p i ∈ interMap , attnMap It's each location p a The matrix composed of interMap It's each location p i The matrix composed of

[0071] p a = α × arctan ( β × p i + γ );

[0072] in α Decided p a Upper and lower limits; β Determines the shape of the curve for attention allocation; γ It is a learnable variable that is slightly adjusted as the convolutional spike neural network error is back-propagated to adjust the ratio of important and unimportant parts perceived by attention.

[0073] Assume that in the above formula, α is -0.2, β is 2.5, γ is 0; then p a and p i Examples of corresponding values can be (-0.8, 0.221), (-0.6, 0.197), (-0.4, 0.157), and (-0.2, 0.093).

[0074] S45. weakening or strengthening the neuron membrane potential at the corresponding position in the convolutional spike neural network according to the importance correction value;

[0075] Specifically, use p a Regulate the neuronal membrane potential. .

[0076] For p a >0 position to enhance the neuronal membrane potential, p a The neuronal membrane potential at the position <0 is weakened, and after the adjustment of the importance correction value, the neuronal membrane potential is regulated by pre-attention. Important parts (positions with greater pre-attention) will reach the threshold and emit pulses earlier than normal, while unimportant positions (positions with less pre-attention) will find it more difficult to emit pulses than normal.

[0077] Finally, perform step S46 and return to step S41, feed forward again and adjust the neuron membrane potential through pre-attention until the convolutional spike neural network completes the output of the final layer.

[0078] The flowchart of the above complete process is as follows Figure 1 shown.

[0079] It should be noted that the above embodiments are only detailed descriptions of the preferred embodiments and principles of the present invention. For ordinary technicians in this field, there will be changes in the specific implementation methods based on the ideas provided by the present invention, and these changes should also be regarded as the scope of protection of the present invention.

Claims

1. A pre-attention extraction method based on convolutional spike neural network, characterized in that: Including steps: S1. Acquire a number of original pulse samples with the same pixel size. The original pulse samples are captured by an event camera, and image changes are recorded as pulses appearing at each pixel position. S2. Set a pre-attention threshold greater than 1, count the number of pulses occurring at each pixel position of each original pulse sample, reset the pixel position where the number of pulses in each original pulse sample is greater than the pre-attention threshold to a state where one pulse occurs, and reset the pixel position where the number of pulses is less than the pre-attention threshold to a silent state; S3. After resetting all the original pulse samples at each pixel position, the number of pulses at each pixel position is accumulated to obtain a pre-attention sample, wherein the value of the pre-attention sample at each pixel position represents the frequency of pulses at the pixel position, and the higher the frequency, the less important the pixel position; S4, feeding the original pulse sample forward through a convolutional spiking neural network, wherein the convolutional spiking neural network is a normal feedforward convolutional spiking neural network; Feeding the pre-attention sample forward through a convolutional neural network, the convolution kernel size of the convolutional neural network is the same as that of the convolutional spiking neural network, so that the receptive field range of neurons at corresponding positions in the convolutional spiking neural network and the convolutional neural network are the same, the convolution kernel weight of the convolutional neural network is 1, and the activation function after convolution is replaced by a normalization transformation; Normalizing the output of the pre-attention sample at each convolution layer of the convolutional neural network into the pre-attention of each pixel in the output of each layer of the convolutional spike neural network; Based on the pre-attention of each pixel point in the output of each layer, the neuron membrane potential of the original pulse sample at the corresponding position is enhanced or weakened.

2. The pre-attention extraction method based on convolutional spike neural network according to claim 1, characterized in that: The step S2 specifically includes the following method: S21, processing the original pulse sample into a matrix of size P×T×W×H, where the sizes of P, T, W, and H are respectively equal to the number of polarities, the pulse time window, the original pulse sample width, and the original pulse sample height; according to the polarity type, occurrence time, width, and height of the original pulse sample, the corresponding positions in the P, T, W, and H dimensions are set to 1, and the other positions are set to 0; S22, adding the P and T dimensions of the original pulse samples to obtain a matrix representing the number of pulse emission times at each pixel position; S23. Set a pre-attention threshold greater than 1, and reset the pixel positions whose total pulse emission times are greater than the pre-attention threshold to 1, otherwise reset to 0.

3. The pre-attention extraction method based on convolutional spike neural network according to claim 2, characterized in that: The step S3 is specifically as follows: The matrices obtained after resetting all the original pulse samples in step S23 are accumulated so that the values at the same pixel position are accumulated together to obtain the matrix of pre-attention samples; the value of the matrix of pre-attention samples at each pixel position represents the frequency of pulse occurrence at that pixel position.

4. The pre-attention extraction method based on convolutional spike neural network according to claim 1, characterized in that: The step S4 specifically includes the following method: S41, feeding the original pulse sample forward one layer through a convolutional spiking neural network, where the convolutional spiking neural network is a normal feedforward convolutional spiking neural network; S42, feeding the pre-attention sample through a convolutional neural network for one layer, wherein the convolution kernel size of the convolutional neural network is the same as that of the convolutional spiking neural network, so that the receptive field range of the neurons at corresponding positions in the convolutional spiking neural network and the convolutional neural network is the same, and the convolution kernel weight of the convolutional neural network is 1; S43, normalizing each output of the current feedforward of the convolutional neural network to [-1, 1] in proportion; S44. Performing a function conversion based on information theory on the normalized value to obtain an importance correction value, wherein the importance correction value indicates the degree to which the neuron membrane potential at the corresponding position in the convolutional spike neural network should be weakened or strengthened; S45. weakening or strengthening the neuron membrane potential at the corresponding position in the convolutional spiking neural network according to the importance correction value; S46. Return to step S41 until the convolutional spike neural network completes the output of the final layer.

5. The pre-attention extraction method based on convolutional spike neural network according to claim 4, characterized in that: The normalization of step S43 is specifically as follows: Assume that one output of the convolutional neural network is p n , p n ∈numMap, the normalized value is p a , p a ∈attnMap, numMap is each position p n The matrix composed of attnMap is the matrix of each position p a The matrix composed of Set the p in numMap n Convert to normal distribution, the converted matrix is recorded as X: Among them, μ represents p in numMap n The mean of ,σ represents the p in numMap n The standard deviation of The elements X in X ij Converted to importance correction value p i : Where min(X) and max(X) represent the minimum and maximum values in the matrix X, respectively.

6. A pre-attention extraction method based on convolutional spike neural network according to claim 4, characterized in that: The function conversion in step S44 based on the information theory principle is specifically as follows: Let the normalized value be p a , p a ∈attnMap, the importance correction value is p i , p i ∈interMap, attnMap is each position p a The matrix composed of interMap is the position p i The matrix composed of p a =α×arctan(β×p i +c); Where α determines p a The upper and lower limits of β determine the shape of the attention allocation curve; γ is a learnable variable that is slightly adjusted with the back propagation of the convolutional spike neural network error to adjust the ratio of important and unimportant parts perceived by attention.

Images