A method for extracting temporal surface features based on pulse events

Abstract

The present invention discloses a method for extracting temporal surface features based on pulse events. First, event pulse samples are obtained. First, the self-information magnitude of each pixel position in the event pulse samples is calculated. Then, the pulse occurrence frequency of each pixel position in each sample is calculated. Furthermore, the spatial position importance of each pixel position in each sample is calculated. After that, the temporal surface features are extracted by combining the spatial position importance of each position. The method of the present invention combines the spatial position importance when extracting temporal surface features, focuses the key points of model calculation on important positions, and improves the problems of the traditional temporal surface feature extraction method being insensitive to spatial information and being overly sensitive to noise or minor changes in the event stream.

Description

Technical Field

[0001] The present invention belongs to the technical field of spiking neural networks, and specifically relates to a method for extracting temporal surface features based on spike events. Background Art

[0002] Compared with traditional frame-based cameras, address event representation (AER) cameras have attracted the attention of many scholars at home and abroad in recent years due to their advantages of high temporal resolution and low power consumption. AER cameras can record visual inputs as asynchronous discrete events, so they are very suitable for combination with spiking neural networks (SNNs), which is biologically reasonable and more efficient on neuromorphic hardware.

[0003] When performing event-based object recognition tasks, the core processing part of spiking neural networks is generally divided into three steps, namely feature extraction, spike coding, and classification. Among them, feature extraction is the basic building block of many advanced vision tasks. It generally connects the data preprocessing process and the learning process and is the basis of the learning process. A good feature extraction algorithm will make the subsequent learning algorithm classification twice the result with half the effort.

[0004] Currently, the main feature extraction methods include temporal surface, HMAX, BOE, PCA, MDS, etc. The temporal surface is a new type of event-based spatio-temporal feature extraction method that uses the rich temporal information provided by spike event sequences to create a spike event temporal surface context and calculates the temporal surface with a temporal decay kernel to represent the historical spike event activities in the local spatial neighborhood. However, due to the defect that the temporal surface is insensitive to the spatial position of spikes, noise will greatly affect the accuracy of the temporal surface features, resulting in this feature extraction method being very severely affected by noise.

[0005] The purpose of the present invention is to address the problem that the temporal surface feature extraction method is insensitive to spatial position. In order to better distinguish the importance of different spatial positions, a method for extracting temporal surface features based on spike events is proposed. This method improves the accuracy of object extraction and also enhances the robustness to noisy images.

[0006] When calculating the temporal surface, we treat events at different positions equally without discrimination, resulting in the model being significantly affected by noise and lacking accurate spatial information. Moreover, the low sensor resolution of current event cameras exacerbates this drawback. In the real world, when people observe a picture, they usually focus on the important positions in the picture, that is, the positions where objects exist, and reduce their attention to irrelevant things. Therefore, the present invention is inspired by the TF-IDF weighting technique commonly used in the fields of information retrieval and data mining. We will distinguish the importance of different positions by mining the importance differences of spatial positions within samples and the same pixels between samples, and focus the model calculation on important positions. Summary of the Invention

[0007] Based on the above-mentioned drawbacks and deficiencies existing in the prior art, one of the objectives of the present invention is to at least solve one or more of the above-mentioned problems existing in the prior art. In other words, one of the objectives of the present invention is to provide a method for extracting temporal surface features based on pulsed events that meets one or more of the foregoing requirements.

[0008] To achieve the above-mentioned invention objective, the present invention adopts the following technical solutions:

[0009] A method for extracting temporal surface features based on pulsed events, the method comprising the steps of:

[0010] S1. Obtain an event pulse sample set, where the event pulse sample set includes a plurality of event pulse samples, and the event pulse samples record visual image changes as pulses that appear at each pixel position at different times;

[0011] S2. Count the number of pulse occurrences at each pixel position in each event pulse sample, calculate the self-information of each pixel position according to the number of pulse occurrences, and obtain a spatial position self-information set, where the spatial position self-information set records the self-information size of each pixel position;

[0012] S3. Count the pulse occurrence frequencies at each pixel position in each event pulse sample, and respectively obtain a pulse occurrence frequency set for each event pulse sample;

[0013] S4. According to the spatial position self-information set and the pulse occurrence frequency set of each event pulse sample, combine the self-information and the frequency to obtain a spatial position importance set for each event pulse sample, where the spatial position importance set records the importance level of each pixel position in each event pulse sample;

[0014] S5. Divide the pixel positions of the event pulse samples into K pixel groups with a fixed number of pixels in both length and width, and each pixel group is a spatial unit;

[0015] S6. For each pulse, count the historical pulses that occurred within its spatial unit in the past Δt time to obtain the historical pulse sequence within the spatio-temporal neighborhood of this pulse;

[0016] S7. For each pulse, add up all the historical pulses in its historical pulse sequence combined with their spatial position importance to obtain the temporal surface feature of this pulse, and the temporal surface feature is used to represent the historical event pulse activity within the spatial neighborhood of a pulse.

[0017] As a preferred solution, step S2 specifically includes the following steps:

[0018] S21. For each of the event pulse samples, respectively count to obtain the pixel position pulse count set of each event pulse sample; the pixel position pulse count set records the number of times a pulse appears at each pixel position in each event pulse sample;

[0019] S22. Integrate the pulse counts at the same pixel positions from the pixel position pulse count sets of each event pulse sample, and calculate the self-information of each pixel position;

[0020] S23. Use the self-information of each pixel position to construct the spatial position self-information set.

[0021] As a further preferred solution, the calculation method of step S21 is specifically as follows:

[0022] Respectively set the pixel position pulse count set:

[0023]

[0024] where represents the number of times a pulse appears at the pixel position (i, j) in the nth event pulse sample; W represents the width of the event pulse sample space, and H represents the height of the event pulse sample space;

[0025] The calculation method of step S22 is specifically as follows:

[0026]

[0027] where I ij represents the self-information of each pixel position; N represents the total number of event pulse samples;

[0028] The specific content of step S23 is:

[0029] Use the above I ij to construct the spatial position self-information set SI:

[0030] SI = {I ij|i = 0, 1, ..., W - 1; j = 0, 1, ..., H - 1}.

[0031] As a preferred solution, the calculation method of step S3 is specifically as follows:

[0032] Construct a pulse occurrence frequency set Fre = {V n |n = 0, 1, ..., N - 1}; Fre is composed of N matrices V n constitute, where V n represents the matrix of pulse frequencies at each pixel position in the nth event pulse sample, represents the frequency of pulse occurrence at position (i, j).

[0033] As a preferred solution, the calculation method of step S4 is specifically as follows:

[0034]

[0035] where represents the importance of the pulse at pixel position (i, j) in the nth event pulse sample, I ij represents the spatial position self - information of the pulse at position (i, j), represents the frequency of pulse occurrence at position (i, j);

[0036] Then construct a spatial position importance set containing N matrices Q n :

[0037]

[0038] where Q n represents the spatial position importance matrix of the nth event pulse sample.

[0039] As a preferred solution, step S6 is specifically as follows:

[0040] For each pulse e k , count the historical pulses that occurred within its spatial unit in the past Δt time to obtain the historical pulse sequence within the spatio - temporal neighborhood of this pulse:

[0041]

[0042] where z l = z k +Δz represents the spatial range of pixel positions, p k represents the polarity of the pulse event, Δz ∈ [-ρ, ρ] 2 ..

[0043] t l represents the historical pulse e within the spatio - temporal neighborhood in the past Δt time lOccurrence time.

[0044] As a further preferred solution, step S7 is specifically:

[0045] For each pulse e k , according to e k in the position importance matrix Q of the sample where it is located n , for each historical pulse in the historical pulse sequence within the spatio-temporal neighborhood of e k , combine its position importance q l at the pixel position where it is located to calculate the time surface feature, and store the time surface feature in a matrix:

[0046]

[0047] where q1 represents the spatial position importance of the pixel position where the historical pulse e k within the spatio-temporal neighborhood of e l is located, and τ represents the decay constant.

[0048] Compared with the prior art, the beneficial effects of the present invention are:

[0049] The method of the present invention distinguishes the importance of different spatial positions while extracting time surface features for spiking neural networks, improves the accuracy of object extraction, and enhances the robustness against noise interference. Brief Description of the Drawings

[0050] Figure 1 is a flowchart for extracting time surface features and transmitting nerve impulses in an embodiment of the present invention;

[0051] Figure 2 is a flowchart for transmitting nerve impulses in an embodiment of the present invention. Specific Embodiments

[0052] In order to more clearly illustrate the embodiments of the present invention, the specific embodiments of the present invention will be described below with reference to the accompanying drawings. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings, and other embodiments can be obtained.

[0053] Embodiment: A method for extracting time surface features based on pulse events according to this embodiment aims at the defect that the existing time surface feature extraction method is insensitive to the spatial position of pulses, and determines the importance of each spatial position based on the spatial position where the pulse event occurs, thereby ignoring the noise pulse interference at unimportant positions.

[0054] The method of this embodiment includes the following steps:

[0055] S1. Obtain an event pulse sample set, where the event pulse sample set includes a number of event pulse samples. An event pulse sample records visual image changes as pulses that occur at various pixel positions at different times.

[0056] The event pulse samples are collected by an Address Event Representation (AER) camera. The AER camera can record the image change of a certain pixel point as the pulse information that appears at that point at a certain moment, and has the advantages of high time resolution and low power consumption, and is very suitable for object recognition tasks. The event pulse samples collected by the same AER camera have the same pixel space size, that is, the same number of pixels in length and width and pixel positions. In this embodiment, the pulse is in the form of e = {t, (x, y), p}, where t is the pulse occurrence time; (x, y) is the pulse occurrence position; p is the polarity of the pulse, and the value is -1 or 1.

[0057] S2. Count the number of pulse occurrences at each pixel position in each event pulse sample, and calculate the self-information of each pixel position according to the number of pulse occurrences to obtain a spatial position self-information set, which records the self-information size of each pixel position.

[0058] Furthermore, step S2 can be divided into the following steps:

[0059] S21. Respectively count the pixel position pulse count sets of each event pulse sample according to each event pulse sample; the pixel position pulse count set records the number of pulse occurrences at each pixel position in each event pulse sample.

[0060] S22. Integrate the pulse counts at the same pixel positions from the pixel position pulse count sets of each event pulse sample, and calculate the self-information of each pixel position.

[0061] S23. Use the self-information of each pixel position to construct a spatial position self-information set.

[0062] This embodiment details an implementation manner of steps S21 - S23:

[0063] S21. Set a matrix X with the same pixel space size as the event pulse sample n in a list imgCount, and the list includes n samples X n , that is, represent imgCount as

[0064] imgCount = {X n |n = 0, 1,..., N - 1};

[0065] And among them

[0066] Indicates the number of pulses occurring at position (i, j). N represents the number of samples, W represents the width of the sample space, and H represents the height of the sample space.

[0067] Suppose there are two event pulse samples, a and b, and the length and width of the pixel space are both 2. In sample a, the pulses occur 1, 2, 3, and 4 times at positions (0, 0), (0, 1), (1, 0), and (1, 1) respectively, and in sample b, the pulses occur 6, 5, 4, and 3 times at the same positions. Then and imgCount = {X a , X [[ID=D8]] b}.

[0068] S22. Integrate and calculate the number of pulses at each same pixel position in samples a and b to obtain the self-information of the pulses occurring at each pixel position.

[0069] The specific calculation formula for self-information is: where I ij represents the self-information of each pixel position; N represents the total number of event pulse samples. In this way, the self-information of the pulses occurring at each pixel position (i, j) is obtained, and the self-information of the pulses occurring at each pixel position (i, j) is constructed into a spatial position self-information set matrix with the same size as the pixel space:

[0070] SI = {I ij | i = 0, 1,..., W - 1; j = 0, 1,..., H - 1};

[0071] where W represents the width of the pixel space and H represents the height of the pixel space.

[0072] So in step S23, the spatial position self-information set obtained from the two event pulse samples a and b in the above S21 is the matrix

[0073] Then perform step S3: Statistically calculate the pulse occurrence frequency at each pixel position in each event pulse sample to obtain the pulse occurrence frequency set of each event pulse sample respectively;

[0074] The specific implementation method of step S3 can be:

[0075] Set up a matrix V n list Fre with the same size as the sample space of the event pulse sample, and use Fre as the pulse occurrence frequency set of each event pulse sample:

[0076] Fre = {V n | n = 0, 1,..., N - 1};

[0077]

[0078] wherein is the pulse occurrence frequency at pixel position (i, j), and V n is the pulse occurrence frequency at each pixel position (i, j) in the nth sample matrix.

[0079] Furthermore, according to the example of step S2 above, using imgCount = {X a , X b}, the obtained pulse occurrence frequency sets are respectively:[[]] Fre = {V a , V b}.

[0080] After completing steps S2 and S3, use the self-information of each pixel position obtained in steps S2 and S3 and the pulse occurrence frequency of each pixel position in each sample to perform step S4. According to the spatial position self-information set and the pulse occurrence frequency set of each event pulse sample, combine the self-information and frequency to obtain the spatial position importance set of each event pulse sample. The spatial position importance set records the importance level of each pixel position in each event pulse sample;

[0081] Spatial position importance is an importance measurement parameter based on the self-information of pixel positions and the probability of pulses occurring at pixel positions. The calculation method can be specifically:[[]]

[0082]

[0083] wherein represents the spatial position importance of the pulse at pixel position (i, j).

[0084] Combine the spatial position importances of all pixel positions on an event pulse sample to obtain the spatial position importance set matrix Q n :[[]]

[0085] Q n represents the spatial position importance matrix of the nth event pulse sample.

[0086] According to the above calculation method, substituting SI and Fre obtained in the specific examples of steps S2 and S3 above into step S4, the spatial position importance matrices of sample a and sample b can be obtained as respectively:[[]]

[0087]

[0088] S5. Divide the pixel positions of the event pulse sample into pixel groups with a fixed number of pixels in both length and width. Each pixel group is a spatial unit;

[0089] Specifically, in this embodiment, the pixel positions of the event pulse samples are divided into pixel groups with a side length of K×K according to the spatial scale, and each pixel group is called a spatial unit C.

[0090] S6. For each pulse, count the historical pulses that occurred in the spatial unit where it is located within the past Δt time, and obtain the historical pulse sequence of the temporal and spatial neighborhood of the pulse;

[0091] Each time a pulse e is received k The traversal occurs at Δt time before the pulse, and the statistics occur at time t l ∈[t k -Δt, t k ), and occurs in the Δz neighborhood near the pixel where the pulse is located, and e k All pulses in the same spatial unit C are pulses e k Historical pulse sequences in the temporal and spatial neighborhoods

[0092]

[0093] where e l :z l Indicates that the historical pulses e in the spatiotemporal neighborhood can be counted l The spatial range is z l , z l =z k +Δz represents the statistical range z l Pulse e k The pixel position z k The surrounding Δz region, Δz∈[-ρ,ρ] 2 . t l Indicates that the historical pulses e in the spatiotemporal neighborhood can be counted l The time range [t k -Δt,t k ).p l Indicates pulse e l polarity.

[0094] Specifically, assuming ρ = 1, K = 3, W = H = 3, events e1 = {(1, 0), 1, 0}, e2 = {(0, 1), 2, 0}, e3 = {(1, 1), 2.5, 0}, if the time window Δt = 1, then when the pulse e3 arrives, the historical pulses in the spatiotemporal neighborhood of e3 are the pulses that occurred around (1, 1) within the time Δt before e3, that is,

[0095] Then, step S7 is performed. For each pulse, all the historical pulses in its historical pulse sequence are combined with their spatial position importance and added together to obtain the temporal surface feature of the pulse. The temporal surface feature is used to represent the historical event pulse activities within the spatial neighborhood of a pulse.

[0096] Specifically, for each pulse e k , according to the position importance matrix Q k of the sample where e n is located, each historical pulse within the spatio-temporal neighborhood of e k is combined with the position importance of its pixel position to calculate the temporal surface feature of e k and store the temporal surface feature of e in a matrix: k The temporal surface feature of e is stored in the matrix:

[0097]

[0098] In the above formula, q l represents the position importance of each historical pulse within the spatio-temporal neighborhood of e k , τ represents the time decay constant, and the calculation result is stored in a matrix of size (2×ρ + 1) 2 .

[0099] Calculate the temporal surface feature of e3 using the spatial position importance matrix obtained by the above step S4 and the neighborhood historical pulse sequence statistically obtained in step S6. Assume that the spatial position importance of pulse e3 is According to the above formula, it can be calculated that That is, the position importance q2 of pulse e2 = 2, the pulse occurrence time t2 = 2; the position importance q3 of pulse e3 = 2, the pulse occurrence time t3 = 2.5, and The accumulation positions are the positions of e3 - e2 and e3 - e3, that is, (1, 1) - (0, 1) = (1, 0) and (1, 1) - (1, 1) = (0, 0).

[0100] Then The temporal surface feature in unit C finally obtained is

[0101] The temporal surface feature extracted by the above method takes into account the different spatial importance of different positions in different samples and can better avoid noise interference. After obtaining the temporal surface feature in step S7, the temporal surface feature can be used to transmit pulses to neurons in the pulse neural network. Its complete flowchart is as Figure 1 shown.

[0102] Further, the method for generating pulses is as Figure 2 shown. The neuron membrane potential can be accumulated and a pulse is sent to the next layer when the threshold is reached. According to the above example, assuming the threshold is 0.3, when e3 arrives, the neuron membrane potentials at positions (0, 1) and (0, 0) can reach the threshold and send pulses. Thus, the pulse information of position (1, 0), time 2, polarity 0; position (0, 0), time 2.5, polarity 0 is transmitted to the next layer, and the entire unit is reset to

[0103] It should be noted that the above embodiments only elaborate in detail on the preferred embodiments and principles of the present invention. For those of ordinary skill in the art, according to the idea provided by the present invention, there will be changes in the specific implementation manners, and these changes should also be regarded as the protection scope of the present invention.

Claims

1. A method for extracting time surface features based on pulse events, characterized in that, The method includes the steps of: S1. Obtain an event pulse sample set, where the event pulse sample set includes a number of event pulse samples, and each event pulse sample records visual image changes as pulses that appear at each pixel position at different times; S2. Count the number of times a pulse appears at each pixel position in each event pulse sample, calculate the self-information of each pixel position based on the number of times the pulse appears, and obtain a spatial position self-information set, where the spatial position self-information set records the size of the self-information of each pixel position; S3. Count the occurrence frequency of pulses at each pixel position in each event pulse sample to obtain a pulse occurrence frequency set for each event pulse sample respectively; S4. According to the spatial position self-information set and the pulse occurrence frequency set of each event pulse sample, combine the self-information and the frequency to obtain a spatial position importance set for each event pulse sample, where the spatial position importance set records the level of importance of each pixel position in each event pulse sample; S5. Divide the pixel positions of the event pulse samples into K pixel groups with a fixed number of pixels in both length and width, and each pixel group is a spatial unit; S6. For each pulse, count the historical pulses that occurred in a certain range near it and in the same spatial unit within the past Δt time to obtain a historical pulse sequence for this pulse; S7. For each pulse, add up all the historical pulses in its historical pulse sequence combined with its spatial position importance to obtain the temporal surface feature of this pulse, and the temporal surface feature is used to represent the historical event pulse activities within the spatial neighborhood of a pulse; The specific steps of step S2 include the following steps: S21. Statistically obtain a pixel position pulse count set for each event pulse sample respectively; the pixel position pulse count set records the number of times a pulse appears at each pixel position in each event pulse sample; S22. Integrate the pulse counts at the same pixel position from the pixel position pulse count sets of each event pulse sample, and calculate the self-information of each pixel position; S23. Use the self-information of each pixel position to construct a spatial position self-information set; The specific calculation method of step S21 is: Set the pixel position pulse number sets respectively Among them represents the number of times a pulse appears at the pixel position (i, j) in the nth event pulse sample; W represents the width of the event pulse sample space, and H represents the height of the event pulse sample space; The specific calculation method of step S22 is: where I ij represents the spatial position self-information of the pulse at each pixel position; N represents the total number of event pulse samples; The specific content of step S23 is: Using the above I ij Construct the spatial position self-information set SI: SI = {I ij | i = 0, 1, …, W - 1; j = 0, 1, …, H - 1}.

2. The method for extracting temporal surface features based on pulse events according to claim 1, wherein The specific calculation method of step S3 is: Construct the pulse occurrence frequency set Fre = {V n | n = 0, 1,..., N - 1}; Fre is composed of N matrices V n . Among them, V n represents the matrix of the pulse frequencies at each pixel position in the nth event pulse sample, and represents the frequency of pulse occurrence at the position (i, j).

3. A method for extracting temporal surface features based on pulse events according to claim 1, characterized in that, The specific calculation method of step S4 is: Among them represents the importance of the pulse at the pixel position (i, j) in the nth event pulse sample, I ij represents the spatial position self-information of the pulse at the position (i, j), represents the frequency of the pulse appearance at the position (i, j); Then construct a set of spatial position importance containing N matrices Q n : where Q n represents the spatial position importance matrix of the nth event pulse sample.

4. A method for extracting temporal surface features based on pulse events according to claim 1, characterized in that, The specific content of step S6 is: For each pulse e k , count the historical pulses that occurred within its spatial unit in the past Δt time to obtain the historical pulse sequence within the spatio-temporal neighborhood of this pulse: where z l = z k + Δz represents the pixel position spatial range, p k represents the polarity of the pulse event, Δz ∈ [-ρ, ρ] 2 ; t l represents the historical pulse e within the spatio-temporal neighborhood at the past time Δt l Occurrence time.

5. A method for extracting time surface features based on pulse events according to claim 4, characterized in that The specific content of step S7 is: For each pulse e k , according to e k in the position importance matrix Q of the sample where it is located n , for each historical pulse in the historical pulse sequence within the spatio-temporal neighborhood of e k , combine its position importance q at its pixel position l to calculate the temporal surface feature, and store the temporal surface feature in the matrix: where q l represents e k the historical pulse e within the spatio-temporal neighborhood l the spatial position importance of the pixel position where it is located, and τ represents the decay constant.

Images