An event stream data classification method based on pulse neural network

Abstract

The application discloses an event stream data classification method based on a pulse neural network, which comprises four steps of event stream data preprocessing, generating corresponding neuron excitation states in a memory generation module according to input pulse signals, extracting a neuron group of a corresponding category and interpreting the neuron group of the corresponding mode, wherein a pulse neuron model uses a LIF neuron model, and an effective network structure for processing event stream data is proposed. The method effectively combines the strong biological interpretability of event stream data and the pulse neural network, can more efficiently utilize the event stream data for classification and recognition, can effectively improve the training efficiency, and achieves a better recognition accuracy.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence and image classification technology, and specifically relates to an event stream data classification method based on spiking neural networks. Background Technology

[0002] In today's digital age, image classification plays a crucial role in the field of computer vision. With the continuous emergence of various types of image data, accurate and efficient classification and recognition of these images has become a core requirement in many application areas, such as autonomous driving, medical diagnosis, and security monitoring. Success in image classification tasks is of great significance in driving innovation and progress across various industries.

[0003] Meanwhile, with the rapid development of image acquisition technology and equipment, a large amount of event stream data is being collected and transmitted. Event stream data is acquired by event cameras, which can efficiently perceive dynamic scenes, especially suitable for rapidly changing environments. By classifying event stream data, useful information can be extracted from high-speed, large-volume data streams, enabling more efficient perception and processing; event cameras consume less power than ordinary cameras, enabling rapid identification and decision-making in dynamic environments. However, due to the special format of event stream data, its classification differs significantly from traditional data classification tasks, presenting considerable challenges. Summary of the Invention

[0004] To overcome the shortcomings of the prior art, the present invention aims to provide an event stream data classification method based on spiking neural networks. This method seeks to address the limitations of traditional methods on event stream data by applying spiking neural networks to image classification tasks on event stream datasets. By fully utilizing the biological interpretability, temporal information processing capabilities, and computational efficiency of spiking neural networks, new meaning can be given to the application of image classification tasks on event stream datasets, further promoting the development and innovation of this field.

[0005] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0006] A pattern recognition method and system based on a spiking neural network includes the following steps:

[0007] Step 1: Event stream data preprocessing to obtain pulse signals and location data; the event stream data is data collected by the event camera, recording events that occur at different times and locations, including relevant event information; the pulse signal is the time data in the preprocessed dataset, which appears in groups with the location data in the preprocessed dataset, where the time data and location data in a group together represent the occurrence of a certain event;

[0008] Step 2: Input the pulse signal into the spiking neural network according to the position data, and the memory generation module learns to generate the corresponding neuron excitation state to complete the training of the spiking neural network;

[0009] Step 3: Extract neuronal clusters corresponding to different categories based on neuronal firing states;

[0010] Step 4: After preprocessing the event stream data to be classified, input it into the spiking neural network trained in Step 2 for recognition. By interpreting the firing status of the corresponding neuron clusters, the classification task is completed.

[0011] In another aspect, the present invention provides a computer program product comprising a computer program that, when executed by a processing unit, implements the steps of the method described.

[0012] In another aspect, the present invention provides a computing device including a storage unit and a processing unit, wherein the storage unit stores a computer program, and the processing unit executes the computer program to implement the steps of the method described.

[0013] In another aspect, the present invention provides an event stream data classification method based on a spiking neural network, including a display device and the aforementioned computing device;

[0014] The computing device includes a processing unit comprising a preprocessing module for performing preprocessing, a training module for performing network training, and a computation module for performing transformation; the preprocessing module takes the event stream data to be identified as input, and the computation module takes the preprocessed event stream data to be identified as input and outputs the obtained classification result;

[0015] The display device displays the classification results of the event stream data.

[0016] Compared with the prior art, the beneficial effects of the present invention are:

[0017] 1. A network construction and training model based on biomimetic design can be effectively applied to the classification problem of event stream datasets.

[0018] 2. The data structure of the event stream dataset is highly compatible with the features of the spiking neural network used, both of which have strong biological interpretability. Combining the two can achieve better classification results.

[0019] 3. By leveraging the low energy consumption and high energy efficiency of spiking neural networks, the network parameters and training costs are greatly reduced compared to traditional deep learning models.

[0020] 4. Due to the sparse firing of spiking neurons, the overall computation speed is accelerated, and classification results can be given faster and more accurately in the classification task of event stream datasets. Attached Figure Description

[0021] Figure 1 This is the overall flowchart of the event stream data classification method based on spiking neural networks proposed in this invention.

[0022] Figure 2 This is a flowchart of the event stream dataset sparsity processing proposed in this invention.

[0023] Figure 3 This is a flowchart of a hierarchical cumulative growth connection algorithm.

[0024] Figure 4 This is a flowchart of a pulse density-based recognition algorithm. Detailed Implementation

[0025] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings and examples.

[0026] This invention discloses an event stream data classification method based on spiking neural networks (SNNs), comprising four main steps: event stream data preprocessing, generating corresponding neuronal excitation states in a memory generation module based on input pulse signals, extracting neuronal clusters of corresponding categories, and interpreting neuronal clusters of corresponding patterns. Spiking neural networks (SNNs), due to their biological neuron simulation characteristics and their closer resemblance to the information processing process of the real human brain, are expected to solve more artificial intelligence problems. Unlike traditional neural networks, spiking neural networks simulate the pulse transmission process of brain neurons, making them excellent in processing temporal information and dynamic pattern recognition. Another significant advantage of spiking neural networks lies in their sparse coding method, i.e., computation is only performed when neuronal activity changes, significantly reducing computational costs and energy consumption. This makes spiking neural networks highly promising for processing large-scale event stream datasets. In this invention, the spiking neuron model can utilize the LIF neuron model, proposing an effective network structure for processing event stream data. This invention effectively combines the strong biological interpretability of event stream data and spiking neural networks, demonstrating excellent performance in event stream data feature extraction, effectively improving training efficiency, and achieving good recognition accuracy.

[0027] Specifically, such as Figure 1 As shown, the event stream data classification method based on spiking neural networks of this invention mainly includes the following steps:

[0028] Step 1: Event stream data preprocessing to obtain pulse signals and position data.

[0029] In this invention, event stream data refers to data collected by an event camera, recording events occurring at different times and locations, including relevant information such as event timestamps. The resulting pulse signals are time data within the processed dataset. This time data appears in groups with the location data within the processed dataset, where the time data and location data within a group together represent the occurrence of an event at a specific point.

[0030] In this embodiment of the invention, the event stream dataset used is the DVS128 Gesture dataset, and the event stream data preprocessing process is as follows:

[0031] Step 1.1: Segmentation and organization of raw event stream data.

[0032] In this step, the segmentation method is as follows: First, the original event stream dataset, i.e., the DVS128 Gesture dataset, is segmented according to the annotation file requirements. For example, all file names ending with .aedat are read, and the training set and test set are divided according to the numbers in the file names. The data of the .aedat files within the specified number range is read and converted into an events array with elements [x,y,p,t]. Events are selected from the events array according to the time range in the CSV file, that is, the eventType, eventCapacity, eventSize, etc. are sliced ​​according to the header format of the corresponding CSV file. These are converted into files of a specific format and stored independently to make each action independent. Then, the segmented event stream data is organized by category.

[0033] In this step, after data preprocessing, a total of 1176 training data and 288 test data were obtained. This step addresses the issue of file collection format in event stream datasets that is not conducive to subsequent processing, i.e., a single file contains information about a subject completing all gestures under a certain lighting condition. This approach is more beneficial for subsequent operations.

[0034] In this step, the method for sorting is to remove non-time and non-location data, such as device information data, from the dataset.

[0035] Step 1.2: Perform sparsification processing on the processed event stream data, which is divided into two processes: event sparsification and time sparsification, to obtain processed time data and location data.

[0036] like Figure 2 As shown, event sparsification involves sampling the original event stream data after step 1.1, retaining only adjacent n*n ( Figure 2In the n-th pixel selection, one of the events involved is taken, and only 1 / m of the corresponding data of that pixel is used for subsequent processing. Temporal sparsity is achieved by scaling the event-sparsed data on a time scale, merging adjacent time points after scaling into a single time point, and arranging them in chronological order to form a time series.

[0037] In this step, the event stream data becomes sparser after sparsification, significantly reducing file size and optimizing the overall process runtime.

[0038] Step 1.3: Generation of pulse signals. The processed time data obtained in Step 1.2 is used as a pulse excitation time sequence and as the pulse signal to be input into the spiking neural network; each processed event stream data will generate a pulse excitation time sequence.

[0039] In this step, the information contained in the event stream dataset is converted into pulses, thus completing the introduction of the event stream dataset into the spiking neural network.

[0040] Step 2: Input the pulse signal into the spiking neural network based on the position data obtained in Step 1. The memory generation module learns and generates the corresponding neuron excitation state to complete the training of the spiking neural network.

[0041] In this invention, the spiking neuron model uses the LIF neuron model. The spiking neural network consists of an input layer, hidden layers, and an output layer, with each layer composed of several LIF neuron models. The memory generation module of this invention is constructed within the spiking neural network, including all input layers and hidden layers in the network; that is, the input layers and the hidden layers constitute the memory generation module. The input layer introduces the event stream dataset processed in step 1 into the spiking neural network, causing the corresponding input neurons to generate pulses, which are then transmitted to the hidden layers for information processing and transformation.

[0042] In the memory generation module, the input layer and hidden layer use a many-to-one connection method, that is, each neuron in the hidden layer randomly selects n neurons from the input layer. inp Connecting to n neurons; where n inp The determination method is as follows:

[0043]

[0044] Where, n i n represents the number of neurons in the input layer. h This indicates the number of neurons in the hidden layer.

[0045] In one embodiment, the training process in this step is as follows:

[0046] Step 2.1: Set network training parameters. In this embodiment, the number of nodes hn = 1100, the excitation threshold st = -63.0, the refractory period rp = 25.0, and the maximum number of synaptic connections per data class sle = 120.

[0047] Step 2.2: Based on the location data obtained in Step 1, i.e. the input neuron number corresponding to each event stream data, the pulse signal corresponding to a certain data in a certain type of event stream data is transmitted to the hidden layer through a many-to-one connection between the input layer and the hidden layer.

[0048] Step 2.3: For the pulse signal transmitted from the input layer, each neuron in the hidden layer accumulates weights within the effective time difference and forms new synaptic connections.

[0049] like Figure 3 As shown, in this embodiment, weight accumulation is performed based on a hierarchical cumulative growth connection algorithm. The hierarchical cumulative growth connection algorithm considers not only neurons in the activated state but also neurons to be activated. It can generate new synaptic connections based on the logical relationship between two neurons that have not yet formed synaptic connections, thereby strengthening this logical relationship. In addition, the hierarchical cumulative growth connection algorithm accumulates weights for neurons in the hidden layer according to their activation batch numbers, and only when the number of a neuron that has not established a relationship with neuron Neur is greater than neuron Neur can weights be accumulated.

[0050] In the hierarchical cumulative growth connection algorithm, the time difference is the time difference between two neurons that can establish a new synaptic connection (i.e., two neurons that have not yet formed a synaptic connection) when they reach the peak point. The algorithm considers a time difference between 1ms and 10ms to be an effective time difference. The functional relationship between the weights available for accumulation within the effective time difference and the time difference is as follows:

[0051]

[0052] Where ω is the weight that can be accumulated, Δt is the time difference within the effective time, and the unit of Δt is ms.

[0053] Step 2.4: Through the transmission of the new synaptic connections, some neurons in the hidden layer reach an excited state.

[0054] Step 2.5: Replace the data within this category and repeat steps 2.2 to 2.4 until the number of hidden layer connections reaches the maximum number of connections sle in the set network training parameters. At this point, the training for this type of event data is complete.

[0055] Step 2.6: Change the data category and repeat steps 2.1 to 2.5 above until training for each category of data is complete.

[0056] Step 3: Extract neuronal clusters corresponding to different categories based on neuronal firing states.

[0057] In one embodiment, the neuron clusters corresponding to different classifications in this step are: a special group of neurons composed of multiple neuron nodes in the spiking neural network after step 2, i.e., after training is completed. It is a local topology formed by the top v neurons in the hidden layer that are fired the most times under a certain type of event stream data input.

[0058] In this step, the obtained pulse signal is fed into the neural network. After learning by the memory generation module, the relevant information of the event stream dataset is stored in the spiking neural network through continuous optimization and adjustment.

[0059] In one embodiment, the extraction process for neuron clusters corresponding to different categories in this step is as follows:

[0060] Step 3.1: Initialize the neuron cluster parameters.

[0061] Step 3.2: Based on the activation status of the hidden layer neurons of the corresponding category in Step 2, sort the activated neurons according to the number of activations, and add the v neurons with the most activations to the neuron cluster of that category.

[0062] Step 3.3: Repeat steps 3.1 to 3.2 to obtain different clusters of hidden layer neurons when each category of event stream data is input.

[0063] Step 3.4: Identify common neurons in each category of neuronal clusters, increase the activation threshold of common neurons, and decrease the activation threshold of non-repetitive neurons.

[0064] In this step, neuron clusters of various categories were extracted. Neuron clusters play a crucial role in spiking neural networks because they embody the network's intelligent logic in responding to external information. This process is key to the learning and memory capabilities of spiking neural networks and is essential for improving the network's accuracy and efficiency.

[0065] Step 4: After preprocessing the event stream data to be classified, input it into the spiking neural network trained in Step 2 for recognition. By interpreting the firing status of the corresponding neuron clusters, the classification task is completed.

[0066] In some embodiments of the present invention, after the data to be classified is preprocessed, it is input into a trained spiking neural network for recognition. By inputting the event stream data to be classified into the spiking neural network, the excitation information of neurons in the hidden layer and the test layer can be obtained, the excitation density of each neuron cluster can be calculated and compared, and the neuron cluster with the highest excitation density can be selected as the network output result.

[0067] like Figure 4 As shown, the classification process is as follows:

[0068] Step 4.1: Input the test set into the spiking neural network and obtain the activation information of neurons in the hidden and output layers.

[0069] Step 4.2: Traverse all neuron clusters, obtain the absolute size of each neuron cluster, and calculate the absolute number of firings for each neuron cluster. The method for determining the absolute size of a neuron cluster is shown in Equation 2 below, V mass a represents the absolute size of the neuron cluster. i This represents the number of neurons shared by i neuron clusters within a neuron cluster. The absolute number of firings is determined as shown in Equation 3 below, where n represents the number of neurons shared by n neuron clusters.

[0070]

[0071]

[0072] Step 4.3: Calculate and compare the excitation density of each neuronal cluster. The method for determining the excitation density ρ of the neuronal cluster is shown in Equation 4.

[0073]

[0074] Step 4.4: Select the cluster of neurons with the highest excitation density as the network output result.

[0075] In some embodiments of the present invention, after preprocessing the data to be classified, it is input into a trained spiking neural network for recognition. The required spatiotemporal representation data of the hidden layer can be obtained through the neuron clusters obtained in step 3. Here, spatiotemporal representation data refers to a combination of data that covers the more important time and space information in the processed input dataset. The obtained spatiotemporal representation data of the hidden layer is then input into an existing classifier for classification, and the classification result is output. When selecting the spatiotemporal representation data of the hidden layer, one can choose to obtain the neuron activation information of all hidden layers, or one can choose to obtain the neuron activation information within the neuron cluster or the neuron activation information outside the neuron cluster. Selecting different ranges of spatiotemporal representation data corresponds to different degrees of feature extraction of the dataset by the spiking neural network, and these data, when input into the existing classification model, will also correspond to different classification results. In this embodiment, selecting spatiotemporal data within and outside the neuron cluster as feature inputs to the SVM classifier has achieved very good classification results.

[0076] The display device is used to display the classification results of the event stream dataset.

[0077] To test the effectiveness of this invention, the original data from the DVS128 Gesture dataset was used as the test input. The method described in this invention yielded good classification results. The DVS128 Gesture dataset contains 11 pose classes, which are numbered 0-10 for ease of presentation. The classification results obtained through different methods involved in this invention are shown in the table below:

[0078]

[0079] In summary, this invention consists of four steps: preprocessing event stream data, generating corresponding neuronal excitation states in the memory generation module based on the input pulse signals, extracting neuronal clusters of corresponding categories, and interpreting neuronal clusters of corresponding patterns. It can be used for classification of event stream datasets and can accurately perform classification tasks. This method effectively combines the strong biological interpretability of event stream data and spiking neural networks, enabling more efficient use of event stream data for classification and recognition, effectively improving training efficiency, and achieving better recognition accuracy.

[0080] The above description is merely a preferred embodiment of the present invention in the specific problem of image classification. The present invention should not be limited to the problem form, implementation examples, and content disclosed in the accompanying drawings. Any equivalent or modified embodiments that do not depart from the spirit of the present invention should fall within the protection scope of the present invention.

Claims

1. A method for classifying event stream data based on a spiking neural network, characterized in that, Includes the following steps: Step 1: Event stream data preprocessing to obtain pulse signals and location data; the event stream data is data collected by the event camera, recording events that occur at different times and locations, including relevant event information; the pulse signal is the time data in the preprocessed dataset, which appears in groups with the location data in the preprocessed dataset, where the time data and location data in a group together represent the occurrence of a certain event; Step 2: Input the pulse signal into the spiking neural network according to the position data, and the memory generation module learns to generate the corresponding neuron excitation state to complete the training of the spiking neural network; Step 3: Extract neuronal clusters corresponding to different categories based on neuronal firing states; Step 4: After preprocessing the event stream data to be classified, input it into the spiking neural network trained in Step 2 for recognition. By interpreting the firing status of the neuron clusters of the corresponding categories, the classification task is completed. Step 1, the event stream data preprocessing process, is as follows: Step 1.1: Segment and organize the raw event stream data; Step 1.2: Perform sparsification on the processed event stream data, which is divided into two processes: event sparsification and time sparsification, to obtain the processed time data and location data; Step 1.3: The time data is a pulse excitation time series, which serves as the input pulse signal for the spiking neural network; Each processed event stream data will generate a pulse excitation time sequence; In the memory generation module, the input layer and hidden layer adopt a many-to-one connection method, that is, the neurons of each hidden layer are randomly selected from the input layer. 1 neuron establishes a connection with it; among which The determination method is as follows: in, This indicates the number of neurons in the input layer. Indicates the number of neurons in the hidden layer; Step 2, the training process is as follows: Step 2.1: Set network training parameters; Step 2.2: Based on the location data obtained in Step 1, i.e. the input neuron number corresponding to each event stream data, the pulse signal corresponding to a certain data in a certain type of event stream data is transmitted to the hidden layer through a many-to-one connection between the input layer and the hidden layer; Step 2.3: For the pulse signal transmitted from the input layer, each neuron in the hidden layer accumulates weights within the effective time difference and forms new synaptic connections; Step 2.4: Through the transmission of the new synaptic connections, some neurons in the hidden layer reach an excited state; Step 2.5: Replace the data within this class and repeat steps 2.2 to 2.4 until the number of hidden layer connections reaches the maximum number of connections set in the network training parameters. At this point, the training of this class of event data is complete. Step 2.6: Repeat steps 2.1 to 2.5 above by changing the data category until training for all data categories is complete; In step 2.3, weight accumulation is performed according to the hierarchical cumulative growth connection algorithm. The hierarchical cumulative growth connection algorithm generates new synaptic connections based on the logical relationship between two neurons that have not yet formed synaptic connections, so as to strengthen the logical relationship. The hidden layer neurons are numbered according to the activation batch number, and weight accumulation can only be performed when the number of the neuron that has not established a relationship with neuron Neur is greater than neuron Neur. In the hierarchical cumulative growth connection algorithm, the time difference is the time difference between two neurons that can establish a new synaptic connection (i.e., two neurons that have not yet formed a synaptic connection) when they reach the peak point. The algorithm considers a time difference between 1ms and 10ms to be an effective time difference. The functional relationship between the weights that can be used for accumulation within the effective time difference and the time difference is as follows: in For the weights that can be used for accumulation, The time difference within the effective time period, where The unit is ms; Step 3, the extraction process is as follows: Step 3.1: Initialize the neuron cluster parameters; Step 3.2: Based on the activation status of the hidden layer neurons of the corresponding category in Step 2, sort the activated neurons according to the number of activations, and add the v neurons with the most activations to the neuron cluster of that category; Step 3.3: Repeat steps 3.1 to 3.2 to obtain different clusters of hidden layer neurons when inputting event stream data for each category; Step 3.4: Identify common neurons in each category of neuronal clusters, increase the activation threshold of common neurons, and decrease the activation threshold of non-repetitive neurons.

2. The event stream data classification method based on spiking neural networks according to claim 1, characterized in that, Step 1.1 involves segmenting the original event stream data by dividing the .aedat files in the event stream dataset according to the annotation file requirements; and organizing the original event stream data by removing non-time and non-location data from the dataset. In step 1.2, the event sparsification method specifically involves sampling the original event stream data processed in step 1.1, retaining only the event involved in one of the adjacent n*n pixels, and taking only 1 / m of the data corresponding to that pixel for subsequent processing; the time sparsification method specifically involves scaling the event sparsified data on a time scale, merging adjacent time points after scaling into a single time point, and arranging them into a time series in chronological order.

3. The event stream data classification method based on spiking neural networks according to claim 1, characterized in that, In step 2, the spiking neural network consists of an input layer, a hidden layer, and an output layer, with each layer consisting of several LIF neuron models. The input layer and the hidden layer together form the memory generation module. The input layer introduces the event stream dataset processed in step 1 into the spiking neural network, causing the corresponding input neurons to generate pulses, which are then transmitted to the hidden layer for information processing and conversion.

4. The event stream data classification method based on spiking neural networks according to claim 1, characterized in that, In step 3, the neuron clusters corresponding to different categories are the neuron populations in the spiking neural network after training. The neuron population is composed of multiple neuron nodes and is a local topology formed by the top v neurons in the hidden layer that are fired the most times under a certain type of event stream data input.

5. The event stream data classification method based on spiking neural networks according to claim 1, characterized in that, In step 4, the event stream data to be classified is input into a spiking neural network to obtain the excitation information of neurons in the hidden layer and output layer, the excitation density of each neuron cluster is calculated and compared, and the neuron cluster with the highest excitation density is selected as the network output result; the required spatiotemporal representation data of the hidden layer is obtained through the neuron clusters obtained in step 3, and the obtained spatiotemporal representation data of the hidden layer is input into the classifier for classification, and the classification result is output.

Images