Asynchronous event stream processing method and device based on graph model and storage medium

Abstract

The invention relates to the technical field of computer visual recognition, in particular to an asynchronous event stream processing method and device based on a graph model and a storage medium. The method comprises the following steps: sampling an ambient light intensity change value through an event camera, and outputting to obtain an asynchronous event stream, the asynchronous event stream being used for indicating light intensity changes corresponding to a plurality of asynchronously sampled pixels; and feature learning is carried out on the asynchronous event flow through a graph convolutional network model, feature data is output and obtained, the graph convolutional network model is used for indicating setting of parameters of an asynchronous event flow generation graph according to a connectivity index, and the connectivity index indicates connectivity of the graph. According to the embodiment of the invention, the connectivity index is provided for describing the connectivity of the graph, the parameter setting of the asynchronous event flow generation graph is guided through the connectivity, the scale of the generation graph, the number of nodes and the number of edges are controlled, data preparation is made for an end-to-end deep learning algorithm, and the processing effect of the asynchronous event flow based on the graph model is improved.

Description

Technical Field

[0001] The present disclosure relates to the field of computer vision recognition technology, and in particular to a method, device and storage medium for asynchronous event stream processing based on a graph model. Background Art

[0002] Neuromorphic engineering is a work that uses large-scale integrated circuit technology to simulate biological nervous systems. Neuromorphic vision sensing (NVS) draws on the neural network structure and visual information sampling and processing mechanism of biological visual systems, and is an important part of its visual perception system.

[0003] Different from traditional frame-based cameras, NVS asynchronously collects information about changes in light intensity in the environment, so it has the characteristics of high temporal resolution and high dynamic range. However, NVS cameras output asynchronous event streams, which are presented as sparse, discrete dot matrices in three-dimensional space. For such asynchronous event streams, current image processing-based deep learning algorithms cannot be directly transferred and applied. How to learn features of asynchronous event streams is an important and core issue in neuromorphic engineering, and related technologies have not yet provided a reasonable and effective processing method. Summary of the invention

[0004] In view of this, the present disclosure proposes a method, device and storage medium for processing asynchronous event streams based on a graph model. The technical solution includes:

[0005] According to one aspect of the present disclosure, a method for processing an asynchronous event stream based on a graph model is provided, the method comprising:

[0006] The ambient light intensity change value is sampled by an event camera, and an asynchronous event stream is outputted, where the asynchronous event stream is used to indicate the light intensity change corresponding to each of the multiple pixels sampled asynchronously;

[0007] The asynchronous event stream is subjected to feature learning through a Graph Convolutional Networks (GCN) model, and feature data is outputted. The Graph Convolutional Network model is used to indicate setting parameters of a graph generated by the asynchronous event stream according to a connectivity index, and the connectivity index indicates connectivity of the graph.

[0008] In a possible implementation, the asynchronous event stream includes multiple asynchronous events, and the event indicates the coordinate position, timestamp and event polarity of the pixel generating the event, and the event polarity indicates an increase or decrease in light intensity corresponding to the pixel.

[0009] In another possible implementation, the method further includes:

[0010] Convert each event in the asynchronous event stream into a node in the graph;

[0011] When the distance between two nodes in the graph is less than a preset distance, the two nodes are connected using an edge;

[0012] The ratio of the number of edges to the number of nodes in the graph is determined as the connectivity index.

[0013] In another possible implementation, the distance between the two nodes is the weighted Euclidean distance between the two nodes, the preset distance is the distance between the node and the neighbor node farthest from the node, and the neighbor node is a node within a circular neighborhood of a target radius with the node as the center.

[0014] In another possible implementation, the parameters of the asynchronous event flow generation graph include at least one of the number of events, neighborhood radius distance and maximum connectivity, wherein the neighborhood radius distance is the distance between the node and the neighboring node farthest from the node, the neighboring nodes are nodes within a circular neighborhood of a target radius with the node as the center, and the maximum connectivity is the maximum number of connections for each of the nodes.

[0015] In another possible implementation, the number of events is logarithmically related to the connectivity index.

[0016] In another possible implementation, the method further includes:

[0017] Setting the depth of the graph convolutional network model according to the connectivity index; and / or,

[0018] The pooling scale of each pooling layer of the graph convolutional network model is set according to the connectivity index.

[0019] According to another aspect of the present disclosure, a graph model-based asynchronous event stream processing device is provided, the device comprising:

[0020] A first output module is used to sample the ambient light intensity change value through an event camera, and output an asynchronous event stream, wherein the asynchronous event stream is used to indicate the light intensity change corresponding to each of the multiple pixels sampled asynchronously;

[0021] The second output module is used to perform feature learning on the asynchronous event stream through a graph convolutional network model and output feature data. The graph convolutional network model is used to indicate the parameters of the asynchronous event stream generation graph to be set according to a connectivity index, and the connectivity index indicates the connectivity of the graph.

[0022] In a possible implementation, the asynchronous event stream includes multiple asynchronous events, and the event indicates the coordinate position, timestamp and event polarity of the pixel generating the event, and the event polarity indicates an increase or decrease in light intensity corresponding to the pixel.

[0023] In another possible implementation, the apparatus further includes: a determining module, configured to:

[0024] Convert each event in the asynchronous event stream into a node in the graph;

[0025] When the distance between two nodes in the graph is less than a preset distance, the two nodes are connected using an edge;

[0026] The ratio of the number of edges to the number of nodes in the graph is determined as the connectivity index.

[0027] In another possible implementation, the distance between the two nodes is the weighted Euclidean distance between the two nodes, the preset distance is the distance between the node and the neighbor node farthest from the node, and the neighbor node is a node within a circular neighborhood of a target radius with the node as the center.

[0028] In another possible implementation, the parameters of the asynchronous event flow generation graph include at least one of the number of events, neighborhood radius distance and maximum connectivity, wherein the neighborhood radius distance is the distance between the node and the neighboring node farthest from the node, the neighboring nodes are nodes within a circular neighborhood of a target radius with the node as the center, and the maximum connectivity is the maximum number of connections for each of the nodes.

[0029] In another possible implementation, the number of events is logarithmically related to the connectivity index.

[0030] In another possible implementation, the device further includes: a setting module, configured to:

[0031] Setting the depth of the graph convolutional network model according to the connectivity index; and / or,

[0032] The pooling scale of each pooling layer of the graph convolutional network model is set according to the connectivity index.

[0033] According to another aspect of the present disclosure, a computing device is provided, the computing device comprising: a processor; a memory for storing processor-executable instructions;

[0034] Wherein, the processor is configured to:

[0035] The ambient light intensity change value is sampled by an event camera, and an asynchronous event stream is outputted, where the asynchronous event stream is used to indicate the light intensity change corresponding to each of the multiple pixels sampled asynchronously;

[0036] The asynchronous event stream is subjected to feature learning through a graph convolutional network model, and feature data is outputted. The graph convolutional network model is used to indicate setting parameters of a graph generated by the asynchronous event stream according to a connectivity index, and the connectivity index indicates connectivity of the graph.

[0037] According to another aspect of the present disclosure, a non-volatile computer-readable storage medium is provided, on which computer program instructions are stored, characterized in that when the computer program instructions are executed by a processor, the method provided by the first aspect or any possible implementation manner of the first aspect is implemented.

[0038] The disclosed embodiment samples the ambient light intensity change value through an event camera, and outputs an asynchronous event stream, which is used to indicate the light intensity change corresponding to each of the multiple pixels sampled asynchronously; the asynchronous event stream is feature learned through a graph convolutional network model, and feature data is output, and the graph convolutional network model is used to indicate the setting of parameters of the asynchronous event stream generation graph according to the connectivity index, and the connectivity index indicates the connectivity of the graph; that is, a connectivity index is proposed to characterize the connectivity of the graph, and the connectivity is used to guide the setting of parameters of the asynchronous event stream generation graph, control the scale of the generated graph, the number of nodes and the number of edges, and prepare data for the end-to-end deep learning algorithm. While ensuring the accuracy of the model, the minimum number of nodes is used, thereby reducing the amount of calculation, and then reducing the training time, inference time and latency of the model, thereby improving the effect of feature learning of the asynchronous event stream using the graph convolutional network model. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate exemplary embodiments, features, and aspects of the disclosure and, together with the description, serve to explain the principles of the disclosure.

[0040] Figure 1 A schematic diagram of the structure of a computing device provided by an exemplary embodiment of the present disclosure is shown.

[0041] Figure 2 A flowchart of an asynchronous event stream processing method based on a graph model provided by an exemplary embodiment of the present disclosure is shown.

[0042] Figure 3 A schematic diagram showing statistical data provided by an exemplary embodiment of the present disclosure is shown.

[0043] Figure 4 A schematic diagram of a fitting curve provided by an exemplary embodiment of the present disclosure is shown.

[0044] Figure 5 A schematic diagram showing pooling effects of different pooling scales provided by an exemplary embodiment of the present disclosure is shown.

[0045] Figure 6 A schematic diagram of the structure of an asynchronous event stream processing device based on a graph model provided by an exemplary embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0046] Various exemplary embodiments, features and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent elements with the same or similar functions. Although various aspects of the embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless otherwise specified.

[0047] The word “exemplary” is used exclusively herein to mean “serving as an example, example, or illustration.” Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments.

[0048] In addition, in order to better illustrate the present disclosure, numerous specific details are given in the following specific embodiments. It should be understood by those skilled in the art that the present disclosure can also be implemented without certain specific details. In some examples, methods, means, components and circuits well known to those skilled in the art are not described in detail in order to highlight the subject matter of the present disclosure.

[0049] How to learn features from asynchronous event streams is an important and core issue in neuromorphic engineering. At present, event-based computer vision research is still in its early stages, and it is not clear which feature representation methods and architectures are most suitable for this problem. In related technologies, the processing methods for asynchronous event streams are mainly concentrated in two directions, namely the Convolutional Neural Network (CNN) method and the Spiking Neural Network (SNN) method.

[0050] The CNN method includes: converting the event stream into an image, and then directly integrating it into the CNN-based visual algorithm. This method enables the rapid application of asynchronous event streams and is also the mainstream method for expressing the features of asynchronous event streams in the early stage. However, due to the low spatial resolution of the NVS device, the texture of the event image is relatively fuzzy, which greatly reduces the performance of visual information processing. On the other hand, the conversion to an image is equivalent to re-densely representing the sparse asynchronous event stream, which violates the original intention of the NVS camera design. The defect of this method is that it abandons the sparse and asynchronous nature of NVS data and does not fully explore the spatiotemporal characteristics of the event stream.

[0051] Both spiking neural networks and NVS cameras are event encodings, and they have a natural adaptability to data processing methods. For this reason, research on asynchronous event streams is mainly concentrated in the field of SNN. However, the biggest challenge for SNN is that it cannot establish a good optimization algorithm and learning rule based on gradient descent and error back propagation like artificial neural networks (ANN), which makes SNN-based methods quite complicated to train. In terms of accuracy, the effect is still behind ANN in most learning tasks.

[0052] The disclosed embodiment performs feature learning on asynchronous event streams through a graph convolutional network model, proposes a connectivity index to characterize the connectivity of the graph, and uses the connectivity to guide the parameter setting of the asynchronous event stream generation graph, controls the scale, number of nodes, and number of edges of the generated graph, and prepares data for end-to-end deep learning algorithms. While ensuring the accuracy of the model, the minimum number of nodes is used to reduce the amount of calculation, thereby reducing the training time, inference time, and latency of the model, thereby improving the effect of feature learning on asynchronous event streams using a graph convolutional network model.

[0053] First, the application scenarios involved in the present disclosure are introduced.

[0054] The event camera (such as NVS camera) provided in the embodiments of the present disclosure is suitable for real-time interactive systems such as autonomous driving and wearable devices, security monitoring and industrial visual inspection with uncontrolled lighting conditions, and other fields, involving target tracking, feature detection, posture estimation under conditions such as high-speed motion and extreme lighting, and other scenarios with strict requirements on latency and power. Applying event cameras to the visual system of unmanned equipment will have better tracking capabilities in high-speed motion scenes and extreme lighting conditions than traditional cameras. Research on target recognition and detection technology based on asynchronous event streams can significantly enhance the unmanned equipment's ability to adapt to complex environments and its ability to perceive surrounding situations, and has important research significance and value in the fields of autonomous driving and smart transportation.

[0055] For the convenience of explanation, the following description is only based on the example of the execution subject of the method provided in the embodiment of the present disclosure being a computing device. Figure 1 , which shows a schematic diagram of the structure of a computing device provided by an exemplary embodiment of the present disclosure.

[0056] The computing device may be a terminal or a server. The terminal includes a mobile terminal or a fixed terminal, such as a mobile phone, a tablet computer, a laptop computer, a desktop computer, etc. The server may be a server, a server cluster consisting of several servers, or a cloud computing service center.

[0057] like Figure 1As shown, the computing device includes a processor 10, a memory 20, and a communication interface 30. Those skilled in the art will appreciate that Figure 1 The structure shown in the figure does not constitute a limitation on the computing device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0058] The processor 10 is the control center of the computing device, and uses various interfaces and lines to connect various parts of the entire computing device. It executes various functions of the computing device and processes data by running or executing software programs and / or modules stored in the memory 20, and calling data stored in the memory 20, thereby controlling the computing device as a whole. The processor 10 can be implemented by a CPU or a graphics processing unit (GPU).

[0059] The memory 20 can be used to store software programs and modules. The processor 10 executes various functional applications and data processing by running the software programs and modules stored in the memory 20. The memory 20 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system 21, a first output module 22, a second output module 23 and an application required for at least one function, etc.; the data storage area may store data created according to the use of the computing device, etc. The memory 20 may be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as a static random access memory (SRAM), an EEPROM, an erasable programmable read-only memory (EPROM), a programmable read-only memory (PROM), a read-only memory (ROM), a magnetic memory, a flash memory, a magnetic disk or an optical disk. Accordingly, the memory 20 may also include a memory controller to provide the processor 10 with access to the memory 20.

[0060] Among them, the processor 10 performs the following functions by running the first output module 22: sampling the ambient light intensity change value through the event camera, and outputting an asynchronous event stream, wherein the asynchronous event stream is used to indicate the light intensity changes corresponding to multiple pixels sampled asynchronously; the processor 10 performs the following functions by running the second output module 23: performing feature learning on the asynchronous event stream through a graph convolutional network model, and outputting feature data, wherein the graph convolutional network model is used to indicate setting parameters of the asynchronous event stream generation graph according to a connectivity indicator, and the connectivity indicator indicates the connectivity of the graph.

[0061] The following uses several exemplary embodiments to introduce the asynchronous event stream processing method based on the graph model provided by the embodiments of the present disclosure.

[0062] Please refer to Figure 2 , which shows a flowchart of an asynchronous event stream processing method based on a graph model provided by an exemplary embodiment of the present disclosure. This embodiment uses this method for Figure 1 The method is illustrated by using the computing device shown in FIG.

[0063] Step 201, sampling the ambient light intensity change value through an event camera, and outputting an asynchronous event stream, where the asynchronous event stream is used to indicate the light intensity changes corresponding to a plurality of asynchronously sampled pixels.

[0064] The computing device obtains the ambient light intensity change value, inputs the ambient light intensity change value into the event camera, and outputs an asynchronous event stream.

[0065] Optionally, the event camera is NVS. NVS samples the ambient brightness change value and has a stronger ability to perceive complex environments than traditional cameras. The asynchronous event stream output by NVS does not have the concept of frames. Given an NVS camera with a spatial address resolution of H×W, each event generated indicates the coordinate position, timestamp and event polarity of the pixel generating the event. The event polarity indicates whether the light intensity corresponding to the pixel increases or decreases. Each event generated can be represented by a four-tuple (x, y, t, p), also called Address-Event Representation (AER). (x, y)∈R H×W is the coordinate position of the pixel where the event occurs, t is the timestamp of the event, and p is the event polarity, which indicates whether the light intensity corresponding to the pixel increases or decreases, where +1 indicates an increase in light intensity, i.e. an ON event or a Positive event, and -1 indicates a decrease in light intensity, i.e. an OFF event or a Negative event. Due to this difference in principle, the NVS camera greatly reduces the transmission bandwidth, has high temporal resolution, and has a very high dynamic range.

[0066] Optionally, the computing device samples the ambient light intensity change value through the NVS camera and outputs a four-tuple sequence, which represents an asynchronous event stream. That is, the asynchronous event stream includes multiple asynchronous events, each of which is independent and unconnected. Schematically, a set of asynchronous event streams output by NVS is represented by a four-tuple sequence: {e i} N , where e i =(x i ,y i ,t i ,pi ),i∈[1,N].

[0067] Step 202, feature learning is performed on the asynchronous event stream through a graph convolutional network model, and feature data is output. The graph convolutional network model is used to indicate setting parameters of the asynchronous event stream generation graph according to a connectivity index, and the connectivity index indicates the connectivity of the graph.

[0068] The computing device inputs the asynchronous event (i.e., the four-tuple sequence) into the graph convolutional network model, and outputs the feature data. Optionally, the asynchronous event stream is a four-tuple sequence, and the feature data is a feature matrix.

[0069] The disclosed embodiment proposes a connectivity index to characterize the connectivity of a graph. It is derived by comparing the number of edges with the number of nodes in the graph. The connectivity is jointly affected by parameters such as the number of events, the neighborhood radius distance, and the maximum connectivity. Among them, the number of events is the total number of events; the neighborhood radius distance means that in a graph neural network, for a node, its neighborhood includes all nodes connected to it and the edges between these nodes. The neighborhood radius distance is the distance between a node and its neighboring node farthest from the node. The neighboring node is a node within a circular neighborhood of a target radius with the node as the center. In a graph convolutional network model, the neighborhood radius distance is usually a pre-specified hyperparameter used to determine which nodes and edges should be considered when performing a convolution operation; the maximum connectivity is the maximum number of connections for each node, that is, the upper limit of the number of nodes connected to a node. For example, the number of events is N, the neighborhood radius distance is R, and the maximum connectivity is D. max .

[0070] Optionally, before the feature learning process of the model, the computing device sets an asynchronous event flow generation graph according to the connectivity index. This step belongs to data preprocessing and is performed separately before model training.

[0071] Optionally, the computing device converts each event in the asynchronous event stream into a node in a graph; when the distance between two nodes in the graph is less than a preset distance, the two nodes are connected by an edge; and the ratio of the number of edges to the number of nodes in the graph is determined as a connectivity indicator.

[0072] Optionally, the distance between two nodes is a weighted Euclidean distance between the two nodes, the preset distance is the distance between the node and a neighbor node farthest from the node, and the neighbor node is a node within a circular neighborhood of a target radius with the node as the center.

[0073] Schematically, a set of asynchronous event streams output by NVS is represented by a sequence of four tuples: i} N , where e i =(x i ,yi ,t i ,p i ),i∈[1,N]. For a graph G=(V,E), V represents the set of nodes, E represents the set of edges, and each event e i =(x i ,y i ,t i ,p i ) is considered as a node v in the graph i =(x i ,y i ,t i ), where v i ∈V. When node v i and v j When the weighted Euclidean distance of node v is less than the neighborhood radius distance R, node v i and v j Use edges to connect them, and the Euclidean distance d between them i,j It is defined as the weighted space-time distance, and the formula is as follows:

[0074]

[0075] Among them, α and β are weight parameters to compensate for the difference in spatial and temporal grid resolution, such as α = 1, β = 0.5×10 4 , because the timing accuracy in the NVS camera is significantly higher than the spatial grid resolution (time accuracy 10e-6, spatial accuracy 10e2), it can be calculated by the maximum connectivity D max To constrain the maximum number of connections per node.

[0076] Optionally, the computing device sets the parameters of the asynchronous event flow generation graph according to the connectivity index, that is, the connectivity is used to guide the parameter setting of the asynchronous event flow generation graph. Schematically, the parameters of the asynchronous event flow generation graph include at least one of the number of events, the neighborhood radius distance, and the maximum connectivity.

[0077] Optionally, after the computing device determines that the connectivity index is a target value, the target value is used as a reference to adjust the parameters of the asynchronous event flow generation graph (including at least one of the number of events, neighborhood radius distance, and maximum connectivity). When the connectivity index is the target value, the corresponding target parameters of the asynchronous event flow generation graph are determined, and the parameters of the asynchronous event flow generation graph are set as the target parameters. Optionally, the number of events is logarithmically related to the connectivity index. In an illustrative example, the relationship between the connectivity index and the number of events is statistically analyzed on two neuromorphic datasets (such as the N-Cars dataset and the NCaltech101 dataset), and the changing trend of the connectivity index is fitted. Statistical data such as Figure 3 The fitting curve is shown as Figure 4As shown in the figure, the number of events and the connectivity index are logarithmically related. As the number of nodes increases, the connectivity index gradually approaches the maximum connectivity D. max At the input stage of the network, a large amount of connectivity must be ensured. As the number of network layers increases, the amount of connectivity in the graph should become smaller and smaller.

[0078] Optionally, the computing device sets the depth of the graph convolutional network model according to the connectivity index; and / or sets the pooling scale of each pooling layer of the graph convolutional network model according to the connectivity index.

[0079] The computing device sets the depth of the graph convolutional network model according to the connectivity index. It is related to the receptive field (English: Receptive Field) by analogy with replacing large-scale convolution kernels with multiple small-scale convolution kernels in CNN. Schematically, the VGG network in CNN uses two 3×3 convolution kernels instead of a 5×5 convolution kernel, or three 3×3 convolution kernels instead of a 7×7 convolution kernel, which can reduce the number of parameters, increase the number of network layers, and improve the nonlinear expression ability of the network. In the graph convolutional network, the connectivity index is also used to guide the setting of the pooling layer to increase the number of network layers. By selecting and setting pooling layers of different sizes at different positions in the network, the multi-scale features of the graph can be learned. The pooling size should be from small to large, which is conducive to the network to more fully extract the structural features of the graph.

[0080] The connectivity of the graph remains unchanged after the convolution layer, but becomes smaller after the pooling layer. Therefore, the computing device can set the pooling scale of each pooling layer of the graph convolutional network model according to the connectivity index.

[0081] In an illustrative example, on the N-Caltech101 dataset, an event graph containing 10,000 nodes is selected. After this graph is sent to the pooling layer, the pooling effects of different pooling scales are as follows: Figure 5 As shown in the results, it can be seen that with the increase of pooling size, the number of nodes after pooling decreases rapidly, the edges become more sparse, and the connectivity index becomes smaller and smaller.

[0082] Optionally, the method of setting the pooling scale of the pooling layer according to the connectivity is automatic setting or manual setting. The computing device can set the pooling scale of the pooling layer according to the connectivity based on empirical data and a fitting curve.

[0083] Optionally, the computing device replaces a pooling layer of a first size with multiple pooling layers of a second size, where the second size is smaller than the first size, that is, multiple small-scale pooling layers are used to replace a large-scale pooling layer. In an illustrative example, a graph of 10,000 nodes is pooled using three pooling layers of sizes 1×1, 2×2, and 2×2, which is equivalent to only passing through a 3×3 pooling layer, and is pooled using three pooling layers of sizes 1×1, 2×2, and 4×4, which is equivalent to only passing through a 5×5 pooling layer. Using the connectivity index as the basis for selecting the pooling scale of the pooling layer can increase the number of network layers and improve the nonlinear expression ability of the network.

[0084] In summary, the embodiment of the present disclosure uses the connectivity index to guide the setting of the parameters of the asynchronous event flow generation graph, controls the scale, number of nodes and number of edges of the generated graph, and prepares data for the end-to-end deep learning algorithm. In addition, the connectivity index is used to evaluate the pooling effect of different pooling scales, and multiple small-scale pooling layers are used instead of a large-scale pooling layer, thereby increasing the number of network layers and improving the nonlinear expression ability of the network.

[0085] The following is an apparatus embodiment of the present disclosure. For parts not described in detail in the apparatus embodiment, reference may be made to the technical details disclosed in the above method embodiment.

[0086] Please refer to Figure 6 , which shows a schematic diagram of the structure of an asynchronous event stream processing device based on a graph model provided by an exemplary embodiment of the present disclosure. The asynchronous event stream processing device based on a graph model can be implemented as all or part of a computing device through software, hardware, or a combination of both. The device includes: a first output module 610 and a second output module 620.

[0087] A first output module 610 is used to sample the ambient light intensity change value through an event camera and output an asynchronous event stream, where the asynchronous event stream is used to indicate the light intensity change corresponding to each of the multiple pixels sampled asynchronously;

[0088] The second output module 620 is used to perform feature learning on the asynchronous event stream through a graph convolutional network model and output feature data. The graph convolutional network model is used to indicate the parameters of the asynchronous event stream generation graph to be set according to the connectivity index, and the connectivity index indicates the connectivity of the graph.

[0089] In a possible implementation, the asynchronous event stream includes multiple asynchronous events, where the event indicates the coordinate position of the pixel generating the event, a timestamp, and an event polarity, and the event polarity indicates an increase or decrease in light intensity corresponding to the pixel.

[0090] In another possible implementation, the apparatus further includes: a determining module, configured to:

[0091] Convert each event in the asynchronous event stream into a node in the graph;

[0092] When the distance between two nodes in the graph is less than the preset distance, the two nodes are connected by an edge;

[0093] The ratio of the number of edges to the number of nodes in the graph is determined as the connectivity index.

[0094] In another possible implementation, the distance between two nodes is the weighted Euclidean distance between the two nodes, the preset distance is the distance between the node and the neighbor node farthest from the node, and the neighbor node is a node within a circular neighborhood of a target radius with the node as the center.

[0095] In another possible implementation, the parameters of the asynchronous event flow generation graph include at least one of the number of events, the neighborhood radius distance, and the maximum connectivity. The neighborhood radius distance is the distance between a node and its neighboring node farthest from the node. The neighboring nodes are nodes within a circular neighborhood of a target radius with the node as the center. The maximum connectivity is the maximum number of connections for each node.

[0096] In another possible implementation, the number of events is logarithmically related to the connectivity index.

[0097] In another possible implementation, the device further includes: a setting module, configured to:

[0098] Setting the depth of the graph convolutional network model based on the connectivity metric; and / or,

[0099] The pooling scale of each pooling layer of the graph convolutional network model is set according to the connectivity index.

[0100] It should be noted that the device provided in the above embodiment only uses the division of the above-mentioned functional modules as an example to implement its functions. In practical applications, the above-mentioned functions can be assigned to different functional modules according to actual needs, that is, the content structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0101] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0102] The embodiments of the present disclosure further provide a computing device, which includes: a processor; and a memory for storing instructions executable by the processor; wherein the processor is configured to: implement the steps executed by the computing device in the above-mentioned various method embodiments.

[0103] The embodiments of the present disclosure further provide a non-volatile computer-readable storage medium on which computer program instructions are stored. When the computer program instructions are executed by a processor, the methods in the above-mentioned various method embodiments are implemented.

[0104] The present disclosure may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present disclosure.

[0105] A computer-readable storage medium may be a tangible device that can hold and store instructions used by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples of computer-readable storage media (a non-exhaustive list) include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a raised structure in a groove on which instructions are stored, and any suitable combination of the foregoing. As used herein, a computer-readable storage medium is not to be interpreted as a transient signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse through a fiber optic cable), or an electrical signal transmitted through a wire.

[0106] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0107] The computer program instructions for performing the operation of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions may be executed completely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be customized by utilizing the state information of the computer-readable program instructions, and the electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present disclosure.

[0108] Various aspects of the present disclosure are described herein with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0109] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0110] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0111] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of the module, program segment or instruction includes one or more executable instructions for realizing the specified logical function. In some alternative implementations, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or action, or can be implemented with a combination of special hardware and computer instructions.

[0112] The embodiments of the present disclosure have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A method for processing asynchronous event streams based on a graph model, characterized in that: The method comprises: The ambient light intensity change value is sampled by an event camera, and an asynchronous event stream is outputted, where the asynchronous event stream is used to indicate the light intensity change corresponding to each of the multiple pixels sampled asynchronously; The asynchronous event stream is subjected to feature learning through a graph convolutional network model, and feature data is outputted. The graph convolutional network model is used to indicate setting parameters of a graph generated by the asynchronous event stream according to a connectivity index, and the connectivity index indicates connectivity of the graph.

2. The method according to claim 1, characterized in that: The asynchronous event stream includes multiple asynchronous events, wherein the event indicates the coordinate position, timestamp and event polarity of the pixel generating the event, and the event polarity indicates whether the light intensity corresponding to the pixel increases or decreases.

3. The method according to claim 1 or 2, characterized in that: The method further comprises: Convert each event in the asynchronous event stream into a node in the graph; When the distance between two nodes in the graph is less than a preset distance, the two nodes are connected by an edge; The ratio of the number of edges to the number of nodes in the graph is determined as the connectivity index.

4. The method according to claim 3, characterized in that: The distance between the two nodes is the weighted Euclidean distance between the two nodes, the preset distance is the distance between the node and the neighboring node farthest from the node, and the neighboring node is a node within a circular neighborhood of a target radius with the node as the center.

5. The method according to any one of claims 1 to 4, characterized in that: The parameters of the asynchronous event flow generation graph include at least one of the number of events, neighborhood radius distance and maximum connectivity, wherein the neighborhood radius distance is the distance between the node and the neighboring node farthest from the node, the neighboring node is the node within a circular neighborhood of a target radius with the node as the center, and the maximum connectivity is the maximum number of connections for each of the nodes.

6. The method according to claim 5, characterized in that The number of events is logarithmically related to the connectivity index.

7. The method according to any one of claims 1 to 6, characterized in that: The method further comprises: Setting the depth of the graph convolutional network model according to the connectivity index; and / or, The pooling scale of each pooling layer of the graph convolutional network model is set according to the connectivity indicator.

8. An asynchronous event stream processing device based on a graph model, characterized in that: The device comprises: A first output module is used to sample the ambient light intensity change value through an event camera, and output an asynchronous event stream, wherein the asynchronous event stream is used to indicate the light intensity change corresponding to each of the multiple pixels sampled asynchronously; The second output module is used to perform feature learning on the asynchronous event stream through a graph convolutional network model and output feature data. The graph convolutional network model is used to indicate the parameters of the asynchronous event stream generation graph to be set according to a connectivity index, and the connectivity index indicates the connectivity of the graph.

9. A computing device, characterized in that The computing device comprises: a processor; a memory for storing instructions executable by the processor; Wherein, the processor is configured to: The ambient light intensity change value is sampled by an event camera, and an asynchronous event stream is outputted, where the asynchronous event stream is used to indicate the light intensity change corresponding to each of the multiple pixels sampled asynchronously; The asynchronous event stream is subjected to feature learning through a graph convolutional network model, and feature data is outputted. The graph convolutional network model is used to indicate setting parameters of a graph generated by the asynchronous event stream according to a connectivity index, and the connectivity index indicates connectivity of the graph.

10. A non-volatile computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images