Segmented interval sampling method and device for asynchronous event stream and storage medium

Abstract

The invention relates to the technical field of computer visual recognition, in particular to a segmented interval sampling method and device for asynchronous event streams and a storage medium. The method comprises the following steps: sensing an ambient light intensity change value through an event camera, and outputting to obtain an asynchronous event stream, the asynchronous event stream comprising a plurality of candidate event streams; a dynamic time interval sampling strategy is adopted for sampling the multiple candidate event streams, multiple sample event streams are obtained, and the sample event streams are used for being converted into sub-graphs and then input into a graph model to be processed. According to the embodiment of the invention, a dynamic time interval sampling mechanism aiming at the asynchronous event stream is designed, the time domain length of the event processing unit is controlled through a dynamic time interval sampling strategy, the long-range time structure of the whole event stream is modeled, and samples in a data set are fully utilized.

Description

Technical Field

[0001] The present disclosure relates to the technical field of computer vision recognition, and in particular to a segmented interval sampling method, device and storage medium for asynchronous event streams. Background Art

[0002] The asynchronous event stream output by neuromorphic vision sensing (NVS) is presented in the form of sparse, discrete dot matrix in three-dimensional space. Before using the graph model method to process it, the event stream needs to be sampled, and the design of the sampling algorithm will directly affect the intrinsic characteristics of the graph and the expression ability of the graph convolutional network. At the same time, the current neuromorphic datasets are limited in number and the number of samples is small, so there is insufficient data for model learning. Designing a good event sampling algorithm can make full use of limited data and improve the learning and generalization ability of the graph model.

[0003] At present, there is still a lack of research on using graph convolutional networks to process NVS data, and there is no experience to learn from. There are generally several sampling methods for asynchronous event streams, including random sampling, uniform sampling, and truncated sampling. Among them, the random sampling and uniform sampling methods randomly extract events from the entire event stream or extract a certain number of events at intervals. These methods make the originally sparse event stream even more sparse, destroying the high-resolution characteristics of the original data; truncated sampling is to take data within a fixed interval each time training, and the features learned by the model each time are highly similar.

[0004] These methods in related technologies all remove the spatiotemporal context information of the asynchronous event stream, resulting in the inability to fully explore the intrinsic characteristics of the data in the subsequent convolution process, greatly reducing the accuracy and performance of the model. Summary of the invention

[0005] In view of this, the present disclosure proposes a segmented interval sampling method, device and storage medium for asynchronous event streams. The technical solution includes:

[0006] According to one aspect of the present disclosure, a segmented interval sampling method for an asynchronous event stream is provided, the method comprising:

[0007] The change value of the ambient light intensity is sensed by an event camera, and an asynchronous event stream is outputted, wherein the asynchronous event stream includes a plurality of candidate event streams;

[0008] The multiple candidate event streams are sampled using a dynamic time interval sampling strategy to obtain multiple sample event streams. The sample event streams are converted into subgraphs and then input into the graph model for processing.

[0009] In a possible implementation, the sampling of the multiple candidate event streams using a dynamic time interval sampling strategy to obtain multiple sample event streams includes:

[0010] Acquire preset parameters, where the preset parameters are used to control the scale and time dimension of the multiple sample event streams;

[0011] According to the preset parameters, the multiple candidate event streams are sampled using the dynamic time interval sampling strategy to obtain the multiple sample event streams.

[0012] In another possible implementation, the preset parameters include an expansion rate and an offset, wherein the expansion rate is used to indicate the number of sampled nodes, and the offset is used to indicate a timestamp of the sampling.

[0013] In another possible implementation manner, the time interval sampling strategy is used to indicate retaining context information of the asynchronous event stream.

[0014] In another possible implementation, the time interval sampling strategy is used to indicate that multiple segmented candidate event streams are fused in different training rounds to obtain spatiotemporal structural features, and the spatiotemporal structural features are stacked to obtain feature representation.

[0015] In another possible implementation, the event camera is an NVS.

[0016] According to another aspect of the present disclosure, a segmented interval sampling device for an asynchronous event stream is provided, the device comprising:

[0017] A perception module, used to perceive the change value of the ambient light intensity through an event camera, and output an asynchronous event stream, wherein the asynchronous event stream includes multiple candidate event streams;

[0018] The sampling module is used to sample the multiple candidate event streams using a dynamic time interval sampling strategy to obtain multiple sample event streams. The sample event streams are used to be converted into subgraphs and then input into the graph model for processing.

[0019] In a possible implementation, the sampling module is further used to:

[0020] Acquire preset parameters, where the preset parameters are used to control the scale and time dimension of the multiple sample event streams;

[0021] According to the preset parameters, the multiple candidate event streams are sampled using the dynamic time interval sampling strategy to obtain the multiple sample event streams.

[0022] In another possible implementation, the preset parameters include an expansion rate and an offset, wherein the expansion rate is used to indicate the number of sampled nodes, and the offset is used to indicate a timestamp of the sampling.

[0023] In another possible implementation manner, the time interval sampling strategy is used to indicate retaining context information of the asynchronous event stream.

[0024] In another possible implementation, the time interval sampling strategy is used to indicate that multiple segmented candidate event streams are fused in different training rounds to obtain spatiotemporal structural features, and the spatiotemporal structural features are stacked to obtain feature representation.

[0025] In another possible implementation, the event camera is an NVS.

[0026] According to another aspect of the present disclosure, a computing device is provided, the computing device comprising: a processor; a memory for storing processor-executable instructions;

[0027] Wherein, the processor is configured to:

[0028] The change value of the ambient light intensity is sensed by an event camera, and an asynchronous event stream is outputted, wherein the asynchronous event stream includes a plurality of candidate event streams;

[0029] The multiple candidate event streams are sampled using a dynamic time interval sampling strategy to obtain multiple sample event streams. The sample event streams are converted into subgraphs and then input into the graph model for processing.

[0030] According to another aspect of the present disclosure, a non-volatile computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the method provided by the first aspect or any possible implementation manner of the first aspect is implemented.

[0031] The disclosed embodiment designs a dynamic time interval sampling mechanism for asynchronous event streams, which senses the change value of ambient light intensity through an event camera and outputs an asynchronous event stream, which includes multiple candidate event streams; a dynamic time interval sampling strategy is used to sample the multiple candidate event streams to obtain multiple sample event streams, which are converted into subgraphs and then input into a graph model for processing; that is, the time domain length of the event processing unit is controlled by a dynamic time interval sampling strategy, the long-range time structure of the entire event stream is modeled, and the samples in the data set are fully utilized. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate exemplary embodiments, features, and aspects of the disclosure and, together with the description, serve to explain the principles of the disclosure.

[0033] Figure 1 A schematic diagram of the structure of a computing device provided by an exemplary embodiment of the present disclosure is shown.

[0034] Figure 2 A flowchart of a segmented interval sampling method for an asynchronous event stream provided by an exemplary embodiment of the present disclosure is shown.

[0035] Figure 3 A schematic diagram showing experimental results of an exemplary embodiment of the present disclosure using a segmented interval sampling method on an N-Cars dataset is shown.

[0036] Figure 4 A schematic diagram showing experimental results of an exemplary embodiment of the present disclosure using a segmented interval sampling method on the N-Caltech101 dataset is shown.

[0037] Figure 5 A schematic diagram of the structure of a segmented interval sampling device for an asynchronous event stream provided by an exemplary embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0038] Various exemplary embodiments, features and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent elements with the same or similar functions. Although various aspects of the embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless otherwise specified.

[0039] The word “exemplary” is used exclusively herein to mean “serving as an example, example, or illustration.” Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments.

[0040] In addition, in order to better illustrate the present disclosure, numerous specific details are given in the following specific embodiments. It should be understood by those skilled in the art that the present disclosure can also be implemented without certain specific details. In some examples, methods, means, components and circuits well known to those skilled in the art are not described in detail in order to highlight the subject matter of the present disclosure.

[0041] First, the application scenarios involved in the present disclosure are introduced.

[0042] The event camera (such as NVS) provided in the embodiments of the present disclosure is suitable for real-time interactive systems such as autonomous driving and wearable devices, security monitoring and industrial visual inspection with uncontrolled lighting conditions, and other fields, involving target tracking, feature detection, posture estimation under conditions such as high-speed motion and extreme lighting, and other scenarios with strict requirements on latency and power. Applying event cameras to the visual system of unmanned equipment will have better tracking capabilities in high-speed motion scenes and extreme lighting conditions than traditional cameras. Research on target recognition and detection technology based on asynchronous event streams can significantly enhance the unmanned equipment's ability to adapt to complex environments and its ability to perceive surrounding situations, and has important research significance and value in the fields of autonomous driving and smart transportation.

[0043] For the convenience of explanation, the following description is only based on the example of the execution subject of the method provided in the embodiment of the present disclosure being a computing device. Figure 1 , which shows a schematic diagram of the structure of a computing device provided by an exemplary embodiment of the present disclosure.

[0044] The computing device may be a terminal or a server. The terminal includes a mobile terminal or a fixed terminal, such as a mobile phone, a tablet computer, a laptop computer, a desktop computer, etc. The server may be a server, a server cluster consisting of several servers, or a cloud computing service center.

[0045] like Figure 1 As shown, the computing device includes a processor 10, a memory 20, and a communication interface 30. Those skilled in the art will appreciate that Figure 1 The structure shown in the figure does not constitute a limitation on the computing device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0046] The processor 10 is the control center of the computing device, and uses various interfaces and lines to connect various parts of the entire computing device. It executes various functions of the computing device and processes data by running or executing software programs and / or modules stored in the memory 20, and calling data stored in the memory 20, thereby controlling the computing device as a whole. The processor 10 can be implemented by a CPU or a graphics processing unit (GPU).

[0047] The memory 20 can be used to store software programs and modules. The processor 10 executes various functional applications and data processing by running the software programs and modules stored in the memory 20. The memory 20 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system 21, a perception module 22, a sampling module 23 and an application required for at least one function, etc.; the data storage area may store data created according to the use of the computing device, etc. The memory 20 may be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as a static random access memory (SRAM), an EEPROM, an erasable programmable read-only memory (EPROM), a programmable read-only memory (PROM), a read-only memory (ROM), a magnetic memory, a flash memory, a magnetic disk or an optical disk. Accordingly, the memory 20 may also include a memory controller to provide the processor 10 with access to the memory 20.

[0048] Among them, the processor 10 performs the following functions by running the perception module 22: the change value of the ambient light intensity is perceived by the event camera, and an asynchronous event stream is output, and the asynchronous event stream includes multiple candidate event streams; the processor 10 performs the following functions by running the sampling module 23: the multiple candidate event streams are sampled using a dynamic time interval sampling strategy to obtain multiple sample event streams, and the sample event streams are used to convert into subgraphs and then input into the graph model for processing.

[0049] The following uses several exemplary embodiments to introduce the asynchronous event stream processing method based on the graph model provided by the embodiments of the present disclosure.

[0050] Please refer to Figure 2 , which shows a flowchart of a segmented interval sampling method for an asynchronous event stream provided by an exemplary embodiment of the present disclosure. This embodiment uses this method for Figure 1 The method is illustrated by using the computing device shown in FIG.

[0051] Step 201 , sensing the change value of the ambient light intensity through an event camera, and outputting an asynchronous event stream, which includes a plurality of candidate event streams.

[0052] The computing device obtains the ambient light intensity change value, inputs the ambient light intensity change value into the event camera, and outputs an asynchronous event stream. Optionally, the event camera is NVS.

[0053] The asynchronous event stream includes multiple asynchronous candidate event streams, and each candidate event stream is independent and unconnected.

[0054] Step 202: a plurality of candidate event streams are sampled using a dynamic time interval sampling strategy to obtain a plurality of sample event streams. The sample event streams are converted into subgraphs and then input into a graph model for processing.

[0055] Before a computing device processes an event stream using a graph model, it needs to sample the event stream. The design of the sampling algorithm will directly affect the intrinsic characteristics of the graph and the expressive power of the graph convolutional network. The disclosed embodiment designs a dynamic time interval sampling strategy, which samples multiple candidate event streams using the dynamic time interval sampling strategy to obtain multiple sample event streams.

[0056] The embodiment of the present disclosure adds the idea of ​​dynamic segmentation to the sampling operation, and controls the scale and time dimension of the subgraph by presetting parameters.

[0057] In one possible implementation, a computing device samples multiple candidate event streams using a dynamic time interval sampling strategy to obtain multiple sample event streams, including: obtaining preset parameters, where the preset parameters are used to control the scale and time dimension of the multiple sample event streams; and sampling multiple candidate event streams using a dynamic time interval sampling strategy according to the preset parameters to obtain multiple sample event streams.

[0058] Optionally, the preset parameters include an expansion rate and an offset, wherein the expansion rate is used to indicate the number of sampled nodes, and the offset is used to indicate the timestamp of the sampling, that is, by setting a progressive expansion rate and a forward offset, the entire sample space can be covered. Schematically, the offset is the proportion of repeated event streams to the sample event streams sampled this time, wherein the repeated event streams are the repeated sample event streams between the sample event streams sampled this time and the sample event streams sampled next time; the expansion rate is used to indicate the number of parts into which the overall sample space is divided.

[0059] Optionally, multiple candidate event streams are distributed along the time dimension. For each sampling, the computing device extracts sampling segments on the time series of the multiple candidate event streams according to preset parameters to obtain multiple sample event streams, and the multiple sample event streams aggregate the information in the corresponding sampling segments.

[0060] Optionally, there is an intersection between multiple sample event streams of two adjacent samplings. Schematically, there is an intersection between multiple sample event streams of this sampling and multiple sample event streams of the previous sampling; and / or there is an intersection between multiple sample event streams of this sampling and multiple sample event streams of the next sampling.

[0061] Optionally, for two adjacent samples in multiple samplings, the ratio of the number of sample event streams with intersection to the total number of samples in the previous segment is the same.

[0062] Illustratively, the segmented interval sampling method provided by the embodiment of the present disclosure is verified on two neuromorphic datasets (such as the N-Cars dataset and the NCaltech101 dataset). On the N-Cars dataset, the input event points are arranged according to the time dimension, and the offset o is used to indicate the proportion of the sample event stream sampled this time that is repeated in the sample event stream sampled next time, that is, the offset o is the proportion of the repeated event stream to the sample event stream sampled this time, where the repeated event stream is the repeated sample event stream between the sample event stream sampled this time and the sample event stream sampled next time; the expansion rate s is used to indicate the number of sampled nodes, and the expansion rate n is used to indicate that the overall sample space is divided into n parts, where n is a positive integer. For example, when the expansion rate s is determined to be 4, it means that the overall sample space is divided into 4 parts, and the number of samples after segmentation is {2000, 4000, 6000, 8 000}, when o is 1, it is used to indicate all repeated sampling, and the sampling sequence is {(0, 2000), (0, 4000), (0, 6000), (0, 8000)}, when o is 0.5, it is used to indicate that the sample event stream of this sampling has 50% repeated samples in the sample event stream of the previous sampling, and the resulting sampling sequence is {(0, 2000), (1000, 5000), (3000, 9000), (6000, 14000)}. Because the N-Cars dataset is small in size, sampling can almost cover all samples without setting an offset, but the method provided by the embodiment of the present disclosure can further improve the accuracy of the model. In an illustrative example, the experimental results after using the segmented interval sampling method on the N-Cars dataset are shown in the figure. Figure 3 As shown. Among them, solid line 31 is the training accuracy curve of the traditional method, and solid line 32 is the recognition accuracy curve of the traditional method. Solid line 33 is the training accuracy curve of the training method using the Progressive Graph Convolution Network (PGCN), which only uses the progressive training method and does not use segmented interval sampling, that is, only the expansion rate is set. The dotted line 34 is an approximate training accuracy curve, and the solid line 35 is the recognition accuracy curve of the PGCN test method. Solid line 36 is the training accuracy curve of the training method using the Progressive Graph Convolution Network with Segmented Interval Sampling (English: DS-PGCN), which sets both the expansion rate and the offset. The dotted line 37 is an approximate training accuracy curve, and the solid line 38 is the recognition accuracy curve of the test method using DS-PGCN. The relationship between the offset and the recognition accuracy on the N-Cars dataset is shown in Table 1. It can be seen that the segmented interval sampling method improves the recognition accuracy to 93.892.

[0063] Table 1

[0064]

[0065]

[0066] In another illustrative example, the experimental results after using the segmented interval sampling method on the N-Caltech101 dataset are as follows: Figure 4 As shown. Among them, solid line 41 is the training accuracy curve of the traditional method, and solid line 42 is the recognition accuracy curve of the traditional method. Solid line 44 is the training accuracy curve of the training method using PGCN, which only uses the progressive training method and does not use segmented interval sampling, that is, only the expansion rate is set. The dotted line 44 is the approximate training accuracy curve, and the solid line 45 is the recognition accuracy curve of the test method using PGCN. Solid line 46 is the training accuracy curve of the training method using DS-PGCN, which sets both the expansion rate and the offset. The dotted line 47 is the approximate training accuracy curve, and the solid line 48 is the recognition accuracy curve of the test method using DS-PGCN. The relationship between the offset and the recognition accuracy on the N-Caltech101 dataset is shown in Table 2. It can be seen that after adopting the segmented interval sampling method, a new sample space is introduced after every 15 training rounds. This is a sample that has never participated in the training before. Therefore, the training accuracy drops rapidly, and it drops more sharply than when only the expansion sampling is added. The setting of the offset o adds the idea of ​​segmentation. By extracting fragments from the time series of the event stream, the sampled samples are distributed along the time dimension, and each segmented structure aggregates the information in each sampled fragment, so that the long-range time structure of the entire event stream can be modeled. In addition, the traditional sampling method only uses part of the data in the data set, while the method provided by the embodiment of the present disclosure uses all the sample data. In a sense, it can be called data amplification. The method provided by the embodiment of the present disclosure amplifies the event stream data set through the segmented interval sampling method. One sample data is segmented and processed into multiple samples, similar to the continuous addition of scaled, cropped and flipped samples in the Convolutional Neural Networks (CNN). This method significantly improves the accuracy of the model, and the highest accuracy reaches 68.354.

[0067] Table 2

[0068]

[0069]

[0070] Optionally, the time interval sampling strategy provided in the embodiment of the present disclosure is used to indicate the preservation of context information of the asynchronous event stream, and is also used to indicate that multiple segmented candidate event streams are fused in different training rounds to obtain spatiotemporal structural features, and the spatiotemporal structural features are stacked to obtain feature representation. Among them, the training round is also called the round, which refers to the training cycle (English: Epoch) during neural network training.

[0071] Optionally, after the computing device samples and obtains multiple sample event streams, the multiple sample event streams are converted into subgraphs of different scales and then input into the graph model, and feature learning is performed on the asynchronous event stream through the graph model to output feature data. Schematically, the graph model is a graph convolutional network (GCN) model.

[0072] In summary, the time interval sampling method provided in the embodiment of the present disclosure can model the long-range temporal structure of the entire event stream, make full use of the limited data in the data set, effectively overcome the overfitting problem in network training, and improve the accuracy of object classification. On the one hand, the dynamic time interval sampling strategy provided in the embodiment of the present disclosure controls the time domain length of the event processing unit, optimizes the generation method of the subgraph, and controls the scale and time dimension of the subgraph through the two parameters of expansion rate and offset respectively. On the other hand, the time interval sampling mechanism can fully retain the contextual information of the original data, fuse multiple segmented subgraphs in different training rounds, obtain more comprehensive spatiotemporal structural features, and then stack these features to form a hierarchical, multi-scale, and highly semantic feature representation.

[0073] The following is an apparatus embodiment of the present disclosure. For parts not described in detail in the apparatus embodiment, reference may be made to the technical details disclosed in the above method embodiment.

[0074] Please refer to Figure 5 , which shows a schematic diagram of the structure of a segmented interval sampling device for an asynchronous event stream provided by an exemplary embodiment of the present disclosure. The device can be implemented as all or part of a computing device through software, hardware, or a combination of the two. The device includes: a perception module 510 and a sampling module 520.

[0075] A perception module 510 is used to perceive the change value of the ambient light intensity through an event camera, and output an asynchronous event stream, which includes multiple candidate event streams;

[0076] The sampling module 520 is used to sample multiple candidate event streams using a dynamic time interval sampling strategy to obtain multiple sample event streams. The sample event streams are converted into subgraphs and then input into the graph model for processing.

[0077] In a possible implementation, the sampling module 520 is further configured to:

[0078] Get preset parameters, which are used to control the scale and time dimension of multiple sample event streams;

[0079] According to preset parameters, a dynamic time interval sampling strategy is adopted to sample multiple candidate event streams to obtain multiple sample event streams.

[0080] In another possible implementation, the preset parameters include an expansion rate and an offset, wherein the expansion rate is used to indicate the number of sampled nodes, and the offset is used to indicate the timestamp of the sampling.

[0081] In another possible implementation, the time interval sampling strategy is used to indicate the preservation of context information of the asynchronous event stream.

[0082] In another possible implementation, the time interval sampling strategy is used to indicate that multiple segmented candidate event streams are fused in different training rounds to obtain spatiotemporal structural features, and the spatiotemporal structural features are stacked to obtain feature representation.

[0083] In another possible implementation, the event camera is an NVS.

[0084] It should be noted that the device provided in the above embodiment only uses the division of the above-mentioned functional modules as an example to implement its functions. In practical applications, the above-mentioned functions can be assigned to different functional modules according to actual needs, that is, the content structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0085] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0086] The embodiments of the present disclosure further provide a computing device, which includes: a processor; and a memory for storing instructions executable by the processor; wherein the processor is configured to: implement the steps executed by the computing device in the above-mentioned various method embodiments.

[0087] The embodiments of the present disclosure further provide a non-volatile computer-readable storage medium on which computer program instructions are stored. When the computer program instructions are executed by a processor, the methods in the above-mentioned various method embodiments are implemented.

[0088] The present disclosure may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present disclosure.

[0089] A computer-readable storage medium may be a tangible device that can hold and store instructions used by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples of computer-readable storage media (a non-exhaustive list) include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a raised structure in a groove on which instructions are stored, and any suitable combination of the foregoing. As used herein, a computer-readable storage medium is not to be interpreted as a transient signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse through a fiber optic cable), or an electrical signal transmitted through a wire.

[0090] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0091] The computer program instructions for performing the operation of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions may be executed completely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be customized by utilizing the state information of the computer-readable program instructions, and the electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present disclosure.

[0092] Various aspects of the present disclosure are described herein with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0093] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0094] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0095] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of the module, program segment or instruction includes one or more executable instructions for realizing the specified logical function. In some alternative implementations, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or action, or can be implemented with a combination of special hardware and computer instructions.

[0096] The embodiments of the present disclosure have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A segmented interval sampling method for asynchronous event streams, characterized in that: The method comprises: The change value of the ambient light intensity is sensed by an event camera, and an asynchronous event stream is outputted, wherein the asynchronous event stream includes a plurality of candidate event streams; The multiple candidate event streams are sampled using a dynamic time interval sampling strategy to obtain multiple sample event streams. The sample event streams are converted into subgraphs and then input into the graph model for processing.

2. The method according to claim 1, characterized in that The method of sampling the multiple candidate event streams using a dynamic time interval sampling strategy to obtain multiple sample event streams includes: Acquire preset parameters, where the preset parameters are used to control the scale and time dimension of the multiple sample event streams; According to the preset parameters, the multiple candidate event streams are sampled using the dynamic time interval sampling strategy to obtain the multiple sample event streams.

3. The method according to claim 2, characterized in that The preset parameters include an expansion rate and an offset, wherein the expansion rate is used to indicate the number of sampled nodes, and the offset is used to indicate the timestamp of the sampling.

4. The method according to any one of claims 1 to 3, characterized in that: The time interval sampling strategy is used to indicate the preservation of context information of the asynchronous event stream.

5. The method according to any one of claims 1 to 4, characterized in that: The time interval sampling strategy is used to indicate that the candidate event streams of multiple segments are fused in different training rounds to obtain spatiotemporal structural features, and the spatiotemporal structural features are stacked to obtain feature representation.

6. The method according to any one of claims 1 to 5, characterized in that: The event camera is a neuromorphic vision sensor NVS.

7. A segmented interval sampling device for asynchronous event stream, characterized in that: The device comprises: A perception module, used to perceive the change value of the ambient light intensity through an event camera, and output an asynchronous event stream, wherein the asynchronous event stream includes multiple candidate event streams; The sampling module is used to sample the multiple candidate event streams using a dynamic time interval sampling strategy to obtain multiple sample event streams. The sample event streams are used to be converted into subgraphs and then input into the graph model for processing.

8. A computing device, characterized in that The computing device comprises: a processor; a memory for storing instructions executable by the processor; Wherein, the processor is configured to: The change value of the ambient light intensity is sensed by an event camera, and an asynchronous event stream is outputted, wherein the asynchronous event stream includes a plurality of candidate event streams; The multiple candidate event streams are sampled using a dynamic time interval sampling strategy to obtain multiple sample event streams. The sample event streams are converted into subgraphs and then input into the graph model for processing.

9. A non-volatile computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images