Jointly learning visual motion and confidence from local patches in event cameras

Abstract

A method may include obtaining a set of events for a set of pixels of a dynamic visual sensor associated with an object; determining a set of voltages for the set of pixels based on the set of events; generating a set of images based on the set of voltages for the set of pixels; inputting the set of images into a first neural network configured to output a visual motion estimate of the object; inputting the set of images into a second neural network configured to output a confidence score for the visual motion estimate output by the first neural network; obtaining a visual motion estimate of the object and a confidence score for the visual motion estimate of the object based on inputting the set of images into the first neural network and the second neural network; and providing the visual motion estimate of the object and the confidence score.

Description

Technical Field

[0001] The present disclosure relates to using an event-based camera (DVS), such as a dynamic vision sensor, to perform motion estimation. Background Art

[0002] Each pixel in an event-based camera reports only when there is a change in the logarithmic light intensity in its field of view that exceeds a threshold. This operation can be performed very quickly, without influence from neighboring pixels, and with minimal influence from absolute light intensity. This creates inherent advantages such as low-latency vision that does not require high power or is restricted to uniform and good lighting environments. In this way, event-based cameras are attractive for motion-based tasks that require precise timing and ideally invariance under extreme changes in lighting conditions. As new data processing techniques are developed, other advantages of event-based vision can be realized, such as the potential for vision with low computational cost due to its sparse, asynchronous output. Summary of the invention

[0003] Technical Solution

[0004] A method may include obtaining a set of events for a set of pixels of a dynamic visual sensor associated with an object; determining a set of voltages for the set of pixels based on the set of events; generating a set of images based on the set of voltages for the set of pixels; inputting the set of images into a first neural network configured to output a visual motion estimate of the object; inputting the set of images into a second neural network configured to output a confidence score for the visual motion estimate output by the first neural network; obtaining a visual motion estimate of the object and a confidence score for the visual motion estimate of the object based on inputting the set of images into the first neural network and the second neural network; and providing a visual motion estimate and a confidence score for the object. BRIEF DESCRIPTION OF THE DRAWINGS

[0005] The above and other aspects, features and aspects of embodiments of the present disclosure will become more apparent from the following description taken in conjunction with the accompanying drawings, in which:

[0006] Figure 1 is a schematic diagram of an example system for obtaining visual motion estimates of an object and confidence scores for the visual motion estimates using a set of neural networks according to an embodiment;

[0007] Figure 2 is a schematic diagram of an apparatus of a system for obtaining visual motion estimates of an object and confidence scores for the visual motion estimates using a set of neural networks according to one embodiment;

[0008] Figure 3 According to the implementation method Figure 2 a schematic diagram of components of the device;

[0009] Figure 4 is a schematic diagram of components of a visual motion estimation apparatus according to an embodiment;

[0010] Figure 5 is a flow chart of a method for obtaining visual motion estimates of an object and confidence scores for the visual motion estimates using a set of neural networks; and

[0011] Figure 6 is a flow chart of a method for determining the dimensions of a set of pixels to be used to generate a set of images using a neural network. DETAILED DESCRIPTION

[0012] Best Mode

[0013] According to one aspect of an exemplary embodiment, a method may include obtaining a set of events for a set of pixels of a dynamic visual sensor associated with an object; determining a set of voltages for the set of pixels based on the set of events; generating a set of images based on the set of voltages for the set of pixels; inputting the set of images into a first neural network configured to output a visual motion estimate of the object; inputting the set of images into a second neural network configured to output a confidence score for the visual motion estimate output by the first neural network; obtaining a visual motion estimate of the object and a confidence score for the visual motion estimate of the object based on inputting the set of images into the first neural network and the second neural network; and providing a visual motion estimate and a confidence score for the object.

[0014] According to one aspect of an exemplary embodiment, a device may include a memory configured to store instructions; and a processor configured to execute the instructions to: obtain a set of events of a set of pixels of a dynamic visual sensor associated with an object, determine a set of voltages of the set of pixels based on the set of events, generate a set of images based on the set of voltages of the set of pixels, input the set of images into a first neural network configured to output a visual motion estimate of the object, input the set of images into a second neural network configured to output a confidence score of the visual motion estimate output by the first neural network, obtain a visual motion estimate of the object and a confidence score of the visual motion estimate of the object based on inputting the set of images into the first neural network and the second neural network, and provide a visual motion estimate and a confidence score of the object.

[0015] According to one aspect of an exemplary embodiment, a non-transitory computer-readable medium may store one or more instructions, which, when executed by one or more processors of a device, cause the one or more processors to: obtain a set of events of a set of pixels of a dynamic visual sensor associated with an object; determine a set of voltages of the set of pixels based on the set of events; generate a set of images based on the set of voltages of the set of pixels; input the set of images into a first neural network configured to output a visual motion estimate of the object; input the set of images into a second neural network configured to output a confidence score of the visual motion estimate output by the first neural network; obtain a visual motion estimate of the object and a confidence score of the visual motion estimate of the object based on inputting the set of images into the first neural network and the second neural network; and provide a visual motion estimate and a confidence score of the object.

[0016] Additional aspects will be set forth in part in the description which follows and, in part, will be obvious from the description, or may be learned by practice of the embodiments disclosed in the present disclosure.

[0017] Invention Mode

[0018] The following detailed description of exemplary embodiments refers to the accompanying drawings.The same reference numbers in different drawings may identify the same or similar elements.

[0019] Exemplary embodiments of the present disclosure relate to methods and systems for obtaining visual motion estimates of an object and confidence scores of the visual motion estimates of the object using a set of neural networks. Exemplary embodiments use subsets of pixels of a pixel array of a dynamic vision sensor to obtain visual motion estimates, and obtain respective visual motion estimate values ​​and confidence scores for each subset of the pixel array. In this way, exemplary embodiments allow for faster and more accurate visual motion estimation at low computational cost, thereby reducing latency and conserving processor and memory resources of a visual motion estimation device.

[0020] Figure 1 is a schematic diagram of an example system for obtaining visual motion estimates of an object and confidence scores for the visual motion estimates using a set of neural networks, according to an embodiment.

[0021] like Figure 1As shown, the visual motion estimation device can generate a set of images 110 based on a set of voltages of a set of pixels of the dynamic visual sensor. In addition, the visual motion estimation device can input the set of images 110 into a first neural network 120, which is configured to output a visual motion estimate 130 of an object detected by the dynamic visual sensor, and can input the set of images 110 into a second neural network 140, which is configured to output a confidence score 150 of the visual motion estimate 130 output by the first neural network 120. The visual motion estimation device can obtain the visual motion estimate 130 of the object and the confidence score 150 of the visual motion estimate 130 of the object based on inputting the set of images 110 into the first neural network 120 and the second neural network 140. In addition, the visual motion estimation device can provide the visual motion estimate 130 and the confidence score 150 of the object.

[0022] Devices in communication with dynamic visual sensors may use visual motion estimates and confidence scores. For example, a device of an unmanned vehicle may utilize visual motion estimates and confidence scores to control the trajectory of the unmanned vehicle to avoid collisions, output warnings, and the like. As another example, a monitoring system may utilize visual motion estimates and confidence scores to monitor specific areas, notify authorized agencies, and the like. As another example, a robot may utilize visual motion estimates and confidence scores to manipulate components to avoid collisions, route planning, and the like. In this way, some implementations herein allow devices associated with event-based cameras to more accurately and quickly determine the motion of the device and / or surrounding objects, and do so while saving processor and / or memory resources based on the reduced computational complexity of some implementations herein.

[0023] Figure 2 is a schematic diagram of an apparatus of a system for obtaining visual motion estimates of an object and confidence scores for the visual motion estimates using a set of neural networks, according to one embodiment. Figure 2 The system includes a DVS 210, a visual motion estimation device 220, a server 230, and a network 240. The DVS 210, the visual motion estimation device 220, and the server 230 may be interconnected via a wired connection, a wireless connection, or a combination of wired and wireless connections.

[0024] The DVS 210 includes one or more devices configured to detect changes in light intensity based on motion of an object relative to the DVS 210. For example, the DVS 210 may be a DVS, a neuromorphic camera, a silicon retina, an imaging sensor, or the like.

[0025] The visual motion estimation device 220 includes one or more devices configured to obtain a visual motion estimation of an object and a confidence score of the visual motion estimation using a set of neural networks. For example, the visual motion estimation device 220 may include one or more computing devices. According to an example embodiment, the DVS 210 and the visual motion estimation device 220 may be integrated. In addition, the visual motion estimation device 220 may be fixed at a specific location. For example, the visual motion estimation device 220 may be a monitoring device for an area. Alternatively, the visual estimation device 220 may be integrated with a moving object such as a drone, a robot, a vehicle, etc.

[0026] Server 230 includes one or more devices configured to communicate with visual motion estimation device 220. For example, server 230 may be a server, a computing device, etc. Server 230 may train and provide the first neural network, the second neural network, and the third neural network to visual motion estimation device 220.

[0027] The network 240 includes one or more wired and / or wireless networks. For example, the network 240 may include a cellular network (e.g., a fifth generation (5G) network, a long term evolution (LTE) network, a third generation (3G) network, a code division multiple access (CDMA) network, etc.), a public land mobile network (PLMN), a local area network (LAN), a wide area network (WAN), a metropolitan area network (MAN), a telephone network (e.g., a public switched telephone network (PSTN)), a private network, an ad hoc network, an intranet, the Internet, a fiber-based network, etc., and / or a combination of these or other types of networks.

[0028] Figure 2 The number and arrangement of devices and networks shown in the figure are provided as examples. In practice, there may be additional devices and / or networks, fewer devices and / or networks, different devices and / or networks, or different devices and / or networks. Figure 2 The devices and / or networks shown may be arranged differently. Figure 2 Two or more of the devices shown in may be implemented in a single device, or Figure 2 A single device shown in the environment 200 may be implemented as multiple, distributed devices. In addition, or alternatively, a group of devices (eg, one or more devices) of the environment 200 may perform one or more functions described as being performed by another group of devices of the environment 200.

[0029] Figure 3 According to the implementation method Figure 2 The apparatus 300 may correspond to the visual motion estimation apparatus 220 and / or the server 230.

[0030] like Figure 3As shown, the apparatus 300 may include a bus 310 , a processor 320 , a memory 330 , a storage component 340 , an input component 350 , an output component 360 , and a communication interface 370 .

[0031] The bus 310 includes components that allow communication between components of the device 300. The processor 320 is implemented in hardware, firmware, or a combination of hardware and software. The processor 320 is a central processing unit (CPU), a graphics processing unit (GPU), an accelerated processing unit (APU), a microprocessor, a microcontroller, a digital signal processor (DSP), a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), or other types of processing components. The processor 320 includes one or more processors that can be programmed to perform functions.

[0032] Memory 330 includes random access memory (RAM), read only memory (ROM), and / or other types of dynamic or static storage devices (e.g., flash memory, magnetic memory, and / or optical memory) that stores information and / or instructions used by processor 320.

[0033] The storage component 340 stores information and / or software related to the operation and use of the device 300. For example, the storage component 340 may include a hard disk (e.g., a magnetic disk, an optical disk, a magneto-optical disk, and / or a solid-state disk), a compact disk (CD), a digital versatile disk (DVD), a floppy disk, a cassette, a magnetic tape, and / or other types of non-transitory computer-readable media and corresponding drives.

[0034] Input component 350 includes, for example, components that allow device 300 to receive information via user input (e.g., a touch screen display, a keyboard, keys, a mouse, buttons, switches, and / or a microphone). Input component 350 may include a sensor for sensing information (e.g., a global positioning system (GPS) component, an accelerometer, a gyroscope, an imaging sensor, and / or an actuator). Input component 350 may be DVS 210.

[0035] Output component 360 includes components (eg, a display, a speaker, and / or one or more light emitting diodes (LEDs)) that provide output information (eg, a speed estimate) from device 300 .

[0036] The communication interface 370 includes transceiver-like components (e.g., a transceiver and / or a separate receiver and transmitter) that enable the device 300 to communicate with other devices, for example, via a wired connection, a wireless connection, or a combination of wired and wireless connections. The communication interface 370 can allow the device 300 to receive information from another device and / or provide information to another device. For example, the communication interface 370 may include an Ethernet interface, an optical interface, a coaxial interface, an infrared interface, a radio frequency (RF) interface, a universal serial bus (USB) interface, a Wi-Fi interface, a cellular network interface, etc.

[0037] The device 300 may perform one or more processes described herein. The device 300 may perform these processes based on a processor 320 executing software instructions stored by a non-transitory computer-readable medium such as a memory 330 and / or a storage component 340. A computer-readable medium is defined herein as a non-transitory memory device. The memory device includes memory space within a single physical memory device or memory space spread across multiple physical memory devices.

[0038] The software instructions may be read into memory 330 and / or storage component 340 from another computer-readable medium or from another device via communication interface 370. When executed, the software instructions stored in memory 330 and / or storage component 340 may cause processor 320 to perform one or more processes described herein.

[0039] In addition, or alternatively, hard-wired circuitry may be used in place of or in combination with software instructions to perform one or more of the processes described herein.Thus, the embodiments described herein are not limited to any specific combination of hardware circuitry and software.

[0040] Figure 3 The number and arrangement of components shown in FIG. 3 are provided as examples. In practice, the apparatus 300 may include more Figure 3 The components shown may be more components, fewer components, different components, or components arranged differently. In addition, or alternatively, a set of components (e.g., one or more components) of device 300 may perform one or more functions described as being performed by another set of components of device 300.

[0041] Figure 4 is a schematic diagram of components of a visual motion estimation apparatus according to an embodiment. Figure 4 It includes an attention neural network 410, a local patch generator 420, a visual motion estimation neural network 430 and a confidence neural network 440.

[0042] The attention neural network 410 may be a neural network configured to determine the size of a group of pixels to generate an image for visual motion estimation and confidence score generation. For example, the attention neural network 410 may determine the size of a P×Q array that is a subset of the M×N pixel array of the DVS 210. The P×Q array may be referred to as a “local patch”. The DVS 210 may include an M×N pixel array. Each pixel of the DVS 210 may include a coordinate value (x, y) in the M×N array. M may be equal to N, greater than N, or less than N. The pixel group may be P×Q pixels, etc. P may be equal to Q, greater than Q, or less than Q. In addition, P×Q may be equal to M×N or less than M×N.

[0043] The DVS 210 includes a set of pixels that operate independently and asynchronously. Each pixel of the DVS 210 stores a predetermined brightness level and compares the current brightness level to the predetermined brightness level. Based on determining that the difference in brightness level meets a threshold, the pixel generates an event including the pixel address of the pixel, the timestamp of the event, the polarity of the brightness change (e.g., positive or negative), the brightness level of the event, etc.

[0044] The attention neural network 410 may be configured to determine the size of the P×Q array based on the DVS 210 , the type of object to be detected, the size of the object to be detected, the number of objects to be detected, the speed of the objects, the speed of the DVS 210 , etc.

[0045] The local patch generator 420 may be configured to generate local patches based on the size of the pixel groups (ie, array P×Q) from an image obtained from the DVS 210. For example, the local patch generator 420 may obtain an input image from the DVS 210 and generate a set of local patches based on the size of the pixel groups.

[0046] The visual motion estimation neural network 430 may be configured to output a visual motion estimate of an object. For example, the visual motion estimation neural network 430 may obtain a set of images and output a visual motion estimate of an object based on the set of images.

[0047] The confidence neural network 440 may be configured to output a confidence score of the visual motion estimation output by the visual motion estimation neural network 430. For example, the confidence neural network 440 may obtain a set of images used by the visual motion estimation neural network 430 to generate the visual motion estimation, and output a confidence score of the visual motion estimation output by the visual motion estimation neural network 430.

[0048] Figure 5 is a flow chart of a method for obtaining visual motion estimates of an object and confidence scores for the visual motion estimates using a set of neural networks. According to some embodiments, Figure 5One or more operations of may be performed by visual motion estimation device 220. Additionally or alternatively, DVS 210 and / or server 230 may perform one or more operations.

[0049] like Figure 5 As shown, the method may include obtaining a set of events of a set of pixels of a dynamic visual sensor associated with an object (operation 510). For example, the visual motion estimation device 220 may obtain a set of events associated with an object based on the output of a set of pixels of the DVS 210.

[0050] Each pixel of the DVS 210 may be configured to detect a change in light intensity. For example, the DVS 210 may detect a change in light intensity based on the motion of an object relative to the DVS 210. The object may include any type of potential object that may be detected by the DVS 210 (e.g., a person, a vehicle, a machine, an animal, an item, etc.). The DVS 210 may be configured to provide events based on the pixels of the DVS 210 that detect the change in light intensity to the visual motion estimation device 220.

[0051] The event may include the coordinate value of the pixel of the DVS 210 that detects the light intensity change, the polarity value of the light intensity change, the light intensity value, and the time when the light intensity change is detected. For example, the visual motion estimation device 220 may obtain the following event sequence (S) from the DVS 210. The event sequence (S) may include the i-th event (s i ) to the Kth event (s K ). The i-th event (s i ) may include detecting the i-th event (s i ) is the pixel coordinate value (x i ,y i ), the polarity value of the ith event (p i ) and the time of the ith event.

[0052]

[0053] like Figure 5 As further shown, the method may include determining a set of voltages for a set of pixels based on a set of events (operation 520). For example, visual motion estimation device 220 may determine a set of voltages for a set of pixels of DVS 210 based on a set of events detected by DVS 210.

[0054] The visual motion estimation device 220 may use the following function to determine the voltage value (v(t)) of a pixel:

[0055]

[0056] The visual motion estimation device 220 may be configured to use a set of time constants τ to determine the voltage value (v(t)) of the pixel. For example, the visual motion estimation device 220 may use a first time constant (τ slow) to determine the first voltage value of the pixel, and may use a second time constant (τ fast) to determine the second voltage value of the pixel. In addition, the visual motion estimation device 220 may be configured to determine the first time constant and the second time constant based on the following function:

[0057]

[0058] As indicated above, "R" may refer to the pixel rate, and "w" may refer to the pixel width of the DVS 210. According to one embodiment, the first time constant may be 20 ms and the second time constant may be 10 ms. However, it should be understood that the visual motion estimation device 220 may be configured to use n time constants having any specific values ​​(e.g., 1, 2, 3, 6, etc.) to determine the n voltage values ​​of the pixel.

[0059] like Figure 5 As further shown, the method may include generating a set of images based on a set of voltages of a set of pixels (operation 530 ). For example, the visual motion estimation device 220 may generate a set of images based on a set of voltages of a set of pixels of the DVS 210 .

[0060] The visual motion estimation device 220 may generate a set of images by grouping the voltages of a set of pixels based on polarity and time constant. For example, the visual motion estimation device 220 may generate a first image by grouping the voltages of a set of pixels based on positive polarity and a first time constant (τslow). In addition, the visual motion estimation device 220 may generate a second image by grouping the voltages of a set of pixels based on negative polarity and a first time constant. In addition, the visual motion estimation device 220 may generate a third image by grouping the voltages of a set of pixels based on positive polarity and a second time constant (τfast). In addition, the visual motion estimation device 220 may generate a fourth image by grouping the voltages of a set of pixels based on negative polarity and a second time constant.

[0061] like Figure 5 As further shown, the method may include inputting a set of images into a first neural network configured to output a visual motion estimate of the object (operation 540). For example, the visual motion estimation device 220 may input a set of images into the visual motion estimation neural network 430.

[0062] The input of the first neural network can be expressed as:

[0063]

[0064] As shown above, "a" and "b" define the number of pixels of a set of images. For example, the image can be a 15×15 image of pixel voltages. It should be understood that the size of the P×Q array can be a different size than 15×15.

[0065] The visual motion estimation device 220 may be configured to determine the size of the pixel group to be used to generate the image group. Figure 6 , the visual motion estimation device 220 may obtain information associated with the object (operation 610), and determine the size of the pixel group based on the information associated with the object using the third neural network (operation 620). For example, the information associated with the object may be the type of object (e.g., car, machine, person, sign, etc.), the size of the object (e.g., measurement value, geometric information, etc.), the number of objects to be detected by the DVS 210, the location of the object, the time frame in which the object detection is to be performed, the physical location of the DVS 210 relative to the object, the device to which the DVS 210 is installed, etc.

[0066] The first neural network can be configured to output a visual motion estimate based on an input image as follows:

[0067]

[0068] The visual motion estimation device 220 may be configured to generate a set of visual motion estimation values ​​for a set of axes. For example, the visual motion estimation device 220 may be configured to generate visual motion estimation values ​​u for θ values ​​p of 0, π / 8, π / 4, 3π / 8, etc. p and v p .

[0069] like Figure 5 As further shown, the method may include inputting a set of images into a second neural network configured to output a confidence score of the visual motion estimate output by the first neural network (operation 550). For example, the visual motion estimation device 220 may input a set of images into the confidence neural network 440.

[0070] The input to the second neural network can be the same as the input to the first neural network and can be expressed as:

[0071]

[0072] The second neural network can be configured to output a confidence score for the visual motion estimate of the first neural network based on a set of images, as follows:

[0073]

[0074] like Figure 5As further shown, the method may include obtaining a visual motion estimate of the object and a confidence score of the visual motion estimate of the object based on inputting the image group into the first neural network and the second neural network (operation 560). For example, the visual estimation device 220 may obtain the visual motion estimate and the confidence score of the object based on the outputs of the first neural network and the second neural network.

[0075] like Figure 5 As further shown, the method may include providing the visual motion estimate and the confidence score of the object (operation 570). For example, the visual motion estimation device 220 may provide the visual motion estimate and the confidence score of the object to a downstream application.

[0076] although Figure 5 Exemplary operations are shown, but the method may include additional operations, fewer operations, different operations, or operations that are different from the Figure 5 The operations shown in the method are arranged differently. Additionally, or alternatively, two or more operations of the method may be performed in parallel.

[0077] The foregoing disclosure provides illustration and description, but is not intended to be exhaustive or to limit the implementations to the precise form disclosed. Modifications and variations are possible in light of the above disclosure or may be acquired from practice of implementation.

[0078] As used herein, the term "component" is intended to be broadly interpreted as hardware, firmware, or a combination of hardware and software.

[0079] Obviously, the systems and / or methods described herein can be implemented with different forms of hardware, firmware, or a combination of hardware and software. The actual dedicated control hardware or software code used to implement these systems and / or methods does not limit the implementation. Therefore, the operation and behavior of the systems and / or methods are described herein without reference to specific software codes. It should be understood that software and hardware can be designed based on the description herein to implement the systems and / or methods.

[0080] Even though particular combinations of features are recited in the claims and / or disclosed in the specification, these combinations are not intended to limit the disclosure of possible implementations. In fact, many of these features may be combined in ways not specifically recited in the claims and / or disclosed in the specification. Although each dependent claim listed below is directly dependent on only one claim, the disclosure of possible implementations includes each dependent claim in combination with every other claim in the claim set.

[0081] Unless explicitly described, the elements, actions or instructions used herein should not be interpreted as critical or necessary. In addition, the articles "one" and "a kind of" as used herein are intended to include one or more items and can be used interchangeably with "one or more". In addition, as used herein, the term "group" is intended to include one or more items (e.g., related items, unrelated items, combinations of related and unrelated items, etc.), and can be used interchangeably with "one or more". In the case of only one item, the term "one" or similar language is used. In addition, as used herein, the terms "have", "have", "have" etc. are intended to be open terms. In addition, unless otherwise expressly stated, the phrase "based on" is intended to mean "based at least in part on".

Claims

1. A method comprising: Acquire a set of events for a set of pixels of a dynamic vision sensor associated with an object; determining a set of voltages for the set of pixels based on the set of events; generating a set of images based on the set of voltages of the set of pixels; inputting the set of images into a first neural network, the first neural network configured to output a visual motion estimate of the object; inputting the set of images into a second neural network, the second neural network configured to output a confidence score for the visual motion estimate output by the first neural network; obtaining the visual motion estimate of the object and the confidence score of the visual motion estimate of the object based on inputting the set of images into the first neural network and the second neural network; as well as The visual motion estimate and the confidence score of the object are provided.

2. The method according to claim 1, wherein: The set of events includes coordinate values ​​of the set of pixels, polarity values ​​of the set of pixels, and time of the set of events.

3. The method of claim 1, further comprising: determining a first set of voltage values ​​based on a first time constant; determining a second set of voltage values ​​based on a second time constant different from the first time constant; as well as The set of images is generated based on the first set of voltage values ​​and the second set of voltage values.

4. The method of claim 1, further comprising: Using a third neural network, a size of the set of pixels to be used to generate the set of images is determined.

5. The method of claim 4, further comprising: Based on the type of the object, the size of the group of pixels is determined.

6. The method of claim 4, further comprising: Based on a size of the object, the size of the group of pixels is determined.

7. The method of claim 4, further comprising: The size of the group of pixels is determined based on the number of objects to be detected by the dynamic vision sensor.

8. An apparatus comprising: a memory configured to store instructions; as well as A processor configured to execute the instructions to: Acquire a set of events for a set of pixels of a dynamic vision sensor associated with an object; determining a set of voltages for the set of pixels based on the set of events; generating a set of images based on the set of voltages of the set of pixels; inputting the set of images into a first neural network, the first neural network configured to output a visual motion estimate of the object; inputting the set of images into a second neural network, the second neural network configured to output a confidence score for the visual motion estimate output by the first neural network; obtaining the visual motion estimate of the object and the confidence score of the motion estimate of the object based on inputting the set of images into the first neural network and the second neural network; as well as The visual motion estimate and the confidence score of the object are provided.

9. The device according to claim 8, wherein: The set of events includes coordinate values ​​of the set of pixels, polarity values ​​of the set of pixels, and time of the set of events.

10. The apparatus of claim 8, wherein the processor is further configured to: determining a first set of voltage values ​​based on a first time constant; determining a second set of voltage values ​​based on a second time constant different from the first time constant; and The set of images is generated based on the first set of voltage values ​​and the second set of voltage values.

11. The apparatus of claim 8, wherein the processor is further configured to: Using a third neural network, a size of the set of pixels to be used to generate the set of images is determined.

12. The apparatus of claim 11, wherein the processor is further configured to: Based on the type of the object, the size of the group of pixels is determined.

13. The apparatus of claim 11, wherein the processor is further configured to: Based on a size of the object, the size of the group of pixels is determined.

14. The apparatus of claim 11, wherein the processor is further configured to: The size of the group of pixels is determined based on a number of a plurality of objects to be detected by the dynamic vision sensor.

15. A computer-readable medium storing instructions, which, when executed by one or more processors of a device, cause the one or more processors to perform operations corresponding to one of the methods of claims 1-7.

Images