Information processing device, information processing method, and information processing program

Abstract

We propose an information processing device that can efficiently classify event data into event data from stationary objects and event data from moving objects. [Solution] The information processing device includes: an acquisition unit that acquires event data relating to events caused by changes in the brightness of reflected light from an object and angular velocity data relating to the angular velocity of an imaging device; a motion correction unit that performs motion correction on the acquired event data based on the acquired event data and angular velocity data to correct for events caused by the movement of the sensor that detected the event data; and a segmentation unit that performs segmentation of the motion-corrected event data into event data caused by stationary objects and event data caused by moving objects.

Description

Technical Field

[0001] The present disclosure relates to an information processing apparatus, an information processing method, and an information processing program.

Background Art

[0002] There is known a technique for classifying data of an object in which an object moving relative to a sensor is detected into data of a stationary object that appears to have moved because the sensor has moved but is actually stationary, and data of a moving object that actually moves.

[0003] For example, there is known a technique for classifying event data (data related to an event generated by a change in the luminance of reflected light from an object), which is an example of object data, into event data caused by a stationary object such as a wall and event data caused by a moving object such as a bicycle. As an example, there is known a technique for classifying corrected event data into event data caused by a stationary object and event data caused by a moving object after correcting an event caused by the movement of a sensor that detected the event data (for example, Non-Patent Document 1).

Prior Art Documents

Non-Patent Documents

[0004]

Non-Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] Conventional technology performs motion correction on event data, which corrects for events caused by the movement of a sensor that detects event data. This motion correction then segments the event data into event data caused by stationary objects and event data caused by moving objects. However, conventional motion correction and segmentation technologies still have room for improvement in efficiently classifying event data into event data caused by stationary objects and event data caused by moving objects.

[0006] For example, if motion correction and segmentation calculations are repeated until event data is completely classified into event data from stationary objects and event data from moving objects, the number of calculations will increase as the number of events increases. Therefore, in motion correction and segmentation techniques, classifying event data into event data from stationary objects and event data from moving objects can be computationally expensive.

[0007] Therefore, this disclosure aims to propose an information processing device, an information processing method, and an information processing system that can efficiently classify event data into event data from stationary objects and event data from moving objects. [Means for solving the problem]

[0008] The information processing device according to this disclosure includes: an acquisition unit that acquires event data relating to events caused by changes in the brightness of reflected light from an object and angular velocity data relating to the angular velocity of an imaging device; a motion correction unit that performs motion correction on the acquired event data based on the acquired event data and angular velocity data to correct for events caused by the movement of the sensor that detected the event data; and a segmentation unit that performs segmentation of the motion-corrected event data into event data caused by stationary objects and event data caused by moving objects. [Brief explanation of the drawing]

[0009] [Figure 1] This is a diagram illustrating the overview of the information processing system according to the first embodiment. [Figure 2] This is a diagram illustrating an example of motion correction and segmentation. [Figure 3] This is a block diagram showing an example of the configuration of an information processing system according to the first embodiment. [Figure 4] This is a diagram illustrating the first example of motion correction. [Figure 5] This is a diagram to explain in detail the first example of motion correction. [Figure 6] This diagram provides a more detailed explanation of the first motion correction example. [Figure 7] This diagram provides a more detailed explanation of the first motion correction example. [Figure 8] This figure illustrates an example of 3D reconstruction of event data and the projection of the 3D reconstructed event data onto the pixel matrix of the EVS. [Figure 9] This diagram illustrates an example of event image generation. [Figure 10] This is a diagram illustrating an example of segmentation. [Figure 11] This is a diagram illustrating the second example of motion correction. [Figure 12] This is a diagram illustrating the third example of motion correction. [Figure 13] This is a diagram illustrating the fourth example of motion correction. [Figure 14] This diagram illustrates the details of the event depth data calculation. [Figure 15] This is a diagram illustrating the fifth example of motion correction. [Figure 16] This is a flowchart showing an example of the information processing flow according to the first embodiment. [Figure 17] This is a block diagram showing an example of the configuration of an information processing system according to the second embodiment. [Figure 18]This figure illustrates an example of motion correction by the motion correction unit of the first embodiment when the EVS undergoes translational movement. [Figure 19] This figure illustrates an example of motion correction when the EVS rotates, using the motion correction unit of the first embodiment. [Figure 20] This figure illustrates an example of motion correction by the motion correction unit of the second embodiment. [Figure 21] This figure illustrates the details of motion correction by the motion correction unit of the second embodiment. [Figure 22] This figure illustrates another example of motion correction by the motion correction unit of the second embodiment. [Figure 23] This is a block diagram showing an example of the configuration of an information processing system according to the third embodiment. [Figure 24] This is a hardware configuration diagram showing an example of a computer that implements the functions of an information processing device. [Modes for carrying out the invention]

[0010] The embodiments of this disclosure will be described in detail below with reference to the drawings. In the following embodiments, the same parts will be denoted by the same reference numerals, and redundant descriptions will be omitted.

[0011] The embodiments of this disclosure will be described below in the following order. 1. First Embodiment 1-1. Overview of the Information Processing System According to the First Embodiment 1-2. Configuration of the Information Processing System According to the First Embodiment 1-3. Information processing flow according to the first embodiment 2. Second Embodiment 3. Third Embodiment 4. Effects of the information processing device related to this disclosure 5. Hardware Configuration 6. Addendum

[0012] (1. First Embodiment) (1-1. Overview of the information processing system according to the first embodiment) The overview of the information processing system 1 according to the first embodiment will be explained using Figure 1. Figure 1 is a diagram illustrating the overview of the information processing system according to the first embodiment.

[0013] Information processing system 1 includes an information processing device 100, a terminal device 200, an EVS (Event-based Vision Sensor) 300, a gyro sensor 400, and a depth sensor 500. The number of EVS 300, gyro sensor 400, and depth sensor 500 included in information processing system 1 may be one or more. Also, the EVS 300, gyro sensor 400, and depth sensor 500 may be coaxial or non-coaxial. If the EVS 300 and depth sensor 500 are coaxial, these sensors may be an integrated sensor.

[0014] The information processing device 100 is, for example, a server with sensor fusion technology. Sensor fusion technology is a technology that performs information processing that cannot be done from a single data point, based on multiple data points detected by multiple sensors such as an EVS 300, a gyro sensor 400, and a depth sensor 500.

[0015] The information processing device 100 classifies the data of objects detected moving relative to the EVS 300 into data for stationary objects and data for moving objects. Stationary objects are objects that appear to move because the EVS 300 is moving, but are actually stationary. For example, stationary objects are walls, etc. Moving objects are objects that are actually moving. For example, moving objects are bicycles, pedestrians, etc.

[0016] Let's return to the explanation of the information processing device 100. The information processing device 100, for example, classifies event data 600, which is an example of object data, into event data 700 caused by stationary objects such as walls, and event data 800 caused by moving objects such as bicycles.

[0017] Event data 600 is data related to events that occurred due to changes in the brightness of reflected light from an object. For example, event data 600 is point cloud data in which objects moving relative to EVS300 are represented as points.

[0018] Specifically, event data 600 includes the location (x, y) of each event corresponding to multiple black spots that occurred in the pixel matrix of the EVS300 sensor array, the time t at which the event occurred, and the polarity p of the event. The polarity p of the event represents a change in brightness, such as an increase or decrease in brightness. For example, if the brightness increases, the polarity p of the event will be +1 (On), and if the brightness decreases, it will be -1 (Off).

[0019] Event data 700 from stationary objects is data related to events (events in Egomotion) detected when, for example, the EVS300 photographs a stationary object while moving. Event data 700 is also an image of a stationary object, such as a wall. Event data 800 is an image of a moving object, such as a bicycle.

[0020] Returning to the explanation of the information processing device 100, the information processing device 100 performs motion correction (motion compensation) on the event data 600 to correct for events caused by the movement of the EVS 300 that detected the event data 600. The information processing device 100 then performs segmentation of the motion-corrected event data 600, classifying it into event data 700 caused by stationary objects and event data 800 caused by moving objects.

[0021] Specifically, the information processing device 100, based on setting data related to motion correction and segmentation settings received from the terminal device 200, corrects the blur caused by the movement of the EVS 300 when detecting the event data 600 as motion correction.

[0022] Next, the information processing device 100 performs segmentation by classifying the shake-corrected event data 600 into event data 700 caused by stationary objects such as walls and event data 800 caused by moving objects such as bicycles. Subsequently, the information processing device 100 transmits the event data 700 and event data 800 to the terminal device 200.

[0023] The terminal device 200 is, for example, a smartphone or a personal computer. For example, the terminal device 200 transmits setting data related to various settings to the information processing device 100 via an application.

[0024] As an example, terminal device 200 transmits data related to segmentation settings (e.g., event contrast threshold) to information processing device 100 as configuration data. As another example, terminal device 200 transmits data related to the format settings of event data 700 for stationary objects and event data 800 for moving objects to information processing device 100 as configuration data. Terminal device 200 also receives event data 700 for stationary objects and event data 800 for moving objects from information processing device 100.

[0025] The EVS300 is a sensor that detects event data 600. The EVS300 can also function as an imaging device such as a camera. The gyro sensor 400 is an angular velocity sensor that measures angular velocity data related to the angular velocity of the EVS300 and depth sensor 500, which also function as imaging devices. The gyro sensor 400 is an IMU (Inertial Measurement Unit) that also measures acceleration data related to the acceleration of the imaging device. The depth sensor 500 is a sensor that measures the distance (depth) between the depth sensor 500 and an object. The depth sensor 500 is, for example, a Time of Flight (ToF) type sensor. The depth sensor 500 can also function as an imaging device.

[0026] The information processing device 100 can classify event data 600 into event data 700 caused by stationary objects and event data 800 caused by moving objects. However, in the motion correction and segmentation techniques described above, there is generally room for improvement in efficiently classifying event data 600 into event data 700 caused by stationary objects and event data 800 caused by moving objects.

[0027] The following explanation uses a technique that repeatedly performs motion correction and segmentation calculations based solely on event data 600, until the event data 600 is completely classified into event data 700 from stationary objects and event data 800 from moving objects. Note that "until the event data 700 from stationary objects and event data 800 from moving objects are completely classified" means until the maximum contrast value of the events converges.

[0028] In this type of technology, for example, if the EVS300 is capturing a large number of moving objects or objects moving at high speed, the EVS300 will detect a large number of events, which may increase the number of calculation iterations. Furthermore, an increase in the number of pixels of the EVS300 could also increase the number of calculation iterations. Therefore, in this type of technology, classifying event data 600 into event data 700 from stationary objects and event data 800 from moving objects may incur computational costs.

[0029] In order to solve the problem of efficiently classifying event data 600 into event data 700 from stationary objects and event data 800 from moving objects, the information processing device 100 acquires angular velocity data in addition to the event data 600.

[0030] Next, the information processing device 100 performs motion correction based on the acquired event data 600 and angular velocity data. Subsequently, the information processing device 100 performs segmentation of the motion-corrected event data 600, classifying it into event data 700 caused by stationary objects and event data 800 caused by moving objects.

[0031] In this way, the information processing device 100 performs motion correction based on angular velocity data in addition to the event data 600, and then classifies the event data 600 into event data 700 from stationary objects and event data 800 from moving objects. As a result, the information processing device 100 does not have to repeatedly perform motion correction and segmentation calculations based solely on the event data 600, and can efficiently classify the event data 600 into event data 700 from stationary objects and event data 800 from moving objects.

[0032] The following describes an example of the information processing device 100 performing the processes shown in steps S1 to S5 of Figure 1.

[0033] The terminal device 200 transmits configuration data related to various settings to the information processing device 100 (step S1). For example, the terminal device 200 transmits data related to segmentation settings (e.g., event contrast threshold) to the information processing device 100 as configuration data. The information processing device 100 acquires the configuration data by receiving it from the terminal device 200.

[0034] Next, the information processing device 100 acquires event data 600, angular velocity data, acceleration data, and depth data (step S2).

[0035] For example, the information processing device 100 acquires event data 600 detected by the EVS 300 by receiving it from the EVS 300. The information processing device 100 also acquires angular velocity data and acceleration data measured by the gyro sensor 400 by receiving them from the gyro sensor 400. Furthermore, the information processing device 100 acquires depth data measured by the depth sensor 500 by receiving it from the depth sensor 500.

[0036] Next, the information processing device 100 performs motion correction on the acquired event data 600 based on the acquired angular velocity data, acceleration data, and depth data (step S3). An example of motion correction will be explained below using Figure 2. Figure 2 is a diagram illustrating an example of motion correction and segmentation.

[0037] For example, based on the acquired angular velocity data, acceleration data, and depth data, the information processing device 100 corrects the blur caused by the movement of the EVS 300 when detecting the acquired event data 600 as motion correction. As a result, the information processing device 100 acquires event data 600X with the blur corrected. The information processing device 100 also generates an event image 900 with the blur corrected from the acquired event data 600.

[0038] Returning to the explanation of Figure 1, the information processing device 100 then performs segmentation of the motion-corrected event data 600, classifying it into event data 700 caused by stationary objects and event data 800 caused by moving objects (step S4). An example of segmentation will be explained below using Figure 2.

[0039] The information processing device 100 calculates the contrast of events in the event image 900. The information processing device 100 removes events from the event image 900 whose calculated event contrast is greater than or equal to the event contrast threshold set by the setting data. For example, the information processing device 100 classifies the event image 900 into event data 700 caused by stationary objects and event data 800 caused by moving objects by performing ego-motion cancellation to remove event data caused by the movement of the EVS 300.

[0040] In this case, the information processing device 100 can further classify the event data 800 generated by each moving object. For example, the information processing device 100 can classify the event data 800 generated by moving objects into event data 800a generated by a first moving object such as a bicycle, and event data 800b generated by a second moving object such as a pedestrian.

[0041] Returning to the explanation of Figure 1, the information processing device 100 then transmits event data 700 from classified stationary objects and event data 800 from moving objects to the terminal device 200 (step S5). For example, the information processing device 100 transmits event data 700 from stationary objects, which are images of stationary objects such as walls, and event data 800 from moving objects such as bicycles to the terminal device 200.

[0042] Specifically, the information processing device 100 transmits event data 800 from a moving object to the terminal device 200, consisting of event data 800a from a first moving object such as a bicycle, and event data 800b from a second moving object such as a pedestrian. The terminal device 200 also receives event data 700 from stationary objects and event data 800 from moving objects from the information processing device 100.

[0043] In this way, the information processing device 100 performs motion correction based on angular velocity data in addition to the event data 600, and then classifies the event data 600 into event data 700 from stationary objects and event data 800 from moving objects. As a result, the information processing device 100 does not have to repeatedly perform motion correction and segmentation calculations based solely on the event data 600, and can efficiently classify the event data 600 into event data 700 from stationary objects and event data 800 from moving objects.

[0044] (1-2. Configuration of the information processing system according to the first embodiment) An example of the configuration of the information processing system 1 according to the first embodiment will be explained using Figure 3. Figure 3 is a block diagram showing an example of the configuration of the information processing system according to the first embodiment. The configuration of the information processing device 100 and the terminal device 200 included in the information processing system 1 will be explained.

[0045] (Configuration of information processing device) The information processing device 100 comprises a communication unit 110, a storage unit 120, and a control unit 130.

[0046] (Communications Department) The communication unit 110 is implemented by, for example, a network interface controller or a network interface card (NIC). The communication unit 110 may also be a USB interface consisting of a USB (Universal Serial Bus) host controller, a USB port, etc. Furthermore, the communication unit 110 may be a wired interface or a wireless interface. For example, the communication unit 110 may be a wireless communication interface using a wireless LAN method or a cellular communication method.

[0047] The communication unit 110 functions as a communication means or transmission means for the information processing device 100. For example, the communication unit 110 is connected to the network N by wire or wireless connection and transmits and receives information to and from external devices such as cloud servers and other information processing terminals via the network N. The network N is implemented using wireless communication standards or methods such as Bluetooth®, the Internet, Wi-Fi®, UWB (Ultra-Wide Band), LPWA (Low Power Wide Area), and ELTRES®.

[0048] For example, the communication unit 110 receives configuration data from the terminal device 200. As one example, the communication unit 110 receives data related to segmentation settings (e.g., event contrast threshold) from the terminal device 200 as configuration data. As another example, the communication unit 110 receives data related to the format settings of event data 700 for stationary objects and event data 800 for moving objects from the terminal device 200 as configuration data.

[0049] Furthermore, the communication unit 110 transmits event data 700 from stationary objects and event data 800 from moving objects to the terminal device 200. For example, the communication unit 110 transmits event data 700 from stationary objects, which are images of stationary objects such as walls, and event data 800 from moving objects such as bicycles to the terminal device 200.

[0050] Specifically, the communication unit 110 transmits event data 800 from a moving object to the terminal device 200, consisting of event data 800a from a first moving object such as a bicycle, and event data 800b from a second moving object such as a pedestrian.

[0051] (Storage part) The memory unit 120 is implemented by, for example, semiconductor memory elements such as RAM (Random Access Memory) and flash memory, or by storage devices such as hard disks and optical discs. For example, the memory unit 120 stores various data such as various setting data, event data 600, event data 700 from stationary objects, and event data 800 from moving objects.

[0052] (Control Unit) The control unit 130 is implemented, for example, by a CPU (Central Processing Unit) or MPU (Micro Processing Unit) executing a program (for example, the information processing program according to this disclosure) stored inside the information processing device 100 using RAM (Random Access Memory) or the like as a working area. The control unit 130 is also a controller and may be implemented by an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or FPGA (Field Programmable Gate Array).

[0053] The control unit 130 includes an acquisition unit 131, a motion correction unit 132, and a segmentation unit 133.

[0054] (Acquisition Department) The acquisition unit 131 acquires event data 600 and angular velocity data. For example, the acquisition unit 131 acquires event data 600 detected by the EVS 300 by receiving it from the EVS 300. The acquisition unit 131 also acquires angular velocity data and acceleration data measured by the gyro sensor 400 by receiving them from the gyro sensor 400. Furthermore, the acquisition unit 131 acquires depth data measured by the depth sensor 500 by receiving it from the depth sensor 500.

[0055] The acquisition unit 131 can also acquire event-based angular velocity data calculated from event data 600 as angular velocity data.

[0056] For example, the acquisition unit 131 acquires multi-viewpoint event data as event data 600, which is generated from multiple viewpoints resulting from the movement of the EVS300's viewpoint along with the movement of the EVS300. The acquisition unit 131 also acquires multi-viewpoint event-based angular velocity data (for example, the rotation angle and displacement of the EVS300) calculated from the feature points of the multiple viewpoints detected from the multi-viewpoint event data, as angular velocity data.

[0057] The acquisition unit 131 can also acquire different-viewpoint depth data as depth data, which is measured from a different viewpoint than the event data 600. For example, the acquisition unit 131 can acquire depth data measured by the EVS 300 and a non-coaxial depth sensor 500 as different-viewpoint depth data.

[0058] The acquisition unit 131 can also acquire same-viewpoint depth data measured from the same viewpoint as the event data 600. For example, the acquisition unit 131 acquires depth data measured by a coaxial depth sensor 500 with the EVS 300 as same-viewpoint depth data. As an example, the acquisition unit 131 acquires depth data measured by a depth sensor 500 integrated with the EVS 300 as same-viewpoint depth data.

[0059] The acquisition unit 131 can also acquire as depth data different from event data 600, by calculating from different viewpoint event data detected from different viewpoints. For example, the acquisition unit 131 acquires event data detected by the EVS 300 as event data 600. In addition, the acquisition unit 131 acquires as different viewpoint event-based depth data different from event data detected by the EVS 300 and a non-coaxial EVS.

[0060] The acquisition unit 131 can also acquire event-based depth data calculated from event data 600 as depth data. For example, the acquisition unit 131 can acquire multi-viewpoint event-based depth data calculated from feature points of multiple viewpoints detected from the multi-viewpoint event data mentioned above as event-based depth data.

[0061] (Motion correction unit) The motion correction unit 132 performs motion correction on the acquired event data 600 based on the acquired angular velocity data.

[0062] For example, the motion correction unit 132 corrects the blur caused by the movement of the EVS 300 when detecting the event data 600, based on the acquired angular velocity data, acceleration data, and depth data. As a result, the motion correction unit 132 acquires event data 600X with the blur corrected. The motion correction unit 132 also generates an event image 900 with the blur corrected from the acquired event data 600.

[0063] (Example of motion correction for the first time) The first motion correction example will be explained below using Figure 4. Figure 4 is a diagram illustrating the first motion correction example. The motion correction unit 132 performs motion correction based on the event data 600 and the different viewpoint angular velocity data, different viewpoint acceleration data, and different viewpoint depth data measured from different viewpoints than the event data 600 (step S10).

[0064] For example, the motion correction unit 132 performs motion correction based on the angular velocity data and acceleration data measured by the EVS 300 and the non-coaxial gyro sensor 400, which are used as different viewpoint angular velocity data and different viewpoint acceleration data. The motion correction unit 132 also performs motion correction based on the different viewpoint depth data measured by the EVS 300 and the non-coaxial depth sensor 500, which are used as different viewpoint depth data.

[0065] The details of the first motion correction example will be explained below using Figure 5. Figure 5 is a diagram for explaining the first motion correction example in detail. The motion correction unit 132 performs motion correction based on the following data. • Multiple viewpoint event data detected from multiple viewpoints resulting from the movement of the EVS300 as the EVS300's viewpoint moves. • Angle velocity data and acceleration data from different viewpoints measured by the EVS300 and the non-coaxial gyro sensor 400. Depth data when the distance 50 between the depth sensor 500, located at approximately the same position as the EVS300, and the object 6000 is distance 50a, distance 50b, distance 50c, distance 50d, and distance 50e.

[0066] Hereinafter, using FIG. 6, the first motion correction example in FIG. 5 will be described in more detail. FIG. 6 is a diagram for explaining the first motion correction example in more detail. The horizontal axis of FIG. 6 is time. Among the times on the horizontal axis, time t ,

[0069] , j , , j , , ,

[0068] , i , time t and time t j corresponds to time t i , time t and time t j in FIG. 5, respectively. Also, the bars and ellipses arranged along the horizontal axis of FIG. 6 are various data acquired at the timings corresponding to the time on the horizontal axis.

[0067] The angular velocity data and acceleration data 4000 are measured by the gyro sensor 400 at least at time t i and time t j . The event data 600 is measured by the EVS300 at least at time t i and time t. The depth data 5000 is measured by the depth sensor 500 at least at time t i and time t j .

[0068] For example, as shown in the upper diagram of FIG. 6, the motion correction unit 132 starts motion correction at time t j which is the same timing as the timing when the depth data 5000 is acquired. The motion correction unit 132 can also start motion correction at time t which is a timing different from the timing when the depth data 500 is acquired, as shown in the lower diagram of FIG. 6. Thereby, since the motion correction unit 132 can correct the events caused by the motion of the EVS300 at the time t when the depth data 5000 is not acquired, the bias of the events can be reduced.

[0069] The motion correction unit 132 can also perform motion correction on the event data weighted by the Gaussian function as the event data 600, as shown in the lower diagram of FIG.. For example, the motion correction unit 132 is at time t i and time t jMotion correction is applied to event data that has been weighted with a Gaussian function so that the weight is maximized for a particular event.

[0070] As a result, the motion correction unit 132 adjusts the time t around the time t i and time t j By utilizing the acquired depth data 5000, and correcting for events caused by the movement of EVS300 at time t where depth data 5000 was not acquired, the bias in events can be reduced. Furthermore, the motion correction unit 132 can also weight the event data 600 using means other than a Gaussian function, as long as the weights are weighted so that the weights peak at the aforementioned times.

[0071] The first correction will be explained in more detail below using Figure 7. Figure 7 is a diagram for explaining the first motion correction example in more detail. Step S10 in Figure 7 corresponds to step S10 in Figure 4. In Figure 7, step S10 includes steps S11 to S16 and step S18. In addition, after steps S11 to S16 of step S10, segmentation is performed by the segmentation unit 133 (step S17).

[0072] (Step S11) The motion correction unit 132 estimates the state of the EVS 300 based on angular velocity data and acceleration data measured by the gyro sensor 400.

[0073] Below, using Figure 5, EVS300 is at time t i ~time t j An example of how the motion correction unit 132 estimates the state of the EVS300 when it moves to time t will be explained. First, the motion correction unit 132 estimates the state of the EVS300 as time t from the following equation (1). j EVS300 posture R j We will estimate the following. Below, we assume that the EVS300 functions as a camera. We also assume that the gyro sensor 400 is an IMU that also functions as an accelerometer.

[0074]

number

[0075] Equation (1) is given by time t i EVS300 posture R i The gyro sensor 400 determines the time t i ~time t j-1 The angular velocity of the EVS300 measured during this time w i ~angular velocity w j-1 By using and, time t j This is the formula for estimating the attitude of the EVS300. R in formula (1) j This is the rotation matrix representing the attitude of EVS300 at time tj. k -b k g -η k gb This is the turnover rate of EVS300 at time tk.

[0076] b k g This is the time t of the gyro sensor 400, which is the IMU. k This is the bias at η. k gb This is an event caused by the movement of EVS300 at time tk. k g ya η k gb This is determined from event data 600 detected in advance by EVS300, angular velocity data measured by gyro sensor 400, and the datasheets of these sensors. Δt represents the sampling period, which is the time required for one measurement by gyro sensor 400.

[0077] Next, the motion correction unit 132 calculates the state of EVS300 as follows from equation (2): time t j EVS300 movement speed v j We estimate this.

[0078]

number

[0079] v in equation (2) j is, time t j This is a vector representing the movement speed of the EVS300. i is, time t i This is a vector representing the velocity of the EVS300. g is the gravitational constant. Δt ij is, time t j and time t i It is the time between these two points. k This is estimated from equation (1), time t k This is the rotation matrix representing the attitude of the EVS300. k -b k a -η k gb is, time t k This is the effective acceleration of the EVS300. k The IMU, which is a gyro sensor 400, measures the time t k This is a vector representing the theoretical acceleration of the EVS300 when measured at [location / device].

[0080] Next, the motion correction unit 132 calculates the state of EVS300 as follows from equation (3): time t j Estimate the position of the EVS300.

[0081]

number

[0082] Equation (3) is given by time t i The position of EVS300 p i Then, the IMU, which is a gyro sensor 400, determines the time t i ~time t j-1 The acceleration a of the EVS300 measured during this period i ~acceleration a j-1 By using the above, the time t j This is the formula for estimating the position of the EVS300. In formula (3), p j is, time t j This is a vector representing the position of EVS300. k is, time t kThis is a vector representing the movement speed of the EVS300.

[0083] (Step S12) Returning to the explanation of Figure 7, the motion correction unit 132 interpolates the attitude of the EVS300. Using Figure 5 below, the EVS300 is at time t i ~time t j An example of interpolation of the attitude of the EVS300 by the motion correction unit 132 when it moves to a specific location will be explained. First, as interpolation of the attitude of the EVS300, the motion correction unit 132 uses the following equation (4) to determine the attitude of the EVS300 to be interpolated at time t i and time t j We estimate the displacement p(s) of the EVS300 interpolated between the two points.

[0084]

number

[0085] Equation (4) is given by time t i The position and time of the EVS300 j Along the translational movement of EVS300 between the position of EVS300 and time t i and time t j This is an equation for estimating the displacement p(s) of the EVS300 interpolated between time t. In equation (4), s is equal to time t. i The position and time of the EVS300 j These are interpolation coefficients that are interpolated into pi and pj to perform interpolation along the translational movement of EVS300 between the position of EVS300 and the time. The interpolation coefficient s(t) at time t can be expressed, for example, as shown in equation (5) below.

[0086]

number

[0087] Next, the motion correction unit 132 calculates the attitude of the EVS300 to be interpolated from the following equation (6) as time t i and time t j The rotation angle q(s) of the EVS300 is estimated by interpolating between the two values.

[0088] [Mathematics]

[0089] Equation (6) is an equation for interpolating the rotation angle q(s) of EVS300 by Slerp (Spherical linear interpolation), which is spherical linear interpolation of quaternions. The q in Equation (6) i and q j are, respectively, the rotation matrices of EVS300 at time t i and time t j of the rotation matrix R of EVS300 i and R j which are quaternions normalized from. θ is the angle that is half of the angular distance represented by the distance between q i and q j expressed in terms of an angle. θ and sin(θ) are obtained, for example, from cos(θ) in Equation (7) below. [[ID=Z7]]

[0090] [Mathematics] [[ID=Z2]] [[ID=Z3]] [[ID=Z4]] [[ID=Z5]] [[ID=Z6]]

[0091] [[ID=Z7]] [[ID=Z8]]The x in Equation (7) [[ID=Z9]] i and x j are, respectively, the positions of events in the x-axis direction in the camera coordinate system of EVS300 that occurred at time t i and time t j in the pixel matrix of EVS300. y i and y j are, respectively, the positions of events in the y-axis direction in the camera coordinate system of EVS300 that occurred at time t i and time t j in the pixel matrix of EVS300. z i and z j are, respectively, the distances (depth) between the depth sensor 500 and the object 6000 at time t i and time t j .

[0092] (Step S13) Returning to the explanation of Figure 7, the motion correction unit 132 adjusts the event data 600 detected by the EVS 300. For example, the motion correction unit 132 weights the event data 600 detected by the EVS 300 using a Gaussian function. The motion correction unit 132 can also adjust the event data 600 based on the classification results after the event data 700 from stationary objects and the event data 800 from moving objects are classified in the segmentation in step S17 (step S18).

[0093] (Step S14) The motion correction unit 132 performs three-dimensional reconstruction of the event data 600. Below, an example of three-dimensional reconstruction of the event data 600 will be explained using Figure 8. Figure 8 is a diagram illustrating an example of three-dimensional reconstruction of event data and projection of the three-dimensionally reconstructed event data onto the pixel matrix of the EVS. As for the three-dimensional reconstruction of the event data 600, the motion correction unit 132 calculates time t from the following equation (8): i ~time t j Perform a 3D reconstruction of all 600 event data points detected up to that point.

[0094]

number

[0095] Equation (8) is given by time t i ~time t j This formula reconstructs the points in the camera coordinate system (x, y, 1) of the 600 individual event data points detected up to that point into 3D points in the world coordinate system (X, Y, Z). w is the time t i or time t j This is the depth. Note that in Figure 6, the depth at time t was not detected by the depth sensor 500, so when the motion correction unit 1132 performs 3D reconstruction, it approximates time t to time t. i or time t j Utilize the depth. K -1 This is the inverse matrix of the internal parameters of the EVS300, which functions as a camera.

[0096] (Step S15) The motion correction unit 132 projects the three-dimensionally reconstructed event data 600 onto the pixel matrix (sensor surface) of the EVS300. An example of the projection of the three-dimensionally reconstructed event data 600 onto the sensor surface of the EVS300 will be explained below using Figure 8. The motion correction unit 132 estimates the position (x', y') of the event data 600 projected onto the sensor surface of the EVS300 from the following equations (9) and (10).

[0097]

number

number

[0098] Equations (9) and (10) are formulas that perform viewpoint transformation of the EVS300 using ΔR and Δp, and then project the 3D point (X, Y, Z) onto the sensor surface of the EVS300 using K. K is an internal parameter of the EVS300. ΔR is the time t i ~time t j This is the amount of rotation of EVS300 during time t. Δp is the amount of rotation of EVS300 during time t. i ~time t j This is the translational displacement of EVS300 between these points.

[0099] Combining equations (1) through (10), we obtain the following equation (11). x' in equation (11) j is, time t j This is the position of event data 600 projected onto the sensor surface of the EVS300. In other words, x' j This is the position vector of event data 600 after motion correction.

[0100]

number

[0101] t tk f t j is, time tk From the attitude and position of the EVS300 to time t j This is a matrix representing the operation to convert the attitude and position of the EVS300. tk f t j This corresponds to the operation of changing the viewpoint of the EVS300 using ΔR and Δp in equation (9). Z(x j ) is the time t in equation (8) j This corresponds to the depth w of Z(x). In other words, Z(x j ) is x j This is the corresponding depth. π -1 This is K in equation (8). -1 It corresponds to this.

[0102] (Step S16) The motion correction unit 132 generates an event image 900, which is an image that has undergone motion correction. An example of event image generation is illustrated using Figure 9. Figure 9 is a diagram illustrating an example of event image generation. The motion correction unit 132 generates an event image 900a by correcting the blur caused by the movement of the EVS 300 when detecting the event data 600a.

[0103] For example, the motion correction unit 132 generates an event image 900a by integrating the event data 600 projected onto the sensor surface of the EVS 300 using the following equation (12).

[0104]

number

[0105] I in equation (12) k is, time t k This is event image 900a. j This is the j-th event data, 600. K is, time t i ~time t j The event data is 600, detected during that period. δ is the Dirac delta function.

[0106] (Second example of motion correction) The second motion correction example will be explained below using Figure 11. Figure 11 is a diagram illustrating the second motion correction example. The motion correction unit 132 calculates the different viewpoint event-based depth data from the different viewpoint event data detected from a different viewpoint than the event data 600 (step S9). For example, the motion correction unit 132 calculates the different viewpoint event-based depth data using a calculation means based on triangulation from the different viewpoint event data detected by EVS300 and the non-coaxial EVS300a, which are examples of different viewpoint event data.

[0107] Next, the motion correction unit 132 performs motion correction based on the event data 600, the angular velocity data from different viewpoints, the acceleration data from different viewpoints, and the event-based depth data from different viewpoints (step S10a).

[0108] (Third example of motion correction) The third motion correction example will be explained below using Figure 12. Figure 12 is a diagram illustrating the third motion correction example. The motion correction unit 132 performs motion correction based on same-viewpoint depth data detected from the same viewpoint as the event data 600 (step S10b). For example, the motion correction unit 132 performs motion correction based on same-viewpoint depth data detected by the depth sensor 500 integrated with the EVS 300 as same-viewpoint depth data.

[0109] (Example of motion correction for the fourth time) The fourth motion correction example will be explained below using Figure 13. Figure 13 is a diagram illustrating the fourth motion correction example. First, before describing the motion correction by the motion correction unit 132 in step S10c, an example of acquiring depth data used by the motion correction unit 132 for motion correction will be explained using the acquisition unit 131.

[0110] The acquisition unit 131 acquires event-based depth data from feature points detected from the event data 600 as depth data (step S8). For example, the acquisition unit 131 calculates multi-viewpoint event-based depth data based on feature points detected from multi-viewpoint event data, which is an example of event data 600, and a calculation means based on triangulation. In this way, the acquisition unit 131 acquires multi-viewpoint event-based depth data.

[0111] Next, the motion correction unit 132 performs motion correction based on the event data 600, the angular velocity data from different viewpoints, the acceleration data from different viewpoints, and the event-based depth data (step S10c). For example, the motion correction unit 132 performs motion correction based on multi-viewpoint event-based depth data as the event-based depth data.

[0112] The details of acquiring event-based depth data will be explained below using Figure 14. Figure 14 is a diagram illustrating the details of calculating event depth data. First, the acquisition unit 131 generates event image 900c, which includes multiple frame images, event images 900c1 to 900c3, from the multi-viewpoint event data (step S81).

[0113] Next, the acquisition unit 131 detects feature points from the event image 900c (step S82). For example, the acquisition unit 131 detects feature points from the event image 900c by tracking a series of feature points, such as event image 900cX, through event images 900c1 to 900c3 using the KLT (Kanade-Lucas-Tomasi) algorithm.

[0114] Next, the acquisition unit 131 acquires event-based depth data from the feature points of the detected event image 900c (step S83). For example, the acquisition unit 131 acquires event depth data by estimating the depth from the 3D coordinates of the feature points of the event image 900c using a calculation means based on triangulation and PnP (Perspective-n-Point) which estimates the real-world orientation of the EVS300.

[0115] (Fifth example of motion correction) The fifth motion correction example will be explained below using Figure 15. Figure 15 is a diagram illustrating the fifth motion correction example. First, before describing the motion correction by the motion correction unit 132 in step S10d, an example of acquiring angular velocity data used by the motion correction unit 132 for motion correction by the acquisition unit 131 will be explained.

[0116] The acquisition unit 131 acquires multi-viewpoint event-based angular velocity data from feature points detected from the event data 600 as angular velocity data (step S7). For example, the motion correction unit 132 calculates the rotation angle and displacement of the EVS 300 as multi-viewpoint event-based angular velocity data, thereby acquiring the rotation angle and displacement of the EVS 300 as angular velocity data.

[0117] Next, the motion correction unit 132 performs motion correction based on the event data 600, multi-viewpoint event-based angular velocity data, different-viewpoint acceleration data, and event-based depth data (step S10d). For example, the motion correction unit 132 performs motion correction based on the rotation angle and displacement of the EVS 300 as multi-viewpoint event-based angular velocity data.

[0118] (Segmentation Department) Returning to the explanation of Figure 3, the segmentation unit 133 performs segmentation. For example, the segmentation unit 133 will explain an example of segmentation using Figure 10 below. Figure 10 is a diagram illustrating an example of segmentation. In Figure 10, the motion-corrected event image 900b includes a stationary object 7000 and a moving object 8000. The segmentation unit 133 also performs segmentation based on the contrast threshold of the motion-corrected event image 900b.

[0119] In this case, the segmentation unit 133 first calculates the contrast of events in the event image 900b. Next, the segmentation unit 133 removes from the event image 900b any events whose calculated event contrast is greater than or equal to the event contrast threshold set by the setting data.

[0120] For example, the segmentation unit 133 extracts event data 700b from the event image 900b by extracting only events from the event image 900b whose contrast is above a threshold using ego-motion cancellation. The segmentation unit 133 also extracts event data 800b from the event image 900b by extracting only events from the event image 900b whose contrast is below a threshold.

[0121] Specifically, the segmentation unit 133 is determined from the following equations (13) and (14): k Time t k Segmentation is performed based on an event contrast threshold that is directly proportional to the variance f(θ) of event image 900b, H(x;θ).

[0122]

number

number

[0123] N in equation (13) e and N p This is the total number of events. e and N p This means they can be the same or different. k σ is a constant calculated from the contrast threshold, corresponding to the contrast threshold. 2 h is the variance. ij is, time t i ~time t j This is the value of each event image up to 900b. μ H This is the mean value of H(x;θ).

[0124] (Terminal device configuration) Returning to the explanation of Figure 3, the terminal device 200 comprises a reception unit 210, a communication unit 220, a storage unit 230, a display unit 240, and a control unit 250.

[0125] (Reception Department) The reception unit 210 is a user interface, etc. For example, the reception unit 210 is a touch panel, etc. The reception unit 210 receives setting data from the user of the terminal device 200. For example, the reception unit 210 receives the event contrast threshold as setting data from the user via an application.

[0126] (Communications Department) The communication unit 220 is implemented by, for example, a network interface controller or NIC. The communication unit 220 may also be a USB interface consisting of a USB host controller, USB port, etc. Furthermore, the communication unit 220 may be a wired interface or a wireless interface. For example, the communication unit 220 may be a wireless communication interface using a wireless LAN method or a cellular communication method.

[0127] The communication unit 220 functions as a communication or transmission means for the terminal device 200. For example, the communication unit 220 is connected to the network N by wire or wireless connection and transmits and receives information to and from external devices such as cloud servers and other information processing terminals via the network N. The network N is implemented using wireless communication standards or methods such as Bluetooth®, the Internet, Wi-Fi®, UWB, LPWA, ELTRES®, etc.

[0128] For example, the communication unit 220 transmits configuration data to the information processing device 100 via an application. As one example, the communication unit 220 transmits data related to segmentation settings (e.g., event contrast threshold) to the information processing device 100 as configuration data. As another example, the communication unit 220 transmits data related to the format settings of event data 700 for stationary objects and event data 800 for moving objects to the information processing device 100 as configuration data.

[0129] Furthermore, the communication unit 220 receives event data 700 from stationary objects and event data 800 from moving objects from the information processing device 100.

[0130] (Storage part) The storage unit 230 is implemented by, for example, a semiconductor memory element such as RAM or flash memory, or a storage device such as a hard disk or optical disc. For example, the storage unit 230 stores various data such as setting data, event data 700 from stationary objects, and event data 800 from moving objects.

[0131] (Display) The display unit 240 is a desktop or the like that displays various data. For example, the display unit 240 is a touch panel or the like that displays various data such as setting data, event data 700 from stationary objects, and event data 800 from moving objects.

[0132] (Control Unit) The control unit 250 is implemented, for example, by a CPU or MPU, which executes a program stored inside the terminal device 200 using RAM or the like as a working area. The control unit 250 is also a controller and may be implemented by an integrated circuit such as an ASIC or FPGA.

[0133] (1-3. Information processing flow according to the first embodiment) An example of the information processing flow by the information processing device 100 according to the first embodiment will be explained using Figure 16. Figure 16 is a flowchart of an example of the information processing flow according to the first embodiment. Hereinafter, when equations (1) to (14) described above are used, Ts is time t i , Ts+T is at time t j This is based on the premise that...

[0134] The motion correction unit 132 initializes the event image 900, which is a motion correction image (step S101). Next, the motion correction unit 132 waits until a certain time T has elapsed (step S102). During this time, the acquisition unit 131 acquires the event data 600, angular velocity data, acceleration data, and depth data.

[0135] Next, the motion correction unit 132 estimates the state of the EVS300 using the acceleration data and angular velocity data acquired within the time frame Ts~Ts+T (step S103). For example, from equations (1) to (3), the motion correction unit 132 estimates that the state of the EVS300 is at time t i ~time t j Estimate the state of the EVS300 when it is moved to [location].

[0136] Next, the motion correction unit 132 interpolates the amount of movement and rotation angle of the EVS300 (step S104). For example, from equations (4) to (7), the motion correction unit 132 calculates that the EVS300 moves at time t i ~time t j Interpolates the attitude of the EVS300 when it moves to [location].

[0137] Next, the motion correction unit 132 uses the depth data and the amount of movement and rotation angle of the EVS 300 to perform a three-dimensional reconstruction of all event data 600 detected within the time frame Ts~Ts+T (step S105). For example, from equation (8), the motion correction unit 132 calculates the time t i ~time t j Perform a 3D reconstruction of all 600 event data points detected up to that point.

[0138] Next, the motion correction unit 132 projects the three-dimensionally reconstructed event data 600 onto the sensor surface of the EVS300 using the internal parameters of the EVS300 (step S106). For example, the motion correction unit 132 estimates the position (x', y') of the event data 600 projected onto the sensor surface of the EVS300 from equations (9) and (10).

[0139] To summarize steps S103 to S106, the motion correction unit 132 performs motion correction based on the acquired event data, angular velocity data, acceleration data, and depth data. For example, the motion correction unit 132 performs motion correction as shown in equation (11).

[0140] Next, the motion correction unit 132 generates a motion-corrected event image 900 by integrating the events projected onto the sensor surface of the EVS 300 (step S107). For example, the motion correction unit 132 generates an event image 900a by integrating the event data 600 projected onto the sensor surface of the EVS 300 according to equation (12).

[0141] Next, the segmentation unit 133 performs segmentation by classifying event data 700 from stationary objects and event data 800 from moving objects using the motion-corrected event image (step S108). For example, from equations (13) and (14), the segmentation unit 133 calculates I k Time t k Segmentation is performed based on an event contrast threshold that is directly proportional to the variance f(θ) of event image 900b, H(x;θ).

[0142] (2. Second Embodiment) The information processing device 100 can also determine whether or not to perform motion correction based on the distance between the depth sensor 500 and the object. Below, an example of the configuration of the information processing system 1A according to the second embodiment will be described with reference to Figure 17.

[0143] Figure 17 is a block diagram showing an example of the configuration of an information processing system according to the second embodiment. The information processing system 1A includes an information processing device 100A instead of an information processing device 100. The information processing device 100A includes a control unit 130A instead of a control unit 130. The control unit 130A includes a motion correction unit 132A instead of a motion correction unit 132.

[0144] (Motion correction unit) The motion correction unit 132A determines whether or not to perform motion correction based on the distance between the depth sensor 500 and the object. Before describing the motion correction by the motion correction unit 132A, an example of motion correction by the motion correction unit 132 of the first embodiment when the EVS 300 is moving in translation will be explained using Figure 18 as a point of comparison. Figure 18 is a diagram illustrating an example of motion correction by the motion correction unit of the first embodiment when the EVS is moving in translation.

[0145] In this case, the motion correction unit 132 performs motion correction regardless of the distance 50 between the EVS 300 and the depth sensor 500, which is located at approximately the same position. For example, the motion correction unit 132 generates a motion-corrected event image 900c by performing motion correction in all cases of the distance 50a between the depth sensor 500 and object 8000a, the distance 50b between object 8000b and object 8000c and the distance 50c between object 8000c and object 8000a.

[0146] Next, using Figure 19, an example of motion correction when the EVS 300 rotates, performed by the motion correction unit 132 of the first embodiment, will be explained. Figure 19 is a diagram illustrating an example of motion correction when the EVS rotates, performed by the motion correction unit of the first embodiment.

[0147] In this case as well, the motion correction unit 132 performs motion correction regardless of the distance 50 between the EVS 300 and the depth sensor 500, which is located at approximately the same position. For example, the motion correction unit 132 generates a motion-corrected event image 900d by performing motion correction in all cases of the distance 50a between the depth sensor 500 and object 8000a, the distance 50b between object 8000b and object 8000c and the distance 50c between object 8000c and object 8000a.

[0148] Next, an example of motion correction by the motion correction unit 132A will be explained using Figure 20. Figure 20 is a diagram illustrating an example of motion correction by the motion correction unit of the second embodiment. The motion correction unit 132A performs motion correction only when the distance between the depth sensor 500 and the object is less than or equal to the threshold 5000X.

[0149] For example, the motion correction unit 132A performs motion correction only when the distance between the depth sensor 500 and the object is less than or equal to the threshold 5000X, and the distance between the depth sensor 500 and the object is distance 50a or distance 50b. The motion correction unit 132A skips motion correction when the distance between the depth sensor 500 and the object is greater than the threshold 5000X, and the distance between the depth sensor 500 and the object is distance 50c.

[0150] The details of motion correction by the motion correction unit 132A will be explained below with reference to Figure 21. Figure 21 is a diagram illustrating the details of motion correction by the motion correction unit in the second embodiment. As the motion correction step S10e, the motion correction unit 132A performs the process in step S13.5 in addition to the processes in steps S11 to S16 and step S18.

[0151] The motion correction unit 132A determines whether the distance between the depth sensor 500 and the object is below a threshold (step S13.5). If the distance between the depth sensor 500 and the object is below a threshold (step S13.5; Yes), the motion correction unit 132A performs the processes in steps S11 to S16 and step S18.

[0152] If the distance between the depth sensor 500 and the object is greater than a threshold (step S13.5; No), the motion correction unit 132A skips steps S11, S12, S14, and S15.

[0153] The motion correction unit 132A can also determine whether or not to perform motion correction based on angular velocity data and acceleration data. Below, another example of motion correction by the motion correction unit 132A will be explained using Figure 22. Figure 22 is a diagram illustrating another example of motion correction by the motion correction unit of the second embodiment. As the motion correction step S10f, the motion correction unit 132A performs the processing of step S13.5f instead of the processing of step S13.5.

[0154] The motion correction unit 132A determines whether the distance between the depth sensor 500 and the object is below a threshold, and whether the angular velocity and acceleration of the EVS300, indicated by the angular velocity data and acceleration data measured by the gyro sensor 400, are above a threshold (step S13.5f). If the distance between the depth sensor 500 and the object is below a threshold, and the angular velocity and acceleration of the EVS300 are above a threshold (step S13.5f; Yes), the motion correction unit 132A performs the processing in steps S11 to S16 and step S18.

[0155] The motion correction unit 132A skips steps S12, S14, and S15 if the distance between the depth sensor 500 and the object is greater than a threshold, or if the angular velocity or acceleration of the EVS 300 is less than a threshold (step S13.5f; No).

[0156] (3. Third Embodiment) The information processing device 100 can estimate its own position based on event data 700 from stationary objects, and can detect moving objects based on event data 800 from moving objects. Below, an example of the configuration of the information processing system 1B according to the third embodiment will be described with reference to Figure 23.

[0157] Figure 23 is a block diagram showing an example of the configuration of an information processing system according to a third embodiment. Information processing system 1B includes an information processing device 100B instead of an information processing device 100. Information processing device 100B includes a control unit 130B instead of a control unit 130. The control unit 130B further includes an estimation unit 134 and a detection unit 135.

[0158] (Estimation Department) The estimation unit 134 estimates its own position based on event data 700 from stationary objects. This will be explained using the case where the information processing device 100B is a delivery robot or an AGV (Automatic Guided Vehicle) as an example. Based on the event data 700 from stationary objects, the estimation unit 134 performs V (Visual)SLAM (Simultaneous Localization and Mapping), which simultaneously estimates the self-position of the delivery robot or the like and generates a 3D (Dimension) map.

[0159] In this way, the estimation unit 134 determines its own position based on event data 700 from stationary objects, from which moving objects such as people and vehicles, which would be noise for VSLAM whose accuracy depends on the tracking accuracy of static landmarks (feature points), have been removed by segmentation. As a result, the estimation unit 134 can generate a 3D map with high accuracy, thereby stabilizing the performance of VSLAM.

[0160] (Detection unit) The detection unit 135 detects moving objects based on event data 800 generated by the moving object. For example, the detection unit 135 detects pedestrians and other vehicles in front of a vehicle such as an autonomous vehicle. This allows the detection unit 135 to avoid collisions between the autonomous vehicle and pedestrians, etc. Another example is the detection unit 135, which recognizes hand gestures in AR (Augmented Reality) and VR (Virtual Reality), and performs 6DoF (Degrees of Freedom) HMD (Head Mount Display) tracking, which is an example of an AR / VR device.

[0161] Generally, if moving objects are detected based on event data 600 that includes event data 700 from stationary objects, the detection accuracy may decrease, or the computational cost of detecting moving objects may increase.

[0162] In contrast, the detection unit 135 detects moving objects based on event data 800 generated by moving objects, from which event data 700 generated by stationary objects has been removed by ego-motion cancellation. As a result, the detection unit 135 can improve the accuracy of moving object detection while also streamlining the calculations for detecting moving objects.

[0163] (4. Effects of the information processing device related to this disclosure) As described above, the information processing device according to this disclosure comprises an acquisition unit (acquisition unit 131 in the embodiment), a motion correction unit (motion correction unit 132 in the embodiment), and a segmentation unit (segmentation unit 133 in the embodiment).

[0164] The acquisition unit acquires event data related to events caused by changes in the brightness of reflected light from an object, and angular velocity data related to the angular velocity of the imaging device. The motion correction unit performs motion correction on the acquired event data based on the acquired event data and angular velocity data, correcting for events caused by the movement of the sensor that detected the event data. The segmentation unit performs segmentation on the motion-corrected event data, classifying it into event data caused by stationary objects and event data caused by moving objects.

[0165] In this way, the information processing device performs motion correction based on angular velocity data in addition to event data, and then classifies the event data into event data from stationary objects and event data from moving objects. As a result, the information processing device does not have to repeatedly perform motion correction and segmentation calculations based solely on event data, and can efficiently classify the event data into event data from stationary objects and event data from moving objects.

[0166] The acquisition unit acquires angular velocity data measured by a gyro sensor (gyro sensor 400 in this embodiment) as angular velocity data.

[0167] As a result, the information processing device can perform motion correction based on angular velocity data measured with high precision by the gyro sensor, thus enabling highly accurate motion correction.

[0168] The acquisition unit acquires event-based angular velocity data calculated from event data as angular velocity data.

[0169] As a result, the information processing device can acquire angular velocity data from event data, allowing it to perform motion correction without having to acquire angular velocity data directly measured by a gyro sensor.

[0170] The acquisition unit acquires multi-viewpoint event data detected from multiple viewpoints as event data, and acquires multi-viewpoint event-based angular velocity data calculated from the feature points of multiple viewpoints detected from the multi-viewpoint event data as angular velocity data.

[0171] As a result, the information processing device can perform motion correction based on event-based angular velocity data from multiple viewpoints, enabling it to perform motion correction with higher accuracy than when motion correction is performed based on event-based angular velocity data calculated from feature points of a single viewpoint.

[0172] The acquisition unit further acquires acceleration data related to the acceleration of the imaging device, and the motion correction unit performs motion correction based on the acquired acceleration data.

[0173] This allows the information processing device to perform motion correction based on acceleration data, thereby reducing the computational cost of motion correction. As a result, the information processing device can, for example, classify event data in real time into event data from stationary objects and event data from moving objects.

[0174] The acquisition unit further acquires depth data regarding the object's depth, and the motion correction unit performs motion correction based on the acquired depth data.

[0175] This allows the information processing device to perform motion correction based on depth data, thereby reducing the computational cost of motion correction. As a result, the information processing device can, for example, classify event data in real time into event data from stationary objects and event data from moving objects.

[0176] The motion correction unit starts motion correction at a different time than when the depth data is acquired.

[0177] This allows the information processing device to compensate for events caused by sensor movement at times when depth data was not acquired, thereby reducing bias in events.

[0178] The motion correction unit performs motion correction on the acquired event data, which has been weighted using a Gaussian function.

[0179] As a result, the information processing device performs motion correction on event data that has been weighted with a Gaussian function such that the weight is greatest at approximately the same time that depth data was acquired. Consequently, the information processing device can utilize depth data acquired at times around the time when depth data was not acquired, and correct for events caused by sensor movement at the time when depth data was not acquired, thereby further reducing event bias.

[0180] The motion correction unit generates an event image in which motion correction has been applied to the acquired event data.

[0181] As a result, the information processing device generates event images corrected for events caused by sensor movement, allowing it to classify event images into those caused by stationary events and those caused by moving objects. Consequently, if the classified event images are used for feature point matching, for example, the information processing device can reduce the error rate of feature point matching. Furthermore, if the classified event images are used for location estimation, the information processing device can improve the accuracy of location estimation.

[0182] The acquisition unit acquires different-viewpoint depth data, measured from a different viewpoint than the event data, as depth data.

[0183] This allows the information processing device to reduce the bias of events compared to when motion correction is performed based on depth data measured from the same viewpoint as the event data, thus enabling highly accurate motion correction.

[0184] The acquisition unit acquires event data detected by the EVS (EVS300 in this embodiment) as event data, and acquires depth data measured by the EVS and a non-coaxial depth sensor (depth sensor 500 in this embodiment) as different-viewpoint depth data.

[0185] As a result, the information processing device can perform motion correction based on depth data measured with high precision by the depth sensor, thereby enabling more accurate motion correction.

[0186] The acquisition unit acquires depth data from the same viewpoint as the event data, as depth data from the same viewpoint.

[0187] This allows the information processing device to reduce computational costs because it can eliminate calculations based on different viewpoints than the event data, unlike when motion correction is performed based on depth data from different viewpoints.

[0188] The acquisition unit acquires the event data detected by the EVS as event data, and acquires the depth data measured by a depth sensor coaxial with the EVS as the same-viewpoint depth data.

[0189] As a result, the information processing apparatus can perform motion correction based on the event data detected with high precision by the EVS and the depth data detected with high precision by the depth sensor, so that the motion correction can be performed with high precision.

[0190] The acquisition unit acquires, as depth data, the different-viewpoint event-based depth data calculated from the different-viewpoint event data detected from a viewpoint different from the event data.

[0191] As a result, since the information processing apparatus can acquire depth data from the event data, it can perform motion correction without acquiring the depth data directly measured by the depth sensor. In addition, since the information processing apparatus can perform motion correction based on the different-viewpoint event-based depth data, it can perform motion correction with higher precision than performing motion correction based on the event-data-based depth data calculated from the event data.

[0192] The acquisition unit acquires the event data detected by the EVS as event data, and acquires the different-viewpoint event-based depth data calculated from the event data detected by an EVS non-coaxial with the EVS as the different-viewpoint event-based depth data.

[0193] As a result, the information processing apparatus can perform motion correction based on the event data detected with high precision by the EVS and the different-viewpoint event-based depth data calculated from the different-viewpoint event data detected with high precision by the EVS non-coaxial with the EVS. Therefore, the information processing apparatus can perform motion correction with higher precision.

[0194] The acquisition unit acquires event-based depth data calculated from event data as depth data.

[0195] This allows the information processing device to acquire depth data from event data, enabling motion correction without acquiring depth data directly measured by a depth sensor.

[0196] The acquisition unit acquires multi-viewpoint event data detected from multiple viewpoints as event data, and acquires multi-viewpoint event-based depth data calculated from the feature points of multiple viewpoints detected from the multi-viewpoint event data as depth data.

[0197] As a result, the information processing device can perform motion correction based on event-based depth data from multiple viewpoints, enabling it to perform motion correction with higher accuracy than when motion correction is performed based on event-based depth data calculated from feature points of a single viewpoint.

[0198] The motion correction unit determines whether or not to perform motion correction based on the distance between the depth sensor, which measures the depth data, and the object.

[0199] This allows the information processing device to skip motion compensation if the distance between the depth sensor and the object is greater than a threshold, thus further streamlining the motion compensation calculation.

[0200] (5. Hardware Configuration) The information processing device 100 etc. related to this disclosure described above is realized by a computer 1000 having a configuration such as that shown in Figure 24. Figure 24 is a hardware configuration diagram showing an example of a computer that realizes the functions of the information processing device. Hereinafter, as an example of the computer 1000, the information processing device 100 according to the embodiment will be described. The computer 1000 has a CPU 1100, RAM 1200, ROM (Read Only Memory) 1300, HDD (Hard Disk Drive) 1400, communication interface 1500, and input / output interface 1600. The various parts of the computer 1000 are connected by a bus 1050.

[0201] The CPU 1100 operates based on programs stored in the ROM 1300 or HDD 1400, and controls various parts. For example, the CPU 1100 loads the programs stored in the ROM 1300 or HDD 1400 into the RAM 1200 and executes processing corresponding to various programs.

[0202] ROM1300 stores boot programs such as the BIOS (Basic Input Output System) executed by CPU1100 when computer 1000 starts up, as well as programs that depend on the computer 1000's hardware.

[0203] HDD1400 is a computer-readable recording medium that non-temporarily records programs executed by CPU1100 and data used by said programs. Specifically, HDD1400 is a recording medium that records an information processing program related to this disclosure, which is an example of program data 1450.

[0204] The communication interface 1500 is an interface for the computer 1000 to connect to an external network 1550 (e.g., the Internet). For example, the CPU 1100 can receive data from other devices or transmit data it generates to other devices via the communication interface 1500.

[0205] The input / output interface 1600 is an interface for connecting the input / output device 1650 and the computer 1000. For example, the CPU 1100 receives data from input devices such as a keyboard or mouse via the input / output interface 1600. The CPU 1100 also transmits data to output devices such as a display, speaker, or printer via the input / output interface 1600. The input / output interface 1600 may also function as a media interface for reading programs recorded on a predetermined recording medium (media). Examples of media include optical recording media such as DVDs (Digital Versatile Discs) and PDs (Phase Change Rewritable Disks), magneto-optical recording media such as MOs (Magneto-Optical Disks), tape media, magnetic recording media, or semiconductor memory.

[0206] For example, when the computer 1000 functions as an information processing device 100 according to the embodiment, the CPU 1100 of the computer 1000 realizes functions such as the control unit 150 by executing an information processing program loaded on the RAM 1200. The HDD 1400 stores the information processing program according to this disclosure and data in the storage unit 120. The CPU 1100 reads and executes the program data 1450 from the HDD 1400, but as another example, these programs may be obtained from other devices via an external network 1550.

[0207] (6. Addendum) This technology can also be configured as follows: (1) An acquisition unit that acquires event data related to events caused by changes in the brightness of reflected light from an object, and angular velocity data related to the angular velocity of the imaging device, Based on the acquired event data and angular velocity data, a motion correction unit performs motion correction to correct an event caused by the movement of the sensor that detected the event data for the acquired event data. A segmentation unit performs segmentation to classify the event data for which the motion correction has been performed into event data by a stationary object and event data by a moving object. An information processing apparatus comprising the above. (2) The acquisition unit acquires, as the angular velocity data, angular velocity data measured by a gyro sensor. The information processing apparatus according to (1) above. (3) The acquisition unit acquires, as the angular velocity data, event-based angular velocity data calculated from the event data. The information processing apparatus according to (1) above. (4) The acquisition unit acquires, as the event data, multi-viewpoint event data detected from a plurality of viewpoints, and acquires, as the angular velocity data, multi-viewpoint event-based angular velocity data calculated from feature points of the plurality of viewpoints detected from the multi-viewpoint event data. The information processing apparatus according to (3) above. (5) The acquisition unit further acquires acceleration data related to the acceleration of the imaging device. The motion correction unit performs the motion correction based further on the acquired acceleration data. The information processing apparatus according to any one of (1) to (4) above. (6) The acquisition unit further acquires depth data related to the depth of the object. The motion correction unit performs the motion correction based further on the acquired depth data. The information processing apparatus according to any one of (1) to (5) above. (7) The motion correction unit starts the motion correction at a timing different from the timing at which the depth data is acquired. The information processing device described in (6) above. (8) The motion correction unit performs the motion correction on the acquired event data, which is weighted by a Gaussian function. The information processing device described in (7) above. (9) The motion correction unit generates an event image in which the motion correction has been applied to the acquired event data. The information processing apparatus described in (7) or (8) above. (10) The acquisition unit acquires, as depth data, different viewpoint depth data measured from a different viewpoint than the event data. The information processing device described in any one of (6) to (9) above. (11) The acquisition unit acquires event data detected by an EVS (Event-based Vision Sensor) as event data, and acquires depth data measured by a non-coaxial depth sensor with the EVS as non-coaxial depth data. The information processing device described in (10) above. (12) The acquisition unit acquires depth data measured from the same viewpoint as the event data as the depth data. The information processing device described in any one of (6) to (9) above. (13) The acquisition unit acquires event data detected by the EVS as event data, and acquires depth data measured by a coaxial depth sensor of the EVS as co-viewpoint depth data. The information processing device described in (12) above. (14) The acquisition unit acquires, as depth data, alternate viewpoint event-based depth data calculated from alternate viewpoint event data detected from a different viewpoint than the event data. The information processing device described in any one of (6) to (9) above. (15) The acquisition unit acquires event data detected by the EVS as event data, and acquires event-based depth data calculated from the event data detected by the EVS and a non-coaxial EVS as multi-viewpoint event-based depth data. The information processing device described in (14) above. (16) The acquisition unit acquires event-based depth data calculated from the event data as the depth data. The information processing device described in any one of (6) to (9) above. (17) The acquisition unit acquires multi-viewpoint event data detected from multiple viewpoints as event data, and acquires multi-viewpoint event-based depth data calculated from the feature points of multiple viewpoints detected from the multi-viewpoint event data as depth data. The information processing device described in (16) above. (18) The motion correction unit determines whether or not to perform motion correction based on the distance between the depth sensor that measured the depth data and the object. The information processing device described in any one of (6) to (9) above. (19) A method of information processing performed by a computer, This involves acquiring event data related to events caused by changes in the brightness of reflected light from an object, and angular velocity data related to the angular velocity of the imaging device. Based on the acquired event data and angular velocity data, motion correction is performed on the acquired event data to correct for events caused by the movement of the sensor that detected the event data. The event data, after motion correction has been applied, is then segmented into event data caused by stationary objects and event data caused by moving objects. Information processing methods, including those mentioned above. (20) Computers, An acquisition unit that acquires event data related to events caused by changes in the brightness of reflected light from an object, and angular velocity data related to the angular velocity of the imaging device, Based on the acquired event data and angular velocity data, a motion correction unit performs motion correction on the acquired event data to correct for events caused by the movement of the sensor that detected the event data. A segmentation unit performs segmentation of the motion-corrected event data into event data caused by stationary objects and event data caused by moving objects. An information processing program that is equipped to function as an information processing device. [Explanation of symbols]

[0208] 1. Information Processing System 100 Information Processing Devices 110 Communications Department 120 Storage section 130 Control Unit 131 Acquisition Department 132 Motion Correction Unit 133 Segmentation Section 200 terminal devices 210 Reception Department 220 Communications Department 230 Storage section 240 Display section 250 Control Unit 300 EVS 400 Gyroscope Sensors 500 Depth Sensor N Network

Claims

1. An acquisition unit that acquires event data related to events caused by changes in the brightness of reflected light from an object, and angular velocity data related to the angular velocity of the imaging device, Based on the acquired event data and angular velocity data, a motion correction unit performs motion correction on the acquired event data to correct for events caused by the movement of the sensor that detected the event data. A segmentation unit performs segmentation of the motion-corrected event data into event data caused by stationary objects and event data caused by moving objects. An information processing device equipped with the following features.

2. The acquisition unit acquires angular velocity data measured by the gyro sensor as the angular velocity data. The information processing apparatus according to claim 1.

3. The acquisition unit acquires event-based angular velocity data calculated from the event data as the angular velocity data. The information processing apparatus according to claim 1.

4. The acquisition unit acquires multi-viewpoint event data detected from multiple viewpoints as event data, and acquires multi-viewpoint event-based angular velocity data calculated from the feature points of multiple viewpoints detected from the multi-viewpoint event data as angular velocity data. The information processing apparatus according to claim 3.

5. The acquisition unit further acquires acceleration data relating to the acceleration of the imaging device, The motion correction unit performs motion correction based on the acquired acceleration data. The information processing apparatus according to claim 1.

6. The acquisition unit further acquires depth data relating to the depth of the object, The motion correction unit performs motion correction based on the acquired depth data. The information processing apparatus according to claim 1.

7. The motion correction unit starts the motion correction at a timing different from the timing at which the depth data is acquired. The information processing apparatus according to claim 6.

8. The motion correction unit performs the motion correction on the acquired event data, which is weighted by a Gaussian function. The information processing apparatus according to claim 7.

9. The motion correction unit generates an event image in which the motion correction has been applied to the acquired event data. The information processing apparatus according to claim 7.

10. The acquisition unit acquires, as depth data, different viewpoint depth data measured from a different viewpoint than the event data. The information processing apparatus according to claim 6.

11. The acquisition unit acquires event data detected by the EVS (Event-based Vision Sensor) as event data, and acquires depth data measured by a depth sensor not coaxial with the EVS as different-viewpoint depth data. The information processing apparatus according to claim 10.

12. The acquisition unit acquires depth data measured from the same viewpoint as the event data as the depth data. The information processing apparatus according to claim 6.

13. The acquisition unit acquires event data detected by the EVS as event data, and acquires depth data measured by a depth sensor coaxial with the EVS as co-viewpoint depth data. The information processing apparatus according to claim 12.

14. The acquisition unit acquires, as depth data, alternate viewpoint event-based depth data calculated from alternate viewpoint event data detected from a different viewpoint than the event data. The information processing apparatus according to claim 6.

15. The acquisition unit acquires event data detected by the EVS as event data, and acquires event-based depth data calculated from the event data detected by the EVS and a non-coaxial EVS as multi-viewpoint event-based depth data. The information processing apparatus according to claim 14.

16. The acquisition unit acquires event-based depth data calculated from the event data as the depth data. The information processing apparatus according to claim 6.

17. The acquisition unit acquires multi-viewpoint event data detected from multiple viewpoints as event data, and acquires multi-viewpoint event-based depth data calculated from the feature points of multiple viewpoints detected from the multi-viewpoint event data as depth data. The information processing apparatus according to claim 16.

18. The motion correction unit determines whether or not to perform motion correction based on the distance between the depth sensor that measured the depth data and the object. The information processing apparatus according to claim 6.

19. A method of information processing performed by a computer, This involves acquiring event data related to events caused by changes in the brightness of reflected light from an object, and angular velocity data related to the angular velocity of the imaging device. Based on the acquired event data and angular velocity data, motion correction is performed on the acquired event data to correct for events caused by the movement of the sensor that detected the event data. The event data, after motion correction has been applied, is then segmented into event data caused by stationary objects and event data caused by moving objects. Information processing methods, including those mentioned above.

20. Computers, An acquisition unit that acquires event data related to events caused by changes in the brightness of reflected light from an object, and angular velocity data related to the angular velocity of the imaging device, Based on the acquired event data and angular velocity data, a motion correction unit performs motion correction on the acquired event data to correct for events caused by the movement of the sensor that detected the event data. A segmentation unit performs segmentation of the motion-corrected event data into event data caused by stationary objects and event data caused by moving objects. An information processing program that is equipped to function as an information processing device.

Images