Optical flow estimation device, optical flow estimation method, and program

Abstract

To provide a technology that efficiently estimates an optical flow using event data.SOLUTION: An optical flow estimation device 1 comprises: an event data acquisition unit 10; a storage unit 12 in which is stored the energy function of an optical flow in which each pixel is defined in a graph structure, the energy function being composed, as cost, of the consistency of time stamps before and after a position change based on the optical flow and the smoothness of the optical flow of nearest neighbor pixels in the graph structure; and a computation unit 11 which includes a plurality of processors that are driven in response to data input. Each of the processors of the computation unit 11 reads an energy function from the storage unit 12, specifies a pixel at which an event has occurred within a prescribed time when event data is inputted, as an active pixel, and performs belief propagation between the active pixels by message passing, then calculating an optical flow that minimizes the energy function.SELECTED DRAWING: Figure 3

Description

Technical Field

[0001] The present invention relates to a technique for estimating optical flow based on event data generated asynchronously.

Background Art

[0002] Optical flow is an important technique for estimating the motion of objects in video recognition. In recent years, a method of estimating optical flow using a large-scale neural network on a GPU has become mainstream. However, in a video, the same image is continuously acquired even if there is no change in the object, resulting in redundant data. In addition, the burden of data transfer and storage is large, and the recognition process also becomes redundant. For more efficient and low-power consumption computing, it is conceivable to distribute storage and operations and achieve less data transfer and parallel operations. For this purpose, innovation is required in both sensors and calculators.

[0003] On the sensor side, a visual sensor called an event camera has emerged in place of the conventional frame-based camera (Non-Patent Document 1). An event camera is a sensor that records only asynchronous luminance changes for each pixel. It has the excellent feature of high temporal resolution in microseconds. The event data output by the event camera has no data for pixels with no luminance change and is spatially sparse data.

[0004] On the calculator side, a graph processor having a large number of cores with local memories and all of which are fully connected has emerged. Such a structure eliminates the need for off-chip memory access and realizes high-speed super parallel operations. It has excellent performance in the probability propagation algorithm in a sparse graph and is expected to be used in deep learning accelerators and Spatial AI.

Prior Art Documents

Non-Patent Documents

[0005]

Non-Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0006] In view of the above background, an object of the present invention is to provide a technique for efficiently estimating optical flow by processing event data having asynchrony and sparsity by an arithmetic unit capable of parallel processing such as a graph processor.

Means for Solving the Problems

[0007] The optical flow estimation device of the present invention includes an event data acquisition unit that acquires event data including the position and time of pixels where a predetermined luminance change has occurred, an optical flow energy function defined in a graph structure in which the nearest pixels are connected by links, the storage unit stores an energy function that uses the consistency of timestamps before and after the position change based on the optical flow and the smoothness of the optical flow of the nearest neighbor pixels in the graph structure as costs, an arithmetic unit having a plurality of processors that are driven in response to the input of data, and an output unit that outputs the calculation result by the arithmetic unit. Each processor of the arithmetic unit reads the energy function from the storage unit, and when the event data acquired by the event data acquisition unit is input, identifies pixels where events have occurred within a predetermined time as active pixels, performs probabilistic propagation among the active pixels, and calculates an optical flow that minimizes the energy function.

[0008] With this configuration, probability propagation can be performed only on the active pixels around the pixels with a predetermined luminance change, and the optical flow can be updated. Since the arithmetic unit includes a processor that is driven in response to the input of data, the processor that does not receive event data is not driven. With the above configuration combined, an efficient optical flow estimation can be realized in which the processor is driven only when there is a change from sensing to optimization.

[0009] In the optical flow estimation device of the present invention, the predetermined time may be set based on a time Δt such that the position change of the optical flow used in the energy function is within 1 pixel. For example, the predetermined time is a constant multiple of Δt. Here, if the constant is too small, the range of probability propagation becomes too small and the optical flow cannot be updated appropriately. If the constant is too large, the range of probability propagation becomes wide and the computational processing load increases. The constant multiplied by Δt is, for example, preferably 4 to 16, and more preferably 8 to 12. By appropriately setting the active pixels in this way, calculations can be performed using sparsity.

[0010] The optical flow estimation device of the present invention includes an event data generation unit that acquires a video captured by a camera, generates timing information of luminance change based on the luminance difference between frames of the acquired video, and generates event data, and may pass the event data generated by the event data generation unit to the event data acquisition unit. By generating event data from camera video in this way, efficient optical flow estimation can be performed using the event data.

[0011] In the optical flow estimation device of the present invention, the processor performs probability propagation by exchanging messages between adjacent active pixels, and the message exchange may be performed when the difference between the message to be transmitted and the previously transmitted message is greater than a predetermined threshold. With this configuration, by limiting the messages exchanged between adjacent pixels to messages that have a large influence on probability propagation, it is possible to suppress the amount of calculation while maintaining the estimation accuracy.

[0012] The optical flow estimation method of the present invention is a method for estimating an optical flow based on event data including the position and time of pixels having a predetermined luminance change by an optical flow estimation device including a plurality of processors driven in response to input of data, wherein the optical flow estimation device reads, from a storage unit, an optical flow energy function defined in a graph structure in which nearest neighbor pixels are connected by links, the optical flow energy function having, as costs, the consistency of time stamps before and after a position change based on the optical flow and the smoothness of the optical flow of nearest neighbor pixels in the graph structure; the optical flow estimation device acquires event data; the processor, when event data is input, identifies pixels in which an event has occurred within a predetermined time as active pixels; the processor performs probability propagation between the active pixels to calculate an optical flow that minimizes the energy function; and the optical flow estimation device outputs data of the calculated optical flow as an operation result.

[0013] The program of the present invention is for a computer having a plurality of processors driven in response to input of data to estimate an optical flow based on event data including the positions and times of pixels having a predetermined luminance change. From a storage unit, an energy function of the optical flow defined in a graph structure in which the nearest pixels are connected by links, the energy function having, as costs, the consistency of time stamps before and after a position change based on the optical flow and the smoothness of the optical flow of the nearest neighbor pixels in the graph structure, a step of reading out the energy function; a step of acquiring event data; a step of, when event data is input, specifying pixels in which an event has occurred within a predetermined time as active pixels; a step of performing probabilistic propagation among the active pixels to calculate an optical flow that minimizes the energy function; and a step of outputting data of the calculated optical flow as an operation result.

Advantages of the Invention

[0014] The present invention can realize an efficient optical flow estimation in which a processor is driven only when there is a change from sensing to optimization.

Brief Description of the Drawings

[0015]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Embodiments for Carrying Out the Invention

[0016] Hereinafter, an optical flow estimation apparatus according to an embodiment of the present invention will be described. First, an overview of the processing performed by the optical flow estimation apparatus will be described.

[0017] [Definition of Markov Random Field] As shown in FIG. 1, a Markov random field is defined by a graph in which each pixel is a node and the nearest neighboring pixels are connected by links. Let the total number of pixels be N, and numbers i = 1, 2, ···, N be assigned to each node in raster scan order, and the set thereof be P. Also, let the undirected link between nodes i and j be labeled (i, j), and the set of all 4-neighborhood links be N. Let the optical flow to be estimated at pixel i be v i Then, the total cost is expressed by an energy function shown in the following equation (1). [Equation]

[0018] Here, V represents a smoothing term representing the smoothness of the optical flow (OF) between adjacent nodes, and D represents a data term representing the fit of the optical flow to the data. By minimizing the energy in equation (1), the optical flow is estimated. In FIG. 1, some pixels are shaded, which will be described later.

[0019] [Definition of Event OF Model and Cost Function] The coordinates, timestamp, and polarity (direction of light and dark change) of the pixels of the event are denoted as x, y, t, p ∈ {+, -}, respectively. The coordinates are data for specifying the pixel where the event occurred, and the timestamp is data indicating the time when the event occurred. As a representation of event data, SAE (Surface of active events) is defined by the latest timestamp of each pixel for each polarity. [Number]

[0020] SAE is a concept that represents the spatio-temporal representation of events and stores the latest event trigger time (time dimension) of pixels (spatial dimension). Note that SAE exists for each of the polarities {+, -}.

[0021] Figure 2 is a diagram for explaining the consistency of event timestamps. As shown in Figure 2, due to the movement of the object, when a new event e k =(x k ,t k ,p k ) enters the i-th node, if the movement vector of that pixel for Δt seconds is v i , the following equation is derived from the timestamp consistency. Note that Δt is determined so that the movement of the object is within 1 pixel. Thereby, the calculation of the optical flow can be performed only by local probability propagation. [Number] From this, the data term is defined as follows. [Number]

[0022] Whether the current event matches which event Δt seconds ago is referred to from the SAE of the same polarity, and the matching cost is calculated. The data term defines the energy such that the optical flow with matching timestamps is taken as the correct answer. This cost can be calculated only by local reference by adjusting Δt to be small.

[0023] Also, the smoothing term is defined as follows for two adjacent nodes i and j. [Number] This indicates that the smaller the difference in optical flow between two adjacent nodes, the smaller the energy.

[0024] [Optimization by Loopy Probability Propagation Method] To perform global optimization of the energy function shown in Equation (1) using local operations, the loopy probability propagation method is used. The loopy probability propagation method is a technique for approximately estimating the MAP solution by repeatedly exchanging messages between adjacent nodes in an active node where the most recent event has occurred. This method is a technique for obtaining the joint distribution of a graph and repeatedly performs the joint distribution and marginalization of adjacent nodes. A "message" is a concept representing local calculations, and the product of messages from connected nodes represents the marginal distribution. A detailed explanation of the loopy probability propagation method is described, for example, in J. Ortiz et al., "Bundle Adjustment on a Graph Processor," IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2416-2425, 2020.

[0025] In this embodiment, the optical parameter v to be estimated i is defined as a discrete variable, and the message m is represented by a matrix of the dimension of the number of its labels. The message m ij sent from node i to node j is as follows.

Equation

[0026] Here, k ∈ N(i)\j indicates the connected nodes of i other than j. After repeatedly exchanging messages by the message exchange algorithm shown later, the belief is as follows.

Equation

[0027] Finally, the estimated optical flow for each pixel j is determined at each node to minimize b(v j ) as follows. [Number]

[0028] In FIG. 1, the pixels shown in black indicate the pixels where events have occurred, and the pixels with diagonal hatching are the active pixels where events have recently occurred. Specifically, they are pixels having time stamps within a predetermined time from the time stamp of the latest event. In the present embodiment, the predetermined time for defining the active pixels is set based on Δt described above, and specifically, it is Δt × 10. The optical flow estimation device according to the present embodiment performs probability propagation by exchanging messages among the active pixels hatched with diagonal lines. When a new event enters, the matching cost is calculated with reference to the SAE and set to the pixel on the graph structure corresponding to that pixel. Then, message exchange is started from the surrounding pixels, and the optical flow is estimated by repeating until there is no update of the message only in the active pixels.

[0029] [Regarding Message Transmission] In the present embodiment, a method of transmitting when the difference from the previous message is large is applied to the event-based algorithm. The nodes where recent events have occurred are set as active nodes, and among them, transmission is performed when the difference from the previous message is larger than a predetermined threshold. These are performed asynchronously each time an event enters. As a result, it becomes an algorithm driven by a large difference from event data to message exchange, and it becomes an efficient algorithm suitable for a graph processor.

[0030] (First Embodiment) FIG. 3 is a diagram showing the configuration of the optical flow estimation device 1 according to the first embodiment. The optical flow estimation device 1 according to the first embodiment includes an event data acquisition unit 10 that acquires event data from the event camera 20, a calculation unit 11 that calculates an optical flow based on the acquired event data, a storage unit 12 that stores various types of data, and an output unit 13 that outputs the optical flow data estimated by the calculation unit 11. The storage unit 12 stores data of an energy function used for the estimation of the optical flow and data of the SAE based on the event data.

[0031] FIG. 4 is a diagram showing the configuration of the calculation unit 11. The calculation unit 11 includes a plurality of processors 30, and each processor 30 is fully connected. Each processor 30 has a local memory 31, eliminates the need for off-chip memory access, and realizes high-speed super parallel calculation.

[0032] FIG. 5 is a diagram showing the operation of the optical flow estimation device 1 according to the first embodiment. Each processor 30 of the optical flow estimation device 1 reads the data of the energy function from the storage unit 12 and expands it in the memory 31 (S10). When event data is acquired from the event camera 20 (S11), the processor 30 into which the event data is input identifies the data of the active pixels based on the SAE data stored in the storage unit 12, and performs probability propagation processing among the active pixels (S12). As described above, the probability propagation processing is performed by message passing. Specifically, a message is sent to an adjacent pixel when the difference from the previous message is large.

[0033] The optical flow estimation device 1 determines whether the probability propagation process has converged (S113). When the difference from the previous message becomes smaller than a predetermined threshold value, it is determined that the probability propagation process has converged. When it is determined that the probability propagation process has converged (YES in S13), the optical flow estimation device 1 stores the estimated optical flow in the storage unit 12. Then, the optical flow estimation device 1 outputs the obtained optical flow data (S14).

[0034] As described above, the configuration of the optical flow estimation device 1 according to the present embodiment has been described. However, a program having a module that realizes the processing performed by the arithmetic unit 11 described above is stored in the RAM or ROM, and the optical flow estimation device 1 described above is realized by the CPU executing the program. Such a program is also included in the scope of the present invention.

[0035] The optical flow estimation device 1 according to the present embodiment can update the optical flow by performing probability propagation by message passing only around the pixels having a predetermined luminance change. Due to the synergistic effect of such a configuration enabling distributed processing and the arithmetic unit 11 capable of high-speed super-parallel arithmetic, an efficient optical flow estimation in which the processor 30 is driven only when there is a change from sensing to optimization can be realized.

[0036] (Second Embodiment) FIG. 6 is a diagram showing the configuration of the optical flow estimation device 2 according to the second embodiment. The basic configuration of the optical flow estimation device 2 according to the second embodiment is the same as that of the optical flow estimation device 1 according to the first embodiment, but the optical flow estimation device 2 according to the second embodiment is different in that it estimates the optical flow based on the video of the frame-based camera 21.

[0037] In the second embodiment, an event data generation unit 14 that generates event data from the video acquired by the camera 21 is provided. The processing after the event data generation unit 14 generates an event is the same as that of the optical flow estimation device 1 in the first embodiment.

[0038] FIG. 7 is a diagram for explaining the generation process of event data. The video data captured by the frame-based camera 21 consists of frames at regular time intervals (upper left in FIG. 7). Focusing on a specific pixel, it can be seen that the pixel has data of luminance values at regular time intervals (upper right in FIG. 7).

[0039] The event data generation unit 14 detects timing information at which a luminance change equal to or greater than a predetermined threshold occurs from the data of luminance values at regular time intervals (lower right in FIG. 7). By performing this process for all the pixels in the video of the camera 21, the timing information of the pixels where a luminance change equal to or greater than the predetermined threshold occurs can be generated as event data (lower left in FIG. 7).

[0040] The optical flow estimation device 2 according to the second embodiment has been described above. The optical flow estimation device 2 according to the second embodiment can realize efficient optical flow estimation in the same manner as in the first embodiment by converting the video of the frame-based camera 21 into event data.

Explanation of Reference Numerals

[0041] 1, 2 Optical flow estimation device 10 Event data acquisition unit 11 Arithmetic unit 12 Storage unit 13 Output unit 14 Event data generation unit 20 Event camera 21 Camera 30 Processor 31 Memory

Claims

1. An event data acquisition unit that acquires event data including the position and time of pixels with a predetermined luminance change; A storage unit that stores an energy function of optical flow defined in a graph structure in which the nearest neighboring pixels are connected by links, the energy function taking as costs the consistency of timestamps before and after a position change of the optical flow and the smoothness of the optical flow of the nearest neighboring pixels in the graph structure; An arithmetic unit having a plurality of processors that are driven in response to input of data; An output unit that outputs the arithmetic result by the arithmetic unit; Comprising: Each processor of the arithmetic unit: Reads out the energy function from the storage unit; When the event data acquired by the event data acquisition unit is input, identifies pixels in which an event has occurred within a predetermined time as active pixels; Performs probabilistic propagation among the active pixels to calculate an optical flow that minimizes the energy function; An optical flow estimation device.

2. The optical flow estimation device according to claim 1, wherein the predetermined time is set based on a time Δt such that a position change of the optical flow used in the energy function falls within one pixel.

3. An event data generation unit that acquires an image captured by a camera, generates timing information of luminance change based on the difference in luminance between frames of the acquired image, and generates event data; Comprising: The optical flow estimation device according to claim 1 or 2, wherein the event data generated by the event data generation unit is passed to the event data acquisition unit.

4. The optical flow estimation device according to any one of claims 1 to 3, wherein the processor performs probabilistic propagation by exchanging messages between adjacent active pixels, and the message exchange is performed when the difference between the message to be transmitted and the message transmitted previously is greater than a predetermined threshold.

5. A method for estimating an optical flow based on event data including the position and time of pixels with a predetermined luminance change by an optical flow estimation device having a plurality of processors that are driven in response to input of data, The optical flow estimation device reads, from a storage unit, an optical flow energy function defined in a graph structure in which adjacent pixels are connected by links, the energy function taking as costs the consistency of timestamps before and after a position change based on the optical flow and the smoothness of the optical flows of neighboring pixels in the graph structure; The optical flow estimation device acquires event data; When event data is input, the processor identifies pixels in which events have occurred within a predetermined time as active pixels; The processor performs probabilistic propagation among the active pixels to calculate an optical flow that minimizes the energy function; The optical flow estimation device outputs data of the calculated optical flow as a calculation result; An optical flow estimation method comprising the steps. Claim 6 In a computer provided with a plurality of processors that are driven in response to input of data, for estimating an optical flow based on event data including the positions and times of pixels at which a predetermined luminance change has occurred, reading, from a storage unit, an optical flow energy function defined in a graph structure in which adjacent pixels are connected by links, the energy function taking as costs the consistency of timestamps before and after a position change based on the optical flow and the smoothness of the optical flows of neighboring pixels in the graph structure; acquiring event data; when event data is input, identifying pixels in which events have occurred within a predetermined time as active pixels; performing probabilistic propagation among the active pixels to calculate an optical flow that minimizes the energy function; outputting data of the calculated optical flow as a calculation result; A program for causing the steps to be executed.

Images