Tracking device, tracking method, and program

Abstract

To provide a tracking device capable of performing efficient calculation by utilizing sparse properties of event data.SOLUTION: A tracking device 1 includes: a storage part 13 for storing a multilayer perceptron in which a relation between event data and a relative position and a relative speed of a tracking object is learned in advance; and an arithmetic part 11 for calculating the relative position and the relative speed of the tracking object from inputted event data. The arithmetic part 11 repeats processing for inputting event data acquired from an event camera and data on tentative relative positions and relative speeds of a tracking object to the multilayer perceptron, and obtaining deviation between the tentative relative positions and relative speeds relative to event data, and processing for correcting data on relative positions and relative speeds of a tracking object in a direction for reducing the deviation, and obtains relative positions and relative speeds when a predetermined completion condition is satisfied as relative positions and speeds of a tracking object.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] The present invention relates to an apparatus for tracking based on event data. [Background technology]

[0002] Motion tracking from spatiotemporal data such as video frames is a fundamental function in robotics applications. Motion tracking can be performed by tracking the position of a specific target from a camera, or by tracking the position of a camera from world coordinates. Applications of motion tracking are diverse, ranging from manipulating a robot arm to localizing a mobile robot.

[0003] Event cameras are a new bio-inspired vision sensor that mimics the biological retina and outputs pixel-by-pixel intensity changes in the form of an asynchronous event stream rather than intensity frames. Due to this unique operating principle, event-based sensing offers significant advantages over conventional cameras, including high dynamic range (HDR), high temporal resolution, and blur-free sensing. As a result, it has the potential to achieve robust tracking even under intense motion and harsh lighting conditions.

[0004] Taking advantage of these characteristics, a method has been proposed to achieve tracking in high-speed, high-dynamic-range environments (Non-Patent Document 1). In Non-Patent Document 1, the Lucas-Kanade tracker (KLT), which was developed for frame-based video data, is extended to sparse intensity difference data from event-based cameras. The method described in Non-Patent Document 1 warps a given photometric 3D map or frame, and updates the position and velocity of the tracked object by taking the difference between the observed and predicted values. [Prior art documents] [Non-patent literature]

[0005] [Non-Patent Document 1] S. Bryner et al. “Event-based,direct camera tracking from a photometric 3d map using nonlinear optimization” ICRA, 2019, pp. 325-331 Summary of the Invention [Problem to be solved by the invention]

[0006] In conventional methods such as the one described in Non-Patent Document 1, sparse events are first converted into dense frames, which are then processed using a dense neural network such as CNN. These conventional methods require dense calculations to calculate the difference, and are unable to fully utilize the sparsity of the event data.

[0007] In view of the above background, an object of the present invention is to provide a tracking device that utilizes the sparsity of event data and is capable of performing efficient calculations. [Means for solving the problem]

[0008] The tracking device of the present invention is a device that tracks a tracking object using event data acquired from an event camera, and includes: a memory unit that stores a multilayer perceptron that has previously learned the relationship between the event data and the relative position and relative velocity of the tracking object, using event data corresponding to the relative position and relative velocity of the tracking object as training data; and a calculation unit that calculates the relative position and relative velocity of the tracking object from the input event data. The calculation unit inputs the event data acquired from the event camera and provisional relative position and relative velocity data of the tracking object to the multilayer perceptron and repeatedly performs the following processes: determining the deviation of the provisional relative position and relative velocity from the event data; correcting the relative position and relative velocity data of the tracking object in a direction that reduces the deviation; and inputting the corrected relative position and relative velocity data of the tracking object to the multilayer perceptron as provisional relative position and relative velocity data of the tracking object to determine the deviation from the event data, and determines the relative position and relative velocity of the tracking object when a predetermined termination condition is met.

[0009] In this way, the relative position and relative velocity of the tracked object are estimated directly from the event data, so there is no need to calculate the current state, and computational efficiency can be improved by taking advantage of sparsity.

[0010] In the tracking device of the present invention, the tracking target may be a plane having a predetermined pattern or a three-dimensional shape.

[0011] In the tracking device of the present invention, the multilayer perceptron may be trained using the relative position and relative velocity of the object to be tracked and corresponding event data that has been blurred as training data.

[0012] A tracking method of the present invention is a method for tracking a tracking object using event data acquired from an event camera, and includes the steps of: preparing a multilayer perceptron that has previously learned the relationship between the event data and the relative position and relative velocity of the tracking object, using event data corresponding to the relative position and relative velocity of the tracking object as training data; inputting the event data acquired from the event camera and provisional relative position and relative velocity data of the tracking object into the multilayer perceptron and determining a deviation of the provisional relative position and relative velocity from the event data; correcting the relative position and relative velocity data of the tracking object in a direction that reduces the deviation, inputting the corrected relative position and relative velocity data of the tracking object into the multilayer perceptron as provisional relative position and relative velocity data of the tracking object, and repeatedly performing a process of determining a deviation of the provisional relative position and relative velocity from the event data; and determining the relative position and relative velocity when a predetermined termination condition is satisfied as the relative position and relative velocity of the tracking object.

[0013] The method for generating a multilayer perceptron of the present invention is a method for generating a multilayer perceptron that represents the relationship between the relative position and relative velocity of a tracked object and event data, and includes the steps of acquiring the relative position and event data of the tracked object, applying blurring to the event data, and training the multilayer perceptron to minimize the error between the event data generated by applying the relative position and relative position data of the tracked object to the multilayer perceptron and the blurred event data. Here, "applying blurring to the event data" means assigning a value corresponding to the distance to the position where an event has occurred, thereby making the position where an event has occurred continuous with a position where no event has occurred.

[0014] In the method for generating a multilayer perceptron of the present invention, when the tracking target is known, the step of acquiring the event data may generate the event data by simulating the tracking target moving randomly.

[0015] A program of the present invention is a program for tracking a tracking object using event data acquired from an event camera, and causes a computer to execute the following steps: preparing a multilayer perceptron that has previously learned the relationship between the event data and the relative position and relative velocity of the tracking object, using event data corresponding to the relative position and relative velocity of the tracking object as training data; inputting the event data acquired from the event camera and provisional relative position and relative velocity data of the tracking object into the multilayer perceptron and determining a deviation of the provisional relative position and relative velocity from the event data; correcting the relative position and relative velocity data of the tracking object in a direction that reduces the deviation, inputting the corrected relative position and relative velocity data of the tracking object into the multilayer perceptron as provisional relative position and relative velocity data of the tracking object, and repeatedly performing a process of determining a deviation of the provisional relative position and relative velocity from the event data; and determining the relative position and relative velocity of the tracking object when a predetermined termination condition is satisfied. [Effects of the Invention]

[0016] According to the present invention, tracking can be performed computationally efficiently by taking advantage of the sparsity of event data. [Brief explanation of the drawings]

[0017] [Figure 1] 1 is a diagram illustrating the principle of processing by a tracking device according to an embodiment of the present invention. [Figure 2] FIG. 1 illustrates inputs and outputs to an IEG. [Figure 3]FIG. 10 is a diagram illustrating a unit of a tracking process target. [Figure 4] FIG. 1 illustrates a method for generating artificial data. [Figure 5] 1 is a diagram illustrating a configuration of a tracking device according to an embodiment. [Figure 6] 10 is a flowchart showing the operation of the tracking device. DETAILED DESCRIPTION OF THE INVENTION

[0018] The tracking device of this embodiment will be described below. The tracking device has the function of tracking the relative position and relative speed of a tracking target. When the camera is fixed, it becomes an application that tracks the tracking target, and when the tracking target is fixed, it becomes an application that estimates the attitude of the camera. In either case, the position and movement speed of the tracking target as seen from the camera are tracked, so the terms relative position and relative speed are used. For simplicity, the terms "position" and "speed" will sometimes be used below.

[0019] [overview] 1 is a diagram showing the processing principle of the tracking device 1 of this embodiment. The tracking device 1 of this embodiment has a trained multilayer perceptron. The multilayer perceptron uses the position and velocity of the tracked object and the corresponding event data as training data and has previously learned the relationship between the event data and the position and velocity of the tracked object.

[0020] The inputs of a multilayer perceptron are event data (the coordinates and time when the event occurred) and data on the position and velocity of the tracked object. The output of the multilayer perceptron is data that represents the deviation between the event data and the position and velocity of the tracked object. Because this multilayer perceptron does not explicitly have the three-dimensional data of the tracked object indicated by the event data, it is referred to in this specification as an implicit event generator (IEG).

[0021] 2 is a diagram showing input and output to the IEG. In the tracking device 1 of this embodiment, observed event data x (three dimensions) and data on the provisional position T and velocity T' of the tracking target (six dimensions each) are input to the IEG, and inference is performed. When the input provisional position T and velocity T' of the tracking target are close to the actual position and velocity of the target appearing in the event data (i.e., close to the correct answer), the deviation data output from the IEG becomes small.

[0022] The position and velocity of the tracked object at the previous time are used as the initial values ​​for the provisional position T and velocity T' of the tracked object. When tracking starts, there is no data from the previous time, so in this case the position and velocity of the tracked object detected by a full search are used.

[0023] The tracking device 1 corrects the data of the position T and velocity T' of the tracked object so as to reduce the deviation data, and repeats the process of determining the deviation data. When a predetermined convergence condition is satisfied, the position and velocity at that time are determined as the position T and velocity T' of the tracked object. The predetermined convergence condition is that the change in loss is smaller than the optimization threshold ε.

[0024] The IEG has a weight Θ set by prior learning. In this embodiment, since tracking is performed using the IEG, the position T and velocity T' of the tracked object are optimized during inference so as to minimize the difference L between the observation and estimation of the event stream {e} as follows:

number

[0025] In this embodiment, unlike conventional methods, it is not necessary to explicitly generate events using the estimated values ​​(T, T') of the current position and velocity, but the difference between the current value and the estimated value is directly calculated implicitly from sparse events.

[0026] 3 is a diagram for explaining the unit of the tracking process. During tracking, event data of an existing small group (window) consisting of M event data is processed as described above, and at the j-th repetition, the event data is j and speed T j Then, in the j+1th iteration, the window is slid by a sliding width K and optimized again to obtain the position T j+1 and speed T j+1 Since the position and velocity of the tracked object do not change much between the jth and j+1th iterations, (T j ,T j By updating and calculating the time t', it is possible to reduce latency compared to the case of a frame camera. In this embodiment, acceleration is not taken into account, but if the window size M is sufficiently small, this is not a problem because the movement within the window can be approximated as uniform motion.

[0027] [Tracking using IEG gradients] First, the symbols used below will be explained. Weight Θ: IEG parameter fixed by learning x i =(x i ,y i ,t i ): the location of the i-th detected intensity change (x i ,y i ) and time t i δ(x i ):x i Intensity changes detected in M: The number of event data included in the window to be processed

number

[0028] During tracking, the loss L shown in the following equation (2) update Optimization is performed so that the absolute value output by IEG becomes small, that is, so that the absolute value output by IEG becomes close to 1. This optimization is repeated until the change in loss becomes smaller than the optimization threshold ε.

number

[0029] [IEG Learning] Next, we will explain the outline of IEG learning. During learning, the IEG is trained using the event intensity change δ as the output. That is, an IEG is prepared that takes the two-dimensional position x = (x, y), time t, position T, and time change T' of the tracking target as input, and outputs the detected intensity change δ∈[+ / - / NaN].

[0030] The training data for learning the IEG will now be described. The target tracked by the tracking device 1 of this embodiment is known. Therefore, event data that occurs when the target moves can be artificially generated, and the artificially generated data is used as training data.

[0031] FIG. 4 is a diagram showing a method for generating artificial data. In FIG. 4, A indicates the tracking target. The tracking target is, for example, a specific pattern as shown in FIG. 4. Here, an example is given in which two-dimensional data with a pattern is used, but the tracking target may also have a three-dimensional shape. When data on the edge of the tracking target is obtained, sparse gradient data as shown in B is obtained.

[0032] Next, for the gradient data of B, if we randomly give time t, movement speed v, and rotation angle w, then each position x iIntensity change δ t C in Figure 4 shows an example where a moving speed is applied in the horizontal direction. On an edge extending horizontally, the intensity of adjacent pixels along the moving direction is the same, so no change in intensity occurs, but when an edge extending vertically moves horizontally, an intensity change occurs.

[0033] Specifically, δ t is calculated as follows: To generate intensity change data at various speeds, a randomly determined speed T' = (v x ,v y ,ω) and the coordinates from the edge of the known object (x e ,y e ), and the differential data of the intensity in that coordinate is ∂I / ∂x=(∂ ix ,∂ iy ) is used. From these values, the edge coordinates (x e ,y e ) t ,y t ) theoretical intensity change value δ t can be calculated as follows:

number

[0034] Next, the intensity change δ t Among them, the threshold value of the event occurrence δ s The intensity change exceeding δ is expressed as a binary event change δ as shown in D. t Let ′=(+1,-1,Nan).

number

[0035] Finally, the binary event change δ t ' is blurred with Gaussian blur to generate artificial data as shown in E. The purpose of Gaussian blurring is to give the data a gradient, making it differentiable. tDepending on the distance (dx, dy) to the point ′, an extended Gaussian function G(dx, dy) with mean 0 and variance σ is used.

number

[0036] To avoid interference with events occurring on other edges, the blur width is set to w, and (x t ′,y t ′)=(x t +dx,y t +dy) Gaussian blur strength change δ a is calculated as follows:

number

[0037] In IEG training, we use a 6-dimensional vector (x, y, t, v x ,v y ,ω) and estimate the intensity change δ g The generated intensity change δ g and the artificially calculated intensity change δ a Error L train The IEG is trained by backpropagating the

number

[0038] 5 is a diagram showing the configuration of a tracking device 1 according to an embodiment. The tracking device 1 includes an event data acquisition unit 10 that acquires event data from an event camera, a calculation unit 11 that tracks the tracking target based on the event data, an output unit 12 that outputs the tracking results, and a storage unit 13 that stores various data. The storage unit 13 stores IEGs used for tracking.

[0039] 6 is a flowchart showing the operation of the tracking device 1 according to the embodiment. The tracking device 1 acquires an event stream transmitted from an event camera (S10). The tracking device 1 acquires event data of one window to be subjected to tracking processing from the event stream (S11). As described above, one window consists of M items of event data.

[0040] The tracking device 1 inputs one window of event data and the provisional position and velocity of the tracking target to the IEG (S12), and calculates the deviation of the provisional position and velocity from the event data (S13). The tracking device 1 determines whether or not a termination condition for the position and velocity calculation process is met (S14), and if it determines that the termination condition is not met (NO in S14), corrects the position and velocity of the tracking target based on the deviation (S15), inputs the corrected position and velocity to the IEG (S12), and repeats the process of calculating the deviation (S13).

[0041] If it is determined that the termination condition is met (YES in S14), the tracking device 1 stores the position and velocity of the tracked object corresponding to the event data of the window in the memory unit 13 (S16), and determines whether there is any further event data to be processed (S17).

[0042] If it is determined that there is further event data to be processed (YES in S17), the window is slid (S18), the event data to be processed is acquired (S11), and processing is performed to obtain the position and velocity of the tracking object corresponding to the window to be processed (S12 to S16).If it is determined that there is no event data (NO in S17), the tracking device 1 outputs the tracking result (S19).

[0043] The above has described in detail the tracking device 1 according to the embodiment. The tracking device 1 according to the embodiment inputs event data to the IEG, estimates deviations between the tentative position and velocity of the tracked object, and corrects the deviations to determine the position and velocity of the tracked object, thereby enabling efficient tracking by utilizing sparsity.

[0044] In the above embodiment, an example was given in which the position and time at which an event occurred are used as event data, but the event data may also use data on polarity (the direction of change in the event) in addition to the position and time.

[0045] In the above embodiment, Gaussian blurring is used as one of the blurring processes when generating the training data, but the blurring process performed on the event data is not limited to Gaussian blurring. Any method can be used as long as it does not set the value to 0 even when the distance from the edge of the event data is small, but rather sets a value according to the distance. [Explanation of symbols]

[0046] 1. Tracking Device 10 Event data acquisition section 11 Arithmetic section 12 Output section 13 Storage section

Claims

1. An apparatus for tracking a tracking target using event data acquired from an event camera, a storage unit that stores a multilayer perceptron that has learned in advance the relationship between the event data and the relative position and relative velocity of the tracking object, using the event data corresponding to the relative position and relative velocity of the tracking object as training data; a calculation unit that calculates the relative position and relative velocity of the tracking object from the input event data, The calculation unit inputting event data acquired from an event camera and data on the provisional relative position and relative velocity of the tracking object into the multilayer perceptron, and determining deviations of the provisional relative position and relative velocity from the event data; a process of correcting the data of the relative position and relative velocity of the tracked object in a direction to reduce the deviation; a process of inputting the corrected data of the relative position and relative velocity of the tracking object into the multi-layer perceptron as provisional data of the relative position and relative velocity of the tracking object to determine a deviation from the event data; The tracking device repeats the above steps and determines the relative position and relative velocity of the object to be tracked when a predetermined termination condition is met.

2. 2. The tracking device according to claim 1, wherein the object to be tracked is a plane or a three-dimensional object having a predetermined pattern.

3. 3. The tracking device according to claim 1, wherein the multilayer perceptron is trained using the relative position and relative velocity of the object to be tracked and corresponding event data that has been subjected to blurring processing as training data.

4. A method for tracking a tracking target using event data acquired from an event camera, comprising: preparing a multi-layer perceptron that has previously learned the relationship between the event data and the relative position and relative velocity of the tracked object using the event data corresponding to the relative position and relative velocity of the tracked object as training data; inputting event data acquired from an event camera and data on a tentative relative position and relative velocity of a tracking object into the multilayer perceptron, and calculating deviations of the tentative relative position and relative velocity from the event data; a step of correcting the data of the relative position and relative velocity of the tracked object in a direction to reduce the deviation, inputting the corrected data of the relative position and relative velocity of the tracked object to the multilayer perceptron as provisional data of the relative position and relative velocity of the tracked object, and repeatedly performing a process of determining deviations of the provisional relative position and relative velocity from the event data; determining the relative position and relative velocity of the tracked object when a predetermined termination condition is satisfied; A tracking method comprising:

5. 1. A method for generating a multi-layer perceptron that represents a relationship between a relative position and a relative velocity of a tracked object and event data, comprising: acquiring relative position and relative velocity of a tracked object and event data; applying blurring to the event data; a step of training the multilayer perceptron so as to minimize an error between event data generated by applying the data on the relative position and relative velocity of the tracked object to the multilayer perceptron and the blurred event data; A generating method comprising:

6. 6. The generating method according to claim 5, wherein the step of acquiring the event data comprises generating the event data by simulating random movement of the tracking target when the tracking target is known.

7. A program for tracking a tracking target using event data acquired from an event camera, the program comprising: preparing a multi-layer perceptron that has previously learned the relationship between the event data and the relative position and relative velocity of the tracked object using the event data corresponding to the relative position and relative velocity of the tracked object as training data; inputting event data acquired from an event camera and data on a tentative relative position and relative velocity of a tracking object into the multilayer perceptron, and calculating deviations of the tentative relative position and relative velocity from the event data; a step of correcting the data of the relative position and relative velocity of the tracked object in a direction to reduce the deviation, inputting the corrected data of the relative position and relative velocity of the tracked object to the multilayer perceptron as provisional data of the relative position and relative velocity of the tracked object, and repeatedly performing a process of determining deviations of the provisional relative position and relative velocity from the event data; determining the relative position and relative velocity of the tracked object when a predetermined termination condition is satisfied; A program that executes the following.

Images