Tracking device and program
The tracking device improves event camera tracking by channeling event data based on polarity, extracting spiking features, and using a particle filter to enhance accuracy for small objects, addressing the challenges of sparse data and noise differentiation.
Patent Information
- Application Number
- JP2022019776
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2022-02-10
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2042-02-10
AI Technical Summary
Conventional tracking technologies using event cameras struggle to achieve accurate tracking of extremely small moving objects due to sparse data and difficulty in distinguishing between signal and noise, leading to decreased detection accuracy.
A tracking device that separates event camera data into channels based on polarity, extracts features using spiking filters, constructs sequential frames, and employs a particle filter for accurate tracking by averaging likelihood across channels.
Enhances tracking accuracy of small moving objects by maintaining temporal resolution and distinguishing between signal and noise, ensuring reliable tracking even for very small objects.
Smart Images

Figure 0007593960000005 
Figure 0007593960000006 
Figure 0007593960000007
Abstract
Description
[Technical field]
[0001] The present invention relates to a tracking device and a program for tracking a moving object using event camera data. [Background technology]
[0002] <About the event camera> In recent years, the event camera disclosed in Non-Patent Document 1 has become known as a technology for generating visual signals of a measurement object at higher speeds. The event camera was developed inspired by the retinal structure of living organisms. As shown in the schematic example in Figure 1, the output of the event camera is an asynchronous data string that indicates the time, pixel position, and polarity (whether the brightness has decreased or increased) when the brightness has changed to or above a certain threshold value TH.
[0003] In Fig. 1, a certain pixel position (x, y) is fixed, and a graph example of the luminance value sensed at this pixel (x, y) is shown, in which data points d1, d2, ..., d8 at times t1, t2, ..., t8 on the graph are output asynchronously and discretely as the output of the event camera from this pixel (x, y). These data points d1, d2, ..., d8 are output at times t1, t2, ..., t8 when the graph crosses horizontal lines (drawn as dotted lines) that divide the luminance values into equal intervals (equal intervals with the width of the threshold value TH described above) in the graph of Fig. 1.
[0004] That is, the time sequence in which the event camera output occurs at a certain pixel position (x, y) is called t k (k=1,2,…), and the time t k The brightness value at d k (k=1,2,…), the adjacent time t k , t k+1 The brightness value at d k and d k+1 The relationship is as follows: "d k+1 =d k +TH" or "d k+1 =d k -TH”
[0005] In other words, at a pixel position (x, y) in the event camera, k In the case of luminance value d k Assume that an event camera data output occurs at time t k After that, the behavior of the time change of the brightness value at the same position (x, y) is monitored, and the brightness value d k The time when the change from exceeds the threshold TH is the next time t k+1 As a result, the next event camera data output occurs.
[0006] As described above, in the event camera output, binary information is also output as polarity for each discretely output data point, indicating whether the brightness value at that time has increased by the threshold value TH from the brightness value at the previous time, or whether it has decreased by the threshold value TH. In the example of Fig. 1, for example, data d1 at time t1 is output triggered by an increase in brightness by the threshold value TH compared to data d0 at the previous time t0, so the polarity is increasing, and data d2 at the next time t2 is output triggered by a decrease in brightness by the threshold value TH compared to the previous time t1, so the polarity is decreasing. In Fig. 1, the polarity of increase and decrease is diagrammatically indicated by an up arrow (↑) for increase and a down arrow (↓) for decrease.
[0007] In the example shown in Figure 1 above, a single pixel (x, y) was fixed and the event camera output was observed. However, since the same output is obtained for all pixels (x, y) within the image range, the event camera output generally becomes data in the following format:
[0008]
number
[0009] In the above data, t k ≦t k+1 The event camera output data is, in other words, at the kth time t kAt the pixel position (x k ,y k ) in polarity p k This means that there has been a change in brightness (increase ↑ or decrease ↓). Note that N is the total amount of data obtained by taking pictures with an event camera over a certain period of time. (If there is no change in the shooting conditions and no noise effects, N=0 may occur and the event camera output may be null data.)
[0010] In addition, t k ≦t k+1 ≦t k+2 Therefore, even if the event camera output occurs at two or more different pixel positions at the same time, the inequality sign "≦" corresponds to the equality sign "=", and two or more different consecutive indexes k, k+1, ... occur at the same time (t k =t k+1 =...), they can be recorded on the event camera data by linking them to different pixel positions and the polarities at the different pixel positions. For example, if event camera data with an increase polarity "↑" occurs at a pixel position (0,0) and event camera data with a decrease polarity "↓" occurs at another pixel position (1,1) at the same time t=1 second, indexes of, for example, k=1,2 are assigned to these two simultaneous time data, and they can be recorded as follows (with t1=t2=1 second): (x1,y1,t1,p1)=(0, 0, 1 second, ↑) (x2,y2,t2,p2)=(1, 1, 1 second, ↓)
[0011] With this configuration, event cameras have the advantage of not outputting pixel information with no change in brightness, in other words, redundant data, as do conventional cameras. This reduces the amount of data communication and lightens the load of image processing, making it possible to generate images of the object to be measured more quickly.
[0012] Therefore, the output of the event camera (x k ,y k ,t k ,p k) is spatially very sparse and has a very small amount of data compared to the output image P(x,y) (0≦x≦W-1, 0≦y≦H-1, where W and H are the number of pixels in the width and height of the image, and P(x,y) is the pixel value) obtained as an image by a conventional camera at each time t (t=1,2,...). In addition, event cameras have the characteristics of having a very high time resolution (for example, on the order of microms) and a high dynamic range compared to conventional cameras. In other words, event cameras have the characteristics of a small amount of data, high time resolution, and high dynamic range (HDR) compared to conventional cameras. These characteristics are very important for detecting moving objects by processing images instantly or in real time.
[0013] <About video tracking technology> A technology has been proposed that tracks objects in each frame by inputting a video and specifying the object to be tracked in the first frame. In recent years, advances in deep learning have significantly improved the accuracy of detection technology, so an approach called Tracking by Detection as described in Non-Patent Document 2 is widely used. In the Tracking by Detection method, the object to be tracked is first detected (Detection) from each frame of the video, and then the same ID is assigned to the same object between frames to obtain the tracking result.
[0014] However, when the moving object captured in the image is a very small moving object with a small size on the image, the detection accuracy tends to deteriorate. In this case, it is preferable to adopt a framework using a particle filter instead of detecting the target object as in Patent Document 1. In the particle filter, many possible next states from the current state are represented by many particles, and the weighted average calculated according to the likelihood of all particles is estimated to be the current state and tracking is performed.
[0015] When using a particle filter, it is necessary to define, depending on the task, a transition model to determine how to transition a particle from its current state to a number of possible next states, and a likelihood to determine how to calculate the similarity between the particle's state and an observed value (e.g., pixel value).
[0016] For example, in Non-Patent Document 3, the location of a moving object to be tracked at the tracking start time is known, the moving object is approximated by an ellipse, and the position, orientation, and size of the ellipse are assumed to be known. The target to be tracked is set as the state of the ellipse that approximates the moving object at each time T. Specifically, the state S of the ellipse at each time T is T is as shown in equation (1), and in Non-Patent Document 3, the state S T Regarding this, we adopted the transition model of equations (2) and (3) below and the likelihood of equations (4) and (5).
[0017]
number
[0018] In equation (1), {x(T), y(T)} are the coordinates of the center of the ellipse, {x'(T), y'(T)} (the upper dot in the equation is represented by an apostrophe ') is the moving speed (first derivative of the coordinates) of the center of the ellipse {x(T), y(T)}, and H x (T) is half the length of the major axis (major axis), and H x (T) is half the length of the minor axis (minor axis), and a'(T) is the angle between the major axis and the horizontal line (x-axis of the coordinate system). All of these values in equation (1) are the values at time T, and the state of the ellipse S T Its position {x(T), y(T)}, velocity {x'(T), y'(T)}, size and shape {H x (T),H y (T)} and the direction a'(T).
[0019] Ellipse state S TIn equations (2) and (3) that give the transition model of , δ is a parameter that is determined in advance, and W is a random number matrix that follows a multivariate normal distribution. The parameters of the multivariate normal distribution are determined in advance.
[0020] Regarding equations (4) and (5) that give the likelihood, the Bhattacharyya distance d[p,q] defined in equation (4) is used. That is, the Bhattacharyya distance d[p,q] between the target color histogram q (called the target histogram) to be tracked in the frame at the tracking start time (for example, T=1) and the candidate color histogram p (called the candidate histogram) of the image area corresponding to the state of each particle in the frame at the current time t is calculated as shown in equations (4) and (5). Each histogram p and q in equations (4) and (5) lists the frequency of appearance in each bin of the histogram as an element of a vector, as shown below, where m is the number of bins of the histogram. (When both histograms p and q are the same, ρ=1 in equation (5) and the Bhattacharyya distance d=0 (minimum value) in equation (4), which means that both histograms p and q are most similar. Conversely, the higher the degree of dissimilarity between both histograms p and q, the larger the value of the Bhattacharyya distance d.) Candidate color histogram p={p (u)} u=1,2,…,m Target color histogram q={q (u)} u=1,2,…,m
[0021] <Using particle filters on event camera data> When applying a particle filter to an event camera, in addition to the transition model and likelihood above, it is also necessary to define the update time. In other words, in video, the frame time is the natural update time, but in event camera data, signals and data are generated asynchronously between each pixel, so the definition of the update time is not clear.
[0022] In addition, Non-Patent Document 4, which attempts to apply a particle filter to event camera data, reports that it is difficult to calculate the likelihood using a single event camera data. In Non-Patent Document 4, a ball is tracked as the tracking target, and the event camera data is accumulated in chronological order using the area of the event camera data corresponding to the state of each particle, the likelihood is calculated, and the time when the maximum likelihood is added is regarded as the update time. [Prior art documents] [Patent documents]
[0023] [Patent Document 1] JP 2009-199363 A [Non-patent literature]
[0024] [Non-Patent Document 1] Lichtsteiner, P.; Posch, C.; Delbruck, T. (February 2008). "A 128×128 120 dB 15μs Latency Asynchronous Temporal Contrast Vision Sensor". IEEE Journal of Solid-State Circuits. 43 (2): 566‐576. [Non-Patent Document 2] Bergmann, Philipp, Tim Meinhardt, and Laura Leal-Taixe. "Tracking without bells and whistles." Proceedings of the IEEE / CVF International Conference on Computer Vision. 2019. [Non-Patent Document 3] Nummiaro, Katja, Esther Koller-Meier, and Luc Van Gool. "An adaptive color-based particle filter." Image and vision computing 21.1 (2003): 99-110. [Non-Patent Document 4] Glover, Arren, and Chiara Bartolozzi. "Robust visual tracking with a freely-moving event camera." 2017 IEEE / RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2017. [Non-Patent Document 5] Lee, Chankyu, et al. "Spike-flownet: event-based optical flow estimation with energy-efficient hybrid neural networks." European Conference on Computer Vision (ECCV). Springer, Cham, 2020. [Non-Patent Document 6] Yusuke Sakami, Yoshio Morino. "Deep Learning in Spiking Neural Networks." Production Research 71.2 (2019): 159-167. Summary of the Invention [Problem to be solved by the invention]
[0025] However, conventional techniques have not always been able to ensure sufficient accuracy in tracking extremely small moving objects in event camera data.
[0026] That is, the inventors have confirmed that, in the conventional technology, when the likelihood is calculated using the shape of the ball or the color histogram of the tracked object, the accuracy of detecting extremely small moving objects decreases. This is thought to be because the event camera data capturing extremely small moving objects is sparse (the amount of data from the event camera is proportional to the number of pixels whose brightness changes, so it is sparse for small moving objects), which significantly reduces the ability to distinguish shapes (Non-Patent Document 4), colors (Non-Patent Document 3), and textures. Here, there is a problem in that it is difficult to distinguish between the signal of an extremely small moving object and noise in the presence of noise. Ultimately, when the tracked object is below a certain size, tracking using a particle filter is difficult.
[0027] Fig. 2 shows the results of tracking a ball from a rotating plate CG (computer graphics) image using the color histogram likelihood method of Non-Patent Document 3, as an example of an experiment conducted by the inventors who confirmed that tracking becomes difficult when the size of the tracking target becomes small. As shown in the examples EX1 to EX4 on the upper side, a CG image is prepared in which circular dots (representing a ball with circular dots) with radii R of 8, 4, 2, and 1 pixels are provided on the circumference of a rotating plate (plate) as a CG image containing a tracking target, and the CG image rotates around the center of the plate as the rotation axis. The results of tracking these circular dots with radii R = 8, 4, 2, and 1 pixels that rotate with the rotation of the plate in this CG image to draw a circular orbit are shown in the examples EX11 to EX14 on the lower side, respectively. As shown in examples EX11 and EX12, respectively, when the circular dot is L (large) size and M (medium) size with a radius R = 8.4 pixels, tracking is successful in the form of a circular orbit, but as shown in examples EX13 and EX14, respectively, when the circular dot is S (small) size and SS (extremely small) size with a radius R = 2.1 pixels, tracking fails, resulting in the circular orbit not being traced and remaining near the initial position. (Note that in the case of the S size in example EX13, tracking is successful for a short interval after the initial time with an arc-shaped orbit, but after that the circle is no longer traced and tracking fails. Note that in each of examples EX11 to EX14 in Figure 2, in addition to the orbits that were successfully / unsuccessfully tracked, the state of the ellipse S at each time T mentioned above is plotted as a dot. Tis also plotted at the same time.)
[0028] In view of the above problems with the conventional technology, an object of the present invention is to provide a tracking device and program that can ensure accuracy in tracking even very small moving objects in event camera data. [Means for solving the problem]
[0029] In order to achieve the above object, the present invention is a tracking device that tracks moving objects from event camera data, comprising: a channel separation unit that separates the event camera data into two channels according to positive and negative polarity; a feature extraction unit that extracts features for each channel by triggering a signal generation at a pixel position when the generation of event camera data is accumulated in the vicinity of each pixel position for each of the channel-separated data; a sequential frame construction unit that constructs a frame for each time by combining the features for each channel in the channel direction and adding them within a time range window corresponding to each time; and a tracking unit that tracks moving objects for the constructed frames for each time. The present invention is also characterized by a program that causes a computer to function as the tracking device. Effect of the Invention
[0030] According to the present invention, event camera data is separated into two channels according to positive or negative polarity, and features are extracted for each channel triggered by nearby signal accumulation at each pixel position. A frame to be used for tracking of moving objects is constructed from these features, and tracking is performed. This makes it possible to ensure accuracy in tracking even extremely small moving objects. [Brief description of the drawings]
[0031] [Figure 1] FIG. 13 is a diagram showing a schematic example of an output from an event camera. [Diagram 2] 13A and 13B are diagrams illustrating an example of an experiment that confirmed that tracking becomes difficult as the size of a tracking target becomes smaller. [Diagram 3]FIG. 2 is a functional block diagram of a tracking device according to an embodiment. [Figure 4] FIG. 1 is a diagram showing a schematic diagram of a photographing situation and obtained data in an illustrative example. [Diagram 5] FIG. 5 is a diagram showing an example of actual data in the example of shooting in FIG. 4. [Figure 6] 11 is a schematic diagram showing the processing contents of a frame construction unit; FIG. [Figure 7] FIG. 1 is a schematic diagram of a spiking filter in an integrate-and-fire model. [Figure 8] FIG. 5 is a diagram showing an example of actual data in the example of shooting in FIG. 4. [Figure 9] 11A and 11B are schematic diagrams illustrating a tracking process using a particle filter in a tracking unit. [Figure 10] FIG. 2 is a diagram for illustrating a schematic diagram of an object for which likelihood calculation is performed in the present embodiment. [Figure 11] FIG. 1 is a diagram illustrating a hardware configuration of a typical computer. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0032] 3 is a functional block diagram of a tracking device 10 according to an embodiment. The tracking device 10 includes an imaging unit 1, a frame constructing unit 2, and a tracking unit 3. The frame constructing unit 2 further includes a channel separating unit 21, a feature extracting unit 22, and a sequential frame constructing unit 23. The tracking unit 3 further includes a predicting unit 31, a likelihood calculating unit 32, and a resampling unit 33.
[0033] The overall operation of the tracking device 10 is as follows. The photographing unit 1 is composed of an event camera as hardware, and acquires real-time event camera data (x, y, t, p) by photographing, and outputs it to the frame constructing unit 2. The frame constructing unit 2 constructs a frame F(T) to be subjected to tracking processing from this event camera data (x, y, t, p) input in real time, and outputs it to the tracking unit 3. The tracking unit 3 applies object tracking processing to the frame F(T) input for each time T (T=1, 2, ...) in real time, and outputs the object tracking results at each time T.
[0034] As described above, the time recorded in the event camera data (a time that may occur irregularly) is defined as time t, and the frame to be tracked is defined as frame F(T) at time T, to distinguish the notation. In general, the frame constructor 3 constructs a frame F(T) at a corresponding time T using an event camera data set {(x, y, t, p)|t=t1, t2, ..., tn} at multiple times t=t1, t2, ..., tn} that span a certain time range (the range of a sliding window, which will be described later).
[0035] On the premise of the overall operation as described above, the following describes the details of the processing of the frame constructing unit 2 and the tracking unit 3 in the tracking device 10. Note that since the processing contents of the frame constructing unit 2 and the tracking unit 3 are common at each time in real time (time t of the event camera data and time T of the frame), the following describes the processing contents common at each time in real time.
[0036] In the following, the very small moving object photographed by the event camera of the photographing unit 1 is a ball (circular dot) that moves in a circular motion similar to that shown in the experimental example of FIG. 2 described above, and the tracking unit 3 tracks this circular dot as an example for explanation. FIG. 4 shows a schematic diagram of the photographing situation and the data obtained in this explanatory example. In a photographing environment E, the photographing unit 1, which is an event camera, photographs a disk (with a circular dot pattern on its circumference) rotating in the direction of the arrow A as the photographing target OB. As a result, in the obtained event camera data D, in a three-dimensional space (x, y, t) consisting of pixel coordinates (x, y) and time coordinate t, a data group that is spiral in the direction of time coordinate t as a result of the rotation of the circular dot is obtained. (In reality, not only the spiral data group that is the original signal, but also noise may occur.) In this explanatory example, the tracking unit 3 is required to appropriately track this rotational motion.
[0037] Here, a very small moving object is an object with a small area, for example, an area between 1 pixel (1x1) and 100 pixels (10x10). As a typical example, a state in which the object moves by 1 pixel or more every 3 ms is assumed. However, this embodiment can be applied to other cases as well.
[0038] Figures 5 and 8 are diagrams showing actual data examples D1 to D3 and Exa, EXb in the shooting example of Figure 4, and will be referred to as appropriate in the following description. Note that in each of the examples in Figures 5 and 8 that relate to event camera data (other than example EXb in Figure 8 that shows an example frame), dark colored points represent polarity p=↑ (increase) in the event camera data (x, y, t, p), and light colored points represent polarity p=↓ (decrease) in the event camera data (x, y, t, p).
[0039] <Frame Construction Part 2> The frame construction unit 2 calculates features from the event camera data (x, y, t, p) captured by an event camera of a very small moving object, and constructs a frame F(T) as an event frame. It is desirable to construct a frame that maintains the features with strong discriminatory ability and the time resolution of the event camera.
[0040] Data example D1 in Figure 5 shows event camera data capturing an L-size (R = 8 pixels) rotating plate. As in example D in Figure 4, the original signal should be a spiral data group, but in reality, as in data example D1 in Figure 5 and data example D2, which is a time-expanded portion of the data example, it is necessary to utilize the characteristics of the noise and signal in order to distinguish between the signal of a very small moving object and the noise in the presence of noise. As can be seen from data examples D1 and D2, the time continuity of the signal (the spiral part) is clearly stronger than the noise. However, in this example, the ball is L-size, and as the size of the ball becomes smaller (for example, as it becomes S-size or SS-size in the example in Figure 2), the signal becomes sparse.
[0041] Based on the above consideration of the data examples D1 and D2 in Fig. 5, the frame constructor 2 specifically performs the following processing in each of the units 21, 22, and 23 to construct the frame F(t). Fig. 6 is a schematic diagram showing the processing contents of the frame constructor 2, and will be referred to as appropriate in the following explanation.
[0042] In the frame construction unit 2 of this embodiment, in order to utilize time continuity, a spiking filter is constructed using a spiking neuron such as that described in Non-Patent Document 6, and the output from the spiking filter is calculated as a spiking feature. Furthermore, in order to deal with sparse signals, a frame F(T) is constructed as a sequential frame accumulated on the time axis. The specific procedure is as follows.
[0043] <Channel separation section 21> The channel separation unit 21 separates the event camera data (x, y, t, p) into channels according to their polarity, and outputs the separated data (x, y, t, ↑), (x, y, t, ↓) for each channel (the first channel with polarity p=↑ (increase, positive, Positive) and the second channel with polarity p=↓ (decrease, negative, Negative)) to the feature extraction unit 22. The reason for performing such channel separation is as follows.
[0044] The asynchronous event camera data output from the event camera is (x, y, t, p). Since the tracked object is moving, it is possible that the location where the event camera data is generated is not limited to a single pixel, but is expanded to adjacent pixels (nearby pixels). In the case of a very small moving object, positive and negative are paired in adjacent pixels as in data example D3 in Figure 5 (data example D3 is an even shorter time range than data example D2). If the mixed positive and negative values are input from adjacent pixels to the spiking filter without channel separation, the addition of the mixed positive and negative values may cancel out the internal state of the spiking filter, and the spiking filter may not fire.
[0045] Therefore, in order to enable the spiking filter to fire appropriately, in this embodiment, the channel separation unit 21 separates the signals according to the polarity (positive or negative) as in process P1 in Fig. 6, separating them into independent channels CH1 and CH2, which are then input to the spiking filters FL1 and FL2 in process P2. (This process P2 is the next process by the feature extraction unit 22.)
[0046] In addition to appropriately firing the spiking filter as described above, the role of channel separation is to appropriately extract this difference as a feature to improve tracking accuracy, since the difference in polarity between positive and negative in the event camera data also reflects the difference in the manner in which a tiny moving object passes through a pixel position (for example, the difference between when a tiny moving object appears at a certain position (x, y) and when it disappears), as can be seen in data example D3 in Figure 5.
[0047] <Feature extraction unit 22> The feature extraction unit 22 similarly applies a spiking filter to each of the channel-separated data (x, y, t, ↑), (x, y, t, ↓) to obtain spiking features (x, y, t, ↑), (x, y, t, ↓) and sequentially outputs them to the frame construction unit 23.
[0048] That is, as shown in Fig. 6 as process P2, the feature extraction unit 22 monitors the accumulation of signal generation in the spatial vicinity range Na at each position a = (x, y) of the first channel of the positive, outputs a firing event (x, y, t, ↑) at the position a = (x, y) at time t when a certain amount of accumulation has occurred, and resets the signal accumulation at the position a = (x, y) to zero, thereby outputting the spiking feature F1 of the first channel. In exactly the same way, the feature extraction unit 22 monitors the accumulation of signal generation in the spatial vicinity range Nb at each position b = (x, y) of the second channel of the negative, outputs a firing event (x, y, t, ↑) at the position b = (x, y) at time t when a certain amount of accumulation has occurred, and resets the signal accumulation at the position b = (x, y) to zero, thereby outputting the spiking feature F2 of the second channel.
[0049] As described above, the spiking features F1 and F2 are also asynchronous data, and can be formally expressed as data (x, y, t, p) similar to the event camera data.
[0050] The spiking filter applied to each separation channel by the feature extraction unit 22 may be an integrate and fire neuron model disclosed in Non-Patent Document 5 and the like. FIG. 7 shows a schematic diagram of a spiking filter of the integrate and fire model. In this example, as the neighborhood N3 of position p3, in addition to position p3, four points p1, p2, p4, and p5 (four neighborhoods) are used, and the occurrence of firing F3 at position p3 is determined by accumulating and monitoring event camera data in the neighborhood N3. In the example of firing F3, at time t A ,t B ,t C Then, the accumulation reaches the threshold value TH, firing occurs, and the accumulation (and the firing count number at each point in the vicinity N3 for determining firing at position p3) is reset to zero.
[0051] In this embodiment, as a setting for applying the spiking filter, for example, a 3x3 pixel window adjacent to the target pixel is set as the neighborhood, and a stride of 1 is set as a two-layer structure as illustrated in FIG. 7. (The side of individual pixels p1 to p5 is a lower layer, and the side aggregated to N3 near p3 is a higher layer, resulting in a two-layer structure.) Note that the stride may be increased to 2, 3, ... to increase the number of layers of the spiking filter to the higher side, such as 3 layers, 4 layers, ..., but a smaller number of layers makes it easier to perform learning in advance. Also, one channel is output for one weight (w1, w2, w3, ...). Regarding weights, for example, in the example of Figure 7, the accumulation amount Accum(p3) used to determine whether pixel p3 is fired is calculated as follows by weighting the number of times n1, n2, n3, n4, and n5 that event camera data occurred at each pixel p1, p2, p3, p4, and p5 in the neighborhood N3 with weights w1, w2, w3, w4, and w5, respectively. Accum(p3) = Σ i=1 5 wi*pi=w1*n1+w2*n2+w3*n3+w4*n4+w5*n5
[0052] In this embodiment, by preparing K sets of weights (K is a pre-set hyperparameter, and the weights may be either pre-set or learned), K channel spiking features F1, F2 are obtained for each of the positive and negative channels. Then, in the next sequential frame construction unit 23, as in process P3 in Fig. 6, the spiking feature F1 (K channel) of the first positive channel CH1 and the spiking feature F2 (K channel) of the second negative channel CH2 are concatenated in the channel direction to obtain a total of 2K channels.
[0053] Note that although the feature extraction unit 22 has been described as using a spiking filter, when it is determined that there is an accumulation of event camera data in the vicinity of each pixel position (x, y), any filter capable of triggering an output of a signal at the pixel position (x, y) may be used. For example, in the case of a spiking filter, as in the example of FIG. 7, when there is a signal generation in the vicinity N3 of each pixel position p3, it is always added, and the accumulated amount is not subtracted unless the threshold value TH is reached. However, it may be subtracted at a fixed amount at regular intervals, or subtracted by multiplying by a fixed ratio r (0 < r < 1), so as to monitor the accumulated amount while periodically subtracting what is considered to be the influence of noise. Also, the behavior of the temporal change of the accumulated amount (such as whether it is determined that the increase rate is large) may be considered for monitoring.
[0054] <Sequential frame construction unit 23> As shown as process P3 in FIG. 6, the sequential frame construction unit 23 constructs a frame F(T) by combining the values over a certain time range up to time T of the spiking features F1, F2 of each positive and negative channel in the channel direction.
[0055] That is, in the case of a very small moving object, since the data is sparse, a certain amount of time accumulation is required. As a proportional example, in Non-Patent Document 5, since there is no overlap on the time axis, the time resolution of the original event camera data is sacrificed to some extent. In the present embodiment, in order to maintain the time resolution, sequential frames are constructed. That is, as in the example of process P3 in FIG. 6 and the data example Exa in FIG. 8, the time zone is divided using a sliding window that overlaps on the time axis, the total number of spiking features at each pixel within the time range of the sliding window is counted, and this total number of spiking features is used as the value of each pixel, and the number of spiking features for each pixel position is output as a one-channel frame (for each channel out of a total of 2K channels).
[0056] In the process P3 in FIG. 6 and the data example Exa in FIG. 8, overlapping sliding windows W(T-1), W(T), and W(T+1) on the time axis are used to construct successive frames F(T-1), F(T), and F(T+1) at adjacent times T-1, T, and T+1, respectively. The data example EXb in FIG. 8 is an example of a frame of one channel constructed using such overlapping windows. In this example, the number of spiking features at each pixel position is about 10, but in this embodiment, each pixel is, for example, a multi-value signal in the range [0, 255] of 8 bits. (In this case, if the number of spiking features counted for each pixel position (x, y) within the window W(T) at each time T exceeds 255, it may be assumed that the number caps out at 255 (as equivalent to a blown-out highlight state in a normal image). If the number frequently exceeds 255, the width of the window W(T) may be set narrower, or a range wider than 8 bits may be used if the computational resources allow.) The time width of the sliding window and the time width of the overlap may be set in advance.
[0057] The sequential frame construction unit 23 performs the above operations for each channel, and constructs the Tth frame F(T)=F as a frame at time T (tensor of size W×H×C) from one sliding window W(T). T (W, H, C) is output to the tracking unit 3 (likelihood calculation unit 32). WxH is the resolution of the event camera (W is the number of pixels in the horizontal direction, and H is the number of pixels in the vertical direction), C is the number of channels, and C = 2K because they are combined in the channel direction.
[0058] <Tracking Unit 3> The tracking unit 3 repeatedly tracks extremely small moving objects in a frame F(T) using a particle filter at each time T in real time. To this end, the following processes are specifically performed in each of the units 31 to 33. Fig. 9 is a schematic diagram of the process of tracking using a particle filter in the tracking unit 3, and will be referred to as appropriate in the following description.
[0059] <Prediction Section 31> The prediction unit 3 uses the transition model to predict the state S of the previous frame F(T-1) for each particle. T-1 to the state S of the current frame F(T) T The state S of the previous frame F(T-1) is predicted and output to the likelihood calculation unit 32. T-1 By using the value output by the resampling unit 33 at the previous time T-1, the prediction unit 31, the likelihood calculation unit 32 and the resampling unit 33 can perform repeated processing for each time T, as shown in FIG. 3.
[0060] As shown in FIG. 9, the prediction process P31 of the prediction unit 3 is performed by using a particle filter to calculate the state S of the previous frame in the state space. T-1 (This state is composed of multiple particles, and in Fig. 9, three particles are used as an example.) T (The predicted multiple states S T From the result, the state S T is determined.)
[0061] In this embodiment, as an example, the very small moving object to be tracked is a ball (circular dot) moving in a circular motion, so to model this ball, the above-mentioned formula (1) similar to that in Non-Patent Document 3 is used to express the state of the particle. The above-mentioned formulas (2) and (3) are also used for the transition model, as in Non-Patent Document 3. However, since the time corresponding to a frame of the event camera is shorter than the video in Non-Patent Document 3, the effect of the moving speed is small. Therefore, the value of δ in formula (3) should be set to a value smaller than the value set in Non-Patent Document 3. The above-mentioned formulas (2) and (3) can be written component by component as follows. (Note that w T-1 is the value for each component in the multivariate normal distribution W, but in the following, we use the same notation for all components to simplify the notation.) Position x(T)=x(T-1)+δ*x'(T-1)+w T-1 , y(T)=y(T-1)+δ*y'(T-1)+ w T-1 Speed x'(T)=x'(T-1)+w T-1 , y'(T)=y'(T-1)+w T-1 Size / shape H x (T)=H x (T-1)+w T-1 , H y (T)=H y (T-1)+w T-1 Orientation a(T)=a(T-1)+w T-1
[0062] As for state S1 at the initial time T=1 when tracking starts, the particle filter method can be applied to the subsequent times T≧2 by manually specifying the state of the ball's initial position.
[0063] <Likelihood calculation unit 32> The likelihood calculation unit 32 calculates the current frame F(T)=F T For (W, H, C), each particle (state S T ) and updates the weight of each particle according to the likelihood. In FIG. 9, the weight update by likelihood calculation for each particle predicted as three candidates from three states is shown as process P32. The main steps are as follows: 1 to 3. Steps 2 and 3 are the same as those of existing particle filter methods.
[0064] (First step: Likelihood calculation) The likelihood may be calculated as the likelihood based on the Bhattacharya distance d[p,q] between the target histogram q and the candidate histogram p of each particle in the above-mentioned formula (4) in the same manner as in Non-Patent Document 3. However, in this embodiment, the distances calculated for each channel, including the positive channel (C=the first K channels out of 2K channels) and the negative channel (C=the last K channels out of 2K channels), are averaged.
[0065] Moreover, the event camera data is affected not only by the magnitude of the movement but also by the direction of movement. Since the direction of movement of the ball is changing, a method different from that of Non-Patent Document 3 may be used in which the target histogram is updated at a constant rate from the information of the corresponding frame and then the Bhattacharyya distance is calculated. That is, in Non-Patent Document 3, in order to calculate the distance d[p,q] in Equation (4), the target histogram q at the initial time T=1 is used for comparison with the candidate histogram p at the current time T. However, in this embodiment, the target histogram q at the previous time T-1 is used for comparison, and the histogram updated to the previous time T-1, which is the most recent time, is used instead of the initial time T=1.
[0066] FIG. 10 is a diagram for illustrating the object of the likelihood calculation described above. In terms of time, the initial time T=1, the current time T at which the likelihood calculation is performed, and frames F(1), F(T), and F(T-1) at the previous time T-1 are illustrated as tensors of size W×H×C. The states S1, S T ,S T-1 The range within the tensor occupied by the i-th (i = 1, 2, ..., N) particle that makes up the tensor is shown as the particle range PR(i,1), PR(i,T), and PR(i,T-1).
[0067] That is, the particle range PR(i,T) at the current time T is the state S in the frame F(T), which is a tensor expressed as a rectangular parallelepiped range. T The range of the elliptical cylinder corresponding to the above is the position x(T), y(T) and the size and shape H x (T),H y The particle ranges PR(i,T-1), PR(i,1) at the previous time T-1 and the initial time T=1 are also in the state S T-1, S1. In this embodiment, a C-dimensional color histogram hist(i,T) is calculated from the particle range PR(i,T) at the current time T and used as a candidate histogram p, and similarly, a C-dimensional color histogram hist(i,T-1) is calculated from the particle range PR(i,T-1) at the previous time T-1 and used as a target histogram q. Candidate histogram p = hist(i,T) Target histogram q = hist(i,T-1)
[0068] As described above, in the method of Non-Patent Document 3, q=hist(i,1) is used without updating it at the initial time T=1, but in this embodiment, q=hist(i,T-1) is used after updating it. Note that here, with regard to Non-Patent Document 3, the way in which the particle filter is used is explained as a comparison with this embodiment, and the histogram with the same data content as this embodiment is not used in Non-Patent Document 3.
[0069] As shown in FIG. 9, for the i-th particle ranges PR(i,T-1) and PR(i,T) at the previous time T-1 and the current time T, the ones that survived the resampling process P33 of the resampling unit 33 and were applied with the time update process P34 are used at the previous time T-1, and the ones predicted by the prediction process P31 of the prediction unit 31 from the survived ones are used at the current time T. Also, the particle range PR(i,1) at the initial time T=1 corresponds to the initial state S1 given as prior knowledge, and the range of each particle that expresses the initial state S1 as a weighted sum may be used. For each histogram p, q, the corresponding ranges PR(i,T) and PR(i,T-1) are C-dimensional in the channel direction, and each channel is given a value within an 8-bit range [0,255], for example, so that a histogram can be obtained for each channel in the same way as in the case of a normal image such as a grayscale image.
[0070] (2nd step: updating weights) The weight of each particle (importance based on likelihood) is updated in this way. For particles with low likelihood calculated as above, the weight of the particle is reduced because the prediction is less accurate, and conversely, for particles with high likelihood, the weight of the particle is increased because the prediction is more accurate. This update is performed assuming that there are a total of N particles, and the weight w of the i-th particle (i=1,2,...,N) at the current time T is T (i) For the i-th particle, the following equations (6) and (7) can be used. d(i) is the Bhattacharyya distance average calculated as above for the i-th particle, and σ(i) is the variance. After calculating with equation (6), the value normalized with equation (7) can be used. (That is, the Bhattacharyya distance of the k-th dimension (1≦k≦C) channel calculated for the i-th particle is d(i) k Then, the Bhattacharya distances of these C i k The mean of |k=1,2,…,C} is d(i) and the variance is σ(i).
[0071]
number
[0072] As shown in the above equations (6) and (7), the smaller the Bhattacharyya distance average d(i) normalized by the Bhattacharyya distance variance σ(i), the more similar the histograms are, so the larger the weight is calculated, which increases the chances of particle i surviving.
[0073] (Third step: Estimating the state of the current frame) Using the weights calculated as above, the states of all particles are weighted-averaged using the following formula (8) and output as the state of the tracked object in the current frame F(T). (Note that the results of the next resampling unit 33 may be weighted-averaged in the same manner as formula (8) and output as the state of the tracked object in the current frame F(T), but since the resampling process deletes particles with small weights, the result will be almost the same.)S (i) T is the state of the i-th particle, and ST is the state of the tracked object in the current frame F(T) calculated by the weighted average of these.
[0074]
number
[0075] <Resampling section 33> The resampling unit 33 re-scatters particles according to the weight of each particle updated by the likelihood calculation unit 32 in the current frame F(T).
[0076] The larger the weight value calculated by the likelihood calculation unit 32 is based on threshold judgment or the like, the more offspring the particle will generate, and the smaller the weight value is based on threshold judgment or the like, the more the particle will be destroyed since it is deemed that the probability of the particle becoming that value is very low. This completes a new sampled probability distribution. The value with the highest probability in this distribution can be used as the final state of the frame at the next time T+1 (the target to which prediction is applied by the prediction unit 31).
[0077] In the example of Figure 9, resampling process P33 shows that only one of three particles with a large weight survives and the two with small weights are eliminated, and process P34 shows that the surviving particle is re-seeded into three particles and then used as the target for prediction process P31 at the next time T+1.
[0078] As described above, according to the present embodiment, the problems of difficulty in distinguishing between signals of extremely small moving objects and noise and sparse event data can be addressed by improving feature extraction and particle filters for event camera data. Specifically, these can be addressed by all or part of the following:
[0079] ● (Feature extraction) Features are extracted for each channel of the event camera data separated according to positive and negative polarity. ●(Feature extraction) Design a spiking feature that uses features that contain time information. ●(Feature extraction) Construct frames sequentially from spiking features. ●(Improvement of particle filter) The definition of likelihood is the sum of the two separate positive and negative channels. ●(Improvement of particle filter) Update the features of the initial tracked object.
[0080] Various supplementary examples, additional examples, alternative examples, etc. will be described below.
[0081] (1) As one application example of the embodiment of the present invention, when monitoring a remote location (such as a factory), it is possible to accurately track extremely small moving objects in the remote location using event camera data. This enables precise monitoring of the remote location without necessarily requiring actual travel to the remote location, and reduces carbon dioxide emissions by saving energy resources required for user travel, which makes it possible to contribute to Goal 13 of the United Nations-led Sustainable Development Goals (SDGs), which is to "take urgent action to combat climate change and its impacts."
[0082] (2) In the above description, the tracking unit 3 uses a particle filter for tracking, but a Kalman filter or the like may also be used. As a typical example, the tracking target in the tracking unit 3 is assumed to be constantly moving at least one pixel every 3 ms, but if the tracking target stops moving, the event camera data will no longer be detected and tracking will be lost. As a solution to this problem, if the event camera data is no longer detected within the screen (not at the edge of the screen), the object can be considered to be stationary and treated as continuing to exist in the position it was last detected in.
[0083] (3) FIG. 11 is a diagram showing an example of a hardware configuration in a general computer device 70. The tracking device 10 can be realized as one or more computer devices 70 having such a configuration. When the tracking device 10 is realized by two or more computer devices 70, information required for processing may be transmitted and received via a network. The computer device 70 includes a CPU (Central Processing Unit) 71 that executes predetermined instructions, a GPU (Graphics Processing Unit) 72 as a dedicated processor that executes part or all of the execution instructions of the CPU 71 in place of the CPU 71 or in cooperation with the CPU 71, a RAM 73 as a main storage device that provides a work area for the CPU 71 (and the GPU 72), a ROM 74 as an auxiliary storage device, a communication interface 75, a display 76, an input interface 77 that accepts user input via a mouse, a keyboard, a touch panel, or the like, a camera 78, and a bus BS for transmitting and receiving data between them.
[0084] Each functional unit of the tracking device 10 can be realized by a CPU 71 and / or a GPU 72 that reads a predetermined program corresponding to the function of each unit from a ROM 74 and executes it. Both the CPU 71 and the GPU 72 are a type of computing device (processor). Here, when display-related processing is performed, a display 76 also operates in conjunction with the above, and when communication-related processing related to data transmission and reception is performed, a communication interface 75 also operates in conjunction with the above. The camera 78 may be implemented as an event camera that realizes the shooting unit 1, but a normal camera (a camera that captures normal images such as RGB images) may also be used in combination for monitoring, confirmation, and other purposes. [Explanation of symbols]
[0085] 10...Tracking device, 1...Photographing unit, 2...Frame construction unit, 3...Tracking unit 21: channel separation unit, 22: feature extraction unit, 23: sequential frame construction unit 31: prediction unit, 32: likelihood calculation unit, 33: resampling unit
Claims
1. A tracking device that tracks moving objects using event camera data, a channel separation unit that separates the event camera data into two channels according to a distinction between positive and negative polarities; a feature extraction unit that extracts features for each channel by triggering a signal generation at a pixel position when generation of event camera data is accumulated in a vicinity of the pixel position for each of the channel-separated data; a sequential frame constructing unit that constructs a frame for each time by combining the features for each channel in a channel direction and adding them within a time range window corresponding to each time; A tracking device comprising: a tracking unit that performs moving object tracking on the constructed frames at each time.
2. The tracking unit performs tracking by using a particle filter, and the processing by the particle filter includes the following steps: A prediction process for predicting a current state from a previous state; a likelihood calculation process for calculating a likelihood for each of the predicted states at the current time; 2. The tracking device according to claim 1, wherein a resampling process is performed for each state at the current time to determine whether the state will survive or disappear at a later time based on the calculated likelihood.
3. The tracking device according to claim 2, characterized in that in the likelihood calculation process, the likelihood is calculated for each of the predicted current state by referring to a corresponding spatial range in the constructed current frame.
4. The tracking device according to claim 2, characterized in that in the likelihood calculation process, for each state (i) at the predicted current time (T), a likelihood of the predicted state (i) at the current time (T) is calculated based on a similarity between a signal distribution (hist(i,T)) of features in a corresponding spatial range (PR(i,T)) in the constructed frame (F(T)) of the current time (T) and, for a state (i) at the previous time (T-1) used in the prediction process to make the prediction, a signal distribution (hist(i,T-1)) of features in a corresponding spatial range (PR(i,T-1)) in the constructed frame (F(T-1)) of the previous time (T-1).
5. 5. The tracking device according to claim 4, characterized in that the signal distribution of the features (hist(i,T), hist(i,T-1)) is evaluated by a histogram.
6. 6. The tracking device according to claim 4, wherein the signal distribution of the features is evaluated for each of the separated channels, and the similarity is calculated using a total of the evaluations for each of the channels.
7. 7. The tracking device according to claim 1, wherein the sequential frame construction unit adds up the number of times the trigger output signal is generated at each pixel position within a time range window corresponding to each time.
8. 8. The tracking device according to claim 1, wherein the feature extraction unit extracts the feature for each channel using a spiking filter based on an integrate-and-fire model.
9. 9. The tracking device according to claim 1, wherein the sequential frame construction section uses, as the time range window corresponding to each time, windows whose time ranges at adjacent times overlap.
10. A program for causing a computer to function as the tracking device according to any one of claims 1 to 9.
Citation Information
Patent Citations
Tracking system using particle filtering
JP2009199363A
Optical flow estimation device, optical flow estimation method, optical flow estimation system, optical flow estimation program, and yaw rate estimation device, yaw rate estimation method, yaw rate estimation system as well as yaw rate estimation program
JP2019204390A
