Object tracking program, apparatus, and method

Abstract

To improve the accuracy of tracking objects.SOLUTION: An object tracking program according to an embodiment causes a computer to function as: means for detecting a position and a type of an object from image data at a first point in time, thereby generating an object detection result at the first point in time, in which the position and the type of the detected object are associated with each other; means for selecting a prediction model for predicting the position of the object at the first point in time, based on an object tracking result at a second point in time earlier than the first point in time, in which the position and the type of the object determined at the second point in time are associated with each other; means for generating a position prediction result at the first point in time, in which the position and the type of the predicted object are associated with each other, based on the selected prediction model and a time series of continuous tracking results including the object tracking result at the second point in time; means for generating an association result by performing association between the object detection result at the first point in time and the position prediction result at the first point in time; and means for generating the object tracking result at the first point in time based on the association result.SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] Embodiments of the present invention relate to an object tracking program, apparatus, and method.

Background Art

[0002] Conventionally, an object tracking device that tracks an object in a captured moving image is known. For example, the object tracking device detects an object from a moving image at each time, and performs association of the same object by evaluating the overlap of temporally continuous object detection results (for example, a rectangular region that substantially encloses the object), and tracks the object.

[0003] In the above-described object tracking device, for example, when a moving object to be tracked is hidden by another object (when occlusion occurs), the object to be tracked cannot be detected, and the tracking of the object may be interrupted. In contrast to this, there is a method of linearly predicting the position of the moving destination from the previous movement trajectory while the object cannot be detected, and continuing the tracking using the predicted position instead of the detected position. This method is relatively effective if the undetected time is within a certain time, for example, for an object moving at a constant speed such as a vehicle or a walking person, because the error (prediction error) between the actual detection position and the predicted position is small.

[0004] However, in the above-described method, since the prediction error is cumulatively accumulated as the undetected time becomes longer, the tracking accuracy of the object may decrease. For example, when a stopped object to be tracked is hidden by a moving other object, the predicted position may shift due to the prediction error even though the object to be tracked has not moved during the undetected time. Therefore, there is a need for a technique for improving the object tracking accuracy while considering the influence of the prediction error.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0006] The problem to be solved by the present invention is to provide an object tracking program, device, and method capable of improving the tracking accuracy of an object.

Means for Solving the Problems

[0007] An object tracking program according to an embodiment causes a computer to function as image data acquisition means, object detection means, prediction model selection means, object prediction means, association means, and tracking processing means. The image data acquisition means acquires image data at a first time. The object detection means generates an object detection result at the first time in which the detected position and type of the object are associated by detecting the position and type of the object from the image data at the first time. The prediction model selection means selects a prediction model for predicting the position of the object at the first time based on the object tracking result at a second time in which the position and type of the object determined at a second time earlier than the first time are associated. The object prediction means generates a position prediction result at the first time in which the predicted position and type of the object are associated by predicting the position of the object at the first time based on the selected prediction model and the time-series continuous tracking results including the object tracking result at the second time. The association means generates an association result by performing an association between the object detection result at the first time and the position prediction result at the first time. The tracking processing means generates an object tracking result at the first time based on the association result

Brief Description of the Drawings

[0008]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Mode for Carrying Out the Invention

[0009] Hereinafter, embodiments of the object tracking device will be described in detail with reference to the drawings.

[0010] (First Embodiment) FIG. 1 is a block diagram illustrating the configuration of an object tracking system 1 including an object tracking device 100 according to the first embodiment. The object tracking system 1 includes an object tracking device 100, an output device 110, and a photographing device 120. The photographing device 120 photographs, for example, a manufacturing site. The object tracking device 100 tracks, for example, products and workers during manufacturing using a moving image of the photographed manufacturing site. The output device 110 displays display data including information about the tracked objects (products and workers).

[0011] The output device 110 is, for example, a monitor. The output device 110 receives display data from the object tracking device 100. The output device 110 displays the display data. Note that the output device 110 is not limited to a monitor as long as it can display the display data. For example, the output device 110 may be a projector. Also, the output device 110 may include a speaker.

[0012] The photographing device 120 is, for example, an image sensor. The photographing device 120 photographs a predetermined area (for example, a manufacturing site) and acquires a moving image. In this embodiment, the data of the moving image acquired by the photographing device 120 is referred to as image data. In other words, the photographing device 120 acquires time-series image data. The photographing device 120 outputs the acquired image data to the object tracking device 100.

[0013] In the following embodiments, as a specific example, it is assumed that a video from the start of photographing to the present is photographed at each time (1, 2,..., t). Also, the current time is described as t.

[0014] FIG. 2 is a block diagram illustrating the configuration of the object tracking device 100 according to the first embodiment. The object tracking device 100 includes an image data acquisition unit 210 (image data acquisition means), an object detection unit 220 (object detection means), a model DB 230, a prediction model selection unit 240 (prediction model selection means), an object prediction unit 250 (object prediction means), an association unit 260 (association means), and a tracking processing unit 270 (tracking processing means).

[0015] Note that the object tracking device 100 may include a memory and a processor (not shown). The memory stores various programs related to the operation of the object tracking device 100 (for example, an object tracking program for tracking an object). The processor realizes functions as image data acquisition means, object detection means, prediction model selection means, object prediction means, association means, and tracking processing means by executing various programs stored in the memory.

[0016] The image data acquisition unit 210 acquires image data at the current time from the imaging device 120. The image data acquisition unit 210 outputs the image data at the current time to the object detection unit 220. Note that the imaging time may be associated with the image data.

[0017] The image data at the current time may mean real-time image data or image data whose acquisition time is the current time. The same applies hereinafter, but in a series of processes in the object tracking device 100, it is assumed that the above two meanings do not coexist. Therefore, in the following specific examples, the image data at the current time will be described as meaning real-time image data.

[0018] The object detection unit 220 receives the image data at the current time from the image data acquisition unit 210. The object detection unit 220 detects the position and type of an object from the image data at the current time. The object detection unit 220 generates an object detection result at the current time, which is an object detection result in which the position and type of the detected object at the current time are associated. The object detection unit 220 outputs the object detection result at the current time to the association unit 260.

[0019] The position of an object is represented, for example, by a rectangular region that roughly encloses the object in the image data. This rectangular region may be referred to as a bounding box. Also, the position of the detected object in the object detection unit 220 may be referred to as the object detection position. That is, the object detection result includes the object detection position.

[0020] Note that, for example, the method described in Reference 1 (Ren, Shaoqing, Kaiming He, Ross Girshick, and Jian Sun. "Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks." Advances in Neural Information Processing Systems. Vol. 28, 2015.) can be used for detecting the position of the object in the object detection unit 220.

[0021] The model DB 230 stores by associating the type of an object, the location where the object exists (for example, the location where the transported object is placed and the location where the person is working), and the prediction model. The model DB 230 outputs information on the prediction model associated with the type and location of the object in response to a read request from the prediction model selection unit 240. Note that the model DB 230 may store a table in which the type of an object, the location, and the prediction model are associated. Also, the information on the prediction model may be information indicating that prediction using the prediction model is not performed.

[0022] Information on the location where the object exists is assumed, for example, to be such that regions for predetermined locations (for example, passages and transported object storage areas) have been detected and set in advance from the moving image. For example, when only the region of the transported object storage area is detected, the region other than the transported object storage area may be set as the passage region. Examples of the prediction model include prediction using a Kalman filter, which is a type of linear prediction.

[0023] The prediction model selection unit 240 acquires the object tracking result at a time before the current time from the tracking processing unit 270. The prediction model selection unit 240 selects a prediction model from the model DB 230 based on the object tracking result at a time before. The prediction model selection unit 240 outputs the information of the selected prediction model to the object prediction unit 250.

[0024] A time before is, for example, time t - 1 with respect to the current time t. The time interval is, for example, the frame interval of a moving image. In the object tracking result, the position (object determination position) and type of the determined object are associated.

[0025] The object prediction unit 250 receives the information of the prediction model from the prediction model selection unit 240 and receives the continuous tracking result at a time before from the tracking processing unit 270. The object prediction unit 250 predicts the position of the object at the current time based on the information of the prediction model and the object tracking result (or object determination position) at a time before included in the continuous tracking result at a time before. The object prediction unit 250 generates a position prediction result at the current time, which is a prediction result of the object with the predicted position and type of the object at the current time associated. The object prediction unit 250 outputs the position prediction result at the current time to the association unit 260. Note that the predicted position of the object in the object prediction unit 250 may be referred to as the object prediction position. That is, the position prediction result includes the object prediction position. Further, when the information of the prediction model is information indicating that prediction is not performed, the object prediction unit 250 may not predict the position of the object at the current position and determine the object tracking result at a time before as the position prediction result at the current time.

[0026] The continuous tracking result is the cumulative time series of the object tracking results from the start time of tracking to a certain time (for example, the time before the current time and the current time) for the same object associated across time. For example, the continuous tracking result at the time before the current time (the continuous tracking result at time t-1) includes the object tracking result at the start time of tracking (the object tracking result at time 1) to the object tracking result at the time before the current time (the object tracking result at time t-1). Also, for example, the continuous tracking result at the current time (the continuous tracking result at time t) includes the object tracking result at the start time of tracking (the object tracking result at time 1) to the object tracking result at the current time (the object tracking result at time t).

[0027] Note that when using a Kalman filter as a prediction model for predicting the position of an object in the object prediction unit 250, for example, the methods described in Reference 2 (Alex Bewley, Zongyuan Ge, Lionel Ott, Fabio Ramos, and Ben Upcroft. Simple online and realtime tracking. In 2016 IEEE international conference on image processing (ICIP), pages 3464-3468. IEEE, 2016.) and the like can be used.

[0028] The association unit 260 receives the object detection result at the current time from the object detection unit 220 and the position prediction result at the current time from the object prediction unit 250. The association unit 260 performs an association process between the object detection result at the current time and the position prediction result at the current time for all object positions (object detection positions and object prediction positions). The association unit 260 generates the association result at the current time including the information of the associated position combinations and the information of the unassociated positions. The association unit 260 outputs the association result at the current time to the tracking processing unit 270.

[0029] The association process calculates an evaluation value by, for example, evaluating the overlap between the rectangular region that is the object detection position and the rectangular region that is the object prediction position, and assigns the object detection position and the object prediction position using this evaluation value. For the evaluation of the overlap, for example, IoU (Intersection over Union) may be used, the similarity of image features (external features of the object) within the rectangular region may be used, or both may be used. For the assignment method, for example, the Hungarian algorithm may be used.

[0030] The information on the combination of the associated positions is the information on the combination of the object detection result and the position prediction result. The information on the unassociated positions is the information on the object detection result or the position prediction result that did not have a combination to be associated.

[0031] The tracking processing unit 270 receives the association result at the current time from the association unit 260. Based on the association result at the current time, the tracking processing unit 270 generates a continuous tracking result at the current time for all object positions. Specifically, the tracking processing unit 270 generates an object tracking result at the current time for all object positions respectively, and generates a continuous tracking result at the current time by adding the object tracking result at the current time to the continuous tracking result at the previous time. The tracking processing unit 270 outputs the continuous tracking result at the current time to the output device 110. Note that before the association unit 260 executes the association process for the image data at the current time, the tracking processing unit 270 outputs the object tracking result at the previous time to the prediction model selection unit 240 and outputs the continuous tracking result at the previous time to the object prediction unit 250.

[0032] The configuration of the object tracking system 1 and the object tracking device 100 including the object tracking device 100 according to the first embodiment has been described above. Next, an operation example of the object tracking device 100 and an operation example of the object tracking device 100 will be described.

[0033] FIG. 3 is an explanatory diagram showing an operation example of the object tracking device 100 according to the first embodiment. In FIG. 3, there are shown the object tracking device 100, a work area WA in a manufacturing site, two transported objects 330 and 340 arranged in the work area WA, a worker 350 working in the work area WA, and a photographing device 120 that photographs the work area WA.

[0034] The work area WA in FIG. 3 includes a storage area 310 for temporarily placing transported objects and a passage 320. The storage area 310 for transported objects and the passage 320 have, for example, different floor colors. The transported object 330 is arranged in the storage area 310 for transported objects, and the transported object 340 is arranged in the passage 320. Also, the worker 350 is working in the passage 320.

[0035] The photographing device 120 outputs image data regarding a photographing area PA that substantially encloses the work area WA to the object tracking device 100. The photographing area PA includes the two transported objects 330 and 340 and the worker 350. That is, the image data includes the two transported objects 330 and 340 and the worker 350.

[0036] In the following specific example, as shown in FIG. 3, tracking of the two transported objects 330 and 340 and the worker 350 included in the image data acquired from the photographing device 120 will be described.

[0037] FIG. 4 is a flowchart showing an operation example of the object tracking device 100 according to the first embodiment. The processing of the flowchart in FIG. 4 starts, for example, when an object tracking program is executed by a user who supervises a manufacturing site. In the following specific example, after the processing regarding the image data at time t−1 is executed by the object tracking program, processing of the image data at time t will be described.

[0038] (Step ST110) The image data acquisition unit 210 acquires the image data at time t. Hereinafter, an example of the image data at time t will be described with reference to FIG. 5.

[0039] FIG. 5 is a diagram illustrating image data 500 at time t in the first embodiment. In the image data 500 at time t, a conveyed object 530 is arranged at the conveyed object storage area 510, a conveyed object 540 is arranged in the passage 520, and a person 550 is shown working in the passage 520.

[0040] (Step ST120) After the image data at time t is acquired, the object detection unit 220 detects the position and type of an object from the image data at time t. Hereinafter, the object detection at time t will be described with reference to FIG. 6.

[0041] FIG. 6 is an explanatory diagram regarding the object detection at time t in the first embodiment. In FIG. 6, a conveyed object storage area 611 and three object detection positions 631, 641, and 651 are shown on the image data 500 at time t. In the image data 500 at time t, the upper left corner is the origin, the right direction from the origin is the X axis, and the downward direction from the origin is the Y axis. This will be the same hereinafter.

[0042] The object detection unit 220 generates an object detection result at time t by detecting the position and type of an object from the image data 500 at time t. The object detection result at time t includes, for example, three object detection positions 631, 641, and 651. The object detection position 631 is a rectangular area that substantially encloses the conveyed object 530. The object detection position 641 is a rectangular area that substantially encloses the conveyed object 540. The object detection position 651 is a rectangular area that substantially encloses the person 550.

[0043] The object detection result at time t can be expressed as D(t) = (d(t), c) for each detected object, for example. d(t) represents the object detection position (rectangular region) at time t. The object detection position is represented by, for example, "x1, y1, x2, y2". Here, the combination of "x1, y1" indicates the upper left coordinates of the rectangular region, and the combination of "x2, y2" indicates the lower right coordinates of the rectangular region. This way of representing the rectangular region will be the same hereinafter. c represents the type of the object. The types of objects are, for example, conveyances and people. The object detection unit 220 outputs the object detection result D(t) = (d(t), c) for each object detected at time t. Hereinafter, the object detection result at time t will be described with reference to FIG. 7.

[0044] FIG. 7 is a table 700 illustrating the object detection result at time t in the first embodiment. The table 700 associates the object detection position with the type of the object. Specifically, in the table 700, the object detection position "x11, y11, x12, y12" is associated with the object type "conveyance 1". Also, the object detection position "x21, y21, x22, y22" is associated with the object type "conveyance 2". Further, the object detection position "x41, y41, x42, y42" is associated with the object type "person". Note that in the object detection result at time t, conveyance 1 corresponds to conveyance 530, and conveyance 2 corresponds to conveyance 540.

[0045] Note that the object detection unit 220 may detect the conveyance storage area 611 from the image data 500 at time t.

[0046] (Step ST130) After the position and type of the object are detected from the image data at time t, the prediction model selection unit 240 selects a prediction model based on the object tracking result at time t - 1. Specifically, the prediction model selection unit 240 selects a prediction model for predicting the position of the object at time t from the model database based on the object tracking result at time t - 1. Hereinafter, the object determination position included in the object tracking result at time t - 1, the object tracking result at time t - 1, and the model database will be described with reference to FIGS. 8, 9, and 10.

[0047] FIG. 8 is an explanatory diagram regarding the object determination position at time t - 1 in the first embodiment. FIG. 8 shows the image data 800 at time t - 1. In the image data 800 at time t - 1, a carrier 830 is arranged in the carrier storage area 810, a carrier 840 is arranged in the passage 820, and a person 850 is shown working in the passage 820. Further, in FIG. 8, a carrier storage area 811 and three object determination positions 831, 841, 851 are shown on the image data 800 at time t - 1.

[0048] The object tracking result at time t - 1 includes, for example, three object determination positions 831, 841, 851. The object determination position 831 is a rectangular area that substantially encloses the carrier 830. The object determination position 841 is a rectangular area that substantially encloses the carrier 840. The object determination position 851 is a rectangular area that substantially encloses the person 850.

[0049] The object tracking result at time t - 1 can be expressed as, for example, T(t - 1)=(s(t - 1), c) for each object whose position is determined. s(t - 1) represents the object determination position (rectangular area) at time t - 1.

[0050] Figure 9 is a table 900 illustrating the object tracking result at time t-1 in the first embodiment. Table 900 associates the object's determined position with the type of the object. Specifically, in table 900, the object's determined position "x11,y11,x12,y12" is associated with the object type "Carrier 1". Also, the object's determined position "x21,y21,x22,y22" is associated with the object type "Carrier 2". Further, the object's determined position "x31,y31,x32,y32" is associated with the object type "Person". Note that in the object tracking result at time t-1, Carrier 1 corresponds to Carrier 830, and Carrier 2 corresponds to Carrier 840.

[0051] Figure 10 is a table 1000 illustrating the model database in the first embodiment. Table 1000 associates the type of the object, the location, and the prediction model. Specifically, in table 1000, the object type "Person", the location "Passage / Carrier Storage Area", and the prediction model "Kalman Filter" are associated. Also, the object type "Carrier", the location "Passage", and the prediction model "Kalman Filter" are associated. Further, the object type "Carrier", the location "Carrier Storage Area", and the prediction model "No Prediction" are associated.

[0052] Note that the object type "Carrier" does not indicate a specific carrier, and both of the aforementioned Carrier 1 and Carrier 2 are applicable. Also, the location "Passage / Carrier Storage Area" includes both the passage and the carrier storage area.

[0053] For example, the prediction model selection unit 240 selects a prediction model from the model database (Table 1000) based on the object tracking result (Table 900) at time t-1. Focusing on the object type "Carry Object 1", the prediction model selection unit 240 identifies that the object type "Carry Object 1" is placed at the location "Carry Object Storage Area" based on the object's determined position "x11, y11, x12, y12", and selects the prediction model "No Prediction". Also, focusing on the object type "Carry Object 2", the prediction model selection unit 240 identifies that the object type "Carry Object 2" is placed at the location "Passageway" based on the object's determined position "x21, y21, x22, y22", and selects the prediction model "Kalman Filter". Further, focusing on the object type "Person", the prediction model selection unit 240 identifies that the object type "Person" is working at the location "Passageway" based on the object's determined position "x31, y31, x32, y32", and selects the prediction model "Kalman Filter".

[0054] (Step ST140) After the prediction model is selected, the object prediction unit 250 predicts the position of the object at time t based on the selected prediction model and the continuous tracking result at time t-1. Hereinafter, the continuous tracking result at time t-1 will be described with reference to FIG. 11.

[0055] FIG. 11 is an explanatory diagram regarding the movement trajectory at time t-1 in the first embodiment. In FIG. 11, from the object tracking result 1100-1 at time 1 to the object tracking result 1100-(t-1) at time t-1 are cumulatively shown in time series in the time axis direction perpendicular to the X-axis and Y-axis. The object tracking result 1100-1 at time 1 includes three object determined positions 1131, 1141, and 1151. The object determined position 1131 corresponds to Carry Object 1, the object determined position 1141 corresponds to Carry Object 2, and the object determined position 1151 corresponds to a person. Note that the movement trajectory may also be called a tracklet.

[0056] In addition, as shown in FIG. 11, three movement trajectories 1132, 1142, and 1152 are shown. The movement trajectory 1132 is obtained by cumulatively arranging in time series the object determination positions at each time from the object determination position 1131 at time 1 to the object tracking result 1100-(t-1) at time t-1 for each object. Similarly, the movement trajectory 1142 starts from the object determination position 1141, and the movement trajectory 1152 starts from the object determination position 1151.

[0057] The movement trajectory at time t-1 can be represented, for example, for each object, as an array {s(1), …, s(t-1)} obtained by cumulatively arranging in time series the object determination positions at each time. In addition, the continuous tracking result at time t-1 can be represented, for each object, as CT(t-1) = ({s(1), …, s(t-1)}, c).

[0058] For example, when focusing on the transported object 1, since the object prediction unit 250 has the prediction model "no prediction", the object determination position at time t-1 is determined as the object prediction position at time t, and a position prediction result at time t is generated. When focusing on the transported object 2, the object prediction unit 250 predicts the object prediction position at time t based on the prediction model "Kalman filter" and the continuous tracking result at time t-1 (which includes the movement trajectory 1142), and generates a position prediction result at time t. When focusing on a person, the object prediction unit 250 predicts the object prediction position at time t based on the prediction model "Kalman filter" and the continuous tracking result at time t-1 (which includes the movement trajectory 1152), and generates a position prediction result at time t.

[0059] The position prediction result at time t can be represented, for example, for each object, as P(t) = (p(t), c). p(t) represents the object prediction position (rectangular area) at time t. Hereinafter, the position prediction result at time t will be described with reference to FIG. 12.

[0060] FIG. 12 is a table 1200 illustrating the position prediction results at time t in the first embodiment. Table 1200 associates the predicted object positions with the types of objects. Specifically, in table 1200, the object predicted position "x11, y11, x12, y12" is associated with the object type "Carrier 1". Also, the object predicted position "x21, y21, x22, y22" is associated with the object type "Carrier 2". Further, the object predicted position "x51, y51, x52, y52" is associated with the object type "Person".

[0061] For Carrier 1, since the object prediction unit 250 does not perform prediction using the prediction model, the object predicted position at time t is the same as the object determined position at time t - 1. Also, for Carrier 2, although the object prediction unit 250 performs prediction using the prediction model, since Carrier 2 is not moved (is stationary), the object predicted position at time t has not changed from the object determined position at time t - 1.

[0062] (Step ST150) After the position of the object at time t is predicted, the association unit 260 generates an association result based on the object detection result at time t and the position prediction result at time t. Hereinafter, the process of step ST150 is referred to as "association process". Below, a specific example of the association process will be described using the flowchart of FIG. 13.

[0063] FIG. 13 is a flowchart showing a specific example of the association process of FIG. 4. The flowchart of FIG. 13 explains the details of the process of step ST150 of FIG. 4.

[0064] (Step ST151) After the position of the object at time t is predicted, the association unit 260 calculates the IoU for all combinations of the object detection result at time t and the position prediction result at time t.

[0065] The object detection result at time t for all objects can be represented, for example, as D(t)(i). Here, i (= 1, …, N(t)) represents the serial number of the object detected at time t, and N(t) represents the total number of objects detected at time t.

[0066] The position prediction result at time t for all objects can be represented, for example, as P(t)(j). Here, j (= 1, …, M(t)) represents the serial number of the position predicted at time t, and M(t) represents the total number of positions predicted at time t.

[0067] IoU is obtained by normalizing the intersection area by dividing it by the total area for two regions. Therefore, the larger the value of IoU, the more the two regions overlap, and the smaller the value of IoU, the less the two regions overlap. Hereinafter, IoU will be described with reference to FIG. 14.

[0068] FIG. 14 is an explanatory diagram regarding IoU in the first embodiment. In FIG. 14(a), an object detection position 1411 and an object prediction position 1421 are shown. In FIG. 14(b), an intersection area 1431 where the object detection position 1411 and the object prediction position 1421 overlap is shown. In FIG. 14(c), a total area 1432 of the object detection position 1411 and the object prediction position 1421 is shown. For example, the association unit 260 calculates IoU based on the intersection area 1431 and the total area 1432.

[0069] (Step ST152) The association unit 260 associates the object detection position with the object prediction position based on the combination and IoU. Specifically, the association unit 260 uses the Hungarian algorithm to associate the object detection position with the object prediction position from all combinations of the object detection result at time t and the position prediction result at time t, with the value of IoU as the cost.

[0070] (Step ST153) The association unit 260 generates an association result including information on combinations of associated positions and information on unassociated positions. Hereinafter, the information on combinations of associated positions will be described with reference to FIG. 15.

[0071] FIG. 15 is a table 1500 exemplifying information on combinations of associated positions in the first embodiment. Table 1500 associates an object detection position, an object prediction position, and the type of the object. Specifically, in table 1500, the object detection position “x11, y11, x12, y12”, the object prediction position “x11, y11, x12, y12”, and the object type “carrier 1” are associated. Also, the object detection position “x21, y21, x22, y22”, the object prediction position “x21, y21, x22, y22”, and the object type “carrier 2” are associated. Also, the object detection position “x41, y41, x42, y42”, the object prediction position “x51, y51x52, y52”, and the object type “person” are associated.

[0072] Examples of the information on unassociated positions include (1) the case where there is no object prediction position associated with the object detection position, or (2) the case where there is no object detection position associated with the object prediction position. In the case of (1) above, for example, it is assumed that a new person walks into the screen. Also, in the case of (2) above, for example, it is assumed that occlusion has occurred for the object to be detected.

[0073] The information on combinations of associated positions at time t can be represented, for example, as C(t) = (D(t)(I(k)), P(t)(J(k))). k (= 1, …, L(t)) represents the serial number of the combination associated with time t. L(t) represents the total number of combinations associated with time t. I(k) represents the serial number of the detected object corresponding to k. J(k) represents the serial number of the predicted position corresponding to k.

[0074] The information of the positions not associated at time t can be expressed, for example, in the case of object detection results, as NCD(t) = D(t)(i (= not in I(k)). Here, i (= not in I(k)) represents the numbers of the objects detected at time t that are not included in I(k), that is, the numbers of the objects that were not associated.

[0075] Also, for example, the information of the positions not associated at time t can be expressed, in the case of position prediction results, as NCP(t) = P(t)(j (= not in J(k)). Here, j (= not in J(k)) represents the numbers of the positions predicted at time t that are not included in J(k), that is, the numbers of the positions that were not associated.

[0076] (Step ST160) After the association result is generated, the tracking processing unit 270 generates a continuous tracking result at time t based on the association result. Hereinafter, the processing of step ST160 is referred to as "continuous tracking result generation processing". Hereinafter, a specific example of the continuous tracking result generation processing will be described with reference to the flowchart of FIG. 16.

[0077] FIG. 16 is a flowchart showing a specific example of the continuous tracking result generation processing of FIG. 4. The flowchart of FIG. 16 explains the details of the processing of step ST160 of FIG. 4.

[0078] (Step ST161) After the association result is generated, the tracking processing unit 270 determines the type of the association result. If the association result is information on the combination of the associated positions, it is determined that there is an association, and the process proceeds to step ST162. If the association result is information on the positions not associated with the object detection result, it is determined that there is only the object detection position, and the process proceeds to step ST165. If the association result is information on the positions not associated with the position prediction result, it is determined that there is only the object prediction position, and the process proceeds to step ST167.

[0079] (Step ST162) After determining that there is an association, the tracking processing unit 270 calculates an object determination position based on the object detection position and the object prediction position. Specifically, the tracking processing unit 270 calculates the object determination position from the object detection position and the object prediction position using the method described in the aforementioned reference 2 or the like.

[0080] (Step ST163) The tracking processing unit 270 generates an object tracking result at time t having the object determination position. The object tracking result at time t can be expressed as, for example, T(t) = (s(t), c) for each object. s(t) represents the object determination position (rectangular region) at time t.

[0081] FIG. 17 is a table 1700 illustrating the object tracking result at time t in the first embodiment. Specifically, in the table 1700, the object determination position "x11, y11, x12, y12" and the object type "carrier 1" are associated. Also, the object determination position "x21, y21, x22, y22" and the object type "carrier 2" are associated. Further, the object determination position "x61, y61, x62, y62" and the object type "person" are associated.

[0082] For carrier 1, the object detection position at time t and the object prediction position at time t indicate the same rectangular region, and the object determination position is also the same rectangular region. This is the same for carrier 2. For a person, since the object detection position at time t and the object prediction position at time t indicate different rectangular regions, the object determination position is also a different rectangular region from them.

[0083] (Step ST164) The tracking processing unit 270 adds the object tracking result at time t to the continuous tracking result at time t - 1 and generates a continuous tracking result at time t. The continuous tracking result at time t can be expressed as, for example, CT(t) = ({s(1),..., s(t - 1), s(t)}, c) for each object. After step ST164, the object tracking program ends.

[0084] (Step ST165) After it is determined that only the object detection position exists, the tracking processing unit 270 generates an object tracking result at time t having the object detection position. Specifically, the tracking processing unit 270 replaces the object detection position with the object determination position and generates an object tracking result at time t. In step ST165, the object tracking result at time t can be expressed as, for example, for each object, T(t) = (d(t), c) = (s(t), c).

[0085] (Step ST166) The tracking processing unit 270 generates a continuous tracking result at time t starting from the object tracking result at time t. In step ST166, the continuous tracking result at time t can be expressed as, for example, for each object, CT(t) = ({s(t)}, c). After step ST166, the object tracking program ends.

[0086] (Step ST167) After it is determined that only the object prediction position exists, the tracking processing unit 270 generates an object tracking result at time t having the object prediction position. Specifically, the tracking processing unit 270 replaces the object prediction position with the object determination position and generates an object tracking result at time t. In step ST167, the object tracking result at time t can be expressed as, for example, for each object, T(t) = (p(t), c) = (s(t), c). After step ST167, the process proceeds to step ST164.

[0087] Note that after step ST164 and step ST166, 1 can be incremented at time t, and the process can return to step ST110 to repeat the same process. Also, the object tracking program may end according to an instruction by the user.

[0088] As described above, the object tracking device according to the first embodiment acquires image data at a first time (for example, time t), detects the position and type of an object from the image data at the first time, and thereby generates an object detection result at the first time in which the detected position and type of the object are associated with each other. Based on an object tracking result at a second time (for example, time t−1) determined in the past relative to the first time, in which the position and type of the object are associated with each other, a prediction model for predicting the position of the object at the first time is selected. The position of the object at the first time is predicted based on the selected prediction model and a time-series continuous tracking result including the object tracking result at the second time, thereby generating a position prediction result at the first time in which the predicted position and type of the object are associated with each other. An association result is generated by performing an association between the object detection result at the first time and the position prediction result at the first time, and an object tracking result at the first time is generated based on the association result.

[0089] Therefore, the object tracking device according to the first embodiment can improve the tracking accuracy of an object by selecting a prediction model at the current time based on the object's determined position at one time before.

[0090] (Second Embodiment) The object tracking device according to the first embodiment selects a prediction model according to the type and location of an object. On the other hand, the object tracking device according to the second embodiment selects a prediction model according to the type, location, and proximity object of the object.

[0091] The object tracking device according to the second embodiment is different from the object tracking device according to the first embodiment in terms of the model database, the processing in the prediction model selection unit 240, and the processing in the object prediction unit 250. Hereinafter, the differences from the first embodiment will be described.

[0092] In the second embodiment, the model DB 230 stores by associating the type of an object, the location where the object exists, neighboring objects, and a prediction model. The model DB outputs information on the prediction model associated with the type, location, and neighboring objects of the object in response to a read request from the prediction model selection unit 240. Note that the model DB 230 may store a table in which the type of the object, the location, the neighboring objects, and the prediction model are associated. Also, the information on the prediction model may be information indicating that prediction using the prediction model is not performed.

[0093] The information on neighboring objects is, for example, information about other objects approaching the object of interest. The neighboring objects are, for example, persons, and it is assumed that a stationary object (e.g., a conveyance) actively approaches a person. In this case, a conveyance that is normally stopped may be moved by a person.

[0094] FIG. 18 is a table 1800 illustrating the model database in the second embodiment. The table 1800 associates the type of an object, the location, neighboring objects, and a prediction model. Specifically, in the table 1800, the object type "person", the location "passage / conveyance storage area", no neighboring objects, and the prediction model "Kalman filter" are associated. Also, the object type "conveyance", the location "passage", no neighboring objects, and the prediction model "Kalman filter" are associated. Also, the object type "conveyance", the location "passage", the neighboring object "person", and the prediction model "the same prediction model as that of the person" are associated. Also, the object type "conveyance", the location "conveyance storage area", no neighboring objects, and the prediction model "no prediction" are associated. Also, the object type "conveyance", the location "conveyance storage area", the neighboring object "person", and the prediction model "the same prediction model as that of the person" are associated.

[0095] FIG. 19 is an explanatory diagram illustrating a state where a person approaches a conveyed object in the second embodiment. Image data 1900 is shown in FIG. 19. In the image data 1900, a conveyed object 1930 is arranged in a conveyed object storage area 1910, a conveyed object 1940 is arranged in a passage 1920, and a state where a person 1950 is working in the passage 1920 is shown. Also, in FIG. 19, a conveyed object storage area 1911 and three object determination positions 1931, 1941, and 1951 are shown on the image data 1900.

[0096] The object tracking result regarding the image data 1900 includes, for example, three object determination positions 1931, 1941, and 1951. The object determination position 1931 is a rectangular area that substantially encloses the conveyed object 1930. The object determination position 1941 is a rectangular area that substantially encloses the conveyed object 1940. The object determination position 1951 is a rectangular area that substantially encloses the person 1950.

[0097] A part of the area of the object determination position 1941 and the object determination position 1951 overlaps. This overlap occurs, for example, when the person 1950 approaches the conveyed object 1940. The determination process for a proximity object is executed by, for example, a prediction model selection unit 240. Specifically, the prediction model selection unit 240 calculates an evaluation value by evaluating the overlap between the object determination position 1941 for the conveyed object 1940 and the object determination position 1951 for the person 1950, and determines that the person 1950 is a proximity object when the evaluation value is equal to or greater than a threshold value. Incidentally, when a plurality of persons approach a conveyed object, the prediction model selection unit 240 may select the person with the largest evaluation value. Also, since conveyed objects may be arranged close to each other, the prediction model selection unit 240 may be configured not to recognize a conveyed object as a proximity object.

[0098] Next, the processing of the object prediction unit 250 for the transported object 1940 will be described. When the prediction model "the same prediction model as for a person" is selected by the prediction model selection unit 240 for the transported object 1940, the object prediction unit 250 generates a position prediction result using, as the object prediction position for the transported object 1940, the object prediction position predicted based on the continuous tracking result for the person 1950. By performing such processing, even when the transported object 1940 is moved by the person 1950, the object tracking device 100 can accurately perform position prediction for the transported object 1940.

[0099] As described above, the object tracking device according to the second embodiment can select a prediction model according to the type, location, and neighboring objects of the object. Therefore, even when a stopped tracked object moves passively, the object tracking device according to the second embodiment can perform position prediction using an appropriate prediction model.

[0100] Therefore, similar to the object tracking device according to the first embodiment, the object tracking device according to the second embodiment can improve the tracking accuracy of the object.

[0101] (Hardware Configuration) FIG. 20 is a block diagram illustrating the hardware configuration of a computer according to an embodiment. The computer 2000 includes, as hardware, a CPU (Central Processing Unit) 2010, a RAM (Random Access Memory) 2020, a program memory 2030, an auxiliary storage device 2040, and an input / output interface 2050. The CPU 2010 communicates with the RAM 2020, the program memory 2030, the auxiliary storage device 2040, and the input / output interface 2050 via a bus 2060.

[0102] The CPU 2010 is an example of a general-purpose processor. The RAM 2020 is used by the CPU 2010 as a working memory. The RAM 2020 includes a volatile memory such as SDRAM (Synchronous Dynamic Random Access Memory). The program memory 2030 stores various programs including an object tracking program. As the program memory 2030, for example, a ROM (Read-Only Memory), a part of the auxiliary storage device 2040, or a combination thereof is used. The auxiliary storage device 2040 stores data non-temporarily. The auxiliary storage device 2040 includes a non-volatile memory such as an HDD or an SSD.

[0103] The input / output interface 2050 is an interface for connecting to or communicating with other devices. The input / output interface 2050 is used, for example, for connecting to or communicating with the output device 110 and the imaging device 120 shown in FIG. 1. Also, for example, the input / output interface 2050 may be used for connecting to or communicating with a storage device and an external device not shown in FIG. 1.

[0104] Each program stored in the program memory 2030 includes computer-executable instructions. When the program (computer-executable instructions) is executed by the CPU 2010, it causes the CPU 2010 to execute a predetermined process. For example, when the object tracking program is executed by the CPU 2010, it causes the CPU 2010 to execute a series of processes described with respect to each step of FIGS. 4, 13, and 16.

[0105] The program may be provided to the computer 2000 in a state stored in a computer-readable storage medium. In this case, for example, the computer 2000 further includes a drive (not shown) for reading data from the storage medium and acquires the program from the storage medium. Examples of the storage medium include magnetic disks, optical disks (such as CD-ROM, CD-R, DVD-ROM, DVD-R, etc.), magneto-optical disks (such as MO), and semiconductor memories. Also, the program may be stored in a server on a communication network, and the computer 2000 may download the program from the server using the input / output interface 2050.

[0106] The processing described in the embodiments is not limited to being performed by a general-purpose hardware processor such as the CPU 2010 executing a program, and may be performed by a dedicated hardware processor such as an ASIC (Application Specific Integrated Circuit). The term processing circuit (processing unit) includes at least one general-purpose hardware processor, at least one dedicated hardware processor, or a combination of at least one general-purpose hardware processor and at least one dedicated hardware processor. In the example shown in FIG. 20, the CPU 2010, the RAM 2020, and the program memory 2030 correspond to the processing circuit.

[0107] Therefore, according to each of the above embodiments, the tracking accuracy of the object can be improved.

[0108] Although some embodiments of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other forms, and various omissions, replacements, and changes can be made without departing from the gist of the invention. These embodiments and their modifications are included in the scope and gist of the invention, and are included in the invention described in the claims and its equivalent scope.

Description of Reference Numerals

[0109] 1…Object tracking system, 100…Object tracking device, 110…Output device, 120…Imaging device, 210…Image data acquisition unit, 220…Object detection unit, 240…Prediction model selection unit, 250…Object prediction unit, 260…Association unit, 270…Tracking processing unit, 310, 510, 810, 1910…Conveyor storage area, 320, 520, 820, 1920…Passageway, 330, 340, 530, 540, 830, 840, 1930, 1940…Conveyor, 350…Operator, 500…Image data, 550…Person, 611, 811, 1911…Conveyor storage area, 631, 641, 651…Object detection position, 700, 900, 1000, 1200, 1500, 1700, 1800…Table, 800, 1900…Image data, 831, 841, 851, 1131, 1141, 1151, 1931, 1941, 1951…Object determination position, 850…Person, 1100-1, 1100-(t-1)…Object tracking result, 1132, 1142, 1152…Movement trajectory, 1411…Object detection position, 1421…Object prediction position, 1431…Intersection area, 1432…Overall area, 1950…Person, 2000…Computer, 2030…Program memory, 2040…Auxiliary storage device, 2050…Input / output interface, 2060…Bus, PA…Imaging area, WA…Working area.

Claims

1. A computer, image data acquisition means for acquiring image data at a first time, object detection means for generating an object detection result at the first time in which the detected position and type of the object are associated by detecting the position and type of the object from the image data at the first time, prediction model selection means for selecting a prediction model for predicting the position of the object at the first time based on the object tracking result at a second time in the past from the first time, in which the position and type of the object are associated, object prediction means for generating a position prediction result at the first time in which the predicted position and type of the object are associated by predicting the position of the object at the first time based on the selected prediction model and a time-series continuous tracking result including the object tracking result at the second time, association means for generating an association result by performing an association between the object detection result at the first time and the position prediction result at the first time, tracking processing means for generating the object tracking result at the first time based on the association result An object tracking program for causing the computer to function as the above.

2. The prediction model selection means selects the prediction model based on the object tracking result at the second time and a model database in which the type of the object, the location where the object exists, and the prediction model are associated. The object tracking program according to claim 1.

3. The selected prediction model is a prediction using a Kalman filter. The object tracking program according to claim 1.

4. When the selected prediction model has no prediction, the object prediction means determines the object tracking result at the second time as the position prediction result at the first time. The object tracking program according to claim 1.

5. The association means calculates an evaluation value by evaluating the overlap in all combinations of the object detection result at the first time and the position prediction result at the first time, and uses the evaluation value to perform an assignment between the object detection result and the position prediction result to generate the association result. The object tracking program according to claim 1.

6. The association means calculates the evaluation value using at least one of IoU (Intersection over Union) and the similarity of image features. The object tracking program according to claim 5.

7. The association means performs the assignment using the Hungarian algorithm and generates the association result. The object tracking program according to claim 5.

8. The association result includes information on combinations of associated positions and information on unassociated positions. The object tracking program according to claim 1.

9. The prediction model selection means selects the prediction model based on the object tracking result at the second time and a model database in which the type of object, the location where the object exists, neighboring objects, and the prediction model are associated. The object tracking program according to claim 1.

10. When there are neighboring objects adjacent to the object in the object tracking result at the second time, the prediction model selection means selects the same prediction model as that of the neighboring objects. The object prediction means sets the position predicted for the neighboring object as the position predicted for the object in the object tracking result at the second time. The object tracking program according to claim 9.

11. An image data acquisition unit that acquires image data at a first time; An object detection unit that generates the object detection result at the first time in which the detected position and type of the object are associated by detecting the position and type of the object from the image data at the first time; A prediction model selection unit that selects a prediction model for predicting the position of the object at the first time based on the object tracking result at the second time in which the position and type of the object determined at a second time earlier than the first time are associated; An object prediction unit that generates the position prediction result at the first time in which the predicted position and type of the object are associated by predicting the position of the object at the first time based on the selected prediction model and a time-series continuous tracking result including the object tracking result at the second time; An association unit that generates an association result by performing an association between the object detection result at the first time and the position prediction result at the first time; A tracking processing unit that generates the object tracking result at the first time based on the association result An object tracking device comprising:

12. The image data acquisition unit acquires image data at a first time; The object detection unit generates the object detection result at the first time in which the detected position and type of the object are associated by detecting the position and type of the object from the image data at the first time; The prediction model selection unit selects a prediction model for predicting the position of the object at the first time based on the object tracking result at the second time in the past from the first time, in which the position and type of the object determined at the second time are associated with each other. The object prediction unit generates a position prediction result at the first time in which the predicted position and type of the object are associated with each other by predicting the position of the object at the first time based on the selected prediction model and the time-series continuous tracking result including the object tracking result at the second time. The association unit generates an association result by performing an association between the object detection result at the first time and the position prediction result at the first time. The tracking processing unit generates an object tracking result at the first time based on the association result. An object tracking method comprising the above.

Images