Moving object exploration device

Abstract

To provide a mobile object search device configured to prevent similar mobile objects from being falsely associated as an identical mobile object.SOLUTION: An execution apparatus acquires a plurality of frames in time series. The execution apparatus searches for a mobile object OB from the frames by using a learning model. The execution apparatus estimates a movement destination of the mobile object in a new frame, on the basis of movement regulation of the same mobile object within past multiple frames. The execution apparatus sets, as a priority search range AR, a range which includes the estimated movement destination and is narrower than the entire range of the frame. In associating a mobile object OB identical with the mobile object OB searched for in the past frames from the new frame, a mobile object OB searched for from within the priority search range AR is associated.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to a moving object search device. [Background technology]

[0002] Patent Document 1 describes a moving object search device. The moving object search device acquires a video captured by an imaging device as a time series of frames. In each frame, the moving object search device searches for moving objects such as people and vehicles using a learning model that has been trained by machine learning. The moving object search device then associates the same moving object between different frames. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2017-228303 Summary of the Invention [Problem to be solved by the invention]

[0004] Suppose a specific moving object is searched for in a specific frame acquired by a moving object search device such as that described in Patent Document 1. Then, suppose that the specific moving object is hidden by another object and does not appear in the next frame in time series relative to the specific frame. In this case, if a moving object similar to the specific moving object is searched for in the next frame, the moving object search device may erroneously associate this similar moving object with the specific moving object. [Means for solving the problem]

[0005] In order to solve the above problem, the present invention is a moving object search device that includes an execution device and a storage device, wherein the storage device stores a learning model for searching for moving objects that has been learned by machine learning, and the execution device is capable of acquiring a plurality of frames in chronological order, searching for the moving object in each of the frames using the learning model, matching the same moving object between different frames, estimating a destination of the moving object in a new frame based on a movement trajectory of the same moving object in a plurality of past frames, setting a range that includes the estimated destination and is narrower than the entire range of the frame as a priority search range, and when matching the moving object in the new frame that is the same as the moving object searched in the past frame, matching the moving object searched within the priority search range.

[0006] According to the above configuration, when associating a moving object in a new frame that is identical to a moving object searched for in a previous frame, the execution device associates the moving object searched for within the priority search range. Therefore, when a moving object similar to a moving object searched for in a previous frame appears outside the priority search range, it is not necessary to associate the similar moving object with the identical moving object. [Brief explanation of the drawings]

[0007] [Figure 1] FIG. 1 is a schematic diagram showing an outline of a vehicle. [Figure 2] FIG. 2 is an explanatory diagram showing the processing contents for a plurality of frames. [Figure 3] FIG. 3 is an explanatory diagram illustrating the size of the priority search range. DETAILED DESCRIPTION OF THE INVENTION

[0008] (One embodiment) An embodiment of a moving object search device will be described below, along with a vehicle equipped with the moving object search device, with reference to the drawings.

[0009] <Vehicle Overview> As shown in Fig. 1, vehicle 10 includes camera 20 and moving object search device 30. Camera 20 detects video ahead of vehicle 10. More specifically, camera 20 is attached to the windshield or the like of vehicle 10. Camera 20 captures video ahead of vehicle 10 through the windshield. The video captured by camera 20 may capture other vehicles traveling ahead of vehicle 10, pedestrians walking around in front of vehicle 10, and the like.

[0010] The moving object search device 30 includes an execution device 31 and a storage device 32. The execution device 31 is a CPU. The storage device 32 is a ROM and a RAM. The execution device 31 acquires a moving image detected by the camera 20 as a plurality of frames FL in time series. Therefore, each frame FL is a still image.

[0011] The storage device 32 stores a search program PR for searching for a moving object OB in a plurality of frames FL. Note that the moving object OB here includes not only an object that is actually moving, but also an object that is moving in consecutive frames FL.

[0012] The storage device 32 stores a learning model LM for searching for a moving object OB that has been learned in advance by machine learning. The learning model LM is, for example, DeepSORT. More details about DeepSORT are described in arXiv:1703.07402.

[0013] <Processing by the moving object search device> The execution device 31 is capable of executing the search program PR stored in the storage device 32. When the execution device 31 acquires a frame FL from the camera 20, it starts executing the search program PR.

[0014] As shown in FIG. 2, when the execution device 31 starts executing the search program PR, the execution device 31 first searches for a moving object OB in each frame FL using the learning model LM.

[0015] The multiple frames FL will be referred to in chronological order from oldest to newest as follows: First, assume that the times t are arranged at equal intervals in the order of time t1, time t2, time t3, and time t4. The frame at time t1 will be referred to as the first frame FL1. The frame at time t2 will be referred to as the second frame FL2. The frame at time t3 will be referred to as the third frame FL3. The frame at time t4 will be referred to as the fourth frame FL4.

[0016] When the execution device 31 acquires the first frame FL1, it searches for a moving object OB in the first frame FL1 using the learning model LM. Specifically, the execution device 31 assigns an individual ID to each of the searched moving objects OB. For example, the execution device 31 assigns "1" as the ID to a specific searched moving object OB.

[0017] Next, when the execution device 31 acquires the second frame FL2, it searches for a moving object OB in the second frame FL2 using the learning model LM. Furthermore, the execution device 31 associates the same moving object OB between different frames FL. Specifically, the execution device 31 assigns the same ID as the ID assigned in the first frame FL1 to the moving object OB searched for in the second frame FL2 that is the same as the moving object OB searched for in the first frame FL1. For example, the execution device 31 assigns "1" as the ID to the moving object OB that is the same as the moving object OB assigned the ID "1" in the first frame FL1. Note that the determination of whether the moving objects OB are the same or not is performed, for example, based on the similarity of the images showing the searched moving objects OB.

[0018] The execution unit 31 calculates a motion vector, which is the motion trajectory of the same moving object OB in multiple frames FL. Specifically, the execution unit 31 calculates the motion vector VE1 of the moving object OB from the first frame FL1 to the second frame FL2 as follows: The magnitude of the motion vector VE1 is set to the length of a line segment connecting the geometric centers of the image ranges showing the same moving object OB found in the first frame FL1 and the second frame FL2. Next, the execution unit 31 sets the starting point of the motion vector VE1 to the geometric center of the moving object OB in the first frame FL1.

[0019] The execution unit 31 estimates the destination of the moving object OB in a new frame FL based on the movement vectors in multiple past frames FL. Specifically, when searching for the moving object OB in the third frame FL3, the execution unit 31 translates the starting point of the movement vector VE1 calculated in the first frame FL1 and the second frame FL2 to the geometric center of the moving object OB in the second frame FL2. As a result, the execution unit 31 estimates the translated vector as the movement vector VE2. The execution unit 31 then estimates the end point of the movement vector VE2 in the third frame FL3 as the destination of the moving object OB in the third frame FL3.

[0020] The execution unit 31 sets a range that includes the estimated destination and is smaller than the entire range of frame FL as the priority search range AR. Specifically, when setting the priority search range AR in the third frame FL3, the execution unit 31 sets the priority search range AR to include the end point of the movement vector VE2. Furthermore, the execution unit 31 sets the range of the priority search range AR to be smaller than the entire range of frame FL.

[0021] Then, when searching for a moving object OB that is the same as the moving object OB searched for in the past frame FL within the new frame FL, the execution device 31 searches within the priority search range AR.

[0022] Specifically, the execution device 31 searches for a moving object OB with an ID of "1" that was searched for in the first frame FL1 and the second frame FL2 in the third frame FL3. At this time, the execution device 31 searches for whether or not a moving object OB with an ID of "1" is present within the priority search range AR. If a moving object OB with an ID of "1" is present within the priority search range AR, the execution device 31 assigns an ID of "1" to the searched moving object OB. On the other hand, if a moving object OB with an ID of "1" is not present within the priority search range AR, the execution device 31 determines that there is no moving object OB with an ID of "1" in the third frame FL3. In other words, even if other moving objects OB exist outside the priority search range AR, the ID of "1" will not be assigned to the other moving objects OB.

[0023] When a moving object OB with an ID of "1" is present in the third frame FL3, the execution device 31 updates the movement vector VE2 and calculates a movement vector VE3 from the past second frame FL2 and third frame FL3. Next, similar to when searching the third frame FL3, the execution device 31 estimates the destination of the moving object OB in the fourth frame FL4 based on the calculated movement vector VE3. Next, when searching for the moving object OB with an ID of "1" in the fourth frame FL4 based on the estimated destination, the execution device 31 sets a priority search range AR for the fourth frame FL4. Then, when associating the moving object OB with an ID of "1" from the fourth frame FL4, the execution device 31 associates the moving object OB searched for within the set priority search range AR.

[0024] On the other hand, when a moving object OB with an ID of "1" is not present in the third frame FL3, the execution device 31 estimates a destination in the fourth frame FL4 based on the movement vector VE1 calculated in the past first and second frames FL1 and FL2. Specifically, the execution device 31 translates the movement vector VE2 so that the estimated destination in the third frame FL3, i.e., the end point of the movement vector VE2, becomes the start point of a new movement vector VE3, thereby setting the movement vector VE3. The execution device 31 then estimates the end point of the movement vector VE3 as the destination of the moving object OB in the fourth frame FL4. Next, the execution device 31 sets a priority search range AR in the fourth frame FL4 based on the estimated destination. Thereafter, in the same manner as when a moving object OB with an ID of "1" is present in the third frame FL3, the execution device 31 sets a priority search range AR and associates the moving object OB with an ID of "1" in the fourth frame FL4. The execution device 31 updates the movement vector VE3 and calculates the movement vector VE4 from the third frame FL3 and the fourth frame FL4.

[0025] Note that when the movement vectors VE2, VE3, and VE4 are calculated as described above, the end points of the movement vectors VE2, VE3, and VE4 may be located outside the range of the frame FL. In this case, the execution unit 31 handles it as if the moving object OB has moved outside the frame FL and the same moving object OB does not exist in the frame FL. In other words, the execution unit 31 does not assign the same ID to the moving object OB in the new frame FL as the moving object OB that existed in the previous frame FL.

[0026] Here, when setting the range of the priority search range AR, the execution device 31 particularly sets it as follows. 3, the execution device 31 sets a larger range for the priority search range AR the further the estimated moving destination of the moving object OB is from the vanishing point VP in the frame FL. Note that the vanishing point VP here is the vanishing point in perspective projection and perspective. In this embodiment, the vanishing point VP is the center point of the frame FL. Furthermore, the execution device 31 sets a larger range for the priority search range AR when the size of the moving object OB to be searched for is larger in the previous frame FL, compared to when the size is smaller.

[0027] Specifically, for example, suppose the execution device 31 estimates that the estimated destination of the moving object OB is point P1, point P2, and point P3, moving away from the vanishing point VP. Also, suppose the execution device 31 sets a priority search range AR1 when the estimated destination of the moving object OB is point P1. At this time, the execution device 31 sets the size of the priority search range AR2, which is set when the estimated destination of the moving object OB is point P2, to be larger than the priority search range AR1. Similarly, the execution device 31 sets the size of the priority search range AR3, which is set when the estimated destination of the moving object OB is point P3, to be larger than the priority search range AR2.

[0028] (Operation of the embodiment) According to the above embodiment, suppose that in a specific frame, for example, the third frame FL3, a moving object OB with an ID of "1" is hidden by another object and does not appear. In this case, even if a moving object OB similar to the moving object OB with an ID of "1" appears outside the priority search range AR in the fourth frame FL4, the execution device 31 does not associate the moving object OB that appears outside the priority search range AR with an ID of "1." On the other hand, if a moving object OB similar to the moving object OB with an ID of "1" appears within the priority search range AR in the fourth frame FL4, the execution device 31 assigns "1" to the ID of the moving object OB.

[0029] (Effects of the embodiment) (1) According to the above embodiment, after the same moving object OB disappears from a plurality of time-series frames FL, in a new frame FL, the execution device 31 associates the moving object OB searched for within the priority search range AR. Therefore, even if a similar moving object OB appears outside the priority search range AR of the new frame FL, it is not associated as the same moving object OB. This prevents the moving object search device 30 from erroneously associating the same moving object OB.

[0030] (2) According to the above embodiment, the execution device 31 sets the size of the priority search range AR to be larger the further away from the vanishing point VP. Since the moving object OB appears larger in the frame FL the further away from the vanishing point VP, the priority search range AR can be set to match the size of the moving object OB in the frame FL.

[0031] (Other embodiments) The above embodiment can be modified as follows: The above embodiment and the following modifications can be combined with each other within the scope of technical compatibility.

[0032] The execution device 31 may set the size of the priority search range AR regardless of the distance between the vanishing point VP and the destination. For example, the execution device 31 may set the size of the priority search range AR to be the same regardless of the distance between the vanishing point VP and the destination.

[0033] The execution device 31 may set the size of the priority search range AR regardless of the size of the moving object OB. For example, the execution device 31 may set the priority search range AR to the same size regardless of the size of the moving object OB.

[0034] The movement trajectory is not limited to one represented by a vector. For example, it may be a trajectory of curved movement. [Explanation of symbols]

[0035] 10... Vehicle 20... Camera 30... Moving object search device 31... Execution device 32... Storage device

Claims

[Claim 1] an execution device and a storage device; the storage device stores a learning model for searching for a moving object that has been learned by machine learning; The execution device acquiring multiple frames in time series; Searching for the moving object in each of the frames using the learning model; Associating the same moving object between different frames; estimating a destination of the moving object in the new frame based on a movement trajectory of the same moving object in the past plurality of frames; setting a range that includes the estimated destination and is narrower than the entire range of the frame as a priority search range; When associating the same moving object in the new frame as the moving object searched for in the past frame, associating the moving object searched for within the priority search range; is executable, When setting the priority search range, the execution device sets the priority search range to a larger range as the estimated destination is farther away from the vanishing point in the frame. Moving object search device.

Images