Tracking device, tracking method, and computer-readable recording medium

Abstract

The present disclosure relates to a tracking device, a tracking method, and a tracking computer program. The tracking device has: a candidate region detection unit (31) that detects each region in which a predetermined object is represented with a confidence level of a predetermined detection threshold or more as a candidate region by inputting an image generated by an imaging unit (2) to a recognizer; a tracking unit (32) that determines whether to establish correspondence between an arbitrary candidate region and the predetermined object based on an optical flow between an object region in which the predetermined object is represented on a past image generated by the imaging unit (2) earlier than the image and each candidate region, and increments a tracking count of the object by one in a case where the establishment of correspondence is possible; and a threshold control unit (33) that sets a detection threshold to be applied to a next image obtained by the imaging unit later than the image to a first value in a case where the tracking count is a predetermined count or less, and sets the detection threshold to be applied to the next image to a second value lower than the first value in a case where the tracking count is more than the predetermined count.

Description

TECHNICAL FIELD

[0001] The present application relates to a tracking device, a tracking method, and a tracking program for tracking an object appearing in an image. BACKGROUND

[0002] There is proposed a technique of tracking an object detected from a series of images generated in time series with a camera (see Japanese Patent Application Publication No. 2020-52695 and Japanese Patent Application Publication No. 2017-102824).

[0003] The object detection device disclosed in Japanese Patent Application Publication No. 2020-52695 tracks a first object detected in a sensor signal preceding a latest sensor signal among a plurality of sensor signals obtained by a sensor in time series, and detects a passing region through which the first object passes in the latest sensor signal. Further, the object detection device controls a confidence threshold value applied with respect to a confidence level regarding a second object appearing in each region in the latest sensor signal in accordance with whether the region is included in the passing region. Also, the object detection device detects the second object in a region in which the confidence level is equal to or higher than the confidence threshold value.

[0004] The tracking device disclosed in Japanese Patent Application Publication No. 2017-102824 uses a plurality of images to calculate an optical flow, and detects a position and a moving direction of a moving body on the basis of the calculated optical flow. In addition, the tracking device detects the position and the moving direction of the moving body on the basis of a plurality of overview images generated on the basis of the plurality of images. Further, the tracking device combines a detection result of the position and the moving direction of the moving body on the basis of the optical flow and a detection result of the position and the moving direction of the moving body on the basis of each overview image, and thereby detects the position and the moving direction of the moving body. Still further, the tracking device estimates a future position and a future moving direction of a tracking moving body to be tracked on the basis of each of the detection results. Also, the tracking device tracks the tracking moving body using any one of an estimated position of the tracking moving body estimated on the basis of the detection result, a position of the moving body detected on the basis of the overview image, and a position of the moving body detected on the basis of the combination of the detection results, and determines the position of the tracking moving body. SUMMARY

[0005] Depending on a relative positional relationship between a camera and an object in tracking, it is sometimes difficult to detect the object in an image generated at a certain time point in the object tracking. In such a case, a tracking device tracking the object sometimes fails in detection of a region in which the object in tracking appears, and as a result, the tracking of the object sometimes fails.

[0006] Therefore, an object of the present application is to provide a tracking device capable of appropriately continuing tracking of an object appearing in images obtained in time series.

[0007] According to one embodiment, a tracking device is provided. The tracking device has: a candidate region detection section that detects, as a candidate region in which a predetermined object is present, each region of at least one region in which a confidence indicating a probability that the predetermined object is present on an image generated by an imaging section is equal to or higher than a predetermined detection threshold, by inputting the image to an identifier; a tracking section that determines whether to establish correspondence between any of the at least one candidate region and an object region in which the predetermined object is present on a past image generated by the imaging section earlier than the image, based on an optical flow between the object region and each of the at least one candidate region, and increases a tracking number of the predetermined object by one in a case where the establishment of correspondence is possible; and a threshold control section that sets the detection threshold to be applied to a next image obtained later than the image by the imaging section to a first value in a case where the tracking number is equal to or lower than a predetermined number, and sets the detection threshold to be applied to the next image to a second value lower than the first value in a case where the tracking number is larger than the predetermined number.

[0008] In the tracking device, preferably, in the case where the tracking number is larger than the predetermined number, the candidate region detection section further detects, as an additional candidate region, a region in which the confidence is equal to or higher than a third value lower than the second value, from the image, and the tracking section determines whether to establish correspondence between the additional candidate region and the predetermined object based on an optical flow between the object region on the past image and the additional candidate region, in a case where it is determined that none of the at least one candidate region detected from the image is a region in which the predetermined object is present.

[0009] Further, in the tracking device, preferably, the threshold control section estimates a position of the predetermined object in the next image based on the object region in each of a plurality of past images generated by the imaging section during a period in which the predetermined object is tracked more than the predetermined number of times, and sets a region including the estimated position as a region to which the detection threshold having the second value is applied.

[0010] According to other embodiments, a tracking method is provided. The tracking method includes: detecting each of at least one region whose confidence indicating a probability of existing predetermined object appearing on an image generated by an imaging section being above a predetermined detection threshold as a candidate region of existing predetermined object appearing, by inputting the image to an identifier; determining whether to establish correspondence of any of the at least one candidate region as a region of predetermined object appearing, based on an optical flow between an object region of predetermined object appearing in a past image generated by the imaging section earlier than the image and each of the at least one candidate region, in a case where the establishing correspondence is possible, increasing a tracking number of the predetermined object by one; in a case where the tracking number is below a predetermined number, setting a detection threshold to be applied to a next image obtained later than the image by the imaging section as a first value; and in a case where the tracking number is more than the predetermined number, setting the detection threshold to be applied to the next image as a second value lower than the first value.

[0011] According to further other embodiments, a tracking computer program is provided. The tracking computer program includes commands for causing a computer to execute: detecting each of at least one region whose confidence indicating a probability of existing predetermined object appearing on an image generated by an imaging section being above a predetermined detection threshold as a candidate region of existing predetermined object appearing, by inputting the image to an identifier; determining whether to establish correspondence of any of the at least one candidate region as a region of predetermined object appearing, based on an optical flow between an object region of predetermined object appearing in a past image generated by the imaging section earlier than the image and each of the at least one candidate region, in a case where the establishing correspondence is possible, increasing a tracking number of the predetermined object by one; in a case where the tracking number is below a predetermined number, setting a detection threshold to be applied to a next image obtained later than the image by the imaging section as a first value; and in a case where the tracking number is more than the predetermined number, setting the detection threshold to be applied to the next image as a second value lower than the first value.

[0012] The tracking device according to the present disclosure has an effect of being able to appropriately continue tracking of an object appearing in images obtained in time series. BRIEF DESCRIPTION OF DRAWINGS

[0013] Figure 1 is a schematic configuration diagram of a vehicle control system in which the tracking device is installed.

[0014] Figure 2 is a hardware configuration diagram of an electronic control device as one embodiment of the tracking device.

[0015] Figure 3 is a functional block diagram of a processor of the electronic control device related to a vehicle control process including the tracking device.

[0016] Figure 4A This is a diagram illustrating the outline of the tracking process involved in the comparative example.

[0017] Figure 4B This is a diagram illustrating the outline of the tracking process involved in this embodiment.

[0018] Figure 5 It is a flowchart of the vehicle control process, including tracking and processing. Detailed Implementation

[0019] The following description, with reference to the accompanying drawings, describes the tracking device, the tracking method executed on the tracking device, and the computer program for tracking. The tracking device inputs each image of a series of images generated in a time sequence using a camera unit to a recognizer, thereby detecting and tracking an object as the detection target. Whenever an image representing the object has been tracked increases, the tracking device increases the tracking count of that object by one. Based on the tracking count, a detection threshold is controlled and compared with the confidence level output by the recognizer, which represents the probability of reflecting the existence of the detected object. Specifically, the tracking device sets the detection threshold applied to objects with more tracking counts than a predetermined number of times to a value lower than the value when the tracking count is less than the predetermined number of times. Therefore, even if the object is difficult to detect from the image for some reason during tracking, the tracking device can easily continue tracking the object.

[0020] The following describes an example of applying a tracking device to a vehicle control system. In this example, the tracking device tracks other vehicles traveling around the vehicle (hereinafter referred to as "surrounding vehicles" for ease of explanation) by performing tracking processing on images obtained using a camera mounted on the vehicle. Furthermore, the tracking results are used for the driving control of the vehicle. Surrounding vehicles are an example of predetermined objects that are detected and tracked.

[0021] Figure 1 This is a schematic diagram of the vehicle control system equipped with a tracking device. Additionally, Figure 2Fig. 1 is a hardware configuration diagram of an electronic control device that is an example of a tracking device. In the present embodiment, a vehicle control system 1 mounted on a vehicle 10 (i.e., a host vehicle) and controlling the vehicle 10 has a camera 2 for capturing a surrounding of the vehicle 10 and an electronic control device (ECU) 3 that is an example of a tracking device. The camera 2 and the ECU 3 are communicable via an in-vehicle network in accordance with a standard such as a controller area network. Further, the vehicle control system 1 can also have a storage device (not shown) that stores map information indicating positions and kinds of terrestrial objects, lane dividers, and the like for automatic driving control of the vehicle 10. Furthermore, the vehicle control system 1 can also have a distance measuring sensor (not shown) such as a LiDAR or a radar. Still further, the vehicle control system 1 can also have a receiver (not shown) for positioning a self position of the vehicle 10 in accordance with a satellite positioning system such as a GPS receiver. Still further, the vehicle control system 1 can also have a navigation device (not shown) for searching a predetermined travel route of the vehicle 10.

[0022] The camera 2 is an example of an imaging section, and has a two-dimensional detector composed of an array of photoelectric conversion elements having sensitivity to visible light such as a CCD or a C-MOS, and an imaging optical system that images an image of a region as a capturing target on the two-dimensional detector. The camera 2 is mounted in a vehicle cabin of the vehicle 10, for example, in a manner facing a front of the vehicle 10. Also, the camera 2 captures a front region of the vehicle 10 at a predetermined capturing period (e.g., 1 / 30 to 1 / 10 seconds) to generate an image in which the front region is captured. The image obtained by the camera 2 can be a color image or a gray scale image. Further, in the vehicle 10, a plurality of cameras having different capturing directions or focal lengths can be provided.

[0023] The camera 2 outputs the generated image and a capturing time (i.e., a time of generation of the image) to the ECU 3 via the in-vehicle network each time the image is generated.

[0024] The ECU 3 controls the vehicle 10. In the present embodiment, the ECU 3 detects a surrounding vehicle from a series of images obtained by the camera 2 in time series. Furthermore, the ECU 3 tracks the surrounding vehicle in accordance with a detection result of the surrounding vehicle from the series of images. Also, the ECU 3 controls the vehicle 10 in accordance with a tracking result of the surrounding vehicle so that the vehicle 10 is automatically driven in a manner avoiding collision with the surrounding vehicle. To this end, the ECU 3 has a communication interface 21, a memory 22, and a processor 23.

[0025] The communication interface 21 is an example of a communication section, and has an interface circuit for connecting the ECU 3 to an in-vehicle network. That is, the communication interface 21 is connected to the camera 2 via the in-vehicle network. Also, the communication interface 21 sends a received image to the processor 23 each time an image is received from the camera 2.

[0026] The memory 22 is an example of a storage section, and has, for example, a volatile semiconductor memory and a non-volatile semiconductor memory. Also, the memory 22 stores various data used in the tracking process executed by the processor 23 of the ECU 3. In the memory 22, as such data, for example, parameters indicating information related to the camera 2, such as the focal length, the photographing direction, and the set height of the camera 2, various parameters for determining the recognizer utilized in the detection of the surrounding vehicle, and the like are stored. In addition, the memory 22 stores various information utilized in the estimation of the distance to the detected surrounding vehicle. For example, the memory 22 stores, for each vehicle type, a vehicle width that is a reference for the vehicle of the vehicle type (hereinafter referred to as a reference vehicle width). Further, the memory 22 stores images received from the camera 2 together with the photographing time thereof for a certain period. Further, the memory 22 stores, for a certain period, a detection list in which various data generated in the course of the tracking process, such as information related to the surrounding vehicle in tracking, are recorded. Further, the memory 22 can also store information utilized in the travel control of the vehicle 10, such as map information.

[0027] The processor 23 is an example of a control section, and has one or a plurality of CPUs (Central Processing Units) and their peripheral circuits. The processor 23 can also have other arithmetic circuits such as a logic arithmetic unit, a numerical value arithmetic unit, or a graphic processing unit. Also, during the travel of the vehicle 10, the processor 23 executes the vehicle control process including the tracking process at a predetermined cycle (for example, several 10 msec to 100 msec). Also, the processor 23 controls the vehicle 10 so that the vehicle 10 performs automatic driving in accordance with the tracking result of the detected surrounding vehicle.

[0028] Figure 3is a functional block diagram of the processor 23 of the ECU 3 related to the vehicle control processing including the tracking processing. The processor 23 has a candidate region detection section 31, a tracking section 32, a threshold control section 33, and a vehicle control section 34. These sections of the processor 23 are, for example, functional modules realized by a computer program acting on the processor 23. Alternatively, these sections of the processor 23 can also be dedicated arithmetic circuits provided to the processor 23. Further, among these sections of the processor 23, the candidate region detection section 31, the tracking section 32, and the threshold control section 33 perform the tracking processing. Furthermore, when a plurality of cameras are provided in the vehicle 10, the processor 23 can perform the tracking processing in accordance with the image obtained by each camera for each camera.

[0029] The candidate region detection section 31 inputs the latest image received from the camera 2 to the recognizer by the ECU 3, and calculates a confidence indicating a probability of exhibiting a surrounding vehicle with respect to each of a plurality of regions on the image. Further, the candidate region detection section 31 detects each of the regions in which the confidence is equal to or higher than a predetermined detection threshold as a candidate region exhibiting a surrounding vehicle, among the plurality of regions.

[0030] The candidate region detection section 31 can use, for example, a so-called deep neural network (DNN) as the recognizer. Specifically, the DNN used as the recognizer can be of an architecture of a convolutional neural network (CNN) type such as Single Shot MultiBox Detector (SSD) or Faster R-CNN. Such a recognizer is pre-learned in accordance with a predetermined learning method such as a backpropagation method using a large number of training images exhibiting surrounding vehicles in a manner of detecting the surrounding vehicles from the images. That is, the recognizer is pre-learned in a manner such that the confidence calculated with respect to the region exhibiting a surrounding vehicle is higher than the confidence calculated with respect to the region not exhibiting a surrounding vehicle. The candidate region detection section 31 can appropriately detect the candidate region from the image by utilizing the DNN thus learned.

[0031] Further, the recognizer outputs a recognition result of the class of the surrounding vehicle (for example, passenger car, large vehicle, motorcycle, and the like) with respect to each of a plurality of regions on the image.

[0032] The candidate region detection section 31 compares the confidence degree calculated by the recognizer with respect to each region with the detection threshold value set by the threshold control section 33 with respect to the region. Also, the candidate region detection section 31 detects as a candidate region a region whose confidence degree is equal to or higher than the detection threshold value. Further, the detection threshold value is set to either a first value (for example, 0.8) or a second value (for example, 0.7) lower than the first value. The low threshold value region in which the detection threshold value is set to the second value is set by the threshold control section 33 with respect to an arbitrary category. Therefore, the candidate region detection section 31 can use the second value as the detection threshold value with respect to a region included in the low threshold value region and in which the category of the surrounding vehicle coincides with the category for which the second value is applied in the low threshold value region, among the regions with respect to which the confidence degree is calculated. Further, the candidate region detection section 31 can set, for example, a region equal to or higher than a predetermined proportion (for example, 0.7 to 1.0) of the low threshold value region as a region included in the low threshold value region. The candidate region detection section 31 outputs information indicating the position and the range of each of the detected candidate regions, the category of the surrounding vehicle appearing in the candidate region, and the calculated confidence degree to the tracking section 32 and the vehicle control section 34. Further, the information indicating the position and the range of the candidate region includes, for example, the coordinates of the upper left end position and the coordinates of the lower right end position of the candidate region.

[0033] Further, in a case where the confidence degree calculated with respect to a region included in the low threshold value region and in which the category of the surrounding vehicle coincides with the category for which the second value is applied in the low threshold value region is equal to or higher than a third value, the candidate region detection section 31 detects the region as an additional candidate region. Further, the third value is set to a value (for example, 0.6) lower than the second value. Also, the candidate region detection section 31 outputs information indicating the position and the range of each of the detected additional candidate regions, the category of the surrounding vehicle appearing in the additional candidate region, and the calculated confidence degree to the tracking section 32 and the vehicle control section 34.

[0034] Further, the recognizer can also be previously learned in a manner of detecting not only the surrounding vehicle but also an object other than the surrounding vehicle that is likely to affect the driving control of the vehicle 10. Also, the candidate region detection section 31 can determine that the object appears in a region in which the confidence degree calculated with respect to the object by inputting the image to the recognizer is equal to or higher than the detection threshold value as the first value. Also, the candidate region detection section 31 can output information indicating the category of the detected object and the region in which the object appears to the vehicle control section 34. Further, such an object includes at least any one of a moving object such as a person, a road sign such as a lane dividing line, a curbstone, a guardrail, and a ground object on or around a road.

[0035] The tracking section 32 determines, with respect to each of the detected candidate regions, whether or not a region detected from a past image generated by the camera 2 in the past compared with the image in which the candidate region is detected, and which corresponds to any of the surrounding vehicles being tracked. At this time, the tracking section 32 calculates, with respect to each of the candidate regions, an optical flow from a region in which the surrounding vehicle being tracked is reflected (hereinafter sometimes referred to as an "object region") in the past image and the candidate region thereof, and determines whether or not correspondence can be established based on the optical flow. Further, the tracking section 32 registers, as the object region in which the surrounding vehicle being tracked is reflected, the candidate region corresponding to any of the surrounding vehicles being tracked, in association with the surrounding vehicle, into the detection list, and increments the number of times of tracking with respect to the surrounding vehicle by one. Furthermore, the tracking section 32 takes, as the object region in which a newly detected surrounding vehicle is reflected, the candidate region in which the confidence is the first value or more of the detection threshold, among the candidate regions which do not correspond to any of the surrounding vehicles being tracked. Further, the tracking section 32 registers, as a newly started surrounding vehicle to be tracked, information indicating the surrounding vehicle reflected in the object region and the region in which the surrounding vehicle is reflected, into the detection list. At this time, the tracking section 32 sets the number of times of tracking of the newly started surrounding vehicle to be tracked to one. In addition, the tracking section 32 ends the tracking of the surrounding vehicle which does not correspond to any of the candidate regions of the image obtained in the most recent certain period, among each of the surrounding vehicles being tracked.

[0036] The tracking section 32 detects a plurality of feature points from an object region (hereinafter referred to as "attention region") in which each of the peripheral vehicles is imaged in the past image in which the peripheral vehicle was last detected, in accordance with a predetermined tracking method, with respect to each of the peripheral vehicles being tracked. Also, the tracking section 32 calculates an optical flow between each of the detected feature points and each of the candidate regions, with respect to each of the peripheral vehicles being tracked, and determines a candidate region in which the attention region is most consistent, from among the candidate regions, based on the calculated optical flow. At this time, the tracking section 32 can also only regard a candidate region in which a peripheral vehicle of the same class as the peripheral vehicle imaged in the attention region is imaged, from among the candidate regions, as a candidate region that is a calculation target of the optical flow. In a case where the degree of difference between the determined candidate region and the attention region is equal to or less than a predetermined degree of difference threshold, the tracking section 32 can establish a correspondence between the determined candidate region and the peripheral vehicle imaged in the attention region. Further, the tracking section 32 can employ, for example, a method based on Lucas-Kanade method, Kanade-Lucas-Tomasi (KLT) method, or Mean-Shift search as the predetermined tracking method. In addition, in the detection of the feature points, the tracking section 32 can use, for example, a filter for feature point detection used in the predetermined tracking method, such as a Harris operator or a SIFT descriptor. In addition, the degree of difference is calculated in accordance with the predetermined tracking method. For example, when the optical flow is calculated in such a manner that a square error of pixel values between each of the feature points of the attention region and corresponding points in the candidate region is minimized, the minimum square error at this time is calculated as the degree of difference. By thus establishing the correspondence, the tracking section 32 can establish a correspondence only with respect to a candidate region having a detection threshold equal to or greater than a first value, with respect to a peripheral vehicle whose tracking number is equal to or less than a predetermined number. On the other hand, with respect to a peripheral vehicle whose tracking number is greater than the predetermined number, the tracking section 32 can also establish a correspondence with respect to a candidate region having a detection threshold equal to or greater than a second value. Therefore, with respect to a peripheral vehicle that has been tracked for a certain period of time and is highly likely to be imaged in the latest image, even if it is difficult to detect the peripheral vehicle from the latest image, the tracking section 32 can continue to track the peripheral vehicle. Further, the predetermined number is set to be equal to or greater than 2, for example, to be equal to or greater than 5 to 10.

[0037] Further, by the above processing, sometimes there is no candidate region corresponding to the attention region in which the surrounding vehicle is tracked more than the predetermined number of times. In such a case, the tracking unit 32 determines whether to correspond the additional candidate region to the attention region based on the optical flow between the attention region and the additional candidate region. At this time, the tracking unit 32 regards, as the corresponding point for each feature point detected from the attention region, a point in which the difference in pixel value between the feature point and the corresponding point within the additional candidate region based on the optical flow is within a predetermined error range. Further, the tracking unit 32 can set, with respect to each feature point, a feature point block (for example, a block of 3 x 3 pixels) including a predetermined number of pixels centered on the feature point. The tracking unit 32 calculates, between the feature point block and the corresponding block within the additional candidate region based on the optical flow, the sum of the absolute values of the differences in pixel value of each corresponding pixel. Also, the tracking unit 32 regards, as the corresponding point corresponding to the feature point, the center pixel of the corresponding block in a case where the sum of the absolute values of the differences in pixel value is within a predetermined error range. The tracking unit 32 can also correspond the additional candidate region to the surrounding vehicle appearing in the attention region in a case where the proportion of the number of feature points for which the corresponding point can be determined to the total number of feature points detected from the attention region is equal to or greater than a predetermined proportion. The tracking unit 32 determines whether to correspond the additional candidate region to the surrounding vehicle under tracking also with respect to such an additional candidate region, and can continue to track the surrounding vehicle even in a case where the detection of the surrounding vehicle is particularly difficult in the latest image.

[0038] The tracking unit 32 saves the updated detection list and the number of times of tracking of each surrounding vehicle under tracking to the memory 22, and notifies the threshold control unit 33.

[0039] The threshold control unit 33 sets, with respect to each surrounding vehicle under tracking, a detection threshold to be applied to the surrounding vehicle in an image obtained by sequentially detecting images of each candidate region (hereinafter sometimes referred to as "next image").

[0040] In the present embodiment, the threshold control section 33 sets the detection threshold to the first value for the surrounding vehicle whose tracking number is the predetermined number or less. In contrast, the threshold control section 33 sets the detection threshold to the second value lower than the first value for the surrounding vehicle whose tracking number is more than the predetermined number. Further, the threshold control section 33 refers to the detection list, estimates the region in which the surrounding vehicle is expected to appear in the next image, and sets the estimated region as the low threshold region. The threshold control section 33 applies a prediction process to the object region in which the surrounding vehicle appears in each past image in tracking, for example, registered in the detection list, thereby estimating the region in which the surrounding vehicle is expected to appear in the next image. As such a prediction process, the threshold control section 33 can use a Kalman Filter. Alternatively, the threshold control section 33 can estimate the region in which the surrounding vehicle is expected to appear in the next image by applying a predetermined extrapolation process to the object region in which the surrounding vehicle appears in each past image in tracking. In this way, by setting the low threshold region in which the detection threshold having the second value is applied in the next image, even in the case where a plurality of surrounding vehicles whose tracking numbers are different exist, the detection threshold having an appropriate value can be applied to each of the surrounding vehicles.

[0041] The threshold control section 33 notifies the candidate region detection section 31 and the tracking section 32 of the detection threshold applied to the next image with respect to each surrounding vehicle in tracking. Further, the threshold control section 33 notifies the candidate region detection section 31 of information indicating the low threshold region with respect to each surrounding vehicle whose tracking number is more than the predetermined number and information indicating the category of the surrounding vehicle.

[0042] Figure 4A is a diagram illustrating an outline of the tracking process according to the comparative example, Figure 4B is a diagram illustrating an outline of the tracking process according to the present embodiment. In the example shown in Figure 4A , the surrounding vehicle 410 appears in each of a series of images 400-1 to 400-n obtained in time series. However, in the nth image 400-n, the confidence (0.75) calculated by the recognizer with respect to the region 401 in which the surrounding vehicle 410 appears is lower than the first value Thl of the detection threshold. As a result, the surrounding vehicle 410 is not detected in the image 400-n, and the tracking of the surrounding vehicle 410 by the tracking device according to the comparative example fails.

[0043] In contrast, in the present embodiment, Figure 4BIn the example shown, the surrounding vehicle 430 is also depicted in each of the series of images 420-1 to 420-n obtained in time series. Also, at the time point of the (n-1)th image 420-(n-1), the number of times of tracking of the surrounding vehicle 430 exceeds the predetermined number of times. Therefore, in the low threshold region 422 in which it is presumed that the surrounding vehicle 430 is depicted in the nth image 420-n, the detection threshold is changed from the first value Thl, which is uniformly applied to the image 420-(n-1), to a second value Th2 that is lower than the first value. Therefore, in the image 420-n, although the confidence (0.75) with respect to the region 421 in which the surrounding vehicle 430 is depicted is lower than the first value Thl, as long as it is the second value Th2 or more and the region 421 is within the low threshold region 422, the region 421 is still detected as a candidate region. As a result thereof, in the image 420-n, the surrounding vehicle 430 is also continuously tracked.

[0044] The vehicle control section 34 refers to the detection list to generate one or more travel predetermined paths (trajectory lines) of the vehicle 10 in a manner to avoid collision between each of the surrounding vehicles under tracking and the vehicle 10. The travel predetermined path is expressed as a set of target positions of the vehicle 10 at each time from the current time to a predetermined time ahead, for example. At this time, the vehicle control section 34 can set the travel predetermined path in a manner to cause the vehicle 10 to travel along a predetermined travel route set by the navigation device or in a manner to cause the vehicle 10 to travel along a lane on which the vehicle 10 is currently traveling. For example, the vehicle control section 34 performs a viewpoint conversion process using information such as the mounting position of the camera 2 to the vehicle 10 to convert the image inner coordinates of each of the surrounding vehicles under tracking into coordinates on the overhead view image (overhead view coordinates) by referring to the detection list. At this time, the vehicle control section 34 can estimate the position of each of the surrounding vehicles under tracking at the time each image is captured by acquiring the position and posture of the vehicle 10 at the time each image is captured, the estimated distance to each of the surrounding vehicles under tracking, and the direction from the vehicle 10 to each of the surrounding vehicles. Further, the vehicle control section 34 can estimate the position and posture of the vehicle 10 by matching the image generated using the camera 2 and the map information. For example, the vehicle control section 34 assumes the position and posture of the vehicle 10 to project a road or a terrestrial object on or around the road detected from the image onto the map information or to project a road or a terrestrial object on or around the vehicle 10 appearing in the map information onto the image. Also, the vehicle control section 34 estimates the position and posture of the vehicle 10 at the time the terrestrial object detected from the image and the terrestrial object appearing on the high-precision map are most consistent as the actual position and posture of the vehicle 10. In addition, the vehicle control section 34 can determine the direction from the vehicle 10 to each of the surrounding vehicles under tracking from the position of the region including each of the surrounding vehicles under tracking on the image and the optical axis direction of the camera 2. Furthermore, the vehicle control section 34 estimates the distance from the vehicle 10 to each of the surrounding vehicles under tracking from the ratio of the size of the region in which each of the surrounding vehicles under tracking appears to the reference size assuming that the distance between the vehicle 10 and each of the surrounding vehicles is a predetermined distance. Alternatively, in the case where the vehicle control system 1 has a distance measuring sensor (not shown) such as a LiDAR or a radar, the distance to each of the surrounding vehicles under tracking can be measured by the distance measuring sensor. In this case, for example, the distance from the position of the distance measuring sensor corresponding to the position from the camera 2 to the center of gravity of the region of the surrounding vehicle of interest appearing on the image is measured as the distance from the vehicle 10 to the surrounding vehicle of interest.

[0045] The vehicle control section 34 predicts the predicted trajectory of each of the surrounding vehicles under tracking at a predetermined time ahead by performing a prediction process using a Kalman Filter or a Particle filter or the like with respect to a series of overhead view coordinates of each of the surrounding vehicles under tracking.

[0046] The vehicle control portion 34 generates a travel scheduled path of the vehicle 10 in such a manner that the predicted value of the distance between the vehicle 10 and any one of the surrounding vehicles until a predetermined time after the surrounding vehicle is above a predetermined distance, based on the predicted trajectories of the surrounding vehicles in the tracking, the position, speed, and posture of the vehicle 10.

[0047] The vehicle control portion 34 controls each portion of the vehicle 10 in such a manner that the vehicle 10 travels along the notified travel scheduled path. For example, the vehicle control portion 34 calculates the acceleration of the vehicle 10 from the travel scheduled path and the current speed of the vehicle 10 measured by a vehicle speed sensor (not shown), and sets the throttle opening or the brake amount in such a manner that the acceleration becomes the set value. Also, the vehicle control portion 34 calculates the fuel injection amount from the set throttle opening, and outputs a control signal corresponding to the fuel injection amount to the fuel injection device of the engine of the vehicle 10. Alternatively, the vehicle control portion 34 calculates the electric power to be supplied to the motor from the set throttle opening, and controls the drive circuit of the motor in such a manner that the motor is supplied with the electric power. Further alternatively, the vehicle control portion 34 outputs a control signal corresponding to the set brake amount to the brake device of the vehicle 10.

[0048] Further, the vehicle control portion 34 calculates the steering angle of the vehicle 10 from the travel scheduled path in the case where the advancing road of the vehicle 10 is changed so that the vehicle 10 travels along the travel scheduled path. Also, the vehicle control portion 34 outputs a control signal corresponding to the steering angle to the actuator (not shown) that controls the steering wheel of the vehicle 10.

[0049] Figure 5 is an action flowchart of the vehicle control processing including the tracking processing, which is executed by the processor 23. The processor 23 executes the vehicle control processing in accordance with the action flowchart shown in Figure 5 in the following action flowchart. In addition, the processes of steps S101 to S108 in the following action flowchart correspond to the tracking processing.

[0050] The candidate region detection section 31 of the processor 23 inputs the latest image obtained from the camera 2 to the recognizer, and calculates the confidence for each of the regions on the image (step S101). The candidate region detection section 31 compares the confidence for each region with the detection threshold applied to the region. As described above, the candidate region detection section 31 applies the detection threshold having the second value to the region containing the peripheral vehicle of the same class as the class applied to the low threshold region, and applies the detection threshold having the first value to the region other than the above. Further, the candidate region detection section 31 detects the region having the confidence equal to or higher than the detection threshold as a candidate region (step S102). Furthermore, in the case where the confidence of the region contained in the low threshold region among the regions not detected as the candidate region is equal to or higher than the detection threshold having the third value, the candidate region detection section 31 detects the region as an additional candidate region (step S103).

[0051] The tracking section 32 of the processor 23 determines whether to establish correspondence between each of the candidate regions and any of the peripheral vehicles under tracking. Further, the tracking section 32 registers the candidate region establishing correspondence with any of the peripheral vehicles under tracking to the detection list in association with the peripheral vehicle as the object region representing the peripheral vehicle. Furthermore, the tracking section 32 increments the number of times of tracking with respect to the peripheral vehicle by one (step S104). Furthermore, the tracking section 32 determines whether to establish correspondence between the additional candidate region and the peripheral vehicle whose number of times of tracking exceeds a predetermined number of times. Further, the tracking section 32 registers the additional candidate region establishing correspondence with the peripheral vehicle whose number of times of tracking exceeds the predetermined number of times to the detection list in association with the peripheral vehicle as the object region representing the peripheral vehicle. Furthermore, the tracking section 32 increments the number of times of tracking with respect to the peripheral vehicle by one (step S105).

[0052] The threshold control section 33 of the processor 23 sets the detection threshold to be applied in the next image to the first value with respect to the tracking vehicle whose number of times of tracking is equal to or lower than the predetermined number of times among the peripheral vehicles under tracking (step S106). In addition, the threshold control section 33 sets the detection threshold to be applied in the next image to the second value lower than the first value with respect to the tracking vehicle whose number of times of tracking is larger than the predetermined number of times (step S107). Furthermore, the threshold control section 33 sets the low threshold region to which the detection threshold serving as the second value is applied (step S108).

[0053] The vehicle control section 34 of the processor 23 performs automatic driving control of the vehicle 10 in such a manner as to avoid collision between each of the peripheral vehicles under tracking and the vehicle 10 (step S109). Then, the processor 23 ends the vehicle control processing.

[0054] As explained above, the tracking device increases the number of times of tracking with respect to the object detected and tracked from a series of images obtained in time series by 1 each time the image in which tracking is completed increases. Also, the tracking device controls the detection threshold value compared with the confidence level indicating the probability that the object existing as a detection target is represented by the recognizer in accordance with the number of times of tracking. In particular, the tracking device sets the value of the detection threshold value applied with respect to the object whose number of times of tracking is more than a predetermined number of times to be lower than the value when the number of times of tracking is equal to or less than the predetermined number of times. Thus, even when the tracking of the object becomes difficult to detect the object from the image due to some reason, the tracking device can easily continue to track the object.

[0055] According to the modification, the threshold value control section 33 can not set the low threshold value region in a case where the position of the low threshold value region estimated with respect to the surrounding vehicle in tracking is outside the next image. The reason is that, in this case, the surrounding vehicle has a high possibility of moving to a position out of the shooting range of the camera 2 at the time of generation of the next image.

[0056] Further, the object as a tracking target is not limited to the surrounding vehicle. The object as a tracking target can be another moving object detectable from the image, for example, can be a person or an animal other than a person.

[0057] The tracking device related to the above-described embodiment or modification can also be applied to a system other than the vehicle control system. For example, the tracking device can be used to track an object represented in a series of images in time series obtained by a monitoring camera provided in such a manner that a predetermined region of the outside or the inside of a room is shot. In this case, the monitoring camera becomes one example of the imaging section.

[0058] The computer program realizing the functions of each section of the processor 23 of the tracking device related to the above-described embodiment or modification can also be provided in the form of a computer-readable removable recording medium such as a semiconductor memory, a magnetic recording medium, or an optical recording medium.

[0059] As described above, those skilled in the art can make various modifications in the scope of the present application while matching the embodiments.

Claims

1. A tracking device, comprising: The candidate region detection unit inputs an image generated by the camera unit into the recognizer and detects each region of at least one region whose confidence level, representing the probability of a predetermined object being projected in the image, is above a predetermined detection threshold as a candidate region. The tracking unit determines whether to establish a correspondence between any candidate region of at least one candidate region and the predetermined object, based on the optical flow between the object region that reflects the predetermined object in a past image generated earlier than the image by the camera unit and each candidate region of at least one candidate region. If the correspondence can be established, the tracking number of the predetermined object is increased by 1. as well as The threshold control unit sets the detection threshold applied to the next image obtained later than the previous image to a first value when the number of tracking attempts is less than a predetermined number; and sets the detection threshold applied to the next image to a second value lower than the first value when the number of tracking attempts is greater than the predetermined number. The tracking unit will select a candidate region with a confidence level of 1 or higher from the candidate regions that have not yet been established with any of the multiple predetermined objects being tracked, as the object region that reflects the newly detected predetermined object.

2. The tracking device according to claim 1, wherein, If the number of tracking attempts exceeds the predetermined number, the candidate region detection unit further detects regions in the image with a confidence level lower than the second value but higher than the third value, as additional candidate regions. If the tracking unit determines that any one of the candidate regions detected in the image is not a region that reflects the predetermined object, it determines whether to establish a correspondence between the additional candidate region and the additional candidate region as a region that reflects the predetermined object based on the optical flow between the object region and the additional candidate region in the past image.

3. The tracking device according to claim 1 or 2, wherein, The threshold control unit infers the position of the predetermined object in the next image based on the object region in each of the multiple past images generated by the camera unit during the period of tracking the predetermined object more times than the predetermined number of times, and sets the region including the inferred position as the region to which the detection threshold with the second value is applied.

4. A tracking method, comprising: By inputting an image generated using a camera into a recognizer, each region of at least one region, whose confidence level representing the probability of a predetermined object being projected in the image is above a predetermined detection threshold, is detected as a candidate region. Based on the optical flow between the object region that reflects the predetermined object in a past image generated earlier than the image by the camera unit and each of the candidate regions of at least one candidate region, it is determined whether to establish a correspondence between any candidate region of at least one candidate region and the predetermined object as the region that reflects the predetermined object. If the correspondence can be established, the number of times the predetermined object is tracked is increased by 1. If the number of tracking attempts is less than a predetermined number, the detection threshold applied to the next image obtained later than the image using the camera unit will be set to a first value. as well as If the number of tracking attempts is greater than the predetermined number, the detection threshold applied to the next image will be set to a second value, which is lower than the first value. The tracking method further includes using a candidate region with a confidence level of 1 or higher from among the candidate regions that do not correspond to any of the plurality of predetermined objects being tracked as the object region reflecting the newly detected predetermined object.

5. A computer-readable recording medium having a tracking computer program recorded thereon, the tracking computer program being used to cause a computer to execute: By inputting an image generated using a camera into a recognizer, each region of at least one region, whose confidence level representing the probability of a predetermined object being projected in the image is above a predetermined detection threshold, is detected as a candidate region. Based on the optical flow between the object region that reflects the predetermined object in a past image generated earlier than the image by the camera unit and each of the candidate regions of at least one candidate region, it is determined whether to establish a correspondence between any candidate region of at least one candidate region and the predetermined object as the region that reflects the predetermined object. If the correspondence can be established, the number of times the predetermined object is tracked is increased by 1. If the number of tracking attempts is less than a predetermined number, the detection threshold applied to the next image obtained later than the image using the camera unit will be set to a first value. as well as If the number of tracking attempts is greater than the predetermined number, the detection threshold applied to the next image will be set to a second value, which is lower than the first value. The tracking computer program also causes the computer to use a candidate region with a confidence level of 1 or higher from among the candidate regions that do not correspond to any of the plurality of predetermined objects being tracked as the object region that reflects the newly detected predetermined object.

Images