Object Tracking Device

Abstract

To provide an object tracking device capable of generating precisely a trajectory of an object within a measurement range by correcting object detection frame information.SOLUTION: An object tracking device (100) that generates a trajectory of an object within a measurement range of a camera (2) includes: an object detector (4) for detecting an object on multiple frames acquired by a sensor; a detection frame reliability calculation unit (8) that calculates the reliability of the detection frames based on flow information among frames of the detection frames in which the object is detected; a detection frame position correction unit (9) that corrects the detection frame information of a low reliable detection frame using the detection frame information of highly reliable detection frame; and a trajectory generation unit (10) that generates the trajectory of the object using the corrected detection frame information.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an object tracking device. [Background technology]

[0002] In recent years, there is an increasing need for image recognition technology that tracks a detected object and generates trajectory information by analyzing images captured by surveillance cameras, in-vehicle cameras, etc. In particular, trajectory information viewed from a bird's-eye view can be easily visualized by projecting it onto a two-dimensional map, and can also be used for managing the target's work behavior and detecting abnormal behavior. As a method for generating the trajectory, a method is given in which the distance from the camera to the detected object is calculated in each frame of a camera image, three-dimensional position information is obtained, and the position information for each frame is integrated. As an example of distance calculation, a method is generally used in which the circumscribing rectangle information of the object obtained by detecting the object from the camera image and the camera parameters are used. Of these, the camera parameters can be estimated by taking a picture in an ideal environment in advance, but the circumscribing rectangle information needs to be estimated from a camera image taken at an actual shooting site. Therefore, in order to generate a highly accurate trajectory, a technology is required that can accurately estimate the position of the circumscribing rectangle (hereinafter referred to as the detection frame) even in various shooting sites. As an example of the technology, there is a method that uses dictionary information created by machine learning such as a convolutional neural network. Machine learning requires the creation of learning data in advance, and although it would be nice to be able to learn a variety of scenes, the variation of scenes is limited in light of the realistic man-hours involved. Therefore, depending on the camera installation environment, there are cases where the correct detection frame information cannot be obtained due to non-detection or erroneous detection of the target, or cases where the detection frame information includes the background of the target and the accurate rectangular position cannot be estimated. Patent Document 1 proposes a method of correcting the detection frame information by performing object tracking processing and object recognition processing in parallel, and Patent Document 2 proposes a method of correcting the rectangular position when the detection target is occluded by an obstacle by extracting image feature points and estimating a movement vector. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] International Publication 2019 / 180917 [Patent Document 2] Patent Publication No. 2006-323437 Summary of the Invention [Problem to be solved by the invention]

[0004] Patent Document 1 describes a technology that calculates the trajectories of multiple targets with high accuracy by performing object recognition at a predetermined frame interval and correcting the ID information of the rectangle while tracking the detected target using object tracking processing to generate the trajectory of the same target, but it is not possible to correct the size or position information of the rectangle. Also, Patent Document 2 describes a technology that can correct the rectangle position by predicting the rectangle position in the next frame and thereafter from the movement vector information obtained from the image features within the circumscribing rectangle of the detected target, but it is not possible to correct the size of the rectangle.

[0005] The present invention is an invention for solving the above-mentioned problems, and aims to output a highly accurate trajectory of a detection target in an object tracking device that generates a trajectory of an object within a measurement range by correcting the detection frame position according to the reliability calculated from flow information. [Means for solving the problem]

[0006] In order to achieve the above object, the present invention provides An object tracking device that generates a trajectory of an object within a measurement range of a sensor, an object detection unit that detects an object in each of a plurality of frames acquired by the sensor; a detection frame reliability calculation unit that calculates a reliability of the detection frame in which the object is detected based on inter-frame flow information of the detection frame; a detection frame position correction unit that corrects detection frame information of a low-reliability detection frame having lower reliability than the high-reliability detection frame by using detection frame information of a high-reliability detection frame having reliability higher than a threshold; a trajectory generating unit that generates a trajectory of the object using the corrected detection frame information; The present invention is characterized by having the following. Effect of the Invention

[0007] By applying the object tracking device of the present invention based on the above-mentioned features, it is possible to correct the detection frame information of the detection target and generate a highly accurate trajectory.

[0008] Further features related to the present invention will become apparent from the description of the present specification and the accompanying drawings. In addition, problems, configurations, and effects other than those described above will become apparent from the following description of the embodiments. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a functional block diagram of a first embodiment of the present invention. [Diagram 2] FIG. 2 is a diagram for explaining an object detection unit 4. [Diagram 3] FIG. 4 is a configuration diagram of the existence probability map determination unit 5. [Figure 4] FIG. 2 is a diagram for explaining the existence probability map creation unit 21. [Diagram 5] An example of how to create a presence probability map. [Figure 6] FIG. 4 is a diagram for explaining the existence probability map interpolation unit 22. [Figure 7] FIG. 11 is a diagram for explaining a flow of adding a detection window. [Figure 8] FIG. 4 is a diagram for explaining a flow calculation unit 7. [Figure 9] FIG. 4 is a diagram for explaining a detection frame reliability calculation unit 8. [Figure 10] FIG. 4 is a diagram showing the configuration of a detection frame position correction unit 9. [Figure 11] FIG. 11 is a functional block diagram of a second embodiment of the present invention. [Figure 12] FIG. 4 is a diagram showing the configuration of a detection target movement direction prediction unit 91. [Figure 13] FIG. 11 is a functional block diagram of a third embodiment of the present invention. [Figure 14] A diagram showing formula 1. [Figure 15] A diagram showing formula 2. [Figure 16] A diagram showing formula 3. [Figure 17] A diagram showing formula 4. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, specific embodiments of the present invention will be described with reference to the drawings.

[0011] <Example 1> FIG. 1 is a block diagram of the first embodiment of the present invention. In this embodiment, an embodiment in which the measurement device is a camera will be described, but the measurement device is not limited to this and can be applied to other sensors such as a stereo camera or a distance sensor. The object tracking device 1 shown in Fig. 1 is a device that corrects the detection frame position based on presence probability map information generated from the object detection results of all frames captured by a camera 2 and reliability information of the detection frame calculated from flow information, etc., and generates a highly accurate trajectory.

[0012] The camera 2 is attached to, for example, the body of an automobile, and captures images of the road surface and other vehicles in front of the vehicle. The object tracking device 1 is a device that acquires images captured by the camera 2 and performs analysis processing to analyze the trajectory of an object in front of the vehicle that is the detection target. The object tracking device 1 is realized in a computer PC prepared separately from the camera 2. The computer PC has an arithmetic unit, a main memory device, and an external memory device, and realizes the functions of a frame collection unit 3, an object detection unit 4, a presence probability map determination unit 5, a detection frame adjustment determination unit 6, a flow calculation unit 7, a detection frame reliability calculation unit 8, a detection frame position correction unit 9, and a trajectory generation unit 10. The object tracking device 1 may be provided integrally inside the camera 2.

[0013] 1, the frame collection unit 3 has a function of collecting images (frames) captured by the camera 2, the object detection unit 4 has a function of detecting the detection target in all frames acquired by the frame collection unit 3, the presence probability map determination unit 5 has a function of generating or interpolating a presence probability map of the detection target from the object detection results of each frame, the detection frame adjustment determination unit 6 has a function of determining whether to add or subtract a detection frame in each frame by using the presence probability map information, the flow calculation unit 7 has a function of calculating flow information between frames, the detection frame reliability calculation unit 8 has a function of calculating the reliability of the detection frame from the flow information within the final detection frame output from the detection frame adjustment determination unit 6, the detection frame position correction unit 9 has a function of correcting the detection frame position and size according to the reliability of the detection frame, and the trajectory generation unit 10 has a function of generating the trajectory of the detection target from the corrected detection frame information. Each of the functions 3, 4, 5, 6, 7, 8, 9, and 10 will be described in detail below.

[0014] The frame collection unit 3 collects frames, which are images captured by the camera 2. There are multiple frames to be collected, and they may be all frames in which the detection target exists, or frames automatically or manually selected from all frames. There are no particular limitations on the selection method, and there are methods such as automatically extracting frames within a specified shooting time, automatically extracting frames in which the detection target exists within a measurement range in a video specified by the user, and manually selecting frames by the user via a GUI (Graphical User Interface) or the like.

[0015] The object detection unit 4 will be described with reference to FIG. The object detection unit 4 detects the detection target in each of the multiple frames collected by the frame collection unit 3. In FIG. 2, 11 indicates a captured image (frame), 12 and 13 indicate detection targets (a person and a car in this example), 14a and 14b indicate examples of detection frames in which an area including an object is estimated from an image by the object detection unit 4, and 15 indicates an example of detection frame information. The object detection unit 4 detects the detection target present in the captured image by utilizing a phenomenon that allows the detection of the object from an image generated in advance from learning data by machine learning or the like. The algorithm for generating the detection dictionary may be a general one such as a convolutional neural network or AdaBoost, and is not particularly limited. The object detection unit 4 detects the object in the multiple frames acquired by the frame collection unit 3 using the dictionary, and saves the detection frame information as shown in 15 of FIG. 2. The detection frame information includes, for each detected target, class information indicating the type of target, reliability information indicating the accuracy of the class information, start points (X, Y) indicating the image coordinates of the upper left corner of the detection frames 14a and 14b, information on the horizontal and vertical widths of the detection frames, etc. Note that methods other than this example may be used as long as they are capable of estimating the position of the detection target in the image.

[0016] FIG. 3 shows a block configuration of the existence probability map determination unit 5. The presence probability map determination unit 5 performs processing to generate an presence probability map indicating the presence probability of a detection target in each of a plurality of frames. The presence probability map determination unit 5 includes a presence probability map creation unit 21 that creates an presence probability map for each frame from the detection window information output from the object detection unit 4, a presence probability map interpolation unit 22 that generates an interpolated presence probability map for the target frame by interpolating information on the presence probability maps of two or more frames before and after the target frame, and a presence probability map selection unit 23 that compares the presence probability map with the interpolated presence probability map and selects a final presence probability map for each frame. Below, 21, 22, and 23 will be described in detail.

[0017] The existence probability map creation unit 21 will be described with reference to FIG. The presence probability map creation unit 21 generates a presence probability map indicating the presence probability of an object in each of a plurality of frames. The presence probability map creation unit 21 calculates, for each frame, a presence probability map for each detection target class from the detection window information.

[0018] The existence probability map creation unit 21 generates an existence probability map using a probability density function calculated by a normal distribution from the position information and reliability information of the detection frame of the object detected by the object detection unit 4, and one or more pieces of information of the object detection results using multiple different dictionaries.

[0019] One method for generating the presence probability map is to divide the captured image 11 into a plurality of small regions as shown in 31a and 31b, calculate the degree of overlap between the detection frames 14a and 14b of each class and each small region as a percentage, and generate presence probability maps as shown in 32a and 32b. The number of small regions is not particularly limited and may be determined in consideration of the specifications of the computer PC that executes the processing, and the presence probability map may be generated by dividing the frame into a plurality of regions after reducing the resolution of the frame in advance.

[0020] Furthermore, the value of the presence probability map is not particularly limited as long as it indicates the proportion of objects of each class present within a small region, and instead of using the degree of overlap with the detection frame, a method utilizing a probability density function such as that shown in Figure 5 may be used.

[0021] FIG. 5 is a diagram for explaining an example of a method for creating the existence probability map. In FIG. 5, 11 is a captured image, 41 is a detection frame, and 42 is the center coordinate (x c , y c ), 43, 44 indicate probability density functions f(x) and f(y) for the X and Y directions of the image coordinate axis used to calculate the existence probability map. The method for creating the probability density function f(x) in the X direction is to multiply the average value shown in Equation 1 in FIG. 14 by the x-coordinate x of the center of the image. c, and a normal distribution with variance σ^2. Similarly, the probability density function f(y) in the Y direction is calculated using Equation 2 in FIG. 15, and the existence probability map f(x, y) can be generated by calculating the product of f(x) and f(y) using Equation 3 in FIG. 16. The value of the variance σ^2 may be determined based on the information on the width and height of the detection frame, or based on the aspect ratio information of the width and height when the actual measurement information of the detection target is known in advance. In addition, a method of adjusting the value of the existence probability of the detection target by multiplying the reliability information of the detection frame by the existence probability map f(x, y) may also be adopted.

[0022] The existence probability map interpolation unit 22 will be described with reference to FIG. The existence probability map interpolation unit 22 performs processing to estimate the existence probability map of a corresponding frame (target frame) from existence probability map information of frames t-1 and t+1 before and after frame t, or from existence probability map information of multiple previous frames, or from existence probability map information of multiple subsequent frames t+1 and t+2. The interpolation method used by the existence probability map interpolation unit 22 is roughly divided into two patterns shown in 51 and 52 of FIG.

[0023] In the method shown in 51 of FIG. 6, an interpolated existence probability map corresponding to the t-th frame 54 is generated from the existence probability maps of the preceding and succeeding frames, that is, the t-1-th frame 53a and the t+1-th frame 53b. For example, the generation method may be a method of calculating the product or average value of the existence probability of the existence probability maps of the t-1-th frame 53a and the t+1-th frame 53b. In the example shown in 51 of FIG. 6, a car existence probability map in the t-1-th frame 53a is generated based on the detection frame information of the detection frame 53a1, and a car existence probability map in the t+1-th frame 53b is generated based on the detection frame information of the detection frame 53b1. The existence probability map interpolation unit 22 calculates the product or average value of the existence probability of the car existence probability map in the t-1-th frame 53a and the car existence probability map in the t+1-th frame 53b, respectively, to generate an interpolated existence probability map of the t-th frame 54.

[0024] 6, an interpolated existence probability map corresponding to the t-th frame 54 is estimated based on the amount of change from the existence probability map of the t+2th frame 53c to the existence probability map of the t+1th frame 53b. For example, after estimating the approximate position of the detection window in the t-th frame 54 from the amount of change in the center position, vertical width, and horizontal width of the t+2th and t+1th detection windows 53c1 and 53b1, the existence probability map corresponding to the t+1th frame 53b is enlarged or reduced by linear interpolation or the like in accordance with the amount of change in the size of the detection window to determine the size and position of the detection window 54a1 in the t-th frame, and an interpolated existence probability map corresponding to the t-th frame 54 is calculated. In this example, a method for calculating an interpolated presence probability map of the t-th frame from frames before and after the t-th frame, or from the t+1th and t+2th frames, has been described. However, it is also possible to calculate an interpolated presence probability map of a frame from multiple frames before and after the frame, or to calculate an interpolated presence probability map of a frame from multiple frames before the frame, such as the t-1th frame and the t-2th frame.

[0025] There are no particular limitations on the method as long as it estimates the presence probability map of the relevant frame (target frame) from the presence probability map information of the preceding and following frames. In this example, in the entire frame group acquired by the frame collection unit 3, an interpolated presence probability map is generated for the first frame by the method shown in 52, an interpolated presence probability map is generated for the final frame by the reverse flow of the method shown in 52, and an interpolated presence probability map is generated for the remaining frames by the method shown in 51.

[0026] The existence probability map selection unit 23 selects a final existence probability map from the existence probability map and the interpolated existence probability map for each frame. The selection method is not particularly limited, and may be, for example, a method of calculating the difference between the existence probability map and the interpolated existence probability map and adopting the value of the interpolated existence probability map only for small regions where the difference is equal to or greater than a threshold, or a method of calculating the product or average of the existence probability map and the interpolated existence probability map and setting the existence probability value of small regions below a threshold to 0.

[0027] By the method described above, the existence probability map determination unit 5 determines the existence probability map for each frame. In this example, the flow has been described in which two maps, an existence probability map and an interpolated existence probability map, are generated for each frame, and then the final value is determined by the existence probability map selection unit 23, but multiple interpolated existence probability maps may be generated for a frame. For example, a method may be used in which, for frames other than the first frame and the last frame in the frame group, three types of interpolated existence probability maps are generated by the method of 51, 52, and the reverse flow of 52, and then the final value is similarly determined by the existence probability map selection unit 23, or a method may be used in which the number of frames before and after used for interpolation is increased to increase the number of patterns of the interpolated existence probability map.

[0028] In addition, in this flow, an interpolated existence probability map is created for all frames, but a method of correcting the existence probability map of the corresponding frame by generating an interpolated existence probability map only for some frames may be adopted. For example, a method of correcting the existence probability map only for frames in which the reliability of the detection window information is equal to or lower than a threshold, a method of counting the number of detection windows of the detection target in each frame and calculating the average value, and then correcting only for frames with a number of detection windows other than the average, a method of correcting only frames selected by the user via a GUI, etc., etc., are not particularly limited. Also, a method of treating an existence probability map generated by acquiring detection window information of the detection target using multiple different dictionaries for each frame as an interpolated existence probability map, and correcting the existence probability map by the existence probability map selection unit 23 may be used.

[0029] The detection frame adjustment determination unit 6 performs a process of determining whether to add or remove a detection frame in each of a plurality of frames using the presence probability map. The detection frame adjustment determination unit 6 uses the presence probability map information output by the presence probability map determination unit 5 to determine whether to add or remove a detection frame in each frame. As a method of determination, when a detection frame does not exist near an area where the presence probability is equal to or greater than a threshold, a circumscribing rectangle of the area where the presence probability is equal to or greater than a threshold is added as a detection frame, and when a detection frame exists in an area where the presence probability is equal to or less than a threshold, the corresponding detection frame is removed. The criterion for determining whether a detection frame exists in the presence probability map is not particularly limited, and may be the content rate of a small area in the detection frame that satisfies the threshold of the presence probability map, or a method in which a detection frame exists near the presence probability map if the Euclidean distance between the center coordinates of the detection frame and the small area that satisfies the threshold of the presence probability map is within a specified range. In addition, when a detection frame is added, the detection frame adjustment determination unit 6 may add a process of increasing the size of the detection frame by a specified margin in anticipation of the position of the detection frame being corrected by the detection frame position correction unit 9 in the subsequent stage.

[0030] Using FIG. 7, a flow of estimating the presence probability map of the t-th frame from the detection window information of the t-1-th and t+1-th frames and adding the detection window 57 to the t-th frame will be described. First, the presence probability map creation unit 21 divides the t-1-th and t+1-th frames 53a and 53b into small regions 54a and 54b, and generates the presence probability maps 55a and 55b from the overlap rate with the detection windows 54a1 and 54b1. Next, the presence probability map 56 of the t-th frame is estimated by calculating the average of the presence probability maps 55a and 55b. Finally, the detection window 57 is determined based on a predetermined threshold and the value of the presence probability map. In the example shown in Figure 7, the value of each small area on the presence probability map is classified into three classes: less than 0.5, 0.5 or more and less than 0.9, and 0.9 or more. Detection window 57 is determined based on the conditions that small areas less than 0.5 are not included in the rectangle, all small areas greater than or equal to 0.9 are included in the rectangle, and small areas greater than or equal to 0.5 and less than 0.9 are included in the rectangle but more than half of them are not included.

[0031] The flow calculation unit 7 will be described with reference to FIG. FIG. 8 shows an example of calculating flow information between frames by analyzing image features of the t-th frame 11a and the t+1-th frame 11b by the flow calculation unit 7. In FIG. 8, 61 indicates a road surface, 62 indicates a vehicle to be measured in the frame 11a, 63 indicates a vehicle to be measured in the frame 11b, 64a and 64b indicate examples of image features in the vehicle 62, 65a and 65b indicate examples of image features in the vehicle 63, 66 indicates flow information between the image feature points 64a and 65a, and 67 indicates flow information between the image feature points 64b and 65b. The flow information is a movement vector of the same image feature point between frames, and indicates the direction and magnitude of the vector. The method of acquiring image features and flow information is not particularly limited as long as it is a method capable of calculating the same image feature point and its flow information between previous and next frames, such as a method of utilizing optical flow by the Lucas-Kanade method or a method of tracking using image features such as SHIFT. In this example, flow information for frames other than the first frame is calculated from the current frame (tth frame) and the previous frame (t-1th frame), and flow information for the first frame is obtained by inverting the sign of the numerical value calculated from the current frame and the next frame (t+1th frame).

[0032] The detection frame reliability calculation unit 8 will be described with reference to FIG. The detection frame reliability calculation unit 8 calculates the reliability of the detection frame based on the flow information between frames within the detection frame of the detected detection target. The detection frame reliability calculation unit 8 calculates the reliability of the detection frame by analyzing the flow information calculated by the flow calculation unit 7 within the detection frame information of the final object detection output from the detection frame adjustment determination unit 6.

[0033] In the captured image 11 in which the detection target vehicle 62 shown in FIG. 9 is traveling on a road surface 61, the final detection frame 70 and image feature points 71a to 71f output from the detection frame adjustment determination unit 6 are examples of image feature points having flow information on the vehicle, and image feature points 72a and 72b are examples of image feature points having flow information on the road surface. Optical flow generally calculates flow information of a moving object between frames. Therefore, when the camera is fixed, such as a surveillance camera, the image feature points and flow information of the moving object are easily calculated, and image feature points and flow information are hardly calculated for a background such as a road surface. Also, in the case of a dynamic camera such as an in-vehicle camera, image feature points and flow information can be calculated in the same way for a moving object, and image feature points are easily calculated for a background such as a road surface compared to a fixed camera, but flow information tends to be small.

[0034] Therefore, in this example, image feature points with flow information larger than a predetermined threshold are considered to be moving object feature points (three-dimensional object feature points), and image feature points with flow information smaller than the threshold are considered to be plane feature points (road surface feature points), and the reliability of the detection frame 70 is calculated from the content rate of moving object feature points and plane feature points in the detection frame 70. That is, the detection frame reliability calculation unit 8 calculates the reliability of the detection frame based on the information of the three-dimensional object feature points and the road surface feature points in the detection frame. As a calculation method, a method of using the ratio of the difference between the number of moving object feature points and the number of plane feature points with respect to the size of the detection frame as shown in (Formula 4) in FIG. 17, or a method of considering the magnitude of flow information for each feature point in (Formula 4) to improve the reliability as there are three-dimensional object feature points with larger flow information and lower the reliability as there are plane feature points with smaller flow information may be adopted. Also, the reliability may be calculated by considering the direction of the flow information in (Formula 4). For example, one method takes advantage of the fact that image feature points of the same moving object have similar flow directions, and increases the reliability of the detection frame the more three-dimensional object feature points that show similar flow directions there are within the detection frame.

[0035] In this example, the reliability of the detection frame is calculated using flow information, but the reliability may be calculated using image information that can distinguish between a moving object (three-dimensional object) and a background. For example, after obtaining the inter-frame difference, it may be determined that the more difference regions there are in the detection frame, the more likely it is that the region is a moving object, and the reliability may be calculated based on the content rate of the difference regions relative to the area of ​​the detection frame, or a method may be used in which edge detection or the like is performed in the detection frame to estimate a candidate contour of the detection target, and a detection frame with a high overlap rate between the circumscribing rectangle of the contour and the detection frame is determined to be a highly reliable detection frame.

[0036] FIG. 10 shows a block diagram of the detection frame position correction unit 9. The detection frame position correction unit 9 performs a process of correcting the detection frame information of a low-reliability detection frame, which has lower reliability than the high-reliability detection frame, using the detection frame information of a high-reliability detection frame, which has a reliability higher than a threshold value. The detection frame position correction unit 9 includes a detection target matching unit 80 that matches detection targets that are considered to be the same from the detection results of all frames, a high-reliability detection frame selection unit 81 that selects a high-reliability detection frame for each identical detection target, and a low-reliability detection frame correction unit 82 that uses the selected high-reliability detection frame information to correct the position information of the remaining low-reliability detection frames for the same detection target. Each function will be described below.

[0037] The detection target matching unit 80 analyzes the detection frame information of all frames and assigns the same ID information to detection frames that are considered to be the same target. As a means for determining whether they are the same target, there is a method for calculating the Euclidean distance of the center coordinates between all detection frames in previous and next frames, and performing a process for all frames in which the closest detection frames are determined to be the same detection target. In addition to this method, there is no particular limitation as long as it is a method that uses the detection results of all frames as input and determines whether they are the same target using a combinatorial optimization algorithm or the like.

[0038] The high-reliability detection frame selection unit 81 collects information on detection frames to which the same ID information has been assigned by the detection target association unit 80, and selects multiple pieces of information on detection frames with high reliability (high-reliability detection frames). The selection method is not particularly limited, and may include a method of selecting all detection frames with a reliability equal to or higher than a predetermined threshold, a method of selecting information on a specified number of detection frames in descending order of reliability, and the like. In addition, information on detection frames with high reliability may be selected by utilizing image features, a GUI, or the like. For example, a method of estimating an abnormal captured image such as whiteout based on image features, and not using information on detection frames of the corresponding image, or a method of allowing a user to manually select a high-reliability detection frame using a GUI, or the like, may be used.

[0039] The low-reliability detection frame correction unit 82 corrects the position information of unselected detection frames that have been assigned the same ID information, using the detection frame information selected by the high-reliability detection frame selection unit 81. The position information of the detection frame is information on the position and size of the detection frame in the captured image, as shown in 15 of FIG. 2. The correction method is not particularly limited as long as it is a method that allows function approximation from multiple points, such as a method of collecting information on the center coordinate position, width, and height of a detection frame with high reliability, and calculating the center coordinate position, width, and height of a detection frame with low reliability by a general interpolation method such as spline interpolation. In addition, a method of utilizing the reliability value as weight information in the interpolation method may also be used.

[0040] The trajectory generating unit 10 calculates the distance between the camera 2 and the detection target by using the detection frame information output from the detection frame position correcting unit 9, and generates a trajectory. A general method that utilizes camera parameters is used as a method for calculating the distance to the detection target. Specifically, the image coordinates of the lower end center of the detection frame in the detection target are converted to camera coordinates using internal parameters including the focal length and distortion correction coefficient among the camera parameters. Then, by assuming that the point of the three-dimensional world coordinates calculated from the camera coordinates of the lower end center using the external parameters indicating the installation attitude and angle of the camera exists on the ground at a height of 0 in the real world, the position of the detection target in the real world coordinates can be estimated, and the distance from the camera to the detection target can be calculated. Note that the distance calculation method is not particularly limited as long as it is a method that can estimate the distance from the camera to the detection target using the detection frame information in the image. It is possible to obtain a trajectory by connecting the three-dimensional world coordinates obtained by calculating the distance from the camera to the target using all the detection frame information of the same target and looking down on it.

[0041] In the first embodiment of the present invention, with the functional configuration described above, in an object tracking device that generates a trajectory of an object within a measurement range, the size of the detection frame is adjusted based on presence probability map information generated from the object detection results of each frame, and the position of the detection frame is corrected according to the reliability calculated from the flow information, making it possible to output a highly accurate trajectory of the detection target.

[0042] The object tracking device 1 of this embodiment detects a detection target in each of multiple frames captured by the camera 2, and calculates the reliability of the detection frame based on the flow information between frames within the detection frame of the detected detection target. Then, the detection frame information of the low-reliability detection frame, which has a higher reliability than a threshold, is used to correct the detection frame information of the low-reliability detection frame, and a trajectory is generated using the corrected detection frame information.

[0043] According to the object tracking device 1 of the present embodiment, it is possible to reduce missed detections and false detections of detection objects in a frame, reduce positional deviations of the detection frame, and generate a highly accurate trajectory of the same detection object. Therefore, for example, in order to analyze the quality of ADAS or the situation when a traffic accident occurs, it is possible to provide a service that is utilized for maintenance and inspection of emergency braking operation, evaluation of whether the behavior is correct, etc., by estimating the trajectory of a preceding vehicle from an on-board image and comparing it with CAN data.

[0044] In this embodiment, the object tracking device 1 has been described as having the presence probability map determination unit 5 and the detection frame adjustment determination unit 6, but these may be omitted. Even if the presence probability map determination unit 5 and the detection frame adjustment determination unit 6 are omitted, the object detection unit 4, the detection frame reliability calculation unit 8, the detection frame position correction unit 9, and the trajectory generation unit 10 are included, so that the detection frame information of the detection target can be corrected based on the detection frame reliability, and a highly accurate trajectory can be generated using the corrected detection frame information. In this embodiment, the object tracking device 1 has the presence probability map determination unit 5 and the detection frame adjustment determination unit 6, so that the detection accuracy of the detection frame can be further improved.

[0045] <Example 2> FIG. 11 is a block diagram of the second embodiment of the present invention. A feature of this embodiment is that the object tracking device 90 has a detection target moving direction prediction unit (object motion estimation unit) 91 that estimates the motion of the object. The object tracking device 90 shown in FIG. 11 is a device that predicts the moving direction of the detection target in advance, and generates a presence probability map, calculates the reliability of the detection frame, and corrects the position by utilizing the prediction result, thereby generating a highly accurate trajectory of the detection target. In FIG. 11, the frame collection unit 3, the object detection unit 4, the presence probability map determination unit 5', the detection frame adjustment determination unit 6, the flow calculation unit 7, the detection frame reliability calculation unit 8', the detection frame position correction unit 9', and the trajectory generation unit 10 have the same or almost the same functions as those in the first embodiment.

[0046] The detection target movement direction prediction unit 91 estimates the movement of the object from one or more pieces of information, such as the trajectory information generated by the trajectory generation unit 10, the movement area candidate information of the detection target, the object movement information around the detection target, and the medium information such as the vehicle on which the camera 2 serving as the measuring device is installed. The detection target movement direction prediction unit 91 shown in FIG. 11 has a function of predicting the movement direction of the detection target based on the result of executing image processing on all frames acquired by the frame collection unit 3 and the object detection result acquired from the object detection unit 4. Hereinafter, a method of generating a presence probability map, calculating the reliability of the detection frame, and correcting the position of the detection frame using the detection target movement direction prediction unit 91 and the acquired surrounding environment information will be described.

[0047] FIG. 12 is a block diagram of the detection object movement direction prediction unit 91. As shown in FIG. 12, the moving area extraction unit 95 has a function of analyzing frame information acquired from the frame collection unit 3 to extract the moving area of ​​the detection target, the surrounding object information analysis unit 96 has a function of recognizing the state of objects around the detection target using the object detection result acquired from the object detection unit 4 and acquiring information such as the moving direction of the surrounding objects, the control data acquisition unit 97 has a function of acquiring control data such as the steering angle and running speed of the vehicle when the camera (measurement device) 2 is installed on a vehicle, and the detection target moving direction output unit 98 has a function of estimating the moving direction of the detection target by analyzing the information acquired by the moving area extraction unit 95, the surrounding object information analysis unit 96, and the control data acquisition unit 97, and outputting the result to a subsequent processing block. The functions 95, 96, and 98 will be described below.

[0048] The moving area extraction unit 95 executes image processing on the frames, detects information on passageways inside buildings, sidewalks outdoors, white lines, etc., and extracts the moving area of ​​the detection target. The extraction method includes detecting obstacles, white lines, etc. by object recognition or straight line detection, and extracting the area surrounded by the obstacles or white lines as the moving area of ​​the detection target.

[0049] The surrounding object information analysis unit 96 analyzes the object detection results, analyzes the detection results of objects that exist in the surroundings other than the detection target in all frames, and acquires information such as the motion state and movement direction of the object. As an acquisition method, the flow described in the first embodiment is applied to objects other than the detection target, and a trajectory of the object is generated to acquire information such as whether the object is stopped or moving, and if it is moving, information on the direction of movement. In addition, a method of outputting the motion states of selected objects rather than all objects may be adopted, and a method of using only the motion states of objects with many highly reliable detection frames may be used.

[0050] The detection target moving direction output unit 98 estimates and outputs the moving direction of the detection target using the acquired moving area of ​​the detection target, the motion state of the surrounding objects, and the control data. Examples of the method of estimating the moving direction include a method of approximating the moving area of ​​the detection target to a rectangle and assuming that the moving object moves back and forth along the long side direction, and determining the moving direction of the detection target in these two directions, and a method of analyzing the motion state and control data of the surrounding objects, assuming that the detection target moves in the same way as the surrounding objects, determining that the detection target is stopped and setting the moving direction to 0 if the surrounding objects are stopped, and determining that the moving direction is moving if the surrounding objects are moving as the moving direction of the detection target.

[0051] A method for correcting information such as the presence probability map, the detection window reliability, and the detection window position from the movement direction of the detection target output by the detection target movement direction prediction unit 91 will be described. For example, as a method for correcting the presence probability map, when the presence probability map of the corresponding frame is determined using the presence probability maps of the previous and next frames in the presence probability map interpolation unit 22, a method for finely correcting the position of the map along the predicted movement direction of the detection target is considered. Also, as a method for correcting the detection window reliability, a method for obtaining the movement direction of the center coordinates of the detection window of the detection target between frames, and correcting the reliability of the detection window of the corresponding frame so that it is higher if it is similar to the predicted movement direction of the detection target. Also, as a method for correcting the detection window position, a method for preferentially selecting a frame in which the movement direction of the center coordinates of the detection window between frames and the predicted movement direction of the detection target are similar when the high reliability detection window selection unit 81 selects a high reliability detection window is considered. In addition to the above-described method, any method is not particularly limited as long as it can be used for correction by comparing the predicted movement direction of the detection target with the movement direction of the detection target used by the flow shown in the first embodiment.

[0052] In the second embodiment of the present invention, with the functional configuration described above, in an object tracking device that generates a trajectory of an object within a measurement range, the moving direction of the detection target is predicted in advance, and the detection frame is adjusted based on presence probability map information generated from the object detection results in each frame while utilizing the information on the moving direction, and the detection frame position is corrected according to the reliability calculated from the flow information, making it possible to output a highly accurate trajectory of the detection target.

[0053] <Example 3> FIG. 13 is a block diagram of the third embodiment of the present invention. The object tracking device 100 shown in Fig. 13 is a device that adjusts the strength of a detection frame based on presence probability map information generated from the object detection result of each frame, generates multiple trajectories of the detection target by correcting the detection frame position according to the reliability calculated from the flow information, and outputs a highly accurate trajectory of the detection target by selecting a highly accurate trajectory from among the multiple trajectories. In Fig. 13, the frame collection unit 3, the object detection unit 4, the presence probability map determination unit 5, the detection frame strength determination unit 6, the flow calculation unit 7, the detection frame reliability calculation unit 8, the detection frame position correction unit 9, and the trajectory generation unit 10 have the same functions as those in the first embodiment. The presence probability map determination unit 5 generates multiple presence probability maps of the detection target, the detection frame reliability calculation unit 8 calculates multiple reliabilities of the detection target, and the detection frame position correction unit 9 generates multiple detection frame positions of the detection target.

[0054] The trajectory generation unit 10 generates a plurality of trajectories using a plurality of presence probability maps, a plurality of detection window reliabilities, and a plurality of detection window position information. The trajectory storage unit 101 has a function of storing the generated trajectories, and the trajectory selection unit 102 has a function of allowing the user to manually select one of the stored trajectories using a GUI or the like.

[0055] In this embodiment, the trajectory generating unit 10 prepares multiple patterns of presence probability maps, detection frame reliability, and detection frame position correction methods for the same detection target in the same frame, and generates multiple trajectories for the same target. For example, the presence probability map determining unit 5 can generate multiple presence probability maps by using different algorithms for the generation or interpolation of the presence probability map, or by changing the pattern of the area selected by the presence probability map selecting unit 23. Furthermore, the detection frame reliability calculating unit 8 can have multiple reliabilities by changing the reliability calculation algorithm, and the detection frame position correcting unit 9 can generate multiple detection frame information by changing the pattern of the association method of the detection target association unit 80 or the pattern of the high-reliability detection frame selection.

[0056] In the trajectory selection unit 102, images of all frames with detection frames added after position correction for the detection target and the trajectories generated from the detection frames are displayed as a set on a GUI screen for each trajectory, allowing the user to select the trajectory that he or she considers optimal.

[0057] In the third embodiment of the present invention, with the functional configuration described above, in an object tracking device that generates a trajectory of an object within a measurement range, the algorithm in each processing block is changed while saving a plurality of trajectories generated by the flow of the first embodiment, and the trajectories are visualized on a GUI screen or the like, so that a user can select a highly accurate trajectory of the detection target.

[0058] Although the embodiments of the present invention have been described above in detail, the present invention is not limited to the above-described embodiments, and various design changes can be made without departing from the spirit of the present invention described in the claims. For example, the above-described embodiments have been described in detail to clearly explain the present invention, and the present invention is not necessarily limited to those having all of the configurations described. In addition, it is possible to replace a part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment. Furthermore, it is possible to add, delete, or replace a part of the configuration of each embodiment with another configuration. [Explanation of symbols]

[0059] Reference Signs List 1 object tracking device, 2 camera (sensor), 3 frame collection unit, 4 object detection unit, 5 presence probability map determination unit, 6 detection frame adjustment determination unit, 7 flow calculation unit, 8 detection frame reliability calculation unit, 9 detection frame position correction unit, 10 trajectory generation unit, 11 captured image, 12, 13 detection target, 14 detection frame, 15 detection frame information

Claims

1. An object tracking device that generates a trajectory of an object within a measurement range of a sensor, an object detection unit that detects an object in each of a plurality of frames acquired by the sensor; a detection frame reliability calculation unit that calculates the reliability of the detection frame based on inter-frame flow information within the detection frame of the detected object; a detection frame position correction unit that corrects detection frame information of a low-reliability detection frame that is less reliable than the high-reliability detection frame, using detection frame information of a high-reliability detection frame that is more reliable than a threshold; a trajectory generation unit that generates the trajectory using the corrected detection frame information; An object tracking device comprising:

2. The object tracking device according to claim 1 , wherein the detection frame reliability calculation unit calculates the reliability of the detection frame based on information on three-dimensional object feature points and road surface feature points within the detection frame.

3. 3. The object tracking device according to claim 1, wherein the detection frame position correction unit corrects the information on the position, vertical width, and horizontal width of the low-reliability detection frame using information on the positions, vertical width, and horizontal width of the plurality of high-reliability detection frames.

4. an existence probability map determination unit that generates an existence probability map indicating the existence probability of the object in each of the plurality of frames; a detection window addition / subtraction determination unit that determines whether to add or delete a detection window in each of the plurality of frames using the presence probability map; The object tracking device according to claim 1, further comprising:

5. The existence probability map determination unit an existence probability map creation unit that creates an existence probability map for each frame from the detection frame information output by the object detection unit; an existence probability map interpolation unit that interpolates information on existence probability maps of two or more frames before and after a target frame included in the plurality of frames to generate an interpolated existence probability map of the target frame; an existence probability map selection unit that compares the existence probability map with the interpolated existence probability map to select a final existence probability map; 5. The object tracking device according to claim 4, further comprising:

6. 6. The object tracking device according to claim 5, wherein the existence probability map creation unit creates the existence probability map using one or more pieces of information from a probability density function calculated by a normal distribution from position information of the detection frame of the object detected by the object detection unit and the reliability information, and object detection results using a plurality of different dictionaries.

7. The object tracking device according to claim 6, further comprising an object motion estimation unit that estimates the motion of the object.

8. 8. The object tracking device according to claim 7, wherein the object motion estimation unit estimates the motion of the object from one or more of the following information: trajectory information generated by the trajectory generation unit; object movement area candidate information; object motion information around the object; and information on a medium in which the sensor is installed.

9. 9. The object tracking device according to claim 8, wherein the presence probability map, the reliability of the detection frame, and the position of the detection frame are corrected again using the object movement information obtained by the object movement estimation unit.

10. the presence probability map determiner generates a plurality of presence probability maps of the object; the detection frame reliability calculation unit calculates a plurality of reliabilities of the object; the detection frame position correction unit corrects positions of a plurality of detection frames of the object; 5. The object tracking device according to claim 4, wherein the trajectory generation unit generates a plurality of trajectories using the plurality of existence probability maps, the plurality of reliabilities, and the plurality of detection window information, and selects any one of the trajectories.

Images