Mobile object recognition method and mobile object recognition device

Abstract

To achieve both accurate recognition and quick recognition of a mobile object included in images of a plurality of frames.SOLUTION: A mobile object recognition device 1 detects a mobile object on the basis of images of a plurality of frames captured in time series by reference and comparison cameras 3 and 4. The mobile object recognition device 1 comprises a feature point tracking unit 21, a variable time width speed estimation unit 24, and a high-precision movement flow determination unit 25. The feature point tracking unit 21 tracks a feature point in the images of the plurality of frames. The variable time width speed estimation unit 24 estimates a relative speed to a vehicle of an object corresponding to the feature point from a difference value between coordinates of the feature point in an image of a frame at a first time and coordinates of the feature point in an image of a frame at a second time preceding the first time by a prescribed time width. The high-precision movement flow determination unit 25 converts the relative speed of the object to an absolute speed. The prescribed time width is determined on the basis of a distance from the reference camera 3 to the object. The mobile object is detected on the basis of the absolute speed of the object.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to a moving object recognition method and a moving object recognition device.

Background Art

[0002] Patent Document 1 describes an invention of an object detection device and an object detection method. In the invention of Patent Document 1, a frame image used for generating an optical flow of a moving body is divided into a first region including an appearance point of the optical flow and a second region other than the first region. In the invention of Patent Document 1, the appearance point refers to a point at which a moving body appears to appear in the frame image. In the invention of Patent Document 1, extension lines of the optical flow of the moving body spreading radially intersect at the appearance point. In the invention of Patent Document 1, as an interval between a set of frame images used for generating the optical flow, a first frame interval of the first region is made larger than a second frame interval of the second region.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In the invention of Patent Document 1, when a distant moving object with a slow moving speed starts to appear in the second region, since the second frame interval is small and the moving speed is slow, the length of the optical flow of the moving object generated from a set of frame images becomes short. When the optical flow is short, it may be difficult to accurately detect the moving object appearing in the second region from the optical flow. In the invention of Patent Document 1, when a fast moving object hidden behind occlusion starts to appear in the first region without appearing in the second region, since the first frame interval is large, it takes time to generate the optical flow of the moving object. When it takes time to generate the optical flow, even if the moving speed of the moving object is fast, the detection of the moving object appearing in the first region may be delayed.

[0005] The present invention has been made in view of the above circumstances, and an object of the present invention is to achieve both accurate recognition and rapid recognition of a moving object appearing in a plurality of frames of images.

Means for Solving the Problems

[0006] A moving object recognition method according to one aspect of the present invention is executed by a computer that detects a moving object based on a plurality of frames of images captured in time series by an imaging device. In this object recognition method, feature points are tracked in a plurality of frames of images. In the object recognition method, the speed of the feature point is determined based on the difference value of the coordinates of the feature point on the image between the frame at the first time and the frame at the second time that is a predetermined time width before the first time, and a moving object is detected based on the speed of the feature point. The predetermined time width is determined based on the distance from the imaging device to the object corresponding to the feature point.

Effects of the Invention

[0007] According to the present invention, it is possible to achieve both accurate recognition and rapid recognition of a moving object appearing in a plurality of frames of images.

Brief Description of the Drawings

[0008]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

DETAILED DESCRIPTION OF THE INVENTION

[0009] Embodiments will be described with reference to the drawings. In the description of the drawings, the same parts are denoted by the same reference numerals and the description thereof is omitted.

[0010] FIG. 1 is a diagram showing an overall configuration example of a moving object detection device to which a moving object detection method according to an embodiment of the present invention is applied. With reference to FIG. 1, the moving object detection device 1 according to the present embodiment will be described. The moving object detection device 1 is mounted on a vehicle, for example. The vehicle is a vehicle having an automatic driving function, but may also be a vehicle without an automatic driving function. Further, the vehicle may be a vehicle capable of switching between an automatic driving function and a manual driving function. The automatic driving function is a function that automatically controls a plurality of vehicle control functions such as a steering control function, a braking force control function, and a driving force control function to perform autonomous driving, but may also be a function that automatically controls only some of the plurality of vehicle control functions to assist the driver's driving.

[0011] The moving object detection device 1 detects a moving object based on images of a plurality of frames captured in time series. The moving object detection device 1 includes an arithmetic processing unit 2, a reference camera 3 as an imaging device, and a comparison camera 4.

[0012] The arithmetic processing unit 2 is a general-purpose microcomputer including a CPU (Central Processing Unit), a memory, and an input / output unit. A computer program for causing the microcomputer to function as the moving object detection device 1 is installed in the microcomputer. By executing the computer program, the microcomputer functions as a plurality of information processing circuits included in the moving object detection device 1. In the present embodiment, an example is shown in which a plurality of information processing circuits included in the moving object detection device 1 are realized by software. Of course, it is also possible to configure a plurality of information processing circuits for executing each information processing by dedicated hardware. Further, the plurality of information processing circuits may be configured by individual hardware. Details of the arithmetic processing unit 2 will be described later.

[0013] The reference camera 3 is a camera that takes the main range in front of the vehicle as the imaging range. The comparison camera 4 is a camera that takes the main range in front of the vehicle as the imaging range. The reference and comparison cameras 3 and 4 are equipped with solid-state imaging devices such as CCD (Charge Coupled Device) and CMOS (Complementary Metal Oxide Semiconductor). The reference and comparison cameras 3 and 4 are a pair of cameras that constitute a stereo camera, and are arranged at a predetermined distance in the vehicle width direction of the vehicle. The reference and comparison cameras 3 and 4 perform imaging at synchronized timings and output images of a plurality of frames captured in time series at a predetermined frame rate.

[0014] The arithmetic processing unit 2 performs predetermined processing based on the images acquired from the reference and comparison cameras 3 and 4. The arithmetic processing unit 2 includes, as a plurality of information processing circuits, a feature point tracking unit 21, a parallax measurement unit 22, a feature point history management unit 23, a variable time-width speed estimation unit 24, a high-precision movement flow determination unit 25, a fixed time-width speed estimation unit 26, and a low-latency movement flow determination unit 27. The arithmetic processing unit 2 further includes, as a plurality of information processing circuits, a feature point update unit 28, a host vehicle behavior estimation unit 31, a speed determination unit 32, and a clustering unit 33. The arithmetic processing unit 2 further includes a storage unit 30 as a data storage. The storage unit 30 may be, for example, an SSD (Solid State Drive) or an HDD (Hard Disk Drive). Information on two models, information on a fixed speed threshold, and information on a fixed time width are stored in the storage unit 30. The two models are a detection speed resolution model and a pixel movement amount resolution model. Each piece of information will be described later.

[0015] The feature point tracking unit 21 is a tracking unit that acquires images from the reference camera 3. The feature point tracking unit 21 extracts a template image including feature points from the image of a certain frame, and tracks the feature points in the images of multiple frames by template matching that identifies a region with a high correlation degree with the template image from the images of frames after a certain frame. When there are multiple feature points in the image, the feature point tracking unit 21 tracks each feature point. Note that the feature point tracking unit 21 may extract a partial region including the feature points in the image as the template image, or may extract the entire region of the image as the template image. The template image may be updated corresponding to the change in the content of the image of the reference camera 3 from which the template image is extracted over time. The update of the template image will be described later.

[0016] The disparity measurement unit 22 is a disparity measurement unit that acquires images from the reference and comparison cameras 3 and 4 respectively. The disparity measurement unit 22 performs stereo matching processing using a pair of images of the same frame output from the reference and comparison cameras 3 and 4. The disparity measurement unit 22 measures the disparity for each frame by stereo matching processing, and identifies the distance of the object corresponding to the feature point from the measured disparity. The distance of the object can be, for example, the distance from the reference camera 3 to the object. The disparity measurement unit 22 calculates the disparity for the entire image region, but may also obtain the disparity for each feature point.

[0017] The feature point history management unit 23 manages the information of the feature points for each frame tracked by the feature point tracking unit 21 for each feature point. The feature point history management unit 23 stores the history of the feature points tracked by the feature point tracking unit 21 as feature point information in the history management area and manages it for each frame. The feature point information includes the coordinate information in the image of each frame and the disparity information of the feature points measured by the disparity measurement unit 22 in each frame. The history management area can be secured, for example, in the storage unit 30. The feature point history management unit 23 can perform the processing related to the management of the feature point history using, for example, a buffer area provided in the memory of the arithmetic processing unit 2.

[0018] The variable time width speed estimation unit 24 and the fixed time width speed estimation unit 26 acquire the history of feature points of a plurality of frames stored in the history management area of the storage unit 30 for frames having a cycle of a time width corresponding to each of the units 24 and 26. The variable time width speed estimation unit 24 and the fixed time width speed estimation unit 26 can cause the feature point history management unit 23 to select and acquire the history of feature points of the frames corresponding to the cycles of their respective time widths from the storage unit 30. The variable time width speed estimation unit 24 and the fixed time width speed estimation unit 26 estimate the relative speed of the object corresponding to the feature point with respect to the vehicle from the history of feature points of the frames corresponding to the cycles of their respective time widths. The high-precision movement flow determination unit 25 and the low-latency movement flow determination unit 27 convert the relative speed of the object corresponding to the feature point into an absolute speed, and determine whether the object corresponding to the feature point is a moving object based on their respective speed thresholds. Hereinafter, each of the units 24 to 27 will be described in detail.

[0019] The variable time width speed estimation unit 24 acquires the history of feature points of the frames corresponding to the cycle of a predetermined time width via the feature point history management unit 23. The predetermined time width is variable. The variable time width speed estimation unit 24 determines the predetermined time width based on the distance specified by the parallax measurement unit 22 for the object corresponding to the feature point for which the history is acquired. The variable time width speed estimation unit 24 can determine the predetermined time width using the pixel movement amount resolution model of the storage unit 30.

[0020] When the object imaged by the reference camera 3 moves, if the position of the pixel of the object shown in the image moves, the movement of the object can be recognized on the image. The moving distance of the object when the object on the image moves by one pixel is shorter for an object closer to the reference camera 3 and longer for an object farther from the reference camera 3. For example, if the distances of the objects are 10 meters, 20 meters, and 40 meters and increase, the frame intervals required to recognize the movement of the object on the image increase by 1 times, 2 times, and 4 times. The pixel movement amount resolution model of the storage unit 30 is a model in which the frame intervals at which the movement of the object shown on the screen imaged by the reference camera 3 can be recognized are defined for each distance of the object and modeled.

[0021] In the pixel movement resolution model, a variable time width is defined for each distance of an object. The variable time width represents a predetermined variable time width in terms of the time interval per frame. The variable time width indicates the minimum frame interval at which the movement of the feature point is recognized on the image. The pixel movement resolution model is a movement amount model that defines the resolution of the movement amount at which the movement of the feature point is recognized on the image for each distance.

[0022] FIG. 2 is a diagram showing an example of the content of the pixel movement resolution model. In FIG. 2, when the movement of an object at a distance of 10 meters can be recognized on the screen at an interval of 1 frame, it shows that in order to recognize objects at distances of 20 meters and 40 meters on the screen, frame intervals of 2 frames and 4 frames are respectively required. The frame interval corresponding to the distance of the object in the pixel movement resolution model may be set based on experimental data, or may be set theoretically from the resolution and focal length of the reference camera 3. The frame interval set in the pixel movement resolution model may be the value obtained by experiment or calculation, or may be a value doubled based on the sampling theorem.

[0023] The variable time width speed estimation unit 24 acquires, as the distance of the object corresponding to the feature point, the distance specified from the parallax by the parallax measurement unit 22 for the feature point tracked by the feature point tracking unit 21. The variable time width speed estimation unit 24 determines a predetermined time width corresponding to the acquired distance of the object based on the pixel movement resolution model. The variable time width speed estimation unit 24 acquires, via the feature point history management unit 23, the history of the feature points of the frame corresponding to the period of the determined predetermined time width. The history of the feature points acquired by the variable time width speed estimation unit 24 includes the coordinates of the feature points on the image in the frame at the first time and the coordinates of the feature points on the image in the frame at the second time, which is a predetermined time width before the first time. The variable time width speed estimation unit 24 generates an optical flow of the feature points from the difference value between the coordinates of the feature points at the first time and the coordinates of the feature points at the second time, and estimates the relative speed of the object corresponding to the feature points with respect to the vehicle.

[0024] FIG. 3 is a diagram showing an example of the relationship between the predetermined time width determined by the variable time width speed estimator 24 and the first time and the second time at which the variable time width speed estimator 24 obtains the difference value of the coordinates of the feature points between frames. T1 to T7 on the horizontal axis of FIG. 3 indicate the timings at which the reference camera 3 captures images of a plurality of frames. For example, when the predetermined time width is 4 frames, the variable time width speed estimator 24 can estimate the relative speed of the object corresponding to the feature point with respect to the vehicle after the time t5 when the image captured by the reference camera 3 occurred more than 4 frames ago. The variable time width speed estimator 24 obtains, for example, the history of the feature points for both frames at times t5 and t1 corresponding to the first time and the second time, and can estimate the relative speed of the object corresponding to the feature point from the difference value of the coordinates of the feature points at times t5 and t1. The variable time width speed estimator 24 can estimate, for example, the relative speed of the object corresponding to the feature point from the difference value of the coordinates of the feature points at times t6 and t2, times t7 and t3 corresponding to the respective first time and second time after time t6 and after time t7. The variable time width speed estimator 24 estimates the relative speed of the object corresponding to the feature point for all the feature points tracked by the feature point tracking unit 21. The variable time width speed estimator 24 can guarantee the resolution of the image for observing the relative speed of the object with respect to the vehicle based on the image of the reference camera 3 by determining a predetermined time width corresponding to the distance of the object.

[0025] The high-precision movement flow determination unit 25 performs high-precision movement flow determination. In high-precision movement flow determination, the high-precision movement flow determination unit 25 converts the relative speed estimated by the variable time-width speed estimation unit 24 for the object corresponding to each feature point into an absolute speed based on the coordinates of the position of the vehicle in the world coordinate system and the distance of the object corresponding to each feature point. The coordinates of the vehicle in the world coordinate system can be obtained, for example, by a GNSS (Global Navigation Satellite System) sensor (not shown) mounted on the vehicle. The relative speed of the object includes the movement component of the feature point generated on the screen due to the behavior of the vehicle. The own vehicle behavior estimation unit 31 estimates this movement component of the feature point. Since the own vehicle behavior estimation unit 31 can obtain the coordinates of the vehicle in the world coordinate system by the GNSS sensor, the movement component of the feature point generated on the screen due to the behavior of the vehicle can be estimated, for example, based on the blur of the captured image by the reference camera 3. The high-precision movement flow determination unit 25 refers to the estimation result of the own vehicle behavior estimation unit 31 and can convert the relative speed of each object into an absolute speed excluding the influence due to the behavior of the vehicle based on the coordinates of the position of the vehicle in the world coordinate system and the distance of the object corresponding to each feature point. The variable time-width speed estimation unit 24 and the high-precision movement flow determination unit 25 can constitute a determination unit for determining the speed of the object corresponding to the feature point.

[0026] The high-precision movement flow determination unit 25 determines an object with an absolute speed equal to or higher than the fixed speed threshold of the storage unit 30 as a moving object, and notifies the clustering unit 33 of the information of the object determined as the moving object. The information of the object notified by the high-precision movement flow determination unit 25 includes the information of the optical flow of the feature point corresponding to the object.

[0027] The fixed-time-width speed estimation unit 26 acquires the history of the feature points of the frame corresponding to the period of the fixed time width in the storage unit 30 via the feature point history management unit 23. The fixed time width indicates the minimum time width for detecting a moving object by the movement of the feature points on the image. The fixed time width is set commonly regardless of the distance of the object. The fixed-time-width speed estimation unit 26 generates an optical flow of the feature points from the difference value of the coordinates of the feature points at both times before and after the fixed time width, and estimates the relative speed of the object corresponding to the feature points with respect to the vehicle. The fixed-time-width speed estimation unit 26 estimates the relative speed of the object corresponding to the feature points for all the feature points tracked by the feature point tracking unit 21. The low-latency moving flow determination unit 27 performs low-latency moving flow determination. In the low-latency moving flow determination, the low-latency moving flow determination unit 27 refers to the estimation result of the host vehicle behavior estimation unit 31, and based on the coordinates of the position of the vehicle in the world coordinate system, converts the relative speed of each object into an absolute speed excluding the influence by the behavior of the vehicle.

[0028] FIG. 4 is a diagram showing an example of the relationship between the fixed time width and the times before and after the fixed time width when the fixed-time-width speed estimation unit 26 obtains the difference value of the coordinates of the feature points between frames. In FIG. 4, the case where the fixed time width is an interval of one frame is shown. The fixed-time-width speed estimation unit 26 can estimate the relative speed of the object corresponding to the feature points from the difference value of the coordinates of the feature points at two consecutive times t2, t1, time t3, t2, and time t4, t3, respectively.

[0029] The low-latency moving flow determination unit 27 acquires the distance specified from the parallax by the parallax measurement unit 22 as the distance of the object corresponding to the feature points for the feature points whose history is acquired by the fixed-time-width speed estimation unit 26. The low-latency moving flow determination unit 27 determines a variable speed threshold corresponding to the acquired distance of the object. The variable speed threshold varies depending on the distance of the object. The variable speed threshold is a threshold that defines the minimum speed for each distance of the object to be detected as a moving object. The low-latency moving flow determination unit 27 can determine the variable speed threshold using the detection speed resolution model in the storage unit 30.

[0030] When an object on the image moves by one pixel, the moving speed of the object is slower for an object closer to the reference camera 3 and faster for an object farther from the reference camera 3. For example, if the distance of the object increases to 10 meters, 20 meters, and 40 meters, the speed of the object whose movement can be recognized on the image increases to 1 times, 2 times, and 4 times. The detection speed resolution model in the memory unit 30 defines and models the moving speed at which the movement of the object shown on the screen captured by the reference camera 3 can be recognized for each distance of the object.

[0031] A variable speed threshold is defined for each distance of the object in the detection speed resolution model. The variable speed threshold indicates the minimum moving speed of an object that can be detected as a moving object. The detection speed resolution model is a speed model that defines the resolution of the speed at which the movement of the feature points is recognized on the image for each distance.

[0032] FIG. 5 is a diagram showing an example of the content of the detection speed resolution model. In FIG. 5, when the fixed time width is one frame, it shows that the minimum speed at which the movement of an object at a distance of 10 meters can be recognized on the screen is plus or minus (±) 1.0 m / s. The variable speed threshold corresponding to the distance of the object in the detection speed resolution model may be set based on experimental data or may be set theoretically from the resolution and focal length of the reference camera 3. The variable speed threshold set in the detection speed resolution model can be obtained, for example, by experiment or calculation.

[0033] The low-latency movement flow determination unit 27 determines an object whose absolute speed is equal to or higher than the variable speed threshold corresponding to the distance as a moving object, and notifies the speed determination unit 32 of the information of the object determined as a moving object. The information of the object notified by the low-latency movement flow determination unit 27 includes the information of the absolute speed converted from the relative speed by the low-latency movement flow determination unit 27 for that object.

[0034] Instead of obtaining the distance specified by the disparity measurement unit 22, the variable time-width speed estimation unit 24 and the fixed time-width speed estimation unit 26 may obtain the distance of an object by another method. For example, when obtaining the relative speed of an object, information on the disparity of feature points regarding the images of the reference camera 3 and the comparison cameras 4 at that time may be obtained from the history management area of the storage unit 30, and the distance of the object corresponding to the feature points may be calculated from the obtained disparity information. For example, the distance of the object corresponding to the feature points in the image estimated using a Kalman filter or the like from a plurality of frames of images of the reference camera 3 may be obtained.

[0035] The high-precision motion flow determination by the high-precision motion flow determination unit 25 and the low-latency motion flow determination by the low-latency motion flow determination unit 27 may be performed, for example, for each imaging cycle of the image by the reference camera 3, or may be performed at intervals of a cycle that is a multiple of the imaging cycle. The high-precision motion flow determination unit 25 and the low-latency motion flow determination unit 27 can perform processes related to the detection of moving objects, such as the determination of moving objects and the clustering of feature points corresponding to the objects determined to be moving objects, during periods when each determination is not performed. The high-precision motion flow determination unit 25 and the low-latency motion flow determination unit 27 may perform processing following the imaging of the images of the frames required for each determination by the reference camera 3, or may perform processing in a separate process in parallel with the processing related to the imaging of the reference camera 3.

[0036] The high-precision motion flow determination unit 25 can accurately obtain the absolute speed of an object from an optical flow with a predetermined time width that can recognize the movement of feature points on the image, generated by the variable time-width speed estimation unit 24. However, since the predetermined time width is equal to or greater than the fixed time width, in order for the variable time-width speed estimation unit 24 to obtain the absolute speed of the object, it is necessary to acquire images before and after the predetermined time width with a waiting time equal to or greater than the fixed time width. For this reason, the determination process by the high-precision motion flow determination unit 25 is a high-latency process compared to the determination process by the low-latency motion flow determination unit 27.

[0037] The low-latency motion flow determination unit 27 acquires images of frames before and after a fixed time width with a waiting time of the fixed time width that is equal to or less than a predetermined time width from the optical flow generated by the fixed-time-width speed estimation unit 26, and can obtain the absolute speed of an object with low latency. If the fixed time width is set to the imaging period for one frame of the image by the reference camera 3, the delay from the acquisition of the image to obtaining the absolute speed of the object can be minimized. However, the minimum value of the absolute speed of the object that the speed estimation unit 26 can obtain from the optical flow with a fixed time width increases as the distance of the object corresponding to the feature point becomes longer. For this reason, the determination process by the low-latency motion flow determination unit 27 is a lower-precision process compared to the determination process by the high-precision motion flow determination unit 25.

[0038] When the object with the absolute speed converted from the relative speed by the low-latency motion flow determination unit 27 corresponds to a moving object, the speed determination unit 32 determines the speed to be adopted as the absolute speed of the object. The method for the speed determination unit 32 to determine the speed to be adopted as the absolute speed will be described later. The speed determination unit 32 notifies the clustering unit 33 of the information of the object for which the speed to be adopted as the absolute speed has been determined. The information of the object notified by the speed determination unit 32 includes the information of the optical flow of the feature point corresponding to the object.

[0039] The clustering unit 33 clusters the feature points corresponding to the objects corresponding to the moving objects based on the optical flow of the feature points notified from the variable-time-width speed estimation unit 24 or the speed determination unit 32. The clustering unit 33 can detect one or more moving objects from the objects corresponding to the feature points by this clustering.

[0040] Hereinafter, with reference to FIGS. 6 and 7, the moving object recognition method executed by the moving object recognition device 1 of the present embodiment will be described. The moving object recognition method is roughly classified into a registration process of registering the information of the feature points shown in FIG. 6 and a determination process of discriminating the moving object shown in FIG. 7. In the present embodiment, the registration process and the determination process are performed in parallel as separate processes, but the registration process and the determination process may be performed serially.

[0041] First, referring to FIG. 6, the registration process will be described. The registration process is executed in accordance with the imaging cycle of the image. The feature point tracking unit 21 acquires an image from the reference camera 3, and the parallax measurement unit 22 acquires images from the reference and comparison cameras 3 and 4, respectively (S10). The feature point tracking unit 21 tracks the already detected feature points and identifies the coordinates of the feature points in the current image (S11). Also, the feature point tracking unit 21 detects new feature points in the current image and identifies the coordinates of the feature points in the current image (S12). The parallax measurement unit 22 acquires images from the reference and comparison cameras 3 and 4, respectively. The parallax measurement unit 22 calculates the parallax of each feature point from the parallax image (S13). The feature point history management unit 23 registers the coordinates of the feature points identified by the feature point tracking unit 21 and the parallax identified by the parallax measurement unit 22 in the history management area of the storage unit 30 (S14). The feature point update unit 28 updates the template for the feature point tracking unit 21 to detect feature points (S15). The details of the template update process will be described later.

[0042] Referring to FIG. 7, the determination process will be described. First, the variable time-width speed estimation unit 24 determines whether the processing cycle has been reached (S20). The processing cycle is the same as the imaging cycle of the image, but may be a cycle larger than the imaging cycle of the image so that one or more images are decimated. When the processing cycle is reached, the processes of steps S21 to S26, or the processes of steps S21, S22, and steps S27 to S35 are performed. The processes of steps S21 to S26, or the processes of steps S21, S22, and steps S27 to S35 are executed for each feature point. On the other hand, when the processing cycle is not reached, the process returns to step S21.

[0043] The variable time-width velocity estimation unit 24 determines a variable time-width as a predetermined time-width for obtaining the velocity of the object corresponding to the feature point (S21). The variable time-width velocity estimation unit 24 uses the feature point history management unit 23 to determine whether there is a history of the feature point up to the time before the variable time-width from the present (S22). If there is no history up to the time before the variable time-width, the process proceeds to the process of step S27 described later. If there is a history up to the time before the variable time-width, the variable time-width velocity estimation unit 24 calculates the relative velocity of the object corresponding to the feature point with respect to the vehicle in the horizontal direction of the image of the reference camera 3 (S23). The high-precision motion flow determination unit 25 converts the relative velocity of the object into an absolute velocity (S24), and labels the feature points corresponding to the objects whose absolute velocity is equal to or higher than the fixed velocity threshold as moving feature points (S25). The velocity determination unit 32 checks whether the feature points corresponding to the objects converted into the absolute velocity in step S24 correspond to moving objects (S26). The velocity determination unit 32 can check whether it corresponds to a moving object depending on whether the high-precision motion flow determination unit 25 labels the feature point as a moving feature point. If it does not correspond to a moving object, the process proceeds to the process of step S28 described later. If it corresponds to a moving object, if there are feature points for which the processes of steps S21 to S26, or the processes of steps S21, S22, and S27 to S35 have not been completed, return to S21. If the processes of steps S21 to S26, or the processes of steps S21, S22, and S27 to S35 have been completed for all feature points, the process proceeds to the process of step S36 described later.

[0044] In step S27, the fixed-time-width speed estimator 26 uses the feature point history management unit 23 to determine whether there is a history of feature points up to a fixed time width before the current time. If there is no history up to the fixed time width before, it returns to S21. If there is a history up to the fixed time width before, the process proceeds to the process of S28. In step S28, the fixed-time-width speed estimator 26 calculates the relative speed of the object corresponding to the feature point with respect to the vehicle in the horizontal direction of the image of the reference camera 3. The low-latency movement flow determination unit 27 converts the relative speed of the object into an absolute speed (S29). The low-latency movement flow determination unit 27 determines a variable speed threshold for determining an object having a speed corresponding to a moving object (S30). The low-latency movement flow determination unit 27 labels the feature points corresponding to the objects whose absolute speed is equal to or higher than the variable speed threshold as moving feature points (S31).

[0045] The arithmetic processing unit 2 increases the fixed time width for the feature points labeled as moving feature points and recalculates the absolute speed of the object corresponding to the feature points (S32). The arithmetic processing unit 2 confirms whether the object with the recalculated absolute speed corresponds to a moving object based on the recalculated absolute speed of the object (S33). The increased fixed time width may be, for example, a time width having a predetermined time width corresponding to the distance of the object as a maximum value. The maximum value of the predetermined time width corresponding to the distance of the object may be used as the increased fixed time width, and the variable time width speed estimator 24 and the high-precision movement flow determination unit 25 may be used to recalculate the absolute speed in step S32. In this case, the arithmetic processing unit 2 can cause the high-precision movement flow determination unit 25 to confirm whether the recalculated absolute speed of the object is equal to or higher than the fixed speed threshold, and perform the confirmation of whether it corresponds to a moving object in step S33.

[0046] If the object with the absolute velocity recalculated by the arithmetic processing unit 2 corresponds to a moving object, the velocity determination unit 32 adopts the recalculated absolute velocity (S34). If the object with the absolute velocity recalculated by the arithmetic processing unit 2 does not correspond to a moving object, the velocity determination unit 32 adopts the original absolute velocity converted from the relative velocity in step S29 (S35). If there are feature points for which the processing from step S21 to step S26, or the processing from step S21, step S22, step S27 to step S35 has not been completed, the process returns to S21. If the processing from step S21 to step S26, or the processing from step S21, step S22, step S27 to step S35 has been completed for all feature points, the process proceeds to the processing of step S36.

[0047] In step S36, the clustering unit 33 clusters the optical flow of the feature points notified by the high-precision movement flow determination unit 25 or the velocity determination unit 32, and detects one or more moving objects. When the velocity determination unit 32 adopts the absolute velocity recalculated using the high-precision movement flow determination unit 25 in step S34, the clustering unit 33 preferentially uses the result of the high-precision movement flow determination over the low-latency movement flow determination to detect a moving object. In this case, an object with an absolute velocity determined by a predetermined time width set based on the pixel movement amount resolution model and the fixed velocity threshold is detected as a moving object with priority over an object with a velocity equal to or higher than the variable velocity threshold.

[0048] Next, the template update process in step S15 of FIG. 6 will be described. Generally, by performing feature point tracking at a high cycle, the change in the shape of the peripheral pixels of the feature point becomes small and the tracking accuracy becomes high. Also, in feature point tracking by template matching, there is an advantage that the search area can be set narrow. However, when tracking a slow object whose movement between one frame is less than one pixel for the template for performing feature point tracking, there is a problem that correct tracking cannot be performed if updated every frame. To counter this problem, an optimal update cycle is set for each distance of the object corresponding to the feature point based on the minimum target detection speed recognized as a sensor. For example, the number of frames required for the feature point corresponding to the object with the target speed to move one pixel on the image at the observed distance is set as the update frame of the template image of the corresponding feature point. In this case, the update of the template image can be performed, for example, at frame intervals that are multiples of a predetermined time width corresponding to the distance of the object.

[0049] FIG. 8 is a diagram showing an example of the content of the template update process. FIG. 8 shows a case where an object with a target speed takes two frames of time to move one pixel at the observed distance. In the time series of FIG. 8, for example, 100% template update is performed at times t1 and t3, which are every two frames of the imaging cycle of the image by the reference camera 3. Also, the frame interval is not fixed and may be changed based on the distance of the target feature point.

[0050] Alternatively, instead of thinning the frame intervals and updating the template image all at once, the template image may be updated for each frame while taking a weighted average. The weighting can be performed on at least one of, for example, a past template image extracted from the image before updating the template image and a new template image extracted from the image at the time of updating the template image. The weighting coefficient of the weighting can be set such that, for example, the weight of the new template image is greater than that of the past template image. For example, when designing such that the information of the past template image is halved at intervals of half of the update frame interval f, the weights can be determined by the following formula when the weight of the past template image is a and the weight of the new template image around the feature point to be tracked in the latest image is b. The weights a and b can be stored, for example, in the memory or storage unit 30 of the arithmetic processing unit 2.

Equation

[0051] FIG. 9 is a diagram showing another example of the content of the template update process. FIG. 9 shows a case where template update is performed for each frame of the imaging cycle of the image by the reference camera 3. In the time series of FIG. 9, for example, at times t1, t2, t3, and t4 of each frame of the imaging cycle of the image by the reference camera 3, template image updates are performed to average the past and new template images with weights a = 25% and b = 75%, respectively.

[0052] Note that this update is not limited to feature point tracking using template matching, and may be applied to the target pixel to be tracked and the target area of the gradient method that constitutes the surrounding pixels having the same movement as it in feature point tracking by the gradient method.

[0053] In this embodiment, the difference value of the coordinates of the feature points between two frames, i.e., the current frame and the past frame, is obtained with a variable predetermined time width corresponding to the distance of the object, and a moving object is detected based on the absolute speed of the object corresponding to the feature points obtained from this difference value. Since the movement of the feature points between frames is recognized with a predetermined time width determined corresponding to the distance of the object, the time width that guarantees the resolution of the image for observing the relative speed of the object with respect to the vehicle based on the image of the reference camera 3 can be appropriately set based on the distance of the object.

[0054] In this embodiment, the predetermined time width is set based on a pixel movement amount resolution model that defines the resolution of the frame interval at which the movement of the feature points is recognized on the image for each distance, and a fixed speed threshold value preset as the minimum speed of the object to be detected as a moving object. By using the pixel movement amount resolution model for setting the predetermined time width, the predetermined time width can be set to an optimal value based on the target performance of detecting the moving object reflected in the image of the reference camera 3. Also, by obtaining the difference value of the coordinates of the feature points between the images before and after the predetermined time width set as the optimal value and determining the speed of the object corresponding to the feature points, it is possible to achieve both accurate recognition and rapid recognition of the moving object reflected in the images of multiple frames.

[0055] In this embodiment, the feature points corresponding to the objects with an absolute speed equal to or higher than the fixed speed threshold are clustered to detect the moving object. By limiting the clustering target to the feature points corresponding to the objects with an absolute speed equal to or higher than the fixed speed threshold, it is possible to cluster the moving singular points corresponding to the moving object and improve the detection accuracy of the moving object by clustering.

[0056] In this embodiment, the distance of the object corresponding to the feature points is specified based on the parallax of a pair of images of the same frame obtained from the reference and comparison cameras 3 and 4 constituting the stereo camera. In this case, the distance of the object that is the basis for setting the predetermined time width can be accurately specified, and the predetermined time width can be set to an appropriate value.

[0057] In this embodiment, the movement component of the feature points generated on the screen is estimated based on the behavior of the vehicle, and the estimated component is excluded from the difference value of the coordinates of the feature points in the images between frames to estimate the relative speed of the object corresponding to the feature points. Thereby, the absolute speed of the object converted from the relative speed can be regarded as the speed from which the influence of the vehicle behavior is excluded, and the absolute speed from which the influence of the host vehicle is removed can be obtained.

[0058] In this embodiment, based on the resolution of the frame interval in which the movement of the feature points defined in the pixel movement amount resolution model for the distance of the object is recognized on the image and the speed of the object calculated by the fixed time width, a variable speed threshold for each distance of the object to be the moving object is set. When the absolute speed of the object is equal to or higher than the variable speed threshold corresponding to the distance of the object, the object is detected as a moving object. For this reason, based on the information of the absolute speed of the object obtained with low latency by the low latency movement flow determination unit 27, a moving object can be detected quickly.

[0059] In this embodiment, an object with an absolute speed determined by a predetermined time width set based on the pixel movement amount resolution model and the fixed speed threshold is detected as a moving object with priority over an object with a speed equal to or higher than the variable speed threshold. In this case, the result of the high-precision movement flow determination with higher accuracy than the low latency movement flow determination can be preferentially used to detect a moving object.

[0060] In this embodiment, for the feature points labeled as moving feature points, the fixed time width is increased to recalculate the absolute speed of the object corresponding to the feature points. Thereby, among the absolute speeds of the objects obtained by the arithmetic processing unit 2, the absolute speeds with high accuracy can be preferentially used for detecting moving objects.

[0061] In this embodiment, the template image of the region including the feature points extracted from the image is updated at a frame interval that is a multiple of the predetermined time width. Thereby, based on the distance of the object and the target performance for detecting the moving object reflected in the image of the reference camera 3, the template image can be updated at an optimal interval.

[0062] In this embodiment, weighting is performed on at least one of a past template image extracted from an image before updating the template image and a new template image extracted from the image when updating the template image. The weighting coefficient for weighting can be set such that, for example, the weight of the new template image is greater than that of the past template image. Thereby, changes due to the update of the template image can be moderated, and minute changes that occur over time in the image of the reference camera 3 can be reflected in the updated template image.

[0063] In this embodiment, the feature point update unit 28 that updates the template image is provided in the arithmetic processing unit 2, but the feature point update unit 28 may be omitted. In this embodiment, the fixed time-width speed estimation unit 26 and the low-latency movement flow determination unit 27 are provided in the arithmetic processing unit 2, but the fixed time-width speed estimation unit 26 and the low-latency movement flow determination unit 27 may be omitted. In that case, the storage unit 30 does not have to store the pixel movement amount resolution model and the information on the fixed time width. Further, when the fixed time-width speed estimation unit 26 and the low-latency movement flow determination unit 27 are omitted, each process from step S26 to step S35 in the flowchart of FIG. 7 is omitted. If there is no history up to the variable time width at step S22, the process returns to step S21. After the labeling at step S25, if there are feature points for which the processes from step S21 to step S25 have not been completed, the process returns to S21. If the processes from step S21 to step S25 have been completed for all feature points, the process proceeds to the process of step S36.

[0064] As described above, although the embodiments of the present invention have been described, it should not be understood that the discussions and drawings forming a part of this disclosure limit this invention. Various alternative embodiments, examples, and operation techniques will be apparent to those skilled in the art from this disclosure.

Explanation of Reference Numerals

[0065] 1 Moving object detection device 3 Reference camera (imaging device, stereo camera) 4 Comparison camera (imaging device, stereo camera) 21 Feature point tracking unit (tracking unit) 24 Variable time-width speed estimation unit (judgment unit) 25 High-precision movement flow judgment unit (judgment unit) 26 Fixed time-width speed estimation unit 27 Low-latency movement flow judgment unit 31 Host vehicle behavior estimation unit 32 Speed determination unit 33 Clustering unit

Claims

1. A moving object recognition method executed by a computer that detects a moving object based on images of a plurality of frames captured in time series by an imaging device, tracking feature points in the images of the plurality of frames, determining the speed of the object corresponding to the feature point from the difference value between the coordinates of the feature point on the image in the frame at the first time and the coordinates of the feature point on the image in the frame at the second time that is a predetermined time width before the first time, detecting the moving object based on the speed, including wherein the predetermined time width is determined based on the distance from the imaging device to the object corresponding to the feature point, Moving object recognition method.

2. The predetermined time width is set based on a movement amount model in which the resolution of the frame interval in which the movement of the feature point is recognized on the image is defined for each distance, and a fixed speed threshold set in advance as the minimum speed of the object to be detected as the moving object. The moving object recognition method according to Claim 1.

3. Detecting the moving object by clustering the feature points corresponding to the object whose speed is equal to or higher than the fixed speed threshold. The moving object recognition method according to Claim 2.

4. The imaging device is composed of a stereo camera including a pair of cameras, wherein the distance is specified based on a pair of images of the same frame output from the pair of cameras. The moving object recognition method according to Claim 1.

5. The imaging device is mounted on a vehicle, and converting the speed into an absolute speed based on the behavior of the vehicle. The moving object recognition method according to Claim 1.

6. Setting a variable speed threshold that is the minimum speed for each distance of the object to be the moving object based on the resolution defined in the movement amount model for the distance and the speed calculated by a fixed time width set in common regardless of the distance, and when the speed determined by the difference value is equal to or higher than the variable speed threshold corresponding to the distance of the object corresponding to the feature point, detecting the object corresponding to the feature point as the moving object. The moving object recognition method according to Claim 2.

7. Detecting the object with the speed determined by the predetermined time width set based on the movement amount model and the fixed speed threshold as the moving object with priority over the object with a speed equal to or higher than the variable speed threshold. The moving object recognition method according to Claim 6.

8. Increasing the fixed time width to the predetermined time width and recalculating the speed The moving object recognition method according to claim 6

9. Updating a template image of a region including the feature points extracted from the image at frame intervals that are multiples of the predetermined time width The moving object recognition method according to claim 1

10. At least one of the past template image extracted from the image of the frame before updating the template image and the new template image extracted from the image of the frame at the time of updating the template image is weighted with a weight coefficient such that the weight of the new template image is greater than that of the past template image, and the template image is updated to an image obtained by averaging the weighted past template image and the new template image The moving object recognition method according to claim 9

11. A moving object recognition device that detects a moving object based on images of a plurality of frames captured in time series by an imaging device, comprising A tracking unit that tracks feature points in the images of the plurality of frames A determination unit that determines the speed of the object corresponding to the feature point from the difference value between the coordinates of the feature point on the image in the frame at the first time and the coordinates of the feature point on the image in the frame at the second time that is a predetermined time width before the first time The predetermined time width is determined based on the distance from the imaging device to the object corresponding to the feature point The moving object is detected based on the speed Moving object recognition device

Images