Object tracking device, object tracking method, and program

Abstract

To provide an object tracking device capable of improving the tracking accuracy of objects existing around a vehicle.SOLUTION: An object tracking device in an embodiment includes: an image acquisition unit that acquires image data including multiple image frames captured in time series by an imaging unit mounted on a moving body; a recognition unit that recognizes an object from the images acquired by the image acquisition unit; an area setting unit that sets an image region that includes the object recognized by the recognition unit; and an object tracking unit that tracks the object based on the amount of time series change of the image area set by the area setting unit. The area setting unit sets the position and the size of the image region to track the object in future image frames based on the amount of time series change of the image area including the object in the past image frame and behavior information of the moving body.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an object tracking device, an object tracking method, and a program. [Background technology]

[0002] A technology has been known in the past that performs signal processing based on pre-learned results on image data of the area ahead of the vehicle captured by an on-board camera, and detects objects present around the vehicle (see, for example, Patent Document 1). In Patent Document 1, a deep neural network (DNN) such as a convolutional neural network is used to detect objects present around the vehicle. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2021-144689 Summary of the Invention [Problem to be solved by the invention]

[0004] However, when tracking an object using images captured by an imaging unit mounted on a moving body, as in conventional technology, the appearance and amount of movement of the tracked object change significantly compared to images captured by a stationary camera, making it difficult to track the object accurately.

[0005] The present invention has been made in consideration of the above circumstances, and one of its objectives is to provide an object tracking device, an object tracking method, and a program that can further improve the accuracy of tracking objects present around a vehicle. [Means for solving the problem]

[0006] The object tracking device, the object tracking method, and the program according to the present invention employ the following configuration. (1): An object tracking device according to one embodiment of the present invention includes an image acquisition unit that acquires image data including a plurality of image frames captured in time series by an imaging unit mounted on a moving body; a recognition unit that recognizes an object from the images acquired by the image acquisition unit; an area setting unit that sets an image area including the object recognized by the recognition unit; and an object tracking unit that tracks the object based on a time-series change in the image area set by the area setting unit, wherein the area setting unit sets the position and size of the image area for tracking the object in future image frames based on a time-series change in the image area including the object in past image frames and behavior information of the moving body.

[0007] (2): In the above aspect (1), the area setting unit estimates the position and speed of the object after the time of recognition based on the amount of change in the position of the object prior to the time of recognition by the recognition unit, and sets the position and size of the image area for tracking the object in future image frames based on the estimated position and speed and behavior information of the moving body prior to the time of recognition.

[0008] (3): In the above aspect (1) or (2), when the object is recognized by the recognition unit, the area setting unit performs projective transformation of the captured image captured by the imaging unit into a bird's-eye image, acquires the position and size of the object in the bird's-eye image, estimates the future position of the object in the bird's-eye image based on the acquired position and size of the object and behavior information of the moving body, associates the estimated position with the captured image, and sets the position and size of the image area for tracking the object in the next image frame.

[0009] (4): In any one of the above aspects (1) to (3), the object tracking unit uses a KCF (Kernelized Correlation Filter) to track the object.

[0010] (5): In any one of the above aspects (1) to (4), the area setting unit increases the size of the image area when the moving body is traveling in a manner that avoids contact with the object compared to when the moving body is not traveling in a manner that avoids contact.

[0011] (6): Another aspect of the object tracking method of the present invention is an object tracking method in which a computer acquires image data including a plurality of image frames captured in time series by an imaging unit mounted on a moving body, recognizes an object from the acquired image data, sets an image area including the recognized object, tracks the object based on the amount of change over time in the set image area, and sets the position and size of the image area for tracking the object in future image frames based on the amount of change over time in the image area including the object in past image frames and behavior information of the moving body.

[0012] (7): Another aspect of the present invention is a program that causes a computer to acquire image data including a plurality of image frames captured in time series by an imaging unit mounted on a moving body, recognize an object from the acquired image data, set an image area including the recognized object, track the object based on the amount of change over time in the set image area, and set the position and size of the image area for tracking the object in future image frames based on the amount of change over time in the image area including the object in past image frames and behavior information of the moving body. [Effects of the Invention]

[0013] According to aspects (1) to (7), it is possible to further improve the accuracy of tracking objects existing around the vehicle. [Brief explanation of the drawings]

[0014] [Figure 1] 1 is a diagram showing an example of the configuration of an object tracking device 100 mounted on a host vehicle M and peripheral devices. [Figure 2]1 is a diagram showing an example of a surrounding situation of a host vehicle M equipped with an object tracking device 100. FIG. [Figure 3] 3 is a diagram showing an example of an image IM10 of the front of the host vehicle M captured by the camera 10 in the surrounding situation shown in FIG. 2. FIG. [Figure 4] FIG. 2 shows an example of the configuration of a region setting section 130. [Figure 5] 10 is a diagram showing an example of the configuration of a grid set by a grid extraction unit 134. FIG. [Figure 6] 10 is a diagram showing an example of a method for extracting a grid G ​​by a grid extraction unit 134. FIG. [Figure 7] 10 is a diagram showing an example of a grid image GI calculated by a grid extraction unit 134. FIG. [Figure 8] 10 is a diagram showing an example of a method for searching for a grid G ​​executed by a region control unit 136. FIG. [Figure 9] FIG. 10 is a diagram showing an example of a bounding box BX superimposed on an image IM10. [Figure 10] FIG. 10 is a schematic diagram for explaining setting of an image area and tracking processing. [Figure 11] 10 is a flowchart illustrating an example of an area setting process. [Figure 12] 10 is a flowchart showing an example of the flow of a driving control process executed by the object tracking device 100. DETAILED DESCRIPTION OF THE INVENTION

[0015] Hereinafter, with reference to the drawings, embodiments of an object tracking device, an object tracking method, and a program of the present invention will be described. The object tracking device of the embodiment is mounted on, for example, a moving body. The moving body may be, for example, a four-wheeled vehicle, a two-wheeled vehicle, micromobility, a robot that moves on its own, or a portable device such as a smartphone that is placed on a moving body that moves on its own or carried by a person. In the following description, the moving body is assumed to be a four-wheeled vehicle, and the moving body will be referred to as "host vehicle M." The object tracking device is not limited to one mounted on a moving body, and may also be one that performs the processing described below based on images captured by a fixed-point observation camera or a smartphone camera.

[0016] 1 is a diagram showing an example of the configuration of an object tracking device 100 and peripheral devices mounted on a host vehicle M. The object tracking device 100 communicates with, for example, a camera 10, an HMI 30, a vehicle sensor 40, and a cruise control device 200.

[0017] The camera 10 is attached to the rear surface of the windshield of the vehicle M, captures images of an area including at least the road in the traveling direction of the vehicle M in time series, and outputs the captured images (captured images) to the object tracking device 100. Note that a sensor fusion device or the like may be interposed between the camera 10 and the object tracking device 100, but a description thereof will be omitted.

[0018] The HMI 30 presents various information to the occupants of the vehicle M under the control of the HMI control unit 150 and accepts input operations by the occupants. The HMI 30 includes, for example, various display devices, speakers, switches, a microphone, a buzzer, a touch panel, keys, etc. The various display devices are, for example, a liquid crystal display (LCD) or an organic electroluminescence (EL) display device. The display device is provided, for example, in the instrument panel near the front of the driver's seat (the seat closest to the steering wheel) and is installed in a position where the occupant can see it through the gap in the steering wheel or over the steering wheel. The display device may also be installed in the center of the instrument panel. The display device may also be a head-up display (HUD). The HUD projects an image onto a portion of the front windshield in front of the driver's seat, allowing the occupant sitting in the driver's seat to see a virtual image. The display device displays an image generated by the HMI control unit 150, which will be described later.

[0019] The vehicle sensors 40 include a vehicle speed sensor that detects the speed of the host vehicle M, an acceleration sensor that detects acceleration, a yaw rate sensor that detects the angular velocity (yaw rate) around a vertical axis, and a direction sensor that detects the direction of the host vehicle M. The vehicle sensors 40 may also include a steering angle sensor that detects the steering angle of the host vehicle M (which may be the angle of the steering wheels or the operation angle of the steering wheel). The vehicle sensors 40 may also include sensors that detect the amount of depression of an accelerator pedal or a brake pedal. The vehicle sensors 40 may also include a position sensor that acquires the position of the host vehicle M. The position sensor is, for example, a sensor that acquires position information (longitude and latitude information) from a GPS (Global Positioning System) device. The position sensor may also be, for example, a sensor that acquires position information using a GNSS (Global Navigation Satellite System) receiver of a navigation device (not shown) installed in the host vehicle M.

[0020] The object tracking device 100 includes, for example, an image acquisition unit 110, a recognition unit 120, an area setting unit 130, an object tracking unit 140, an HMI control unit 150, and a storage unit 160. These components are realized by, for example, a hardware processor such as a CPU (Central Processing Unit) executing a program (software). Some or all of these components may be realized by hardware (including circuitry) such as an LSI (Large Scale Integration), an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or a GPU (Graphics Processing Unit), or may be realized by a combination of software and hardware. The program may be stored in advance in a storage device (a storage device with a non-transitory storage medium) such as an HDD (Hard Disk Drive) or flash memory, or may be stored in a removable storage medium (a non-transitory storage medium) such as a DVD or CD-ROM, and installed by inserting the storage medium into a drive device.

[0021] The storage unit 160 may be realized by the various storage devices described above, or a solid-state drive (SSD), an electrically erasable programmable read-only memory (EEPROM), a read-only memory (ROM), or a random-access memory (RAM). The storage unit 160 stores, for example, information required for object tracking in the embodiment, tracking results, map information, programs, and various other information. The map information may include, for example, road shapes (road width, curvature, gradient), the number of lanes, intersections, lane center information, lane boundary (dividing line) information, and the like. The map information may also include point-of-interest (POI) information, traffic regulation information, address information (address and postal code), facility information, telephone number information, and the like.

[0022] The image acquisition unit 110 acquires images (hereinafter referred to as camera images) captured in time series by the camera 10. The image acquisition unit 110 may store the acquired camera images in the storage unit 160.

[0023] The recognition unit 120 recognizes the surrounding conditions of the host vehicle M based on the camera images acquired by the image acquisition unit 110. For example, the recognition unit 120 recognizes the type, position, speed, acceleration, etc. of objects present around the host vehicle M (within a predetermined distance). Objects include, for example, other vehicles (including motorcycles, etc.), traffic participants such as pedestrians and bicycles, and road structures. Road structures include, for example, road signs, traffic signals, curbs, medians, guardrails, fences, walls, and railroad crossings. The position of an object is recognized as a position on an absolute coordinate system with a representative point of the host vehicle M (such as the center of gravity or the center of the drive shaft) as the origin, and is used for control. The position of an object may be represented by a representative point such as the center of gravity or a corner of the object, or by a represented area. The "state" of an object may include the acceleration or jerk of the object, or the "behavioral state" (for example, whether the object is changing lanes or is about to change lanes). In the following description, the object is assumed to be an "other vehicle."

[0024] The recognition unit 120 may also recognize crosswalks, stop lines, other traffic signs (speed limits, road signs), etc. drawn on the road on which the host vehicle M is traveling. The recognition unit 120 may also recognize road dividing lines (hereinafter referred to as dividing lines) that divide each lane included in the road on which the host vehicle M is traveling, or may recognize the traveling lane of the host vehicle M from the nearest dividing lines existing on each of the left and right sides of the host vehicle M. The recognition unit 120 may recognize dividing lines by analyzing an image captured by the camera 10, or may refer to map information stored in the memory unit 160 from the position information of the host vehicle M detected by the vehicle sensor 40 and recognize information about surrounding dividing lines and traveling lanes from the position of the host vehicle M, or may combine both recognition results.

[0025] Furthermore, the recognition unit 120 recognizes the position and attitude of the host vehicle M with respect to the traveling lane. For example, the recognition unit 120 may recognize the deviation of the reference point of the host vehicle M from the center of the lane and the angle of the vehicle body with respect to a line connecting the centers of the lanes in the traveling direction of the host vehicle M as the relative position and attitude of the host vehicle M with respect to the traveling lane. Alternatively, the recognition unit 120 may recognize the position of the reference point of the host vehicle M with respect to either side edge of the traveling lane (a road dividing line or a road boundary) as the relative position of the host vehicle M with respect to the traveling lane.

[0026] Furthermore, the recognition unit 120 may analyze the image captured by the camera 10 and recognize the orientation of the body of the other vehicle relative to the front direction of the vehicle M or the direction in which the lane extends, the vehicle width, the position and orientation of the wheels of the other vehicle, etc., based on feature information (for example, edge information, color information, information on the shape and size of an object, etc.) obtained from the analysis results. The orientation of the body is, for example, the yaw angle of the other vehicle (the angle of the body relative to a line connecting the centers of the lanes in the direction in which the other vehicle is traveling).

[0027] When an object is recognized by the recognition unit 120, the region setting unit 130 sets an image region including the object in the camera image. The shape of the image region may be, for example, a rectangular shape such as a bounding box, or may be another shape (for example, a circle). Furthermore, the region setting unit 130 sets the position and size of the image region when the object tracking unit 140 tracks the object in a future image frame based on the amount of change over time in the image region including the object in a past image frame and behavior information of the host vehicle M.

[0028] The object tracking unit 140 tracks an object included in a future image frame based on the image region set by the region setting unit 130.

[0029] The HMI control unit 150 notifies the occupant of predetermined information via the HMI 30, and acquires information received by the HMI 30 through the operation of the occupant. For example, the predetermined information notified to the occupant includes information related to the traveling of the host vehicle M, such as information about the state of the host vehicle M and information about driving control. Information about the state of the host vehicle M includes, for example, the speed, engine speed, and shift position of the host vehicle M. The predetermined information may also include information about the tracking results of an object, information for warning of the possibility of contact with an object, and information for prompting driving operations to avoid contact. The predetermined information may also include information unrelated to the driving control of the host vehicle M, such as television programs, content (e.g., movies) stored on a storage medium such as a DVD, etc.

[0030] For example, the HMI control unit 150 may generate an image containing the above-mentioned specified information and display the generated image on the display device of the HMI 30, or may generate audio indicating the specified information and output the generated audio from the speaker of the HMI 30.

[0031] The driving control device 200 is, for example, an automatic driving control device that controls one or both of the steering and speed of the host vehicle M to autonomously drive the host vehicle M, or a driving support device that performs inter-vehicle distance control, automatic braking control, automatic lane change control, lane keeping control, etc. For example, the driving control device 200 operates an automatic driving control device or a driving support device based on information obtained by the object tracking device 100 to perform driving control such as avoiding contact between the host vehicle M and the object being tracked.

[0032] [Functions of the object tracking device] Next, the details of the function of the object tracking device 100 will be described. Fig. 2 is a diagram showing an example of the surrounding situation of the host vehicle M equipped with the object tracking device 100. As an example, Fig. 2 shows a scene in which the host vehicle M equipped with the object tracking device 100 is traveling at a speed VM along the extension direction of the road RD1 (the X-axis direction in the figure), and a motorbike B (an example of a target) is traveling across the road RD1 in front of the host vehicle M. As an example, the tracking of the motorbike (motorcycle) B by the object tracking device 100 will be described below.

[0033] Fig. 3 is a diagram showing an example of an image IM10 of the area ahead of the host vehicle M captured by the camera 10 in the surrounding conditions shown in Fig. 2. The image acquisition unit 110 acquires image data including a plurality of frames that represent the surrounding conditions of the host vehicle M, captured in time series by the camera 10 mounted on the host vehicle M. More specifically, for example, the image acquisition unit 110 acquires image data from the camera 10 at a frame rate of about 30 Hz, but this is not limited to this.

[0034] The recognition unit 120 performs image analysis processing on the image IM10, acquires feature information for each object included in the image (for example, feature information based on color, size, shape, etc.), and recognizes the motorbike B by matching the acquired feature information with feature information of a predetermined target. Furthermore, the recognition of the motorbike B may include, for example, a determination process using artificial intelligence (AI) or machine learning.

[0035] The region setting unit 130 sets an image region (bounding box) that includes bike B included in image IM10. FIG. 4 is a diagram showing an example of the configuration of the region setting unit 130. The region setting unit 130 includes, for example, a difference calculation unit 132, a grid extraction unit 134, a region control unit 136, and a region prediction unit 138. For example, the difference calculation unit 132, the grid extraction unit 134, and the region control unit 136 are functions that are used to set an image region that includes bike B recognized by the recognition unit 120, and the region prediction unit 138 is a function that is used to set an image region in the next image frame.

[0036] The difference calculation unit 132 calculates the difference in pixel values ​​for multiple frames acquired by the image acquisition unit 110, and calculates a difference image DI between the multiple frames by binarizing the calculated difference into a first value (e.g., 1) and a second value (e.g., 0).

[0037] More specifically, first, the difference calculation unit 132 performs gray conversion on the multiple frames acquired by the image acquisition unit 110, converting the RGB image into a grayscale image. Next, based on the speed of the vehicle M during the shooting interval when the multiple frames were captured, the difference calculation unit 132 aligns the frame captured at the previous time point (hereinafter may be referred to as the "previous frame") with the frame captured at the current time point (hereinafter may be referred to as the "current frame") by enlarging the frame captured at the previous time point (hereinafter may be referred to as the "previous frame") around the vanishing point of the frame.

[0038] For example, the difference calculation unit 132 estimates the travel distance of the host vehicle M from the speed (average speed) of the host vehicle M measured between the previous time point and the current time point, and enlarges the previous frame by an enlargement factor corresponding to the travel distance, with a vanishing point as the center. The vanishing point is, for example, an intersection point formed by extending both sides of the lane of travel of the host vehicle M included in the image frame. The difference calculation unit 132 also enlarges the previous frame by an enlargement factor corresponding to the travel distance of the host vehicle M measured between the previous time point and the current time point. At this time, the size of the enlarged previous frame is larger than before enlargement, so the difference calculation unit 132 returns the size of the enlarged previous frame to its original size by trimming the edges of the enlarged previous frame.

[0039] The difference calculation unit 132 may correct the previous frame by taking into consideration the yaw rate of the host vehicle M during the shooting interval between the previous frame and the current frame, in addition to the speed of the host vehicle M during the shooting interval between the previous frame and the current frame. More specifically, the difference calculation unit 132 may calculate the difference between the yaw angle of the host vehicle M at the time the previous frame was acquired and the yaw angle of the host vehicle M at the time the current frame was acquired, based on the yaw rate during the shooting interval, and align the previous frame with the current frame by shifting the previous frame in the yaw direction by an angle corresponding to the difference.

[0040] Next, the difference calculation unit 132 aligns the previous frame with the current frame, and then calculates the difference in pixel values ​​between the previous frame and the current frame. If the calculated difference value for each pixel is equal to or greater than a specified value, the difference calculation unit 132 assigns to the pixel a first value indicating that the pixel is a candidate for the target object. On the other hand, if the calculated difference value is less than the specified value, the difference calculation unit 132 assigns to the pixel a second value indicating that the pixel is not a candidate for a moving object.

[0041] The grid extraction unit 134 sets a grid for each of a plurality of pixels in the difference image DI calculated by the difference calculation unit 132, and when the density (proportion) of pixels having the first value in each of the set grids is equal to or greater than a threshold, extracts the grid G. The grid G ​​is a set of a plurality of pixels defined as a grid in the difference image DI.

[0042] FIG. 5 is a diagram illustrating an example of the configuration of a grid set by the grid extraction unit 134. For example, as shown in FIG. 5, the grid extraction unit 134 sets the size of the grid G ​​to approximately 10×10 pixels (an example of a “first size”) for areas of the difference image DI where the distance from the camera 10 is equal to or less than a first distance (e.g., 10 m), sets the size of the grid G ​​to approximately 8×8 pixels (an example of a “second size”) for areas where the distance from the camera 10 is greater than the first distance but equal to or less than a second distance (e.g., 20 m), and sets the size of the grid G ​​to approximately 5×5 pixels (an example of a “third size”) for areas where the distance from the camera 10 is greater than the second distance. This is because the greater the distance from the camera 10, the smaller the change in the area captured by the camera 10, and therefore the finer the size of the grid G ​​must be set to detect a moving object. Setting the size of the grid G ​​according to the distance from the camera 10 in the difference image DI allows for more accurate detection of a moving object.

[0043] Fig. 6 is a diagram showing an example of a method for extracting a grid G ​​by the grid extraction unit 134. For each of a plurality of grids G, the grid extraction unit 134 determines whether the density of pixels having a first value is equal to or greater than a threshold value (for example, approximately 85%), and for a grid G ​​for which it is determined that the density of pixels having a first value is equal to or greater than the threshold value, the grid extraction unit 134 extracts all of the pixels constituting the grid G ​​(sets them to a first value), as shown in the upper part of Fig. 6. On the other hand, for a grid G ​​for which it is determined that the density of pixels having a first value is less than the threshold value, the grid extraction unit 134 discards all of the pixels constituting the grid G ​​(sets them to a second value), as shown in the lower part of Fig. 6.

[0044] In the above description, the grid extraction unit 134 determines whether the density of pixels having the first value is equal to or greater than a single threshold for each of the plurality of grids G. However, the present invention is not limited to such a configuration, and the grid extraction unit 134 may change the threshold depending on the distance from the camera 10 in the difference image DI. For example, generally, the closer the distance from the camera 10, the greater the change in the area captured by the camera 10 and the more likely an error will occur. Therefore, the grid extraction unit 134 may set a higher threshold the closer the distance from the camera 10. Furthermore, the grid extraction unit 134 may make the determination using any statistical value based on the pixels having the first value, rather than being limited to the density of pixels having the first value.

[0045] The grid extraction unit 134 calculates a grid image GI by performing a process (grid replacement process) on the difference image DI in which all pixels of a grid where the density of pixels having a first value is equal to or greater than a threshold value are set to the first value. FIG. 7 is a diagram showing an example of the grid image GI calculated by the grid extraction unit 134. Note that in the example of FIG. 7, for the sake of convenience of explanation, part of the background image is left intact, but in reality, the components of the grid image GI shown in FIG. 7 are grids rather than pixels. In this way, by performing the grid replacement process on the difference image DI, a grid representing bike B is detected.

[0046] The region control unit 136 searches for a set of grids G that are extracted by the grid extraction unit 134 and that satisfy a predetermined criterion, and sets a bounding box for the set of grids G that has been found.

[0047] 8 is a diagram showing an example of a method for searching for grids G executed by the region control unit 136. The region control unit 136 first searches for a set of grids G whose bottom ends have a certain length L1 or more from the grid image GI calculated by the grid extraction unit 134. At this time, as shown in the left part of FIG. 8, in order to determine that a set of grids G has a bottom end of the certain length L1 or more, the region control unit 136 does not necessarily need to use the condition that the set contains no grids G missing; for example, the region control unit 136 may determine that the set has a bottom end of the certain length L1 or more on the prerequisite that the density of grids G included in the bottom end is equal to or greater than a reference value.

[0048] Next, when the region control unit 136 identifies a set of grids G having bottom ends that are equal to or greater than a certain length L1, it determines whether the set of grids G has a height that is equal to or greater than a certain length L2. That is, by determining whether the set of grids G has a bottom end that is equal to or greater than a certain length L1 and a height that is equal to or greater than a certain length L2, it is possible to identify whether the set of grids G corresponds to an object such as a motorcycle, a pedestrian, or a four-wheeled vehicle. In this case, the combination of the constant length L1 of the bottom end and the constant length L2 of the height is set as a unique value for each object such as a motorcycle, pedestrian, or four-wheeled vehicle.

[0049] Next, when the region control unit 136 identifies a set of grids G having a bottom end equal to or greater than a certain length L1 and a height equal to or greater than a certain length L2, it sets a bounding box for the set of grids G. Next, the region control unit 136 determines whether the density of the grids G included in the set bounding box is equal to or greater than a threshold. If the region control unit 136 determines that the density of the grids G included in the set bounding box is equal to or greater than a threshold, it detects the bounding box as a target object and superimposes the detected region on the image IM10.

[0050] 9 is a diagram showing an example of a bounding box BX superimposed on image IM10. By the above-described processing, it is possible to more accurately set a bounding box BX that includes the image area of ​​bike B, for example, as shown in FIG. 9. The image shown in FIG. 9 may be output to HMI 30 by HMI control unit 150.

[0051] In addition, instead of (or in addition to) the above-mentioned method, the region setting unit 130 may set the bounding box BX from the features of an object in the image using a method that uses known artificial intelligence (AI), machine learning, or deep learning.

[0052] The region prediction unit 138 sets the position and size of the image region for tracking the bike in future image frames based on the amount of change over time in the bounding box BX containing the bike B in past image frames and behavior information of the host vehicle M. For example, the region prediction unit 138 estimates the position and speed of the bike B after the time of recognition based on the amount of change in the position of the bike B in times prior to the time of recognition by the recognition unit 120, and sets the position and size of the image region for tracking the bike B in future image frames based on the estimated position and speed and behavior information (e.g., position, speed, yaw rate) of the host vehicle M prior to the time of recognition.

[0053] The object tracking unit 140 tracks bike B in the next image frame based on the amount of change over time in the image area set by the area setting unit 130. For example, the object tracking unit 140 searches for bike B in the image area (bounding box) predicted by the area prediction unit 138, and if the degree of match between the feature amount for bike B and the feature amount of the object in the bounding box is equal to or greater than a predetermined degree (threshold), it recognizes that the object in the bounding box is bike B and tracks bike B.

[0054] The object tracking unit 140 uses a KCF (Kernelized Correlation Filter) as an object tracking method. KCF is a type of object tracking algorithm that returns the area in an image with the best response by using a filter that is trained as needed based on the frequency components of the image when a series of images and an area of ​​interest to be tracked in the image are input.

[0055] For example, KCF can learn and track objects at high speed while suppressing memory usage by using FFT (Fast Fourier Transform). For example, tracking methods using a general two-class classifier perform classification processing by randomly sampling a search window from around the predicted position of the object. On the other hand, KCF analytically processes a group of images in which the search window is densely shifted by one pixel at a time using FFT, thereby achieving faster processing than methods using two-class classifiers.

[0056] The tracking method is not limited to KCF, and other methods such as Boosting, CSRT (Channel and Spatial Reliability Tracking), MEDIANFLOW, TLD (Tracking Learning Detection), and MIL (Multiple Instance Learning) may also be used. However, among these object tracking algorithms, KCF is most preferable in terms of tracking accuracy and processing speed. In particular, in fields where driving control of the host vehicle M is performed (autonomous driving and driving assistance), quick and highly accurate control according to the surrounding conditions of the host vehicle M is an important element, so KCF is particularly effective in fields where driving control is performed as in the embodiment.

[0057] Next, the setting of an image region in region prediction unit 138 and the tracking process in the set image region will be described. Fig. 10 is a schematic diagram for explaining the setting of an image region and the tracking process. The example of Fig. 10 shows frame IM20 of a camera image at the current time (t) and a bounding box BX(t) including bike B at the current time (t).

[0058] The region prediction unit 138 determines the amount of change in the position and size of the bounding box BX(t) between frames based on the position and size of the bounding box BX(t) recognized by the recognition unit 120 and the position and size of the bounding box BX(t-1) recognized in an image frame at a previous time (t-1). Next, the region prediction unit 138 estimates the position and size of the bounding boxes BX(t+1) and BX(t+2), which are regions of interest in the future (for example, the next frame (time (t+1) or the frame after that (t+2))), based on the determined amount of change. The object tracking unit 140 searches for regions based on the estimated bounding boxes BX(t+1) and BX(t+2) that match the previously recognized feature quantities to a predetermined degree or more, and recognizes the region that matches the predetermined degree or more as the motorbike B. In this way, the motorbike B can be recognized with high accuracy even if the size of the object in the image changes due to differences in orientation or angle, etc., due to the behavior of the vehicle M or the object itself.

[0059] FIG. 11 is a flowchart showing an example of region setting processing by the region prediction unit 138. In the example of FIG. 11, the region prediction unit 138 performs projective transformation on a camera image (e.g., image IM20 in FIG. 10) acquired by the image acquisition unit 110 into a bird's-eye image (overhead image) (e.g., image IM30 in FIG. 10) (step S100). In the processing of step S100, the region prediction unit 138 converts, for example, a coordinate system of a camera image with a forward viewing angle (camera coordinate system) into a coordinate system (vehicle coordinate system) in which the host vehicle M is viewed from above and is based on the position of the host vehicle M. Next, the region prediction unit 138 acquires the position and size of a tracking target object (motorcycle B in the above example) from the transformed image (step S102). Next, the region prediction unit 138 acquires behavior information (e.g., speed, yaw rate) of the host vehicle M in the past few frames using the vehicle sensor 40 (step S104), and estimates changes in the position and speed of the host vehicle M based on the acquired behavior information (step S106). In the process of step S106, for example, by performing a process such as a Kalman filter or linear interpolation on the behavior information, the amount of change can be estimated with higher accuracy.

[0060] Next, the region prediction unit 138 updates the coordinates (position) of future bike B in the bird's-eye image based on the estimated amount of change (step S108). Next, the region prediction unit 138 obtains the size of the tracked object at the updated coordinates from the size of the tracked object obtained in the processing of step S102 (step S110), associates the position and size of the future tracked object with the camera image, and sets a future image region on the camera image where the tracked object is estimated to exist in the future (region of interest for tracking) (step S112). This ends the processing of this flowchart. By performing object recognition in the next frame in the region of interest set in this way, the possibility that the tracked object (bike B) will be included in the region of interest increases, thereby further improving tracking accuracy.

[0061] The driving control device 200 estimates the risk of contact between the motorcycle and the host vehicle M based on the tracking results from the object tracking unit 140 and behavior information about the host vehicle M. Specifically, the driving control device 200 derives a time to collision (TTC) using the relative position (relative distance) and relative speed between the host vehicle M and the motorcycle B, and determines whether the derived time to collision TTC is less than a threshold. The time to collision TTC is a value calculated, for example, by dividing the relative speed from the relative distance. If the time to collision TTC is less than the threshold, the driving control device 200 determines that there is a possibility of contact between the host vehicle M and the motorcycle B, and executes driving control of the host vehicle M to avoid contact. In this case, the driving control device 200 generates a trajectory for the host vehicle M by steering control so as to avoid the motorcycle B detected by the object tracking unit 140, and causes the host vehicle M to travel along the generated trajectory. When the host vehicle M is traveling to avoid contact with the motorcycle B, the region prediction unit 138 may increase the size of the image region of the tracked object in the next image frame compared to when the host vehicle M is not traveling to avoid contact. This makes it possible to prevent the tracking accuracy of the tracked object from deteriorating even when the behavior of the host vehicle M changes significantly due to contact avoidance control.

[0062] Furthermore, instead of (or in addition to) the steering control described above, the driving control device 200 may stop the host vehicle M before the position of the motorcycle B (before the crosswalk shown in FIG. 2) until the motorcycle B crosses the road RD1. Furthermore, if the time to contact TTC is equal to or greater than a threshold, the driving control device 200 determines that the host vehicle M and the motorcycle B will not come into contact with each other, and does not execute contact avoidance control. In this way, in this embodiment, the detection results by the object tracking device 100 can be suitably used for automatic driving or driving assistance of the host vehicle M.

[0063] For example, the HMI control unit 150 outputs the content executed by the cruise control device 200 to the HMI 30 to notify the occupants of the host vehicle M. Furthermore, when an object is detected, the HMI control unit 150 may notify the occupants by displaying the detected content and the predicted position and size based on a bounding box on the HMI 30. This allows the occupants to understand how the host vehicle M is predicting the future behavior of the surrounding object.

[0064] [Processing flow] Next, a description will be given of the flow of processing executed by the object tracking device 100 of the embodiment. Note that the processing of this flowchart may be executed repeatedly at a predetermined timing, for example.

[0065] FIG. 12 is a flowchart showing an example of the flow of a driving control process executed by the object tracking device 100. In the example of FIG. 12, the image acquisition unit 110 acquires a camera image (step S200). Next, the recognition unit 120 recognizes an object from the camera image (step S202). Next, the area setting unit 130 sets an image area (area of ​​interest) for tracking the object from the camera image based on the position and size of the object (step S204). Next, the area is predicted, and the object is tracked using the predicted area (step S206).

[0066] Next, the cruise control device 200 determines whether cruise control of the host vehicle M is necessary based on the tracking result (step S208). If it is determined that cruise control is necessary, the cruise control device 200 executes cruise control based on the tracking result (step S210). For example, the process of step S210 is avoidance control that is executed when it is determined that there is a possibility that the host vehicle M will come into contact with an object in the near future. Note that in the process of step S210, cruise control is executed that also includes the recognition result of the surrounding situation of the host vehicle M by the recognition unit 120. This ends the process of this flowchart. Also, if it is determined in the process of step S208 that cruise control is not necessary, the process of this flowchart ends.

[0067] According to the embodiment described above, the object tracking device 100 comprises an image acquisition unit 110 that acquires image data including a plurality of image frames captured in time series by an imaging unit mounted on the moving body, a recognition unit 120 that recognizes an object from the image acquired by the image acquisition unit 110, an area setting unit 130 that sets an image area including the object recognized by the recognition unit 120, and an object tracking unit 140 that tracks the object based on the amount of change over time in the image area set by the area setting unit 130. The area setting unit 130 sets the position and size of the image area for tracking the object in future image frames based on the amount of change over time in the image area including the object in past image frames and behavior information of the moving body, thereby further improving the accuracy of tracking objects present around the vehicle.

[0068] Furthermore, according to the embodiment, by correcting the position and size (dimensions) of the area to be used as the area of ​​interest in the next frame when updating the image frame based on the behavior information of the vehicle, the possibility that the object to be tracked will be included in the area of ​​interest can be increased, and the tracking accuracy in each frame can be further improved.

[0069] Furthermore, according to the embodiment, the tracking accuracy can be further improved by performing corrections that reflect the behavior of the moving object in the object tracking process using KCF, which inputs images from a camera (mobile camera) mounted on the moving object. For example, according to the embodiment, by tracking the target object by adding adjustment processing of the attention area (image area of ​​the tracking target) according to the behavior of the host vehicle based on KCF, it is possible to flexibly respond to and track changes in the apparent position and size of the object between frames of the camera 10. Therefore, the tracking accuracy can be improved compared to object tracking using preset template matching.

[0070] The above-described embodiment can be expressed as follows. a storage medium storing computer readable instructions; a processor connected to the storage medium; The processor executes the computer-readable instructions to: acquiring image data including a plurality of image frames captured in time series by an imaging unit mounted on the moving object; Recognizing an object from the acquired image; Setting an image region including the recognized object; Tracking the object based on the amount of change in the image area over time; setting a position and a size of an image area in a future image frame for tracking the object based on a time-series change amount of an image area including the object in a past image frame and behavior information of the moving object; Object tracking device.

[0071] The above describes the form for carrying out the present invention using an embodiment, but the present invention is not limited to such an embodiment, and various modifications and substitutions can be made within the scope that does not deviate from the gist of the present invention. [Explanation of symbols]

[0072] 10...camera, 30...HMI, 40...vehicle sensor, 100...object tracking device, 110...image acquisition unit, 120...recognition unit, 130...area setting unit, 132...difference calculation unit, 134...grid extraction unit, 136...area control unit, 138...area prediction unit, 140...object tracking unit, 150...HMI control unit, 160...memory unit, 200...driving control device

Claims

1. an image acquisition unit that acquires image data including a plurality of image frames captured in time series by an imaging unit mounted on the moving object; a recognition unit that recognizes an object from the image data acquired by the image acquisition unit; an area setting unit that sets an image area including the object recognized by the recognition unit; an object tracking unit that tracks the object based on a time-series change amount of the image area set by the area setting unit, the region setting unit sets a position and a size of an image region for tracking the object in a future image frame based on a time-series change amount of an image region including the object in a past image frame and behavior information of the moving object; the region setting unit, when a driving control device that controls one or both of the steering and the speed of the moving body to cause the moving body to autonomously drive the moving body performs driving that avoids contact between the moving body and the object, increases the size of the image region that tracks the object in the future image frame compared to when the driving that avoids contact is not performed; Object tracking device.

2. the region setting unit estimates a position and a speed of the object after the time of recognition based on an amount of change in the position of the object in the past before the time of recognition by the recognition unit, and sets a position and a size of an image region for tracking the object in future image frames based on the estimated position and speed and behavior information of the moving body in the past before the time of recognition; The object tracking device according to claim 1 .

3. The region setting unit When the object is recognized by the recognition unit, a captured image captured by the imaging unit is projectively transformed into a bird's-eye view image, and a position and a size of the object in the bird's-eye view image are acquired; a future position of the object in the bird's-eye view image based on the acquired position and size of the object and behavior information of the moving body, and the estimated position is associated with the captured image to set the position and size of an image area for tracking the object in the next image frame; The object tracking device according to claim 1 or 2.

4. the object tracking unit uses a KCF (Kernelized Correlation Filter) to track the object; The object tracking device according to any one of claims 1 to 3.

5. an image acquisition unit that acquires image data including a plurality of image frames captured in time series by an imaging unit mounted on the moving object; a recognition unit that recognizes an object from the image data acquired by the image acquisition unit; an area setting unit that sets an image area including the object recognized by the recognition unit; an object tracking unit that tracks the object based on a time-series change amount of the image area set by the area setting unit, the region setting unit sets a position and a size of an image region for tracking the object in a future image frame based on a time-series change amount of an image region including the object in a past image frame and behavior information of the moving object; The region setting unit aligning two chronologically consecutive image frames from among the plurality of image frames to calculate a difference in pixel values ​​between the frames, and generating a difference image in which a value indicating whether or not each pixel is a candidate for the target object is assigned depending on whether or not the calculated difference value for each pixel is equal to or greater than a specified value; a plurality of grids each including a plurality of pixels are set in the difference image so that the grid size corresponds to the distance from the imaging unit, and a grid having a density of values ​​indicating whether or not the grid is a candidate for the target object equal to or greater than a threshold is extracted from the plurality of set grids; setting a bounding box for a set of grids among the extracted grids that satisfy a predetermined criterion; If the density of the grid included in the set bounding box is equal to or greater than a threshold, the position and size of the bounding box are set as the position and size of the image area in which the object is tracked. Object tracking device.

6. The computer acquiring image data including a plurality of image frames captured in time series by an imaging unit mounted on the moving object; Recognizing an object from the acquired image data; Setting an image region including the recognized object; Tracking the object based on the amount of change in the image area over time; determining a position and a size of an image area in a future image frame for tracking the object based on a time-series change amount of an image area including the object in a past image frame and behavior information of the moving object; a driving control device that controls one or both of the steering and the speed of the moving body to autonomously drive the moving body, when the moving body drives to avoid contact between the moving body and the object, increases the size of an image area that tracks the object in the future image frame compared to when the moving body does not drive to avoid contact; Object tracking method.

7. The computer acquiring image data including a plurality of image frames captured in time series by an imaging unit mounted on the moving object; Recognizing an object from the acquired image data; Setting an image region including the recognized object; Tracking the object based on the amount of change in the image area over time; determining a position and a size of an image area in a future image frame for tracking the object based on a time-series change amount of an image area including the object in a past image frame and behavior information of the moving object; aligning two chronologically consecutive image frames from among the plurality of image frames to calculate a difference in pixel values ​​between the frames, and generating a difference image in which a value indicating whether or not each pixel is a candidate for the target object is assigned depending on whether or not the calculated difference value for each pixel is equal to or greater than a specified value; a plurality of grids each including a plurality of pixels are set in the difference image so that the grid size corresponds to the distance from the imaging unit, and a grid having a density of values ​​indicating whether or not the grid is a candidate for the target object equal to or greater than a threshold is extracted from the plurality of set grids; setting a bounding box for a set of grids among the extracted grids that satisfy a predetermined criterion; If the density of the grid included in the set bounding box is equal to or greater than a threshold, the position and size of the bounding box are set as the position and size of the image area in which the object is tracked. Object tracking method.

8. On the computer, acquiring image data including a plurality of image frames captured in time series by an imaging unit mounted on the moving object; Recognizing an object from the acquired image data; setting an image region including the recognized object; tracking the object based on the amount of change in the image area over time; setting a position and a size of an image area for tracking the object in a future image frame based on a time-series change amount of an image area including the object in a past image frame and behavior information of the moving object; a driving control device that controls one or both of the steering and the speed of the moving body to autonomously drive the moving body, when the moving body drives to avoid contact between the moving body and the object, increases the size of an image area that tracks the object in the future image frame compared to when the moving body does not drive to avoid contact; program.

9. On the computer, acquiring image data including a plurality of image frames captured in time series by an imaging unit mounted on the moving object; Recognizing an object from the acquired image data; setting an image region including the recognized object; tracking the object based on the amount of change in the image area over time; setting a position and a size of an image area for tracking the object in a future image frame based on a time-series change amount of an image area including the object in a past image frame and behavior information of the moving object; two time-series consecutive image frames among the plurality of image frames are aligned to calculate a difference in pixel values ​​between the frames, and a difference image is generated in which a value indicating whether or not each pixel is a candidate for the target object is assigned to each pixel depending on whether or not the difference value calculated for each pixel is equal to or greater than a specified value; a plurality of grids each including a plurality of pixels are set in the difference image so that the size of the grid corresponds to the distance from the imaging unit, and a grid having a density of values ​​indicating whether or not the grid is a candidate for the target object equal to or greater than a threshold is extracted from the plurality of set grids; setting a bounding box for a set of grids that satisfy a predetermined criterion among the extracted grids; if the density of the grid included in the set bounding box is equal to or greater than a threshold, the position and size of the bounding box are set as the position and size of the image area in which the object is tracked; program.

Images