Image processing device

Abstract

Provided is an image processing device capable of reliably tracking objects related to vehicle control while suppressing processing load and a storage capacity of an image when tracking using the texture of camera images in a situation in which a large number of objects exist nearby. Objects that enter a predetermined area near a camera device 200 or a vehicle 100 equipped with the camera device 200 are subjected to image tracking in reverse chronological order, and objects which can be reliably tracked by nearby object tracking are excluded from image tracking to perform minimal image tracking and suppress image capacity for tracing back into the past.

Description

Image processing device

[0001] The present invention relates to an image processing device.

[0002] Prior art includes a technology such as that shown in Patent Document 1, which discloses a technology for continuously detecting and tracking an object (target) that may be a control target.

[0003] JP 2018-80938 A

[0004] In recent years, vehicles equipped with autonomous driving (AD) systems and advanced driver-assistance systems (ADAS) have become increasingly common. To realize these systems, images from cameras mounted on the vehicle are processed to recognize the vehicle's external environment. AD and ADAS cameras estimate not only the position of an object but also its speed to perform collision detection, requiring tracking of the same object. In object tracking, nearby object tracking, which determines and tracks objects that are nearby on a time axis as the same object, can lead to object transfers in situations where multiple objects are present in the vicinity and crowded the area. Therefore, image tracking is used, which identifies and tracks the same object based on its appearance (the texture of the camera image). However, image tracking imposes a heavy processing load, making it difficult to track all objects.

[0005] The present invention has been made in consideration of the above-mentioned problems, and aims to provide an image processing device that can reliably track objects related to vehicle control while reducing processing load and image storage capacity when tracking using the texture of a camera image in a situation where there are many objects in the vicinity.

[0006] In order to solve the above problem, the image processing device of the present invention includes an image acquisition unit that acquires images captured by an imaging device mounted on the vehicle itself, an image storage unit that stores the images captured at past times, an object detection unit that detects objects based on the images, and a first tracking unit that tracks the objects based on feature amounts of the objects included in the plurality of images captured in time series, wherein the first tracking unit references images of one or more first tracked objects included in the objects detected by the object detection unit among the plurality of images stored in the image storage unit, and searches for feature amounts of the objects from images captured at a first time point to images captured at a second time point earlier than the first time point, thereby tracking the first tracked objects and outputting the tracking results.

[0007] According to the present invention, an object detected in the vicinity of a possible control target is tracked by tracing past images. The problem of increasing the storage capacity of past images is prevented by narrowing down the data necessary for image tracking in the following order and deleting unnecessary data. This allows reliable tracking of objects related to vehicle control while reducing the processing load and image storage capacity.

[0008] Problems, configurations, and effects other than those described above will become apparent from the following description of the embodiments.

[0009] 1 is a diagram showing an example of the operation of the camera device according to the present embodiment, illustrating the detection positions of a vehicle and an object; a flowchart of the camera device according to the present embodiment; a block diagram of the camera device according to the present embodiment; an overhead view of the detection positions of a vehicle and an object, illustrating the operation of the camera device according to the present embodiment, narrowed down to the detection positions of objects of a type other than a specific type; an overhead view of the detection positions of a vehicle and an object, illustrating the operation of the camera device according to the present embodiment, narrowed down to objects that can be determined to be the same object using a nearby object tracking method; an overhead view of the detection positions of a vehicle and an object, illustrating the operation of the camera device according to the present embodiment, narrowed down to objects that can be determined to be the same object of a specific type using a nearby object tracking method; an overhead view of the detection positions of a vehicle and an object, illustrating the operation of the camera device according to the present embodiment, illustrating the maximum movement range of an object of a specific type using a nearby object tracking method; an overhead view of the detection positions of a vehicle and an object, illustrating the operation of the camera device according to the present embodiment, illustrating the collision detection area and collision avoidance determination area when moving forward. 1 is a bird's-eye view of the detection positions of a vehicle and an object, illustrating a state showing an example of operation of the camera device according to this embodiment, and illustrating the intrusion direction in determining the collision avoidance determination area when moving forward. 2 is a bird's-eye view of the detection positions of a vehicle and an object, illustrating a state showing an example of operation of the camera device according to this embodiment, and illustrating the collision determination area and collision avoidance determination area when moving backward. 3 is a bird's-eye view of the detection positions of a vehicle and an object, illustrating a state showing an example of operation of the camera device according to this embodiment, and illustrating the intrusion direction in determining the collision avoidance determination area when moving backward. 4 is a bird's-eye view of the detection positions of a vehicle and an object, illustrating a state showing an example of operation of the camera device according to this embodiment, and illustrating narrowing down objects that have entered the collision avoidance determination area when moving forward by tracking nearby objects.

[0010] Hereinafter, embodiments of the present invention will be described with reference to the drawings. The examples are illustrative of the present invention, and for clarity of explanation, appropriate omissions and simplifications have been made. The present invention can be implemented in various other forms. Unless otherwise specified, each component may be singular or plural.

[0011] In the embodiments, processing performed by executing a program may be described. Here, a computer executes the program using a processor (e.g., a CPU or a GPU) and performs processing defined by the program using storage resources (e.g., memory) and interface devices (e.g., communication ports). Therefore, the entity performing the processing by executing the program may be the processor. Similarly, the entity performing the processing by executing the program may be a controller, device, system, computer, or node having a processor. The entity performing the processing by executing the program may be any computing unit, and may include a dedicated circuit that performs specific processing. Here, the dedicated circuit may be, for example, an FPGA (Field Programmable Gate Array), an ASIC (Application Specific Integrated Circuit), or a CPLD (Complex Programmable Logic Device).

[0012] A program may be installed on a computer from a program source. The program source may be, for example, a program distribution server or a computer-readable storage medium. When the program source is a program distribution server, the program distribution server may include a processor and storage resources for storing the program to be distributed, and the processor of the program distribution server may distribute the program to be distributed to other computers. In addition, in the embodiments, two or more programs may be realized as one program, or one program may be realized as two or more programs.

[0013] An embodiment of a camera device equipped with an image processing device of the present invention will be described with reference to FIGS.

[0014] [Explanation of the relationship between a vehicle and surrounding objects taken as an example] Fig. 1 is an overhead view of a vehicle and the detection positions of objects, illustrating an example of the operation of a camera device according to this embodiment. Objects having the reference numerals described below are positioned as shown in Fig. 1 and move over time. A camera device 200 according to this embodiment is mounted on a vehicle 100. Reference numeral 100 denotes a vehicle (host vehicle) on which the camera device 200 is mounted.

[0015] Reference numeral 101 denotes a collision detection area set around the vehicle 100. When an object is detected in this collision detection area 101, the same object is identified by tracing back images and the way in which the object has moved is determined.

[0016] Reference numeral 102 denotes an object that has entered the collision detection area 101 and is a pedestrian. Reference numerals 103 and 104 denote objects that have not entered the collision detection area 101 and are also pedestrians. Reference numerals 105, 106, 107, and 108 denote vehicles, such as a preceding vehicle 105, a following vehicle 106, and oncoming vehicles 107 and 108, respectively. Reference numeral 109 denotes a person riding a bicycle (cyclist). Reference numeral 110 denotes a pedestrian.

[0017] Reference numeral 111 denotes a legend for the position of an object (object detection result) detected and measured by the camera device 200, including a distance measurement error. Reference numeral 112 denotes a legend for the arrow indicating the true position of the object and its moving direction for the object 111. Objects 102 to 110 are represented by the legends 111 and 112, respectively.

[0018] In the example of FIG. 1, 102, 103, and 104 are located in areas close to each other at a certain time, and the probability of mistracking is high.

[0019] [Explanation of Flow] The camera device (image processing device) according to this embodiment is characterized by basically performing operations in accordance with the flowchart shown in FIG.

[0020] Step S201 is image acquisition processing, which is processing for acquiring images captured by an imaging device such as a camera. There are one or more imaging devices, and the images are divided into a plurality of field of view areas for each imaging device and saved.

[0021] S202 is an object detection process, which is a process for detecting an object that appears in the image acquired in S201 (image acquisition process).

[0022] S203 is an identification process, which is a process for identifying the type of object detected in S202 (object detection process). The types include, for example, a crossing vehicle, other vehicles, a crossing motorcycle, other motorcycles, a pedestrian, a cyclist, etc. In S203 (identification process), the orientation of the object is also identified, for example, into four directions or eight directions.

[0023] S204 is an area setting process, which sets a collision determination area, a collision avoidance determination area, etc. around the vehicle 100 to determine the risk of collision between the vehicle 100 and an object.

[0024] S205 is a tracking method selection process, which selects whether to perform nearby object tracking processing or image tracking processing.

[0025] Step S206 is a nearby object tracking process, in which nearby objects of the same type detected in the images of the previous and next frames are identified as the same object and tracked.

[0026] S207 is an image tracking process in which objects detected in images of previous and next frames are identified as the same object if the image features (quantities) match, and are tracked.

[0027] S208 is an image deletion process in which unnecessary images are deleted. In S208 (image deletion process), image areas (partial areas or the entire area of ​​the image) that include only nearby objects that have been determined to be tracked and objects for which tracking processes (nearby object tracking process, image tracking process) have been completed are deleted.

[0028] S209 is an image saving process, which is a process for saving an image.

[0029] Step S210 is object history storage processing, which stores the object history, including the object's position, type, orientation, tracking method, and tracking completion status.

[0030] S211 is a speed calculation process, in which the speed is calculated using the object history information stored in S210 (object history storage process).

[0031] S212 is the output process of the object recognition results, in which the position, type, and speed of the detected, identified, and calculated object are output to a brake control device, steering control device, etc. installed in the vehicle to control the vehicle and realize, for example, automatic braking.

[0032] [Explanation of Block Diagram] Fig. 3 is a block diagram of a camera device according to this embodiment. Reference numeral 200 denotes the camera device according to this embodiment. The camera device 200 of this embodiment includes an imaging device 200A and an image processing device 200B.

[0033] Reference numeral 200A denotes an imaging device mounted on the vehicle 100. The imaging device 200A is a camera equipped with an imaging sensor (such as a CMOS (Complementary Metal Oxide Semiconductor)) that converts light into an electrical signal. The information converted into an electrical signal by the imaging sensor is further converted into image data representing the captured image within the imaging device 200A. The images captured by the imaging device 200A are transmitted to the image processing device 200B at predetermined intervals. The imaging device 200A captures images of the outside world (surroundings) of the vehicle 100 and transmits (outputs) the images to the image processing device 200B.

[0034] An image processing device 200B processes images received from the imaging device 200A. The image processing device 200B includes an image acquisition unit 201, an object detection unit 202, an identification unit 203, an area setting unit 204, a tracking method selection unit 205, a nearby object tracking unit 206, an image tracking unit 207, an image deletion unit 208, an image storage unit 209, an object history storage unit 210, a speed calculation unit 211, and a result output unit 212.

[0035] An image acquisition unit 201 acquires an image captured by the imaging device 200A.

[0036] An object detection unit 202 detects an object from the image acquired by the image acquisition unit 201 .

[0037] An identification unit 203 identifies the type of object detected by the object detection unit 202 .

[0038] Reference numeral 204 denotes a region setting unit, which is a processing unit that sets collision determination regions 101, 704 and collision avoidance determination regions 600, 601, 602, 603, 700, 701, 702, 703, etc., around the vehicle 100 to determine the risk of collision between the vehicle 100 and an object (see also FIGS. 8 to 11 ). The region setting unit 204 acquires wheel speed, steering angle, actual steering angle, yaw rate sensor, inertial sensor, GPS position information, etc. from the vehicle 100, and determines the speed and turning radius of the host vehicle. When the host vehicle is moving forward, regions 101, 600, 601 are set in front of the vehicle, and regions 602, 603 are set behind or to the sides of the vehicle, as shown in FIG. 8 . When the host vehicle is moving backward, regions 704, 700, 701 are set behind the vehicle, and regions 702, 703 are set in front of or to the sides of the vehicle, as shown in FIG. 10 . The distances 621 to 620, 612 to 611, 613 to 614, 612 to 610, 613 to 615, 721 to 720, 712 to 711, 713 to 714, 712 to 710, and 713 to 715 are increased in correlation with the speed of the vehicle. By increasing the distances in correlation with the speed, when the vehicle is traveling at a low speed, the speed of pedestrians and cyclists is relatively high compared to the speed of the vehicle, and there is a high possibility of a collision with the vehicle if a pedestrian or cyclist suddenly jumps out, so this increases the number of situations where the brakes are applied and reduces accidents. Similarly, by lengthening the distance in correlation with the speed, when a vehicle is traveling at high speed, it is more likely that a pedestrian or cyclist who has avoided a collision will be able to pass by without a collision, rather than changing direction, jumping out, and colliding again; on the other hand, automatic braking while traveling at high speed can reduce the possibility of a rear-end collision with a following vehicle or the vehicle becoming uncontrollable and resulting in a self-inflicted accident.

[0039] Furthermore, when the host vehicle is turning right with respect to the traveling direction while moving forward, the smaller the turning radius, the wider the collision detection region 101 in the traveling direction of the vehicle can be, thereby increasing the accuracy of collision detection during turning. Conversely, when the host vehicle is turning left with respect to the traveling direction while moving forward, the smaller the turning radius, the wider the collision detection region 101 in the traveling direction of the vehicle can be, thereby increasing the accuracy of collision detection during turning.

[0040] Furthermore, when the host vehicle is turning to the right in the traveling direction while backing up, the smaller the turning radius, the wider the collision detection region 704 in the traveling direction of the vehicle can be, thereby increasing the accuracy of collision detection during turning, by changing the ratio of the distance from 713 to 714 to the distance from 712 to 711 so that the distance from 712 to 711 is larger. Conversely, when the host vehicle is turning to the left in the traveling direction while backing up, the smaller the turning radius, the wider the collision detection region 704 in the traveling direction of the vehicle can be, by changing the ratio of the distance from 713 to 714 to the distance from 712 to 711 so that the distance from 713 to 714 is larger.

[0041] A tracking method selection unit 205 selects whether to perform nearby object tracking processing or image tracking processing. In other words, the tracking method selection unit 205 selects, for each detected and identified object, a tracking target object for nearby object tracking processing, a tracking target object for image tracking processing, or a non-tracking target object other than the tracking target object for nearby object tracking processing and the tracking target object for image tracking processing.

[0042] A nearby object tracking unit 206 compares images of different frames taken in succession in time, identifies nearby objects of the same type as the same object, and tracks them. In other words, the nearby object tracking unit 206 tracks one or more tracking target objects of the nearby object tracking process that are included in the object detected by the object detection unit 202, based on the type of object captured in multiple images taken in time series.

[0043] An image tracking unit 207 compares images from different frames taken in succession in time, identifies and tracks images with matching image features (quantities) as the same object. In other words, the image tracking unit 207 tracks an object based on the feature quantities of the object contained in multiple images captured in time series. The image tracking unit 207 references, from the multiple images stored in the image storage unit 209, images in which one or more tracking target objects of the image tracking process included in the object detected by the object detection unit 202 are captured, and tracks the tracking target objects of the image tracking process by searching for the feature quantities of the object in images captured in the past from the image captured at the current time (going back in time), and outputs the tracking results.

[0044] An image storage unit 209 divides an image captured at a previous time into multiple regions and stores the divided images in the memory M209. The image may be divided, for example, by cutting out and storing the region of interest (ROI) of the object identified by the identification unit 203. When storing the images, the images are stored in a ring buffer-like manner in a working memory that performs image processing such as identification processing. In this way, images must be stored using the working memory for image cropping, resizing, normalization, etc., during identification processing, but if the images are stored in a ring buffer-like manner that is originally used as the working memory, there is no need to store them again.

[0045] An image deletion unit 208 deletes, from the image areas stored in the memory M209, image areas in which an object tracked by the nearby object tracking unit 206 has been captured (partial or all of the image) and image areas that have been completely tracked (partial or all of the image up to the time when tracking by the nearby object tracking unit 206 or the image tracking unit 207 has been completed).

[0046] An object history storage unit 210 stores the history of an object in a memory M210.

[0047] A speed calculation unit 211 calculates the speed of an object using historical information about the object.

[0048] 212 is a result output unit that outputs the position, type, and speed of the detected, identified, and calculated object to the vehicle 100, and controls the vehicle using a brake control device, steering control device, etc. provided on the vehicle 100, thereby realizing, for example, automatic braking.

[0049] [Explanation of Features of the Embodiment with Examples] <Tracking Method for Types Other Than the Specific Type: Nearby Object Tracking> FIG. 4 is an overhead view of the detection positions of vehicles and objects illustrating an example of the operation of a camera device according to this embodiment, narrowing down the detection positions of objects of types other than the specific type. In this embodiment, a tracking process is selected depending on the type of object. As described above, two types of tracking processes are provided: near object tracking and image tracking. Near object tracking has a low processing load, but there is a risk of mistracking if an object of the same type is present nearby. Image tracking has a high processing load, but it can identify objects with similar image features, so there is a low risk of mistracking. Other vehicles (traveling in the forward and opposite lanes) and other motorcycles are unlikely to suddenly appear and do not appear to move significantly in the image coordinates. They also rarely change direction suddenly. Therefore, there are rarely multiple objects nearby, and the risk of mistracking when tracking nearby objects is low. Near object tracking is used for objects of a type with a low risk of mistracking when tracking these nearby objects (objects 105, 106, 107, and 108). When near object tracking is selected, images of the objects (partially or entirely) are deleted. This reduces the processing load and the size of stored images.

[0050] <Determining Identical Objects in Nearby Object Tracking by Setting a Movement Prediction Range> Figure 5 is an overhead view of the detection positions of vehicles and objects, illustrating an example of the operation of a camera device according to this embodiment, narrowing down the objects that can be determined to be the same object using a nearby object tracking method. In this embodiment, the tracking method for objects of a type that appear and move significantly on the image coordinates, such as pedestrians, cyclists, crossing vehicles, and crossing motorcycles (objects 109 and 110), determines a maximum range (hereinafter also referred to as a tracking range) within which each type of object moves at a predetermined speed. If only one object of the same type exists within that tracking range in an image captured at a certain time and an image captured at an earlier time, the object is determined to be the same object and tracked. Reference numerals 300 to 306 represent the maximum range (tracking range) for object 110, and reference numerals 308 to 314 represent the maximum range (tracking range) for object 109. The ranges (estimated predicted circles) of the amounts of movement of the objects 109 and 110, 315 and 307, respectively, are determined as the maximum ranges (tracking ranges).

[0051] 6 is an overhead view of the detection positions of a vehicle and an object, illustrating an example of the operation of the camera device according to this embodiment, in which the objects are narrowed down to those that can be determined to be the same object of a specific type using the nearby object tracking method. This is an example that can further improve the accuracy of nearby object tracking from that shown in FIG. 5. By assuming that the maximum range of movement of an object is moving in the direction (orientation) identified by the object classifier, the number of objects that exist within the maximum range of movement is reduced, thereby increasing the number of nearby objects that can be tracked.

[0052] 7 is an overhead view of the detection positions of a vehicle and an object, illustrating an example of the operation of the camera device according to this embodiment, and illustrates the maximum movement range of a specific type of object in a nearby object tracking method. For object position 420 at a certain time in 109, an assumed movement speed of 503 is assumed, and a maximum speed error range 410 for this is used for nearby object tracking. Reference numeral 501 denotes an image captured by camera 500, which is an imaging device. 420 is identified as an object moving in the left-right direction as viewed from camera 500. As a result, 503 is estimated as the assumed movement speed of an object moving in a direction perpendicular to line 502 connecting 500 and 420. As a result, as shown in FIG. 6 , for object position 420 at a certain time in 109, the range of movement amount 410 (assumed prediction ellipse) can be defined as the tracking range, and similarly, tracking ranges 411 to 416 can be defined thereafter.

[0053] <Detected in Collision Avoidance Judgment Area, Determined to be the Same Object by Object Proximity Tracking Depending on the Intrusion Direction (When Moving Forward)> Figure 8 is an overhead view of the detection positions of a vehicle and an object, illustrating an example of the operation of the camera device according to this embodiment, and illustrates the collision judgment area and collision avoidance judgment area when moving forward. Reference numerals 600, 601, 602, and 603 indicate collision avoidance judgment areas set around the vehicle 100. An object that enters this collision avoidance judgment area is judged to have the potential for collision avoidance, and is tracked by object proximity tracking (in other words, selected as a tracking target object for object proximity tracking processing), and is excluded from the image tracking targets. Reducing the number of image tracking targets reduces the processing load and image storage capacity.

[0054] Regarding the setting of distances 610 to 615, 620, and 621, the distances are set farther away depending on the speed and steering angle of the vehicle 100. When the speed of the vehicle 100 is fast, the brakes should be applied if there are pedestrians or the like nearby, which can reduce the accident rate, compared to when the speed of the vehicle 100 is slow. When the vehicle 100 has a steering angle, applying the brakes over a wider range can also reduce the accident rate, compared to when there is no steering angle. Furthermore, when the speed range is high or there is a steering angle, the driver and passengers will feel less uncomfortable even if the brakes are controlled for nearby objects, even if they are not subject to collision.

[0055] FIG. 9 is an overhead view of the detection positions of the vehicle and the object, illustrating an example of the operation of the camera device according to this embodiment, and explains the direction of entry into the collision avoidance determination area when moving forward.

[0056] An object is not determined to be an object for which collision has been avoided simply by being detected as having entered these collision avoidance determination areas. As shown in Figure 9, an object in a position and traveling direction that has entered a collision avoidance determination area 600 is determined to be an object for which collision has been avoided if it has entered across the bottom edge (vehicle side), a collision avoidance determination area 601 is determined to be an object for which collision has been avoided if it has entered across the top edge (vehicle side), a collision avoidance determination area 602 is determined to be an object for which collision has been avoided if it has entered across the right edge and bottom edge, and a collision avoidance determination area 603 is determined to be an object for which collision has been avoided.

[0057] <When detected within the collision avoidance judgment area, the object is tracked nearby depending on the direction of entry and determined to be the same object (when reversing)>

[0058] 10 is an overhead view of the detection positions of a vehicle and an object, illustrating an example of the operation of the camera device according to this embodiment, and illustrates a collision determination area and a collision avoidance determination area when reversing. Reference numerals 700, 701, 702, and 703 denote collision avoidance determination areas set around the vehicle 100. Reference numeral 704 denotes a collision determination area set around the vehicle 100. An object that enters this collision avoidance determination area is determined to have a possibility of collision avoidance, and is tracked using object proximity tracking (in other words, selected as a tracking target object for object proximity tracking processing), and is excluded from the image tracking targets. Reducing the number of image tracking targets reduces the processing load and image storage capacity.

[0059] The distances set for 710 to 715, 720, and 721 are set farther away depending on the speed and steering angle of the vehicle 100. This is for the same reasons as for 610 to 615, 620, and 621.

[0060] FIG. 11 is an overhead view of the detection positions of the vehicle and the object, illustrating an example of the operation of the camera device according to this embodiment, and explains the direction of entry into the collision avoidance determination area when reversing.

[0061] An object is not determined to be an object for which collision has been avoided simply by being detected as having entered these collision avoidance determination areas. As shown in Figure 11, an object in a position and traveling direction that has entered a collision avoidance determination area 700 is determined to be an object for which collision has been avoided if it has entered across the bottom edge (vehicle side), a collision avoidance determination area 701 is determined to be an object for which collision has been avoided if it has entered across the top edge (vehicle side), a collision avoidance determination area 702 is determined to be an object for which collision has been avoided if it has entered across the left and bottom edges, and a collision avoidance determination area 703 is determined to be an object for which collision has been avoided.

[0062] 12 is a bird's-eye view of the detection positions of a vehicle and objects, illustrating an example of the operation of the camera device according to this embodiment, and explains how objects that have entered the collision avoidance determination area while moving forward are narrowed down by nearby object tracking. Reference numerals 805, 804, 803, 802, and 801 denote the detection positions (object detection results) of the object 104 in each frame in chronological order. The object 104 enters the collision avoidance determination area 601 at 801, and 811 is the maximum range (tracking range) of 801. Since 811 includes 802 from the previous frame, it is determined that 801 and 802 are the same object. Furthermore, since the line segment connecting 801 and 802 intersects with the top edge of the collision avoidance determination area 601, it can be seen that the object entered the collision avoidance determination area 601 from the top edge of the collision avoidance determination area 601. Similarly, maximum range (tracking range) 812 of 802 includes only 803 in the previous frame, and is therefore determined to be the same object. Maximum range (tracking range) 813 of 803 includes only 804 in the previous frame, and is therefore determined to be the same object. Maximum range (tracking range) 814 of 804 includes two objects, 805 and 825 (object detection results for object 102), in the previous frame, and the same object cannot be identified. Since 801, 802, 803, and 804 are tracked as objects for which collision has been avoided, in the tracking process for these, they are tracked using object proximity tracking (in other words, selected as tracking target objects in object proximity tracking processing) and are excluded from image tracking, which reduces the number of image tracking operations and also enables the deletion of image areas (partial areas or all areas) that only include these objects, thereby reducing capacity.

[0063] [Outline of the embodiment and explanation of modifications] In the present embodiment, as described above, the camera device 200 includes an object detection unit 202 that detects an object, an image tracking unit 207 that verifies and tracks the same object using images, a speed calculation unit 211 that calculates the speed of the object, an object history storage unit 210 that stores history information of past objects, and an image storage unit 209 that stores past images, and is characterized by tracking the same object using past images. As a result, even if tracking all detected objects is difficult in terms of processing time, by assuming that it will be restarted in a later frame, processing can be performed in frames with remaining processing time, making it possible to track more objects.

[0064] Furthermore, by calculating the velocity using past information of the same object, accuracy can be improved by using past history. Furthermore, by calculating the velocity by removing the information from past information of the same object that is farthest from the linear approximation line, outlier removal during velocity calculation reduces the error in velocity calculation even if a sudden distance measurement error occurs. Since the outlier removal removes at least the farthest object, it is characterized by excluding the most distant object. Furthermore, by tracking the same object using past images for objects detected within a predetermined range determined to have a high collision probability, objects with a low collision risk are excluded from tracking, thereby reducing the load of the tracking process. The system includes a nearby object tracking unit 206 that determines that objects in nearby positions or previous and subsequent frames of image coordinates are the same object, and separates the objects tracked by the nearby object tracking unit 206 from those tracked by the image tracking unit 207. Since independently existing pedestrians can be determined as nearby objects and there is no risk of them jumping onto other objects, verification by image tracking, which requires a high processing load, is not necessary, thereby reducing the processing load. Delete past images for which tracking of detected objects has been completed. By deleting images that have already undergone tracking processing, the storage capacity of images can be reduced.

[0065] Furthermore, in a configuration with multiple cameras, even if images are taken at the same time, images that have completed tracking of detected objects can be deleted. By subdividing the items to be deleted for each camera, storage capacity can be further reduced even if there are images taken at the same time that have not yet been tracked.

[0066] By dividing the camera image into multiple regions and deleting the images from the regions after all tracking processes for detected objects have been completed, it is possible to delete images with angles of view where there is a high probability of the presence of that type of object, further reducing storage capacity.

[0067] The system is provided with a movable range determination unit that determines the range of movement that can be expected at an object's speed from the object's position at a certain time by determining the measured speed in advance, and by tracking the object as the same object if there is only one object in the previous or subsequent frame within the movable range, it is possible to determine that the object is the same object on the assumption that the object will not move at a speed that exceeds its possible speed, thereby reducing the number of objects that require image tracking, reducing the processing load, and reducing the image storage capacity.

[0068] Furthermore, by providing an identification unit 203 that identifies the type of object and using only nearby object tracking as a tracking method for a specific type, nearby object tracking can be performed according to the type, thereby further reducing the processing load and storage capacity.

[0069] By deleting only images of objects detected by nearby object tracking, not only the processing load of tracking but also capacity is reduced. A collision avoidance judgment area where it is determined that a collision has been avoided for the vehicle is predetermined, nearby objects are tracked going back in time from an object that has entered the collision avoidance judgment area, and objects that are identified as the same object are excluded from the image tracking. By tracking nearby objects for which it can be determined that a collision has been avoided and excluding them from the image tracking, the processing load is reduced.

[0070] When determining the movable range, by assuming a speed according to the type and setting the speed according to the type, it is possible to track a larger number of nearby objects more reliably, thereby further reducing the processing load.

[0071] The identification unit 203 identifies objects by direction, and by assuming different speeds for the movable range in the right and left directions and the approaching and moving away directions from the camera's viewpoint according to the direction of the object, the direction can also be identified, thereby more nearby objects can be tracked more reliably and the processing load can be further reduced. By saving at least the position, type, tracking method, and tracking implementation status as the object history, it is possible to trace back the speed estimation if the position is known, and by saving the tracking method and the tracking implementation status of the object, it is possible to avoid unnecessary tracking two or three times, and by doing at least these two things, the storage capacity of the history information can be further reduced.

[0072] [Summary] As described above, the image processing device 200B according to this embodiment includes an image acquisition unit 201 that acquires images captured by the imaging device 200A mounted on the vehicle 100, an image storage unit 209 that stores the images captured at past times, an object detection unit 202 that detects objects (around the vehicle) based on the images, and a first tracking unit (image tracking unit 207) that tracks the objects based on feature amounts of the objects included in the multiple images captured in time series. The first tracking unit (image tracking unit 207) references images of one or more first tracking target objects included in the objects detected by the object detection unit 202 among the multiple images stored in the image storage unit 209, and searches for feature amounts of the objects from images captured at a first time point (current time) to images captured at a second time point (past time) prior to the first time point, thereby tracking the first tracking target objects and outputting the tracking results. In other words, the image tracking is performed retroactively.

[0073] The image processing device 200B according to this embodiment also includes an identification unit 203 that identifies the type of the object based on the image, a second tracking unit (nearby object tracking unit 206) that tracks one or more second tracking target objects included in the objects detected by the object detection unit 202 based on the type of the object captured in the multiple images captured in time series, and an image deletion unit 208 that deletes some or all of the images in which the second tracking target objects tracked by the second tracking unit (nearby object tracking unit 206) are captured from the image storage unit 209. That is, the images are deleted after tracking using nearby object tracking based on the object type.

[0074] Furthermore, the second tracking unit (nearby object tracking unit 206) sets a tracking range depending on the type of the object, and if only one object of the same type is present within the set tracking range in an image captured at a first time and an image captured at a second time before the first time, tracks the object as being the same object. That is, it determines that the two objects are the same object by tracking nearby objects using an assumed prediction circle or the like.

[0075] Moreover, the image processing device 200B according to this embodiment includes a selection unit (tracking method selection unit 205) that selects, for each of the objects, either the first tracking target object, the second tracking target object, or a non-tracking target object other than the first tracking target object and the second tracking target object.

[0076] In addition, the image processing device 200B according to this embodiment includes an area setting unit 204 that sets a collision avoidance judgment area around the host vehicle to determine the risk of collision between the host vehicle and an object, and the selection unit (tracking method selection unit 205) selects an object that exists in the collision avoidance judgment area as the second object to be tracked.

[0077] In addition, the selection unit (tracking method selection unit 205) selects, for each object, either the first tracked object, the second tracked object, or the non-tracked object, depending on the position and traveling direction of the vehicle and the position and traveling direction of the object.

[0078] Furthermore, the image processing device 200B according to this embodiment includes an image deletion unit 208 that deletes a partial region or the entire region of an image stored in the image storage unit 209, and the image deletion unit 208 deletes a partial region or the entire region of the image up to the time when tracking by the first tracking unit (image tracking unit 207) or the second tracking unit (nearby object tracking unit 206) is completed.

[0079] In addition, the image processing device 200B according to this embodiment includes an object history storage unit 210 that stores object history information including at least the position, type, and tracking execution status of the object, and the first tracking unit (image tracking unit 207) tracks the first tracking target object based on the object history information stored in the object history storage unit 210.

[0080] In other words, the image processing device 200B according to this embodiment performs image tracking going back in time of objects that enter a predetermined area near the camera device 200 or the vehicle 100 that is equipped with it, and excludes objects that can be reliably tracked by nearby object tracking from the image tracking, thereby performing the minimum amount of image tracking necessary and reducing the image volume required for tracking going back in time.

[0081] According to this embodiment, an object detected in the vicinity of a possible control target is tracked by tracing past images. The problem of an increase in the storage capacity of past images is reduced by narrowing down the data necessary for image tracking in the above order and deleting unnecessary data. This reduces the processing load and image storage capacity, while reliably tracking objects related to vehicle control.

[0082] It should be noted that the present invention is not limited to the above-described embodiment, and includes various modifications. For example, the above-described embodiment has been described in detail to clearly explain the present invention, and the present invention is not necessarily limited to an embodiment having all of the described configurations.

[0083] Furthermore, the above-described configurations, functions, processing units, processing means, etc. may be partially or entirely implemented in hardware, for example, by designing them as integrated circuits. The above-described configurations, functions, etc. may also be implemented in software, with a processor interpreting and executing a program that implements each function. Information such as the programs, tables, and files that implement each function can be stored in a memory, a storage device such as a hard disk or SSD (Solid State Drive), or a recording medium such as an IC card, SD card, or DVD.

[0084] In addition, the control lines and information lines shown are those that are considered necessary for the explanation, and do not necessarily show all the control lines and information lines in the product. In reality, it can be assumed that almost all components are interconnected.

[0085] DESCRIPTION OF SYMBOLS 100... Vehicle (host vehicle) 101... Collision determination area 200... Camera device 200A... Imaging device 200B... Image processing device 201... Image acquisition unit 202... Object detection unit 203... Identification unit 204... Area setting unit 205... Tracking method selection unit (selection unit) 206... Nearby object tracking unit (second tracking unit) 207... Image tracking unit (first tracking unit) 208... Image deletion unit 209... Image storage unit 210... Object history storage unit 211... Speed ​​calculation unit 212... Result output unit 300 to 306, 308 to 314... Maximum range of nearby object tracking (tracking range) 410 to 416... Maximum range taking into account the direction of nearby object tracking (tracking range) 600 to 603... Collision avoidance determination area when moving forward 700 to 703... Collision avoidance determination area when moving backward

Claims

1. An image processing apparatus comprising: an image acquisition unit that acquires images captured by an imaging device mounted on the vehicle; an image storage unit that stores the images captured at past points in time; an object detection unit that detects objects based on the images; and a first tracking unit that tracks the objects based on feature amounts of the objects included in a plurality of the images captured in time series, wherein the first tracking unit references, from the plurality of images stored in the image storage unit, images in which one or more first tracked objects included in the objects detected by the object detection unit are captured, and tracks the first tracked objects by searching for feature amounts of the objects from images captured at a first point in time to images captured at a second point in time prior to the first point in time, and outputs the tracking results.

2. An image processing device according to claim 1, comprising: an identification unit that identifies the type of object based on the image; a second tracking unit that tracks one or more second tracking target objects included in the objects detected by the object detection unit based on the type of object captured in the multiple images captured in time series; and an image deletion unit that deletes some or all of the images in which the second tracking target objects tracked by the second tracking unit are captured from the image storage unit.

3. An image processing device according to claim 2, wherein the second tracking unit sets a tracking range according to the type of object, and if there is only one object of the same type within the set tracking range in an image captured at a first time and an image captured at a second time earlier than the first time, the second tracking unit tracks the object as being the same object.

4. An image processing device according to claim 2, comprising a selection unit that selects, for each of the objects, either the first tracking target object, the second tracking target object, or a non-tracking target object other than the first tracking target object and the second tracking target object.

5. An image processing device according to claim 4, comprising an area setting unit that sets a collision avoidance judgment area around the host vehicle to determine the risk of collision between the host vehicle and an object, and the selection unit selects an object that exists in the collision avoidance judgment area as the second object to be tracked.

6. An image processing device according to claim 4, wherein the selection unit selects, for each object, either the first tracked object, the second tracked object, or the non-tracked object according to the position and traveling direction of the host vehicle and the position and traveling direction of the object.

7. An image processing device according to claim 2, comprising an image deletion unit that deletes a partial area or the entire area of an image stored in the image storage unit, wherein the image deletion unit deletes a partial area or the entire area of the image up to the time when tracking by the first tracking unit or the second tracking unit is completed.

8. An image processing device according to claim 1, comprising an object history storage unit that stores object history information including at least the position, type, and tracking execution status of the object, and the first tracking unit tracks the first tracked object based on the object history information stored in the object history storage unit.

Images