Image processing apparatus

Abstract

To provide an image processing apparatus capable of reliably tracking an object related to vehicle control, while suppressing a processing load and image storage capacity, when tracking is performed using camera image texture in a situation where many objects exist in the vicinity.SOLUTION: An image processing apparatus performs image tracking of an object that has entered a predetermined region near a camera device 200 or a vehicle 100 equipped with the camera device, in a direction going back in time, performs image tracking by the minimum necessary amount by excluding objects that can be reliably tracked by nearby object tracking from image tracking, and suppresses the image storage capacity required for retroactive tracking.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an image processing device. [Background technology]

[0002] Prior art includes a technology such as that shown in Patent Document 1, which discloses a technology for continuously detecting and tracking an object (target) that may be a control target. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2018-80938 Summary of the Invention [Problem to be solved by the invention]

[0004] In recent years, vehicles equipped with autonomous driving (AD) systems and advanced driver-assistance systems (ADAS) have become increasingly common. To realize these systems, images from cameras mounted on the vehicle are processed to recognize the vehicle's external environment. AD and ADAS cameras estimate not only the object's position but also its speed to perform collision detection, which requires tracking of the same object. In object tracking, nearby object tracking, which determines that objects that are nearby on a time axis are the same object and tracks them, can lead to object transfers in situations where there are many nearby objects and the area is crowded. Therefore, image tracking is used, which identifies and tracks the same object based on its appearance (the texture of the camera image). However, image tracking imposes a heavy processing load, making it difficult to track all objects.

[0005] The present invention has been made in consideration of the above-mentioned problems, and aims to provide an image processing device that can reliably track objects related to vehicle control while reducing processing load and image storage capacity when tracking using the texture of a camera image in a situation where there are many objects in the vicinity. [Means for solving the problem]

[0006] In order to solve the above problem, the image processing device of the present invention includes an image acquisition unit that acquires images captured by an imaging device mounted on the vehicle itself, an image storage unit that stores the images captured at past times, an object detection unit that detects objects based on the images, and a first tracking unit that tracks the objects based on feature amounts of the objects included in the plurality of images captured in time series, wherein the first tracking unit references images of one or more first tracked objects included in the objects detected by the object detection unit among the plurality of images stored in the image storage unit, and searches for feature amounts of the objects from images captured at a first time point to images captured at a second time point earlier than the first time point, thereby tracking the first tracked objects and outputting the tracking results. [Effects of the Invention]

[0007] According to the present invention, an object detected in the vicinity of a possible control target is tracked by tracing past images. The problem of increasing the storage capacity of past images is prevented by narrowing down the data necessary for image tracking in the following order and deleting unnecessary data. This allows reliable tracking of objects related to vehicle control while reducing the processing load and image storage capacity.

[0008] Problems, configurations, and effects other than those described above will become apparent from the following description of the embodiments. [Brief explanation of the drawings]

[0009] [Figure 1] 10A and 10B are overhead views of the detection positions of a vehicle and an object, illustrating an example of the operation of the camera device according to the present embodiment. [Figure 2] 4 is a flowchart of the camera device according to the present embodiment. [Figure 3] FIG. 1 is a block diagram of a camera device according to an embodiment of the present invention. [Figure 4] 10 is a bird's-eye view of the detection positions of the vehicle and the object, illustrating an example of the operation of the camera device according to the present embodiment, narrowed down to the detection positions of objects of a type other than a specific type. [Figure 5] 10 is a bird's-eye view of the detection positions of the vehicle and the object, illustrating an example of the operation of the camera device according to the present embodiment, narrowed down to those that can be determined to be the same object using the nearby object tracking method. FIG. [Figure 6] 10 is a bird's-eye view of the detection positions of vehicles and objects, illustrating an example of the operation of the camera device according to this embodiment, narrowed down to those that can be determined to be the same object of a specific type using the nearby object tracking method. [Figure 7] 10 is a bird's-eye view of the detection positions of a vehicle and an object, illustrating an example of the operation of the camera device according to the present embodiment, and explains the maximum movement range of a specific type of object in the nearby object tracking method. [Figure 8] 10A and 10B are overhead views of the detection positions of the vehicle and the object, illustrating an example of the operation of the camera device according to the present embodiment, and illustrate a collision determination area and a collision avoidance determination area when moving forward. [Figure 9] 10 is a bird's-eye view of the detection positions of the vehicle and the object, illustrating an example of the operation of the camera device according to the present embodiment, and explains the direction of entry into the collision avoidance determination area when moving forward. [Figure 10] 10 is a bird's-eye view of the detection positions of the vehicle and the object, illustrating an example of the operation of the camera device according to the present embodiment, and illustrates a collision determination area and a collision avoidance determination area when reversing. FIG. [Figure 11] 10 is a bird's-eye view of the detection positions of the vehicle and the object, illustrating an example of the operation of the camera device according to the present embodiment, and explains the direction of entry into the collision avoidance determination area when reversing. FIG. [Figure 12] FIG. 10 is an overhead view of the vehicle and object detection positions illustrating an example of the operation of the camera device according to this embodiment, illustrating how objects that have entered the collision avoidance determination area when moving forward are narrowed down by tracking nearby objects. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, embodiments of the present invention will be described with reference to the drawings. The examples are illustrative of the present invention, and for clarity of explanation, appropriate omissions and simplifications have been made. The present invention can be implemented in various other forms. Unless otherwise specified, each component may be singular or plural.

[0011] In the embodiments, processing performed by executing a program may be described. Here, a computer executes the program using a processor (e.g., a CPU or a GPU) and performs processing defined by the program using storage resources (e.g., a memory) and interface devices (e.g., a communication port). Therefore, the entity performing the processing by executing the program may be the processor. Similarly, the entity performing the processing by executing the program may be a controller, device, system, computer, or node having a processor. The entity performing the processing by executing the program may be any computing unit, and may include a dedicated circuit that performs specific processing. Here, the dedicated circuit may be, for example, an FPGA (Field Programmable Gate Array), an ASIC (Application Specific Integrated Circuit), or a CPLD (Complex Programmable Logic Device).

[0012] A program may be installed on a computer from a program source. The program source may be, for example, a program distribution server or a computer-readable storage medium. When the program source is a program distribution server, the program distribution server may include a processor and a storage resource for storing the program to be distributed, and the processor of the program distribution server may distribute the program to be distributed to other computers. In addition, in an embodiment, two or more programs may be realized as one program, or one program may be realized as two or more programs.

[0013] An embodiment of a camera device equipped with an image processing device of the present invention will be described with reference to FIGS.

[0014] [Explanation of the relationship between the vehicle and surrounding objects used as an example] FIG. 1 is an overhead view of a vehicle and the detection positions of objects, illustrating an example of the operation of a camera device according to this embodiment. Objects having the reference symbols described below are located as shown in FIG. 1 and move over time. The camera device 200 of this embodiment is mounted on a vehicle 100. 100 is a vehicle (host vehicle) on which the camera device 200 is mounted.

[0015] Reference numeral 101 denotes a collision detection area set around the vehicle 100. When an object is detected in this collision detection area 101, the same object is identified by tracing back images and the way in which the object has moved is determined.

[0016] Reference numeral 102 denotes an object that has entered the collision detection area 101 and is classified as a pedestrian. Reference numerals 103 and 104 denote objects that have not entered the collision detection area 101 and are also classified as pedestrians. Reference numerals 105, 106, 107, and 108 denote vehicles, such as a preceding vehicle 105, a following vehicle 106, and oncoming vehicles 107 and 108, respectively. Reference numeral 109 denotes a person riding a bicycle (cyclist). Reference numeral 110 denotes a pedestrian.

[0017] Reference numeral 111 denotes a legend for the position of an object (object detection result) detected and measured by the camera device 200, including distance measurement errors. Reference numeral 112 denotes a legend for the arrow indicating the true position of the object and its direction of movement for the object 111. Objects 102 to 110 are represented by the legends 111 and 112, respectively.

[0018] In the example of FIG. 1, 102, 103, and 104 are located in areas close to each other at a certain time, and the probability of mistracking is high.

[0019] [Flow description] The (image processing device of) the camera device according to this embodiment is characterized in that it basically operates in accordance with the flowchart shown in FIG.

[0020] S201 is an image acquisition process, which acquires images captured by an imaging device such as a camera. There are one or more imaging devices, and the images are divided into multiple field of view areas for each imaging device and saved.

[0021] S202 is an object detection process, which is a process for detecting an object that appears in the image acquired in S201 (image acquisition process).

[0022] S203 is an identification process, which is a process for identifying the type of object detected in S202 (object detection process). The types include, for example, a crossing vehicle, other vehicles, a crossing motorcycle, other motorcycles, a pedestrian, a cyclist, etc. In S203 (identification process), the orientation of the object is also identified, for example, into four directions or eight directions.

[0023] S204 is an area setting process, which sets a collision determination area, a collision avoidance determination area, etc. around the vehicle 100 to determine the risk of collision between the vehicle 100 and an object.

[0024] S205 is a tracking method selection process, which selects whether to perform nearby object tracking processing or image tracking processing.

[0025] S206 is a nearby object tracking process, in which objects of the same type that are detected in the images of the previous and next frames are identified as the same object and tracked.

[0026] S207 is an image tracking process in which objects detected in images of previous and next frames are identified as the same object if they have matching image features (quantities), and are tracked.

[0027] S208 is an image deletion process, which deletes unnecessary images. In S208 (image deletion process), image areas (partial areas or the entire area of the image) that only include nearby objects that have been decided to be tracked and objects that have completed tracking processes (nearby object tracking process, image tracking process) are deleted.

[0028] S209 is an image saving process, which is a process for saving an image.

[0029] S210 is an object history saving process, which saves the object history, including the object's position, type, orientation, tracking method, and tracking completion status.

[0030] S211 is a speed calculation process in which the speed is calculated using the object history information saved in S210 (object history saving process).

[0031] S212 is the output process of the object recognition results, and outputs the detected, identified, and calculated object position, type, and speed to the brake control device, steering control device, etc. installed in the vehicle to control the vehicle and realize automatic braking, for example.

[0032] [Block diagram explanation] 3 is a block diagram of a camera device according to this embodiment. Reference numeral 200 denotes the camera device according to this embodiment. The camera device 200 of this embodiment includes an imaging device 200A and an image processing device 200B.

[0033] 200A is an imaging device mounted on the vehicle 100. The imaging device 200A is a camera equipped with an imaging sensor (such as a CMOS: Complementary Metal Oxide Semiconductor) that converts light into an electrical signal. The information converted into an electrical signal by the imaging sensor is further converted into image data representing the captured image within the imaging device 200A. The images captured by the imaging device 200A are transmitted to the image processing device 200B at predetermined intervals. The imaging device 200A captures images of the outside world (surroundings) of the vehicle 100 and transmits (outputs) the images to the image processing device 200B.

[0034] An image processing device 200B processes images received from the imaging device 200A. The image processing device 200B includes an image acquisition unit 201, an object detection unit 202, a classification unit 203, an area setting unit 204, a tracking method selection unit 205, a nearby object tracking unit 206, an image tracking unit 207, an image deletion unit 208, an image storage unit 209, an object history storage unit 210, a speed calculation unit 211, and a result output unit 212.

[0035] An image acquisition unit 201 acquires an image captured by the imaging device 200A.

[0036] An object detection unit 202 detects an object from the image acquired by the image acquisition unit 201 .

[0037] Reference numeral 203 denotes an identification unit that identifies the type of object detected by the object detection unit 202 .

[0038] Reference numeral 204 denotes a region setting unit, which is a processing unit that sets collision determination regions 101, 704 and collision avoidance determination regions 600, 601, 602, 603, 700, 701, 702, 703, etc., around the vehicle 100 to determine the risk of collision between the vehicle 100 and an object (see also FIGS. 8 to 11, etc.). The region setting unit 204 acquires wheel speed, steering angle, actual steering angle, yaw rate sensor, inertial sensor, GPS position information, etc. from the vehicle 100, and determines how the host vehicle is traveling, as well as the speed and turning radius. When the host vehicle is moving forward, regions 101, 600, 601 are set in front of the vehicle, and regions 602, 603 are set behind or to the sides of the vehicle, as shown in FIG. 8. When the host vehicle is moving backward, regions 704, 700, 701 are set behind the vehicle, and regions 702, 703 are set in front of or to the sides of the vehicle, as shown in FIG. 10. The distances from 621 to 620, 612 to 611, 613 to 614, 612 to 610, 613 to 615, 721 to 720, 712 to 711, 713 to 714, 712 to 710, and 713 to 715 are increased in correlation with the speed of the vehicle. By increasing the distances in correlation with the speed, when the vehicle is traveling at a low speed, the speed of pedestrians and cyclists is relatively high compared to the speed of the vehicle, and there is a high possibility of a collision with the vehicle if a pedestrian or cyclist suddenly jumps out, so the number of situations where the brakes are applied increases, thereby reducing accidents. Similarly, by lengthening the distance in correlation with the speed, when a vehicle is traveling at high speed, it is more likely that a pedestrian or cyclist who has avoided a collision will be able to pass by without a collision, rather than changing direction, jumping out, and colliding again; on the other hand, automatic braking while traveling at high speed can reduce the possibility of a rear-end collision with a following vehicle or the vehicle becoming uncontrollable and resulting in a self-inflicted accident.

[0039] Furthermore, when the host vehicle is turning right with respect to the traveling direction while moving forward, the smaller the turning radius, the wider the collision detection region 101 in the traveling direction of the vehicle can be, thereby increasing the accuracy of collision detection during turning. Conversely, when the host vehicle is turning left with respect to the traveling direction while moving forward, the smaller the turning radius, the wider the collision detection region 101 in the traveling direction of the vehicle can be, thereby increasing the accuracy of collision detection during turning.

[0040] Furthermore, when the host vehicle is turning to the right in the traveling direction while backing up, the smaller the turning radius, the wider the collision detection region 704 in the traveling direction of the vehicle can be, thereby increasing the accuracy of collision detection during turning, by changing the ratio of the distance from 713 to 714 to the distance from 712 to 711 so that the distance from 712 to 711 is larger. Conversely, when the host vehicle is turning to the left in the traveling direction while backing up, the smaller the turning radius, the wider the collision detection region 704 in the traveling direction of the vehicle can be, by changing the ratio of the distance from 713 to 714 to the distance from 712 to 711 so that the distance from 713 to 714 is larger.

[0041] A tracking method selection unit 205 selects whether to perform nearby object tracking processing or image tracking processing. In other words, the tracking method selection unit 205 selects, for each detected and identified object, a tracking target object for nearby object tracking processing, a tracking target object for image tracking processing, or a non-tracking target object other than the tracking target object for nearby object tracking processing and the tracking target object for image tracking processing.

[0042] A nearby object tracking unit 206 compares images of different frames taken in succession in time, identifies nearby objects of the same type as the same object, and tracks them. In other words, the nearby object tracking unit 206 tracks one or more tracking target objects of the nearby object tracking process that are included in the object detected by the object detection unit 202, based on the type of object captured in multiple images taken in time series.

[0043] An image tracking unit 207 compares images from different frames taken in time, and identifies and tracks those images with matching image features (quantities) as the same object. In other words, the image tracking unit 207 tracks an object based on the feature quantities of the object contained in multiple images taken in time series. The image tracking unit 207 references, from the multiple images stored in the image storage unit 209, images in which one or more tracking target objects of the image tracking process included in the object detected by the object detection unit 202 are captured, and tracks the tracking target object of the image tracking process by searching for the feature quantities of the object in images taken in the past from the image taken at the current time (going back in time), and outputs the tracking results.

[0044] Reference numeral 209 denotes an image storage unit, which divides an image captured at a previous time into multiple regions and stores the divided images in the memory M209. The image may be divided, for example, by cutting out and saving the region of interest (ROI) of the object identified by the identification unit 203. When saving, the images are saved in a ring buffer-like manner in the working memory where image processing such as classification processing is performed. In this way, images must be saved using the working memory for purposes such as cropping, resizing, and normalization of images during classification processing, but if the image is saved in a ring buffer-like manner that is originally used as the working memory, there is no need to save the images again.

[0045] An image deletion unit 208 deletes, from the image areas stored in the memory M209, image areas in which an object tracked by the nearby object tracking unit 206 has been captured (partial or all of the image) and image areas that have been completely tracked (partial or all of the image up to the time when tracking by the nearby object tracking unit 206 or the image tracking unit 207 has been completed).

[0046] An object history storage unit 210 stores the object history in a memory M210.

[0047] A speed calculation unit 211 calculates the speed of an object using historical information about the object.

[0048] 212 is a result output unit that outputs the position, type, and speed of the detected, identified, and calculated object to the vehicle 100, and controls the vehicle using a brake control device, steering control device, etc. provided on the vehicle 100, thereby realizing, for example, automatic braking.

[0049] [Explain the features of the implementation example using examples] <Use tracking methods for types other than specific types as nearby object tracking> FIG. 4 is a bird's-eye view of the detection positions of vehicles and objects illustrating an example of the operation of the camera device according to this embodiment, narrowing down the detection positions of objects of a type other than a specific type. In this embodiment, a tracking process is selected depending on the type of object. As described above, two types of tracking processes are provided: nearby object tracking and image tracking. While nearby object tracking has a low processing load, it carries the risk of mistracking if an object of the same type is present nearby. While image tracking has a high processing load, it can identify objects with similar image features, reducing the risk of mistracking. Other vehicles (traveling in the forward and opposite lanes) and other motorcycles are unlikely to suddenly appear and do not appear to move significantly in the image coordinate system. They also rarely change direction of movement suddenly. Therefore, there are rarely multiple objects in the vicinity, and the risk of mistracking in nearby object tracking is low. Nearby object tracking is used for these types of objects (objects 105, 106, 107, and 108) that have a low risk of mistracking in nearby object tracking. When these nearby object tracking methods are selected, the images in which the object is captured (partly or entirely) are deleted, which reduces the processing load and the size of the stored images.

[0050] <Determining the identity of an object by tracking its proximity using a predicted movement range> FIG. 5 is an overhead view of the detection positions of vehicles and objects, illustrating an example of the operation of a camera device according to this embodiment. The nearby object tracking method narrows down the objects to those that can be determined to be the same object. In this embodiment, the tracking method for objects of a type that suddenly appear and move significantly on the image coordinates—pedestrians, cyclists, crossing vehicles, and crossing motorcycles (objects 109 and 110)—determines a maximum range (hereinafter also referred to as a tracking range) within which each type of object can move at a predetermined speed. If only one object of the same type exists within the tracking range in an image captured at a certain time and an image captured at an earlier time, the object is determined to be the same object and tracked. 300 to 306 represent the maximum range (tracking range) for object 110, and 308 to 314 represent the maximum range (tracking range) for object 109. The maximum ranges (tracking ranges) for objects 109 and 110 are determined to be the ranges (estimated predicted circles) of the movement amounts of 315 and 307, respectively.

[0051] Fig. 6 is an overhead view of the detection positions of vehicles and objects, illustrating an example of the operation of the camera device according to this embodiment, in which the objects are narrowed down to those that can be determined to be the same object of a specific type using the nearby object tracking method. This is an example that can further improve the accuracy of nearby object tracking from Fig. 5, by assuming that the maximum movement range of an object is moving in the direction (orientation) identified by the object classifier, thereby reducing the number of objects that exist within the maximum movement range and increasing the number of nearby objects that can be tracked.

[0052] 7 is an overhead view of the detection positions of a vehicle and an object, illustrating an example of the operation of the camera device according to this embodiment, and illustrates the maximum movement range of a specific type of object in a nearby object tracking method. For object position 420 at a certain time in 109, an assumed movement speed of 503 is assumed, and a range of maximum speed error 410 for this is used for nearby object tracking. 501 indicates an image captured by camera 500, which is an imaging device. 420 is identified as an object moving left and right as seen from camera 500. As a result, 503 is estimated as the assumed movement speed of an object moving in a direction perpendicular to line 502 connecting 500 and 420. As a result, as shown in FIG. 6, for object position 420 at a certain time in 109, the range of movement amount 410 (assumed prediction ellipse) can be defined as the tracking range, and similarly, tracking ranges 411 to 416 can be defined thereafter.

[0053] <When detected in the collision avoidance judgment area, the object is tracked nearby depending on the direction of entry and determined to be the same object (when moving forward)> 8 is a bird's-eye view of the detection positions of a vehicle and an object, illustrating an example of the operation of the camera device according to this embodiment, and illustrates the collision determination area and collision avoidance determination area when moving forward. Reference numerals 600, 601, 602, and 603 indicate collision avoidance determination areas set around the vehicle 100. An object that enters this collision avoidance determination area is determined to have the potential for collision avoidance, and is tracked using object proximity tracking (in other words, selected as a tracking target object for object proximity tracking processing), and is excluded from the image tracking targets. Reducing the number of image tracking targets reduces the processing load and image storage capacity.

[0054] Regarding the distance settings of 610 to 615, 620, and 621, the distances are set farther away depending on the speed and steering angle of the vehicle 100. When the speed of the vehicle 100 is fast, the brakes should be applied if there are pedestrians or the like nearby, which can reduce the accident rate, compared to when the speed of the vehicle 100 is slow. When the vehicle 100 has a steering angle, applying the brakes over a wider range can reduce the accident rate compared to when there is no steering angle. Furthermore, when the speed range is high or there is a steering angle, the driver and passengers will feel less uncomfortable even if the brakes are controlled for nearby objects, even if they are not subject to collision.

[0055] FIG. 9 is an overhead view of the detection positions of the vehicle and the object, illustrating an example of the operation of the camera device according to this embodiment, and explains the approach direction in the collision avoidance determination area when moving forward.

[0056] An object is not judged as an object for which collision has been avoided simply by being detected as having entered these collision avoidance judgment areas. As shown in Fig. 9, an object in a position and traveling direction that has entered a collision avoidance judgment area 600 by straddling the bottom side (vehicle side), a collision avoidance judgment area 601 by straddling the top side (vehicle side), a collision avoidance judgment area 602 by entering from the right side and bottom side, and a collision avoidance judgment area 603 by entering from the right side and top side, is deemed to have been collision-avoided.

[0057] <When detected within the collision avoidance judgment area, the object is tracked nearby depending on the direction of entry and determined to be the same object (when reversing)>

[0058] FIG. 10 is an overhead view of the detection positions of a vehicle and an object, illustrating an example of the operation of the camera device according to this embodiment, and illustrates the collision determination area and collision avoidance determination area when reversing. 700, 701, 702, and 703 are collision avoidance determination areas set around the vehicle 100. 704 is a collision determination area set around the vehicle 100. An object that enters this collision avoidance determination area is determined to have the potential for collision avoidance, and is tracked using object proximity tracking (in other words, selected as a tracking target object for object proximity tracking processing), and is excluded from the image tracking targets. Reducing the number of image tracking targets reduces the processing load and image storage capacity.

[0059] To explain the distance setting for 710-715, 720, and 721, the distance is set farther depending on the speed and steering angle of the vehicle 100. This is for the same reasons as for 610-615, 620, and 621.

[0060] FIG. 11 is an overhead view of the detection positions of the vehicle and the object, illustrating an example of the operation of the camera device according to this embodiment, and explains the direction of entry into the collision avoidance determination area when reversing.

[0061] An object is not judged as an object for which collision has been avoided simply by being detected as having entered these collision avoidance judgment areas. As shown in Fig. 11, an object in a position and traveling direction that has entered a collision avoidance judgment area 700 is deemed to have been collision-avoided if it has straddled the bottom edge (vehicle side), a collision avoidance judgment area 701 is deemed to have been collision-avoided if it has straddled the top edge (vehicle side), a collision avoidance judgment area 702 is deemed to have been collision-avoided if it has straddled the left and bottom edges, and a collision avoidance judgment area 703 is deemed to have been collision-avoided if it has straddled the left and top edges.

[0062] <Exclusion of objects that enter the collision avoidance detection area> FIG. 12 is a bird's-eye view of the detection positions of a vehicle and an object, illustrating an example of the operation of the camera device according to this embodiment, and explains how objects that have entered the collision avoidance determination area while moving forward are narrowed down by nearby object tracking. 805, 804, 803, 802, and 801 are the detection positions (object detection results) of object 104 in each frame, respectively, in chronological order. 801 is where object 104 enters the collision avoidance determination area 601, and 811 is the maximum range (tracking range) of 801. 811 includes 802 from the previous frame, so 801 and 802 are determined to be the same object. Furthermore, the line segment connecting 801 and 802 intersects with the top edge of the collision avoidance determination area 601, indicating that object 104 entered the collision avoidance determination area 601 from the top edge of the collision avoidance determination area 601. Similarly, 812, the maximum range (tracking range) of 802, includes only 803 in the previous frame, so they are determined to be the same object. The maximum range (tracking range) 813 of 803 included only 804 in the previous frame, and it is determined that they are the same object. The maximum range (tracking range) 814 of 804 included two objects, 805 and 825 (object detection results for object 102), in the previous frame, and it is not possible to identify them as the same object. Since 801, 802, 803, and 804 are tracked as objects for which collisions have been avoided, in the tracking process for these, they are tracked using object proximity tracking (in other words, they are selected as objects to be tracked in object proximity tracking processing) and are excluded from image tracking, which reduces the number of image tracking operations and also enables the deletion of image areas (partial areas or all areas) that only include these objects, thereby reducing capacity.

[0063] [Summary of the embodiment and explanation of modifications] In this embodiment, as described above, camera device 200 includes object detection unit 202 that detects objects, image tracking unit 207 that verifies and tracks the same object in images, speed calculation unit 211 that calculates the speed of the object, object history storage unit 210 that stores history information of past objects, and image storage unit 209 that stores past images, and is characterized by tracking the same object using past images. As a result, even if tracking all detected objects is difficult in terms of processing time, by assuming that it will be restarted in a later frame, processing can be performed in frames with remaining processing time, making it possible to track more objects.

[0064] Furthermore, by calculating the velocity using past information of the same object, accuracy can be improved by using past history. Furthermore, by calculating the velocity by removing the information from past information of the same object that is farthest from the linear approximation line, outlier removal during velocity calculation reduces the error in velocity calculation even if a sudden distance measurement error occurs. Since the outlier removal process removes at least the furthest object, it is characterized by excluding the furthest object. Furthermore, by tracking the same object using past images for objects detected within a predetermined range determined to have a high collision probability, objects with a low collision risk are excluded from tracking, thereby reducing the load of the tracking process. The system includes a nearby object tracking unit 206 that determines that objects in nearby positions or previous and subsequent frames of image coordinates are the same object, and separates the objects tracked by the nearby object tracking unit 206 from those tracked by the image tracking unit 207. Since independently existing pedestrians can be determined as nearby objects and there is no risk of them jumping onto other objects, verification using image tracking, which requires a high processing load, is not necessary, thereby reducing the processing load. Delete past images for which tracking of detected objects has been completed. By deleting images that have already undergone tracking processing, the storage capacity of images can be reduced.

[0065] Furthermore, in a configuration with multiple cameras, even if images are taken at the same time, images that have completed tracking of detected objects can be deleted. By subdividing the items to be deleted for each camera, storage capacity can be further reduced even if there are images taken at the same time that have not yet been tracked.

[0066] By dividing the camera image into multiple regions and deleting the images from the regions after all tracking processes for detected objects have been completed, it is possible to delete images with angles of view where there is a high probability of the presence of that type of object, further reducing storage capacity.

[0067] The system is provided with a movable range determination unit that determines the range of movement that can be expected at an object's speed from the object's position at a certain time by determining the measured speed in advance, and by tracking the object as the same object if there is only one object in the previous or subsequent frame within the movable range, it is possible to determine that the object is the same object on the assumption that the object will not move at a speed that exceeds its possible speed, thereby reducing the number of objects that require image tracking, reducing the processing load, and reducing the image storage capacity.

[0068] Furthermore, by providing an identification unit 203 that identifies the type of object and using only nearby object tracking as a tracking method for a specific type, nearby object tracking can be performed according to the type, thereby further reducing the processing load and storage capacity.

[0069] By deleting only images of objects detected by nearby object tracking, not only the processing load of tracking but also capacity is reduced. A collision avoidance judgment area where it is determined that a collision has been avoided for the vehicle is predetermined, nearby objects are tracked going back in time from an object that has entered the collision avoidance judgment area, and objects that are identified as the same object are excluded from the image tracking. By tracking nearby objects for which it can be determined that a collision has been avoided and excluding them from the image tracking, the processing load is reduced.

[0070] When determining the movable range, by assuming a speed according to the type and setting the speed according to the type, it is possible to track a larger number of nearby objects more reliably, thereby further reducing the processing load.

[0071] The identification unit 203 identifies objects by direction, and by assuming different speeds for the movable range in the right and left directions and the approaching and moving away directions from the camera's viewpoint according to the object's direction, the direction can also be identified, thereby more nearby objects can be tracked more reliably, and the processing load can be further reduced. By saving at least the position, type, tracking method, and tracking implementation status as object history, it is possible to trace back speed estimation if the position is known, and by saving the tracking method and the tracking implementation status of the object, it is possible to avoid unnecessary tracking two or three times, and by doing at least these two things, the storage capacity of history information can be further reduced.

[0072] [summary] As described above, the image processing device 200B according to this embodiment includes an image acquisition unit 201 that acquires images captured by the imaging device 200A mounted on the vehicle 100, an image storage unit 209 that stores the images captured at past times, an object detection unit 202 that detects objects (around the vehicle) based on the images, and a first tracking unit (image tracking unit 207) that tracks the objects based on feature amounts of the objects included in the multiple images captured in time series. The first tracking unit (image tracking unit 207) refers to images in which one or more first tracking target objects included in the objects detected by the object detection unit 202 are captured among the multiple images stored in the image storage unit 209, and searches for feature amounts of the objects in images captured at a first time point (current time) and images captured at a second time point (past time) before the first time point, thereby tracking the first tracking target objects and outputting the tracking results. In other words, it performs image tracking going back to the past.

[0073] The image processing device 200B according to this embodiment also includes an identification unit 203 that identifies the type of the object based on the image, a second tracking unit (nearby object tracking unit 206) that tracks one or more second tracking target objects included in the objects detected by the object detection unit 202 based on the type of the object captured in the multiple images captured in time series, and an image deletion unit 208 that deletes some or all of the images in which the second tracking target objects tracked by the second tracking unit (nearby object tracking unit 206) are captured from the image storage unit 209. That is, the image is deleted after tracking by nearby object tracking based on the object type.

[0074] Furthermore, the second tracking unit (nearby object tracking unit 206) sets a tracking range according to the type of the object, and if only one object of the same type exists within the set tracking range in an image captured at a first time and an image captured at a second time before the first time, the second tracking unit tracks the object as being the same object. That is, it determines that the two objects are the same object by tracking nearby objects using an assumed prediction circle or the like.

[0075] Moreover, the image processing device 200B according to this embodiment includes a selection unit (tracking method selection unit 205) that selects, for each of the objects, either the first tracking target object, the second tracking target object, or a non-tracking target object other than the first tracking target object and the second tracking target object.

[0076] In addition, the image processing device 200B according to this embodiment includes an area setting unit 204 that sets a collision avoidance judgment area around the host vehicle to determine the risk of collision between the host vehicle and an object, and the selection unit (tracking method selection unit 205) selects an object that exists in the collision avoidance judgment area as the second object to be tracked.

[0077] Further, the selection unit (tracking method selection unit 205) selects, for each object, either the first tracked object, the second tracked object, or the non-tracked object according to the position and traveling direction of the host vehicle and the position and traveling direction of the object.

[0078] Furthermore, the image processing device 200B according to this embodiment includes an image deletion unit 208 that deletes a partial region or the entire region of an image stored in the image storage unit 209, and the image deletion unit 208 deletes a partial region or the entire region of an image up to the time when tracking by the first tracking unit (image tracking unit 207) or the second tracking unit (nearby object tracking unit 206) is completed.

[0079] In addition, the image processing device 200B according to this embodiment includes an object history storage unit 210 that stores object history information including at least the position, type, and tracking execution status of the object, and the first tracking unit (image tracking unit 207) tracks the first tracking target object based on the object history information stored in the object history storage unit 210.

[0080] In other words, the image processing device 200B according to this embodiment performs image tracking going back in time of an object that enters a predetermined area near the camera device 200 or the vehicle 100 that is equipped with the camera device 200, and excludes objects that can be reliably tracked by nearby object tracking from the image tracking, thereby performing the minimum amount of image tracking necessary and reducing the image volume required for tracking going back in time.

[0081] According to this embodiment, an object detected in the vicinity of a possible control target is tracked by tracing past images. The problem of an increase in the storage capacity of past images is reduced by narrowing down the data necessary for image tracking in the above order and deleting unnecessary data. This reduces the processing load and image storage capacity, while reliably tracking objects related to vehicle control.

[0082] It should be noted that the present invention is not limited to the above-described embodiment, and includes various modifications. For example, the above-described embodiment has been described in detail to clearly explain the present invention, and the present invention is not necessarily limited to an embodiment having all of the described configurations.

[0083] Furthermore, the above-described configurations, functions, processing units, processing means, etc. may be partially or entirely implemented in hardware, for example, by designing them as integrated circuits. The above-described configurations, functions, etc. may also be implemented in software, with a processor interpreting and executing a program that implements each function. Information such as the programs, tables, and files that implement each function can be stored in a memory, a storage device such as a hard disk or SSD (Solid State Drive), or a recording medium such as an IC card, SD card, or DVD.

[0084] In addition, the control lines and information lines shown are those that are considered necessary for the explanation, and do not necessarily show all the control lines and information lines in the product. In reality, it can be assumed that almost all components are interconnected. [Explanation of symbols]

[0085] 100...Vehicle (own vehicle) 101...Collision detection area 200...Camera equipment 200A...imaging device 200B...Image processing device 201...Image acquisition unit 202...Object detection unit 203...Identification unit 204...Area setting section 205...Tracking method selection unit (selection unit) 206... Nearby object tracking unit (second tracking unit) 207...Image tracking unit (first tracking unit) 208...Image deletion section 209...Image storage section 210...Object History Storage Unit 211…Speed calculation unit 212...Result output section 300~306, 308~314...Maximum range for nearby object tracking (tracking range) 410~416...Maximum range (tracking range) that takes into account the direction of nearby object tracking 600~603...Collision avoidance judgment area when moving forward 700~703: Collision avoidance judgment area when reversing

Claims

1. an image acquisition unit that acquires an image captured by an imaging device mounted on the vehicle; an image storage unit for storing the images captured in the past; an object detection unit that detects an object based on the image; a first tracking unit that tracks the object based on feature amounts of the object included in the plurality of images captured in time series, The first tracking unit referring to an image in which one or more first tracking target objects included in the objects detected by the object detection unit are captured, among the plurality of images stored in the image storage unit; tracking the first tracking target object by searching for a feature amount of the object from an image captured at a first time point to an image captured at a second time point prior to the first time point; outputting the results of said tracking; Image processing device.

2. 2. The image processing device according to claim 1, an identification unit that identifies the type of the object based on the image; a second tracking unit that tracks one or more second tracking target objects included in the objects detected by the object detection unit based on the types of the objects captured in the plurality of images captured in time series; an image deletion unit that deletes from the image storage unit a part or all of images of the second tracking target object tracked by the second tracking unit, Image processing device.

3. 3. The image processing device according to claim 2, The second tracking unit setting a tracking range according to the type of the object; If only one object of the same type is present within the set tracking range in an image captured at a first time and an image captured at a second time before the first time, the object is tracked as being the same object. Image processing device.

4. 3. The image processing device according to claim 2, a selection unit that selects, for each of the objects, one of the first tracking target object, the second tracking target object, or a non-tracking target object other than the first tracking target object and the second tracking target object; Image processing device.

5. 5. The image processing device according to claim 4, an area setting unit that sets a collision avoidance determination area around the host vehicle to determine a collision risk between the host vehicle and an object; the selection unit selects an object present in the collision avoidance determination area as the second object to be tracked. Image processing device.

6. 5. The image processing device according to claim 4, the selection unit selects, for each of the objects, one of the first tracking target object, the second tracking target object, or the non-tracking target object according to a position and a traveling direction of the host vehicle and a position and a traveling direction of the object; Image processing device.

7. 3. The image processing device according to claim 2, an image deletion unit that deletes a partial region or the entire region of the image stored in the image storage unit; the image deletion unit deletes a partial region or an entire region of the image up to the time when tracking by the first tracking unit or the second tracking unit is completed. Image processing device.

8. 2. The image processing device according to claim 1, an object history storage unit that stores object history information including at least the position, type, and tracking execution status of the object; the first tracking unit tracks the first tracking target object based on the object history information stored in the object history storage unit; Image processing device.

Images