Image processing apparatus

Abstract

Provided is an image processing device capable of suppressing a processing load and an image storage capacity and reliably performing tracking of an object related to vehicle control when tracking using a texture of a camera image in a situation in which an object is present in the vicinity. For an object that enters a predetermined area in the vicinity of a camera device (200) and a vehicle (100) on which the camera device is mounted, image tracking is performed in a direction of a tracking time, and an object that can be reliably tracked by a nearby object tracking is excluded as a target of the image tracking, thereby performing a minimum necessary image tracking and suppressing an image capacity for tracking in the past.

Description

Technical Field

[0001] This invention relates to an image processing apparatus. Background Technology

[0002] As a prior art, there is a technology as shown in Patent Document 1, which discloses the continuous detection and tracking of objects (targets) that have the potential to become controlled objects.

[0003] Prior technology documents

[0004] Patent documents

[0005] Patent Document 1: Japanese Patent Application Publication No. 2018-80938 Summary of the Invention

[0006] The problem that the invention aims to solve

[0007] In recent years, vehicles equipped with Autonomous Driving (AD) and Advanced Driver-Assistance Systems (ADAS) have become increasingly common. To implement these systems, images from cameras mounted on the vehicle are processed to identify the vehicle's external environment. In AD and ADAS cameras, not only are the positions and speeds of objects estimated, and collision detection is performed; therefore, tracking of the same object is necessary. In object tracking, if nearby objects located close to each other on the timeline are identified as the same object and tracked accordingly, shifting can occur in crowded situations where many objects are nearby. Therefore, image tracking is used to identify and track the same object based on its appearance (texture in the camera image). However, image tracking is computationally intensive, and tracking all objects is difficult.

[0008] The present invention was made in view of the above-mentioned problems, and its object is to provide an image processing apparatus that can suppress processing load and image storage capacity when tracking using the texture of a camera image in a situation where most objects are present nearby, and can reliably perform object tracking related to vehicle control.

[0009] Methods for solving problems

[0010] To address the aforementioned issues, the image processing apparatus of the present invention comprises: an image acquisition unit that acquires images captured by a camera device mounted on the vehicle; an image storage unit that stores the images captured at past time points; an object detection unit that detects objects based on the images; and a first tracking unit that tracks the objects based on feature values ​​of the objects contained in a plurality of images captured in a time sequence. The first tracking unit refers to images of one or more first tracking target objects included in the objects detected by the object detection unit captured in the plurality of images stored by the image storage unit, explores the feature values ​​of the objects by comparing images captured at a first time point with images captured at a second time point earlier than the first time point, tracks the first tracking target objects, and outputs the tracking result.

[0011] The effects of the invention

[0012] According to the present invention, image tracking is performed on nearby objects that may become controllable objects in a retrospective manner. At this time, the data required for image tracking is filtered in the following order, and unnecessary data is deleted to suppress the increase in the storage capacity of problematic past images. Therefore, the processing load and image storage capacity can be suppressed, and object tracking processing related to vehicle control can be reliably performed.

[0013] The issues, components, and effects beyond those mentioned above become clear through the following description of implementation forms. Attached Figure Description

[0014] Figure 1 This is a top view illustrating the detection positions of vehicles and objects in an example of the operation of the camera device in this embodiment.

[0015] Figure 2 This is a flowchart of the camera device in this embodiment.

[0016] Figure 3 This is a block diagram of the camera device in this embodiment.

[0017] Figure 4 The description shows a top view of the detection positions of vehicles and objects in an example of the operation of the camera device in this embodiment, and the detection positions of objects filtered as other than a specific type.

[0018] Figure 5 This is a top view illustrating the detection positions of vehicles and objects in an example of the operation of the camera device in this embodiment, and is a view selected to be determined as the same object by the nearby object tracking method.

[0019] Figure 6This is a top view illustrating the detection positions of vehicles and objects in an example of the operation of the camera device in this embodiment, and is a diagram that filters out objects of the same type that can be identified as a specific kind by the nearby object tracking method.

[0020] Figure 7 This is a top view illustrating the detection positions of vehicles and objects in an example of the operation of the camera device in this embodiment, and a diagram illustrating the maximum movement range of a specific type of object in the nearby object tracking method.

[0021] Figure 8 This is a top view illustrating the detection positions of the vehicle and the object in an example of the operation of the camera device in this embodiment, and a diagram illustrating the collision determination area and the collision avoidance determination area when moving forward.

[0022] Figure 9 This is a top view illustrating the detection positions of the vehicle and object in an example of the operation of the camera device in this embodiment, and a diagram illustrating the intrusion direction of the collision avoidance determination area during forward movement.

[0023] Figure 10 This is a top view illustrating the detection positions of the vehicle and the object in an example of the operation of the camera device in this embodiment, and a diagram illustrating the collision determination area and the collision avoidance determination area when reversing.

[0024] Figure 11 This is a top view illustrating the detection positions of the vehicle and object in an example of the operation of the camera device in this embodiment, and a diagram illustrating the intrusion direction of the collision avoidance determination area during reversal.

[0025] Figure 12 This is a top view illustrating the detection positions of the vehicle and the object in an example of the operation of the camera device in this embodiment, and a diagram illustrating the process of filtering objects entering the collision avoidance determination area during forward movement by tracking nearby objects. Detailed Implementation

[0026] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. The embodiments are for illustrative purposes only, and omissions and simplifications have been appropriately made for clarity of description. The present invention can also be implemented in a variety of other ways. Unless otherwise specified, each constituent element may be single or multiple.

[0027] In embodiments, the processing performed by an executable program is sometimes described. Here, the computer executes the program via a processor (e.g., CPU, GPU), using storage resources (e.g., memory) and interface devices (e.g., communication ports) to perform the processing specified by the program. Therefore, the entity performing the processing by the executable program can also be the processor. Similarly, the entity performing the processing by the executable program can also be a controller, device, system, computer, or node having a processor. The entity performing the processing by the executable program can be any arithmetic unit, but it can also include dedicated circuitry for performing specific processing. Here, dedicated circuitry includes, for example, FPGA (Field Programmable Gate Array), ASIC (Application Specific Integrated Circuit), CPLD (Complex Programmable Logic Device), etc.

[0028] The program can also be installed onto the computer from a program source. The program source can be, for example, a program distribution server or a storage medium that the computer can read. When the program source is a program distribution server, the program distribution server includes a processor and storage resources for storing the program to be distributed. The processor of the program distribution server can also distribute the program to other computers. Furthermore, in this embodiment, two or more programs can be implemented as a single program, and a single program can be implemented as two or more programs.

[0029] Reference Figures 1-12 An embodiment of a camera device equipped with the image processing apparatus of the present invention will be described.

[0030] [An explanation of the relationship between the vehicle and its surrounding objects, presented as an example]

[0031] Figure 1 This is a top view illustrating the detection positions of a vehicle and an object, showing an example of the operation of the camera device in this embodiment. Objects with the symbols described below are located as follows: Figure 1 The position shown moves over time. In this embodiment, the camera device 200 is mounted on a vehicle 100. 100 is the vehicle (this vehicle) on which the camera device 200 is mounted.

[0032] 101 is a collision determination area set around the vehicle 100. Objects detected in the collision determination area 101 are identified as the same object by tracing back to the past and image tracking, and it is determined how the object moved.

[0033] Object 102 is a pedestrian that has entered collision detection area 101. Objects 103 and 104 are pedestrians that have not entered collision detection area 101. Objects 105, 106, 107, and 108 are vehicles, specifically the first vehicle 105, the following vehicle 106, and the oncoming vehicles 107 and 108. Object 109 is a cyclist. Object 110 is a pedestrian.

[0034] 111 is a legend mark representing the position (object detection result) of the object detected and measured by the camera device 200, including the ranging error. 112 is a legend mark representing the true position of the object relative to the object at 111 and an arrow indicating its direction of movement. Objects 102 to 110 are represented by legend marks 111 and 112, respectively.

[0035] exist Figure 1 In the example, 102, 103, and 104 exist in a region close to each other at a certain moment, which is a state with a high probability of false tracking.

[0036] [Process Description]

[0037] The camera device (image processing device) of this embodiment is characterized in that it substantially conforms to... Figure 2 The actions are performed according to the flowchart shown.

[0038] S201 is image acquisition processing, which involves acquiring images captured by camera devices such as video cameras. There is one or more video devices, and each video device is divided into multiple field-of-view areas and stored.

[0039] S202 is object detection processing, which is the process of detecting objects in the image acquired in S201 (image acquisition processing).

[0040] S203 is the identification process, which identifies the type of object detected in S202 (object detection process). Types include, for example, vehicles crossing the road, other vehicles, motorcycles crossing the road, other motorcycles, pedestrians, cyclists, etc. Furthermore, in S203 (identification process), the orientation of the object is also determined, for example, in 4 or 8 directions.

[0041] S204 is a region setting process, which is a process of setting up collision determination areas, collision avoidance determination areas, etc. around the vehicle 100 to determine the collision risk between the vehicle 100 and objects.

[0042] S205 is the tracking method selection process, which selects whether to perform nearby object tracking or image tracking.

[0043] S206 is a nearby object tracking process, which identifies objects of the same type that are nearby in the images of consecutive frames as the same object and tracks them.

[0044] S207 is image tracking processing, which is a process that identifies objects with consistent image features (quantities) as the same object and tracks them when they are detected in images of consecutive frames.

[0045] S208 is image deletion processing, which is the process of deleting unwanted images. In S208 (image deletion processing), only the image region (partial or full region of the image) of the object that was determined to be tracked by nearby objects and the object that completed the tracking processing (nearby object tracking processing, image tracking processing) was deleted.

[0046] S209 is image saving processing, which is the process of saving images.

[0047] S210 is the object history saving process, which saves the history of an object. As an object's history, it saves the object's position, type, orientation, tracking method, tracking completion status, etc.

[0048] S211 is the speed calculation process, which uses the object's history information saved in S210 (object history saving process) to calculate the speed.

[0049] S212 is the result output processing of object recognition. For vehicle braking control devices and steering control devices, it outputs the position, type and speed of the detected, recognized and calculated objects to control the vehicle and, for example, realize automatic braking.

[0050] [Explanation of the flowchart]

[0051] Figure 3 This is a block diagram of the camera device of this embodiment. 200 is the camera device of this embodiment. The camera device 200 of this embodiment includes a camera device 200A and an image processing device 200B.

[0052] 200A is a camera device mounted on vehicle 100. Camera device 200A is a camera equipped with a camera sensor (CMOS: Complementary Metal Oxide Semiconductor, etc.) that converts light into electrical signals. The information converted into electrical signals by the camera sensor is further converted into image data representing the captured image within camera device 200A. Images captured by camera device 200A are sent to image processing device 200B at predetermined intervals. Camera device 200A captures images of the external environment (surroundings) of vehicle 100 and sends (outputs) the images to image processing device 200B.

[0053] 200B is an image processing apparatus that processes images received from the camera device 200A. The image processing apparatus 200B includes: an image acquisition unit 201, an object detection unit 202, an identification unit 203, a region setting unit 204, a tracking method selection unit 205, a nearby object tracking unit 206, an image tracking unit 207, an image deletion unit 208, an image storage unit 209, an object history storage unit 210, a speed calculation unit 211, and a result output unit 212.

[0054] 201 is the image acquisition unit, which acquires images captured by the camera device 200A.

[0055] 202 is an object detection unit that detects objects from the image acquired by the image acquisition unit 201.

[0056] 203 is the identification unit, which identifies the types of objects detected by the object detection unit 202.

[0057] 204 is the area setting unit, which is a processing unit that sets up collision judgment zones 101 and 704, and collision avoidance judgment zones 600, 601, 602, 603, 700, 701, 702, and 703 around the vehicle 100 to determine the collision risk between the vehicle 100 and objects (also referred to as...). Figures 8-11 (etc.). In the area setting unit 204, wheel speed, steering angle, actual steering angle, yaw angle sensor, inertial sensor, GPS position information, etc., are obtained from the vehicle 100 to determine how the vehicle is driving, its speed, and its turning radius. For example... Figure 8 As shown, when the vehicle is moving forward, there are configuration setting areas with 101, 600, 601 in front of the vehicle, and 602, 603 behind and to the side of the vehicle. Figure 10As shown, when the vehicle is reversing, areas 704, 700, and 701 are positioned behind the vehicle, while areas 702 and 703 are positioned in front of or even to the side of the vehicle. The distances 621 to 620, 612 to 611, 613 to 614, 612 to 610, 613 to 615, 721 to 720, 712 to 711, 713 to 714, 712 to 710, and 713 to 715 are lengthened in relation to the vehicle's speed. By lengthening these distances in relation to the speed, when the vehicle is traveling at low speeds, the relatively high speeds of pedestrians and bicycles compared to the vehicle increase the likelihood of collisions due to pedestrians or bicycles flying out, thus increasing the need for braking and reducing the risk of accidents. Furthermore, by similarly increasing the distance in relation to the speed, when the vehicle is traveling at high speed, the likelihood of the vehicle passing by without colliding is higher than that of pedestrians and bicycles that have already attempted to avoid a collision, changing direction, flying out, and colliding again. On the other hand, it can reduce the possibility of rear-end collisions caused by automatic braking at high speeds, and the possibility of the vehicle becoming uncontrollable, thus leading to self-damage accidents.

[0058] Furthermore, when the vehicle is moving forward and making a rightward turn relative to the direction of travel, the smaller the turning radius, the greater the ratio of the distance 613 to 614 to the distance 612 to 611 becomes, thus widening the collision determination area 101 in the direction of vehicle travel and further improving the accuracy of collision determination during the turn. Conversely, when the vehicle is moving forward and making a leftward turn relative to the direction of travel, the smaller the turning radius, the greater the ratio of the distance 613 to 614 to the distance 612 to 611 becomes, thus widening the collision determination area 101 in the direction of vehicle travel and further improving the accuracy of collision determination during the turn.

[0059] Furthermore, when the vehicle reverses while turning to the right in the direction of travel, the smaller the turning radius, the greater the ratio of the distance 713 to 714 to the distance 712 to 711 becomes, thus widening the collision determination area 704 in the direction of vehicle travel and further improving the accuracy of collision determination during turning. Conversely, when the vehicle reverses while turning to the left in the direction of travel, the smaller the turning radius, the greater the ratio of the distance 713 to 714 to the distance 712 to 711 becomes, thus widening the collision determination area 704 in the direction of vehicle travel and further improving the accuracy of collision determination during turning.

[0060] 205 is a tracking method selection unit that selects whether to perform nearby object tracking processing or image tracking processing. In other words, in the tracking method selection unit 205, for each detected and identified object, the tracking object to be tracked is selected from the following: the tracking object to be tracked by nearby object tracking processing, the tracking object to be tracked by image tracking processing, or any object other than the tracking object to be tracked by nearby object tracking processing and the tracking object to be tracked by image tracking processing.

[0061] 206 is a nearby object tracking unit that compares images of different frames before and after a certain time, identifies nearby objects of the same type as the same object, and tracks them. In other words, in the nearby object tracking unit 206, based on the types of objects captured in multiple images taken in a time sequence, it tracks one or more nearby object tracking objects included in the objects detected by the object detection unit 202.

[0062] Image tracking unit 207 compares images from different frames before and after a given time period, identifies objects with consistent image features (quantities) as the same object, and tracks them. In other words, image tracking unit 207 tracks objects based on the feature quantities of objects contained in multiple images captured in a time sequence. Image tracking unit 207 refers to images of one or more target objects detected by object detection unit 202 within multiple images stored in image storage unit 209. It explores the feature quantities of objects from past time points based on images captured at the current time point (tracing back to the past), tracks the target objects of image tracking processing, and outputs the tracking result.

[0063] 209 is the image storage unit, which divides images captured at past time points into multiple regions and stores them in memory M209. Image segmentation methods include, for example, cropping out and storing the Region of Interest (ROI) of the object identified by the recognition unit 203. During storage, image processing, such as recognition processing, is performed on the working memory, which is in a circular buffer configuration. Therefore, while image cropping, resizing, and normalization during recognition processing require the use of the working memory to store the image, if the working memory is originally a circular buffer, it is unnecessary to save it again.

[0064] 208 is an image deletion unit that deletes the image area (part or all) of the image area of ​​the object being tracked by the nearby object tracking unit 206 that was captured and stored in the image area of ​​the memory M209, and the image area that has become a state where all tracking has been completed (part or all of the image area up to the moment when the tracking by the nearby object tracking unit 206 or the image tracking unit 207 is completed).

[0065] 210 is the object history storage unit, which stores the object's history in memory M210.

[0066] 211 is the velocity calculation unit, which uses the object's history information to calculate the object's velocity.

[0067] 212 is the result output unit, which outputs the position, type and speed of the object detected, identified and calculated to the vehicle 100, and controls the vehicle through the braking control device and steering control device provided by the vehicle 100, and realizes automatic braking, for example.

[0068] [Illustrate the features of the embodiments through examples]

[0069] Set the tracking method for all types except specific types to nearby object tracking.

[0070] Figure 4 This is a top view illustrating the detection positions of vehicles and objects in an example of the operation of the camera device in this embodiment, and a diagram showing the detection positions of objects filtered to be of types other than a specific type. In this embodiment, tracking processing is selected based on the type of object. As described above, there are two types of tracking processing: nearby object tracking and image tracking. Nearby object tracking has a low processing load, but there is a risk of mistracking if similar objects are present nearby. Image tracking has a high processing load, but it can identify objects with similar features in the image, thus reducing the risk of mistracking. Vehicles and motorcycles traveling in other lanes (in the same or opposite direction) are less likely to fly out, and their movement is not reflected in the image coordinates. Furthermore, sudden changes in direction of movement are rare. Therefore, the presence of numerous objects nearby is rare, and the risk of mistracking due to nearby object tracking is also low. Nearby object tracking is used for objects of types with a low risk of mistracking (objects 105, 106, 107, 108). By selecting these nearby objects for tracking, images (part or all) of those objects are also deleted. This reduces the processing load and decreases the size of the stored images.

[0071] <Determining the same object being tracked in the vicinity by setting the prediction range of movement>

[0072] Figure 5This is a top view illustrating the detection positions of vehicles and objects in an example of the operation of the camera device in this embodiment, and is a view selected to be identified as the same object by the nearby object tracking method. In this embodiment, the tracking method for objects of the type that fly out and move significantly on the image coordinates, namely pedestrians, cyclists, vehicles crossing, and motorcycles crossing (objects 109, 110), determines the maximum range (hereinafter also referred to as the tracking range) that moves at a predetermined speed for each type. If only one object of the same type exists in the tracking range between an image captured at a certain moment and an image captured at a moment earlier than that moment, the object is identified as the same object and tracked. 300 to 306 are the maximum range (tracking range) of object 110, and 308 to 314 are the maximum range (tracking range) of object 109. The maximum range (tracking range) of the movement amount of objects 109 and 110 is determined as the range (assuming prediction circle) of 315 and 307, respectively.

[0073] Figure 6 This is a top view illustrating the detection positions of vehicles and objects in an example of the operation of the camera device in this embodiment, and is a diagram that filters out objects of the same type that can be identified as a specific category using the nearby object tracking method. From... Figure 5 In examples that can further improve the accuracy of nearby object tracking, the number of objects within that maximum range of motion is reduced and the number of objects that can be tracked nearby is increased by placing the maximum range of motion of the object relative to the orientation determined by the object's identifier.

[0074] Figure 7 This is a top view illustrating the detected positions of the vehicle and object, representing the state of an example of the operation of the camera device in this embodiment. It is also a diagram illustrating the maximum movement range of a specific type of object in the nearby object tracking method. For the position 420 of an object at a certain moment in 109, an assumed movement speed of 503 is assumed, and a maximum speed error range 410 is set for it for nearby object tracking. 501 represents the image projected by the camera 500, which is the imaging device. 420 is identified as an object moving in the left-right direction as observed from the camera 500. Therefore, 503 estimates the assumed movement speed of the object moving in a direction perpendicular to the line 502 connecting 500 and 420. Thus, as... Figure 6 As shown, relative to the position 420 of the object at a certain moment of 109, the range of the amount of movement of 410 (assuming a predicted ellipse) is determined as the tracking range, and thereafter the tracking ranges of 411 to 416 can also be determined.

[0075] <Detected within the collision avoidance detection zone, the same object is determined by tracking its vicinity based on the direction of intrusion (in the case of forward movement).>

[0076] Figure 8 This is a top view illustrating the detection positions of the vehicle and objects in an example of the operation of the camera device in this embodiment, and a diagram illustrating the collision determination area and collision avoidance determination area during forward movement. 600, 601, 602, and 603 represent collision avoidance determination areas set around the vehicle 100. Objects entering these collision avoidance determination areas are determined to have a possibility of collision avoidance, and are tracked using object proximity tracking (in other words, objects selected for object proximity tracking processing), and excluded from the objects tracked in the image. Because the number of objects tracked in the image is reduced, the processing load and image storage capacity are decreased.

[0077] If we explain the distance settings of 610-615, 620, and 621, the distances are set at a greater distance based on the vehicle speed and steering angle of vehicle 100. When vehicle 100 is traveling at a high speed, braking should be applied if pedestrians or other objects are nearby, compared to a slower speed, thus reducing the accident rate. Furthermore, with a steering angle of vehicle 100, braking is applied over a wider range compared to a non-collision situation, further reducing the accident rate. Moreover, at higher speeds and with a steering angle, even objects nearby that are not likely to collide are controlled, reducing discomfort for the driver and passengers.

[0078] Figure 9 This is a top view illustrating the detection positions of the vehicle and object in an example of the operation of the camera device in this embodiment, and a diagram illustrating the intrusion direction of the collision avoidance determination area during forward movement.

[0079] Simply entering these collision avoidance detection zones and being detected will not be considered an object avoiding collisions. For example... Figure 9 As shown, objects in the following situations are considered to have avoided collision: when entering the collision avoidance determination area 600 by crossing below (vehicle side), when entering the collision avoidance determination area 601 by crossing above (vehicle side), when entering the collision avoidance determination area 602 by entering from the right and below, and when entering the collision avoidance determination area 603 by entering from the right and above.

[0080] <Detected within the collision avoidance detection area, the same object is determined by tracking its vicinity based on the direction of intrusion (in the case of retreating).>

[0081] Figure 10This is a top view illustrating the detection positions of the vehicle and object in an example of the operation of the camera device in this embodiment, and a diagram illustrating the collision determination area and collision avoidance determination area during reversal. 700, 701, 702, and 703 are collision avoidance determination areas set around the vehicle 100. 704 is a collision determination area set around the vehicle 100. Objects entering this collision avoidance determination area are determined to have a possibility of collision avoidance, and are tracked using object proximity tracking (in other words, objects selected for object proximity tracking processing) and excluded from the image tracking objects. Because the number of image tracking objects is reduced, the processing load and image storage capacity are reduced.

[0082] If we were to explain the distance settings for 710–715, 720, and 721, the distance would be set at a distance based on the vehicle speed of 100 and the size of the steering angle. This is for the same reason as for 610–615, 620, and 621.

[0083] Figure 11 This is a top view illustrating the detection positions of the vehicle and object in an example of the operation of the camera device in this embodiment, and a diagram illustrating the intrusion direction of the collision avoidance determination area during reversal.

[0084] Objects that merely enter these collision avoidance decision zones but are not detected as needing to avoid collisions. For example... Figure 11 As shown, objects in the following situations are considered to have avoided collision: when entering the collision avoidance determination area 700 from below (vehicle side), when entering the collision avoidance determination area 701 from above (vehicle side), when entering the collision avoidance determination area 702 from the left and below, and when entering the collision avoidance determination area 703 from the left and above.

[0085] <Except for objects that have entered the collision avoidance detection area>

[0086] Figure 12This is a top view illustrating the detection positions of the vehicle and objects in an example of the operation of the camera device in this embodiment, and a diagram illustrating the filtering of objects that have entered the collision avoidance determination area during forward movement by tracking nearby objects. 805, 804, 803, 802, and 801, in chronological order, represent the detection positions (object detection results) of object 104 in each frame. Since 801 intrudes into the collision avoidance determination area 601, and 811 is the maximum range (tracking range) of 801, which includes 802 from the previous frame, 801 and 802 are determined to be the same object. Furthermore, since the line segment connecting points 801 and 802 intersects the top edge of the collision avoidance determination area 601, it is determined that the intrusion into the collision avoidance determination area 601 occurred from the top edge. Similarly, the maximum range (tracking range) 812 of 802 only includes 803 in the previous frame, and is therefore determined to be the same object. Within the maximum range (tracking range) 813 of 803, only 804 was included in the previous frame, and they are determined to be the same object. Within the maximum range (tracking range) 814 of 804, both objects 805 and 825 (object detection results of object 102) were included in the previous frame, and they cannot be determined to be the same object. Since 801, 802, 803, and 804 are tracked as objects that have avoided collisions, in the tracking processing of these, tracking is performed by object proximity tracking (in other words, the tracking object is selected as the object of object proximity tracking processing). In addition to being the object of image tracking, the number of image tracking operations can be reduced, and the image regions (partial regions or the entire region) that only reflect these objects can be deleted, and the file size can also be reduced.

[0087] [Summary of the embodiments and description of variations]

[0088] In this embodiment, as described above, the camera device 200 includes: an object detection unit 202 that detects objects; an image tracking unit 207 that verifies and tracks the same object using images; a speed calculation unit 211 that calculates the speed of the object; an object history storage unit 210 that stores past object history information; and an image storage unit 209 that stores past images and uses those past images to track the same object. The camera device 200 is characterized by the above features. Therefore, even if tracking all detected objects is difficult in terms of processing time, by reprocessing them in later frames, processing can be performed in frames with ample processing time, making it possible to track more objects.

[0089] Furthermore, by using past information about the same object to perform velocity calculations, accuracy can be improved if past history is used. Additionally, by removing information about the furthest distance from the linear approximation line from the past information about the same object during velocity calculation, outlier removal can be performed during velocity calculation, thereby reducing velocity calculation errors even in the event of sudden ranging errors. When performing outlier removal, only the furthest information is removed; therefore, the feature is the removal of the furthest information. Furthermore, by using past images to track the same object for objects detected within a predetermined range where collision probability is high, the tracking processing load is reduced by targeting objects with low collision risk. A nearby object tracking unit 206 is provided to determine that objects in adjacent frames with similar positions and image coordinates are the same object, distinguishing between objects tracked by the nearby object tracking unit 206 and objects tracked by the image tracking unit 207. Since pedestrians who can be determined to exist independently through nearby objects do not pose a risk of moving elsewhere, there is no need for verification through image tracking, which has a high processing load, thus reducing the processing load. Delete images from past images where object tracking was fully completed at that particular moment. By deleting information that has already undergone tracking, the image storage size can be reduced.

[0090] Furthermore, in a configuration with multiple cameras, even images from the same moment are processed by deleting images in which the tracking of the detected objects has been completed. This allows for the subdivision of the objects to be deleted for each camera, thereby further reducing the storage capacity even if there are images incompletely tracked at the same moment.

[0091] By dividing the camera image into multiple regions and deleting the image of the region where the tracking processing of the detected object has been completed, the region can be divided, thereby deleting images of angles with a high probability of the existence of that type of object, thus further reducing the storage capacity.

[0092] The device includes a motion range determination unit that pre-determines the measured speed of an object and determines the range of assumed movement at that speed based on the object's position at a certain moment. By tracking the same object in the previous or subsequent frames within the motion range as if it were a single object, and assuming that there is no movement exceeding the object's possible speed, the device can be identified as the same object. This reduces the number of objects required for image tracking, lightens the processing load, and also reduces the image storage capacity.

[0093] Furthermore, it includes an identification unit 203 that identifies the type of object and tracks it using only nearby object tracking as a specific type of tracking method. By tracking nearby objects according to the type, the processing load and storage capacity can be further reduced.

[0094] By deleting images that only detect objects tracked through nearby object tracking, not only is the tracking processing load reduced, but the capacity is also eliminated. For a pre-determined collision avoidance decision zone for a vehicle, nearby object tracking is performed by tracing back from objects that have entered the collision avoidance decision zone. Objects identified as the same object are excluded from the image tracking objects. This reduces the processing load by performing nearby object tracking on objects that can be determined to have avoided a collision and excluding them from image tracking.

[0095] When determining the possible range of movement, by assuming a speed corresponding to the type, and thus setting the speed according to the type, it is possible to track nearby objects more reliably and more frequently, thereby reducing the processing load.

[0096] The identification unit 203 identifies objects in each direction. Based on the object's direction, it assumes different speeds for the possible movement range from the camera's viewpoint in the right, left, approach, and departure directions, thereby also identifying the direction. This allows for more reliable and extensive tracking of nearby objects, thus reducing the processing load. By storing at least the object's location, type, tracking method, and tracking implementation status as its history, the information stored as the object's history allows for speed estimation if the location is known. By recording the tracking method and the tracking implementation status, unnecessary tracking is avoided by repeating the process multiple times. By performing at least these two steps, the storage capacity of the history information can be further reduced.

[0097] [Summarize]

[0098] As explained above, the image processing apparatus 200B of this embodiment includes: an image acquisition unit 201 that acquires images captured by a camera device 200A mounted on the vehicle 100; an image storage unit 209 that stores the images captured at past time points; an object detection unit 202 that detects objects (around the vehicle) based on the images; and a first tracking unit (image tracking unit 207) that tracks objects based on the feature values ​​of the objects contained in a plurality of images captured in a time sequence. The first tracking unit (image tracking unit 207) refers to images of one or more first tracking target objects included in the objects detected by the object detection unit 202 from the plurality of images stored in the image storage unit 209, explores the feature values ​​of the objects by comparing images captured at a first time point (current time point) with images captured at a second time point (past time point) earlier than the first time point, thereby tracking the first tracking target objects and outputting the tracking result. That is, image tracking is performed by tracing back to the past.

[0099] Furthermore, the image processing apparatus 200B of this embodiment includes: a recognition unit 203 that identifies the type of object based on the image; a second tracking unit (nearby object tracking unit 206) that tracks one or more second tracking object objects included in the objects detected by the object detection unit 202 based on the types of objects captured in a plurality of images captured in a time sequence; and an image deletion unit 208 that deletes part or all of the images captured by the second tracking unit (nearby object tracking unit 206) of the second tracking object from the image storage unit 209. That is, tracking is performed based on the type of object using nearby object tracking, and the images are deleted.

[0100] Furthermore, the second tracking unit (nearby object tracking unit 206) sets a tracking range based on the type of the object. In the image captured at the first moment and the image captured at a second moment earlier than the first moment, if only one object of the same type exists within the set tracking range, that object is treated as the same object and tracked accordingly. That is, it uses a hypothetical prediction circle or similar method to determine that the objects are the same through nearby object tracking.

[0101] Furthermore, the image processing apparatus 200B of this embodiment includes a selection unit (tracking method selection unit 205), which selects, for each of the objects, the first tracking target object, the second tracking target object, or any object other than the first tracking target object and the second tracking target object.

[0102] Furthermore, the image processing apparatus 200B of this embodiment includes a region setting unit 204, which sets a collision avoidance determination region around the vehicle to determine the collision risk between the vehicle and an object, and the selection unit (tracking method selection unit 205) selects an object existing in the collision avoidance determination region as the second tracking object.

[0103] Furthermore, the selection unit (tracking method selection unit 205) selects, for each object, the first tracking object, the second tracking object, or an object other than the tracking object, based on the position and direction of travel of the vehicle and the position and direction of travel of the object.

[0104] Furthermore, the image processing apparatus 200B of this embodiment includes an image deletion unit 208, which deletes a portion or the entire area of ​​an image stored in the image storage unit 209. The image deletion unit 208 deletes a portion or the entire area of ​​the image up to the moment when the tracking by the first tracking unit (image tracking unit 207) or the second tracking unit (nearby object tracking unit 206) is completed.

[0105] Furthermore, the image processing apparatus 200B of this embodiment includes an object history storage unit 210, which stores object history information including at least the object's location, type, and tracking execution status. The first tracking unit (image tracking unit 207) tracks the first tracking object based on the object history information stored in the object history storage unit 210.

[0106] That is, in this embodiment, the image processing apparatus 200B performs image tracking in the direction of tracing time for objects entering a predetermined area near the camera device 200 and the vehicle 100 it is mounted on. Except for objects that can be reliably tracked by tracking nearby objects, the image tracking is performed with the minimum necessary amount of image tracking, thereby suppressing the image capacity used for tracing back to the past.

[0107] According to this embodiment, for objects detected nearby that may become control targets, image tracking is performed by tracing back to the past. At this time, the storage capacity of past images that pose a problem increases. Therefore, data required for image tracking is filtered in the aforementioned order, and unnecessary data is deleted to suppress this process. This reduces processing load and image storage capacity, and reliably enables the tracking of objects related to vehicle control.

[0108] Furthermore, the present invention is not limited to the embodiments described above, but includes various modifications. For example, the embodiments described above are provided for ease of understanding of the present invention and are not necessarily limited to having all the described configurations.

[0109] Furthermore, the aforementioned components, functions, processing units, and processing methods can be partially or entirely implemented in hardware, for example, by designing integrated circuits. Alternatively, the aforementioned components and functions can be implemented in software by having a processor interpret and execute programs that perform each function. The programs, tables, files, and other information implementing these functions can be stored in storage devices such as memory, hard disks, and SSDs (Solid State Drives), or on recording media such as IC cards, SD cards, and DVDs.

[0110] Furthermore, control lines and information lines represent what the specifications deem necessary, but may not necessarily represent all control lines and information lines on the product. In practice, it can be assumed that almost all the components are interconnected.

[0111] Symbol Explanation

[0112] 100… vehicles (this vehicle)

[0113] 101… Collision Detection Area

[0114] 200… camera device

[0115] 200A…Camera device

[0116] 200B…Image Processing Device

[0117] 201…Image Acquisition Department

[0118] 202…Object Detection Department

[0119] 203…Identification Department

[0120] 204…Regional Setting Department

[0121] 205… Tracking Methods Selection Section (Selection Section)

[0122] 206…Nearby Object Tracking Unit (Second Tracking Unit)

[0123] 207…Image Tracking Department (First Tracking Department)

[0124] 208…Image Deletion Section

[0125] 209…Image Storage Department

[0126] 210…Object History Preservation Department

[0127] 211… Speed ​​Calculation Department

[0128] 212…Result Output Section

[0129] Maximum tracking range (tracking range) for nearby objects: 300-306, 308-314...

[0130] 410~416… considers the maximum range of the orientation of nearby objects (tracking range).

[0131] Collision avoidance zone 600~603… when moving forward

[0132] Collision avoidance judgment area when retreating from 700 to 703...

Claims

1. An image processing apparatus, characterized in that, have: The image acquisition unit acquires images captured by the camera device mounted on the vehicle; An image storage unit that stores images captured at past points in time; An object detection unit detects objects based on the image. as well as A first tracking unit tracks the object based on feature quantities of the object contained in a plurality of images captured in a time-series manner. The first tracking unit takes an image of one or more first tracking objects contained in an object detected by the object detection unit from among the multiple images stored in the image storage unit. It explores the feature quantity of the object by comparing the image taken at the first time point with the image taken at a second time point earlier than the first time point, tracks the first tracking object, and outputs the tracking result.

2. The image processing apparatus according to claim 1, characterized in that, It includes: a recognition unit that identifies the type of object based on the image; The second tracking unit tracks one or more second tracking object objects included in the objects detected by the object detection unit, based on the types of objects captured in a plurality of images taken in a time sequence. as well as The image deletion unit deletes part or all of the image captured by the image storage unit of the second tracking object being tracked by the second tracking unit.

3. The image processing apparatus according to claim 2, characterized in that, The second tracking unit sets a tracking range based on the type of the object. In the image captured at the first moment and the image captured at a second moment earlier than the first moment, if there is only one object of the same type in the set tracking range, the object is tracked as the same object.

4. The image processing apparatus according to claim 2, characterized in that, The device includes a selection unit that selects, for each of the objects, the first tracked object, the second tracked object, or any object other than the first tracked object and the second tracked object.

5. The image processing apparatus according to claim 4, characterized in that, The vehicle is equipped with a zone setting unit that sets a collision avoidance determination zone around the vehicle to determine the collision risk between the vehicle and an object. The selection unit selects an object existing in the collision avoidance determination area as the second tracking object.

6. The image processing apparatus according to claim 4, characterized in that, The selection unit selects, for each object, the first tracking object, the second tracking object, or an object other than the tracking object, based on the position and direction of travel of the vehicle and the position and direction of travel of the object.

7. The image processing apparatus according to claim 2, characterized in that, It includes an image deletion unit that deletes a portion or the entire area of ​​an image stored in the image storage unit. The image deletion unit deletes a portion or the entire area of ​​the image up to the moment when the first tracking unit or the second tracking unit completes its tracking.

8. The image processing apparatus according to claim 1, characterized in that, The system includes an object history storage unit, which stores object history information including at least the object's location, type, and tracking execution status. The first tracking unit tracks the first tracking object based on the object history information stored in the object history storage unit.

Images