Image processing device, vehicle control system, and image processing method

Abstract

This image processing device comprises: an object detection unit that detects, as a rectangular region, a region including a three-dimensional object from an image captured by an imaging device mounted on a vehicle; a tracking unit that generates, as a detection result, from a plurality of images captured in time series by the imaging device, time-series information of a plurality of rectangular regions in which the same three-dimensional object is detected, and time-series information of the angle of view of the imaging device when the rectangular region is detected; and a three-dimensional object region estimation unit that estimates the size of the three-dimensional object related to the rectangular region on the basis of the time-series change in the detection result.

Description

Image processing device, vehicle control system, and image processing method

[0001] The present invention relates to an image processing device, a vehicle control system, and an image processing method.

[0002] In recent years, driving assistance systems have been developed that detect objects using images captured by a camera mounted on a vehicle and assist the driver according to the surrounding conditions. For example, an AEB (Autonomous Emergency Braking) function in a driving assistance system determines a collision based on an overlap between the paths of the vehicle and another vehicle, and activates the brakes of the vehicle if there is a possibility of a collision. In order to correctly determine a collision with another vehicle, it is necessary to accurately determine the size and direction of the other vehicle. An object detection device used in such a driving assistance system is described, for example, in Patent Document 1.

[0003] Patent Document 1 describes an object detection device that includes "an acquisition unit that acquires an image captured by an imaging unit that can capture images of the surrounding environment including the road surface on which the vehicle is located; a setting unit that sets a three-dimensional object boundary line that indicates the boundary between the road surface and an object area recognized as an area where a three-dimensional object exists on the road surface included in the captured image, and a rectangular bounding box for selecting a predetermined target object from the three-dimensional objects included in the object area; and a control unit that extracts a partial boundary line included in the bounding box from the three-dimensional object boundary line, and estimates and outputs a focus point to focus on on the partial boundary line to avoid contact between the vehicle and the target object."

[0004] Japanese Patent Application Laid-Open No. 2021-56717

[0005] As described above, a technology has been disclosed for avoiding contact with another vehicle based on the bounding box of a detected other vehicle. However, the bounding box is the smallest rectangular frame that surrounds the other vehicle in an image, and it is not possible to distinguish between the rear and side of the other vehicle or estimate the direction of the other vehicle. For this reason, when calculating the size and direction of the other vehicle based on the bounding box, it is assumed that only the rear of the other vehicle is captured within the bounding box and that the direction of the other vehicle is the same as the direction of the own vehicle. However, if the direction of the other vehicle is not the same as the direction of the own vehicle, it is not possible to accurately estimate the area of ​​the other vehicle based on the bounding box extracted based on the above assumption, resulting in an erroneous collision determination.

[0006] The present invention has been made to solve the above-mentioned problems, and an object of the present invention is to accurately estimate the size and orientation of other vehicles using a bounding box, thereby suppressing false alarms and malfunctions in driving assistance systems.

[0007] The image processing device of the present invention includes an object detection unit that detects an area including a three-dimensional object as a rectangular area from an image captured by an imaging device mounted on a vehicle, a tracking unit that generates, as detection results, time-series information of multiple rectangular areas in which the same three-dimensional object is detected from multiple images captured in time series by the imaging device and time-series information of the angle of view of the imaging device at the time of detecting the rectangular area, and a three-dimensional object area estimation unit that estimates the size of the three-dimensional object related to the rectangular area based on time-series changes in the detection results. Note that the above image forming device is one aspect of the present invention, and a vehicle control system and an image processing method that reflect one aspect of the present invention are also configured in the same way as the above image processing device.

[0008] According to the present invention having the above configuration, it is possible to accurately estimate the size and direction of other vehicles using a bounding box, thereby suppressing false alarms and malfunctions in driving assistance systems. Other issues, configurations, and effects will become clear from the description of the following embodiments.

[0009] 1 is a diagram showing an example of a bounding box when another vehicle is located far away. FIG. 2 is a diagram showing an example of a bounding box when another vehicle is approaching. FIG. 3 is a block diagram showing an example of the functional configuration of a vehicle control system 1 according to an embodiment of the present invention. FIG. 4 is a diagram showing the horizontal size of a bounding box in which another vehicle has been detected. FIG. 5 is a diagram for explaining the shape of a rectangular area that can exist based on the horizontal size of the rectangular area at the current time only. FIG. 6 is a diagram showing time-series changes in the horizontal size of rectangular areas of different shapes. FIG. 7 is a diagram for explaining a method for estimating the size of another vehicle in an image processing device according to an embodiment of the present invention. FIG. 8 is a diagram for explaining a method for calculating the size and orientation of another vehicle in an image processing device according to an embodiment of the present invention. FIG. 9 is a block diagram showing an example of the hardware configuration of an image processing device according to an embodiment of the present invention. FIG. 10 is a flowchart showing the procedure of a solid object area estimation process in an image processing device according to an embodiment of the present invention. FIG. 11 is a diagram for explaining an application of an image processing device according to an embodiment of the present invention to verification of a 3D bounding box estimation result.

[0010] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. In this specification and drawings, components having substantially the same functions or configurations are designated by the same reference numerals, and redundant description will be omitted. The present invention is applicable to, for example, a computing device for vehicle control capable of communicating with an on-board ECU (Electronic Control Unit) for an Advanced Driver Assistance System (ADAS) or Autonomous Driving (AD).

[0011] First, a problem with collision detection based on a conventional 2D bounding box will be described. Figures 1A and 1B are diagrams for explaining the problem with collision detection based on a conventional 2D bounding box. Hereinafter, a 2D bounding box will be abbreviated as "bounding box." Figure 1A shows a bounding box when another vehicle is located far away. Figure 1B shows a bounding box when the other vehicle is approaching.

[0012] The bounding box represents a rectangular area including the other vehicle V2, detected in a coordinate system with a bird's-eye view of the host vehicle V1. FIG. 1A shows a bounding box Bx1 that detects the other vehicle V2 located far in front of the host vehicle V1. In the driving scene shown in FIG. 1A, a small proportion of the side of the other vehicle V2 is captured in the captured image. In this case, even if the size of the bounding box Bx1 is assumed to correspond to the size of the other vehicle V2, there is no significant deviation, and therefore a collision determination based on the bounding box Bx1 will not result in an erroneous determination.

[0013] However, as the host vehicle V1 approaches the other vehicle V2, the proportion of the side of the other vehicle V2 captured in the captured image increases, as shown in Figure 1B. In this case, if it is assumed that the size of the bounding box Bx2 detecting the other vehicle V2 corresponds to the size of the other vehicle V2, the estimated lateral size of the other vehicle V2 will be significantly different from the actual lateral size, resulting in an erroneous collision determination based on the bounding box Bx2. If an erroneous collision determination is made, the host vehicle V1 may perform an operation such as changing lanes to avoid the other vehicle V2, even though the other vehicle V2 is parked in a safe shoulder strip.

[0014] The present invention has been made to solve the above-mentioned problems, and a vehicle control system 1 according to one embodiment of the present invention will be described below.

[0015] [Example of Functional Configuration of Vehicle Control System] Fig. 2 is a block diagram showing an example of the functional configuration of the vehicle control system 1 according to this embodiment. The vehicle control system 1 includes an imaging device 10, an image processing device 20, and a vehicle control unit 30. The imaging device 10, the image processing device 20, and the vehicle control unit 30 are connected in this order. Note that Fig. 2 shows an example in which the vehicle control unit 30 is arranged as an independent device outside the image processing device 20, but the present invention is not limited to this. The vehicle control unit 30 may also be included inside the image processing device 20.

[0016] The imaging device 10 is a device with an imaging function, such as a camera mounted on a vehicle. When the imaging device 10 is installed, for example, behind the windshield or rearview mirror of the vehicle, the imaging range (detection range) is in the front direction of the vehicle. The imaging device 10 can select (set) an appropriate angle of view, resolution, etc. depending on the type and size of the object to be detected and the expected driving scene. Note that the installation location of the imaging device 10 is not limited to behind the windshield or rearview mirror of the vehicle. The installation location of the imaging device 10 can be changed or the number of imaging devices 10 can be increased depending on the range to be detected. The imaging device 10 outputs images of three-dimensional objects captured in time series to the image processing device 20.

[0017] 2, the image processing device 20 is an example of an ECU (Electronic Control Unit), and includes an object detection unit 21, a tracking unit 22, a detection result storage unit 23, and a three-dimensional object area calculation unit 24. The object detection unit 21, the tracking unit 22, the detection result storage unit 23, and the three-dimensional object area calculation unit 24 are connected in this order.

[0018] The object detection unit (object detection unit 21) detects an area containing a three-dimensional object as a rectangular area from an image captured by an imaging device (imaging device 10) mounted on a vehicle. This rectangular area is, in other words, an area surrounded by a bounding box. The bounding box is estimated, for example, using a neural network model. An example of a neural network model is YOLO. The object detection unit 21 outputs time-series information of the detected rectangular area, i.e., position information of the bounding box in the time-series information, to the tracking unit 22 as a detection result. Here, the position information of the bounding box is, for example, information on the width and height of the bounding box in a coordinate system on the captured image with the upper left vertex of the rectangular area as the origin. Note that the position information of the bounding box is not limited to information on the width and height of the bounding box in the above-mentioned coordinate system, and any format that can represent the position of the bounding box on the image may be used. Furthermore, the bounding box may be estimated using any detection method other than a neural network model.

[0019] The tracking unit (tracking unit 22) also generates, as detection results, time-series information on multiple rectangular areas in which the same three-dimensional object is detected, and time-series information on the angle of view of the imaging device (imaging device 10) at the time of detecting the rectangular area, from multiple images captured in time series by the imaging device (imaging device 10). Below, calculation of the rectangular area in which the other vehicle, a three-dimensional object, is detected, i.e., the bounding box, will be described. Here, in the coordinate system on the captured image, the depth distance from the subject vehicle to the other vehicle is defined as Z, the lateral distance from the subject vehicle to the other vehicle is defined as X, and the coordinates on the image of the upper left vertex of the bounding box are defined as (i, j). Assuming the road surface is flat, Z and X can be calculated using the following equations (1) and (2), respectively.

[0020]

[0021]

[0022] In the above formulas (1) and (2), f is the focal length of the camera, H is the height of the camera from the ground, c is the pixel pitch, and (c ｘ , c ｙ ) indicates the camera center.

[0023] Furthermore, using the distance Z calculated by the above formula (1), the bounding box in which the other vehicle is detected from the captured image, i.e., the horizontal size W' of the rectangular area containing the other vehicle (see Figure 3A described below), can be calculated using the following formula (3).

[0024]

[0025] In the above formula (3), W (see FIG. 3A described later) is the size of the three-dimensional object related to the rectangular area, i.e., the size of the other vehicle on the image. Note that if only the rear surface of the other vehicle is included in the bounding box, W' will have the same value as W. On the other hand, if both the rear surface and the side surface of the other vehicle are included in the bounding box, W' will have a different value from W.

[0026] The tracking unit 22 stores, as detection results, in the detection result storage unit 23, time-series information about the calculated rectangular area (bounding box) (such as the horizontal size W' and the position information of the bounding box) and time-series information about the angle of view of the imaging device when the rectangular area was detected. The detection results may also be stored in the detection result storage unit 23, including time-series information such as the position of the other vehicle in three-dimensional space and an image area containing an image of the other vehicle cut out from the image as a template. If the behavior of the subject vehicle can be acquired from a CAN (Controller Area Network), the behavior of the subject vehicle may also be included in the detection results and stored. If the type of the detected other vehicle (passenger car, truck, etc.) can be obtained, the type of the other vehicle may also be included in the detection results and stored. If the speed of the other vehicle can be estimated based on the detection results stored in time series, the estimated speed of the other vehicle may also be included in the detection results and stored. When storing the detection results, the tracking unit 22 stores detection results in which the same three-dimensional object is detected from multiple images captured in time series by the imaging device 10, linking the detection results to the three-dimensional object.

[0027] In determining the three-dimensional object, the tracking unit 22 uses, for example, the position of the three-dimensional object in three-dimensional space. If a new three-dimensional object is detected near a previously detected position, the tracking unit 22 determines that the three-dimensional object at the previously detected position and the new three-dimensional object are the same object. Furthermore, if the behavior of the host vehicle can be acquired from the CAN, the tracking unit 22 may calculate the predicted position of another vehicle taking into account the movement of the host vehicle, and if a new three-dimensional object is detected near the predicted position, determine that the two are the same object. Furthermore, if the position and template of the three-dimensional object on the captured image are stored, the tracking unit 22 may determine that the two are the same object by tracking the other vehicle on the captured image.

[0028] As shown in FIG. 2, the three-dimensional object region calculation unit 24 includes a three-dimensional object region estimation unit 241, a time-series information selection unit 242, and an estimation parameter selection unit 243.

[0029] The three-dimensional object area estimation unit (three-dimensional object area estimation unit 241) estimates the size of the three-dimensional object associated with the rectangular area based on time-series changes in the detection results. The three-dimensional object area estimation unit (three-dimensional object area estimation unit 241) also estimates the orientation of the three-dimensional object. Specifically, the three-dimensional object area estimation unit 241 estimates parameters such as the size and orientation of the three-dimensional object associated with the rectangular area based on the horizontal size W' of the rectangular area included in the detection results. The size of the three-dimensional object associated with the rectangular area includes the width and length of the three-dimensional object associated with the rectangular area. In the following description, the width and length of the three-dimensional object associated with the rectangular area are abbreviated as the "width" and "length" of the three-dimensional object (other vehicle), respectively. A detailed description will be provided below using Figures 3A, 3B, 3C, 4, and 5.

[0030] 3A, 3B, and 3C are diagrams illustrating changes in the width W' of a rectangular area associated with time-series changes in the depth distance from the host vehicle V1 to the other vehicle V2 observed by the host vehicle V1. Here, the time-series change is the amount of change between the observed value at each time and the observed value at the immediately preceding time. FIG. 3A shows the width W' of the bounding box Bx in which the host vehicle V1 detects the other vehicle V2. FIG. 3B is a diagram illustrating possible shapes of rectangular areas based on the width W' of the rectangular area at only the current time. FIG. 3C is a diagram illustrating time-series changes in the width W' of rectangular areas of different shapes.

[0031] 3A shows an example in which the other vehicle is located to the right of the subject vehicle and is facing the same direction as the subject vehicle. The size of the other vehicle V2 within the rectangular area (bounding box Bx) is width W and length L, as shown in the figure. The image of the other vehicle V2 enclosed by the bounding box Bx includes the rear and side surfaces of the other vehicle V2. Therefore, the horizontal dimension W' of the bounding box Bx is larger than the width W of the other vehicle V2.

[0032] Based on the width W' shown in Figure 3A, there are two possible shapes of the rectangular area: shape Q1 and shape Q2 shown in Figure 3B. Therefore, the width W and length L of the other vehicle V2 cannot be uniquely determined.

[0033] As the depth distance from the host vehicle V1 to the other vehicle V2 changes over time, the width W' of each of the shapes Q1 and Q2 shown in FIG. 3B also changes. The time-varying characteristics of the width W' of each shape with respect to the depth distance from the host vehicle V1 to the other vehicle V2 are shown in FIG. 3C. As shown in FIG. 3C, the width W' of each of the shapes Q1 and Q2 is the same at a depth distance of 20 m. As the depth distance decreases, the width W' of both shapes becomes different. Furthermore, the closer the host vehicle V1 is to the other vehicle V2, the larger the width W' of both shapes becomes. The width W and length L of the other vehicle V2 estimated when the difference in the change in the width W' of each of the rectangular regions of the multiple shapes is most consistent are optimal.

[0034] FIG. 4 is a diagram illustrating a method for estimating the size of another vehicle in the image processing device 20 according to this embodiment. In this embodiment, when the estimation parameter selection unit (estimation parameter selection unit 243) determines that the orientation of the three-dimensional object (another vehicle) is the same as the orientation of the vehicle (host vehicle), it selects the width and length of the three-dimensional object associated with the rectangular region. The three-dimensional object region estimation unit (three-dimensional object region estimation unit 241) estimates the width and length of the three-dimensional object associated with the rectangular region selected by the estimation parameter selection unit (estimation parameter selection unit 243). FIG. 4 illustrates an example in which another vehicle V2 is located to the right of the host vehicle V1 and the orientation of the other vehicle V2 is the same as the orientation of the host vehicle V1. The origin of the coordinate system illustrated in FIG. 4 is the position of the host vehicle V1. The X-axis indicates a direction perpendicular to the orientation of the host vehicle V1. The Z-axis indicates the orientation of the host vehicle V1. Time t1 represents the time when the other vehicle V2 is first detected, and times t2 to tn represent the times at which the other vehicle V2 is observed (imaged) after time t1. ｔ１ ~θ ｔｎ represents the angle of view of the image capturing device 10 at each observation time. ｔ１ ~W' ｔｎ represents the time-series data of the rectangular area (bounding box) width W' obtained at each observation time. An enlarged view of the other vehicle V2 is shown in the upper right corner of the coordinate system. The rectangular area width W' can be expressed by the following equation (4).

[0035]

[0036] From equation (4), the following equation (5) can be obtained for calculating the width W and length L of the other vehicle V2.

[0037]

[0038] The above-described method for estimating the width W and length L of the other vehicle V2 is an example, and the present invention is not limited to this. Any method that can estimate the width W and length L using time-series data of the lateral size W' of the rectangular area may be applied.

[0039] In the description of FIG. 4 , an example was described in which the direction of the other vehicle is the same as the direction of the host vehicle, but in reality, the direction of the other vehicle may differ from the direction of the host vehicle. When the estimation parameter selection unit (estimation parameter selection unit 243) determines that the direction of the three-dimensional object (other vehicle) differs from the direction of the vehicle (host vehicle), it selects the width and length of the three-dimensional object (other vehicle) associated with the rectangular region. The three-dimensional object region estimation unit (three-dimensional object region estimation unit 241) estimates the width and length of the three-dimensional object (other vehicle) associated with the rectangular region, assuming that the direction of the three-dimensional object (other vehicle) is constant over time. A specific description will be given below using FIG. 5 .

[0040] FIG. 5 is a diagram for explaining a method for estimating the size and heading direction of another vehicle in the image processing device 20 according to this embodiment. FIG. 5 shows an example in which the heading direction of the other vehicle is constant and the angle between the heading direction of the other vehicle and the heading direction of the subject vehicle is α (α≠0). The x-axis, z-axis, t1 to tn, W, L, and W' shown in FIG. 5 are the same as those explained in FIG. 4, and therefore redundant explanations will be omitted. Since the heading direction of the other vehicle and the heading direction of the subject vehicle are different, as shown in FIG. 5, the parameters representing the angle of view of the imaging device 10 at each observation time are θ L ｔｎ and θ R ｔｎ There are two types: θ L ｔｎ As shown in the figure, θ represents the angle of view at which the leftmost side (the side closest to the origin of the x-axis) of the other vehicle V2 can be observed. R ｔｎrepresents the angle of view at which the rightmost side of the other vehicle V2 (the side furthest from the origin of the x-axis) can be observed. An enlarged view of the other vehicle V2 is shown in the upper right corner of the coordinate system.

[0041] At each observation time, the horizontal size W' of the rectangular area that is the observation value ｔｎ , can be expressed by the following formula (6).

[0042]

[0043] Using equation (6) and a nonlinear optimization method, (W, L, α) can be calculated. Note that the estimation above assumes that the estimated parameters (W, L, α) are constant over time. Therefore, it is difficult to apply this method to a scene in which the direction α of the other vehicle V2 changes.

[0044] In the above estimation, when the three-dimensional object region estimation unit 241 uses all the detection results stored in the detection result storage unit 23, the observed value W' ｔｎ , θ ｔｎ Therefore, detection results with little change in the area are included in the calculation. This causes a problem of reduced estimation accuracy by the three-dimensional object area estimation unit 241 and increased calculation costs. To address this problem, the image processing device 20 according to this embodiment is provided with a time-series information selection unit 242. The time-series information selection unit 242 selects detection results from the detection result storage unit 23 to be used in estimating the size of a three-dimensional object in the three-dimensional object area estimation unit 241. The three-dimensional object area estimation unit (three-dimensional object area estimation unit 241) estimates the size of a three-dimensional object related to a rectangular area based on the time-series change in the detection results selected by the time-series information selection unit (time-series information selection unit 242).

[0045] The time-series information selection unit (time-series information selection unit 242) selects the detection result to be used when estimating the size of the three-dimensional object based on the amount of change in at least one of the width direction length of the rectangular area and the angle of view of the imaging device (imaging device 10) included in the detection result. Specifically, the time-series information selection unit 242 selects W' ｔｎ , θ ｔｎ The detection result at the time when the amount of change of at least one of the above is equal to or greater than a predetermined threshold is selected and output to the three-dimensional object region estimation unit 241. ｔｎ , θ ｔｎThe time-series information selection unit 242 may select data that satisfies a predetermined change amount condition from the data stored in the detection result storage unit 23, such as the position of other vehicles in three-dimensional space, the behavior of the vehicle itself, and the traveling speed.

[0046] Furthermore, depending on the driving scene, when estimating the size of an other vehicle, it may not be necessary to estimate all of the parameters of the other vehicle, namely its width W, length L, and orientation α. ​​For this reason, the image processing device 20 according to this embodiment is provided with an estimation parameter selection unit 243. The estimation parameter selection unit 243 selects a parameter to be estimated from the parameters W, L, and α. The three-dimensional object region estimation unit (three-dimensional object region estimation unit 241) estimates the parameter to be estimated selected by the estimation parameter selection unit (estimation parameter selection unit 243).

[0047] Specifically, the estimation parameter selection unit (estimation parameter selection unit 243) selects at least one of the width length, vertical length, and orientation direction of the three-dimensional object related to the rectangular area as estimation target parameters for the three-dimensional object area estimation unit (three-dimensional object area estimation unit 241) based on at least one of the type of the three-dimensional object (other vehicle), road information related to the road on which the vehicle (host vehicle) is traveling, and the positional relationship between the vehicle and the three-dimensional object. For example, in the example shown in FIG. 4 where the orientation direction of the other vehicle and the orientation direction of the host vehicle are the same, the estimation parameter selection unit 243 selects only W and L as estimation parameters. Selection of estimation parameters similar to the example shown in FIG. 4 can be applied to, for example, scenes where vehicles run side by side on expressways or major trunk roads.

[0048] 5, the estimation parameter selection unit 243 selects W, L, and α as estimation parameters. Selection of estimation parameters similar to the example shown in FIG. 5 can be applied to, for example, a scene in which another vehicle overtakes the host vehicle.

[0049] Furthermore, for example, if the other vehicle is traveling in the same driving lane as the host vehicle and the road shape is straight (assuming a highway or major trunk road), only the back of the other vehicle will be captured within the bounding box. In this case, the width W is equal to the observed value W'. Therefore, if the estimation parameter selection unit (estimation parameter selection unit 243) determines that the three-dimensional object (other vehicle) is located in the driving lane of the vehicle (host vehicle) and in front of the vehicle, the three-dimensional object region estimation unit (three-dimensional object region estimation unit 241) sets the width of the rectangular region to the width of the three-dimensional object related to the rectangular region.

[0050] Furthermore, for example, if the type of the other vehicle is known and the width W and length L of the other vehicle can be determined from the type, the width W and length L may be used. For example, if the type of the other vehicle is known to be a large truck, the length L may be determined to be 12 m, and only W and α may be used as estimation parameters. As described above, by limiting the estimation parameters according to each driving scene, the calculation formula for estimating the three-dimensional object area can be simplified, the estimation accuracy can be improved, and the calculation cost can be reduced.

[0051] The control unit (vehicle control unit 30) outputs a control command to control the operation of the vehicle based on at least one of the size and orientation of the three-dimensional object related to the rectangular area estimated by the three-dimensional object area estimation unit (three-dimensional object area estimation unit 241).

[0052] [Example of Hardware Configuration of Processing Device] Next, a description will be given of an example of the hardware configuration of the image processing device 20. Fig. 6 is a block diagram showing an example of the hardware configuration of the image processing device 20 according to this embodiment.

[0053] 6, the image processing device 20 includes a CPU (Central Processing Unit) 20a, a ROM (Read Only Memory) 20b, a RAM (Random Access Memory) 20c, a storage device 20d, and a communication interface 20e. The CPU 20a, the ROM 20b, the RAM 20c, the storage device 20d, and the communication interface 20e are connected via a bus B so as to be able to transmit and receive information data to and from each other.

[0054] The CPU 20a reads from the ROM 20b, loads into the RAM 20c, and executes program code of software that realizes the functions of each component of the image processing device 20. The CPU 20a realizes the functions of the object detection unit 21, tracking unit 22, and three-dimensional object area calculation unit 24 of the image processing device 20 shown in Fig. 2. Note that a GPU (Graphics Processing Unit) may be used instead of the CPU 20a, or the CPU 20a and the GPU may be used together.

[0055] The ROM 20b is configured as a storage medium such as a nonvolatile memory, and stores programs and data that are executed and referenced by the CPU 20a.

[0056] The RAM 20c is configured by a storage medium such as a volatile memory, and temporarily stores information (data) required for each process performed by the CPU 20a.

[0057] The storage device 20d is a computer-readable, non-transitory recording medium storing the program executed by the CPU 20a, such as a hard disk drive (HDD). The storage device 20d realizes the function of the detection result storage unit 23 of the image processing device 20 shown in FIG. 2. The storage device 20d stores programs for the CPU 20a to execute various functions, an operating system (OS), controller programs, and other data. Note that some of the programs and data stored in the storage device 20d may be stored in the ROM 20b. The computer-readable, non-transitory recording medium storing the program executed by the CPU 20a is not limited to an HDD, and may be, for example, a solid state drive (SSD), a compact disc (CD)-ROM, a digital versatile disc (DVD)-ROM, or other recording medium.

[0058] The communication interface 20e may be, for example, a network interface card (NIC), and various data may be transmitted and received between devices via a CAN, dedicated line, or the like connected to a terminal of the NIC.

[0059] [Three-dimensional object region estimation processing] Next, a procedure for three-dimensional object region estimation processing in the image processing device 20 will be described. Fig. 7 is a flowchart showing the procedure for three-dimensional object region estimation processing in the image processing device 20 according to this embodiment. The processing described below starts when it is detected that a detection result of a new three-dimensional object is to be stored in the detection result storage unit 23.

[0060] First, the three-dimensional object region calculation unit 24 acquires detection results from the detection result storage unit 23 (step S10). In this process, the three-dimensional object region calculation unit 24 acquires, for example, the detection results of all other vehicles stored in the detection result storage unit 23. The three-dimensional object region calculation unit 24 may also determine the possibility of a collision based on the positional relationship between the other vehicles and the vehicle itself, assign a higher priority to other vehicles with a higher probability of collision, and acquire the detection results of other vehicles in descending order of priority. In this process, the three-dimensional object region calculation unit 24 outputs the acquired detection results of other vehicles to the time-series information selection unit 242.

[0061] Next, the time-series information selection unit 242 selects the horizontal size W' of the rectangular region, which is the observation value. ｔｎ and the angle of view θ of the imaging device 10 ｔｎ It is determined whether there is data in which the amount of change in at least one of the above is equal to or greater than a predetermined threshold value registered in advance (step S11).

[0062] In the judgment of step S11, if the time series information selection unit 242 determines that there is no data in which the change in the observation value is equal to or greater than a predetermined threshold value registered in advance (NO judgment of S11), the three-dimensional object area estimation process ends.

[0063] On the other hand, in the determination of step S11, if the time-series information selection unit 242 determines that there is data in which the amount of change in the observation value is equal to or greater than a predetermined threshold value registered in advance (YES determination of S11), the estimation parameter selection unit 243 performs processing of step S12. In the processing of step S12, the estimation parameter selection unit 243 determines whether there is a parameter that can be limited among the parameters of the other vehicle, namely the width W, the length L, and the orientation α, depending on whether or not there is stored information about the driving scene and the type, size, etc. of the other vehicle.

[0064] In the determination of step S12, if the time-series information selection unit 242 determines that there are parameters that can be limited (YES determination of S12), it selects parameters that cannot be limited from the width W, length L, and orientation α as estimation parameters (step S13). Also, in this process, the time-series information selection unit 242 outputs the selected estimation parameters to the three-dimensional object region estimation unit 241.

[0065] On the other hand, if the time-series information selection unit 242 determines in step S12 that there are no parameters that can be limited (NO in step S12), it selects all parameters (width W, length L, and orientation α) as estimation parameters (step S14). In this process, the time-series information selection unit 242 outputs the selected estimation parameters to the three-dimensional object region estimation unit 241.

[0066] After the processing of step S13 or step S14, the solid object area estimation unit 241 estimates the estimation parameters, i.e., the solid object area (step S15). In this processing, the solid object area calculation unit 24 outputs the estimated solid object area (W, L, α) of the other vehicle to the vehicle control unit 30. After the processing of step S15, the solid object area estimation processing ends.

[0067] The vehicle control unit 30 determines a collision with another vehicle based on the three-dimensional object area input from the three-dimensional object area calculation unit 24, and performs display and warning to the driver, vehicle control to avoid a collision, and the like.

[0068] [Effects] As described above, the image processing device 20 according to this embodiment performs a solid object region estimation process when the change in at least one of the observed values, the horizontal size W' of the 2D bounding box, and the angle of view θ of the imaging device at the time of observation, is equal to or greater than a predetermined threshold. In the solid object region estimation process, the image processing device 20 determines which of the other vehicle's width W, length L, and orientation α can be defined, which are parameters for determining the other vehicle region. The image processing device 20 uses the undefined parameters as estimation parameters and calculates the estimation parameters based on time-series information on the horizontal size W' of the 2D bounding box and the angle of view θ of the imaging device. In other words, the image processing device 20 provides the actual size of the other vehicle calculated based on time-series information on the 2D bounding box, rather than providing the size of the 2D bounding box as the size of the detected other vehicle, for false alarm and malfunction suppression control in the driving assistance system. By performing collision judgment based on the actual size of the other vehicle, the driving assistance system can suppress false alarms and malfunctions. Therefore, according to the image processing device 20 of this embodiment, it is possible to accurately estimate the size of another vehicle using a bounding box, and to suppress false alarms and malfunctions in the driving assistance system.

[0069] [Application Example 1] Here, an example of applying the present invention to verifying the estimation result of a 3D bounding box will be described. For example, the object detection unit (object detection unit 21) of the image processing device 20 detects a 3D area including a three-dimensional object as a 3D bounding box. The three-dimensional object area estimation unit (three-dimensional object area estimation unit 241) uses the size of the three-dimensional object associated with the estimated rectangular area to determine whether the 3D bounding box is accurate. This will be described in detail with reference to FIG. 8. FIG. 8 is a diagram for explaining an application example of the image processing device 20 according to this embodiment to verifying the estimation result of a 3D bounding box.

[0070] A 3D bounding box is typically a three-dimensional frame surrounding an other vehicle. However, if the vehicle has a curved shape, a 3D bounding box with incorrect vertices, such as the 3D bounding box Bx3 shown in FIG. 8, may be estimated. With such a 3D bounding box Bx3, the estimation results may be unstable depending on the vehicle shape, making it impossible to accurately calculate the vehicle's width. However, since the left and right edge positions L1 and R1 of the 3D bounding box Bx3 are accurately estimated, the vehicle's width W' can be calculated based on the left and right edge positions L1 and R1. The vehicle's width W, length L, and orientation θ can be estimated using time-series information on the calculated width W'. The estimation results based on the 3D bounding box (values ​​of W, L, and θ) can then be compared with the estimation results of the image processing device 20 to determine whether the 3D bounding box is accurate.

[0071] Specifically, for example, if the difference between the 3D bounding box estimation result and the estimation result of the image processing device 20 is less than a certain value, the three-dimensional object area estimation unit 241 of the image processing device 20 determines that the 3D bounding box is accurate. A 3D bounding box determined to be accurate is output to the vehicle control unit 30 and used in vehicle control. On the other hand, if the difference between the 3D bounding box estimation result and the estimation result of the image processing device 20 is equal to or greater than a certain value, the three-dimensional object area estimation unit 241 determines that the 3D bounding box is inaccurate. A 3D bounding box determined to be inaccurate is not used. If the three-dimensional object area estimation unit (three-dimensional object area estimation unit 241) determines that the 3D bounding box is inaccurate, it can correct the 3D bounding box based on the size of the three-dimensional object associated with the estimated rectangular area.

[0072] Furthermore, the three-dimensional object region estimation unit 241 of the image processing device 20 may estimate an optimal three-dimensional object region based on the 3D bounding box estimation result and the estimation result of the image processing device 20. Specifically, for example, the three-dimensional object region estimation unit 241 assigns weights to both the 3D bounding box estimation result and the estimation result of the image processing device 20 based on their respective reliability. Furthermore, the three-dimensional object region estimation unit 241 calculates a new estimation result based on the weights of both and the estimation result, and outputs it to the vehicle control unit 30 as an optimal three-dimensional object region. Here, the reliability can be calculated using appropriate indicators such as, for example, the stability of the time-series data of the estimation result, the output reliability of the AI ​​that detects the 3D bounding box, and the fitting error of the estimation result. A larger weight is assigned to the one with higher reliability. A larger weight is assigned to the one with lower reliability.

[0073] As described above, by applying the present invention, it is possible to determine whether the 3D bounding box used in vehicle control is accurate, and to avoid vehicle control problems caused by the use of an inaccurate 3D bounding box. Furthermore, by calculating a new optimal estimation result for the three-dimensional object region based on the estimation result of the 3D bounding box and the estimation result of the image processing device 20, it is possible to estimate an even more optimal three-dimensional object region.

[0074] [Application Example 2] Here, an example of application of the present invention to host vehicle control based on the relative relationship with another vehicle other than collision determination in AEB will be described. Examples of host vehicle control based on the relative relationship with another vehicle include preceding vehicle following control and automatic parking control. In preceding vehicle following control, automatic parking control, and the like, it is necessary to determine the relative relationship with the other vehicle. When determining the relative relationship with the other vehicle, if the size and orientation of the other vehicle can be known, the accuracy of determining the relative relationship can be improved. For example, when performing preceding vehicle following control, the size and orientation of the other vehicle calculated by the image processing device 20 of the present invention can be used to more accurately determine the relative relationship with the preceding vehicle, thereby enabling smoother vehicle speed control. Similarly, when performing automatic parking control, the size and orientation of the other vehicle calculated by the image processing device 20 of the present invention can be used to more accurately determine the relative relationship with the preceding vehicle, thereby enabling smoother automatic parking control.

[0075] Furthermore, the present invention is not limited to the above-described embodiments, and various other applications and modifications are possible without departing from the spirit and scope of the present invention as defined in the claims. For example, the above-described embodiments provide a detailed and specific description of the configuration of an image processing device in order to clearly explain the present invention, and the present invention is not necessarily limited to devices that include all of the described components. Furthermore, it is possible to replace some of the components of the embodiments described here with components of other embodiments, and it is also possible to add other components to the configuration of one embodiment. Furthermore, it is also possible to add, delete, or replace other components with respect to some of the configuration of one embodiment. Furthermore, the control lines and information lines shown are those considered necessary for the explanation, and do not necessarily represent all control lines and information lines in the actual product. In reality, it is reasonable to assume that almost all components are interconnected.

[0076] 1... Vehicle control system, 10... Imaging device, 20... Image processing device, 21... Object detection unit, 22... Tracking unit, 23... Detection result storage unit, 24... Solid object area calculation unit, 241... Solid object area estimation unit, 242... Time series information selection unit, 243... Estimation parameter selection unit, 30... Vehicle control unit

Claims

1. An image processing device comprising: an object detection unit that detects an area containing a three-dimensional object as a rectangular area from an image captured by an imaging device mounted on a vehicle; a tracking unit that generates, as detection results, time-series information of multiple rectangular areas in which the same three-dimensional object is detected from multiple images captured in time series by the imaging device, and time-series information of the angle of view of the imaging device at the time the rectangular area was detected; and a three-dimensional object area estimation unit that estimates the size of the three-dimensional object related to the rectangular area based on the time-series changes in the detection results.

2. The image processing device according to claim 1, wherein the size of the three-dimensional object in the rectangular area includes the width and length of the three-dimensional object in the rectangular area.

3. The image processing device according to claim 2, wherein the three-dimensional object area estimation unit also estimates the orientation of the three-dimensional object.

4. An image processing device as described in claim 3, comprising an estimation parameter selection unit that selects at least one of the width direction length, the vertical direction length, and the orientation direction of the three-dimensional object related to the rectangular area as estimation target parameters for the three-dimensional object area estimation unit based on at least one of the type of the three-dimensional object, road information related to the road on which the vehicle is traveling, and the positional relationship between the vehicle and the three-dimensional object.

5. The image processing device described in claim 4, wherein the estimation parameter selection unit selects the width and length of the three-dimensional object related to the rectangular area when it determines that the orientation direction of the three-dimensional object is the same as the orientation direction of the vehicle, and the three-dimensional object area estimation unit estimates the width and length of the three-dimensional object related to the rectangular area selected by the estimation parameter selection unit.

6. The image processing device described in claim 5, wherein when the estimation parameter selection unit determines that the three-dimensional object is located in the vehicle's driving lane and in front of the vehicle, the three-dimensional object area estimation unit sets the width length of the rectangular area to the width length of the three-dimensional object related to the rectangular area.

7. The image processing device described in claim 4, wherein the estimation parameter selection unit selects the width and length of the three-dimensional object related to the rectangular area when it determines that the orientation of the three-dimensional object differs from the orientation of the vehicle, and the three-dimensional object area estimation unit estimates the width and length of the three-dimensional object related to the rectangular area, assuming that the orientation of the three-dimensional object is constant over time.

8. An image processing device as described in claim 2, further comprising a time series information selection unit that selects the detection results to be used when estimating the size of the three-dimensional object based on the amount of change in at least one of the width direction length of the rectangular area and the angle of view of the imaging device, which are included in the detection results, and the three-dimensional object area estimation unit estimates the size of the three-dimensional object related to the rectangular area based on the time series change in the detection results selected by the time series information selection unit.

9. The image processing device according to claim 1, wherein the object detection unit detects a 3D area including the three-dimensional object as a 3D bounding box, and the three-dimensional object area estimation unit uses the size of the three-dimensional object associated with the estimated rectangular area to determine whether the 3D bounding box is accurate.

10. An image processing device as described in claim 9, wherein the three-dimensional object area estimation unit corrects the 3D bounding box based on the size of the three-dimensional object associated with the estimated rectangular area when it determines that the 3D bounding box is inaccurate.

11. A vehicle control system including the image processing device of claim 1, further comprising a vehicle control unit that outputs a control command to control the operation of the vehicle, wherein the vehicle control unit outputs the control command based on at least one of the size of the three-dimensional object related to the rectangular area estimated by the three-dimensional object area estimation unit and the orientation direction of the three-dimensional object.

12. An image processing method comprising the steps of: detecting an area containing a three-dimensional object as a rectangular area from an image captured by an imaging device mounted on a vehicle; generating, as detection results, time series information of a plurality of rectangular areas in which the same three-dimensional object is detected from a plurality of images captured in time series by the imaging device, and time series information of the angle of view of the imaging device at the time of detection of the rectangular area; and estimating the size of the three-dimensional object related to the rectangular area based on the time series changes in the detection results.

Images