Object detection device, object detection method, and computer program for object detection

Abstract

To provide an object detection device with which it is possible to improve the accuracy of detecting objects shown in images.SOLUTION: The object detection device comprises: a detection unit 31 that detects, from an image generated by an imaging unit 2 mounted on a vehicle 10 and showing the surroundings of the vehicle 10, a candidate region in which a prescribed object may possibly be represented, and identifies the type of an object represented in the candidate region; a first distance estimation unit 32 that finds a first estimated distance between the vehicle 10 and the object represented in the candidate region, on the basis of a standard size corresponding to the type identified regarding the object represented in the candidate region and the size of the candidate region; a second distance estimation unit 33 that finds a second estimated distance between the vehicle 10 and the object represented in the candidate region, on the basis of a lower end position of the candidate region in the image, an installed height of the imaging unit 2, and a position in the image equivalent to an azimuth parallel to a road surface; and a determination unit 34 that determines that the object represented in the candidate region is an erroneously detected one, when a difference between the first and second estimated distances is equal to or greater than a prescribed threshold.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to an object detection device, an object detection method, and an object detection computer program for detecting an object shown in an image. [Background technology]

[0002] A technology has been proposed that detects a specific object located within a specific area from an image obtained by photographing the area and measures the distance to the specific object (see Patent Document 1).

[0003] The image processing method disclosed in Patent Document 1 detects the type of object in a captured image. Furthermore, this image processing method calculates a first distance to the object based on a predetermined constant obtained by photographing a chart placed a predetermined distance from the imaging position, the length of the object in the captured image, and a reference length of the object according to the type of the detected object. This image processing method also calculates a second distance to the object based on the imaging range and the ground contact point of the object. This image processing method then calculates the distance to the object using the first distance and the second distance when a provisional distance, which is either the first distance or the second distance, is below a threshold, and uses the first distance as the distance to the object when the provisional distance is equal to or greater than the threshold. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] International Publication No. 2014 / 171052 Summary of the Invention [Problem to be solved by the invention]

[0005] Depending on the object to be detected, there may be similar objects that have similar appearances to the object. If such similar objects are shown in an image, they may be mistakenly detected as the object to be detected.

[0006] Therefore, an object of the present invention is to provide an object detection device that can improve the accuracy of detecting an object shown in an image. [Means for solving the problem]

[0007] According to one embodiment, there is provided an object detection device including: a storage unit that stores a reference size in real space for each type of predetermined object; a detection unit that detects candidate areas that may contain predetermined objects present around the vehicle from an image of the vehicle's surroundings generated by an image capture unit mounted on the vehicle and identifies the type of object represented in the candidate area; a first distance estimation unit that calculates a first estimated distance between the vehicle and the object represented in the candidate area based on the reference size corresponding to the identified type of object represented in the candidate area and the size of the candidate area; a second distance estimation unit that calculates a second estimated distance between the vehicle and the object represented in the candidate area based on the position of a bottom edge of the candidate area in the image, the installation height of the image capture unit, and a position on the image corresponding to an orientation parallel to the road surface; and a determination unit that determines that the object represented in the candidate area has been erroneously detected if the difference between the first estimated distance and the second estimated distance is equal to or greater than a predetermined threshold.

[0008] According to another embodiment, there is provided an object detection method including: detecting a candidate area that may represent a predetermined object present around the vehicle from an image representing the surroundings of the vehicle generated by an image capture unit mounted on the vehicle, identifying a type of the object represented in the candidate area, calculating a first estimated distance between the vehicle and the object represented in the candidate area based on a reference size in real space corresponding to the identified type of object represented in the candidate area and the size of the candidate area; calculating a second estimated distance between the vehicle and the object represented in the candidate area based on a position of a bottom edge of the candidate area in the image, an installation height of the image capture unit, and a position on the image corresponding to an orientation parallel to the road surface; and determining that the object represented in the candidate area has been erroneously detected if a difference between the first estimated distance and the second estimated distance is equal to or greater than a predetermined threshold.

[0009] According to yet another embodiment, there is provided a computer program for object detection, the computer program including instructions for causing a processor mounted on the vehicle to execute the following steps: detect, from an image of the vehicle's surroundings generated by an image capture unit mounted on the vehicle, a candidate area that may contain a predetermined object present around the vehicle, identify the type of the object depicted in the candidate area, calculate a first estimated distance between the vehicle and the object depicted in the candidate area based on the size of the candidate area and a reference size in real space corresponding to the identified type of object depicted in the candidate area, calculate a second estimated distance between the vehicle and the object depicted in the candidate area based on the position of the bottom edge of the candidate area in the image, the installation height of the image capture unit, and a position on the image corresponding to an orientation parallel to the road surface; and determine that the object depicted in the candidate area has been erroneously detected if the difference between the first estimated distance and the second estimated distance is equal to or greater than a predetermined threshold. [Effects of the Invention]

[0010] The object detection device according to the present disclosure has the effect of improving the accuracy of detecting an object shown in an image. [Brief explanation of the drawings]

[0011] [Figure 1] 1 is a schematic configuration diagram of a vehicle control system in which an object detection device is implemented. [Figure 2] FIG. 1 is a hardware configuration diagram of an electronic control device that is an embodiment of an object detection device. [Figure 3] FIG. 2 is a functional block diagram of a processor of an electronic control unit, which is involved in vehicle control processing including object detection processing. [Figure 4] FIG. 10 is a diagram illustrating an example of object detection. [Figure 5] FIG. 10 is a diagram illustrating another example of object detection. [Figure 6] 10 is an operational flowchart of a vehicle control process including an object detection process. DETAILED DESCRIPTION OF THE INVENTION

[0012] The following describes an object detection device, an object detection process executed by the object detection device, and a computer program for object detection, with reference to the drawings. The object detection device detects a candidate area from an image generated by an imaging unit mounted on a vehicle, where a specific object to be detected may be present, and identifies the type of object represented in the candidate area. The object detection device then estimates a first estimated distance, which is the distance between the vehicle and the object represented in the candidate area, based on the reference size in real space of the identified object type and the size of the candidate area on the image. The object detection device also estimates a second estimated distance, which is the distance between the vehicle and the object represented in the candidate area, based on the position of the bottom edge of the candidate area in the image, the installation height of the imaging unit, and the position on the image corresponding to the orientation parallel to the road surface. If the difference between the first estimated distance and the second estimated distance is equal to or greater than a predetermined threshold, the object detection device determines that the object represented in the candidate area has been erroneously detected. If the object represented in the candidate area and its type are correctly detected, the distances between the vehicle and the object estimated by the two methods described above should be approximately equal. Therefore, this object detection device can determine whether an object shown in a candidate area has been detected correctly or incorrectly, based on the difference in distance between the vehicle and the object estimated using the two methods.

[0013] An example in which the object detection device is applied to a vehicle control system will be described below. In this example, the object detection device performs object detection processing on an image acquired by a camera mounted on the vehicle to detect a predetermined object present around the vehicle and estimate the distance to the detected object. The detection result and the estimated value of the distance to the detected object are then used for vehicle driving control or driving assistance. In the following embodiment, the distance from the camera mounted on the vehicle to the detected object is defined as the distance from the vehicle to the object.

[0014] The specified object is an object that may affect the driving control of the vehicle, and may be, for example, another vehicle, a pedestrian, a road sign, or a structure installed on the road (such as a pylon, signboard, or block).

[0015] FIG. 1 is a schematic configuration diagram of a vehicle control system in which an object detection device is implemented. FIG. 2 is a hardware configuration diagram of an electronic control device, which is an example of an object detection device. In this embodiment, a vehicle control system 1 mounted on a vehicle 10 (i.e., the vehicle itself) and controlling the vehicle 10 includes a camera 2 for capturing images of the surroundings of the vehicle 10 and an electronic control unit (ECU) 3, which is an example of an object detection device. The camera 2 and the ECU 3 are communicably connected via an in-vehicle network conforming to a standard such as a controller area network. The vehicle control system 1 may further include a storage device (not shown) that stores high-precision maps representing information used for autonomous driving control, such as the positions and types of road markings and road signs. The vehicle control system 1 may further include a receiver (not shown), such as a GPS receiver, for determining the self-position of the vehicle 10 in accordance with a satellite positioning system. The vehicle control system 1 may also include a navigation device (not shown) for searching for a planned driving route for the vehicle 10.

[0016] Camera 2 is an example of an imaging unit, and includes a two-dimensional detector configured with an array of photoelectric conversion elements, such as a CCD or C-MOS, that are sensitive to visible light, and an imaging optical system that forms an image of the area to be photographed on the two-dimensional detector. Camera 2 is mounted, for example, inside the passenger compartment of vehicle 10 so as to face forward of vehicle 10. Camera 2 photographs the area ahead of vehicle 10 at predetermined photographing intervals (for example, 1 / 30 to 1 / 10 seconds) and generates an image of the area ahead. The image obtained by camera 2 may be a color image or a grayscale image. Note that vehicle 10 may be equipped with multiple cameras with different photographing directions or focal lengths.

[0017] Every time the camera 2 generates an image, it outputs the generated image and the time of photographing (that is, the time of generation of the image) to the ECU 3 via the in-vehicle network.

[0018] The ECU 3 controls the vehicle 10. In this embodiment, the ECU 3 controls the vehicle 10 so that the vehicle 10 automatically drives based on the distance to an object detected from an image obtained by the camera 2. To this end, the ECU 3 includes a communication interface 21, a memory 22, and a processor 23.

[0019] The communication interface 21 is an example of a communication unit, and has an interface circuit for connecting the ECU 3 to an in-vehicle network. That is, the communication interface 21 is connected to the camera 2 via the in-vehicle network. Every time the communication interface 21 receives an image from the camera 2, it passes the received image to the processor 23.

[0020] The memory 22 is an example of a storage unit and includes, for example, a volatile semiconductor memory and a nonvolatile semiconductor memory. The memory 22 stores various data used in the object detection process executed by the processor 23 of the ECU 3. The memory 22 stores, as such data, parameters representing information about the camera 2, such as the focal length, shooting direction, and installation height of the camera 2, as well as various parameters for identifying a classifier used in object detection. The memory 22 also stores various information used in distance estimation. For example, the memory 22 stores, for each type of predetermined object to be detected, a reference size in real space for that type of object. This reference size may be a height or width size. The memory 22 also stores, for a certain period of time, images received from the camera 2 along with the time of capture. The memory 22 also stores, for a certain period of time, various data generated during the object detection process. The memory 22 may also store information used for driving control of the vehicle 10, such as map information.

[0021] The processor 23 is an example of a control unit and includes one or more central processing units (CPUs) and their peripheral circuits. The processor 23 may further include other arithmetic circuits such as a logic operation unit, a numerical operation unit, or a graphics processing unit. The processor 23 executes object detection processing on images received from the camera 2 at predetermined intervals while the vehicle 10 is traveling. The processor 23 then controls the vehicle 10 to automatically drive the vehicle 10 based on the object detection results.

[0022] 3 is a functional block diagram of the processor 23 of the ECU 3, which is related to vehicle control processing including object detection processing. The processor 23 includes a detection unit 31, a first distance estimation unit 32, a second distance estimation unit 33, a determination unit 34, and a vehicle control unit 35. Each of these units included in the processor 23 is a functional module implemented by, for example, a computer program running on the processor 23. Alternatively, each of these units included in the processor 23 may be a dedicated arithmetic circuit provided in the processor 23. Furthermore, among these units included in the processor 23, the detection unit 31, the first distance estimation unit 32, the second distance estimation unit 33, and the determination unit 34 execute the object detection processing. Note that if multiple cameras are provided on the vehicle 10, the processor 23 may execute the object detection processing for each camera based on images acquired by that camera.

[0023] The detection unit 31 inputs the latest image received from the camera 2 to the classifier at predetermined intervals. As a result, the detection unit 31 detects a candidate area (for example, a circumscribing rectangle of the predetermined object) in the image where a predetermined object may be depicted, and identifies the type of the predetermined object depicted in the candidate area. Note that the detection unit 31 may detect multiple candidate areas from one image.

[0024] The detection unit 31 may use a deep neural network (DNN) with a convolutional neural network (CNN) architecture, such as Single Shot MultiBox Detector or Faster R-CNN, as such a classifier. Alternatively, the detection unit 31 may use a DNN with a self-attention network (SAN) architecture, such as Vision Transformer, as such a classifier. These classifiers output a confidence level representing the likelihood that a specific object type is represented for each region in an image. The classifier then detects a region in an image where the confidence level calculated for a specific object type is higher than a specific detection threshold as a candidate region that may represent that type of object. The classifier is trained in advance using a large number of training images depicting a specific object type to be detected, according to a specific training method, such as backpropagation.

[0025] The detection unit 31 notifies the first distance estimation unit 32 and the second distance estimation unit 33 of the position and range on the image of each detected candidate area, as well as the type of object that may be represented in the candidate area.

[0026] For each candidate area detected by the detection unit 31, the first distance estimation unit 32 reads from the memory 22 a reference size corresponding to the type identified for the object depicted in that candidate area. Then, for each candidate area, the first distance estimation unit 32 calculates a first estimated distance between the vehicle 10 and the object depicted in that candidate area based on the reference size corresponding to the type of object depicted in that candidate area and the size of the candidate area on the image. Specifically, the first distance estimation unit 32 can calculate the first estimated distance d1 according to the following equation: d1[m] = f[pixel]×rs[m] / cs[pixel] (1) Here, rs represents a reference size corresponding to the type identified for the object represented in the candidate area, and cs is the size of the candidate area on the image expressed in pixel units. The size of the candidate area corresponds to the size of the object in the direction expressed as the reference size rs. For example, if the reference size rs is the height direction size of the object, the size cs of the candidate area is the vertical height direction size of the candidate area on the image. Also, if the reference size rs is the horizontal direction size of the object, the size cs of the candidate area is the horizontal direction size of the candidate area on the image. Also, f is the focal length of camera 2 expressed in pixel size units.

[0027] Note that the first distance estimation unit 32 may calculate the first estimated distance using the size on the image (hereinafter referred to as the reference image size) that corresponds to the reference size of the object when the distance between the object shown in the candidate area and the camera 2 is a predetermined reference distance. In this case, the first distance estimation unit 32 can calculate the first estimated distance by multiplying the reference distance by the ratio of (reference image size / candidate area size).

[0028] The first distance estimation unit 32 notifies the determination unit 34 of the position and range of each candidate area detected by the detection unit 31, the type of object represented in that candidate area, and the first estimated distance calculated for that candidate area.

[0029] For each detected candidate area, the second distance estimation unit 33 calculates a second estimated distance between the vehicle 10 and the object depicted in that candidate area based on the position of the bottom edge of that candidate area on the image, the installation height of the camera 2, and the position on the image corresponding to the orientation parallel to the road surface. Specifically, the second distance estimation unit 33 can calculate the second estimated distance d2 according to the following equation. d2[m] = f[pixel]×h[m] / (cb[pixel]-FOE[pixel]) (2) Here, f is the focal length of camera 2 expressed in pixel size units in the vertical direction of the image, and h is the installation height of camera 2 from the road surface. Also, cb represents the position of the bottom edge of the candidate area in the vertical direction of the image, and FOE represents the position of the vanishing point in the vertical direction of the image. The vanishing point FOE represents the position on the image corresponding to the direction parallel to the road surface.

[0030] To detect vanishing points on the image, the second distance estimation unit 33 inputs the image to a classifier that has been trained in advance to detect lane markings. This allows the second distance estimation unit 33 to detect two or more lane markings. The second distance estimation unit 33 may use a classifier similar to that described for the detection unit 31, or may use a CNN for semantic segmentation, such as a Fully Convolutional Network (FCN) or U-net. Alternatively, the classifier used by the detection unit 31 to detect a predetermined object may be trained in advance to also detect lane markings.

[0031] The second distance estimation unit 33 approximates each detected lane marking with a straight line by applying a Hough transform to the set of pixels representing the lane marking. The second distance estimation unit 33 then detects the intersection of the two linearly approximated lane markings as the vanishing point. Note that the second distance estimation unit 33 may also detect the vanishing point using other methods for detecting vanishing points on an image.

[0032] Furthermore, if the variation in the pitch direction tilt of the vehicle 10 is negligible, the second distance estimation unit 33 may use the coordinates of a position on the image corresponding to a direction parallel to the road surface (hereinafter referred to as parallel position coordinates) instead of the FOE in equation (2). The parallel position coordinates are calculated in advance based on the shooting direction of the camera 2 and stored in the memory 22. Therefore, the second distance estimation unit 33 may read the parallel position coordinates from the memory 22 and substitute them into equation (2).

[0033] When the vehicle 10 is stopped, the second distance estimation unit 33 determines that the fluctuation in the tilt of the vehicle 10 in the pitch direction is negligible. Alternatively, when the difference between the angle in the pitch direction measured by an angle sensor (not shown) mounted on the vehicle 10 and a predetermined reference angle is within a predetermined allowable range, the second distance estimation unit 33 may determine that the fluctuation in the tilt of the vehicle 10 in the pitch direction is negligible.

[0034] The second distance estimation unit 33 notifies the determination unit 34 of the position and range of each candidate area detected by the detection unit 31 and the second estimated distance calculated for that candidate area.

[0035] For each candidate area detected by the detection unit 31, the determination unit 34 calculates the absolute value of the difference (hereinafter simply referred to as the difference) between the first estimated distance and the second estimated distance calculated for that candidate area. If the difference is equal to or greater than a predetermined threshold, the determination unit 34 determines that the object shown in that candidate area has been erroneously detected. On the other hand, if the difference is less than the predetermined threshold, the determination unit 34 determines that the object shown in that candidate area has been correctly detected.

[0036] 4 and 5 are diagrams illustrating an example of object detection according to this embodiment. In the example shown in Fig. 4, an object 401 to be detected is correctly detected from an image 400, and the type of the object 401 is also correctly identified. Therefore, the difference between a first estimated distance d1 from the vehicle 10 to the object 401, which is calculated based on the reference size rs of the object 401, and a second estimated distance d2, which is calculated based on the position cb of the bottom edge of the candidate area 402 in which the object 401 is represented, is less than the threshold Th. Therefore, the object 401 is determined to have been correctly detected.

[0037] In the example shown in FIG. 5, a similar object 502, which has an appearance similar to that of the object 501, is erroneously detected as the object 501 in the image 500. The size of the similar object 502 in real space is different from the reference size of the object 501. Therefore, the first estimated distance d1 calculated based on the reference size of the object 501 is significantly different from the actual distance between the vehicle 10 and the similar object 502. In contrast, the position of the bottom edge of the candidate area 503 in which the similar object 502 is depicted represents the direction to the position where the similar object 502 contacts the road surface as seen from the camera 2. Therefore, the second estimated distance d2 calculated based on the position of the bottom edge of the candidate area 503 is almost accurate. As a result, the difference between the first estimated distance d1 and the second estimated distance d2 is equal to or greater than the threshold value Th. Therefore, the object 501 is determined to have been erroneously detected.

[0038] The determination unit 34 determines, from among the individual detected candidate areas, a candidate area in which an object has been correctly detected as an object area. For each object area, the determination unit 34 determines either the first estimated distance or the second estimated distance, or the average value, as the distance from the vehicle 10 to the object depicted in that object area. The determination unit 34 then notifies the vehicle control unit 35 of the type of object depicted in the object area and the distance to the object. The determination unit 34 may also notify the vehicle control unit 35 of both the first estimated distance and the second estimated distance. Furthermore, the determination unit 34 notifies the vehicle control unit 35 of the position and range of the object area on the image, or the orientation of the object area as seen from the camera 2, which corresponds to the center of gravity of the object area. Furthermore, for a candidate area in which an object has been erroneously detected, from among the individual detected candidate areas, the determination unit 34 may also notify the vehicle control unit 35 of the determination result and the position and range of the candidate area on the image.

[0039] The vehicle control unit 35 controls the vehicle 10 to automatically drive so that the vehicle 10 does not collide with the detected object, based on the object detection result and the distance between the vehicle 10 and the object. For example, if the detected object is a stationary object such as a signboard and is located on the path of the vehicle 10, the vehicle control unit 35 decelerates the vehicle 10 when the distance to the detected object becomes equal to or less than a predetermined distance. The vehicle control unit 35 then stops the vehicle 10 before it reaches the position of the detected object. Alternatively, the vehicle control unit 35 may control the vehicle 10 to change lanes from the lane in which the vehicle 10 is traveling to a lane adjacent to the lane in which the vehicle 10 is traveling.

[0040] If the difference between the traveling direction of the vehicle 10 and the direction of the detected object as seen from the camera 2 is less than a predetermined angle threshold, the vehicle control unit 35 determines that the object is located on the path of the vehicle 10. In this case, the vehicle control unit 35 may decrease the predetermined angle threshold as the distance to the object increases. Alternatively, as described for the detection of the vanishing point in the second distance estimation unit 33, lane markings may be detected from the image. In this case, the vehicle control unit 35 identifies the detected lane markings closest to the vehicle 10 on each of the left and right sides of the vehicle 10 as the lane markings that define the vehicle's own lane. The vehicle control unit 35 then determines that the area between the identified lane markings on the image represents the vehicle's own lane. The vehicle control unit 35 may determine that the detected object is located on the path of the vehicle 10 if the lower end of the object area representing the detected object at least partially overlaps with the area representing the vehicle's own lane.

[0041] Furthermore, if the detected object is a moving object such as another vehicle or a pedestrian, the vehicle control unit 35 predicts the trajectory of the moving object for a predetermined time ahead and controls the vehicle 10 so that it moves at least a certain distance away from the predicted trajectory. To this end, the vehicle control unit 35 tracks the detected moving object in a series of time-series images generated by the camera 2 over the most recent predetermined period, and predicts the trajectory of the moving object based on the tracking results.

[0042] The vehicle control unit 35 applies a tracking process based on optical flow, such as the Lucas-Kanade algorithm or the KLT algorithm, to the object region of interest in the latest image and the object region in the previous image. As a result, the vehicle control unit 35 associates object regions in which the same moving object is represented across a time-series series of images. To this end, the vehicle control unit 35 extracts multiple feature points from the object region of interest in the latest image by applying a feature point extraction filter, such as SIFT or a Harris operator, to the object region of interest. The vehicle control unit 35 then calculates the optical flow by identifying corresponding points in the object region in the previous image for each of the multiple feature points according to the applied tracking method. Alternatively, the vehicle control unit 35 may associate object regions in which the same moving object is represented by applying another tracking method, which is used to track moving objects detected from images, to the object region of interest in the latest image and the object region in the previous image.

[0043] The vehicle control unit 35 converts the coordinates of the moving object in the image into coordinates on the bird's-eye image (bird's-eye coordinates) by performing a viewpoint conversion process using information such as the mounting position of the camera 2 on the vehicle 10. The vehicle control unit 35 estimates the position of the moving object at the time of generating each image based on the position and traveling direction of the vehicle 10 at the time of generating each image, the distance to the detected moving object, and the direction from the vehicle 10 to the moving object. The vehicle control unit 35 then arranges the estimated positions of the moving object in chronological order to estimate the trajectory of the moving object being tracked. Furthermore, the vehicle control unit 35 can estimate the predicted trajectory of the moving object up to a predetermined time in the future by performing a prediction process using a Kalman filter or a particle filter based on the trajectory of the moving object being tracked during a predetermined recent period. The vehicle control unit 35 may use either the first estimated distance or the second estimated distance, whichever produces the smaller error in the prediction process, as the distance from the vehicle 10 to the moving object. Furthermore, the vehicle control unit 35 can use the position of the vehicle 10 at the time of image generation measured by a GPS receiver (not shown) mounted on the vehicle 10 as the position of the vehicle 10 at the time of image generation. Furthermore, the vehicle control unit 35 can use the traveling direction of the vehicle 10 at the time of image generation measured by a direction sensor (not shown) mounted on the vehicle 10 as the traveling direction of the vehicle 10 at the time of image generation. Alternatively, the vehicle control unit 35 can compare each image with map information to estimate the position and traveling direction of the vehicle 10 at the time of image generation.

[0044] The vehicle control unit 35 generates a planned traveling trajectory based on the predicted trajectory of the moving object being tracked so that the predicted value of the distance between the moving object being tracked and the vehicle 10 up to a predetermined time ahead is equal to or greater than a predetermined distance.

[0045] Once the planned travel route is set, the vehicle control unit 35 controls each unit of the vehicle 10 so that the vehicle 10 travels along the planned travel route. For example, the vehicle control unit 35 calculates the deceleration of the vehicle 10 based on the planned travel route and the current vehicle speed of the vehicle 10 measured by a vehicle speed sensor (not shown), and sets the accelerator opening or braking amount to achieve that deceleration. The vehicle control unit 35 then calculates the fuel injection amount based on the set accelerator opening, and outputs a control signal corresponding to the fuel injection amount to a fuel injection device of the engine of the vehicle 10. Alternatively, the vehicle control unit 35 calculates the amount of power to be supplied to the motor based on the set accelerator opening, and controls the motor drive circuit so that the amount of power is supplied to the motor. Furthermore, the vehicle control unit 35 outputs a control signal corresponding to the set braking amount to the brake of the vehicle 10.

[0046] Furthermore, when the vehicle 10 changes its course so that the vehicle 10 travels along a planned travel route, the vehicle control unit 35 determines the steering angle of the vehicle 10 according to the planned travel route. Then, the vehicle control unit 35 outputs a control signal corresponding to the steering angle to an actuator (not shown) that controls the steered wheels of the vehicle 10.

[0047] The vehicle control unit 35 does not use the object determined to have been erroneously detected in controlling the vehicle 10. Alternatively, if the object determined to have been erroneously detected is located at a predicted position obtained by tracking an object detected from a past image, the vehicle control unit 35 may use the erroneously detected object as the object being tracked in controlling the vehicle 10.

[0048] Fig. 6 is an operational flowchart of a vehicle control process including an object detection process, which is executed by the processor 23. The processor 23 executes the vehicle control process at predetermined intervals in accordance with the operational flowchart shown in Fig. 6. In the operational flowchart shown below, the processes of steps S101 to S106 correspond to the object detection process.

[0049] The detection unit 31 of the processor 23 inputs the latest image obtained from the camera 2 into a classifier to detect candidate areas in which a specified object may be depicted, and identifies the type of the specified object depicted in the candidate area (step S101).

[0050] The first distance estimation unit 32 of the processor 23 calculates a first estimated distance d1 between the vehicle 10 and the object depicted in the candidate area based on a reference size corresponding to the type of object depicted in the candidate area and the size of the candidate area on the image (step S102).

[0051] The second distance estimation unit 33 of the processor 23 calculates a second estimated distance d2 between the vehicle 10 and the object depicted in the candidate area based on the position of the bottom edge of the candidate area on the image, the installation height of the camera 2, and the position on the image corresponding to the direction parallel to the road surface (step S103).

[0052] The determination unit 34 of the processor 23 determines whether the difference |d1-d2| between the first estimated distance d1 and the second estimated distance d2 is less than a predetermined threshold Th (step S104). If the difference |d1-d2| is less than the predetermined threshold Th (step S104-Yes), the determination unit 34 determines that the object represented in the candidate area has been correctly detected (step S105). On the other hand, if the difference |d1-d2| is equal to or greater than the predetermined threshold Th (step S104-No), the determination unit 34 determines that the object represented in the candidate area has been erroneously detected (step S106). Note that if multiple candidate areas are detected from the image, the processor 23 may perform the processes of steps S102 to S106 for each candidate area.

[0053] After step S105 or S106, the vehicle control unit 35 of the processor 23 controls the vehicle 10 based on the object detection result (step S107), and the processor 23 then ends the vehicle control process.

[0054] As described above, this object detection device estimates the distance between the vehicle and an object depicted in a candidate area detected from an image generated by an imaging unit mounted on the vehicle using two different methods.The object detection device then determines whether the object depicted in the candidate area was detected correctly or erroneously based on the difference between the distances estimated by each method.As a result, this object detection device can improve the accuracy of detecting objects depicted in images and the accuracy of estimating the distance between the vehicle and the object.

[0055] According to a modified example, the determination unit 34 may control the threshold value to be compared with the difference between the first estimated distance and the second estimated distance, depending on the type of object identified in the candidate area. In this case, a reference table indicating the relationship between the object type and the threshold value is stored in advance in the memory 22. The determination unit 34 may then refer to the reference table to determine the threshold value corresponding to the type of object identified in the candidate area of ​​interest. According to this modified example, the determination unit 34 can set the threshold value to be compared with the difference between the first estimated distance and the second estimated distance to a more appropriate value. As a result, the determination unit 34 can more accurately determine whether the object depicted in the candidate area has been correctly detected or erroneously detected.

[0056] In the above embodiment or modification, if the type of object depicted in the candidate area has a known height, the second distance estimation unit 33 may use the position of the upper end of the candidate area on the image instead of the position of the lower end of the candidate area to determine the second estimated distance. In this case, the object detection device can achieve the same effect as in the above embodiment.

[0057] A computer program that realizes the functions of each part of the processor 23 of the object detection device according to the above embodiment or variant may be provided in a form recorded on a computer-readable portable recording medium such as a semiconductor memory, a magnetic recording medium or an optical recording medium.

[0058] As described above, those skilled in the art can make various modifications to the embodiments within the scope of the present invention. [Explanation of symbols]

[0059] 1. Vehicle control system 10 vehicles 2 Cameras 3 Electronic control device (object detection device) 21 Communication Interface 22 Memory 23 processors 31 Detector 32 First distance estimation unit 33 Second distance estimation unit 34 Judgment section 35 Vehicle control unit

Claims

1. a storage unit that stores a reference size in real space for each type of predetermined object; a detection unit that detects a candidate area in which the predetermined object existing around the vehicle may be depicted from an image showing the surroundings of the vehicle generated by an imaging unit mounted on the vehicle, and identifies the type of object depicted in the candidate area; a first distance estimation unit that calculates a first estimated distance between the vehicle and the object represented in the candidate area based on a reference size corresponding to a type identified for the object represented in the candidate area, among the reference sizes for each type, and the size of the candidate area; a second distance estimation unit that calculates a second estimated distance between the vehicle and an object depicted in the candidate area based on a position of a lower end of the candidate area in the image, an installation height of the imaging unit, and a position on the image corresponding to a direction parallel to a road surface; a determination unit that determines whether a difference between the first estimated distance and the second estimated distance is equal to or greater than a predetermined threshold; a vehicle control unit that controls the vehicle based on at least one of the first estimated distance and the second estimated distance when the difference is less than the predetermined threshold value to prevent a collision between the object and the vehicle; and When the difference is equal to or greater than the predetermined threshold, the vehicle control unit calculates a predicted position at the time of image generation for a tracking object that has been detected in a plurality of past images previously obtained by the imaging unit and is being tracked, and when the position of the object estimated based on the orientation corresponding to the candidate area and the first estimated distance or the second estimated distance matches the predicted position for the tracking object, the vehicle control unit determines that the object is the tracking object and controls the vehicle to prevent a collision between the tracking object and the vehicle. Object detection device.

2. detecting a candidate area in which a predetermined object existing around the vehicle may be depicted from an image showing the surroundings of the vehicle generated by an imaging unit mounted on the vehicle, and identifying the type of object depicted in the candidate area; calculating a first estimated distance between the vehicle and the object represented in the candidate area based on a reference size in real space corresponding to the type identified for the object represented in the candidate area and the size of the candidate area; calculating a second estimated distance between the vehicle and the object depicted in the candidate area based on a position of a lower end of the candidate area in the image, an installation height of the imaging unit, and a position on the image corresponding to a direction parallel to a road surface; determining whether a difference between the first estimated distance and the second estimated distance is greater than or equal to a predetermined threshold; If the difference is less than the predetermined threshold, controlling the vehicle based on at least one of the first estimated distance and the second estimated distance to prevent a collision between the object and the vehicle; If the difference is equal to or greater than the predetermined threshold, a predicted position at the time of image generation is obtained for a tracking object that has been detected in a plurality of past images previously obtained by the imaging unit and is being tracked, and if the position of the object estimated based on the orientation corresponding to the candidate area and the first estimated distance or the second estimated distance is the predicted position for the tracking object, the object is determined to be the tracking object, and the vehicle is controlled so that the tracking object does not collide with the vehicle. The object detection method includes:

3. detecting a candidate area in which a predetermined object existing around the vehicle may be depicted from an image showing the surroundings of the vehicle generated by an imaging unit mounted on the vehicle, and identifying the type of object depicted in the candidate area; calculating a first estimated distance between the vehicle and the object represented in the candidate area based on a reference size in real space corresponding to the type identified for the object represented in the candidate area and the size of the candidate area; calculating a second estimated distance between the vehicle and the object depicted in the candidate area based on a position of a lower end of the candidate area in the image, an installation height of the imaging unit, and a position on the image corresponding to a direction parallel to a road surface; determining whether a difference between the first estimated distance and the second estimated distance is greater than or equal to a predetermined threshold; If the difference is less than the predetermined threshold, controlling the vehicle based on at least one of the first estimated distance and the second estimated distance to prevent a collision between the object and the vehicle; If the difference is equal to or greater than the predetermined threshold, a predicted position at the time of image generation is obtained for a tracking object that has been detected in a plurality of past images previously obtained by the imaging unit and is being tracked, and if the position of the object estimated based on the orientation corresponding to the candidate area and the first estimated distance or the second estimated distance is the predicted position for the tracking object, the object is determined to be the tracking object, and the vehicle is controlled to prevent a collision between the tracking object and the vehicle. A computer program for object detection that causes a processor mounted on the vehicle to execute the above.

Images