Detection device, detection method, and storage medium

Abstract

One of the purposes of the present invention is to provide a detection device, method, and storage medium that can more reliably suppress a reduction in the accuracy of identifying objects around a vehicle. The detection device comprises: a first feature point extraction unit that extracts feature points of an object detected by the first object detection unit as first feature points based on a detection result of a first object detection unit that detects an object in front of the vehicle; a second feature point extraction unit that extracts feature points of the object as second feature points based on a detection result of a second object detection unit whose detection range includes a portion or all of the detection range of the first object detection unit; and a determination unit that determines that there is an obstacle to the detection of the object, based on the detection result of the first object detection unit and the second object detection unit, for extracting a feature point group that lacks a corresponding feature point from a point group that has obtained the first feature point and a point group that has obtained the second feature point.

Description

Technical Field

[0001] The present invention relates to a detection device, a detection method and a storage medium. Background Art

[0002] Conventionally, in order to prevent a decrease in the detection performance of the vehicle's surrounding information, a technology has been developed that operates a cleaning device when dirt is detected based on the defect rate of the surrounding information (Patent Document 1).

[0003] Prior art literature

[0004] Patent Literature

[0005] Patent Document 1: Japanese Patent Application Laid-Open No. 2016-179767 Summary of the Invention

[0006] Problems to be solved by the invention

[0007] However, in the existing technology, since the defect rate of the peripheral information output by the sensor is detected based on the output status of the signal of the sensor based on the cleaning object, it is sometimes not possible to obtain a good defect rate depending on the type and status of the detection object, and sometimes it is unavoidable to reduce the recognition rate of the peripheral information.

[0008] The present invention has been made in consideration of such circumstances, and one of its objects is to provide a detection device, a detection method, and a storage medium that can more reliably suppress a decrease in recognition accuracy of objects around a host vehicle.

[0009] Solutions to Problems

[0010] The detection device, detection method, and storage medium of the present invention adopt the following structure. (1): A detection device of one embodiment of the present invention comprises: a first feature point extraction unit, which extracts the feature points of the object detected by the first object detection device as first feature points based on the detection result of a first object detection device that detects an object in front of the vehicle; a second feature point extraction unit, which extracts the feature points of the object as second feature points based on the detection result of a second object detection device that detects the object and whose detection range includes a part or all of the detection range of the first object detection device; and a determination unit, which determines that there is an obstacle to the detection of the object in the object detection device related to the detection result of the first object detection device and the second object detection device for extracting the feature point group that lacks the corresponding feature point in the first feature point group and the second feature point group.

[0011] (2) In the above-mentioned embodiment (1), the determination unit determines that the obstruction factor exists when the first feature point or the second feature point without a corresponding feature point is detected continuously for a predetermined time period or longer.

[0012] (3): Based on the above scheme (1) or (2), the judgment unit detects the feature points that should have been detected, i.e., the missing feature points, in a time series for the first feature point or the second feature point that does not have a corresponding feature point, and totals the number of the missing feature points detected for each partial area into which the detection range is divided. When the total number of the missing feature points is greater than a threshold value, it is determined that the obstacle exists in the partial area where the total number is obtained.

[0013] (4): Based on any one of the above schemes (1) to (3), the first object detection device is a camera that detects objects in front of the vehicle through the front window from inside the vehicle, and the obstacle to object detection based on the first object detection device is water droplets attached to the front window.

[0014] (5): Based on the above scheme (4), the second object detection device is a radar device located on the back of the radio wave transmission portion provided in a part of the bumper, and the obstacle to object detection based on the second object detection device is an attachment relative to the radio wave transmission portion.

[0015] (6): Based on the above solution (4), the second object detection device is a laser radar device installed on the bumper, and the obstacle to object detection based on the second object detection device is an attachment relative to the light-emitting part or the light-receiving part of the laser radar device.

[0016] (7): Based on any one of the above schemes (4) to (6), a heater control unit is further provided, which operates the heater device that blows warm air to the front window according to the determination result of the determination unit, and the heater control unit operates the heater device when the determination unit determines that the obstruction factor exists in the object detection of the first object detection device.

[0017] (8): A detection method according to one embodiment of the present invention causes a computer to perform the following processing: extracting the feature points of the object detected by the first object detection device as first feature points based on the detection results of a first object detection device that detects an object in front of the vehicle; extracting the feature points of the object as second feature points based on the detection results of a second object detection device that detects the object and whose detection range includes a part or all of the detection range of the first object detection device; and determining that there is an obstacle to the detection of the object in the object detection device with respect to the detection results of the first object detection device and the second object detection device for extracting the feature point group of the first feature point group and the second feature point group that lacks the corresponding feature points.

[0018] (9): A storage medium of one embodiment of the present invention stores a program, wherein the program causes a computer to perform the following processing: extracting feature points of the object detected by a first object detection device as first feature points based on a detection result of a first object detection device that detects an object in front of the vehicle; extracting feature points of the object as second feature points based on a detection result of a second object detection device that is used to detect the object and whose detection range includes a part or all of the detection range of the first object detection device; and determining that there is an obstacle to the detection of the object in the object detection device with respect to the detection result involving the extraction of a feature point group that has obtained the first feature point and a feature point group that has missing corresponding feature points in the first object detection device and the second object detection device.

[0019] Effects of the Invention

[0020] According to the schemes (1) to (9) above, feature points of the object detected by the first object detection device are extracted as first feature points based on the detection results of the first object detection device for detecting the object in front of the vehicle; feature points of the object are extracted as second feature points based on the detection results of the second object detection device for detecting the object and whose detection range includes part or all of the detection range of the first object detection device; and the object detection device determines that there are obstacles to the detection of the object based on the detection results of the first object detection device and the second object detection device for extracting the feature point group that lacks the corresponding feature points in the point group that has obtained the first feature point, i.e., the first feature point group, and the point group that has obtained the second feature point, i.e., the second feature point group. This makes it possible to more reliably suppress the reduction in the recognition accuracy of objects around the vehicle. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 It is a configuration diagram of a vehicle system using the vehicle control device according to the embodiment.

[0022] Figure 2 This is a functional structure diagram of the first control unit and the second control unit.

[0023] Figure 3 This is a diagram showing an example of the correspondence between the driving mode and the control state and task of the host vehicle.

[0024] Figure 4 This is a block diagram showing an example of a configuration that the object recognition device according to the embodiment has for detecting an obstruction factor.

[0025] Figure 5 This is a diagram showing an example of a case where the obstacle determination unit determines that there is “no obstacle” in the object recognition device according to the embodiment.

[0026] Figure 6 This is a diagram showing an example of a case where the obstacle determination unit determines that “an obstacle exists” in the object recognition device according to the embodiment.

[0027] Figure 7 This is a flowchart showing an example of the flow of non-corresponding feature point detection processing executed by the object recognition device according to the embodiment.

[0028] Figure 8 This is a flowchart showing an example of the flow of an obstacle determination process executed by the object recognition device according to the embodiment.

[0029] Figure 9 This is an image diagram showing a specific example of the obstacle determination process according to the embodiment.

[0030] Description of reference numerals:

[0031] 1...Vehicle System, 10...Camera, 12...Radar Device, 14...LIDAR, 20...Communication Device, 30...HMI, 40...Vehicle Sensor, 50...Navigation Device, 51...GNSS Receiver, 52...Navigation HMI, 53...Route Determination Unit, 54...First Map Information, 60...MPU, 61...Recommended Lane Determination Unit, 62...Second Map Information, 70...Driver Monitoring Camera, 80...Driving Operation Components, 82...Steering Wheel, 84...Steering Wheel Grip Sensor, 100...Automatic Driving Control Unit, 120...First Control Unit, 130...Recognition Unit, 140... Action plan generating unit, 150...Mode determining unit, 152...Driver state determining unit, 154...Mode changing processing unit, 160...Second control unit, 162...Acquisition unit, 164...Speed control unit, 166...Steering control unit, 200...Travel driving force output device, 210...Braking device, 220...Steering device, 300...Heater device, 16...Object recognition device, 610...Storage unit, 620...Control unit, 621...First feature point extraction unit, 622...Second feature point extraction unit, 623...Feature point overlapping unit, 624...Obstacle factor determining unit, 625...Heater control unit. DETAILED DESCRIPTION

[0032] Hereinafter, embodiments of a detection device, a detection method, and a storage medium according to the present invention will be described with reference to the accompanying drawings.

[0033] [Overall structure]

[0034] Figure 1 This is a structural diagram of a vehicle system 1 utilizing a vehicle control device according to an embodiment. The vehicle equipped with vehicle system 1 is, for example, a two-wheeled, three-wheeled, or four-wheeled vehicle, and its driving source is an internal combustion engine such as a diesel engine or a gasoline engine, an electric motor, or a combination thereof. The electric motor operates using power generated by a generator coupled to the internal combustion engine, or power discharged from a secondary battery or fuel cell.

[0035] The vehicle system 1 includes, for example, a camera 10, a radar device 12, a LIDAR (Light Detection and Ranging) 14, an object recognition device 16, a communication device 20, an HMI (Human Machine Interface) 30, a vehicle sensor 40, a navigation device 50, an MPU (Map Positioning Unit) 60, a driver monitoring camera 70, a driving operating part 80, an automatic driving control device 100, a driving force output device 200, a braking device 210, a steering device 220, and a heater device 300. These devices and equipment are interconnected through multiple communication lines such as CAN (Controller Area Network) communication lines, serial communication lines, wireless communication networks, etc. It should be noted that, Figure 1 The structure shown is merely an example, and part of the structure may be omitted or other structures may be further added.

[0036] Camera 10 is, for example, a digital camera utilizing a solid-state imaging element such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor). Camera 10 is mounted at any location on a vehicle equipped with vehicle system 1 (hereinafter, the vehicle itself). To capture images of the front, camera 10 is mounted on the upper portion of the windshield, behind the rearview mirror, or elsewhere in the vehicle. Camera 10, for example, periodically and repeatedly captures images of the surroundings of the vehicle itself. Camera 10 may also be a stereo camera.

[0037] The radar device 12 transmits radio waves, such as millimeter waves, toward the periphery of the vehicle and detects the radio waves (reflected waves) reflected by objects, thereby detecting at least the object's position (range and direction). The radar device 12 can be installed anywhere on the vehicle. The radar device 12 can also detect the position and speed of an object using FM-CW (Frequency Modulated Continuous Wave) technology. For example, the radar device 12 can be installed behind a radio wave transmission portion provided in a portion of the bumper.

[0038] LIDAR 14 irradiates light (or electromagnetic waves with a wavelength close to that of light) around the vehicle and measures the scattered light. LIDAR 14 detects the distance to an object based on the time between light emission and light reception. The irradiated light is, for example, a pulsed laser. LIDAR 14 can be installed anywhere on the vehicle.

[0039] The object recognition device 16 performs sensor fusion processing on the detection results from some or all of the camera 10, radar device 12, and LIDAR 14 to identify the position, type, speed, etc. of the object. The object recognition device 16 outputs the recognition results to the automatic driving control device 100. The object recognition device 16 can output the detection results from the camera 10, radar device 12, and LIDAR 14 directly to the automatic driving control device 100. The object recognition device 16 can also be omitted from the vehicle system 1.

[0040] Furthermore, in addition to the object recognition function described above, the object recognition device 16 of this embodiment also has a function of detecting the presence of objects that could obstruct object detection by the camera 10, the radar device 12, and the LIDAR 14. For example, with respect to object detection by the camera 10, an object that exists within the detection range (photographing range) of the camera 10 and obstructs the view of the object to be detected may become an obstruction.

[0041] For example, water droplets adhering to the front window or lens can be an example of a factor hindering object detection by the camera 10. Furthermore, if the radar device 12 is installed behind a radio wave transmission portion provided in a portion of the bumper, dirt adhering to the radio wave transmission portion can be an example of a factor hindering object detection by the radar device 12. Furthermore, dirt adhering to the light emitting unit or light receiving unit of the LIDAR 14 can be an example of a factor hindering object detection by the LIDAR 14.

[0042] The object recognition device 16 determines the presence of the aforementioned obstructions based on detection results from object detection devices such as the camera 10, radar device 12, and LIDAR 14. The method for determining the presence of obstructions will be described in detail later. The object recognition device 16 is an example of a "detection device." The object recognition device 16 can also be separated into an object recognition device that has object recognition capabilities and a detection device that detects the presence of obstructions.

[0043] The communication device 20 communicates with other vehicles around the vehicle using, for example, a cellular network, Wi-Fi network, Bluetooth (registered trademark), DSRC (Dedicated Short Range Communication), or communicates with various server devices via a wireless base station.

[0044] The HMI 30 presents various information to the occupants of the vehicle and receives input operations from the occupants. The HMI 30 includes various display devices, speakers, buzzers, touch panels, switches, buttons, and the like.

[0045] The vehicle sensor 40 includes a vehicle speed sensor that detects the speed of the vehicle, an acceleration sensor that detects acceleration, a yaw rate sensor that detects angular velocity about a vertical axis, an azimuth sensor that detects the orientation of the vehicle, and the like.

[0046] The navigation device 50 includes, for example, a GNSS (Global Navigation Satellite System) receiver 51, a navigation HMI 52, and a route determination unit 53. The navigation device 50 stores first map information 54 in a storage device such as an HDD (Hard Disk Drive) or a flash memory. The GNSS receiver 51 determines the position of the vehicle based on signals received from GNSS satellites. The position of the vehicle can also be determined or supplemented by an INS (Inertial Navigation System) that utilizes the output of the vehicle sensor 40. The navigation HMI 52 includes a display device, a speaker, a touch panel, buttons, etc. The navigation HMI 52 can also be partially or entirely shared with the aforementioned HMI 30. The route determination unit 53, for example, refers to the first map information 54 to determine a route (hereinafter referred to as a route on the map) from the position of the vehicle determined by the GNSS receiver 51 (or an arbitrary position input) to the destination input by the occupant using the navigation HMI 52. The first map information 54 is information that represents the shape of the road by, for example, representing road segments and nodes connected by the segments. The first map information 54 may also include road curvature, POI (Point of Interest) information, and the like. The route on the map is output to the MPU 60. The navigation device 50 may also provide route guidance using the navigation HMI 52 based on the route on the map. The navigation device 50 may also be implemented as a function of a terminal device such as a smartphone or tablet computer held by the passenger. The navigation device 50 may also transmit the current location and destination to a navigation server via the communication device 20, and obtain a route equivalent to the route on the map from the navigation server.

[0047] The MPU 60, for example, includes a recommended lane determination unit 61, which stores second map information 62 in a storage device such as a HDD or flash memory. The recommended lane determination unit 61 divides the route on the map provided by the navigation device 50 into multiple blocks (e.g., every 100 meters in the vehicle's direction of travel) and determines a recommended lane for each block by referring to the second map information 62. The recommended lane determination unit 61 determines the lane to travel on from the left. If the route on the map branches, the recommended lane determination unit 61 determines the recommended lane so that the vehicle can travel on a reasonable route to the branch destination.

[0048] The second map information 62 is higher-precision map information than the first map information 54. The second map information 62 includes, for example, information about lane centers or lane boundaries. Furthermore, the second map information 62 may include road information, traffic restriction information, address information (address, postal code), facility information, phone number information, and information about prohibited sections where Mode A or Mode B, described later, is prohibited. The second map information 62 can be updated at any time by communicating with other devices via the communication device 20.

[0049] The driver monitoring camera 70 is, for example, a digital camera utilizing a solid-state imaging element such as a CCD or CMOS. The driver monitoring camera 70 is mounted at any location within the vehicle in a position and orientation capable of capturing a frontal image (in an orientation that captures the face) of the head of a passenger (hereinafter referred to as the driver) seated in the driver's seat of the vehicle. For example, the driver monitoring camera 70 is mounted above a display device located in the center of the vehicle's instrument panel.

[0050] The driving operating parts 80 include, for example, an accelerator pedal, a brake pedal, a shift lever, and other operating parts in addition to the steering wheel 82. A sensor that detects the amount of operation or the presence or absence of operation is installed in the driving operating parts 80, and the detection results are output to the automatic driving control device 100, or part or all of the driving drive force output device 200, the braking device 210, and the steering device 220. The steering wheel 82 is an example of an "operating part that receives the steering operation performed by the driver." The operating part does not necessarily need to be annular, and can also be in the form of a special-shaped steering wheel, a joystick, a button, etc. A steering wheel grip sensor 84 is installed on the steering wheel 82. The steering wheel grip sensor 84 is implemented by an electrostatic capacitance sensor, etc., and outputs a signal to the automatic driving control device 100 that can detect whether the driver is gripping the steering wheel 82 (that is, contacting it in a state where force can be applied).

[0051] The automatic driving control device 100 includes, for example, a first control unit 120 and a second control unit 160. The first control unit 120 and the second control unit 160 are respectively implemented by executing a program (software) by a hardware processor such as a CPU (Central Processing Unit). In addition, some or all of these components can be implemented by hardware (including circuitry) such as LSI (Large Scale Integration), ASIC (Application Specific Integrated Circuit), FPGA (Field-Programmable Gate Array), GPU (Graphics Processing Unit), or by the collaboration of software and hardware. The program can be pre-stored in a storage device (a storage device having a non-transitory storage medium) such as an HDD or flash memory of the automatic driving control device 100, or can be stored in a removable storage medium such as a DVD or CD-ROM, and installed in the HDD or flash memory of the automatic driving control device 100 by assembling the storage medium (non-transitory storage medium) in a drive device.

[0052] Figure 2 This is a functional structure diagram of the first control unit 120 and the second control unit 160. The first control unit 120 includes, for example, an identification unit 130, an action plan generation unit 140, and a mode determination unit 150. The first control unit 120 implements, for example, functions based on AI (Artificial Intelligence) and functions based on a pre-given model in parallel. For example, the function of "identifying intersections" can be achieved by executing the recognition of intersections based on deep learning and the like in parallel with the recognition based on pre-given conditions (the presence of signals, road signs, etc. that can be pattern-matched), and scoring both for comprehensive evaluation. In this way, the reliability of autonomous driving can be ensured.

[0053] The recognition unit 130 recognizes the position, velocity, acceleration, and other states of objects in the vicinity of the vehicle based on information input from the camera 10, radar device 12, and LIDAR 14 via the object recognition device 16. The position of the object is, for example, recognized as a position on an absolute coordinate with a representative point of the vehicle (center of gravity, drive shaft center, etc.) as the origin and used for control. The position of the object can be represented by a representative point such as the center of gravity or a corner of the object, or by an area. The "state" of the object can also include the acceleration, jerk, or "action state" of the object (for example, whether a lane change is being made or is about to be made).

[0054] In addition, the recognition unit 130, for example, recognizes the lane in which the vehicle is traveling (driving lane). For example, the recognition unit 130 recognizes the driving lane by comparing the pattern of road dividing lines obtained from the second map information 62 (for example, the arrangement of solid lines and dotted lines) with the pattern of road dividing lines around the vehicle recognized based on the image captured by the camera 10. It should be noted that the recognition unit 130 is not limited to recognizing road dividing lines, but can also recognize the driving lane by recognizing the boundaries of the driving road (road boundaries) including road dividing lines, shoulders, curbs, central median strips, guardrails, etc. In this recognition, the position of the vehicle obtained from the navigation device 50 and the processing results processed by the INS can also be added. In addition, the recognition unit 130 recognizes temporary stop lines, obstacles, red lights, toll booths, and other road phenomena.

[0055] When identifying a lane, the recognition unit 130 identifies the position and posture of the vehicle relative to the lane. For example, the recognition unit 130 may identify the deviation of the vehicle's reference point from the center of the lane and the angle formed by the vehicle's travel direction with respect to a line connecting the lane centers as the vehicle's relative position and posture relative to the lane. Alternatively, the recognition unit 130 may identify the position of the vehicle's reference point relative to any side edge of the lane (road dividing line or road boundary) as the vehicle's relative position relative to the lane.

[0056] The action plan generation unit 140 generates a target trajectory for the vehicle to automatically (independent of the driver's operation) travel in the future, in a manner that allows the vehicle to travel on the recommended lane determined by the recommended lane determination unit 61 in principle and to cope with the surrounding conditions of the vehicle. The target trajectory includes, for example, a speed element. For example, the target trajectory is represented by a trajectory in which the locations (track points) that the vehicle should arrive at are arranged in order. Track points are locations that the vehicle should arrive at at predetermined driving distances (for example, a few [m]) along the way. Different from this, target speeds and target accelerations at predetermined sampling times (for example, a few tenths [sec]) are generated as part of the target trajectory. In addition, track points can also be positions that the vehicle should arrive at at the sampling moment at predetermined sampling times. In this case, the information on the target speed and target acceleration is represented by the intervals between track points.

[0057] When generating a target trajectory, the action plan generator 140 can set an autonomous driving event. These events include constant speed driving, low-speed following, lane change, diverging, merging, and takeover. The action plan generator 140 generates a target trajectory corresponding to the activated event.

[0058] The mode determination unit 150 determines the driving mode of the host vehicle to be one of a plurality of driving modes that assign different tasks to the driver. The mode determination unit 150 includes, for example, a driver state determination unit 152 and a mode change processing unit 154. The individual functions of these units will be described later.

[0059] Figure 3 : This is a diagram showing an example of the correspondence between the driving mode and the control state and task of the vehicle. Among the driving modes of the vehicle, there are five modes, for example, mode A to mode E. With regard to the control state, that is, the degree of automation of the driving control of the vehicle, mode A is the highest, followed by mode B, mode C, and mode D in order, with mode E being the lowest. On the contrary, with regard to the tasks assigned to the driver, mode A is the lightest, followed by mode B, mode C, and mode D in order, with mode E being the heaviest. It should be noted that in modes D and E, since they are control states that are not automatic driving, the automatic driving control device 100 has the responsibility to end the control involved in automatic driving until it is transferred to driving assistance or manual driving. The following is an example of the contents of each driving mode.

[0060] In mode A, the vehicle is in an automatic driving state, and the driver is not required to monitor the front or control the steering wheel 82 (steering control in the figure). However, even in mode A, the driver is required to be able to quickly switch to a manual driving posture according to the requirements from the system centered on the automatic driving control device 100. It should be noted that the automatic driving mentioned here means that steering and acceleration and deceleration are controlled independently of the driver's operation. The front refers to the space in the direction of travel of the vehicle that can be visually identified through the windshield. Mode A is a driving mode that can be executed when the following conditions are met, and is sometimes called TJP (Traffic Jam Pilot). The condition is that the vehicle is traveling at a specified speed (for example, about 50 [km / h]) or less on a motor vehicle-only road such as an expressway, and there is a preceding vehicle to be followed. If this condition is not met, the mode determination unit 150 changes the driving mode of the vehicle to mode B.

[0061] In mode B, the state is set to driving support, and the driver is assigned the task of monitoring the front of the vehicle (hereinafter referred to as front monitoring), but is not assigned the task of holding the steering wheel 82. In mode C, the state is set to driving support, and the driver is assigned the task of monitoring the front and the task of holding the steering wheel 82. Mode D is a driving mode that requires a certain degree of driving operation by the driver with respect to at least one of the steering and acceleration and deceleration of the vehicle. For example, in mode D, driving support such as ACC (Adaptive Cruise Control) and LKAS (Lane Keeping Assist System) is performed. In mode E, the state is set to manual driving, in which driving operations by the driver are required for both steering and acceleration and deceleration. Of course, in both modes D and E, the driver is assigned the task of monitoring the front of the vehicle.

[0062] The automatic driving control device 100 (and the driving support device (not shown)) performs automatic lane changes according to the driving mode. In the automatic lane change, there are automatic lane changes based on system requirements (1) and automatic lane changes based on driver requirements (2). In the automatic lane change (1), there are automatic lane changes for overtaking when the speed of the preceding vehicle is greater than a certain threshold compared to the speed of the own vehicle, and automatic lane changes for traveling toward the destination (automatic lane changes caused by the recommended lane being changed). Automatic lane change (2) is to change the lane of the own vehicle in the operating direction when the driver operates the direction indicator when conditions such as speed and positional relationship with surrounding vehicles are met.

[0063] In mode A, the automatic driving control device 100 does not execute either automatic lane change (1) or (2). In modes B and C, the automatic driving control device 100 executes both automatic lane change (1) and (2). In mode D, the driving support device (not shown) does not execute automatic lane change (1) but executes automatic lane change (2). In mode E, neither automatic lane change (1) nor (2) is executed.

[0064] If the driver does not perform the task associated with the determined driving mode (hereinafter referred to as the current driving mode), the mode determination unit 150 changes the driving mode of the host vehicle to a driving mode with a higher task load.

[0065] For example, in Mode A, if the driver is in a physical position that prevents them from switching to manual driving in response to a system request (e.g., continuously looking around outside the permitted area or detecting signs of driving difficulty), the mode determination unit 150 controls the vehicle using the HMI 30 to urge the driver to switch to manual driving. If the driver does not respond, the vehicle is brought to a gradual stop near the roadside, thereby discontinuing automated driving. After discontinuing automated driving, the vehicle enters Mode D or E, where it can be manually started by the driver. The same applies to "discontinuing automated driving" below. In Mode B, if the driver is not monitoring the road ahead, the mode determination unit 150 controls the vehicle using the HMI 30 to urge the driver to monitor the road ahead. If the driver does not respond, the vehicle is brought to a gradual stop near the roadside, thereby discontinuing automated driving. In Mode C, if the driver is not monitoring the road ahead or is not gripping the steering wheel 82, the mode determination unit 150 controls the vehicle using the HMI 30 to urge the driver to monitor the road ahead and / or grip the steering wheel 82. If the driver does not respond, the vehicle is brought to a gradual stop near the roadside, thereby discontinuing automated driving.

[0066] The driver state determination unit 152 monitors the driver's state for the aforementioned mode change and determines whether the driver's state is appropriate for the task. For example, the driver state determination unit 152 analyzes images captured by the driver monitoring camera 70 and performs posture estimation processing to determine whether the driver's body posture is not suitable for transitioning to manual driving in response to the system's request. Furthermore, the driver state determination unit 152 analyzes images captured by the driver monitoring camera 70 and performs line of sight estimation processing to determine whether the driver is looking ahead.

[0067] The mode change processing unit 154 performs various processes for mode change. For example, the mode change processing unit 154 instructs the action plan generation unit 140 to generate a target trajectory for roadside stop, instructs a driving support device (not shown) to operate, and controls the HMI 30 to urge the driver to take action.

[0068] The second control unit 160 controls the travel driving force output device 200 , the braking device 210 , and the steering device 220 so that the host vehicle passes through the target trajectory generated by the action plan generating unit 140 at a predetermined timing.

[0069] return Figure 2, the second control unit 160 includes, for example, an acquisition unit 162, a speed control unit 164, and a steering control unit 166. The acquisition unit 162 acquires the information of the target track (track point) generated by the action plan generation unit 140, and stores it in a memory (not shown). The speed control unit 164 controls the driving force output device 200 or the braking device 210 based on the speed element attached to the target track stored in the memory. The steering control unit 166 controls the steering device 220 according to the curvature of the target track stored in the memory. The processing of the speed control unit 164 and the steering control unit 166 is implemented, for example, by a combination of feedforward control and feedback control. As an example, the steering control unit 166 performs a combination of feedforward control corresponding to the curvature of the road in front of the vehicle and feedback control based on the deviation from the target track.

[0070] The driving force output device 200 outputs the driving force (torque) used to propel the vehicle to the drive wheels. The driving force output device 200 includes, for example, a combination of an internal combustion engine, an electric motor, and a transmission, as well as an ECU (Electronic Control Unit) that controls them. The ECU controls the aforementioned components based on information input from the second control unit 160 or from the driving control element 80.

[0071] The brake device 210 includes, for example, a brake caliper, a hydraulic cylinder that transmits hydraulic pressure to the brake caliper, an electric motor that generates hydraulic pressure in the hydraulic cylinder, and a brake ECU. The brake ECU controls the electric motor according to information input from the second control unit 160 or information input from the driving operating element 80, so that a braking torque corresponding to the braking operation is output to each wheel. The brake device 210 may include a mechanism that transmits the hydraulic pressure generated by operating the brake pedal included in the driving operating element 80 to the hydraulic cylinder via the master hydraulic cylinder as a backup. It should be noted that the brake device 210 is not limited to the structure described above, and may also be an electronically controlled hydraulic brake device that controls the actuator according to information input from the second control unit 160 to transmit the hydraulic pressure of the master hydraulic cylinder to the hydraulic cylinder.

[0072] The steering system 220 includes, for example, a steering ECU and an electric motor. The electric motor applies force to, for example, a rack-and-pinion mechanism to change the direction of the steered wheels. The steering ECU drives the electric motor based on information input from the second control unit 160 or from the driving operating element 80 to change the direction of the steered wheels.

[0073] The heater device 300 is a device that blows warm air to the front window to remove water droplets adhering to the front window. In this embodiment, the heater device 300 is configured to start or stop the air supply in addition to the driver's operation, and can also start or stop the air supply in accordance with the instruction of the object recognition device 16.

[0074] Figure 4 This is a block diagram showing an example of a structure that the object recognition device 16 of the present embodiment has for detecting obstructing factors. The object recognition device 16, for example, includes a storage unit 610 and a control unit 620. The control unit 620 is implemented by, for example, a hardware processor such as a CPU executing a program (software). In addition, some or all of these components can be implemented by hardware (including a circuit unit; circuitry) such as an LSI, ASIC, FPGA, GPU, etc., or can be implemented by the collaboration of software and hardware. The program can be pre-stored in a storage device such as an HDD, a flash memory, etc. of the automatic driving control device 100 (a storage device having a non-temporary storage medium), or can be stored in a removable storage medium such as a DVD, CD-ROM, etc., and installed in the HDD, flash memory, etc. of the automatic driving control device 100 by assembling the storage medium (non-temporary storage medium) in the drive device.

[0075] The storage unit 610 is implemented, for example, by an HDD, a flash memory, an EEPROM (Electrically Erasable Programmable Read Only Memory), a ROM (Read Only Memory), or a RAM (Random Access Memory). The storage unit 610 stores information including, for example, references and thresholds used in the object recognition device 16's process of detecting an obstacle.

[0076] The control unit 620 includes, for example, a first feature point extraction unit 621, a second feature point extraction unit 622, a feature point overlapping unit 623, an obstruction factor determination unit 624, and a heater control unit 625. The first feature point extraction unit 621 extracts feature points of an object detected by the camera 10 as first feature points based on the object detection result of the camera 10. Similarly, the second feature point extraction unit 622 extracts feature points of an object detected by the radar device 12 as second feature points based on the object detection result of the radar device 12. It should be noted that the second feature point extraction unit 622 may also extract feature points of an object detected by the LIDAR 14 as second feature points based on the object detection result of the LIDAR 14.

[0077] Here, the camera 10 is an example of a "first object detection device" that detects objects in front of the vehicle based on electromagnetic wave reception results, while the radar device 12 and the LIDAR 14 are examples of a "second object detection device." Furthermore, the radar device 12 and the LIDAR 14 are positioned and positioned so as to encompass part or all of the camera 10's detection range. The first feature point extraction unit 621 and the second feature point extraction unit 622 respectively output information representing the extracted first and second feature point groups to the feature point superposition unit 623.

[0078] The feature point overlapping unit 623 overlaps the first and second feature point groups in the same coordinate space to extract groups of feature points representing the same object (hereinafter referred to as "corresponding feature points"). Furthermore, the feature point overlapping unit 623 detects feature points in the first or second feature point groups that are not extracted as corresponding feature points as "non-corresponding feature points." Specifically, a non-corresponding feature point is a feature point that belongs to one of the first and second feature point groups but has no corresponding feature point in the other feature point group.

[0079] The obstruction factor determination unit 624 determines the presence of an obstruction to object detection by the camera 10 and the radar device 12 based on the detection results of the non-corresponding feature points detected by the feature point overlapping unit 623. Specifically, if the feature point overlapping unit 623 detects a non-corresponding feature point, the obstruction factor determination unit 624 determines that an obstruction exists for the object detection mechanism of the feature point group in the first feature point group and the second feature point group that did not detect the non-corresponding feature point.

[0080] Figure 5 1 is a diagram showing an example of a case where the obstacle determination unit 624 determines that “there is no obstacle” for any of the detection mechanisms. Figure 5 Indicates that at time t 10 and t 11 (>t 10 )’s detection results from the camera 10 and the radar device 12. Figure 5 The detection results are shown when the host vehicle is traveling in a lane R1 divided by road dividing lines L1 and L2 in a direction from the front side of the drawing toward the back side of the drawing. Vehicle M1 in the figure is traveling ahead of the host vehicle.

[0081] In this example, the obstacle determination unit 624 at time t 10 and t 11At any given moment, the same object (roadside tree T1 on the left side of lane R1) is detected in both the detection results from camera 10 and radar device 12. In this case, the same feature points are extracted from roadside tree T1 by camera 10 and radar device 12, so the feature point overlap unit 623 does not detect any non-corresponding feature points. Therefore, in this case, the obstruction factor determination unit 624 determines that there is no obstruction for both the camera 10 and radar device 12 detection mechanisms.

[0082] In contrast, Figure 6 1 is a diagram showing an example of a case where the obstacle determination unit 624 determines that “an obstacle exists” with respect to object detection by the camera 10 . Figure 6 Indicates time t 20 and t 21 (>t 20 )’s detection results from the camera 10 and the radar device 12. Figure 6 Indicates that the vehicle is Figure 5 The results of object detection performed under the same driving conditions. The obstacle W1 in the figure is, for example, water droplets adhering to the front window.

[0083] In this example, at time t 20 , the water drop W1 does not block the view of the roadside tree T1, so at this time, the obstruction factor determination unit 624 determines that there is no obstruction factor for any of the detection mechanisms of the camera 10 and the radar device 12. 21 Water droplets W1 obstruct the view of roadside tree T1. Therefore, the feature point group F1 of roadside tree T1 extracted in the detection results by camera 10 represents the feature points of the portion not obstructed by water droplets W1. Meanwhile, radar device 12 detects the entire roadside tree T1. Therefore, feature point group F3 of roadside tree T1's feature point group F2 extracted in the detection results by radar device 12, which does not correspond to feature point group F1, represents non-corresponding feature points. Therefore, in this case, obstruction factor determination unit 624 determines that "obstruction factor exists" in object detection by camera 10.

[0084] Heater control unit 625 controls heater device 300 based on the presence or absence of an obstruction determined by obstruction determination unit 624. Specifically, heater control unit 625 operates heater device 300 if obstruction determination unit 624 determines that an obstruction is present. This allows the obstruction to be removed if the detected obstruction is water droplets adhering to the front window.

[0085] [Method for determining impeding factors]

[0086] Figure 7This flowchart illustrates an example of the process of detecting non-corresponding feature points (hereinafter referred to as "non-corresponding feature point detection process") performed by the object recognition device 16 of this embodiment. First, the feature point superimposition unit 623 obtains information on the first feature point group from the first feature point extraction unit 621 and obtains information on the second feature point group from the second feature point extraction unit 622 (step S101).

[0087] Next, the feature point overlapping unit 623 selects a feature point from the first feature point group (step S102), and determines whether a corresponding feature point exists in the second feature point group for the selected first feature point (step S103). If, for the selected first feature point, it is determined that a corresponding feature point does not exist in the second feature point group, the feature point overlapping unit 623 records the selected first feature point as a non-corresponding feature point (step S104), and the process proceeds to step S105. On the other hand, if, in step S103, it is determined that a corresponding feature point exists in the second feature point group for the selected first feature point, the feature point overlapping unit 623 skips step S104 and proceeds to step S105.

[0088] Next, the feature point overlap unit 623 determines whether all feature points included in the first feature point group have been checked for corresponding feature points (step S105). If it is determined that there are unchecked feature points in the first feature point group, the feature point overlap unit 623 returns the process to step S102 and repeats steps S102 to S104 until there are no unchecked feature points. On the other hand, if it is determined in step S105 that all feature points included in the first feature point group have been checked for corresponding feature points, the feature point overlap unit 623 proceeds to step S106.

[0089] Next, the feature point overlapping unit 623 selects a feature point from the second feature point group (step S106), and determines whether a corresponding feature point exists in the first feature point group for the selected second feature point (step S107). If, for the selected second feature point, it is determined that a corresponding feature point does not exist in the first feature point group, the feature point overlapping unit 623 records the selected second feature point as a non-corresponding feature point (step S108), and the process proceeds to step S109. On the other hand, if, in step S107, it is determined that a corresponding feature point exists in the first feature point group for the selected second feature point, the feature point overlapping unit 623 skips step S108 and proceeds to step S109.

[0090] Next, the feature point overlap unit 623 determines whether all feature points included in the second feature point group have been checked for corresponding feature points (step S109). If it is determined that there are unchecked feature points in the second feature point group, the feature point overlap unit 623 returns to step S106 and repeats steps S106 to S108 until there are no unchecked feature points. On the other hand, if it is determined in step S109 that all feature points included in the second feature point group have been checked for corresponding feature points, the feature point overlap unit 623 terminates the non-corresponding feature point detection process.

[0091] It should be noted that Figure 7 The non-corresponding feature point detection process described in the preceding section is a process performed on each detection result obtained in time series by the camera 10 and the radar device 12. The feature point superposition unit 623 performs a feature point superposition on the detection results obtained by the camera 10 and the radar device 12 at each time point. Figure 7 The non-corresponding feature point detection process is performed, and the coordinates of the detected non-corresponding feature points are recorded by establishing a corresponding relationship with the detection mechanism (in this embodiment, the camera 10 or the radar device 12). In this way, by cumulatively recording the detection information of the non-corresponding feature points, the obstacle factor determination unit 624 can count the number of non-corresponding feature points generated during the specified period, and can determine whether there are obstacles based on the number of non-corresponding feature points generated during the specified period. Specifically, the obstacle factor determination unit 624 can perform the following Figure 8 The obstacle determination process shown is used to detect whether there is an obstacle.

[0092] In addition, Figure 7 In the example of , the feature point overlapping unit 623 performs the inspection of the second feature point after the inspection of the first feature point. However, either inspection of the feature point may be performed first, or both inspections may be performed in parallel.

[0093] Figure 8This is a flowchart showing an example of the process of the obstruction factor determination processing performed by the object recognition device 16 of the present embodiment. First, the obstruction factor determination unit 624 obtains the coordinate information of all the non-corresponding feature points detected by the camera 10 and the radar device 12 from the feature point overlapping unit 623 (step S201). Next, the obstruction factor determination unit 624 selects the non-corresponding feature points of the processing object from the non-corresponding feature point group for which the coordinate information has been obtained (step S202), and for the selected non-corresponding feature points, the coordinates of the corresponding feature points that should have been detected (hereinafter referred to as "missing feature points") are recorded on the recording surface corresponding to the detection mechanism (step S203). Here, the recording surface is a memory area for recording the existence of the missing feature points and the positions that should have been detected, and is ensured for each detection mechanism of the camera 10 and the radar device 12.

[0094] Next, the obstacle determination unit 624 totals the number of missing feature points recorded on the recording surface of each detection mechanism, for each partial area set within the detection range, and determines whether there is a partial area where the total number is greater than a threshold value (step S204). If it is determined that there is a partial area where the total number of missing feature points is greater than the threshold value, the obstacle determination unit 624 determines that "there is an obstacle" with respect to the detection mechanism that is supposed to detect the missing feature points in that partial area (step S205), thereby terminating the obstacle determination process. On the other hand, if it is determined in step S204 that there is no partial area where the total number of missing feature points is greater than the threshold value, the obstacle determination unit 624 determines that there is no obstacle (step S206), thereby terminating the obstacle determination process.

[0095] Figure 9 It is an image diagram showing a specific example of the obstacle determination process. Here, for example, it is assumed that frames representing the detection results are input to the object recognition device 16 at a predetermined time interval in the order of N (N represents an integer greater than 1), N+1, and N-+-2 for the camera 10 and the radar device 12, respectively. Hereinafter, the frames input in the order of N, N+1, and N+2 will be referred to as N frames, N+1 frames, and N-+-2 frames, respectively, and each frame represents the detection results obtained by the camera 10 and the radar device 12 for the common detection area. In addition, Figure 9 The example shows a case where the camera 10 captures the obstacle B2 during the detection of the N frame, the N+1 frame, and the N+2 frame.

[0096] In this case, the obstruction factor determination unit 624 reserves recording surfaces M1 and M2 for the camera 10 and radar device 12, respectively, corresponding to the detection range common to each detection mechanism, as recording surfaces for recording missing feature points. Next, the obstruction factor determination unit 624 records the corresponding missing feature point on the recording surface M1 of the camera 10 for each second feature point determined to be a non-corresponding feature point in the second feature point group extracted based on the object detection results of the radar device 12.

[0097] exist Figure 9 In the example, there is an obstacle B2 in the object detection of the camera 10. Therefore, in the second feature point group, non-corresponding feature points F11-1, F11-2, F12-1 to F12-3, and F13 are detected at the position A1 corresponding to the obstacle B2, and the corresponding missing feature points D11-1, D11-2, D12-1 to D12-3, and D13 are recorded on the recording surface M1. Figure 9 Recording plane M1 shown shows the result of recording missing feature points for frames N, N+1, and N+2. In recording plane M1, missing feature points D11-1 and D11-2 correspond to non-corresponding feature points F11-1 and F11-2 detected in frame N of radar device 12. Furthermore, missing feature points D12-1 through D12-3 correspond to non-corresponding feature points F12-1 through F12-3 detected in frame N+1 of radar device 12. Furthermore, missing feature point D13 corresponds to non-corresponding feature point F13 detected in frame N+2 of radar device 12.

[0098] In this case, the obstruction factor determination unit 624 may also determine that an obstruction factor exists in the fourth area (partial area indicated by (4) in the figure) and the seventh area (partial area indicated by (7) in the figure) of the camera 10 where the missing feature points are recorded between the N frame and the N+2 frame. In addition, the obstruction factor determination unit 624 may also determine that an obstruction factor exists in the area where the missing feature points are detected at the same position for more than a specified time. For example, when the missing feature points D11-1 and D11-2 detected in the N frame are continuously detected in the N+1 frame and the N+2 frame, the obstruction factor determination unit 624 may determine that an obstruction factor exists in the object detection in the fourth area and the seventh area where the missing feature points D11-1 and D11-2 are recorded.

[0099] Alternatively, the obstacle determination unit 624 may sum the number of missing feature points detected in time series for each partial area within the detection range, and determine that an obstacle exists in the object detection of the partial area where the sum is greater than a threshold. For example, Figure 9This shows an example of a situation where nine partial areas, namely the first to the ninth, are set within the detection range (areas indicated by (1) to (9) in the figure). It should be noted that Figure 9 The method of dividing the partial regions in is just an example, and the number, size, shape, etc. of the partial regions can be determined arbitrarily.

[0100] In the example of recording surface M1, in the fourth region, one missing feature point D11-1 is recorded during the processing of frame N, one missing feature point D12-1 is further recorded during the processing of frame N+1, and one missing feature point D13 is further recorded during the processing of frame N+2. In this case, if the threshold is 2, the obstruction factor determination unit 624 determines that an obstruction exists during object detection in the fourth region if the number of missing feature points in the region is two or more during the processing of frames N+1 and N+2. If the threshold is 3, the obstruction factor determination unit 624 determines that an obstruction exists if the number of missing feature points in the region is three or more during the processing of frame N+2.

[0101] On the other hand, in the seventh area of the recording surface M1, one missing feature point D11-2 is recorded at the processing time of frame N, two missing feature points D12-2 and D12-3 are further recorded at the processing time of frame N+1, and no new missing feature points are recorded at the processing time of frame N+2. In this case, the number of missing feature points in the area becomes 2 or 3 or more at the processing time of frame N+1. Figure 9 In the example of , when the threshold value is 2 or 3, the obstacle determination unit 624 determines that an obstacle exists at the processing timing of the N+1 frame during object detection in the seventh region.

[0102] The object recognition device 16 of the embodiment thus configured includes a first feature point extraction unit 621 that extracts feature points of the detected object as first feature points based on the detection results of the camera 10; a second feature point extraction unit 622 that extracts feature points of the detected object as second feature points based on the detection results of the radar device 12; and an obstruction factor determination unit 624 that determines the presence of an obstruction factor for the detection of the object in the object detection device that has obtained detection results related to the extraction of a feature point group corresponding to a missing feature point in a first feature point group or a second feature point group (i.e., a feature point group with a missing feature point). Furthermore, with this configuration, the object recognition device 16 of the embodiment can compare the detection results of objects by multiple object detection devices, determine the presence of an obstruction factor based on the difference in the detection results, and identify the location of the obstruction factor. Therefore, the object recognition device 16 of the embodiment can more reliably suppress a decrease in the accuracy of object recognition around the vehicle.

[0103] It should be noted that in the above embodiment, the case where water droplets adhering to the front window are assumed as an example of an obstructing factor, and the heater device 300 is operated as an example of a means for removing the water droplets, is described. However, the means for removing the obstructing factor is not limited to the heater device 300. For example, if dirt adhering to the outer side of the front window is assumed as an obstructing factor, a wiper device (not shown) may be operated as an example of a means for removing the dirt.

[0104] Furthermore, in the above embodiment, for simplicity, the missing feature points are recorded on a two-dimensional recording surface. However, the missing feature points may also be recorded on a three-dimensional space corresponding to the detection range (hereinafter referred to as the "recording space"). In this case, the blocking factor determination unit 624 may be configured to divide the three-dimensional recording space into partial spaces and determine the presence of blocking factors based on the occurrence of missing feature points in each partial space.

[0105] The above-described embodiment can also be expressed as follows.

[0106] A detection device, comprising: a storage device for storing a program and a hardware processor,

[0107] The hardware processor performs the following processing by executing the program:

[0108] Based on a detection result of a first object detection device for detecting an object in front of the vehicle, feature points of the object detected by the first object detection device are extracted as first feature points.

[0109] extracting feature points of the object as second feature points based on a detection result of a second object detection device for detecting the object and having a detection range that includes a part or all of the detection range of the first object detection device;

[0110] An object detection device that obtains detection results related to the extraction of a feature point group in which corresponding feature points are missing from the point group of the first feature points, i.e., the first feature point group, and the point group of the second feature points, i.e., the second feature point group, determines that there is an obstacle to the detection of the object.

[0111] While specific embodiments of the present invention have been described above, the present invention is not limited to these embodiments at all, and various modifications and substitutions can be made without departing from the spirit of the present invention.

Claims

1. A detection device, wherein: The detection device comprises: a first feature point extraction unit for extracting, based on a detection result of a first object detection device for detecting an object in front of the host vehicle, feature points of the object detected by the first object detection device as first feature points; a second feature point extraction unit configured to extract feature points of the object as second feature points based on a detection result of a second object detection device for detecting the object and having a detection range that includes a part or all of the detection range of the first object detection device; as well as a determination unit for determining that there is an obstacle to detection of the object in the object detection device that has obtained a detection result related to extraction of a feature point group of a first feature point group and a feature point group of a second feature point group that lacks a corresponding feature point, in the first object detection device and the second object detection device; The determination unit detects the feature points that should have been detected, i.e., the missing feature points, in a time series for the first feature point or the second feature point that does not have a corresponding feature point, and totals the number of the missing feature points detected for each partial area into which the detection range is divided. When the total number of the missing feature points is greater than a threshold value, it is determined that the obstacle exists in the partial area where the total number is obtained.

2. The detection device according to claim 1, wherein: The determination unit determines that the obstruction factor exists when the first feature point or the second feature point having no corresponding feature point is detected at the same position continuously for a predetermined time or longer.

3. The detection device according to claim 1 or 2, wherein: The first object detection device is a camera that detects objects in front of the vehicle through the front window from inside the vehicle. The factor hindering the object detection by the first object detection device is water droplets attached to the front window.

4. The detection device according to claim 3, wherein: The second object detection device is a radar device located behind a radio wave transmission portion provided on a portion of the bumper. The factor hindering the object detection by the second object detection device is an object attached to the radio wave transmitting portion.

5. The detection device according to claim 3, wherein: The second object detection device is a laser radar device installed on the bumper. The factor hindering the object detection by the second object detection device is an attachment relative to the light emitting part or the light receiving part of the laser radar device.

6. The detection device according to claim 3, wherein: The detection device further includes a heater control unit configured to operate a heater device that blows warm air toward the front window based on a determination result of the determination unit. The heater control unit operates the heater device when the determination unit determines that the obstacle exists in object detection by the first object detection device.

7. A detection method, wherein: The detection method enables the computer to perform the following processing: extracting, based on a detection result of a first object detection device that detects an object in front of the vehicle, feature points of the object detected by the first object detection device as first feature points; extracting feature points of the object as second feature points based on a detection result of a second object detection device for detecting the object and having a detection range that includes a part or all of the detection range of the first object detection device; For the object detection device in the first object detection device and the second object detection device that has obtained detection results related to extraction of a feature point group that lacks corresponding feature points in the first feature point group and the second feature point group, determining that there is an obstacle to detection of the object; as well as For the first feature point or the second feature point that does not have a corresponding feature point, the feature points that should have been detected, i.e., the missing feature points, are detected in a time series, and the number of the missing feature points detected is totaled for each partial area into which the detection range is divided. When the total number of the missing feature points is above a threshold, it is determined that the obstacle exists in the partial area where the total number is obtained.

8. A storage medium storing a program, wherein: The program causes the computer to perform the following processing: extracting, based on a detection result of a first object detection device that detects an object in front of the vehicle, feature points of the object detected by the first object detection device as first feature points; extracting feature points of the object as second feature points based on a detection result of a second object detection device for detecting the object and having a detection range that includes a part or all of the detection range of the first object detection device; For the object detection device in the first object detection device and the second object detection device that has obtained detection results related to extraction of a feature point group that lacks corresponding feature points in the first feature point group and the second feature point group, determining that there is an obstacle to detection of the object; as well as For the first feature point or the second feature point that does not have a corresponding feature point, the feature points that should have been detected, i.e., the missing feature points, are detected in a time series, and the number of the missing feature points detected is totaled for each partial area into which the detection range is divided. When the total number of the missing feature points is above a threshold, it is determined that the obstacle exists in the partial area where the total number is obtained.

Images