Object marker detection device and vehicle equipped with the same

Abstract

The present invention relates to an object detection device and a vehicle equipped with the same. The object detection device identifies stationary objects within the detection range of an external sensor based on map information. The device then compares the image of the stationary object detected by the external sensor with the stationary object identified from the map information to determine whether the image of the stationary object detected by the external sensor contains an undetected area. If an undetected area is identified, the device identifies undetectable objects between the stationary object and the vehicle.

Description

Technical Field

[0001] The present invention relates to an object marker detection device mounted on a vehicle and a vehicle equipped with the object marker detection device. Background Art

[0002] Various methods for accurately distinguishing objects around a vehicle have been proposed. For example, Japanese Patent Application Laid-Open No. 2019-095290 discloses a technology for accurately distinguishing between walls and valid objects. In the prior art described in Japanese Patent Application Laid-Open No. 2019-095290, when a point group (first point group) longer than a reference length is measured by LiDAR, the point group is classified as a wall. In addition, a second point group other than the first point group classified as a wall is classified as an object candidate. Furthermore, when there are multiple first point groups representing a single wall and a specified condition is satisfied between the endpoints of the first point group and the endpoints of the second point group, the second point group is determined to be a valid object.

[0003] However, external sensors, including LiDAR, have conditions that make it difficult to detect objects due to their characteristics. If object detection by external sensors is not possible, target object candidates cannot be obtained. Summary of the Invention

[0004] The present invention provides an object detection device capable of identifying objects around a vehicle even under conditions where object detection by external sensors is difficult, and also provides a vehicle equipped with the object detection device.

[0005] The object detection device of the present invention is a vehicle-mounted object detection device comprising: an external sensor for acquiring information related to the vehicle's external conditions; a storage device storing map information; and an information processing device. The information processing device processes the information acquired by the external sensor and the map information. The processing performed by the information processing device includes a first process, a second process, and a third process. In the first process, the information processing device identifies stationary objects within the detection range of the external sensor based on the map information. In the second process, the information processing device compares the image of the stationary object detected by the external sensor with the stationary object identified from the map information to determine whether the image of the stationary object detected by the external sensor contains an undetected area. In the third process, if an undetected area is confirmed, the information processing device identifies undetectable objects between the stationary object and the vehicle.

[0006] According to the object detection device having the above configuration, when there are undetectable objects around the vehicle that cannot be directly detected by external sensors, the undetectable objects can be indirectly identified by using the detection results of stationary objects obtained by the external sensors.

[0007] The external sensor may be a LiDAR. In this case, the information processing device determines whether the image of the stationary object detected by the LiDAR contains no areas where point clusters have not been acquired, i.e., unacquired point cluster areas. In LiDAR, unacquired point cluster areas are the aforementioned undetected areas. If the image of the stationary object contains unacquired point cluster areas and the unacquired point cluster area contains a point cluster with a predetermined number of points or more, the information processing device identifies an undetectable object between the stationary object and the vehicle.

[0008] The external sensor may be a camera. In this case, the information processing device determines whether the image of the stationary object detected by the camera contains no area where image recognition is impossible, i.e., an unrecognizable area. In a camera, an unrecognizable area is the aforementioned undetected area. If the image of the stationary object contains an unrecognizable area and the unrecognizable area corresponds to a pixel group with a predetermined number or more, the information processing device identifies the unrecognizable object as being between the stationary object and the vehicle.

[0009] Furthermore, when a moving object detected by an external sensor is not detected by the external sensor, even though the moving object has not been confirmed to have left the detection range of the external sensor, the information processing device may identify an undetectable object as being within the undetectable range of the external sensor. Thus, even when there are no nearby stationary objects that can be used to identify the undetectable object, a moving object that has moved into the undetectable range, i.e., an undetectable object, can be identified based on the movement of the moving object within the detection range of the external sensor.

[0010] The first vehicle provided by the present invention is a vehicle capable of autonomous driving, equipped with this object detection device. The first vehicle includes a vehicle control device that controls the vehicle based on object markers, including undetectable objects detected by the object detection device. This vehicle not only processes objects detected by external sensors as autonomous driving object markers, but also processes undetectable objects identified by the object detection device as autonomous driving object markers, thereby enabling safer autonomous driving.

[0011] The second vehicle provided by the present invention is an autonomously driven vehicle equipped with the object detection device. The second vehicle includes an alarm device that issues an alarm to the outside of the vehicle during vehicle start-up if the object detection device identifies an undetectable object. This second vehicle, by issuing an alarm for an undetectable object that cannot be detected by external sensors, enables safer vehicle start-up.

[0012] The third vehicle provided by the present invention is equipped with this object detection device. The third vehicle includes a notification device that, when an undetectable object is identified by the object detection device, notifies a person monitoring the vehicle of the presence of the undetectable object. This monitoring person includes a remote support operator who remotely supports an autonomous vehicle via a communication network from a remote location, a remote driver who remotely drives a remotely driven vehicle from a remote location via a communication network, and a driver who directly drives the vehicle while riding in the vehicle. With the third vehicle, the presence of an undetectable object that cannot be detected by external sensors is notified to the monitoring person, allowing the monitoring person to monitor the vehicle while maintaining an eye on their surroundings.

[0013] As described above, the object detection device of the present invention can indirectly identify undetectable objects around a vehicle that cannot be directly detected by external sensors by using detection results of stationary objects obtained by external sensors. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Hereinafter, features, advantages, and technical and industrial significance of exemplary embodiments of the present invention will be described with reference to the accompanying drawings, wherein like reference numerals denote like elements, and wherein:

[0015] Figure 1 FIG. 1 is a diagram showing an example of a vehicle to which the object detection device is applied.

[0016] Figure 2 This is a diagram for explaining a first example in which target object detection by LiDAR is impossible.

[0017] Figure 3 This is a diagram for explaining a second example in which target object detection by LiDAR is impossible.

[0018] Figure 4A This is a diagram for explaining a method of recognizing an undetectable object that cannot be detected by LiDAR by utilizing stationary objects existing around a vehicle.

[0019] Figure 4B This is a diagram for explaining a method of recognizing an undetectable object that cannot be detected by LiDAR by utilizing stationary objects existing around a vehicle.

[0020] Figure 5A This is a diagram for explaining a method of identifying an undetectable object that cannot be detected by LiDAR by using stationary objects existing around a vehicle, and is a diagram showing the effect of the first example.

[0021] Figure 5B This is a diagram for explaining a method of identifying an undetectable object that cannot be detected by LiDAR by using stationary objects existing around a vehicle, and is a diagram showing the effect of the first example.

[0022] Figure 6A This is a diagram for explaining a method of identifying an undetectable object that cannot be detected by LiDAR by using stationary objects existing around a vehicle, and is a diagram showing the effect of the second example.

[0023] Figure 6B This is a diagram for explaining a method of identifying an undetectable object that cannot be detected by LiDAR by using stationary objects existing around a vehicle, and is a diagram showing the effect of the second example.

[0024] Figure 7 This is a block diagram showing the configuration of an autonomous driving vehicle to which the object detection device according to the first embodiment is applied.

[0025] Figure 8 1 is a flowchart showing a process of identifying an undetectable object performed by the object target detection device according to the first embodiment.

[0026] Figure 9A This figure is used to explain a method for identifying an undetectable object that hinders the start of an autonomous driving vehicle.

[0027] Figure 9B This figure is used to explain a method for identifying an undetectable object that hinders the start of an autonomous driving vehicle.

[0028] Figure 10 This is a flowchart showing a starting process of an autonomous vehicle to which the object detection device according to the first embodiment is applied.

[0029] Figure 11 This is a flowchart showing a notification determination procedure when the target object detection device according to the first embodiment is applied to a warning notification to a supervisor.

[0030] Figure 12A This diagram is used to explain a method of identifying an undetectable object existing in the detection range of the LiDAR using time-series data of a moving object detected by the LiDAR, and shows a state in which a moving object is detected by the LiDAR.

[0031] Figure 12BThis diagram is used to explain a method of identifying an undetectable object existing in the detection range of the LiDAR using time-series data of a moving object detected by the LiDAR, and shows a state in which a moving object is detected by the LiDAR.

[0032] Figure 13A This diagram is used to explain a method of identifying an undetectable object existing in the detection range of the LiDAR using time-series data of a moving object detected by the LiDAR, and shows a state in which the moving object becomes undetectable by the LiDAR.

[0033] Figure 13B This diagram is used to explain a method of identifying an undetectable object existing in the detection range of the LiDAR using time-series data of a moving object detected by the LiDAR, and shows a state in which the moving object becomes undetectable by the LiDAR.

[0034] Figure 14 This is a block diagram showing the configuration of an autonomous driving vehicle to which the object detection device according to the second embodiment is applied.

[0035] Figure 15 1 is a flowchart showing a process of identifying an undetectable object by the object target detection device according to the second embodiment.

[0036] Figure 16A This is a diagram for explaining a method of recognizing an undetectable object that cannot be detected by a camera by utilizing stationary objects existing around a vehicle.

[0037] Figure 16B This is a diagram for explaining a method of recognizing an undetectable object that cannot be detected by a camera by utilizing stationary objects existing around a vehicle. DETAILED DESCRIPTION

[0038] In each embodiment described below, the same reference numerals are used for elements shared by the various figures, and repeated descriptions are omitted or simplified. In addition, when numerical values such as the number, quantity, amount, and range of each element are mentioned in the embodiments shown below, the present invention is not limited to the numerical values mentioned, except for cases where it is specifically stated or clearly determined in principle to be such numerical values. In addition, with respect to the structures described in the embodiments shown below, except for cases where it is specifically stated or clearly determined in principle to be such structures, they are not necessarily required in the present invention.

[0039] 1. First Implementation

[0040] 1-1. Overview

[0041] First, use Figures 1 to 6B The outline of the first embodiment is described with reference to FIG.

[0042] exist Figure 1 The figure depicts a situation where two vehicles 2 and 110 are traveling side by side on a road 40. Of the two vehicles 2 and 110, at least vehicle 2 is an autonomous driving vehicle to which the object detection device of the present invention is applied. Vehicle 2 is provided with an external sensor 8 for acquiring information related to the external conditions of vehicle 2. The external sensor 8 is, for example, a LiDAR (Light Detection And Ranging), a camera, a fusion of LiDAR and a camera, or a fusion of these with a millimeter-wave radar. Figure 1 In the example shown, the external sensor 8 is mounted on the roof of the vehicle 2 and senses the side of the vehicle 2 from the roof of the vehicle 2. Figure 1 In the figure, range 100, which fan-shapedly extends from the roof of vehicle 2, schematically represents the detection range of external sensor 8. If vehicle 110 is within detection range 100 of external sensor 8, vehicle 110 is detected as a target object. For example, when vehicle 2 changes lanes, the relative position and relative speed of vehicle 110 traveling in the adjacent lane are reflected in the calculation of vehicle 2's target trajectory.

[0043] However, even if an object exists in the detection range 100 of the external sensor 8, it may not be detected by the external sensor 8 depending on the detection conditions. The situation in which the object cannot be detected varies depending on the type of external sensor 8. Figures 2 to 6B In the description of the outline of the first embodiment, it is assumed that the external sensor 8 is a LiDAR. In the following description of the outline, the external sensor is expressed as LiDAR 8.

[0044] Figure 2 This figure is used to illustrate a first example in which the LiDAR 8 cannot detect an object. When an object exists within the detection range 100 of the LiDAR 8, the laser light emitted from the LiDAR 8 is reflected by the surface of the object. The LiDAR 8 receives the reflected light and obtains point cloud data of the object. However, if Figure 2 As shown, when the distance from object 111 to LiDAR 8 is extremely short, for example, within a few tens of centimeters, it is difficult to distinguish between the point cloud derived from object 111 and the sensor noise of LiDAR 8. The point cloud derived from object 111 is also removed by the sensor noise filtering process. In other words, even if object 111 is very close to vehicle 2 and is within the detection range 100 of LiDAR 8, it may not be directly detected by LiDAR 8.

[0045] Figure 3This figure is used to illustrate a second example in which LiDAR 8 cannot detect an object. As described above, LiDAR 8 receives the reflected light of the irradiated laser and obtains a point cloud of the object. Therefore, it is difficult to obtain a point cloud from an object with low laser reflectivity, such as a black object that absorbs light. For example, assume that white vehicle 112 and black vehicle 113 are traveling in adjacent lanes of vehicle 2. Even if both vehicles 112 and 113 enter the detection range 100 of LiDAR 8, although a point cloud from white vehicle 112 can be obtained, it is difficult to obtain a point cloud from black vehicle 113. In other words, in the case of an object 113 that absorbs laser light, even if it exists within the detection range 100 of LiDAR 8, it may not be directly detected by LiDAR 8.

[0046] As in the first and second examples above, depending on the detection conditions, there may be undetectable objects around the vehicle 2 that cannot be directly detected by the LiDAR 8. Figures 4A to 6B As explained in FIG. 1 , in the first embodiment, the stationary objects existing around the vehicle 2 are used to identify undetectable objects that cannot be detected by the LiDAR 8. Figure 4A 、 Figure 5A as well as Figure 6A is a top view showing the positional relationship between the vehicle 2 and various objects. Figure 4B 、 Figure 5B as well as Figure 6B is with Figure 4A 、 Figure 5A as well as Figure 6A Corresponding output image of LiDAR 8. It should be noted that the output image of LiDAR 8 refers to an image obtained by converting point cloud data into two-dimensional image data.

[0047] like Figure 4A As shown, sometimes there are stationary objects 115 such as walls, guardrails, and fences along the road 40 on the shoulder of the road 40. When the stationary object 115 is within the detection range 100 of the LiDAR 8, a point group 105 from the stationary object 115 is obtained in the LiDAR 8. Generally speaking, when the vehicle 2 is traveling, stationary objects 115 such as walls, guardrails, and fences will not exist very close to the vehicle 2. In addition, generally speaking, the color of such stationary objects 115 is not black enough to absorb laser light. Therefore, as long as the stationary object 115 enters the detection range 100 of the LiDAR 8, as shown in FIG. Figure 4B As shown, the point group 105 capturing the stationary object 115 will be output from the LiDAR 8.

[0048] Figure 5A 、 Figure 5B1 is a diagram showing an example of applying the method of using a stationary object 115 to identify an undetectable object that cannot be detected by the LiDAR 8 to the first example. Figure 5A In the example shown, a stationary object 115 exists along the shoulder of road 40, and an undetectable object 111 exists inside stationary object 115 and very near vehicle 2. In this state, the laser beam is blocked by undetectable object 111, so a point cloud cannot be obtained from the portion of stationary object 115 that is shadowed by undetectable object 111.

[0049] When the LiDAR 8, the stationary object 115, and the undetectable object 111 are Figure 5A In the case of the positional relationship shown in Figure 5B The output image of the LiDAR 8 shown in FIG. Specifically, within the point cloud 105 capturing the stationary object 115, an undetected point cloud region 101 appears where no point cloud has been obtained. Undetected point cloud region 101 is an undetected region where the stationary object 115 has not been detected. Whether undetected point cloud region 101 is an undetected region where the point cloud 105 for the stationary object 115 is not obtained despite the presence of the stationary object 115, or a region where the stationary object 115 does not exist in the first place, can be determined based on map information. In other words, by comparing the image of the stationary object 115 detected by the LiDAR 8 with the stationary object identified from the map information, it can be determined whether the undetected region is included. If an undetected region is identified in the point cloud 105 for the stationary object 115, the undetectable object 111 between the stationary object 115 and the vehicle 2 can be indirectly identified.

[0050] Figure 6A 、 Figure 6B 1 is a diagram showing an example of applying a method of using a stationary object 115 to identify an undetectable object that cannot be detected by the LiDAR 8 to the second example. Figure 6A In the example shown, a stationary object 115 is located along the shoulder of road 40. Two vehicles 112 and 113 are located between vehicle 2 and stationary object 115. As described above, vehicle 113 is a black vehicle that absorbs the laser light, making it difficult to obtain a point cloud. In other words, the object cannot be detected. Because the laser light is blocked by vehicles 112 and 113, a point cloud cannot be obtained from the portion of stationary object 115 that is shadowed by these vehicles.

[0051] When the LiDAR 8, the stationary object 115, and the two vehicles 112 and 113 are Figure 6A In the case of the positional relationship shown in Figure 6BThe output image of the LiDAR 8 shown in FIG. Specifically, within the point cloud 105 capturing the stationary object 115, there appears the point cloud 102 capturing the vehicle 112 and an undetected point cloud region 103 where no point cloud has been captured. Undetected point cloud region 103 represents an undetected region where the stationary object 115 was not detected due to the shadow cast by the black vehicle 113. In this example, by comparing undetected point cloud region 103 with the stationary objects identified from the map information, it can be determined whether the undetected point cloud region 103 represents an undetected region where the point cloud 105 for the stationary object 115 was not captured despite the presence of the stationary object 115. If an undetected region is identified in the point cloud 105 for the stationary object 115, it is possible to indirectly identify the undetectable object 113 between the stationary object 115 and the vehicle 2.

[0052] 1-2. Components of Autonomous Driving Vehicles

[0053] Figure 7 This block diagram shows the configuration of vehicle 2, an autonomous vehicle that employs the object detection device according to the first embodiment. Vehicle 2 includes a vehicle control device 10 for controlling vehicle 2; onboard sensors that input information to vehicle control device 10; actuators 4 that operate in response to signals output from vehicle control device 10; and an alarm device 5 that issues an alarm to the exterior of vehicle 2.

[0054] The on-board sensors include a GPS (Global Positioning System) receiver 6, an internal sensor 7, and an external sensor 8. The GPS receiver 6 measures the current position (e.g., latitude and longitude) of the vehicle 2 by receiving signals from GPS satellites. The internal sensor 7 is a sensor that detects the driving state of the vehicle 2. The internal sensor 7 includes: an inertial measurement unit (IMU) that detects the angles and accelerations of the three axes related to the movement of the vehicle 2; and a vehicle speed measuring device that calculates the vehicle speed based on the rotation speed of the wheels. The external sensor 8 can be, for example, any one of LiDAR, a camera, a fusion of LiDAR and a camera, and a fusion of these with a millimeter-wave radar. Object detection is performed based on the information obtained by the external sensor 8.

[0055] Specifically, actuator 4 includes a steering actuator for steering vehicle 2, a drive actuator for driving vehicle 2, and a brake actuator for braking vehicle 2. Specifically, warning device 5 is a speaker that generates sound, a display device that displays information, or a combination thereof. Warning device 5 uses sound, display, or a combination thereof to encourage pedestrians around vehicle 2 to leave vehicle 2.

[0056] The vehicle control device 10 is an ECU (Electronic Control Unit) having at least one processor 11 and at least one memory 12. The memory 12 includes a main storage device and an auxiliary storage device. The memory 12 stores programs that can be executed by the processor 11 and various data associated with the programs. The processor 11 executes the programs stored in the memory 12, thereby realizing various functions in the vehicle control device 10. The program includes a program for enabling the vehicle control device 10 to function as an information processing device of an object detection device to identify undetectable objects. It should be noted that the ECU constituting the vehicle control device 10 may be a collection of multiple ECUs.

[0057] The data stored in the memory 12 includes map information. Map information is managed by a map database (map DB) 21. The map information managed by the map DB 21 includes, for example, road location information, road shape information (e.g., types of turns and straight sections, curvature of turns), information on intersection junctions, target route information for the vehicle, and information on road structures. Information on road structures includes information on stationary objects such as walls, white lines, poles, billboards, signs, guardrails, and fences that can be acquired by the external sensor 8. The map DB 21 is pre-stored in an auxiliary storage device such as an SSD or HDD. However, map information can be downloaded from an external server via the Internet, or referenced from map information on an external server.

[0058] The vehicle control device 10 includes a self-position estimating unit 22, a stationary object identifying unit 23, an object detecting unit 24, an undetectable object identifying unit 25, a driving plan generating unit 26, and a driving control unit 27 as components related to vehicle control during parking. These components are implemented as functions of the vehicle control device 10 when the processor 11 executes a program stored in the memory 12.

[0059] The own position estimating unit 22 estimates the position of the vehicle 2 on the map based on the position information of the vehicle 2 received by the GPS receiver 6, the information related to the driving state of the vehicle 2 detected by the internal sensor 7, and the map information obtained from the map database 21. The information related to the driving state includes, for example, vehicle speed information, acceleration information, and yaw rate information. Furthermore, the own position estimating unit 22 can also estimate the position of the vehicle 2 based on the relative position of a feature detected by the external sensor 8 relative to the vehicle 2, the information related to the driving state of the vehicle 2 detected by the internal sensor 7, and the position of the detected feature on the map.

[0060] The stationary object recognition unit 23 queries the map database 21 for the position of the vehicle 2 estimated by the vehicle position estimation unit 22, and identifies stationary objects within the detection range of the external sensor 8. Specifically, the detection range of each of the multiple external sensors 8 provided on the vehicle 2 is pre-registered. The stationary object recognition unit 23 determines whether a stationary object is within the detection range of the external sensor 8 based on map information. If a stationary object is within the detection range, the stationary object's position on the map is obtained.

[0061] The object detection unit 24 uses pattern matching, deep learning and other methods to detect moving objects around the vehicle 2 based on the information received from the external sensor 8, and determines the location of the moving object. If the external sensor 8 is a LiDAR, the information received from the external sensor 8 is point group data. If the external sensor 8 is a camera, the information received from the external sensor 8 is image data. The moving objects detected by the object detection unit 24 include vehicles, motorcycles, bicycles, pedestrians, animals, etc. However, even if there is a moving object within the detection range of the external sensor 8, as described in the overview item, the moving object may not be detected depending on the detection conditions. Moving objects that cannot be detected by the external sensor 8 are identified by the undetectable object recognition unit 25 described below.

[0062] The undetectable object recognition unit 25 uses the information received from the external sensor 8, the position information of the stationary object recognized from the map information, and the position information of the moving object detected by the object detection unit 24 to recognize the undetectable object existing in the detection range of the external sensor 8. Specifically, Figure 5A 、 Figure 5B and Figure 6A 、 Figure 6B As described above, the undetectable object identification unit 25 compares the image of the stationary object detected by the external sensor 8 with the stationary object identified from the map information. When the external sensor 8 is a LiDAR, the image of the stationary object is an image obtained by converting point cloud data into two-dimensional image data.

[0063] Through this comparison, the undetectable object recognition unit 25 determines whether the image of the stationary object detected by the external sensor 8 contains an undetected area. If an undetected area is found, the undetectable object recognition unit 25 identifies the undetectable object between the stationary object and the vehicle 2. The undetectable object identified by the undetectable object recognition unit 25 is detected as a target object along with the moving object detected by the object detection unit 24.

[0064] The driving plan generation unit 26 obtains the target route recorded in the map database 21, the position of vehicle 2 identified by the vehicle position estimation unit 22, the position of the mobile object detected by the object detection unit 24, and the position of the undetectable object identified by the undetectable object identification unit 25. The driving plan generation unit 26 generates a driving plan along the pre-set target route based on at least the object information including the position of the mobile object and the position of the undetectable object, as well as the map information in the map database 21. The driving plan generation unit 26 preferably generates the driving plan assuming a plurality of coordinate coordinates (p, v) consisting of two elements: a target position p in a coordinate system fixed to vehicle 2 and a velocity v for each target point. If the driving plan generation unit 26 detects an object that interferes with the driving plan of vehicle 2, it updates the driving plan to avoid collision with the object by steering or reducing speed.

[0065] The driving control unit 27 automatically controls the driving of the vehicle 2 based on the driving plan generated by the driving plan generating unit 26. The driving control unit 27 outputs a control signal corresponding to the driving plan to the actuator 4. In this way, the driving control unit 27 controls the driving of the vehicle 2 so that the vehicle 2 automatically drives according to the driving plan.

[0066] 1-3. Unable to detect the object recognition process

[0067] Next, use Figure 8 The recognition process of the undetectable object in the first embodiment is described. Figure 8 In the flowchart, the process of identifying an undetectable object by the vehicle control device 10 as the object detection device of the first embodiment is shown. Figure 8 In the description of the recognition process of the undetectable object, it is assumed that the external sensor 8 is a LiDAR. In the description of the recognition process below, the external sensor is expressed as LiDAR 8.

[0068] according to Figure 8 As shown in the flowchart, the vehicle control device 10 estimates the vehicle 2's own position on the map based on information acquired by the GPS receiver 6, internal sensor 7, and external sensor 8, and map information obtained from the map database 21 (step S101). Next, the vehicle control device 10 detects moving objects around the vehicle 2 based on the information acquired by the external sensor 8 (step S102). Furthermore, based on the vehicle's own position estimated in step S101, the vehicle control device 10 identifies stationary objects within the detection range of the LiDAR 8 based on the map information from the map database 21 (step S103).

[0069] Next, the vehicle control device 10 determines whether the image of the stationary object detected by the LiDAR 8 contains an undetected area (step S104). More specifically, the vehicle control device 10 determines whether the image of the stationary object detected by the LiDAR 8 contains an unobtained point group area where the point group is not obtained. In addition, when the unobtained point group area is equivalent to a point group with a prescribed number of points or more, the vehicle control device 10 determines the unobtained point group area as an undetected area where the stationary object is not detected. The area of the unobtained point group area is proportional to the size of the undetectable object and inversely proportional to the distance between the vehicle 2 and the undetectable object. In order to be able to identify extremely nearby objects more accurately, the prescribed number of points is set to a small value as long as it can be distinguished from noise.

[0070] It should be noted that areas where point clusters are not acquired can also be caused by packet loss in in-vehicle communications. Whether these areas are caused by packet loss can be verified by checking the packet reception interval. For example, the LiDAR8 typically transmits data at 10ms intervals, but sometimes it takes 30ms to receive the data. In this case, verifying packet loss in the 20ms preceding the data reception time is sufficient.

[0071] If the image of the stationary object detected by the LiDAR 8 does not include an undetected area, then there is no undetectable object within the detection range of the LiDAR 8. Therefore, if the result of the determination in step S104 is negative, the recognition process according to the flowchart ends.

[0072] If the result of the determination in step S104 is positive, the vehicle control device 10 determines whether or not the undetected area where no stationary object is detected does not include a point group of a moving object (step S105). Figure 6B As shown in the example, the vehicle control device 10 determines whether all undetected areas where the point cloud 105 from the stationary object 115 is not obtained are areas where the point cloud 102 from the moving object 112 is obtained or areas where no valid point cloud is obtained, such as the area 103 .

[0073] If the point group of the moving object is included in the undetected area, then there is no undetectable object in the undetected area. Therefore, if the result of the determination in step S105 is negative, the recognition process based on the flowchart ends.

[0074] If the undetected area does not contain a point cluster of a moving object, the vehicle control device 10 identifies an undetectable object between the stationary object and the vehicle (step S106). According to the above process, if there is a stationary object within the detection range of the LiDAR 8, the detection results of the stationary object can be used to indirectly identify the undetectable object, even if the LiDAR 8 cannot directly detect the undetectable object.

[0075] It should be noted that the undetected area can be considered the shadow of an undetectable object, and therefore the shape of the undetected area represents the shape of the undetectable object. Therefore, by applying object recognition methods such as pattern matching and deep learning to the shape of the undetected area, the type of undetectable object can also be identified based on the shape of the undetected area. By incorporating the type of undetectable object in addition to its location in the driving plan, more appropriate autonomous driving can be achieved in response to the external conditions of vehicle 2.

[0076] 1-4. Application to Launch Control for Autonomous Vehicles

[0077] The undetectable object that cannot be detected by the external sensor 8 is an existence that should be paid attention to in order to ensure the safety when the vehicle 2 starts. Figure 9A As shown, when vehicle 2 is a bus and stops at bus stop 41, passenger 116 who has gotten off vehicle 2 may walk very close to vehicle 2. As described above, if the distance between passenger 116 and vehicle 2 is too close, even if passenger 116 enters detection range 100 of external sensor 8, external sensor 8 may not be able to detect passenger 116.

[0078] Although the passenger 116 is near the vehicle 2, the vehicle 2 cannot be started. Therefore, if the vehicle control device 10 identifies an undetectable object around the vehicle 2, it issues an alarm to the outside of the vehicle via the alarm device 5. The vehicle control device 10 then starts the vehicle 2 after confirming that the undetectable object has moved to a position where it does not interfere with the vehicle 2.

[0079] During the start control of the vehicle 2 , the vehicle control device 10 recognizes that the object cannot be detected according to the aforementioned process. Here, it is also assumed that the external sensor 8 is a LiDAR, and in the following description, the external sensor will be referred to as LiDAR 8 .

[0080] exist Figure 9A In the example shown, a passenger 116 stands in front of the vehicle 2, and a stationary object 117 exists behind the passenger 116. In the output image of the LiDAR 8 in this case, as shown in FIG. Figure 9BAs shown, within the point cloud 107 that captures the stationary object 117, an undetected point cloud region 106 appears where no point cloud has been captured. Undetected point cloud region 106 is an undetected region of the point cloud 107 where the stationary object 117 has not been captured, and can be determined based on map information. When an undetected region is identified within the point cloud 107 of the stationary object 117, the vehicle control device 10 identifies an undetectable object 116 located in front of the vehicle 2. Undetectable object 116 is a person (passenger), and can be identified by applying object recognition methods such as pattern matching and deep learning to the shape of the undetected region.

[0081] exist Figure 10 In FIG. 1 , a flowchart is shown of a process of starting the vehicle 2 by the vehicle control device 10 as the object detection device of the first embodiment. In this flowchart, the same processes as those in the process of identifying an undetectable object are marked with the same Figure 8 Hereinafter, for the processing already described in the description of the recognition process of the undetectable object, the repeated description will be omitted or simplified.

[0082] according to Figure 10 In the flowchart shown, if a moving object is detected near vehicle 2 in step S102, the vehicle control device 10 determines whether the distance between the detected moving object and vehicle 2 is less than a predetermined distance (step S201). The predetermined distance used in this determination is set based on the minimum distance that ensures sufficient safety between vehicle 2 and surrounding objects when vehicle 2 is in motion. If the distance between the moving object and vehicle 2 is less than the predetermined distance, the vehicle control device 10 issues an alarm via the alarm device 5, prompting the moving object near vehicle 2 to move away from vehicle 2 (step S202).

[0083] If the distance between the detected moving object and the vehicle 2 is not less than the prescribed distance, the vehicle control device 10 performs the processing of steps S103 and S104. If no moving object is detected in step S102, the process proceeds from step S201 to steps S103 and S104.

[0084] In step S104, a determination is made as to whether the image of the stationary object detected by LiDAR 8 contains an undetected area. If the image of the stationary object detected by LiDAR 8 does not contain an undetected area, it can be determined that no undetectable objects exist within the detection range of LiDAR 8. Furthermore, since the determination result in step S201 is negative, there are no moving objects closer than the specified distance to vehicle 2. Therefore, the vehicle control device 10 determines that there is no problem with starting vehicle 2 and starts vehicle 2 (step S204).

[0085] If the image of the stationary object detected by LiDAR 8 contains an undetected area, the vehicle control device 10 proceeds to step S105. If a point cluster of a mobile object is included in the undetected area where no stationary object is detected, this point cluster is a point cluster of mobile objects at a distance greater than a predetermined distance from vehicle 2. In other words, in this case, there are neither undetectable objects nor mobile objects within a distance that would hinder the start of vehicle 2 around vehicle 2. Therefore, the vehicle control device 10 determines that there is no problem with starting vehicle 2 and starts vehicle 2 (step S204).

[0086] If the undetected area, where no stationary object is detected, does not contain a point cluster of a moving object, the undetected area is formed by an undetectable object between the stationary object and the vehicle. If an undetectable object is detected around vehicle 2, vehicle 2 cannot be safely started. In this case, the vehicle control device 10 issues an alarm via the alarm device 5, prompting the object near vehicle 2 to move away from the vehicle 2 (step S202).

[0087] When an alarm is issued regarding an object near vehicle 2, the vehicle control device 10 determines whether the object near vehicle 2, including the undetectable object, has moved to a position where it does not interfere with vehicle 2, that is, whether it has moved to a position at least a predetermined distance away from vehicle 2 (step S203). The vehicle control device 10 continues issuing an alarm via the alarm device 5 until the object near vehicle 2 moves to a position where it does not interfere with vehicle 2. If it is confirmed that the object near vehicle 2 has moved to a position where it does not interfere with vehicle 2, the vehicle control device 10 determines that there is no problem with starting vehicle 2 and starts vehicle 2 (step S204).

[0088] 1-4. Application of warning notification to vehicle observers

[0089] In the description so far, the vehicle 2 to which the object detection device of the first embodiment is applied is an autonomous driving vehicle that drives autonomously. However, the object detection device of the first embodiment can also be used for autonomous driving vehicles that are remotely supported from a remote location via a communication network, remotely driven vehicles that are remotely driven from a remote location via a communication network, and vehicles that are directly driven by a driver. For example, if the object detection device identifies an undetectable object, the presence of the undetectable object can be notified to the vehicle's monitor to prompt vigilance. The monitor referred to here is a remote support operator who remotely supports the autonomous driving vehicle, a remote driver who remotely drives the remotely driven vehicle, or a driver who is on board the vehicle and directly drives the vehicle.

[0090] exist Figure 11In the flowchart, the process of notification judgment in the case of applying the object detection device of the first embodiment to the warning notification to the monitor is shown. In this flowchart, the same processing as the processing in the recognition process of the undetectable object is marked with Figure 10 Hereinafter, for the processing already described in the description of the recognition process of the undetectable object, the repeated description will be omitted or simplified.

[0091] according to Figure 11 In the flowchart shown, when a moving object is detected around vehicle 2 in step S102, it is determined whether the distance between the detected moving object and vehicle 2 is less than a specified distance (step S301). The specified distance used in this determination is the minimum distance that can fully ensure safety between vehicle 2 and objects around the vehicle when vehicle 2 starts. When the distance between the moving object and vehicle 2 is less than the specified distance, the observer of vehicle 2 is prompted to be alert to the surroundings via a notification device (step S302). As the notification device, for example, a speaker that notifies with sound, a head-up display that notifies with visual information, an HMI (Human Machine Interface) such as an instrument panel can be used.

[0092] If the distance between the detected moving object and vehicle 2 is not less than the specified distance, steps S103 and S104 are performed. In step S104, a determination is made as to whether the image of the stationary object detected by LiDAR 8 contains an undetected area. If the determination result in step S104 is negative, it can be determined that there are no undetectable objects within the detection range of LiDAR 8. Furthermore, since the determination result in step S301 is negative, there are no moving objects less than the specified distance from vehicle 2. Therefore, in this case, no warning notification is issued to the observer of vehicle 2.

[0093] If the determination result of step S104 is affirmative, the vehicle control device 10 proceeds to step S105. If a point cluster of a mobile object is included in the undetected area where no stationary object is detected, the point cluster is a point cluster of a mobile object that is at least a predetermined distance from vehicle 2. Therefore, if the determination result of step S105 is negative, no warning notification is issued to the observer of vehicle 2.

[0094] If the result of the determination in step S105 is affirmative, an undetectable object that cannot be detected by the LiDAR 8 exists between the stationary object and the vehicle 2. In this case, a warning notification is issued to the person monitoring the vehicle 2 via the HMI to alert the person monitoring the undetectable object around the vehicle 2 (step S302).

[0095] 2. Second Implementation

[0096] 2-1. Overview

[0097] use Figure 12A 、 Figure 12B and Figure 13A 、 Figure 13B The outline of the second embodiment will be described.

[0098] The object detection device of the first embodiment indirectly identifies undetectable objects that cannot be directly detected by external sensors by using the detection results of stationary objects obtained by external sensors. However, depending on the environment in which the vehicle is located, there may be no available stationary objects around the vehicle. The second embodiment is a proposal for a method of identifying undetectable objects in the absence of available stationary objects. It should be noted that when using Figure 12A 、 Figure 12B and Figure 13A 、 Figure 13B In the description of the outline of the second embodiment, it is assumed that the external sensor is LiDAR.

[0099] As described in the first embodiment, when the distance from the LiDAR to the object is extremely short, it is difficult to distinguish between the point group obtained from the object and the sensor noise of the LiDAR. Figure 12A As shown in FIG, if the object 118 is located in the undetectable range 120 at a very close distance from the LiDAR 8, the LiDAR 8 cannot obtain a valid point cloud from the object 118. In addition, if there is no stationary object around the vehicle 2, a point cloud from the stationary object cannot be obtained. Figure 12B As shown, in the output image of the LiDAR 8 , the point cloud unobtainable region 109 where no point cloud can be obtained expands.

[0100] However, the undetectable range 120 is located within the detection range 100 of the LiDAR 8. Therefore, the object 118 does not enter the undetectable range 120 directly, but Figure 13A As shown, the object 118 passes through the area outside the non-detectable range 120 within the detection range 100 and enters the non-detectable range 120. Figure 13A In the case of the position shown, the result will be as follows Figure 13B The output image of the LiDAR 8 is shown. That is, before the object 118 enters the undetectable range 120, the point group 108 corresponding to the object 118 is obtained.

[0101] While the object 118 is detected by the LiDAR 8, the moving speed of the object 118 relative to the vehicle 2 can be estimated based on the movement of the point group 108 in the output image of the LiDAR 8. If the moving speed of the object 118 is estimated, even if Figure 13B Even if the point group shown disappears from the output image of LiDAR 8, the position of object 118 can be estimated. Based on the estimated position of object 118, it can be determined whether object 118 has entered undetectable range 120. In other words, undetectable object 118 existing in undetectable range 120 of LiDAR 8 can be indirectly identified.

[0102] 2-2. Components of Autonomous Driving Vehicles

[0103] Figure 14 This is a block diagram showing the configuration of vehicle 2, an autonomous vehicle employing the object detection device according to the second embodiment. In the second embodiment, the vehicle control device 10 further includes a time series database (time series DB) 29. Information on moving objects detected by the object detection unit 24 is registered in time series in the time series DB 29. The information registered in the time series DB 29 is used by the undetectable object identification unit 25.

[0104] In the second embodiment, the undetectable object identification unit 25 uses the position information of mobile objects detected by the object detection unit 24 and the position and speed information of mobile objects in past frames registered in the time series database 29 to identify undetectable objects within the detection range of the external sensor 8. Specifically, the undetectable object identification unit 25 determines whether a mobile object detected in a past frame and registered in the time series database 29 is also detected by the object detection unit 24 this time. If the mobile object detected in the past frame is not detected this time, the undetectable object identification unit 25 determines whether the mobile object has entered the undetectable range of the external sensor 8 based on the speed of the mobile object registered in the time series database 29. If it is estimated that the mobile object has entered the undetectable range of the external sensor 8, the undetectable object identification unit 25 identifies the undetectable object.

[0105] 2-2. Identification process that cannot detect objects

[0106] Next, use Figure 15 The recognition process of the undetectable object in the second embodiment is described. Figure 15 In FIG. 1 , a flow chart is used to show the process of identifying an undetectable object by the vehicle control device 10 as the object detection device of the second embodiment. Figure 15In the description of the recognition process of the undetectable object, it is assumed that the external sensor 8 is a LiDAR. In the description of the recognition process below, the external sensor is expressed as LiDAR 8.

[0107] according to Figure 15 In the flowchart shown, the vehicle control device 10 detects a moving object around the vehicle 2 based on information acquired by the LiDAR 8 (step S401). The vehicle control device 10 determines whether a moving object is detected in step S401 (step S402). If a moving object is detected, the process proceeds to step S403.

[0108] If a moving object is detected in step S401, the vehicle control device 10 references the time series DB 29 to obtain the position information of the detected moving object in the most recent past frame (step S403). The vehicle control device 10 then estimates the speed of the moving object based on the current position information of the moving object and the position information obtained from the time series DB 29 (step S404), and updates the speed information of the moving object registered in the time series DB 29 (step S405). If the position information of the moving object detected in step S401 is not registered in the time series DB 29, the position information is registered in the time series DB 29 instead of the processing in steps S404 and S405.

[0109] The vehicle control device 10 repeatedly executes the loop from step S401 to step S405 until no moving object is detected in step S401. Then, if no moving object is detected, the vehicle control device 10 proceeds to step S406 to perform the next determination. Figure 15 In the flowchart shown, the process may proceed to step S406 without even detecting a moving object by the LiDAR 8. However, the possibility that a moving object directly enters the undetectable range of the LiDAR 8, that is, the very close range where the LiDAR 8 cannot obtain a point cloud from the object, can be ignored.

[0110] In step S406, the vehicle control device 10 determines whether the moving object has entered the undetectable range of the LiDAR 8. Whether the moving object has entered the undetectable range can be determined based on the moving object information registered in the time series DB 29, specifically, the position and velocity of the immediately preceding moving object that is no longer detected by the LiDAR 8. For example, if the direction of movement of the immediately preceding moving object that is no longer detected is in the direction of the undetectable range, the likelihood that the moving object has entered the undetectable range is high. However, if the direction of movement of the immediately preceding moving object that is no longer detected is outside the detection range of the LiDAR 8, the likelihood that the moving object has moved outside the detection range of the LiDAR 8 is high.

[0111] If a moving object has entered the undetectable range, it cannot be detected by LiDAR 8. Therefore, if it is estimated that the moving object has entered the undetectable range, the vehicle control device 10 identifies the undetectable object within the undetectable range of LiDAR 8 (step S407). According to the above process, if there are no stationary objects around the vehicle 2 that can be used to identify undetectable objects, a moving object that has moved into the undetectable range of LiDAR 8, i.e., an undetectable object, can be identified based on the movement of the moving object within the detection range of LiDAR 8.

[0112] 3. Other Implementation Methods

[0113] In the above embodiments, the examples in which external sensor 8 is a LiDAR have been specifically described. However, in the object detection device according to the present invention, external sensor 8 is not limited to a LiDAR. Specifically, external sensor 8 may be a camera, a combination of a LiDAR and a camera, or a combination that incorporates a millimeter-wave radar.

[0114] For example, Figure 16A 、 Figure 16B This figure is used to explain a method for identifying an undetectable object that cannot be detected by the camera 8 by using the stationary object 115 when the external sensor 8 is a camera. Figure 16A In the case of the positional relationship shown, the object 111 enters the detection range (viewing angle) 130 of the camera 8. However, since the object 111 is located very close to the camera 8, the entire object 111 cannot be included in the detection range 130 of the camera 8.

[0115] In this case, if Figure 16B As shown, in the output image of camera 8, image 121, which captures a portion of object 111, appears within image 125, which captures stationary object 115. However, object 111 cannot be identified through image recognition based on image 121, which captures only a portion of object 111. Therefore, image 121 is treated as an unrecognizable area, where image recognition is not possible, in the processing of the output image of camera 8.

[0116] Whether the image recognition impossible area 121 is an undetected area where the image 125 of the stationary object 115 is not obtained despite the presence of the stationary object 115, or an area where the stationary object 115 does not exist in the first place, can be determined based on the map information. In other words, by comparing the image of the stationary object 115 detected by the camera 8 with the stationary object recognized from the map information, it can be determined whether the undetected area is included.

[0117] Furthermore, whether or not image-unrecognizable region 121 is sensor noise of camera 8 can be determined based on the number of pixels in image-unrecognizable region 121. Specifically, if image-unrecognizable region 121 corresponds to a pixel group having a predetermined number or greater, it can be determined that image-unrecognizable region 121 is not sensor noise but an undetected region where image 125 of stationary object 115 is not obtained.

[0118] When an undetected area is confirmed in the image 125 of the stationary object 115, the undetectable object 111 between the stationary object 115 and the vehicle 2 can be indirectly identified. As described above, even when the external sensor 8 is a camera, the stationary object 115 detected by the camera 8 can be used to identify an undetectable object that cannot be detected by the camera 8.

[0119] It should be noted that when the recognition method for non-detectable objects of the second embodiment is applied to a camera, a stereo camera that measures distance and speed is used. However, even a non-stereoscopic (monaural) camera can be used if it can be used in conjunction with a sensor that can measure the speed of an object.

Claims

1. An object detection device, mounted on a vehicle, characterized in that it comprises: an external sensor for acquiring information related to an external condition of the vehicle; a storage device storing map information; and an information processing device for processing the information acquired by the external sensor and the map information, The information processing device identifies a stationary object existing within a detection range of the external sensor based on the map information, The information processing device compares the image of the stationary object detected by the external sensor with the stationary object identified from the map information to determine whether the image of the stationary object detected by the external sensor includes an undetected area. When the undetected area is confirmed, the information processing device identifies an undetectable object existing between the stationary object and the vehicle. When the moving object detected by the external sensor is not detected by the external sensor even though it is not confirmed that the moving object has gone out of the detection range of the external sensor, the information processing device identifies the undetectable object existing in the undetectable range of the external sensor.

2. The object mark detection device according to claim 1, characterized in that: The external sensor is a lidar, When the image of the stationary object detected by the laser radar contains an unobtained point group area in which the point group is not obtained and the unobtained point group area is equivalent to a point group with a specified number of points or more, the information processing device identifies the undetectable object existing between the stationary object and the vehicle.

3. The object detection device according to claim 1, characterized in that: The external sensor is a camera, When the image of the stationary object detected by the camera contains an unrecognizable area where image recognition cannot be performed and the unrecognizable area is equivalent to a pixel group with a specified number of pixels or more, the information processing device identifies the undetectable object existing between the stationary object and the vehicle.

4. A vehicle equipped with the object marker detection device according to any one of claims 1 to 3 and capable of autonomous driving, wherein: A vehicle control device is provided, which controls the vehicle based on object markers including the undetectable object detected by the object marker detection device.

5. A vehicle equipped with the object marker detection device according to any one of claims 1 to 3 and capable of autonomous driving, wherein: An alarm device is provided for issuing an alarm outside the vehicle when the vehicle starts moving when the object detection device recognizes the undetectable object.

6. A vehicle equipped with the object marker detection device according to any one of claims 1 to 3, wherein: A notification device is provided for notifying a person monitoring the vehicle of the presence of the undetectable object when the undetectable object is recognized by the object marker detection device.

Images