Pavement marking detection device, notification system provided with the device, and pavement marking detection method

Abstract

A road marking detection device is applied to a vehicle including a first camera that captures an image of at least a region in front of the vehicle, and a second camera that captures an image of at least a region in front of the vehicle and is capable of capturing a road surface region closer to the vehicle than a road surface region of the image captured by the first camera. The road marking detection device includes a processor that performs a process of detecting, by the second camera, a road marking corresponding to a detected temporary stop sign when the temporary stop sign in front of the vehicle is detected by the first camera.

Description

Technical Field

[0001] This disclosure relates to a road marking detection device for use on vehicles, a notification system equipped with the device, and a road marking detection method. Background Technology

[0002] Japanese Unexamined Patent Application Publication No. 2007-257305A discloses a vehicle exterior recognition device. This device calculates the distance between a vehicle and a stop line in front of the vehicle, as well as the distance between the vehicle and a temporary stop sign, based on images captured by a camera. To improve the reliability of the distance detection results, the device uses one calculated distance to determine whether the other calculated distance is appropriate. Summary of the Invention

[0003] The technology described in JP 2007-257305 A has the following problem when using a camera to detect the stop line corresponding to a temporary stop sign: When there is a vehicle in front of the vehicle, at the moment the stop line falls into the camera's field of view, the stop line may be obscured by the vehicle in front, and the camera may not be able to detect it. Subsequently, as the vehicle approaches the stop line further, even if the stop line is no longer obscured by the vehicle in front, it may still be outside the camera's field of view. As a result, there is a possibility that the camera may not be able to detect the stop line.

[0004] The present invention was made in view of the above-mentioned problems, and the object of the present invention is to reliably detect road markings even when there is a vehicle ahead when the vehicle (this vehicle) approaches a road marking such as a stop line.

[0005] The road marking detection device according to this disclosure is applied to a vehicle, the vehicle including: a first camera that captures images of at least an area in front of the vehicle; and a second camera that captures images of at least an area in front of the vehicle and is capable of capturing images of a road area closer to the vehicle than the road area captured by the first camera, wherein the road marking detection device includes a processor that performs processing by the second camera to detect road markings corresponding to the detected temporary stop sign when the first camera detects a temporary stop sign in front of the vehicle.

[0006] A notification system according to one aspect of this disclosure includes the road marking detection device, notification device, and processor described above. The notification device executes a notification to attract the attention of a vehicle's driver. The processor controls the notification device to execute a notification to attract the driver's attention regarding a temporary stop of the vehicle when the distance between the detected road marking and the vehicle is less than or equal to a distance threshold and the vehicle's speed is greater than or equal to a speed threshold.

[0007] According to another aspect of this disclosure, a notification system includes the road marking detection device, notification device, interior camera, and processor described above. The notification device executes a notification to attract the attention of the vehicle's driver. The interior camera is mounted inside the vehicle and captures an image of the driver. When the processor detects, after detecting a road marking, that the driver is not looking carefully ahead based on the image from the interior camera, it controls the notification device to execute a notification to attract the driver's attention to the temporary stopping of the vehicle.

[0008] The road marking detection method of this disclosure is applied to a vehicle, which includes a first camera and a second camera. The first camera captures images of at least a region in front of the vehicle, and the second camera captures images of at least a region in front of the vehicle, and is capable of capturing images of a road area closer to the vehicle than the road area captured by the first camera. The road marking detection method includes: when a temporary stop sign is detected in front of the vehicle by the first camera, the second camera detects a road marking corresponding to the detected temporary stop sign.

[0009] The second camera is capable of capturing images of road areas closer to the vehicle than the images captured by the first camera. According to this disclosure, even when there is a vehicle ahead of the vehicle (this vehicle) approaching a road marking such as a stop line, the second camera can reliably detect the road marking as the vehicle later approaches the road marking further. Attached Figure Description

[0010] The features, advantages, and technical and industrial significance of exemplary embodiments of the present invention will now be described with reference to the accompanying drawings, in which like reference numerals denote like elements, and wherein:

[0011] Figure 1 This is a diagram schematically illustrating an example of the configuration of a vehicle according to an embodiment;

[0012] Figure 2 This is an example diagram showing a vehicle approaching an intersection;

[0013] Figure 3 This is a flowchart illustrating the processes related to preventative safety assistance according to an embodiment;

[0014] Figure 4 This is a diagram schematically illustrating an example of the configuration of a vehicle according to a variant of an embodiment; and

[0015] Figure 5 This is a flowchart illustrating a modified example of a process related to preventative safety assistance according to an embodiment. Detailed Implementation

[0016] Embodiments of this disclosure will now be described with reference to the accompanying drawings. However, when referring to the number, quantity, amount, range, etc., of each element in the embodiments shown below, the technical concept of this disclosure is not limited to the number mentioned herein unless expressly stated in principle or explicitly specified by such number.

[0017] 1. Vehicle Configuration Example

[0018] Figure 1 This is a schematic diagram illustrating an example of the configuration of a vehicle 1 according to an embodiment. The vehicle 1 includes an Electronic Control Unit (ECU) 10, a first camera 20, a second camera 22, and a notification device 30.

[0019] ECU 10 is a computer that controls vehicle 1. ECU 10 includes one or more processors (hereinafter referred to as processors) 12 and one or more storage devices (hereinafter referred to as storage devices) 14. Processor 12 performs various processes. For example, processor 12 includes a Central Processing Unit (CPU). Storage device 14 stores various information required for the processes performed by processor 12. Examples of storage devices 14 include volatile memory, non-volatile memory, hard disk drive (HDD), and solid state drive (SSD). Processor 12 executes various programs to implement the various processes performed by processor 12. These programs are stored in storage device 14. Multiple ECUs 10 can be configured.

[0020] Vehicle 1 also includes a vehicle speed sensor 16 for detecting vehicle speed V (vehicle 1 speed).

[0021] The first camera 20 and the second camera 22 each capture images of the area in front of the vehicle 1. According to this disclosure, each of the "first camera" and "second camera" can capture images of at least the area in front of the vehicle, and can also capture images of areas in other directions (e.g., side areas) in addition to the area in front of the vehicle 1.

[0022] The first camera 20 is mounted, for example, on vehicle 1 to identify various targets, such as road signs (e.g., Figure 2The temporary stop sign 4 shown, vehicles in front of vehicle 1 (vehicles ahead, oncoming vehicles, etc.), and pedestrians, etc. The first camera 20 is installed, for example, inside the passenger compartment of vehicle 1 (e.g., behind the interior rearview mirror) and captures images of the area in front of vehicle 1 through the windshield. The "first camera" according to this disclosure can be mounted, for example, on the roof of vehicle 1.

[0023] For example, the second camera 22 is one of a plurality of cameras. Each camera is arranged in front, rear, right, or left of the vehicle 1 for the basic purpose of assisting the driver in confirming the surroundings of the vehicle 1 when the vehicle 1 is traveling at low speeds (such as when the driver stops and starts the vehicle). The second camera 22 is, for example, a wide-angle fisheye camera and is mounted on the front bumper. For the above purposes, the "second camera" according to this disclosure may be, for example, at least one of two cameras located on the right and left sides of the vehicle 1 (e.g., below the rearview mirror of the door for rearward confirmation).

[0024] The second camera 22 is positioned vertically below the first camera 20. Furthermore, the second camera 22 has a viewing angle capable of capturing images from a lower area than those captured by the first camera 20. Therefore, as... Figure 1 As shown, the road surface area R2 in the image captured by the second camera 22 includes a road surface area R2A that is closer to the vehicle 1 than the road surface area R1 in the image captured by the first camera 20.

[0025] The notification device 30 performs a notification to attract the attention of the driver of vehicle 1. The notification device 30 is, for example, a display for visually attracting the driver's attention. The display is, for example, a display mounted on the dashboard of vehicle 1 or a head-up display (HUD) that displays information on the windshield of vehicle 1. In addition, instead of or attached to a display, the notification device 30 may include one or both of a buzzer and a speaker for audibly attracting the driver's attention.

[0026] The notification device 30 is controlled by the ECU 10 (processor 12). Processor 12 also corresponds to an example of a processor included in the "road marking detection device" according to this disclosure. Instead of such an example, the processor included in the road marking detection device may be different from the processor used to control the notification device. Furthermore, in Figure 1 In the example shown, processor 12 and notification device 30 correspond to an example of a "notification system" according to this disclosure.

[0027] 2. Preventative safety aids

[0028] Figure 2This diagram shows an example of a vehicle 1 approaching an intersection 2. A temporary stop sign 4 is installed on one side of the travel path 3 near the intersection 2. A stop line 6 and a "stop" mark 7 are drawn on the travel path 3 as road markings 5 ​​corresponding to the temporary stop sign 4.

[0029] As a preventative safety aid for vehicle 1, processor 12 executes a notification as needed to draw the driver's attention to vehicle 1's temporary stop at intersection 2. Specifically, to draw the driver's attention, processor 12 first performs processing to detect the temporary stop sign 4 using image recognition from the first camera 20. Next, processor 12 attempts to detect road markings 5 ​​using image recognition from the first camera 20.

[0030] However, the first camera 20 has a viewing angle suitable for recognizing various targets, including road signs such as temporary stop sign 4, vehicles in front of vehicle 1, and pedestrians, and is mounted in a position suitable for recognizing these targets. Therefore, when the road marking 5 is in the road surface area R1 (see...) of the first camera 20... Figure 1 When road marking 5 is obscured by a vehicle in front, road marking 5 may be outside the field of view of the first camera 20 as vehicle 1 approaches further.

[0031] Therefore, in this embodiment, when the first camera 20 detects a temporary stop sign 4 in front of the vehicle 1, the processor 12 performs the processing of the road marking 5 corresponding to the detected temporary stop sign 4, which is detected by the second camera 22.

[0032] Figure 3 This is a flowchart illustrating the processes related to preventative safety assistance according to an embodiment. The processes in this flowchart are repeatedly executed while vehicle 1 is in motion.

[0033] exist Figure 3 In step S100, the ECU 10 (processor 12) determines whether the temporary stop sign 4 has been detected using image recognition from the first camera 20. If the temporary stop sign 4 is not detected, the process proceeds to return.

[0034] On the other hand, when the temporary stop sign 4 is detected in step S100, the process proceeds to step S102. In step S102, the processor 12 determines whether the road marking 5 has been detected using image recognition from the first camera 20. More specifically, for example, the road marking 5 to be detected by the process in step S102 may be both the stop line 6 and the "stop" sign 7. The road marking 5 to be detected may be only one of the stop line 6 and the "stop" sign 7. This also applies to the road marking 5 to be detected by the second camera 22 in the following step S104.

[0035] When the determination result in step S102 is yes (i.e., when the road marking 5 is detected by the first camera 20), the process proceeds to step S106. On the other hand, when the determination result is no (i.e., when the first camera 20 does not detect the road marking 5), the process proceeds to step S104.

[0036] In step S104, the processor 12 determines whether the road marking 5 has been detected using image recognition from the second camera 22. If the determination in step S104 is negative (i.e., when neither the first camera 20 nor the second camera 22 detects the road marking 5), the process proceeds to the return step. If the determination in step S104 is positive (i.e., when the road marking 5 is detected by the second camera 22), the process proceeds to step S106.

[0037] In step S106, the processor 12 determines whether the distance D between the road marking 5 detected by the first camera 20 or the second camera 22 and the vehicle 1 is less than or equal to a predetermined distance threshold THd. When the road marking 5 is detected by the first camera 20, the distance D can be calculated geometrically, for example, based on the ground clearance of the first camera 20 and the downward angle of the road marking 5 relative to the first camera 20. The same applies when the road marking 5 is also detected by the second camera 22.

[0038] Each of the distance threshold THd and the speed threshold THv, as described below, is a predetermined value that allows determination of whether the driver intends to stop based on the distance D and the vehicle speed V. The distance threshold THd is, for example, 5 m. The speed threshold THv is, for example, 15 kph.

[0039] If the determination result in step S106 is negative (distance D > distance threshold THd), the process proceeds to return. That is, the notification executed by the notification device 30 is not performed. As a result, it is possible to suppress excessive notifications that may annoy the driver. On the other hand, if the determination result is positive (distance D ≤ distance threshold THd), the process proceeds to step S108.

[0040] In step S108, the processor 12 determines whether the vehicle speed V is greater than or equal to a predetermined speed threshold THv. If the determination result is negative, and the vehicle 1 is approaching the road marking 5 until the distance D becomes below the distance threshold THd, since the vehicle speed V decreases below the speed threshold THv, it is determined that the driver intends to stop. Therefore, the process proceeds to return. That is, the notification executed by the notification device 30 is not executed. As a result, it is possible to suppress excessive notifications that may annoy the driver.

[0041] On the other hand, when the determination result in step S108 is yes (vehicle speed V ≥ speed threshold THv), the process proceeds to step S110. When the vehicle speed V is higher than or equal to the speed threshold THv, the process still proceeds to step S110 even if the distance D is less than or equal to the distance threshold THd. In this case, it is determined that the driver has no intention to stop. Therefore, in step S110, the processor 12 controls the notification device 30 to execute a notification to draw the driver's attention to the temporary stopping of the vehicle. As a result, the notification can be executed in appropriate scenarios.

[0042] The distance threshold THd and speed threshold THv used in steps S106 and S108, respectively, can be changed, for example, based on the driver's operation of the Human Machine Interface (HMI) device (e.g., a touch panel). Specifically, for example, the distance threshold THd and speed threshold THv can be changed according to the driver's operation of the HMI device to change the sensitivity of the notification from the notification device 30 at any level (e.g., low, medium, and high).

[0043] 3. Effects

[0044] As described above, according to this embodiment, similarly to the first camera 20, the second camera 22 is also used to detect road markings 5 ​​as needed. The second camera 22 can capture images of the road surface area R1 that is closer to the road surface area R2A of the vehicle 1 than the images captured by the first camera 20 (see...). Figure 1 Therefore, even if the first camera 20 cannot detect the road marking 5 due to the presence of a vehicle ahead when vehicle 1 (this vehicle) approaches a road marking 5 such as a stop line 6, the second camera 22 can still detect the road marking 5. Furthermore, even when vehicle 1 is approaching the road marking 5, the road marking 5 can be placed within the field of view of the second camera 22. Therefore, using the "road marking detection device" according to this embodiment, road markings such as stop lines 6 can be reliably detected.

[0045] Furthermore, according to this embodiment, as described above, since a second camera 22 is used in addition to the first camera 20 to increase the chance of detecting the road marking 5, the chance of determining the driver's intention to stop can be increased. Therefore, the notification device 30 can be used to more effectively attract the driver's attention.

[0046] 4. Variations of notifications executed by a notification device

[0047] The notification executed by the notification device 30 can be executed as follows.

[0048] Figure 4This diagram schematically illustrates an example configuration of a modified vehicle 8 according to an embodiment. In addition to the additionally provided inward-facing camera 24, vehicle 8 is configured to... Figure 1 The vehicle 1 shown is configured in the same way. An inward-facing camera 24 is mounted inside the vehicle 8 and captures images of the driver.

[0049] Figure 5 This is a flowchart illustrating a modified example of a process related to preventative safety assistance according to an embodiment. Except for the process of performing step S200 instead of steps S106 and S108, the processes in this flowchart are similar to... Figure 3 The process is the same as the flowchart shown.

[0050] exist Figure 5 If the determination result in step S102 or step S104 is yes, the process proceeds to step S200. In step S200, the processor 12 acquires driver information based on the image from the inward-facing camera 24, such as gaze, degree of eye opening, facial orientation, and posture. Based on the driver information, the processor 12 determines whether the driver is not looking forward attentively, for example, whether the driver is looking elsewhere.

[0051] If the determination result of step S200 is negative (i.e., when the processor 12 does not detect that the driver is not looking ahead attentively), the process proceeds to return. That is, the notification executed by the notification device 30 is not executed. As a result, it is possible to suppress excessive notifications that could annoy the driver.

[0052] On the other hand, if the determination result in step S200 is yes (i.e., when the processor 12 detects that the driver is not looking ahead seriously), the process proceeds to step S110, and the notification device 30 executes the notification. As a result, the notification can be executed in appropriate scenarios.

Claims

1. A road marking detection device, applied to a vehicle, said vehicle comprising: A first camera captures images of at least the area in front of the vehicle; as well as The second camera captures images of at least the area in front of the vehicle, and is capable of capturing images of the road surface area closer to the vehicle than those captured by the first camera. The road marking detection device includes a processor, wherein the processor: When the first camera detects a temporary stop sign in front of the vehicle, and the first camera does not detect a road marking corresponding to the detected temporary stop sign, the process of detecting the road marking by the second camera is executed.

2. Notification system, including: The road marking detection device according to claim 1; A notification device that performs a notification to draw the attention of the driver of the vehicle; as well as The processor controls the notification device to execute a notification to draw the driver's attention to the temporary stopping of the vehicle when the distance between the detected road marking and the vehicle is less than or equal to a distance threshold and the vehicle speed is greater than or equal to a speed threshold.

3. Notification system, including: The road marking detection device according to claim 1; A notification device that performs a notification to draw the attention of the driver of the vehicle; as well as An inward-facing camera, mounted inside the vehicle, captures images of the driver. as well as The processor, upon detecting the road markings and then detecting, based on the image from the inward-facing camera, that the driver is not looking seriously ahead, controls the notification device to issue a notification to draw the driver's attention to the temporary stopping of the vehicle.

4. A road marking detection method applied to a vehicle, the vehicle including a first camera and a second camera, the first camera capturing images of at least a region in front of the vehicle, the second camera capturing images of at least a region in front of the vehicle, and capable of capturing images of a road area closer to the vehicle than the road area captured by the first camera, the road marking detection method comprising: When the first camera detects a temporary stop sign in front of the vehicle, and the first camera does not detect a road marking corresponding to the detected temporary stop sign, the second camera detects the road marking.

Images