Road marking detection device, notification system including same, and road marking detection method
A dual-camera system for vehicles ensures continuous detection of road markings by using a second camera to overcome obstructions, enhancing safety through reliable road marking detection and driver notification.
Patent Information
- Application Number
- JP2022002396
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2022-01-11
- Publication Date
- 2024-11-13
- Estimated Expiration
- 2042-01-11
AI Technical Summary
Existing vehicle systems struggle to reliably detect road markings like stop lines when obstructed by preceding vehicles, leading to potential missed detections as the markings move out of the camera's field of view.
Employing a dual-camera system where a first camera detects road signs and a second camera, positioned lower and with a wider angle, captures closer road surface areas, ensuring continuous detection of road markings even when obstructed by preceding vehicles.
Ensures reliable detection of road markings and accurate determination of driver intent to stop, reducing the likelihood of missed notifications and enhancing safety by using both cameras to maintain visibility of road markings and driver attention.
Smart Images

Figure 0007582214000001 
Figure 0007582214000002 
Figure 0007582214000003
Abstract
Description
[Technical field]
[0001] The present disclosure relates to a road sign detection device applied to a vehicle, a notification system including the same, and a road sign detection method. [Background technology]
[0002] Patent Document 1 discloses an exterior recognition device. This exterior recognition device calculates the distance to a stop line and a stop sign ahead of the vehicle based on an image captured by a camera. In order to improve the reliability of the distance detection result, the exterior recognition device uses one of the calculated distances as a reference to determine whether the other is appropriate. [Prior art documents] [Patent documents]
[0003] [Patent Document 1] JP 2007-257305 A Summary of the Invention [Problem to be solved by the invention]
[0004] When detecting a stop line corresponding to a stop sign using a camera, the technology described in Patent Document 1 has the following problem. That is, if there is a preceding vehicle ahead of the vehicle, the stop line may be blocked by the preceding vehicle at the timing when the stop line falls within the angle of view of the camera, and the stop line may not be detected by the camera. Then, when the vehicle approaches the stop line even further after that, even if the stop line is no longer blocked by the preceding vehicle, the stop line may be outside the angle of view of the camera. As a result, the stop line may not be detected by the camera.
[0005] The present disclosure has been made in consideration of the above-mentioned problems, and has an objective to enable reliable detection of road markings such as stop lines even when a vehicle (host vehicle) approaches a road marking such as a stop line and a leading vehicle is present. [Means for solving the problem]
[0006] A road sign detection device according to the present disclosure is applied to a vehicle equipped with a first camera that captures an image of at least the area ahead of the vehicle, and a second camera that captures an image of at least the area ahead of the vehicle and is capable of capturing an image of a road surface area closer to the vehicle than the road surface area captured by the first camera. P Equipped with a processor The processor: The first camera detects a stop sign in front of the vehicle. and a process of determining whether the stop sign is detected by the first camera. In this case, the road markings corresponding to the detected stop sign are a process of determining whether the road marking is detected by the first camera, and if the road marking is not detected by the first camera, The process of detecting the object using the second camera and, Run do.
[0007] A notification system according to an aspect of the present disclosure includes the above-described road marking detection device, a notification device, and a processor. The notification device issues a notification to alert a driver of a vehicle. The processor controls the notification device to issue a notification to alert a driver of the vehicle to stop the vehicle when a distance between the detected road marking and the vehicle is equal to or less than a distance threshold and the vehicle speed is equal to or greater than a speed threshold.
[0008] A notification system according to another aspect of the present disclosure includes the above-described road marking detection device, a notification device, an inward-facing camera, and a processor. The notification device issues a notification to alert a driver of the vehicle. The inward-facing camera is mounted inside the vehicle and captures an image of the driver. After a road marking is detected, if the processor detects that the driver is not paying attention to the road ahead based on an image from the inward-facing camera, the processor controls the notification device to issue a notification to alert the driver to stop the vehicle.
[0009] The road marking detection method according to the present disclosure is applied to a vehicle equipped with a first camera that captures at least an image ahead of the vehicle, and a second camera that captures at least an image ahead of the vehicle and is capable of capturing an image of a road surface area closer to the vehicle than the road surface area captured by the first camera. and determining whether the stop sign is detected by the first camera. In this case, the road markings corresponding to the detected stop sign are determining whether the road marking is detected by the first camera; and if the road marking is not detected by the first camera, is detected by the second camera. and, Includes. Effect of the Invention
[0010] The second camera can capture an image of a road surface area closer to the vehicle than the road surface area captured by the first camera. According to the present disclosure, even if a leading vehicle is present when the vehicle (host vehicle) approaches a road surface marking such as a stop line, the second camera can be used to reliably detect the road surface marking when the host vehicle subsequently approaches the road surface marking. [Brief description of the drawings]
[0011] [Figure 1] 1 is a diagram illustrating an example of a configuration of a vehicle according to an embodiment; [Diagram 2] FIG. 1 is a diagram illustrating an example of an intersection where a vehicle approaches. [Diagram 3] 4 is a flowchart showing a process related to preventive safety support according to the embodiment. [Figure 4] FIG. 10 is a diagram illustrating an example of a configuration of a vehicle according to a modified example of the embodiment. [Diagram 5] 13 is a flowchart showing a process related to preventive safety support according to a modified example of the embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0012] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. However, when the number, quantity, amount, range, etc. of each element is mentioned in the following embodiments, the technical idea of the present disclosure is not limited to the mentioned number unless otherwise specified or clearly specified in principle.
[0013] 1. Example of vehicle configuration 1 is a diagram illustrating an example of a configuration of a vehicle 1 according to an embodiment. The vehicle 1 includes an electronic control unit (ECU) 10, a first camera 20, a second camera 22, and an alarm device 30.
[0014] The ECU 10 is a computer that controls the vehicle 1. The ECU 10 includes one or more processors (hereinafter simply referred to as processors) 12 and one or more storage devices (hereinafter simply referred to as storage devices) 14. The processor 12 executes various processes. For example, the processor 12 includes a CPU (Central Processing Unit). The storage device 14 stores various information required for processing by the processor 12. Examples of the storage device 14 include a volatile memory, a non-volatile memory, an HDD (Hard Disk Drive), an SSD (Solid State Drive), etc. The processor 12 executes various programs to realize various processes by the processor 12. The various programs are stored in the storage device 14. It should be noted that there may be multiple ECUs 10.
[0015] The vehicle 1 also includes a vehicle speed sensor 16 that detects the vehicle speed V (the speed of the vehicle 1).
[0016] The first camera 20 and the second camera 22 capture an image in front of the vehicle 1. Note that each of the "first camera" and the "second camera" according to the present disclosure may be configured to capture an image in other directions (e.g., sides) as well as in front of the vehicle 1, as long as it captures an image in at least the front.
[0017] The first camera 20 is mounted on the vehicle 1 to recognize various objects such as road signs (e.g., stop sign 4 shown in FIG. 2), vehicles ahead of the vehicle 1 (leading vehicles and oncoming vehicles, etc.), and pedestrians, for example. The first camera 20 is installed, for example, inside the vehicle 1 (for example, behind the inner mirror) and captures an image of the area ahead of the vehicle 1 through the windshield. Note that the "first camera" according to the present disclosure may be mounted on the roof of the vehicle 1, for example.
[0018] As an example, the second camera 22 is one of a plurality of cameras arranged on the front, rear, left and right sides of the vehicle 1 for the basic purpose of assisting the driver in checking the surroundings of the vehicle 1 when parking, starting, or running at low speed, etc. The second camera 22 is, for example, a wide-angle fisheye camera and is installed in the front bumper. Note that the "second camera" according to the present disclosure may be, for example, at least one of two cameras arranged on the left and right sides of the vehicle 1 (for example, below the door mirror for checking the rear) for the above purpose.
[0019] The second camera 22 is disposed vertically lower than the first camera 20. In addition, the second camera 22 has an angle of view that allows it to capture an image further downward than the first camera 20. For this reason, as shown in FIG. 1, the road surface area R2 captured by the second camera 22 includes a road surface area R2A that is closer to the vehicle 1 than the road surface area R1 captured by the first camera 20.
[0020] The notification device 30 issues a notification to the driver of the vehicle 1 to alert him / her. The notification device 30 is, for example, a display device for visually alerting the driver. The display device is, for example, a display mounted on the instrument panel of the vehicle 1, or a head-up display (HUD) for displaying information on the windshield of the vehicle 1. The notification device 30 may also include, instead of or in addition to the display device, one or both of a buzzer and a speaker for auditorily alerting the driver.
[0021] The notification device 30 is controlled by the ECU 10 (processor 12). The processor 12 also corresponds to an example of a processor included in a "road sign detection device" according to the present disclosure. Alternatively, the processor included in the road sign detection device may be different from the processor that controls the notification device. In the example shown in FIG. 1, the processor 12 and the notification device 30 correspond to an example of an "alert system" according to the present disclosure.
[0022] 2. Preventive safety support 2 is a diagram showing an example of an intersection 2 where a vehicle 1 approaches. A stop sign 4 is installed on the side of a road 3 near the intersection 2. In addition, a stop line 6 and a "Stop" sign 7 are painted on the road 3 as road markings 5 corresponding to the stop sign 4.
[0023] As one of the preventive safety supports for the vehicle 1, the processor 12 executes a notification as necessary to alert the driver to stop at the intersection 2. Specifically, in order to issue this alert, the processor 12 first executes a process of detecting a stop sign 4 by image recognition using the first camera 20. Next, the processor 12 attempts to detect a road marking 5 by image recognition using the first camera 20.
[0024] However, the first camera 20 has an angle of view suitable for recognizing the above-mentioned various objects, such as road signs such as stop signs 4, vehicles ahead of the vehicle 1, and pedestrians, and is mounted at a position suitable for such recognition. For this reason, if the road marking 5 is blocked by the leading vehicle when it is within the road area R1 (see FIG. 1) of the first camera 20, when the vehicle 1 subsequently approaches the road marking 5, the road marking 5 may be outside the angle of view of the first camera 20.
[0025] Therefore, in this embodiment, the processor 12 is configured to execute a process of detecting a road marking 5 corresponding to the detected stop sign 4 by the second camera 22 when a stop sign 4 ahead of the vehicle 1 is detected by the first camera 20.
[0026] 3 is a flowchart showing a process related to the preventive safety support according to the embodiment. The process of this flowchart is repeatedly executed while the vehicle 1 is traveling.
[0027] 3, in step S100, the ECU 10 (processor 12) determines whether or not a stop sign 4 has been detected by image recognition using the first camera 20. As a result, if the stop sign 4 has not been detected, the process proceeds to return.
[0028] On the other hand, if a stop sign 4 is detected in step S100, the process proceeds to step S102. In step S102, the processor 12 determines whether or not a road marking 5 has been detected by image recognition using the first camera 20. More specifically, the road marking 5 to be detected by the process of step S102 is, for example, both a stop line 6 and a "Stop" sign 7. The road marking 5 to be detected may be only one of the stop line 6 and the "Stop" sign 7. The same applies to the road marking 5 to be detected by the second camera 22 in the next step S104.
[0029] If the determination result in step S102 is Yes (i.e., if the road marking 5 is detected by the first camera 20), the process proceeds to step S106. On the other hand, if the determination result is No (i.e., if the road marking 5 is not detected by the first camera 20), the process proceeds to step S104.
[0030] In step S104, the processor 12 determines whether or not a road marking 5 has been detected by image recognition using the second camera 22. As a result, if the determination result in step S104 is No (i.e., if the road marking 5 has not been detected by either the first camera 20 or the second camera 22), the process proceeds to RETURN. On the other hand, if the determination result in step S104 is Yes (i.e., if the road marking 5 has been detected by the second camera 22), the process proceeds to step S106.
[0031] In step S106, the processor 12 determines whether or not the distance D between the road marking 5 detected by the first camera 20 or the second camera 22 and the vehicle 1 is equal to or less than a predetermined distance threshold THd. When the road marking 5 is detected by the first camera 20, the distance D can be calculated geometrically, for example, from the height of the first camera 20 above the ground and the depression angle of the road marking 5 with respect to the first camera 20. The same applies to the case when the road marking 5 is detected by the second camera 22.
[0032] The distance threshold THd and a speed threshold THv described later are values that are determined in advance so that the presence or absence of the driver's intention to stop can be determined based on the distance D and the vehicle speed V. The distance threshold THd is, for example, 5 m. The speed threshold THv is, for example, 15 kph.
[0033] If the determination result in step S106 is No (distance D>distance threshold THd), the process proceeds to RETURN. That is, no notification is made by the notification device 30. This makes it possible to prevent the driver from feeling annoyed by excessive notifications. On the other hand, if the determination result is Yes (distance D≦distance threshold THd), the process proceeds to step S108.
[0034] In step S108, the processor 12 judges whether the vehicle speed V is equal to or greater than a predetermined speed threshold THv. If the result of this judgment is No, the vehicle speed V has dropped to or below the speed threshold THv in a situation in which the vehicle 1 is approaching the road marking 5 until the distance D becomes equal to or less than the distance threshold THd, and therefore it is judged that the driver has an intention to stop. Therefore, the process proceeds to RETURN. That is, the notification device 30 does not issue a notification. This makes it possible to prevent the driver from feeling annoyed by excessive notifications.
[0035] On the other hand, if the determination result in step S108 is Yes (vehicle speed V≧speed threshold THv), the process proceeds to step S110. The process proceeds to step S110 when the vehicle speed V is equal to or greater than the speed threshold THv even though the distance D is equal to or less than the distance threshold THd. In such a case, it is determined that the driver does not intend to stop. Therefore, in step S110, the processor 12 controls the notification device 30 to issue a notification to the driver to call his or her attention to the stop sign. This allows the notification to be issued in an appropriate scene.
[0036] The distance threshold THd and the speed threshold THv used in steps S106 and S108, respectively, may be changed based on, for example, the driver's operation of an HMI (Human Machine Interface) device (e.g., a touch panel). Specifically, for example, the distance threshold THd and the speed threshold THv may be changed in order to change the sensitivity of the notification by the notification device 30 to any number of levels (e.g., low, medium, and high) by the driver's operation of the HMI device.
[0037] 3.Effects As described above, according to this embodiment, not only the first camera 20 but also the second camera 22 is used as necessary to detect the road marking 5. The second camera 22 can capture an image of a road area R2A closer to the vehicle 1 than the road area R1 captured by the first camera 20 (see FIG. 1). Therefore, even if the first camera 20 cannot detect the road marking 5 when the vehicle 1 (own vehicle) approaches the road marking 5 such as a stop line 6 due to the presence of a leading vehicle, the road marking 5 can be detected by the second camera 22. In addition, even if the vehicle 1 approaches the vicinity of the road marking 5, it is possible to fit the road marking 5 within the angle of view of the second camera 22. In this way, according to the "road marking detection device" of this embodiment, the road marking 5 such as the stop line 6 can be reliably detected.
[0038] Furthermore, according to this embodiment, as described above, the second camera 22 is used in addition to the first camera 20 to increase the chances of detecting the road markings 5, and therefore the chances of determining whether the driver intends to stop can be increased. This makes it possible to more effectively call attention using the notification device 30.
[0039] 4. Modifications regarding notifications by notification devices The notification by the notification device 30 may be performed as follows.
[0040] Fig. 4 is a diagram illustrating an example of the configuration of a vehicle 8 according to a modified example of the embodiment. Vehicle 8 is configured similarly to vehicle 1 illustrated in Fig. 1, except that it is additionally equipped with inward-facing camera 24. Inward-facing camera 24 is mounted inside vehicle 8 and captures an image of the driver.
[0041] 5 is a flowchart showing a process related to a preventive safety support according to a modified embodiment of the present invention. The process of this flowchart is the same as the process of the flowchart shown in FIG. 3, except that the process of step S200 is executed instead of the processes of steps S106 and S108.
[0042] 5, if the determination result in step S102 or step S104 is Yes, the process proceeds to step S200. In step S200, processor 12 acquires driver information such as line of sight, eye opening, face direction, and posture based on the image from inward-facing camera 24. Then, processor 12 determines whether the driver is inattentive to the road ahead, such as looking away, based on the driver information.
[0043] If the determination result in step S200 is No (i.e., if the driver's inattention to the road ahead is not detected), the process proceeds to Return. That is, no notification is made by the notification device 30. This makes it possible to prevent the driver from feeling annoyed by excessive notifications.
[0044] On the other hand, if the determination result in step S200 is Yes (that is, if the driver's inattention to the road ahead is detected), the process proceeds to step S110, and the notification device 30 issues a notification. This allows the notification to be issued in an appropriate scene. [Explanation of symbols]
[0045] 1, 8 cars 2. Intersection 3. Running track 4 stop sign 5 Road markings 6 stop line 7. "Stop" Sign 8 Vehicles 10 Electronic Control Unit (ECU) 12 processors 14 Storage device 16 Vehicle speed sensor 20 Camera No. 1 22 Second Camera 24 Inward-facing camera 30. Notification device
Claims
1. A first camera that captures an image of at least a front area of the vehicle; a second camera that captures an image of at least a front area of the vehicle and is capable of capturing an image of a road surface area closer to the vehicle than the road surface area captured by the first camera; A road marking detection device applied to a vehicle comprising: A processor is provided. The processor, determining whether a stop sign in front of the vehicle is detected by the first camera; a process of determining, when the stop sign is detected by the first camera, whether or not a road marking corresponding to the detected stop sign has been detected by the first camera; detecting the road marking by the second camera when the road marking is not detected by the first camera; Run Road marking detection device.
2. The road sign detection device according to claim 1 ; An alarm device that issues an alarm to alert a driver of the vehicle; a processor that controls the notification device to issue a notification to call the driver's attention to a temporary stop of the vehicle when a distance between the detected road marking and the vehicle is equal to or less than a distance threshold and a vehicle speed of the vehicle is equal to or greater than a speed threshold; Equipped Notification system.
3. The road sign detection device according to claim 1 ; An alarm device that issues an alarm to alert a driver of the vehicle; An inward-facing camera mounted in a passenger compartment of the vehicle for capturing an image of the driver; a processor that controls the notification device to provide a notification to alert the driver to a temporary stop of the vehicle when it is detected that the driver is not paying attention to a road ahead based on an image from the inward-facing camera after the road marking is detected; Equipped Notification system.
4. 1. A road marking detection method applied to a vehicle including a first camera that images at least a road ahead of the vehicle, and a second camera that images at least a road ahead of the vehicle and is capable of imaging a road surface area closer to the vehicle than a road surface area imaged by the first camera, determining whether a stop sign in front of the vehicle is detected by the first camera; When the stop sign is detected by the first camera, determining whether or not a road marking corresponding to the detected stop sign has been detected by the first camera; detecting the road marking with the second camera when the road marking is not detected by the first camera; Includes Pavement marking detection method.
Citation Information
Patent Citations
Vehicle alarm unit
JP2006260038A
Vehicle exterior recognition system
JP2007257305A
Stop line recognition device
JP2011123613A
Voice guidance notification control device and voice guidance notification control method
JP2019008443A
