Road marking detection device
The lane marking detection device addresses deviations in machine learning-based detection by displaying known and current lane markings and identifying anomalies, ensuring accurate user notification of discrepancies and system issues.
Patent Information
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- TOYOTA JIDOSHA KK
- Filing Date
- 2023-08-09
- Publication Date
- 2026-05-19
AI Technical Summary
Existing partition line detection systems using machine learning models can produce deviations in detection results due to insufficient learning, necessitating a way to inform users of discrepancies in the detection accurately.
A lane marking detection device that utilizes a lane marking detection model, including a known lane marking recognition unit, a lane marking detection unit, and an information providing unit to display known and current lane markings when deviations exceed a threshold, along with model and sensor anomaly determination units to identify and notify users of anomalies.
The device effectively informs users of discrepancies in lane marking detection by displaying known and current markings, allowing for accurate identification of model and sensor anomalies, thereby enhancing user understanding of detection status.
Smart Images
Figure 0007861725000001 
Figure 0007861725000002 
Figure 0007861725000003
Abstract
Description
Technical Field
[0001] The present invention relates to a partition line detection device.
Background Art
[0002] Conventionally, Japanese Patent Application Laid-Open No. 2006-311299 is known as a technical document related to a partition line detection device. In this publication, in a device that detects a parking partition line of a parking partition where a vehicle parks based on an image captured by imaging means, when the parking partition line detected by the parking partition line detection means is incomplete, it is shown that a past image stored in the past image storage means is superimposed on the current image and displayed.
Prior Art Document
Patent Document
[0003]
Patent Document 1
Summary of the Invention
Problems to be Solved by the Invention
[0004] By the way, it has been considered to use a machine learning model in partition line detection. However, if there is insufficient learning of the machine learning model or the like, a deviation occurs in the detection result of the partition line. For this reason, it is required to appropriately inform the user that a deviation has occurred in the detection result of the partition line when detecting the partition line by the machine learning model.
Means for Solving the Problems
[0005] One aspect of the present invention is a lane marking detection device that detects lane markings in front of a vehicle using a lane marking detection model, which is a machine learning model used for lane marking detection, comprising: a known lane marking recognition unit that recognizes known lane markings in front of the vehicle based on past lane marking detection information or map information including lane marking information and the vehicle's position information; a lane marking detection unit that detects lane markings in front of the vehicle using the lane marking detection model based on the detection results of the vehicle's external sensors; and an information providing unit that displays images of known lane markings and lane markings to the vehicle's user when the amount of widthwise displacement between the known lane markings and lane markings is greater than or equal to an image display threshold. A model anomaly determination unit determines that there is an anomaly in the lane line detection model when the comparison result between known lane lines and lane lines is similar to a pre-stored model anomaly determination pattern. It is equipped with.
[0006] According to one aspect of the present invention, when there is a difference between a known boundary line, which is a boundary line detected in the past or a boundary line on a map, and the boundary line currently being detected, the device can appropriately inform the user that there is a discrepancy in the boundary line detection results by displaying the known boundary line and the current boundary line as images to the user.
[0007] A lane marking detection device according to one aspect of the present invention, a sensor abnormality determination unit determines that there is an abnormality in the external sensor when an abnormal area is detected continuously for a certain period of time in at least a part of the detection range of the external sensor. Furthermore, it is equipped with , The model anomaly detection unit is, When the discrepancy between a known lane mark and the current lane mark exceeds the image display threshold, and the sensor abnormality detection unit does not determine that there is an abnormality in the external sensor, the lane mark detection model is determined to be abnormal. death The abnormal region may be a region where the amount of deviation in the width direction between known lane lines exceeds the sensor abnormality threshold, or a region where known lane lines exist but are not detected. [Effects of the Invention]
[0008] According to one aspect of the present invention, it is possible to appropriately inform the user that there is a discrepancy in the detection results of the road markings. [Brief explanation of the drawing]
[0009] [Figure 1] Block diagram of a road mark detection device according to one embodiment. [Figure 2] This figure shows an example of a model anomaly detection pattern. [Figure 3] This figure shows other examples of model anomaly detection patterns. [Figure 4] This figure shows an example of a case where a lane marking is misdetected due to dirt accumulation. [Figure 5] This flowchart shows an example of the process for displaying road markings in images. [Figure 6] This flowchart shows an example of an anomaly notification process. [Modes for carrying out the invention]
[0010] Embodiments of the present invention will be described below with reference to the drawings.
[0011] [Configuration of the road marking detection device] Figure 1 is a block diagram showing a lane marking detection device 100 according to one embodiment. As shown in Figure 1, the lane marking detection device 100 is mounted on the vehicle 1 and is a device that detects lane markings in front of the vehicle 1 as it travels on the road. The lane marking detection device 100 transmits the lane marking detection result to the vehicle 1's automated driving system or driver assistance system, for example. Lane markings are white lines (including dashed lines, etc.) that form lanes on the road. The lane marking detection device 100 may be configured as part of the automated driving system or driver assistance system.
[0012] The lane marking detection device 100 is equipped with an ECU 20 [Electronic Control Unit]. The ECU 20 is an electronic control unit having a CPU [Central Processing Unit] and a memory unit. The memory unit consists of, for example, ROM [Read-Only Memory], RAM [Random Access Memory], EEPROM [Electrically Erasable Programmable Read-Only Memory], etc. The ECU 20 realizes various functions by executing programs stored in the memory unit with the CPU. The ECU 20 may be composed of multiple electronic units.
[0013] As shown in Figure 1, the ECU 20 is connected to the GNSS receiver 10, external camera 11, radar sensor 12, map database 13, lane marking database 14, and HMI 15 [Human Machine Interface].
[0014] The GNSS receiver 10 measures the position of the vehicle 1 (for example, the latitude and longitude of the vehicle 1) by receiving signals from positioning satellites. The GNSS receiver 10 transmits the measured position information of the vehicle 1 to the ECU 20.
[0015] The external camera 11 is an imaging device (external sensor) that captures images of the external conditions of the vehicle 1. The external camera 11 is installed, for example, on the back of the windshield of the vehicle 1 and captures images of the area in front of the vehicle 1. The external camera 11 may be installed on the side or rear of the vehicle 1 and configured to capture images of the area around the vehicle 1. The external camera 11 transmits the captured images of the area outside the vehicle 1 to the ECU 20.
[0016] The radar sensor 12 is a detection device (external sensor) that detects objects around the host vehicle 1 using radio waves (e.g., millimeter waves) or light. The radar sensor 12 includes, for example, millimeter wave radars or lidars [LiDAR: Light Detection and Ranging] provided in multiple directions of the host vehicle 1. The radar sensor 12 transmits radio waves or light around the host vehicle and detects an object by receiving the radio waves or light reflected by the object. The radar sensor 12 transmits information on the detected object to the ECU 20. The object includes a lane marking.
[0017] The map database 13 is a database that stores map information. The map database 13 is formed, for example, in a storage device such as an HDD [Hard Disk Drive] mounted on the host vehicle 1. The map information includes road position information, road shape information (e.g., types of curves, straight sections, curvature of curves, etc.), position information of intersections and branch points, and the like.
[0018] In addition, the map information includes position information of the lane marking as information on the lane marking of the road. The information on the lane marking may include information on the type of the lane marking (solid line, broken line, double solid line, center line of the lane, lane boundary line, etc.). Note that the map database 13 is not limited to being mounted on the host vehicle 1 and may be formed in a server communicable with the host vehicle 1.
[0019] The lane marking database 14 is a database that stores past lane marking detection information. The lane marking database 14 is also formed, for example, in the storage device of the host vehicle 1. The past lane marking detection information is the position information of the lane marking detected by the lane marking detection device 100 in the past. The past lane marking detection information is stored in association with the position on the map of the host vehicle 1. Note that the lane marking database 14 may be configured to share past lane marking detection information acquired by other vehicles through a server. The lane marking database 14 is not limited to being mounted on the host vehicle 1 and may be formed in a server communicable with the host vehicle 1.
[0020] Furthermore, the ECU 20 of the lane marking detection device 100 does not necessarily need to be connected to the lane marking database 14 if the map information includes lane marking information. Conversely, if the lane marking detection device 100 can acquire past lane marking detection information, the map information does not need to include lane marking information.
[0021] The HMI15 is an interface for inputting and outputting information between the ECU20 and the driver. The HMI15 includes, for example, a display and speakers installed in the vehicle cabin. The HMI15 outputs images from the display and audio from the speakers in response to control signals from the ECU20. The display may be a MID (Multi-Information Display) or a HUD (Head-Up Display). The HMI15 may also be equipped with various indicators.
[0022] Next, the functional configuration of the ECU20 will be described. As shown in Figure 1, the ECU20 includes a known lane line recognition unit 21, a lane line detection unit 22, a model anomaly determination unit 23, a sensor anomaly determination unit 24, and an information provision unit 25.
[0023] The known lane marking recognition unit 21 recognizes known lane markings in front of the vehicle 1 based on past lane marking detection information or map information containing lane marking information stored in the lane marking database 14 and the position information of the vehicle 1 measured by the GNSS receiver unit 10. Known lane markings are lane markings that have become known from past lane marking detection information or map information. The position information of the vehicle 1 may be obtained by SLAM (Simultaneous Localization and Mapping) or the like.
[0024] The known lane marking recognition unit 21 recognizes known lane markings in front of the vehicle 1, for example, based on past lane marking detection information and the position information of the vehicle 1. The known lane marking recognition unit 21 uses the position information of the vehicle 1 to obtain past lane marking detection information corresponding to a certain distance in front of the vehicle 1 from the lane marking database 14. The known lane marking recognition unit 21 recognizes known lane markings in front of the vehicle 1 by projecting the position of the lane markings in the past lane marking detection information onto a map coordinate system based on the position of the vehicle 1, for example.
[0025] The known lane marking recognition unit 21 may recognize known lane markings in front of the vehicle 1 based on map information including lane marking information and the position information of the vehicle 1. The known lane marking recognition unit 21 acquires lane marking information corresponding to a certain distance in front of the vehicle 1 from the map information. The known lane marking recognition unit 21 recognizes known lane markings in front of the vehicle 1 by, for example, projecting the lane markings included in the lane marking information onto a map coordinate system based on the position of the vehicle 1. Note that the method of recognizing known lane markings is not limited to the above.
[0026] The known lane marking recognition unit 21 may determine, based on the location information of the vehicle 1 and the map information, whether or not lane marking information corresponding to the current location of the vehicle 1 is included in the map information. If lane marking information corresponding to the current location of the vehicle 1 is included in the map information, the known lane marking recognition unit 21 may prioritize the lane marking information in the map information over past lane marking detection information in the lane marking database 14. In this case, the known lane marking recognition unit 21 recognizes known lane markings based on the map information.
[0027] Alternatively, the known lane marking recognition unit 21 may compare the update date and time of the map information corresponding to the current location of the vehicle 1 with the acquisition date and time of past lane marking detection information in the lane marking database 14 corresponding to the current location of the vehicle 1, and prioritize the newer one. If the acquisition date and time of the past lane marking detection information is newer than the update date and time of the map information, the known lane marking recognition unit 21 recognizes the known lane marking based on the past lane marking detection information in the lane marking database 14.
[0028] The lane marking detection unit 22 detects the lane markings in front of the vehicle 1 from the lane marking detection model 22a based on at least one of the images captured by the external camera 11 and the detection results of the radar sensor 12.
[0029] The road boundary detection model 22a is a machine learning model trained using deep learning to output road boundary detection results from at least one of the images captured by the external camera 11 and the detection results of the radar sensor 12. The road boundary detection model 22a is composed of a neural network, such as a convolutional neural network [CNN]. The neural network may include multiple layers, including multiple convolutional layers and pooling layers. The road boundary detection model 22a may also be composed of a recurrent neural network [RNN] that uses past detection results, including the detection result of the previous road boundary, as input.
[0030] The lane marking detection unit 22 inputs at least one of the images captured by the external camera 11 and the detection results of the radar sensor 12 to the lane marking detection model 22a, thereby obtaining the detection result of the lane markings in front of the vehicle 1 as an output from the lane marking detection model 22a.
[0031] The model anomaly determination unit 23 determines whether there is an anomaly in the lane line detection model 22a if the amount of deviation in the width direction between the known lane line recognized by the known lane line recognition unit 21 and the lane line detected by the lane line detection unit 22 is greater than or equal to the anomaly determination threshold. The anomaly determination threshold is a threshold set in advance to determine anomalies in lane line detection. The amount of deviation can be calculated as the average, median, or maximum value of the distance between the nearest known lane line and the lane line in the width direction. The amount of deviation is measured, for example, in a planar coordinate system or map coordinate system (geographic coordinate system) based on the vehicle 1. If a known lane line and a lane line are detected on the left and right sides of the vehicle 1, the larger of the deviation amounts of the left and right known lane lines and lane lines is used for the determination.
[0032] The model anomaly determination unit 23 determines whether or not there is an anomaly in the boundary line detection model 22a based on the known boundary lines recognized by the known boundary line recognition unit 21 and the boundary lines detected by the boundary line detection unit 22. Specifically, the model anomaly determination unit 23 determines that there is an anomaly in the boundary line detection model 22a when, for example, the comparison result between the known boundary lines and the boundary lines is similar to a pre-stored model anomaly determination pattern. The model anomaly determination pattern is a record of a specific pattern observed when the boundary line detection model 22a shows an anomaly. When the boundary line detection model 22a is functioning normally, the known boundary lines and the boundary lines detected by the boundary line detection model 22a match or are similar. On the other hand, when the boundary line detection model 22a is anomaly, the amount of discrepancy (deviation) between the known boundary lines and the boundary lines detected by the boundary line detection model 22a increases.
[0033] In this case, the comparison results between known lane markings and the actual lane markings show a different trend than the false detection of lane markings when the external sensor malfunctions. Therefore, by focusing on the comparison results between known lane markings and the actual lane markings, it is possible to determine whether a large discrepancy between known lane markings and the actual lane markings is due to a malfunction of the external sensor or a malfunction of the lane marking detection model 22a.
[0034] Here, Figure 2 shows an example of a model anomaly detection pattern. Figure 2 shows the actual left lane markings in front of the vehicle 1, namely the left real lane marking La and the right real lane marking Lb, the left known lane marking Ka and the right known lane marking Kb, and the left lane marking Da and the right lane marking Db detected by the lane marking detection unit 22. The left known lane marking Ka and the right known lane marking Kb are in close agreement with the left real lane marking La and the right real lane marking Lb.
[0035] In the situation shown in Figure 2, the left lane marking Da and the right lane marking Db are detected as meandering curves. When such meandering lane markings are detected, it is highly likely that the problem lies with the lane marking detection model 22a rather than with the external sensor. The model anomaly determination unit 23 determines that there is an anomaly in the lane marking detection model 22a when it detects meandering left lane marking Da and right lane marking Db compared to the left lane marking Ka and right lane marking Kb, which are approximately straight lines, as this is similar to the model anomaly determination pattern shown in Figure 2.
[0036] Figure 3 shows another example of a model anomaly detection pattern. In Figure 3, the left lane marking Da and the right lane marking Db are detected as bending in an arc in the longitudinal direction of the vehicle 1. When such bending lane markings are detected, there is a high probability that the lane marking detection model 22a is an anomaly. The model anomaly detection unit 23 may determine that there is an anomaly in the lane marking detection model 22a if the left lane marking Da and the right lane marking Db are detected as bending in the longitudinal direction compared to the left lane marking Ka and the right lane marking Kb, which are approximately straight lines, as this is similar to the model anomaly detection pattern shown in Figure 3.
[0037] The model anomaly detection pattern is not limited to the embodiments shown in Figures 2 and 3, but encompasses a variety of patterns. The model anomaly detection pattern may be set based on the simulation results of the lane marking detection model 22a and the detection results from actual test drives.
[0038] In addition, the model anomaly determination patterns may include patterns in which, despite the known lane markings being within the detection range of the external sensor, the left lane marking Da and / or the right lane marking Db detected by the lane marking detection unit 22 are located outside the detection range of the external sensor (for example, outside the imaging range of the external camera 11). If the left lane marking Da and / or the right lane marking Db are located outside the detection range of the external sensor, there is a high probability that the lane marking detection model 22a is an anomaly.
[0039] Furthermore, the model anomaly determination unit 23 may determine whether or not there is an anomaly in the lane line detection model 22a based on the comparison result between the known lane line and the lane line, regardless of the model anomaly determination pattern. The model anomaly determination unit 23 may also determine that there is an anomaly in the lane line detection model 22a when the known lane line is a nearly straight line or curve that does not meander, and the maximum value of the width between the right end and left end of the meandering lane line detected by the lane line detection unit 22 is greater than or equal to a certain value, and the number of meanders (corresponding to the degree of the shape of the lane line) is greater than or equal to a certain number.
[0040] Similarly, the model anomaly determination unit 23 may determine that there is an anomaly in the lane line detection model 22a when the known lane line is approximately a straight line or a curve (a curve with a curvature less than a certain degree that does not include sharp curves), and the lane line includes multiple coordinate points that are separated horizontally due to a return, as shown in Figure 3. The model anomaly determination unit 23 may also determine that there is an anomaly in the lane line detection model 22a when the number of multiple coordinate points separated horizontally is greater than or equal to a threshold. The coordinate points may be counted in a planar coordinate system or a map coordinate system, or they may be counted on the captured image. The model anomaly determination unit 23 may also determine that there is an anomaly in the lane line detection model 22a when the lane line detected by the lane line detection unit 22 is located outside the detection range of the external sensor, even though the known lane line is located within the detection range of the external sensor.
[0041] Furthermore, the model anomaly determination unit 23 may determine that there is an anomaly in the road mark detection model 22a when the amount of deviation in the width direction between the known road mark recognized by the known road mark recognition unit 21 and the road mark detected by the road mark detection unit 22 is greater than or equal to an anomaly determination threshold (image display threshold), and when the sensor anomaly determination unit 24, described later, determines that there is no anomaly in the external sensor.
[0042] The sensor anomaly determination unit 24 determines whether there is an anomaly in an external sensor such as the external camera 11 or radar sensor 12 if the amount of deviation in the width direction between the known lane markings recognized by the known lane marking recognition unit 21 and the lane markings detected by the lane marking detection unit 22 is greater than or equal to an anomaly determination threshold. The anomaly determination threshold may be the same value as the anomaly determination threshold of the model anomaly determination unit 23, or it may be a different value.
[0043] The sensor abnormality determination unit 24 determines that there is an abnormality in the external sensor, for example, when an abnormal signal is transmitted from the external camera 11 or the radar sensor 12. The sensor abnormality determination unit 24 may also determine that there is an abnormality in the external camera 11 if frequent noise, brightness abnormalities, chromaticity abnormalities, etc. occur in the image captured by the external camera 11. Based on the detection results of the radar sensor 12, the sensor abnormality determination unit 24 may also determine that there is an abnormality in the radar sensor 12 if frequent noise, clutter fixation, etc. occur.
[0044] The sensor abnormality determination unit 24 may also determine whether or not a preset sensor abnormality determination condition has been met based on the image captured by the external camera 11 or the detection result of the radar sensor 12. The sensor abnormality determination condition is a condition used to determine an abnormality in the external sensor of the vehicle 1. For example, the sensor abnormality determination unit 24 determines that the sensor abnormality determination condition has been met if the image captured by the external camera 11 shows frequent noise, brightness abnormalities, or chromaticity abnormalities, or if the radar sensor 12 shows frequent noise or clutter fixation occurs.
[0045] Furthermore, the sensor abnormality determination unit 24 determines that the sensor abnormality determination condition has been met when an abnormal area is detected continuously for a certain amount of time within a portion of the imaging range of the external camera 11. The detection range of the radar sensor 12 may be used instead of the imaging range of the external camera 11. An abnormal area is, for example, an area on the image captured by the external camera 11 where a detection abnormality occurs due to dirt or water droplets adhering to the lens of the external camera 11. An abnormal area may also be an area within the detection range of the radar sensor 12 where a detection abnormality occurs due to dirt or water droplets adhering to the detection part of the radar sensor 12. Driving time is the time counted while the vehicle 1 is in motion. Driving time is not counted when the vehicle 1 is stopped.
[0046] Figure 4 shows an example of a case where a lane marking is misdetected due to dirt accumulation. Figure 4 shows the abnormal region F caused by lens dirt. In the situation shown in Figure 4, the lane marking is not detected due to the influence of the abnormal region F (dirt), and the curb is mistakenly detected as lane marking Db.
[0047] The sensor anomaly determination unit 24 recognizes an area where the amount of widthwise deviation between known lane lines and lane lines is greater than or equal to the sensor anomaly threshold, or an area where known lane lines exist but are not detected (an area within the imaging range) as an anomaly area. The sensor anomaly threshold is a threshold value smaller than the anomaly determination threshold. The sensor anomaly determination unit 24 determines that the sensor anomaly determination condition has been met if an anomaly area F is detected continuously for a certain amount of time at a predetermined position within the imaging range, even though the vehicle 1 is in motion.
[0048] Abnormal areas can also be caused by the adhesion of water droplets. In the case of water droplets, the detection of the lane markings may not be directly obstructed, and the lane markings may still be detectable. However, the refraction of light may distort the position of the lane markings, leading to incorrect detection.
[0049] The information provision unit 25 displays images of the known lane markings and the lane markings to the user of the vehicle 1 if the amount of widthwise deviation between the known lane markings recognized by the known lane marking recognition unit 21 and the lane markings detected by the lane marking detection unit 22 is greater than or equal to the image display threshold. The image display threshold is a threshold set for determining whether to display images of the known lane markings and the lane markings. The image display threshold may be the same value as the abnormality determination threshold, or it may be a smaller value than the abnormality determination threshold.
[0050] The information provision unit 25 controls the HMI 15 to display images of known lane markings and lane markings on the HMI 15's display. The information provision unit 25 displays images as shown in Figures 2 to 4, for example. The information provision unit 25 may also display abnormal area F if an abnormal area F is detected by the sensor abnormality determination unit 24, as shown in Figure 4. The user can understand the lane marking detection status by comparing it with the known lane markings, as both known lane markings (past lane markings for reference) and lane markings (lane markings currently detected by the vehicle 1 and used for autonomous driving and driver assistance) are displayed in the image.
[0051] Furthermore, the information providing unit 25 does not necessarily need to overlay the image on the view in front of the vehicle 1 as shown in Figures 2 to 4. The information providing unit 25 may display known lane markings and lane markings as images in a bird's-eye view or plan view. The information providing unit 25 may also virtually display the display corresponding to the vehicle 1 and the known lane markings and lane markings in a bird's-eye view or plan view.
[0052] The information provision unit 25 may notify the user of a lane marking detection error if the known lane marking recognition unit 21 recognizes a known lane marking in front of the vehicle 1, but the lane marking detection unit 22 cannot detect the lane marking in front of the vehicle 1. A lane marking detection error notification is a notification to inform the user that a lane marking cannot be detected despite the presence of a known lane marking. If a lane marking cannot be detected in a location where a known lane marking has been recognized, it is highly likely that the lane marking detection device 100 has an error, except in exceptions such as road construction zones. For this reason, the information provision unit 25 controls the HMI 15 to notify the user of the lane marking detection error by displaying an image on the display and / or outputting audio from the speaker.
[0053] The information provision unit 25 may notify the user of a model anomaly if the model anomaly determination unit 23 determines that there is an anomaly in the lane line detection model 22a. A model anomaly notification is a notification to inform the user that there is an anomaly in the lane line detection model 22a. The information provision unit 25 notifies the user of the model anomaly by controlling the HMI 15 and displaying an image on the display and / or outputting audio from the speaker.
[0054] The information provision unit 25 may notify the user of a sensor abnormality if the sensor abnormality determination unit 24 determines that there is an abnormality in the external sensor. A sensor abnormality notification is a notification to inform the user that there is an abnormality in the external sensor. The information provision unit 25 controls the HMI 15 to notify the user of the sensor abnormality by displaying an image on the display and / or outputting audio from the speaker.
[0055] The information provision unit 25 may perform interpolation processing (rewriting processing) of the abnormal region F when the sensor abnormality determination unit 24 detects an abnormal region F and displays an image to the user. Interpolation processing is the process of replacing the lane markings (lane markings currently detected by the vehicle 1 and used for autonomous driving and driver assistance) within the abnormal region F of the imaging range with known lane markings.
[0056] The information provision unit 25 may, for example, when the sensor anomaly detection unit 24 detects an anomaly area F, read the portion of the previously known lane markings corresponding to the anomaly area F that corresponds to the current position of the vehicle, and detect those portions as lane markings. The information provision unit 25 may choose to interpolate a rectangular area including the anomaly area F, the lower half of the imaging range including the anomaly area F, or the entire imaging range. The information provision unit 25 displays the interpolated known lane markings and the image including the lane markings to the user.
[0057] [Processing method for lane line detection device] Next, the processing method of the lane marking detection device 100 according to this embodiment will be described with reference to the drawings. Figure 5 is a flowchart showing an example of lane marking image display processing. Lane marking image display processing is performed, for example, when the vehicle 1 requires lane marking detection in autonomous driving or driver assistance.
[0058] As shown in Figure 5, in S10, the ECU 20 of the lane marking detection device 100 recognizes the known lane markings in front of the vehicle 1 using the known lane marking recognition unit 21. The known lane marking recognition unit 21 recognizes the known lane markings in front of the vehicle 1 based on past lane marking detection information or map information including lane marking information stored in the lane marking database 14 and the position information of the vehicle 1 measured by the GNSS receiver unit 10. After that, the ECU 20 proceeds to S11.
[0059] In S11, the ECU 20 detects the lane markings in front of the vehicle 1 using the lane marking detection unit 22. The lane marking detection unit 22 inputs at least one of the images captured by the external camera 11 and the detection results of the radar sensor 12 to the lane marking detection model 22a, thereby obtaining the detection result of the lane markings in front of the vehicle 1 as an output from the lane marking detection model 22a. After that, the ECU 20 proceeds to S12.
[0060] In S12, the ECU 20 determines whether or not a lane marking has been detected at a location where a known lane marking has been recognized by the information provision unit 25. If the ECU 20 determines that a lane marking has not been detected at a location where a known lane marking has been recognized, it proceeds to S13. If the ECU 20 does not determine that a lane marking has not been detected at a location where a known lane marking has been recognized (i.e., the lane marking has been detected), it proceeds to S14.
[0061] In S13, the ECU 20 notifies the user of a lane marking detection error via the information provision unit 25. The information provision unit 25 controls the HMI 15 to notify the user of the lane marking detection error by displaying an image and / or outputting sound. After that, the ECU 20 terminates the lane marking image display process.
[0062] In S14, the ECU 20 determines whether the amount of widthwise displacement between the known lane markings recognized by the known lane marking recognition unit 21 and the lane markings detected by the lane marking detection unit 22 is greater than or equal to the image display threshold. If the ECU 20 determines that the amount of widthwise displacement between the known lane markings and the lane markings is greater than or equal to the image display threshold, it proceeds to S15. If the ECU 20 determines that the amount of widthwise displacement between the known lane markings and the lane markings is not greater than or equal to the image display threshold, it terminates the lane marking image display process.
[0063] In S15, the ECU 20, via the information provision unit 25, displays images of known lane markings and lane markings to the user of its vehicle 1. The information provision unit 25 controls the HMI 15 to display images of known lane markings and lane markings on the HMI 15's display. The information provision unit 25 displays images as shown in Figures 2 to 4, for example. After that, the ECU 20 terminates the lane marking image display process.
[0064] Figure 6 is a flowchart showing an example of an anomaly notification process. The anomaly notification process is executed, for example, when the image displayed in S15 of Figure 5 occurs. The anomaly notification process may also be executed regardless of the image display, when lane markings are detected in autonomous driving or driver assistance.
[0065] As shown in Figure 6, in S20, the ECU 20 determines whether the sensor abnormality determination condition has been met by the sensor abnormality determination unit 24. The sensor abnormality determination unit 24 determines that the sensor abnormality determination condition has been met, for example, when an abnormal area is detected continuously for a certain amount of time in a part of the imaging range of the external camera 11. If the ECU 20 determines that the sensor abnormality determination condition has been met, it proceeds to S21. If the ECU 20 determines that the sensor abnormality determination condition has not been met, it proceeds to S22.
[0066] In S21, the ECU 20 notifies the user of its own vehicle 1 of a sensor malfunction via the information provision unit 25. The information provision unit 25 controls the HMI 15 to notify the user of the sensor malfunction through image display and / or audio output. After that, the ECU 20 terminates the malfunction notification process.
[0067] In S22, the ECU20 uses the model anomaly determination unit 23 to determine whether the comparison result between the known lane markings and the lane markings is similar to the model anomaly determination pattern. If the ECU20 determines that the comparison result between the known lane markings and the lane markings is similar to the model anomaly determination pattern, it proceeds to S23. If the ECU20 does not determine that the comparison result between the known lane markings and the lane markings is similar to the model anomaly determination pattern, it terminates the anomaly notification process.
[0068] In S23, the ECU 20 notifies the user of its own vehicle 1 of a model anomaly via the information provision unit 25. The information provision unit 25 controls the HMI 15 to notify the user of the model anomaly through image display and / or audio output. After that, the ECU 20 terminates the anomaly notification process.
[0069] According to the lane line detection device 100 of this embodiment described above, when there is a difference between a known lane line, which is a lane line detected in the past or a lane line on a map, and the lane line currently being detected, the device can appropriately inform the user that there is a discrepancy in the lane line detection results by displaying the known lane line and the current lane line as images to the user.
[0070] Furthermore, according to the lane marking detection device 100, if a known lane marking in front of the vehicle 1 is recognized but the lane marking detection unit 22 cannot detect the lane marking in front of the vehicle 1, the device notifies the user that the lane marking is not detectable despite the presence of a known lane marking, so the user can understand the status of lane marking detection.
[0071] Furthermore, according to the lane marking detection device 100, the model anomaly determination unit 23 can determine whether or not there is an anomaly in the lane marking detection model 22a based on the comparison results between known lane markings and the lane markings, thereby notifying the user of any model anomalies. Similarly, according to the lane marking detection device 100, if the sensor anomaly determination unit 24 determines that the sensor anomaly determination conditions have been met, the user can be notified of any sensor anomalies, thereby enabling the user to understand any anomalies in the external sensors. The sensor anomaly determination unit 24 can determine an anomaly in the external sensor if an anomaly area is detected continuously for a certain period of time in at least a portion of the detection range of the external sensor.
[0072] Although embodiments of the present invention have been described above, the present invention is not limited to the embodiments described above. The present invention can be implemented in various forms, starting with the embodiments described above, by making various changes and improvements based on the knowledge of those skilled in the art.
[0073] The lane marking detection device 100 does not necessarily need to have a model anomaly determination unit 23. Similarly, the lane marking detection device 100 does not necessarily need to have a sensor anomaly determination unit 24. Furthermore, the lane marking detection device 100 does not necessarily need to notify the user if a known lane marking in front of the vehicle 1 is recognized, but the lane marking detection unit 22 cannot detect the lane marking in front of the vehicle 1. [Explanation of symbols]
[0074] 1...Vehicle, 20...ECU, 21...Known lane marking recognition unit, 22...Lane marking detection unit, 22a...Lane marking detection model, 23...Model anomaly determination unit, 24...Sensor anomaly determination unit, 25...Information provision unit, 100...Lane marking detection device, Da, Db...Lane markings, F...Anomaly area, Ka, Kb...Known lane markings.
Claims
1. A lane marking detection device that detects lane markings in front of the vehicle using a lane marking detection model, which is a machine learning model used for lane marking detection, A known lane marking recognition unit recognizes known lane markings in front of the vehicle based on past lane marking detection information or map information including lane marking information and the vehicle's position information, A lane marking detection unit detects the lane markings in front of the vehicle using the lane marking detection model based on the detection results of the vehicle's external sensors, An information providing unit that, when the amount of widthwise displacement between the known lane marking and the lane marking is greater than or equal to an image display threshold, displays an image of the known lane marking and the lane marking to the user of the vehicle, A model anomaly determination unit determines that there is an anomaly in the boundary line detection model when the comparison result between the known boundary line and the boundary line is similar to a pre-stored model anomaly determination pattern. A road marking detection device equipped with the following features.
2. The lane marking detection device according to claim 1, wherein the information providing unit recognizes the known lane marking in front of the vehicle by the known lane marking recognition unit, but the lane marking detection unit cannot detect the lane marking in front of the vehicle, and the unit notifies the user that the lane marking is not detectable despite the presence of the known lane marking.
3. The system further includes a sensor abnormality determination unit that determines that there is an abnormality in the external sensor when an abnormal area is detected continuously for a certain period of time in at least a portion of the detection range of the external sensor, The boundary line detection device according to claim 1 or 2, wherein the abnormal region is a region in which the amount of deviation in the width direction between the known boundary line and the boundary line is greater than or equal to the sensor abnormality threshold, or a region in which the boundary line is not detected despite the presence of the known boundary line.
4. The system further includes a sensor abnormality determination unit that determines that there is an abnormality in the external sensor when an abnormal area is detected continuously for a certain period of time in at least a portion of the detection range of the external sensor, The model anomaly determination unit determines that there is an anomaly in the boundary line detection model when the amount of deviation between the known boundary line and the boundary line is greater than or equal to the image display threshold, and the sensor anomaly determination unit does not determine that there is an anomaly in the external sensor. The boundary line detection device according to claim 1 or 2, wherein the abnormal region is a region in which the amount of deviation in the width direction between the known boundary line and the boundary line is greater than or equal to the sensor abnormality threshold, or a region in which the boundary line is not detected despite the presence of the known boundary line.