Lane marking detection apparatus

Abstract

To properly inform a user that there is a deviation in the detection results of lane markings.SOLUTION: A lane marking detection apparatus for detecting a lane marking ahead of a self vehicle using a lane marking detection model that is a machine learning model used for lane marking detection, comprises: a known lane marking recognition unit for recognizing a known lane marking ahead of the self vehicle based on past lane marking detection information or map information including lane marking information and position information of the self vehicle; a lane marking detection unit for detecting the lane marking ahead of the self vehicle using the lane marking detection model based on a detection result from an external sensor of the self vehicle; and an information providing unit for displaying an image of the known lane marking and the lane marking to a user of the self vehicle if a deviation amount in a width direction between the known lane marking and the lane marking is equal to or greater than an image display threshold.SELECTED DRAWING: Figure 5

Description

[Technical field]

[0001] The present invention relates to a marking line detection device. [Background technology]

[0002] Japanese Patent Application Publication No. 2006-311299 is known as a technical document related to a parking space line detection device. This publication describes that in a device that detects parking space lines of a parking space where a vehicle is parked based on an image captured by an imaging means, if the parking space lines detected by the parking space line detection means are incomplete, a past image stored in a past image storage means is displayed by superimposing it on a current image. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2006-311299 A Summary of the Invention [Problem to be solved by the invention]

[0004] By the way, the use of machine learning models for marking line detection is being considered. However, if the machine learning model is insufficiently trained, deviations in the detection results of marking lines will occur. For this reason, there is a need to properly inform users of deviations in the detection results of marking lines when the machine learning model detects marking lines. [Means for solving the problem]

[0005] One aspect of the present invention is a lane line detection device that detects lane lines ahead of the vehicle using a lane line detection model, which is a machine learning model used for lane line detection, and includes a known lane line recognition unit that recognizes known lane lines ahead of the vehicle based on past lane line detection information or map information including lane line information and position information of the vehicle, a lane line detection unit that detects lane lines ahead of the vehicle using the lane line detection model based on the detection results of an external sensor of the vehicle, and an information provision unit that displays images of the known lane line and the lane line to a user of the vehicle if the widthwise deviation between the known lane line and the lane line is greater than or equal to an image display threshold.

[0006] According to one embodiment of the lane detection device of the present invention, when there is a difference between a known lane line, which is a lane line detected in the past or a lane line on a map, and a currently detected lane line, an image of the known lane line and the lane line can be displayed to the user, thereby appropriately informing the user that there is a discrepancy in the lane line detection results.

[0007] In one embodiment of the lane marking detection device of the present invention, the device further includes a sensor anomaly judgment unit that judges that there is an abnormality in the external sensor when an abnormal area is detected continuously for a certain driving time in at least a portion of the detection range of the external sensor, and a model anomaly judgment unit that judges that there is an abnormality in the lane marking detection model when the amount of deviation between the known lane marking and the lane marking is greater than or equal to an image display threshold and the sensor anomaly judgment unit does not judge that there is an abnormality in the external sensor, and the abnormal area may be an area where the amount of deviation in the width direction between the known lane marking and the lane marking is greater than or equal to the sensor anomaly threshold, or an area where the lane marking is not detected despite the presence of the known lane marking. Effect of the Invention

[0008] According to one aspect of the present invention, it is possible to appropriately inform a user that a deviation has occurred in the detection results of lane markings. [Brief description of the drawings]

[0009] [Figure 1] 1 is a block diagram showing a lane marking detection device according to an embodiment; [Diagram 2] FIG. 11 is a diagram showing an example of a model abnormality determination pattern. [Diagram 3] FIG. 13 is a diagram showing another example of the model abnormality determination pattern. [Figure 4] 11 is a diagram showing an example of a case where an erroneous detection of a lane marking occurs due to adhesion of dirt; [Diagram 5] 13 is a flowchart showing an example of a lane line image display process. [Figure 6] 13 is a flowchart illustrating an example of an abnormality notification process. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0011] [Configuration of lane marking detection device] FIG. 1 is a block diagram showing a lane marking detection device 100 according to an embodiment. As shown in FIG. 1, the lane marking detection device 100 is mounted on a host vehicle 1 and detects lane marks ahead of the host vehicle 1 traveling on a road. The lane marking detection device 100 transmits, for example, a detection result of the lane marking to an automatic driving system or a driving assistance system of the host vehicle 1. A lane marking is a white line (including a broken line, etc.) that forms a lane on a road. The lane marking detection device 100 may be configured as a part of the automatic driving system or the driving assistance system.

[0012] The lane marking detection device 100 includes an ECU 20 (Electronic Control Unit). The ECU 20 is an electronic control unit having a CPU (Central Processing Unit) and a storage unit. The storage unit is composed of, for example, a ROM (Read Only Memory), a RAM (Random Access Memory), and an EEPROM (Electrically Erasable Programmable Read-Only Memory). The ECU 20 realizes various functions by having the CPU execute programs stored in the storage unit. The ECU 20 may be composed of multiple electronic units.

[0013] As shown in FIG. 1, the ECU 20 is connected to a GNSS receiver 10, an external camera 11, a radar sensor 12, a map database 13, a lane marking database 14, and an HMI 15 [Human Machine Interface].

[0014] The GNSS receiver 10 receives signals from positioning satellites to measure the position of the vehicle 1 (for example, the latitude and longitude of the vehicle 1). The GNSS receiver 10 transmits the measured position information of the vehicle 1 to the ECU 20.

[0015] The external camera 11 is an imaging device (external sensor) that captures an image of an external situation of the host vehicle 1. The external camera 11 is provided, for example, on the back side of the windshield of the host vehicle 1, and captures an image in front of the host vehicle 1. The external camera 11 may be provided on the side or rear of the host vehicle 1, and configured to be able to capture an image of the surroundings of the host vehicle 1. The external camera 11 transmits the captured image of the outside of the host vehicle 1 to the ECU 20.

[0016] The radar sensor 12 is a detection device (external sensor) that detects objects around the vehicle 1 by using radio waves (e.g., millimeter waves) or light. The radar sensor 12 includes, for example, a millimeter wave radar or a LiDAR (Light Detection and Ranging) provided in multiple directions around the vehicle 1. The radar sensor 12 detects objects by transmitting radio waves or light to the surroundings of the vehicle and receiving the radio waves or light reflected by the objects. The radar sensor 12 transmits information about the detected objects to the ECU 20. The objects include lane markings.

[0017] The map database 13 is a database that stores map information. The map database 13 is formed, for example, in a storage device such as a HDD (Hard Disk Drive) mounted on the vehicle 1. The map information includes road position information, road shape information (for example, types of curves, straight sections, curvature of curves, etc.), and position information of intersections and branching points.

[0018] The map information also includes information about the lane markings of the road, including position information about the lane markings. The information about the lane markings may include information about the type of lane markings (solid line, broken line, double solid line, roadway center line, lane boundary line, etc.). The map database 13 is not limited to being mounted on the vehicle 1, and may be formed in a server capable of communicating with the vehicle 1.

[0019] The lane line database 14 is a database that stores past lane line detection information. The lane line database 14 is also formed, for example, in a storage device of the host vehicle 1. The past lane line detection information is position information of lane lines detected in the past by the lane line detection device 100. The past lane line detection information is stored in association with the position of the host vehicle 1 on a map. Note that the lane line database 14 may be configured to be capable of sharing past lane line detection information acquired by other vehicles via a server. The lane line database 14 is not limited to being mounted on the host vehicle 1, and may be formed in a server capable of communicating with the host vehicle 1.

[0020] Note that, when the map information includes the lane line information, the ECU 20 of the lane line detection device 100 does not necessarily need to be connected to the lane line database 14. Conversely, when the lane line detection device 100 can acquire past lane line detection information, the map information does not need to include the lane line information.

[0021] The HMI 15 is an interface for inputting and outputting information between the ECU 20 and the driver. The HMI 15 includes, for example, a display and a speaker provided in the vehicle cabin. The HMI 15 outputs images from the display and sounds from the speaker in response to a control signal from the ECU 20. The display may be a multi-information display (MID) or a head-up display (HUD). The HMI 15 may include various indicators.

[0022] Next, a description will be given of the functional configuration of the ECU 20. As shown in Fig. 1, the ECU 20 has a known lane marking recognition unit 21, a lane marking detection unit 22, a model abnormality determination unit 23, a sensor abnormality determination unit 24, and an information provision unit 25.

[0023] The known lane line recognition unit 21 recognizes known lane lines ahead of the vehicle 1 based on past lane line detection information or map information including lane line information stored in the lane line database 14 and position information of the vehicle 1 measured by the GNSS receiver unit 10. A known lane line is a lane line that has become known from past lane line detection information or map information. Note that the position information of the vehicle 1 may be obtained by SLAM [Simultaneous Localization and Mapping] or the like.

[0024] The known lane line recognition unit 21 recognizes known lane lines ahead of the host vehicle 1, for example, based on past lane line detection information and position information of the host vehicle 1. The known lane line recognition unit 21 acquires past lane line detection information corresponding to a certain distance ahead of the host vehicle 1 from the lane line database 14 using the position information of the host vehicle 1. The known lane line recognition unit 21 recognizes known lane lines ahead of the host vehicle 1, for example, by projecting the positions of the lane lines in the past lane line detection information onto a map coordinate system based on the position of the host vehicle 1.

[0025] The known lane line recognition unit 21 may recognize known lane lines ahead of the vehicle 1 based on map information including lane line information and position information of the vehicle 1. The known lane line recognition unit 21 acquires lane line information corresponding to a certain distance ahead of the vehicle 1 from the map information. The known lane line recognition unit 21 recognizes known lane lines ahead of the vehicle 1, for example, by projecting the lane lines included in the lane line information onto a map coordinate system based on the position of the vehicle 1. Note that the method of recognizing known lane lines is not limited to the above.

[0026] The known lane line recognition unit 21 may determine whether or not the map information contains lane line information corresponding to the current position of the vehicle 1, based on the position information and map information of the vehicle 1. If the map information contains lane line information corresponding to the current position of the vehicle 1, the known lane line recognition unit 21 may prioritize the lane line information in the map information rather than past lane line detection information in the lane line database 14. In this case, the known lane line recognition unit 21 recognizes the known lane lines based on the map information.

[0027] Alternatively, the known lane line recognition unit 21 may compare the update date and time of the map information corresponding to the current position of the vehicle 1 with the acquisition date and time of past lane line detection information in the lane line database 14 corresponding to the current position of the vehicle 1, and give priority to the newer one. If the acquisition date and time of the past lane line detection information is newer than the update date and time of the map information, the known lane line recognition unit 21 recognizes the known lane line based on the past lane line detection information in the lane line database 14.

[0028] The lane marking detection unit 22 detects lane marks ahead of the host vehicle 1 from a lane marking detection model 22 a based on at least one of the captured image of the external camera 11 and the detection result of the radar sensor 12 .

[0029] The lane marking detection model 22a is a machine learning model that has been trained by deep learning so as to output a lane marking detection result from at least one of the image captured by the external camera 11 and the detection result of the radar sensor 12. The lane marking detection model 22a is configured from a neural network such as a convolutional neural network [CNN: Convolutional Neural Network]. The neural network may include multiple layers including multiple convolution layers and pooling layers. The lane marking detection model 22a may be configured using a recurrent neural network [RNN: Recurrent neural network] that also uses past detection results including the detection result of the previous lane marking as input.

[0030] The lane marking detection unit 22 inputs at least one of the captured image of the external camera 11 and the detection results of the radar sensor 12 into the lane marking detection model 22a, and obtains the detection results of the lane markings ahead of the vehicle 1 as output from the lane marking detection model 22a.

[0031] The model abnormality judgment unit 23 judges whether or not there is an abnormality in the lane line detection model 22a when the amount of deviation in the width direction between the known lane line recognized by the known lane line recognition unit 21 and the lane line detected by the lane line detection unit 22 is equal to or greater than the abnormality judgment threshold. The abnormality judgment threshold is a threshold value set in advance for judging an abnormality in the lane line detection. The deviation amount can be calculated as the average, median or maximum value of the width direction separation distance between the nearest known lane line and the lane line. The deviation amount is measured, for example, in a plane coordinate system or a map coordinate system (geographic coordinate system) based on the vehicle 1. When the known lane line and lane line are detected on the left and right sides of the vehicle 1, the larger deviation amount of the left and right known lane lines and lane lines is used for the judgment.

[0032] The model anomaly determination unit 23 determines whether or not there is an anomaly in the lane line detection model 22a based on the known lane lines recognized by the known lane line recognition unit 21 and the lane lines detected by the lane line detection unit 22. Specifically, the model anomaly determination unit 23 determines that there is an anomaly in the lane line detection model 22a, for example, when the comparison result between the known lane lines and the lane lines resembles a model anomaly determination pattern stored in advance. The model anomaly determination pattern is a recorded specific pattern that is seen when the lane line detection model 22a indicates an anomaly. When the lane line detection model 22a is functioning normally, the known lane lines and the lane lines detected by the lane line detection model 22a match or are similar. On the other hand, when the lane line detection model 22a becomes abnormal, the deviation (divergence) between the known lane lines and the lane lines detected by the lane line detection model 22a increases.

[0033] In this case, the comparison result between the known lane markings and the lane markings has a tendency different from the erroneous detection of the lane markings when the external sensor is abnormal. Therefore, by focusing on the comparison result between the known lane markings and the lane markings, when the deviation between the known lane markings and the lane markings is large, it is possible to determine whether the deviation is due to an abnormality in the external sensor or an abnormality in the lane marking detection model 22a.

[0034] Here, Fig. 2 is a diagram showing an example of a model abnormality determination pattern. Fig. 2 shows a real left lane line La and a real right lane line Lb, which are the actual left lane lines in front of the vehicle 1, a known left lane line Ka and a known right lane line Kb, and a left lane line Da and a right lane line Db detected by the lane line detection unit 22. The known left lane line Ka and the known right lane line Kb approximately coincide with the real left lane line La and the real right lane line Lb.

[0035] In the situation shown in Fig. 2, the left-side lane line Da and the right-side lane line Db are detected as meandering curves. When such meandering lane lines are detected, it is highly likely that there is an abnormality in the lane line detection model 22a, not in the external sensor. When the left-side lane line Da and the right-side lane line Db are detected as meandering lines compared to the left-side known lane line Ka and the right-side known lane line Kb, which are substantially straight lines, the model abnormality determination unit 23 determines that there is an abnormality in the lane line detection model 22a, as it is similar to the model abnormality determination pattern shown in Fig. 2.

[0036] Fig. 3 is a diagram showing another example of the model abnormality determination pattern. In Fig. 3, the left-side lane line Da and the right-side lane line Db are detected as turning back in an arch shape in the longitudinal direction of the vehicle 1. When such turned-back lane lines are detected, there is a high possibility that the lane line detection model 22a is abnormal. When the left-side lane line Da and the right-side lane line Db are detected as turning back in the longitudinal direction compared with the left-side known lane line Ka and the right-side known lane line Kb, which are substantially straight lines, the model abnormality determination unit 23 may determine that there is an abnormality in the lane line detection model 22a as being similar to the model abnormality determination pattern shown in Fig. 3.

[0037] The model abnormality determination pattern is not limited to the aspects shown in Figures 2 and 3, and various patterns are included. The model abnormality determination pattern may be set based on the simulation results of the lane marking detection model 22a or the detection results in actual test driving.

[0038] In addition, the model abnormality determination pattern may include a pattern in which the left-side lane line Da and / or the right-side lane line Db detected by the lane line detection unit 22 are located outside the detection range of the external sensor (e.g., outside the imaging range of the external camera 11) even though the known lane line is located within the detection range of the external sensor. If the left-side lane line Da and / or the right-side lane line Db are located outside the detection range of the external sensor, there is a high possibility that there is an abnormality in the lane line detection model 22a.

[0039] The model anomaly determination unit 23 may determine the presence or absence of an anomaly in the lane line detection model 22a from the comparison result between the known lane line and the lane line, regardless of the model anomaly determination pattern. The model anomaly determination unit 23 may be configured to determine that there is an anomaly in the lane line detection model 22a when the known lane line is a substantially straight line or curved line that does not meander, the maximum value of the width between the right end and the left end of the meandering lane line detected by the lane line detection unit 22 is equal to or greater than a certain value, and the number of meanders (corresponding to the shape order of the lane line) is equal to or greater than a certain number, when the known lane line is a substantially straight line or curved line that does not meander.

[0040] Similarly, the model anomaly determination unit 23 may be configured to determine that there is an anomaly in the lane line detection model 22a when the known lane line is substantially straight or curved (a curve with less than a certain curvature that does not include a sharp curve) and includes a plurality of coordinate points that are horizontally spaced apart by turning back as shown in FIG. 3. The model anomaly determination unit 23 may be configured to determine that there is an anomaly in the lane line detection model 22a when the number of the plurality of coordinate points that are horizontally spaced apart is equal to or greater than a threshold value. The coordinate points may be counted in a planar coordinate system or a map coordinate system, or may be counted on a captured image. The model anomaly determination unit 23 may be configured to determine that there is an anomaly in the lane line detection model 22a when the lane line detected by the lane line detection unit 22 is outside the detection range of the external sensor, even though the known lane line is within the detection range of the external sensor.

[0041] In addition, the model abnormality judgment unit 23 may be configured to judge that there is an abnormality in the lane line detection model 22a when the amount of widthwise deviation between the known lane line recognized by the known lane line recognition unit 21 and the lane line detected by the lane line detection unit 22 is equal to or greater than the abnormality judgment threshold (image display threshold) and the sensor abnormality judgment unit 24 described later judges that there is no abnormality in the external sensor.

[0042] The sensor abnormality determination unit 24 determines the presence or absence of an abnormality in an external sensor such as the external camera 11 or the radar sensor 12 when the amount of deviation in the width direction between the known lane mark recognized by the known lane mark recognition unit 21 and the lane mark detected by the lane mark detection unit 22 is equal to or greater than the abnormality determination threshold. The abnormality determination threshold may be the same value as the abnormality determination threshold of the model abnormality determination unit 23, or may be a different value.

[0043] The sensor abnormality determination unit 24 determines that an abnormality exists in the external sensor, for example, when an abnormal signal is transmitted from the external camera 11 or the radar sensor 12. The sensor abnormality determination unit 24 may determine that an abnormality exists in the external camera 11, when frequent noise, abnormal brightness, abnormal chromaticity, or the like occurs in the image captured by the external camera 11. The sensor abnormality determination unit 24 may determine that an abnormality exists in the radar sensor 12, based on the detection result of the radar sensor 12, when frequent noise occurs, clutter sticks, or the like occurs.

[0044] The sensor abnormality determination unit 24 may determine whether or not a preset sensor abnormality determination condition is satisfied based on the captured image of the external camera 11 or the detection result of the radar sensor 12. The sensor abnormality determination condition is a condition used to determine an abnormality of the external sensor of the vehicle 1. The sensor abnormality determination unit 24 determines that the sensor abnormality determination condition is satisfied when, for example, frequent noise, brightness abnormality, or chromaticity abnormality occurs in the image captured by the external camera 11 or frequent noise, clutter fixation, or the like occurs in the radar sensor 12.

[0045] Furthermore, the sensor abnormality determination unit 24 determines that the sensor abnormality determination condition is satisfied when an abnormal area is detected continuously in a part of the imaging range of the external camera 11 for a certain driving time. The detection range of the radar sensor 12 may be used instead of the imaging range of the external camera 11. The abnormal area is, for example, an area where a detection abnormality occurs on the imaging image of the external camera 11 due to dirt or water droplets adhering to the lens of the external camera 11. The abnormal area may be an area where a detection abnormality occurs within the detection range of the radar sensor 12 due to dirt or water droplets adhering to the detection section of the radar sensor 12. The driving time is the time counted while the host vehicle 1 is driving. When the host vehicle 1 is stopped, the driving time is not counted.

[0046] Fig. 4 is a diagram showing an example of a case where a marking line is erroneously detected due to dirt. Fig. 4 shows an abnormal area F caused by dirt on the lens. In the situation shown in Fig. 4, the marking line cannot be detected due to the influence of the abnormal area F (dirt), and the curb is erroneously detected as the marking line Db.

[0047] The sensor abnormality determination unit 24 recognizes as an abnormal area an area where the amount of deviation in the width direction between the known lane marking and the lane marking is equal to or greater than the sensor abnormality threshold, or an area (area on the imaging range) where the known lane marking is present but is not detected. The sensor abnormality threshold is a threshold value that is smaller than the abnormality determination threshold. The sensor abnormality determination unit 24 determines that the sensor abnormality determination condition is satisfied when an abnormal area F is detected continuously at a predetermined position within the imaging range for a certain driving time even though the vehicle 1 is traveling.

[0048] Abnormal areas can also be caused by water droplets. Water droplets do not directly block the detection of the lane markings, and it may be possible to detect the lane markings. However, light refraction can distort the position of the lane markings, resulting in an incorrect detection.

[0049] When the amount of deviation in the width direction between the known lane marking recognized by the known lane marking recognition unit 21 and the lane marking detected by the lane marking detection unit 22 is equal to or greater than the image display threshold, the information providing unit 25 displays an image of the known lane marking and the lane marking to the user of the vehicle 1. The image display threshold is a threshold set for determining whether to display an image of the known lane marking and the lane marking. The image display threshold may be the same value as the abnormality determination threshold, or may be a value smaller than the abnormality determination threshold.

[0050] The information providing unit 25 controls the HMI 15 to display images of the known lane markings and the lane markings on the display of the HMI 15. The information providing unit 25 performs image display as shown in Figs. 2 to 4, for example. When an abnormal area F is detected by the sensor abnormality determination unit 24 as shown in Fig. 4, the information providing unit 25 may display the abnormal area F. By displaying both the known lane markings (past lane markings that serve as reference) and the lane markings (lane markings that are currently detected by the vehicle 1 and are used for autonomous driving and driving assistance) in an image, the user can understand the detection status of the lane markings by comparing them with the known lane markings.

[0051] The information providing unit 25 does not necessarily have to display an image superimposed on the view ahead of the vehicle 1 as shown in Figs. 2 to 4. The information providing unit 25 may be in a mode in which the known lane markings and the lane markings are displayed as images in a bird's-eye view or plan view. The information providing unit 25 may be in a mode in which the display corresponding to the vehicle 1 and the known lane markings and the lane markings are virtually displayed in a bird's-eye view or plan view.

[0052] When the known lane marking recognition unit 21 recognizes a known lane marking ahead of the vehicle 1 but the lane marking detection unit 22 cannot detect the lane marking ahead of the vehicle 1, the information providing unit 25 may notify the user of a lane marking detection error. The notification of a lane marking detection error is a notification to inform the user that a lane marking cannot be detected despite the existence of a known lane marking. When a lane marking cannot be detected in a location where a known lane marking has been recognized, with the exception of road construction sections, there is a high possibility that an error exists in the lane marking detection device 100. For this reason, the information providing unit 25 controls the HMI 15 to notify the user of the lane marking detection error by displaying an image on the display and / or outputting an audio signal from the speaker.

[0053] When the model abnormality determination unit 23 determines that there is an abnormality in the lane line detection model 22a, the information provision unit 25 may notify the user of the model abnormality. The notification of the model abnormality is a notification to inform the user that there is an abnormality in the lane line detection model 22a. The information provision unit 25 controls the HMI 15 to notify the user of the model abnormality by displaying an image on the display and / or outputting an audio message from the speaker.

[0054] When the sensor abnormality determination unit 24 determines that there is an abnormality in the external sensor, the information providing unit 25 may notify the user of the sensor abnormality. The notification of the sensor abnormality is a notification to inform the user that there is an abnormality in the external sensor. The information providing unit 25 controls the HMI 15 to notify the user of the sensor abnormality by displaying an image on the display and / or outputting audio from the speaker.

[0055] When the sensor abnormality determination unit 24 detects the abnormal area F, the information providing unit 25 may execute a complementation process (rewriting process) of the abnormal area F when displaying an image to the user. The complementation process is a process of replacing the lane markings (the lane markings currently detected by the vehicle 1 and used for autonomous driving or driving assistance) within the abnormal area F in the imaging range with known lane markings.

[0056] For example, when the sensor abnormality determination unit 24 detects an abnormal area F, the information providing unit 25 may read the portions of past known lane lines corresponding to the current position of the vehicle that correspond to the abnormal area F, and detect those portions as lane lines. Note that the information providing unit 25 may complement a rectangular area including the abnormal area F, may complement the lower half of the imaging range including the abnormal area F, or may complement the entire imaging range. The information providing unit 25 displays to the user the known lane lines and an image including the lane lines that have been subjected to the complementing process.

[0057] [Processing method of lane marking detection device] Next, a processing method of the lane marking detection device 100 according to this embodiment will be described with reference to the drawings. Fig. 5 is a flowchart showing an example of lane marking image display processing. The lane marking image display processing is executed, for example, when the host vehicle 1 needs to detect lane marks in autonomous driving or driving assistance.

[0058] As shown in Fig. 5, in S10, the ECU 20 of the lane line detection device 100 recognizes known lane lines ahead of the vehicle 1 using the known lane line recognition unit 21. The known lane line recognition unit 21 recognizes known lane lines ahead of the vehicle 1 based on map information including past lane line detection information or lane line information stored in the lane line database 14 and position information of the vehicle 1 measured by the GNSS receiver unit 10. The ECU 20 then proceeds to S11.

[0059] In S11, the ECU 20 detects a lane marking ahead of the vehicle 1 using the lane marking detection unit 22. The lane marking detection unit 22 inputs at least one of the captured image of the external camera 11 and the detection result of the radar sensor 12 to the lane marking detection model 22a, thereby acquiring the detection result of the lane marking ahead of the vehicle 1 as an output from the lane marking detection model 22a. After that, the ECU 20 proceeds to S12.

[0060] In S12, the ECU 20 determines whether or not the demarcation line has not been detected at the location where the known demarcation line was recognized by the information providing unit 25. If the ECU 20 determines that the demarcation line has not been detected at the location where the known demarcation line was recognized, the ECU 20 proceeds to S13. If the ECU 20 does not determine that the demarcation line has not been detected at the location where the known demarcation line was recognized (if the demarcation line has been detected), the ECU 20 proceeds to S14.

[0061] In S13, the ECU 20 notifies the user of the lane marking detection error through the information providing unit 25. The information providing unit 25 notifies the user of the lane marking detection error through image display and / or audio output by controlling the HMI 15. After that, the ECU 20 ends the lane marking image display process.

[0062] In S14, the ECU 20 determines whether the amount of widthwise deviation between the known demarcation line recognized by the known demarcation line recognition unit 21 and the demarcation line detected by the demarcation line detection unit 22 by the information provision unit 25 is equal to or greater than the image display threshold. If the ECU 20 determines that the amount of widthwise deviation between the known demarcation line and the demarcation line is equal to or greater than the image display threshold, the ECU 20 proceeds to S15. If the ECU 20 does not determine that the amount of widthwise deviation between the known demarcation line and the demarcation line is equal to or greater than the image display threshold, the ECU 20 terminates the demarcation line image display processing.

[0063] In S15, the ECU 20 causes the information providing unit 25 to display images of the known lane markings and the lane markings to the user of the vehicle 1. The information providing unit 25 controls the HMI 15 to display images of the known lane markings and the lane markings on the display of the HMI 15. The information providing unit 25 performs image display as shown in Figs. 2 to 4, for example. Thereafter, the ECU 20 ends the lane marking image display process.

[0064] Fig. 6 is a flowchart showing an example of an abnormality notification process. The abnormality notification process is executed, for example, when the image display of S15 in Fig. 5 is performed. The abnormality notification process may be executed when a lane marking is detected in autonomous driving or driving assistance, regardless of the image display.

[0065] 6, in S20, the ECU 20 determines whether or not the sensor abnormality determination condition is satisfied by the sensor abnormality determination unit 24. The sensor abnormality determination unit 24 determines that the sensor abnormality determination condition is satisfied when, for example, an abnormal area is detected continuously for a certain driving time in a part of the imaging range of the external camera 11. If it is determined that the sensor abnormality determination condition is satisfied, the ECU 20 proceeds to S21. If it is not determined that the sensor abnormality determination condition is satisfied, the ECU 20 proceeds to S22.

[0066] In S21, the ECU 20 notifies the user of the vehicle 1 of the sensor abnormality through the information providing unit 25. The information providing unit 25 notifies the user of the sensor abnormality through image display and / or audio output by controlling the HMI 15. After that, the ECU 20 ends the abnormality notification process.

[0067] In S22, the ECU 20 determines whether the comparison result between the known lane marking and the lane marking is similar to the model abnormality judgment pattern by the model abnormality judgment unit 23. If the ECU 20 determines that the comparison result between the known lane marking and the lane marking is similar to the model abnormality judgment pattern, the ECU 20 proceeds to S23. If the ECU 20 does not determine that the comparison result between the known lane marking and the lane marking is similar to the model abnormality judgment pattern, the ECU 20 ends the abnormality notification process.

[0068] In S23, the ECU 20 notifies the user of the vehicle 1 of the model abnormality through the information providing unit 25. The information providing unit 25 notifies the user of the model abnormality through image display and / or audio output by controlling the HMI 15. After that, the ECU 20 ends the abnormality notification process.

[0069] According to the lane detection device 100 of this embodiment described above, when there is a difference between a previously detected lane line or a known lane line, which is a lane line on a map, and a currently detected lane line, an image of the known lane line and the lane line can be displayed to the user, thereby appropriately informing the user that there is a discrepancy in the lane line detection results.

[0070] In addition, according to the lane marking detection device 100, if a known lane marking ahead of the vehicle 1 is recognized but the lane marking detection unit 22 is unable to detect the lane marking ahead of the vehicle 1, the user is notified that a known lane marking exists but the lane marking cannot be detected, allowing the user to understand the detection status of the lane marking.

[0071] Furthermore, according to the lane line detection device 100, the model anomaly determination unit 23 determines whether or not there is an anomaly in the lane line detection model 22a from the comparison result between the known lane lines and the lane lines, so that the user can be notified of the model anomaly. Similarly, according to the lane line detection device 100, when the sensor anomaly determination unit 24 determines that the sensor anomaly determination condition is satisfied, the user is notified of the sensor anomaly, so that the user can become aware of the anomaly of the external sensor. The sensor anomaly determination unit 24 can determine an anomaly of the external sensor when an abnormal area is detected continuously for a certain period of time in at least a part of the detection range of the external sensor.

[0072] Although the embodiment of the present invention has been described above, the present invention is not limited to the above-mentioned embodiment. The present invention can be embodied in various forms including the above-mentioned embodiment and various modifications and improvements based on the knowledge of those skilled in the art.

[0073] The lane line detection device 100 does not necessarily have to include the model abnormality determination unit 23. Similarly, the lane line detection device 100 does not necessarily have to include the sensor abnormality determination unit 24. Furthermore, when a known lane line ahead of the host vehicle 1 is recognized but the lane line detection unit 22 cannot detect the lane line ahead of the host vehicle 1, the lane line detection device 100 does not necessarily have to notify the user. [Explanation of symbols]

[0074] 1... host vehicle, 20... ECU, 21... known lane marking recognition unit, 22... lane marking detection unit, 22a... lane marking detection model, 23... model abnormality judgment unit, 24... sensor abnormality judgment unit, 25... information provision unit, 100... lane marking detection device, Da, Db... lane markings, F... abnormality area, Ka, Kb... known lane markings.

Claims

1. A lane marking detection device that detects a lane marking ahead of a vehicle using a lane marking detection model that is a machine learning model used for lane marking detection, a known lane marking recognition unit that recognizes a known lane marking, which is a known lane marking ahead of the vehicle, based on map information including past lane marking detection information or lane marking information and position information of the vehicle; a lane marking detection unit that detects a lane marking ahead of the host vehicle using the lane marking detection model based on a detection result from an external sensor of the host vehicle; an information providing unit that displays images of the known lane marking and the lane marking to a user of the vehicle when a deviation amount in a width direction between the known lane marking and the lane marking is equal to or greater than an image display threshold; A lane marking detection device comprising:

2. The lane detection device of claim 1, wherein when the known lane line recognition unit recognizes the known lane line ahead of the vehicle but the lane line detection unit cannot detect the lane line ahead of the vehicle, the information providing unit notifies the user that the lane line cannot be detected despite the existence of the known lane line.

3. The lane line detection device according to claim 1 or 2, further comprising a model abnormality judgment unit that judges that there is an abnormality in the lane line detection model when the comparison result between the known lane line and the lane line is similar to a pre-stored model abnormality judgment pattern.

4. a sensor abnormality determination unit that determines that the external sensor is abnormal when an abnormal area is detected continuously for a certain running time in at least a part of the detection range of the external sensor, The lane marking detection device of claim 1 or 2, wherein the abnormal area is an area in which the widthwise deviation between the known lane marking and the lane marking is greater than or equal to a sensor abnormality threshold, or an area in which the known lane marking is present but the lane marking is not detected.

5. a sensor abnormality determination unit that determines that the external sensor is abnormal when an abnormal area is detected continuously for a certain running time in at least a part of a detection range of the external sensor; and a model anomaly determination unit that determines that there is an anomaly in the lane line detection model when the sensor anomaly determination unit does not determine that there is an anomaly in the external sensor in a case where the deviation amount between the known lane line and the lane line is equal to or greater than the image display threshold value, The lane marking detection device of claim 1 or 2, wherein the abnormal area is an area in which the widthwise deviation between the known lane marking and the lane marking is greater than or equal to a sensor abnormality threshold, or an area in which the known lane marking is present but the lane marking is not detected.

Images