Lane marking recognition device

Abstract

To properly recognize a lane marking.SOLUTION: A lane marking recognition apparatus 10 includes: a mark recognition unit 13 which recognizes a mark on a road on the basis of detection information detected by an in-vehicle detector 2 that detects surroundings of a vehicle; and a lane marking recognition unit 14 which recognizes a lane marking on the road on the basis of the mark recognized by the mark recognition unit 13. The lane marking recognition unit 14 sets a coordinate system in accordance with a current position and an advancing direction of the vehicle, generates a function to approximate multiple marks in the coordinate system after excluding a mark that is located a predetermined interval or more away from other marks in a direction which intersects the advancing direction, out of multiple marks recognized by the mark recognition unit 13, and interpolates a lane marking in a section that is not recognized by the mark recognition unit 13 on the basis of a curve of the function.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a lane marking recognition device that recognizes lane markings on which a vehicle having an automatic driving function or a driving assistance function is traveling. [Background technology]

[0002] One known example of this type of device is one that recognizes lane markings based on camera images (see, for example, Patent Document 1). In conventional technology, road shapes are approximated using polynomial approximation functions based on the results of lane marking detection by a video camera. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Special Publication No. 2021-508901 Summary of the Invention [Problem to be solved by the invention]

[0004] With conventional technology, there were cases where the information obtained from the camera images did not allow for accurate recognition of lane markings. [Means for solving the problem]

[0005] A lane marking recognition device according to one aspect of the present invention includes a marking recognition unit that recognizes markings on a road based on detection information detected by an on-board detector that detects the situation around the vehicle, and a lane marking recognition unit that recognizes lane markings on the road based on the markings recognized by the marking recognition unit. a distance information acquisition unit that acquires distance information from the vehicle to the sign; The lane marking recognition unit sets a coordinate system according to the current position and traveling direction of the vehicle, and excludes, from among the plurality of signs recognized by the sign recognition unit, signs that deviate from other signs by a predetermined distance or more in a direction intersecting with the traveling direction, and then generates a function that approximates the plurality of signs in the coordinate system, and interpolates the lane markings of sections not recognized by the sign recognition unit based on the curve of the function. At the same time, the lane marking is recognized based on a first marking recognized in a first distance range on the vehicle side and a second marking recognized in a second distance range farther than the first distance range. . [Effects of the Invention]

[0006] According to the present invention, it is possible to appropriately recognize lane markings. [Brief explanation of the drawings]

[0007] [Figure 1] 1 is a block diagram illustrating a schematic configuration of a lane marking recognition device according to an embodiment of the present invention; [Figure 2A] 4A and 4B are diagrams illustrating recognition of lane lines by a lane line recognition unit. [Figure 2B] 4A and 4B are diagrams illustrating recognition of lane lines by a lane line recognition unit. [Figure 3A] 4A and 4B are diagrams illustrating recognition of lane lines by a lane line recognition unit. [Figure 3B] 4A and 4B are diagrams illustrating recognition of lane lines by a lane line recognition unit. [Figure 4A] 6 is a flowchart illustrating an example of processing by a program executed in a calculation unit. [Figure 4B] 10 is a flowchart illustrating step S30 in detail. DETAILED DESCRIPTION OF THE INVENTION

[0008] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. The lane marking recognition device according to an embodiment of the invention is applied to vehicles that have driving assistance functions or automatic driving functions for the vehicle driver, and recognizes lane markings that define the lane in which the vehicle is traveling (which may also be called a lane). The vehicle to which the lane marking recognition device according to the embodiment is applied may be referred to as the host vehicle to distinguish it from other vehicles. The host vehicle may be an engine vehicle having an internal combustion engine (engine) as a driving source, an electric vehicle having a traction motor as a driving source, or a hybrid vehicle having an engine and a traction motor as driving sources.

[0009] "Driving assistance" in the embodiments corresponds to assistance for the driver's driving operation, and is equivalent to automated driving levels 1 to 4 defined by the Society of Automotive Engineers (SAE). Furthermore, "automated driving" in the embodiments corresponds to driving control of the vehicle independent of the driver's driving operation, and is equivalent to automated driving level 5 defined by the SAE.

[0010] When the vehicle is undergoing driving assistance or autonomous driving, its driving behavior, such as driving speed and direction of travel, and the external environment around the vehicle are repeatedly detected at predetermined intervals (which may also be called periods), and a target driving route for the vehicle is generated according to the detected information.The vehicle is then controlled to travel along the generated target driving route.

[0011] The external environment of the vehicle is detected by external sensors such as a camera, a LiDAR, etc. The external sensors are attached to the vehicle at a predetermined mounting position and angle (which may also be called attitude information) during the manufacture of the vehicle.

[0012] The lane marking recognition device estimates and recognizes the positions of external objects, including road markings, based on detection information from the external sensors, taking into account the attitude information of the external sensors. Road markings include white lines (including lines of different colors such as yellow), curb lines, road studs, lane dividers, etc., and these markings define the driving lanes of the road. Road white lines include road exterior lines and lane boundary lines, etc. Curb lines are lines made of stones or the like that separate the roadway from the sidewalk, etc. In the embodiment, the lines corresponding to the boundary lines of the driving lanes defined by the above markings are called dividing lines.

[0013] Generally, when lane markings are recognized based on detection information from an external sensor while the vehicle is traveling, the recognition results (position estimation results) of the lane markings contain a certain amount of error. Furthermore, because the external sensor repeatedly detects the external conditions of the vehicle at a predetermined cycle, variations occur between detection information obtained at different times. Therefore, when lane markings are recognized by simply superimposing detection information from multiple times, the recognition error increases and the recognized lane markings vary greatly. In order to reduce such recognition errors and variations, in the embodiment, detection information obtained by external sensors and detected near the vehicle with high accuracy is used preferentially.

[0014] Furthermore, on actual roads, there are cases where markings on the road are partially missing due to peeling or fading, or where markings are obscured by vehicles around the vehicle, etc. As a result, part of the lane markings are recognized as broken, discontinuous lane marks. In order to interpolate such discontinuous sections of the lane markings, in this embodiment, the lane marking shape is estimated using an approximate curve estimation method. In this process, detection information corresponding to so-called outliers (markings detected in a direction intersecting the direction of travel (called the vehicle width direction) that are separated from other marks by a predetermined distance or more) is excluded before the lane marking shape is estimated and the discontinuous sections of the lane markings are interpolated. Such a lane marking recognition device will be described in detail below.

[0015] FIG. 1 is a block diagram that shows a schematic diagram of a lane marking recognition device 10 according to an embodiment of the invention. As shown in FIG. 1, the lane marking recognition device 10 is mainly composed of an electronic control unit (ECU). The ECU is configured as a computer having a processing unit 11 such as a CPU, a storage unit 12 such as RAM and ROM, an I / O interface, and other peripheral circuits. The ECU that constitutes the lane marking recognition device 10 is configured as part of a group of multiple ECUs that are mounted on the host vehicle (vehicle 1) and control the operation of the host vehicle. The lane marking recognition process by the lane marking recognition device 10 is initiated, for example, when the host vehicle is started or started and the ECU is activated, and is repeatedly executed at a predetermined interval.

[0016] The lane marking recognition device 10 is connected to an external sensor 2 that is mounted on the vehicle and detects the external situation of the vehicle, a driving actuator 3, and a behavior sensor 4. The external sensor 2 detects the external environment ahead of the vehicle, centered on the direction of travel of the vehicle. The external sensor 2 is configured by a camera having an imaging element (image sensor) such as a CCD or CMOS, and capturing images of the area ahead of the vehicle. The camera may be a monocular camera or a stereo camera, and continuously captures images of the space ahead of the vehicle at a predetermined frame rate, and sequentially outputs frame image data (simply referred to as camera images) as detection information to the calculation unit 11. In addition, the external sensor 2 may be configured as a lidar that emits laser light and measures the distance and direction to an object from the time it takes for the emitted light to hit the object and return, and detects the reflected brightness at each measurement point.

[0017] The calculation unit 11 estimates the position (current position) of the vehicle based on the camera image as detection information detected by the external sensor 2, for example, according to an algorithm of SLAM (Simultaneous Localization and Mapping) technology, and creates a map including the lane markings around the vehicle.

[0018] The driving actuator 3 includes a steering mechanism such as a steering gear that steers the vehicle, a drive mechanism such as an engine or a motor that drives the vehicle, and a braking mechanism such as a brake that brakes the vehicle. The behavior sensor 4 detects the driving behavior of the vehicle, such as the driving speed and direction of travel, and outputs a detection signal to the calculation unit 11.

[0019] The ECU calculation unit 11 constituting the lane marking recognition device 10 has, as functional components, a marking recognition unit 13, a lane marking recognition unit 14, a driving control unit 15, and a map information update unit 16. The memory unit 12 stores, in addition to map information for autonomous driving, information such as the vehicle's driving trajectory information, various control programs, and thresholds used in the programs.

[0020] The sign recognition unit 13 recognizes the position of a sign on the road in the traveling direction of the vehicle based on the camera image input from the external sensor 2. The lane marking recognition unit 14 recognizes lane markings corresponding to the boundary lines of the driving lanes defined by the markings, based on the positions of the markings recognized by the marking recognition unit 13.

[0021] The driving control unit 15 controls the driving actuator 3 based on the lane markings recognized by the lane marking recognition unit 14. For example, the driving control unit 15 generates a target driving route for the vehicle so that the vehicle passes through the center of the left and right lane markings, and controls the driving actuator 3 so that the vehicle travels along the generated target driving route.

[0022] The map information update unit 16 updates the position information of the lane markings included in the map information for autonomous driving stored in the memory unit 12 based on the lane markings recognized by the lane marking recognition unit 14. Map information is updated under the condition that the computational load of the driving control unit 15 is below a predetermined value, such as when the engine or driving motor is stopped or parked, immediately after starting (starting) the vehicle, or when the vehicle is in hands-on mode and not in automatic driving mode. When the map information is updated by the map information update unit 16, the calculation unit 11 estimates the position (current position) of the vehicle based on the updated map information and the external environment detected by the external sensor 2. In this case, the accuracy of recognizing the lane markings included in the updated map information improves, thereby improving the accuracy of estimating the position of the vehicle.

[0023] <Recognizing lane markings using markings near the vehicle> 2A and 2B are diagrams illustrating lane markings recognized by the lane marking recognition unit 14. As shown in FIGS. 2A and 2B, the lane marking recognition unit 14 sets a coordinate system in which the position of the host vehicle at a given time is the origin Ot, the X axis represents the direction of travel of the host vehicle, and the Y axis represents the vehicle width direction. FIGS. 2A and 2B show an example in which the host vehicle travels straight ahead in the positive direction on the X axis, with the X axis corresponding to the vehicle's travel trajectory. The lane marking recognition unit 14 uses multiple frames of camera images acquired by the external sensor 2 while the vehicle is traveling to recognize lane markings based on the positions of the marks recognized in each frame of the camera image multiple times (corresponding to the number of camera image frames) at a predetermined cycle (corresponding to the cycle at which the external sensor 2 detects the external situation). In other words, the lane markings around the host vehicle are recognized multiple times (corresponding to the number of camera image frames) at different positions on the road based on the marks recognized from those positions. For example, if the vehicle travels 1 m during the above cycle, the lane markings are recognized multiple times (corresponding to the number of frames of the camera image) based on the markings recognized from that position on the road every 1 m traveled.

[0024] The lane marking L in Figure 2A is a superimposition of lane markings based on the positions of all markings from near to far from the vehicle, recognized in each frame of the camera image, recognized multiple times at the above-mentioned predetermined cycle (corresponding to the number of camera image frames). For convenience, only the lane marking on the left side of the driving lane is shown, but the same applies to the lane marking on the right side of the driving lane.

[0025] In contrast, the lane marking Lb in Figure 2B is a lane marking based on the position of the markings recognized in the camera image of each frame, in a nearby area within a predetermined distance (e.g., 10 m) from the vehicle, recognized multiple times (corresponding to the number of frames in the camera image) at the above-mentioned predetermined period, and then superimposed on top of each other. The distance from the vehicle (external sensor 2) to the sign is estimated based on the position of the sign captured in the camera image. The distance from the vehicle (external sensor 2) to the sign may be calculated based on a detection value of a lidar.

[0026] Comparing Figures 2A and 2B, the lane marking L in Figure 2A has greater variance than the lane marking Lb in Figure 2B. This is thought to be because the accuracy of detection of the position by the external sensor 2 decreases as the distance from the vehicle increases, resulting in greater error in the detection information compared to markings in the vicinity of the vehicle. In this embodiment, to reduce the effect of such errors, the lane marking recognition unit 14 recognizes lane marks based on the positions of markings recognized in the vicinity of the vehicle (within the aforementioned 10 m). The lane marking recognition unit 14 recognizes lane marks based on the positions of markings recognized in the vicinity of the vehicle, at least in curved sections of the road. This is because the variance in recognized lane marks is more likely to be greater in curved sections of the road than in straight sections of the road.

[0027] <Interpolation of sections where no sign is detected> Figure 3 A 2A and 2B are diagrams illustrating other lane markings recognized by the lane marking recognition unit 14. Similarly to FIGS. 2A and 2B, the lane marking recognition unit 14 sets a coordinate system in which the position of the host vehicle at a certain point in time is the origin Ot, the X-axis indicates the direction of travel of the host vehicle, and the Y-axis indicates the vehicle width. FIGS. 3A and 3B show an example in which the host vehicle travels straight ahead in the positive direction on the X-axis, with the X-axis corresponding to the vehicle's travel trajectory. Differences from the case of FIG. 2B include the presence of sections 33 and 34 in which part of the markings on the road are missing, and the markings recognized by the marking recognition unit 13 include markings 31 and 32 that are abnormally positioned due to erroneous detection or the like. The lane marking recognition unit 14 recognizes lane marks based on the positions of signs recognized within the vicinity of the vehicle (within the above-mentioned 10 m).

[0028] The lane marking Lb in Fig. 3A is a superimposition of lane markings based on the positions of the markings recognized in the camera image of each frame, recognized multiple times at the predetermined cycle (corresponding to the number of frames of the camera image) and recognized in the vicinity of the vehicle. Note that, as in Fig. 2B, only the lane marking on the left side of the driving lane is shown, but the same applies to the lane marking on the right side of the driving lane.

[0029] In contrast, the lane marking line Lint in Figure 3B corresponds to a curve that is approximated using a curve fitting method such as the least squares method to determine the positions of the markings recognized in the vicinity of the vehicle from among the markings recognized in the camera images of each frame, after excluding the markings that correspond to the above-mentioned outliers.

[0030] The lane marking recognition unit 14 removes markings corresponding to outliers using, for example, a RANSAC (Random Sample Consensus) algorithm. Furthermore, the road shape of a typical curved road is designed using a clothoid curve, whose curvature changes at a constant rate, and some sections of the clothoid curve corresponding to the road shape can be approximated using a cubic function or the like. Therefore, the lane marking recognition unit 14 generates information on the cubic function and uses this approximate curve to interpolate lane markings in sections where no markings are detected. When approximating the road shape using a cubic function, it is possible to reduce the computational load compared to approximating the road shape using a clothoid curve.

[0031] <Explanation of the flowchart> An example of processing executed by the calculation unit 11 of Fig. 1 in accordance with a predetermined program will be described with reference to the flowcharts of Fig. 4A and Fig. 4B. Fig. 4A shows the lane marking recognition processing, which starts when the ECU is started as described above and is repeated at a predetermined interval. Fig. 4B shows the details of step S30 of Fig. 4A.

[0032] In step S10 of FIG. 4A, the calculation unit 11 acquires a camera image as detection information from the camera (external sensor 2), and the process proceeds to step S20.

[0033] In step S20, the calculation unit 11 causes the sign recognition unit 13 to recognize the sign based on the camera image, and the process proceeds to step S30.

[0034] In step S30, the calculation unit 11 causes the demarcation line recognition unit 14 to recognize the demarcation line based on the marking recognized by the marking recognition unit 13, and then proceeds to step S40.

[0035] In step S40, the calculation unit 11 causes the map information update unit 16 to record (update) the position information of the lane markings to be included in the map information for autonomous driving stored in the memory unit 12 based on the lane markings recognized by the lane marking recognition unit 14, and terminates the processing shown in Figure 4A.

[0036] In step S31 of Fig. 4B, the calculation unit 11 selects a sign that is within a predetermined distance from the vehicle from among the signs recognized by the sign recognition unit 13 based on the camera image, and proceeds to step S32. As described above, the distance from the vehicle to the sign can be estimated based on the distance from the camera (external sensor 2) to the sign. Alternatively, the distance can be calculated based on the detection value of the lidar (external sensor 2).

[0037] In step S32, the calculation unit 11 excludes the indicators corresponding to outliers from the selected indicators, and the process proceeds to step S33.

[0038] In step S33, the calculation unit 11 performs approximate curve estimation based on the positions of the markings after removing the markings corresponding to outliers, recognizes the lane markings, and ends the processing in FIG. 4B.

[0039] According to the embodiment described above, the following effects can be obtained. (1) The lane marking recognition device 10 includes a marking recognition unit 13 that recognizes markings on the road (white lines, road studs, lane dividers, etc.) based on camera images as detection information detected by an external sensor 2, which is an on-board detector that detects the conditions around the vehicle, and a lane marking recognition unit 14 that recognizes lane markings on the road based on the markings recognized by the marking recognition unit 13.The lane marking recognition unit 14 sets a coordinate system according to the current position and direction of travel of the vehicle, and excludes, from the multiple markings recognized by the marking recognition unit 13, markings that deviate from other markings by more than a predetermined distance in the vehicle width direction that intersects with the direction of travel.Then, it generates a function that approximates the multiple markings in the coordinate system, and interpolates the lane markings of sections not recognized by the marking recognition unit 13 based on the curve of the function. With this configuration, it is possible to appropriately interpolate and recognize lane markings even in sections where markings are not recognized by the marking recognition unit 13. By using the position information of lane markings recognized by such lane marking recognition device 10, the accuracy of map information for autonomous driving can be improved.

[0040] (2) The lane marking recognition device 10 described in (1) above further includes a calculation unit 11 as a distance information acquisition unit that acquires distance information from the vehicle to the marking, and the lane marking recognition unit 14 recognizes the lane marking based on the first marking recognized in the nearby area as the first distance area on the vehicle side and the second marking recognized in the second distance area that is farther away than the nearby area. This configuration makes it possible to minimize variations in the recognized lane markings and to appropriately recognize the lane markings.

[0041] (3) In the lane marking recognition device 10 described in (2) above, the lane marking recognition unit 14 excludes, from among the multiple first signs recognized in the nearby area, those that deviate from other signs by more than a predetermined distance in the vehicle width direction that intersects with the direction of travel, and then generates a function that approximates the multiple first signs recognized in the nearby area in a coordinate system, and interpolates the lane markings of sections not recognized by the sign recognition unit 13 based on the curve of the function. This configuration makes it possible to suppress the influence of the first markings in the neighboring areas that correspond to outliers and to appropriately interpolate the lane markings.

[0042] (4) In the lane marking recognition device 10 described in (3) above, the lane marking recognition unit 14 recognizes lane markings at least in curved sections of a road based on the first markings. This configuration makes it possible to reduce the variation in recognized lane markings in curved sections where the variation in recognized lane markings tends to be greater compared to straight sections of the road.

[0043] The above embodiment can be modified in various ways, and modifications will be described below. (Variation 1) In the above description of the embodiment, the vicinity area of ​​the vehicle in which the sign recognition unit 13 recognizes signs is set to within 10 m from the vehicle, but the upper limit of the vicinity range may be changed as appropriate within a range of 5 m to 10 m.

[0044] (Variation 2) In the above explanation, an example was given in which markings whose recognition position (distance from the vehicle) exceeds the vicinity area are not used in recognizing the lane markings, but the following configuration may also be used: That is, a weight greater than 1 is assigned to markings whose recognition position (distance from the vehicle) is within the vicinity area, and a weight less than 1 is assigned to markings whose recognition position (distance from the vehicle) is outside the vicinity area, and then both of these markings are used to recognize the lane markings. Markings that are weighted greater than 1 contribute significantly to lane marking recognition, making it possible to properly recognize the lane markings.

[0045] (Variation 3) The process of step S32 for removing outliers and the process of step S33 for estimating an approximate curve may be omitted, for example, in cases where the behavior sensor 4 that detects the traveling behavior of the vehicle, such as the traveling speed and traveling direction, detects that the vehicle is traveling in a straight line.

[0046] (Variation 4) The process of step S31, which selects signs within the vicinity of the vehicle from among the signs recognized by the sign recognition unit 13, may be omitted if the process of step S32, which removes outliers, and the process of step S33, which estimates an approximate curve, are not omitted.

[0047] The above description is merely an example, and the present invention is not limited to the above-described embodiment and modifications as long as the features of the present invention are not impaired. One or more of the above-described embodiment and modifications can be arbitrarily combined, and modifications can also be combined with each other. [Explanation of symbols]

[0048] 1 vehicle (host vehicle), 2 external sensor, 3 driving actuator, 4 behavior sensor, 10 lane marking recognition device, 11 calculation unit, 12 memory unit, 13 sign recognition unit, 14 lane marking recognition unit, 15 driving control unit, 16 map information update unit

Claims

1. a marking recognition unit that recognizes markings on a road based on detection information detected by an on-board detector that detects the situation around the vehicle; a marking line recognition unit that recognizes marking lines of the road based on the markings recognized by the marking recognition unit; a distance information acquisition unit that acquires distance information from the host vehicle to the sign, The lane marking recognition unit sets a coordinate system according to the current position and direction of travel of the vehicle, and excludes, from the multiple signs recognized by the sign recognition unit, those that are more than a predetermined distance away from other signs in a direction intersecting the direction of travel, then generates a function that approximates the multiple signs in the coordinate system, interpolates the lane markings in sections not recognized by the sign recognition unit based on the curve of the function, and recognizes the lane marking based on the first sign of a first sign recognized in a first distance range on the vehicle side and a second sign recognized in a second distance range farther than the first distance range.

2. In the lane marking recognition device described in claim 1, The lane marking recognition device is characterized in that the lane marking recognition unit excludes, from the multiple first signs recognized in the first distance range, any signs that are more than a predetermined distance away from other signs in a direction intersecting the direction of travel, then generates a function that approximates the multiple first signs recognized in the first distance range in the coordinate system, and interpolates the lane markings in sections not recognized by the sign recognition unit based on the curve of the function.

3. In the lane marking recognition device described in claim 2, The lane marking recognition device is characterized in that the lane marking recognition unit recognizes the lane marking based on the first marking at least in a curved section of the road.

Images