Road marking recognition device

Abstract

The road marking recognition device includes: a sign recognition unit that recognizes signs on the road based on detection information from an onboard detector that detects the surrounding conditions of the vehicle; and a road marking recognition unit that recognizes road markings based on the signs recognized by the sign recognition unit. The road marking recognition unit sets up a coordinate system corresponding to the vehicle's current position and direction of travel. After removing signs identified by the sign recognition unit that deviate from the other signs by a predetermined interval in a direction intersecting the direction of travel, it generates a function that approximates the multiple signs in the coordinate system. Based on the curve of this function, it interpolates the road markings in the intervals not recognized by the sign recognition unit.

Description

Technical Field

[0001] This invention relates to a road marking recognition device, which identifies road markings in the driving lanes of vehicles with autonomous driving or driver assistance functions. Background Technology

[0002] As such a device, there are known devices that identify road markings based on camera images. Such a device is described, for example, in Patent Document 1. In Patent Document 1, the road shape is approximated using a polynomial approximation function based on the detection results of the road markings by the camera.

[0003] However, in the existing technology, it is sometimes impossible to correctly identify road markings based on information obtained from camera images.

[0004] Existing technical documents

[0005] Patent documents

[0006] Patent Document 1: Japanese Patent Publication No. 2021-508901 (JP2021-508901A). Summary of the Invention

[0007] The road marking recognition device of the present invention comprises: a sign recognition unit that recognizes signs on the road based on detection information detected by an on-board detector that detects the surrounding conditions of the vehicle; and a road marking recognition unit that recognizes road markings based on the signs recognized by the sign recognition unit. The road marking recognition unit sets a coordinate system corresponding to the current position and direction of travel of the vehicle, and after removing signs that deviate from other signs by a predetermined interval in the direction intersecting the direction of travel from the multiple signs recognized by the sign recognition unit, it generates a function that approximates the multiple signs in the coordinate system, and interpolates the road markings in the intervals not recognized by the sign recognition unit based on the curve of the function. Attached Figure Description

[0008] The objectives, features, and advantages of the present invention are further illustrated by the following description of embodiments in conjunction with the accompanying drawings.

[0009] Figure 1 This is a block diagram schematically illustrating a road marking recognition device according to an embodiment of the present invention.

[0010] Figure 2A This is an explanation Figure 1 The road marking recognition unit identifies the first map of road markings.

[0011] Figure 2B This is an explanation Figure 1 The road marking recognition unit identifies the second image of the road markings.

[0012] Figure 3A This is an explanation Figure 1 The third image of the road marking recognition unit identifies road markings.

[0013] Figure 3B This is an explanation Figure 1 The fourth diagram of the road marking recognition system identifies road markings.

[0014] Figure 4A This explains the use of Figure 1 A flowchart illustrating an example of the processing performed by the arithmetic unit.

[0015] Figure 4B This is a detailed explanation. Figure 4A The flowchart of S30. Detailed Implementation

[0016] Hereinafter, embodiments of the invention will be described with reference to the accompanying drawings.

[0017] The road marking recognition device of the invention is applicable to vehicles with driver assistance or autonomous driving functions for the vehicle's driver, and identifies road markings that define the vehicle's driving lane (also known as the roadway).

[0018] It should be noted that sometimes the vehicle using the road marking recognition device in this implementation method is referred to as "this vehicle" to distinguish it from other vehicles. This vehicle can be any of the following: an engine vehicle with an internal combustion engine as the driving source, an electric vehicle with a drive motor as the driving source, or a hybrid vehicle with both an engine and a drive motor as driving sources.

[0019] In this implementation, "driving assistance" refers to assistance with the driver's driving operations, equivalent to automated driving below the specified level as defined by SAE (Society of Automotive Engineers). Conversely, "automated driving" in this implementation refers to the vehicle's driving control independent of the driver's operations, equivalent to automated driving above the specified level as defined by SAE.

[0020] In driver assistance or autonomous driving systems, the vehicle repeatedly detects its driving behavior, such as speed and direction of travel, as well as the surrounding external conditions at predetermined intervals (also known as cycles), and generates a target driving path based on the detected information. The vehicle is then controlled to travel along this generated target driving path.

[0021] The external conditions of this vehicle are detected by external sensors such as cameras and LiDAR. These external sensors are installed on the vehicle at specified positions and angles (also known as attitude information) during the manufacturing process.

[0022] The road marking recognition device takes into account the posture information of external sensors and uses the detection information of external sensors to estimate and identify the position of external targets, including road markings.

[0023] Road markings include white lines (including lines of different colors such as yellow), curb lines, road studs, and traffic barriers, which define the lanes of traffic. White lines include outer lane lines and lane boundary lines. Curb lines are lines connecting lanes and sidewalks, etc. In this implementation, the lines corresponding to the boundary lines of the lanes defined by the aforementioned markings are called road markings.

[0024] Generally, when road markings are identified based on information detected by external sensors while the vehicle is in motion, the identification result (position estimation result) contains a certain amount of error. Furthermore, because external sensors repeatedly detect the vehicle's external conditions at predetermined intervals, discrepancies can occur between successive detections. Therefore, directly overlapping multiple detections to identify road markings increases the identification error or the deviation of the identified road markings.

[0025] To reduce such recognition errors and deviations, in the implementation method, high-precision detection information from the vicinity of the vehicle obtained by external sensors is preferred.

[0026] Furthermore, on actual roads, sometimes road markings are partially missing due to peeling, fading, or other reasons, or they may be obscured by surrounding vehicles. Therefore, these may be perceived as interrupted road markings.

[0027] To interpolate discontinuous intervals in such road markings, the implementation method uses an approximate curve estimation method to estimate the shape of the road markings. In this case, after excluding detection information equivalent to so-called deviation values ​​(marks detected at intervals or more from other marks in a direction intersecting the direction of travel (called the vehicle width direction), the road marking shape is estimated, and the discontinuous intervals of the road markings are interpolated.

[0028] The following details such a road marking recognition device.

[0029] Figure 1 This is a block diagram schematically illustrating a road marking recognition device 10 according to an embodiment of the present invention. Figure 1As shown, the road marking recognition device 10 is mainly composed of an electronic control unit (ECU). The ECU is configured as a computer having an arithmetic unit 11 such as a CPU, a storage unit 12 such as RAM (random access memory) and ROM (read-only memory), an I / O interface, and other peripheral circuits. The ECU constituting the road marking recognition device 10 is part of a group of multiple ECUs mounted on the vehicle (vehicle 1) and controlling the vehicle's operations. The road marking recognition process performed by the road marking recognition device 10 begins, for example, when the vehicle is started or the ECU is activated, and is repeatedly executed at a predetermined cycle.

[0030] The road marking recognition device 10 is connected to an external sensor 2, a driving actuator 3, and a behavior sensor 4, which are mounted on the vehicle and detect the external conditions of the vehicle.

[0031] External sensor 2 detects the external conditions in front of the vehicle, centered on the vehicle's direction of travel. External sensor 2 may have an imaging element (image sensor) such as a CCD (charge-coupled device) or CMOS (complementary metal-oxide-semiconductor), and is composed of a camera that captures images of the space in front of the vehicle. The camera may be a single-lens reflex camera or a stereo camera, and continuously captures images of the space in front of the vehicle at a predetermined frame rate, outputting the frame image data (referred to as camera images) as detection information sequentially to the processing unit 11.

[0032] It should be noted that the external sensor 2 can also be composed of a lidar that uses a laser to measure the distance and direction to the object based on the time it takes for the laser light to return from the object, and detects the reflectance brightness at each measurement point.

[0033] The computing unit 11 estimates the position (current position) of the vehicle based on the camera image, which is the detection information detected by the external sensor 2, and creates a map containing the road lines around the vehicle using algorithms such as SLAM (Simultaneous Localization and Mapping).

[0034] The driving actuator 3 includes a steering mechanism such as a steering gear that steers the vehicle, a driving mechanism such as an engine or electric motor that drives the vehicle, and a braking mechanism such as a brake that brakes the vehicle. The behavior sensor 4 detects the vehicle's driving behavior, such as its speed and direction of travel, and outputs the detection signals to the arithmetic unit 11.

[0035] The ECU's arithmetic unit 11, which constitutes the road marking recognition device 10, includes a sign recognition unit 13, a road marking recognition unit 14, a driving control unit 15, and a map information update unit 16 as its functional structure. The storage unit 12 stores map information used for autonomous driving, as well as information such as the vehicle's driving trajectory, various control programs, and thresholds used in the programs.

[0036] The sign recognition unit 13 identifies the position of road signs in the direction of travel of the vehicle based on camera images input from external sensor 2.

[0037] The road marking recognition unit 14 identifies the road markings corresponding to the boundary lines of the driving lanes defined by the signs, based on the position of the signs identified by the sign recognition unit 13.

[0038] The driving control unit 15 controls the driving actuator 3 based on the road markings identified by the road marking recognition unit 14. For example, it generates a target driving path for the vehicle by passing through the center of the left and right road markings of the driving lane, and controls the driving actuator 3 to drive along the generated target driving path.

[0039] The map information update unit 16 updates the location information of the road lines included in the map information for autonomous driving stored in the storage unit 12 based on the road lines identified by the road line recognition unit 14.

[0040] The map information is updated based on the condition that the computational load of the driving control unit 15 is below a specified value, such as when the vehicle has just started (started), when the engine and drive motor are stopped, when the vehicle is parked, or when manual control is not in use for automatic driving.

[0041] When the map information is updated by the map information updating unit 16, the calculation unit 11 estimates the vehicle's position (current position) based on the updated map information and the external conditions detected by the external sensor 2. In this case, by improving the recognition accuracy of road markings included in the updated map information, the accuracy of the vehicle's position estimation is also improved.

[0042] Identification of road markings using signs near this vehicle.

[0043] Figure 2A and Figure 2B This is a diagram illustrating the road markings identified by the road marking recognition unit 14. For example... Figure 2A and Figure 2B As shown, the road marking recognition unit 14 sets a coordinate system with the vehicle's position at a certain moment as the origin Ot, the vehicle's direction of travel as the X-axis, and the vehicle width direction as the Y-axis. Figure 2A and Figure 2BThis is an example of the vehicle traveling straight in the positive direction along the X-axis, which corresponds to the vehicle's travel trajectory. The road marking recognition unit 14 uses multiple frames of camera images acquired by the external sensor 2 during driving to recognize road markings based on the positions of signs identified in each frame of the camera image multiple times (corresponding to the number of frames of the camera image) at a predetermined period (corresponding to the period during which the external sensor 2 detects external conditions). That is, at different locations on the road, the road markings around the vehicle are recognized multiple times (corresponding to the number of frames of the camera image) based on the signs identified from those locations. For example, when the vehicle travels 1m during the aforementioned period, the road markings are recognized multiple times (corresponding to the number of frames of the camera image) based on the signs identified from those locations on the road for every 1m traveled.

[0044] Figure 2A The road marking L is formed by repeatedly identifying the positions of all signs from the vicinity of the vehicle to the distance based on the camera images in each frame, according to the aforementioned prescribed period (corresponding to the number of camera frames), and then aligning these multiple road markings. It should be noted that, for convenience, only the road markings on the left side of the driving lane are shown; the same applies to the road markings on the right side of the driving lane.

[0045] In contrast, Figure 2B The road marking Lb is formed by repeatedly identifying road markings within a specified distance (e.g., 10m) of the nearby area of ​​the vehicle based on the signs identified in each frame of the camera image, and then merging these multiple road markings.

[0046] The distance from this vehicle (external sensor 2) to the sign is estimated based on the position of the sign as reflected in the camera image. The distance from this vehicle (external sensor 2) to the sign can also be calculated based on the detection value of the lidar.

[0047] Compare Figure 2A and Figure 2B , Figure 2A Road marking L ratio Figure 2BThe deviation of the road marking Lb is large. This can be attributed to the fact that the further a sign is from the vehicle, the lower the accuracy of the external sensor 2 in detecting its position, resulting in a larger error in the detected information compared to signs closer to the vehicle. In this embodiment, to reduce the impact of such errors, the road marking recognition unit 14 recognizes the road markings based on the positions of signs identified in the vicinity of the vehicle (within the aforementioned 10m). In particular, the road marking recognition unit 14 recognizes the road markings based on the positions of signs identified in the vicinity of the vehicle, at least in the curved sections of the road. This is because the deviation of the road markings recognized in the curved sections of the road is more likely to be larger than that in the straight sections.

[0048] Interpolation for intervals where no markers were detected

[0049] Figure 3A and Figure 3B This is a diagram illustrating other road markings identified by the road marking recognition unit 14. (And...) Figure 2A and Figure 2B Similarly, the road marking recognition unit 14 sets up a coordinate system with the vehicle's position at a certain moment as the origin Ot, the vehicle's direction of travel as the X-axis, and the vehicle width direction as the Y-axis. Figure 3A and Figure 3B This is an example of the vehicle traveling straight in the positive direction along the X-axis, where the X-axis corresponds to the vehicle's travel trajectory. Figure 2B The difference lies in the fact that there are missing sections 33 and 34 of the road signs and that the signs identified by the sign recognition unit 13 include signs 31 and 32 with abnormal positions caused by false detection, etc.

[0050] It should be noted that the road marking recognition unit 14 recognizes road markings based on the position of the markings identified in the vicinity of the vehicle (within the aforementioned 10m).

[0051] Figure 3A The road marking Lb is formed by repeatedly identifying road markings based on the positions of signs identified in the vicinity of the vehicle from the signs identified in each frame of the camera image, and then aligning these multiple road markings. It should be noted that... Figure 2B The same applies, except that only the road markings on the left side of the driving lane are shown; the road markings on the right side of the driving lane are also the same.

[0052] In contrast, Figure 3B The road marking Lint corresponds to the curve obtained by approximating the position of the sign from the signs identified in the vicinity of the vehicle based on the camera images of each frame, after excluding signs with similar deviation values.

[0053] The road marking recognition unit 14 uses an algorithm such as RANSAC (Random Sample Consensus) to remove markings that are equivalent to the deviation value.

[0054] Furthermore, the shape of a typical curved road is designed using a spiral curve with a curvature varying proportionally. A portion of this spiral curve, corresponding to the road shape, can be approximated using a cubic function or similar method. Therefore, the road marking recognition unit 14 generates information about a cubic function and interpolates the road markings in sections where no markings were detected based on this approximate curve. Using a cubic function to approximate the road shape reduces the computational load compared to using a spiral curve.

[0055] <Flowchart Explanation>

[0056] Reference Figure 4A and Figure 4B The flowchart illustrates the process according to a predetermined procedure. Figure 1 An example of the processing performed by the arithmetic unit 11. Figure 4A The road marking recognition process is shown, as described above, which begins when the ECU starts and is repeated at a predetermined cycle. Figure 4B Show Figure 4A Details of the S30.

[0057] exist Figure 4A In S10 (S: processing step), the arithmetic unit 11 obtains a camera image as detection information from the camera (external sensor 2) and enters S20.

[0058] In S20, the arithmetic unit 11 identifies the sign based on the camera image through the sign recognition unit 13 and proceeds to S30.

[0059] In S30, the arithmetic unit 11 identifies road markings based on the signs identified by the sign recognition unit 13 via the road marking recognition unit 14, and then proceeds to S40.

[0060] In S40, the computing unit 11, through the map information update unit 16, records (updates) the location information of the road lines included in the map information for autonomous driving stored in the storage unit 12, based on the road lines identified by the road line recognition unit 14, and then ends the process. Figure 4A The processing.

[0061] exist Figure 4BIn step S31, the arithmetic unit 11 selects a sign from the signs identified by the sign recognition unit 13 based on the camera image, whose distance from the vehicle is within a predetermined distance, and proceeds to step S32. As described above, the distance from the vehicle to the sign can be estimated based on the distance from the camera (external sensor 2) to the sign. Furthermore, calculations can also be performed based on the detection values ​​from the lidar (external sensor 2).

[0062] In S32, the arithmetic unit 11 removes the flags that correspond to the deviation value from the selected flags and proceeds to S33.

[0063] In S33, the arithmetic unit 11 performs an approximate curve estimation based on the position of the markers after removing markers with deviation values, identifies the road markings, and ends the process. Figure 4B The processing.

[0064] By adopting the implementation methods described above, the following effects can be achieved.

[0065] (1) The road marking recognition device 10 includes: a sign recognition unit 13, which recognizes signs (white lines, road studs, traffic barriers, etc.) on the road based on camera images that are detection information detected by an external sensor 2, wherein the external sensor is a vehicle-mounted detector that detects the surrounding conditions of the vehicle; and a road marking recognition unit 14, which recognizes road markings based on the signs recognized by the sign recognition unit 13. The road marking recognition unit 14 sets a coordinate system corresponding to the current position and direction of travel of the vehicle. After removing signs that are more than a predetermined interval (predetermined distance) away from other signs in the vehicle width direction intersecting the direction of travel, the sign recognition unit 13 generates a function that approximates the multiple signs in the coordinate system. The function curve is used to interpolate the road markings in the intervals not recognized by the sign recognition unit 13.

[0066] Because of this configuration, even in areas where the sign recognition unit 13 fails to recognize a sign, road markings can be appropriately interpolated and recognized. By using the location information of road markings recognized by such a road marking recognition device 10, the accuracy of map information used for autonomous driving can be improved.

[0067] (2) The road marking recognition device 10 also includes a calculation unit 11, which serves as a distance information acquisition unit and acquires distance information from the vehicle to the sign. The road marking recognition unit 14 recognizes the road marking based on the first sign identified in a first distance area near the vehicle and the second sign identified in a second distance area farther than the nearby area.

[0068] Because of this design, the deviation of the identified road markings can be minimized, thus enabling proper identification of road markings.

[0069] (3) After removing the sign that deviates more than a specified interval (specified distance) from other signs in the vehicle width direction intersecting the direction of travel from the multiple first signs identified in the nearby area, the road marking recognition unit 14 generates a function in the coordinate system that approximates the multiple first signs identified in the nearby area, and interpolates the road markings in the section not identified by the sign recognition unit 13 according to the curve of the function.

[0070] Because of this configuration, the influence of the first marker in the nearby area with a deviation value can be suppressed, thereby appropriately interpolating the road markings.

[0071] (4) The road marking recognition unit 14 recognizes the road markings based on the first mark at least in the curve section of the road.

[0072] Because of this design, the deviation of the identified road markings can be reduced in curved sections where the deviation is more likely to be larger than that of the straight sections of the road.

[0073] The above-described embodiments can be modified in various ways. The following describes some modifications.

[0074] (Variation Example 1)

[0075] In the description of the above embodiment, the vicinity of the vehicle whose sign is identified by the sign recognition unit 13 is set to within 10m of the vehicle. However, the upper limit of the vicinity range can be appropriately changed within the range of 5m to 10m. That is, when setting the vicinity range within a specified distance from the vehicle, the specified distance can be appropriately set.

[0076] (Variation Example 2)

[0077] In the above explanation, the example of signs whose recognition location (distance from the vehicle) exceeds the vicinity of the surrounding area illustrates an example that is not used for road marking recognition. However, it can also be configured as follows: Signs whose recognition location (distance from the vehicle) is within the vicinity of the vehicle (within a specified distance) are assigned a weight greater than 1, while signs whose recognition location (distance from the vehicle) is outside the vicinity of the vehicle (greater than the specified distance) are assigned a weight less than 1. These two types of signs are then used for road marking recognition. Signs assigned a weight greater than 1 contribute significantly to road marking recognition, thus enabling appropriate recognition of road markings.

[0078] (Variation Example 3)

[0079] The processing of S32 for removing deviation values ​​and the processing of S33 for approximate curve estimation can be omitted, for example, when the behavior sensor 4, which detects the vehicle's driving behavior such as speed and direction, detects that the vehicle is traveling in a straight direction.

[0080] (Variation Example 4)

[0081] The process of S31, which selects a sign in the vicinity of the vehicle among the signs identified by the sign recognition unit 13, can be omitted if the processes of S32, which removes deviation values, and S33, which performs approximate curve estimation, are not omitted.

[0082] It is possible to combine one or more of the above-described embodiments and variations, and to combine the variations with each other.

[0083] Using this invention, road markings can be properly identified.

[0084] The present invention has been described above in conjunction with preferred embodiments, but those skilled in the art should understand that various modifications and alterations can be made without departing from the scope of the following claims.

Claims

1. A road marking recognition device, characterized in that, have: The sign recognition unit (13) identifies road signs based on detection information detected by the external sensor (2) that detects the surrounding conditions of the vehicle. The road marking recognition unit (14) identifies the road markings based on the signs identified by the sign recognition unit (13); and The distance information acquisition unit acquires distance information from the vehicle to the sign; The road marking recognition unit (14) sets a coordinate system corresponding to the current position and direction of travel of the vehicle. After removing signs that deviate from the multiple signs identified by the sign recognition unit (13) by a predetermined interval relative to other signs in the direction intersecting the direction of travel, it generates a function that approximates the multiple signs in the coordinate system. Based on the curve of the function, it interpolates the road markings in the intervals not identified by the sign recognition unit (13). The road marking recognition unit (14) recognizes the road markings based on the first mark in a first distance region on the vehicle side obtained by the distance information acquisition unit and the first mark in a second distance region that is farther than the first distance region.

2. The road marking recognition device according to claim 1, characterized in that, After removing signs that deviate more than a predetermined interval from other signs in the direction intersecting the direction of travel from the plurality of first signs identified in the first distance region, the road marking recognition unit (14) generates a function in the coordinate system that approximates the plurality of first signs identified in the first distance region, and interpolates the road markings in the interval not identified by the sign recognition unit (13) according to the curve of the function.

3. The road marking recognition device according to claim 2, characterized in that, The road marking recognition unit (14) recognizes the road markings based on the first mark at least in the curve section of the road.

4. The road marking recognition device according to claim 1, characterized in that, The road marking recognition unit (14) recognizes the road marking by giving a greater weight to the first mark than the second mark among the first mark identified in the first distance region on the side of the vehicle and the second mark identified in the second distance region that is farther than the first distance region, which are obtained by the distance information acquisition unit.

5. The road marking recognition device according to any one of claims 1 to 4, characterized in that, It also includes a map information update unit (16), which updates the location information of the road lines contained in the pre-stored map information based on the road lines identified by the road line recognition unit (14).

Images