Driving support system and program

Abstract

To properly recognize road lane boundaries and, consequently, provide appropriate driving assistance. [Solution] The ECU 10 performs driving support control of the vehicle based on the left and right boundaries of the lane in which the vehicle is traveling. The ECU 10 includes a road marking line recognition unit 12 that extracts edge points from images captured by the camera 21 based on brightness changes and recognizes road marking lines of the lane based on the extracted edge points, a road shape determination unit 13 that determines the road shape of the road on which the vehicle is traveling from images captured by the camera 21, and a boundary selection unit 14 that selects a control target boundary to be used for driving support control from the road marking lines recognized by the road marking line recognition unit 12 based on the road shape determined by the road shape determination unit 13.

Description

Technical Field

[0001] The disclosure in this specification relates to a driving support device and a program for supporting the driving of a vehicle.

Background Art

[0002] As a vehicle driving support device, there is known a technique in which an edge point sequence corresponding to the left and right boundaries of the own lane is extracted based on a captured image of an in-vehicle camera, a lane line is recognized from the edge point sequence, and further, based on the lane line, driving support processing is performed so that the vehicle does not deviate from the own lane. Also, as a technique for recognizing a driving road, for example, when a vehicle travels in a provisional shared section such as a provisional two-lane section where oncoming traffic is allowed, a technique for appropriately recognizing the lane boundary in the provisional shared section has been proposed (for example, Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] For example, in a provisional two-lane section, the lane boundary on the road center side is composed of a composite line consisting of a plurality of lane lines, and in the composite line, the plurality of lane lines are drawn in parallel at positions close to each other. In this case, when LKA (Lane Keep Assist) control is performed as driving support control, there is a concern that adjacent lane lines may be alternately recognized as lane boundaries, causing the vehicle to wobble. In a provisional shared section, a composite line of a white line and a yellow line may be drawn as the lane boundary on the oncoming lane side, and a wavy surface composed of unevenness may be provided on the road surface. It is considered that the recognition accuracy of the lane line may decrease due to the disorder of the arrangement of the edge point sequence.

[0005] It is conceivable that the above-mentioned problems could occur on roads where vehicles travel, even in road configurations other than those of provisional shared sections. For example, in composite lines, white lines and yellow lines may be combined with colored lines of green, blue, red, etc. If fading or dirt occurs on each line in a composite line that includes these colored lines, there is a concern that this could cause vehicle instability in LKA control.

[0006] This disclosure is made in view of the above circumstances and aims to provide a driving assistance device and program that can properly recognize road lane boundaries and, consequently, provide appropriate driving assistance. [Means for solving the problem]

[0007] The driving assistance device of this disclosure is A driving assistance device applied to a vehicle equipped with a camera that photographs the area in front of the vehicle, which performs driving assistance control of the vehicle based on the left and right boundaries of the lane in which the vehicle is traveling, A road marking recognition unit extracts edge points from an image captured by the aforementioned camera based on brightness changes, and recognizes the road marking lines of the current lane based on the extracted edge points. A road shape determination unit determines the road shape of the road on which the vehicle is traveling from the image captured by the camera, A boundary selection unit selects a control target boundary to be used for the driving support control from the road marking lines recognized by the road marking line recognition unit, based on the road shape determined by the road shape determination unit, It is equipped with.

[0008] The road configurations on which vehicles travel are diverse, and the shape of the lane boundary may differ depending on the road configuration. Furthermore, when performing vehicle driving assistance control based on the left and right boundaries of the vehicle's lane, depending on the road configuration, it may be difficult to correctly recognize the boundary lines, which are road markings, from the sequence of edge points extracted from the camera image, raising concerns that this may hinder driving assistance control. In consideration of this, the system recognizes the road markings on the left and right of the vehicle's lane based on edge points extracted from the camera image, and simultaneously determines the road configuration of the road the vehicle is traveling on from the camera image. Based on this road configuration, it selects the control target boundary to be used for driving assistance control from among the road markings on the left and right of the vehicle's lane recognized from the edge points. In this case, by determining the road configuration from the camera image (vehicle front image) and then selecting the road markings to be used for driving assistance control from among the road markings on the left and right of the vehicle's lane, it is possible to appropriately determine which road markings should be used for driving assistance control. As a result, the system can properly recognize road lane boundaries and, consequently, provide appropriate driving assistance. [Brief explanation of the drawing]

[0009] [Figure 1] A diagram illustrating the overview of the vehicle's driving assistance system. [Figure 2] A diagram illustrating the overview of LKA control. [Figure 3] A diagram showing an image captured by a camera. [Figure 4] A diagram showing an image captured by a camera. [Figure 5] A diagram showing an image captured by a camera. [Figure 6] A diagram showing an image captured by a camera. [Figure 7] A diagram showing an image captured by a camera. [Figure 8] A diagram showing a road with a bicycle lane. [Figure 9] A flowchart illustrating the processing procedure for driving assistance control. [Figure 10] Figure 9 is followed by a flowchart showing the processing procedure for the driving assistance control. [Figure 11]A flowchart showing the processing procedure of driving support control in another example. [Figure 12] A diagram showing a captured image of a camera. [Figure 13] A flowchart showing the processing procedure of driving support control in another example.

Embodiments for Carrying Out the Invention

[0010] Hereinafter, embodiments of a driving support device according to the present disclosure will be described with reference to the drawings. In this embodiment, for example, a driving support system for performing vehicle driving support in a vehicle such as a passenger car, a truck, or a bus is constructed.

[0011] As shown in FIG. 1, the driving support system according to this embodiment includes an ECU 10 (Electronic Control Unit) as a driving support device, a camera 21, and a controlled device 30.

[0012] The camera 21 is an in-vehicle camera composed of a well-known CCD camera or the like, is mounted near the upper part of the windshield inside the host vehicle, and can capture a predetermined imaging range in front of the host vehicle. The camera 21 continuously captures images at a preset time interval and generates image data in front of the host vehicle. The camera 21 can capture color images. The camera 21 may be a monocular camera or a stereo camera. In addition to the camera 21, the vehicle has a radar device as a distance measuring device, a speed sensor for detecting the vehicle speed, a steering angle sensor for detecting the vehicle steering angle, etc., but illustration thereof is omitted here.

[0013] The controlled device 30 includes an alarm device 31 and a steering device 32. The alarm device 31 issues a warning to the driver of the host vehicle by any one of voice, lamp lighting, or display. In addition to or instead of the warning of the alarm device 31, a configuration may be adopted in which the driver is warned by steering vibration. The steering device 32 is a device that causes the host vehicle to perform automatic steering by an electric motor separately from manual steering of the host vehicle by the driver.

[0014] The ECU 10 is an electronic control unit equipped with a well-known microcomputer composed of a CPU, ROM, RAM, flash memory, etc. The microcomputer provides various arithmetic functions. The functions provided by the microcomputer can be provided by software recorded in a physical memory device and a computer that executes it, only software, only hardware, or a combination thereof. The microcomputer executes, for example, a program stored in a non-transitory tangible storage medium as a storage unit provided by itself. The program includes a road marking line recognition process for recognizing road marking lines including boundary dividing lines such as white lines drawn on the road, and a program related to LKA control processing for suppressing lane departure of the host vehicle. In addition, programs executable by the ECU 10 may include an object recognition process for recognizing objects around the host vehicle, a process for avoiding collisions with objects around the host vehicle or reducing damage during collisions, and a program related to a process for controlling the traveling speed of the host vehicle. When the program is executed, a method corresponding to the program is executed. The storage unit is, for example, a non-volatile memory. Note that the program stored in the storage unit can be updated via a network such as the Internet, for example.

[0015] As driving support control, the ECU 10 recognizes the boundary dividing lines of the lanes, which are road marking lines, based on the image captured by the camera 21, and based on the boundary dividing lines, executes LKA control to suppress the departure of the host vehicle from the lane during driving. At this time, the ECU 10 causes the warning device 31 to give a warning or the steering device 32 to perform automatic steering based on the positional relationship between the host vehicle and the boundary dividing lines within the host lane, thereby suppressing the lane departure of the host vehicle.

[0016] More specifically, in Figure 2, the vehicle's lane LA is demarcated by the left lane marking LL and the right lane marking LR, with warning activation lines LL1 and LR1 defined inside the left lane marking LL (right side) and inside the right lane marking LR (left side), respectively. The direction of travel of the vehicle CA is upward in the figure. In this case, when the vehicle CA drifts to the left within the vehicle's lane LA, the ECU 10 triggers a warning from the warning device 31 based on the fact that the left front end of the vehicle CA has reached the warning activation line LL1. Furthermore, based on the fact that the left front end of the vehicle CA has passed the warning activation line LL1 and reached the left lane marking LL, the ECU 10 operates the steering device 32 to pull the vehicle back towards the center of the lane. The same applies to the driving scenario where the vehicle CA drifts to the right within the vehicle's lane LA.

[0017] Next, the detailed configuration of LKA control in this embodiment will be described. In Figure 1, the ECU 10 includes an image acquisition unit 11, a road marking line recognition unit 12, a road shape determination unit 13, a boundary selection unit 14, and a driving control unit 15. The road marking lines include lane markings (boundary lane markings) consisting of white or yellow lines, but also include colored lines other than white and yellow. The colored lines do not necessarily demarcate lane boundaries, but rather serve as road markings to alert drivers, etc.

[0018] The image acquisition unit 11 acquires images captured by the camera 21. At this time, for example, the image shown in Figure 3 is acquired. The image in Figure 3 is an image taken when the vehicle is traveling through a provisional opening section, that is, an image showing an example of a provisional opening section. A provisional opening section is a road configuration in which only a limited number of lanes of a road planned to have four or more lanes are temporarily opened to traffic. In the image in Figure 3, it is a provisional two-lane section with one lane on each side, facing each other with a median strip in between. In a provisional opening section, a solid line is drawn on the left side in the direction of travel, and at least a yellow line is drawn in the median strip on the right side in the direction of travel. In the median strip, between the yellow line on the side of the vehicle's lane and the yellow line on the side of the opposing lane, white lines may be drawn individually for each lane, or lane separators or curbs may be installed.

[0019] In Figure 3, the vehicle's own lane 51, which the vehicle is traveling in, and the oncoming lane 52, which the oncoming vehicle is traveling in, are shown in the provisional section of the road. The left boundary of the vehicle's own lane 51 is the left lane marking 53, and the right boundary of the vehicle's own lane 51 is the median strip 54. The left lane marking 53 is, for example, a white line. The median strip 54 is composed of a composite line consisting of multiple lane markings (road marking lines) and three-dimensional structures such as traffic cones. Specifically, the median strip 54 includes, as a composite line on the vehicle's own lane 51 side, a yellow lane marking 61 on the inside of the vehicle's lane and a white lane marking 62 drawn parallel to and adjacent to the outside of the yellow lane marking 61, and as a composite line on the oncoming lane 52 side, a yellow lane marking 63 on the inside of the oncoming lane and a white lane marking 64 drawn parallel to and adjacent to the outside of the yellow lane marking 63. Furthermore, in the central median strip 54, a curb 65 and a pole cone 66, which serves as a lane separator, are arranged at predetermined intervals along the direction of the lane, between the composite lines on the side of the own lane 51 and the composite lines on the side of the opposing lane 52, respectively.

[0020] The road marking recognition unit 12 extracts edge points from the image acquired by the image acquisition unit 11 to estimate the left and right boundaries of the current lane 51, and recognizes the lane markings on the left and right of the current lane based on the extracted edge points. Specifically, edge points are extracted for each lane marking based on changes in brightness values, and each lane marking is recognized by the sequence of edge points connecting these edge points. In Figure 4, the left lane marking 53 is recognized as the lane marking on the left side of the current lane, and the adjacent yellow lane marking 61 and white lane marking 62 are recognized as the lane markings on the right side of the current lane.

[0021] The road configuration determination unit 13 determines the road configuration of the road on which the vehicle is traveling from the image acquired by the image acquisition unit 11. In this embodiment, the road configuration determination unit 13 extracts features of the boundary structure on both the left and right sides of the vehicle's lane from the entire captured image based on learning by deep learning, and determines the road configuration from these boundary structure features. At this time, the road configuration determination unit 13 performs machine learning processing by deep learning on the image in front of the vehicle to extract features of the boundary structure on both the left and right sides of the vehicle's lane. For example, as shown in Figure 5, features of the boundary structure X1 on the left side of the vehicle's lane and features of the boundary structure X2 on the right side of the vehicle's lane are extracted, and based on the extraction results, it is determined that the road configuration of the vehicle's lane currently being traveled on is a provisional shared section.

[0022] The road shape determination unit 13 calculates structural features such as the type and color of road markings, the number of road markings, and elevated structures visible in the image, along with a probability indicating that it is a provisionally opened section, and integrates these to determine whether or not it is a provisionally opened section. Note that when driving through intersections, junctions, ETC gates, or when guide signs or supplementary signs are present, the accuracy of determining whether it is a provisionally opened section may decrease, so it is advisable to lower the probability indicating that it is a provisionally opened section.

[0023] Since deep learning-based machine learning on images is well-known, a brief supplementary explanation will be provided. For example, a large number of images taken of the area in front of the vehicle are acquired, and the boundary structures on both sides of the vehicle's lane are classified into multiple categories according to the type of road configuration. Then, deep learning is performed on these classified images. The number of images used, the model, the number of layers in the model, etc., can be appropriately selected based on publicly known techniques. A neural network can be used as the deep learning model. A neural network can be composed of multiple layers consisting of interconnected nodes. This network can have tens to hundreds of hidden layers. The deep learning model is called a deep neural network, and it can learn using large amounts of labeled data and the structure of the neural network. A convolutional neural network can be used as the deep neural network.

[0024] The boundary selection unit 14 selects a control target boundary, which is the boundary used for driving control, from the road marking lines recognized by the road marking line recognition unit 12, based on the road configuration determined by the road configuration determination unit 13. For example, if the road configuration determination unit 13 determines that the road configuration of the lane currently being driven in is a temporary shared section, the control target boundary is selected from the road marking lines recognized by the road marking line recognition unit 12 based on the determination result that it is a temporary shared section.

[0025] In the provisional opening section, one of the boundaries on the left and right sides of the lane is the same as that of general roads outside the provisional opening section, but the other boundary is a provisional boundary that has been temporarily established, which may result in a difference in the accuracy of boundary recognition. In Figure 4, a single left-hand lane marking 53, the same as that of general roads, is drawn as the boundary on the left side of the lane, while a composite line including multiple lane markings 61 and 62 is drawn as the boundary on the right side of the lane. In this case, on the right side of the lane, the edge points extracted for each lane marking 61 and 62 are close to each other, so it is possible that the edge points of each lane marking 61 and 62 will be confused and an incorrect sequence of edge points will be created. For example, if each lane marking 61 and 62 on the provisional boundary side becomes faded, the edge points of each lane marking 61 and 62 will be extracted alternately in the lane direction, and the lane markings will be recognized in a wavy state. Furthermore, at temporary boundaries, the road surface may be made wavy to form rumble strips, and lane markings may be drawn on top of them. In such configurations, edge points cannot be properly extracted, resulting in lane markings being recognized in a wavy state. As described above, it is thought that the accuracy of boundary recognition is lower at temporary boundaries on the right side of the lane compared to boundaries (lane markings) on the left side of the lane.

[0026] Therefore, when the boundary selection unit 14 determines that the road is a provisional shared section, it selects the boundary on the side of the left or right boundary of its own lane that is different from the provisional boundary as the boundary to be controlled. In the road shown in Figure 3, the left lane marking 53 is selected as the boundary to be controlled.

[0027] The road configuration determination unit 13 may perform the following determinations in addition to determining whether or not it is a provisional opening section. Here, we will explain how the road configuration determination unit 13 determines the road configuration when the vehicle's lane travels on a road with a configuration different from that of a provisional opening section.

[0028] The road shape determination unit 13 determines that the left and right boundaries of its own lane are composite lines consisting of multiple road marking lines, and that the road marking lines of the composite line include colored lines other than white and yellow.

[0029] Figure 6 is a photographic image of a road configuration where the left and right boundaries of the vehicle's lane 51 are composed of multiple road marking lines, and these multiple road marking lines include colored lines other than white and yellow lines. In Figure 6, vehicle CB is a preceding vehicle traveling in front of the vehicle. On the left side of the vehicle's lane, a composite line is drawn, consisting of a white boundary line 71 and a green line 72 located inside the lane beyond the white line 71. On the right side of the vehicle's lane, a composite line is drawn, consisting of a white boundary line 73 and a green line 74 located inside the lane beyond the white line 73. The white lines 71, 73 and the green lines 72, 74 are solid lines. This type of lane configuration is called a "keep green line" and is provided to encourage drivers to keep their lanes in order to alleviate congestion and reduce the risk of accidents.

[0030] In addition to green lines, colored lines may also be red or blue lines, and composite lines including colored lines may be composite lines that combine red and blue lines. For example, a composite line may be one in which a red line is drawn inside the lane of a white boundary line, or a composite line may be one in which a blue line is drawn inside the lane of a white boundary line. In composite lines, the white or yellow line and the colored lines other than the white and yellow lines may be drawn parallel to each other without any gaps, or they may be drawn parallel to each other with a predetermined gap (about several tens of mm) in between.

[0031] The road shape determination unit 13 can determine the color of each line based on the relationship between the magnitudes of the RGB component values ​​(R value, G value, B value) for each color. In the case of white, the differences between the R value, G value, and B value are small, and the three component values ​​are approximately uniform. Therefore, if the differences between the R value, G value, and B value are all less than a preset difference threshold, the line color is determined to be white. On the other hand, for colors other than white, there is a fixed relationship between the magnitudes of the R value, G value, and B value depending on the color. For example, in the case of yellow, there is a relationship of R value > G value > B value, or R value > G value and R value > B value. Similarly, for green, red, and blue, there is a predetermined relationship between the magnitudes of the R value, G value, and B value. Therefore, it is preferable to pre-determine the relationship between the magnitudes of the R value, G value, and B value for each color used for road marking lines, and determine the line color based on that relationship.

[0032] The boundary selection unit 14, when the road shape determination unit 13 determines that the boundaries on the left and right sides of the current lane are composite lines and that the road markings of those composite lines include colored lines other than white and yellow, selects the road markings excluding those colored lines as the controlled boundary. More specifically, it selects the road markings (lane lines) that are outside the colored lines and are white or yellow as the controlled boundary.

[0033] When a composite line includes colored lines other than white and yellow lines, these colored lines are more likely to serve as a warning to the driver while driving than to indicate a lane boundary. In this case, there is a concern that LKA control or similar functions may be performed based on the colored lines, potentially leading to unnecessary driving assistance. For example, the driving range of the vehicle within its own lane may be excessively restricted, causing the driver to feel uncomfortable. In light of this, the boundary selection unit 14 selects road marking lines excluding colored lines as the control target boundary, and more specifically, selects road marking lines that are outside the colored lines and are either white or yellow as the control target boundary.

[0034] In the driving path shown in Figure 6, the white line 71 outside the green line 72 on the left side of the lane is selected as the control target boundary. Also, the white line 73 outside the green line 74 on the right side of the lane is selected as the control target boundary.

[0035] Furthermore, composite lines consisting of a combination of white or yellow lane markings and colored lines may take forms other than those shown in Figure 6. For example, contrary to Figure 6, a road with a green line drawn outside the white line in the lane is also conceivable. In this case as well, the boundary selection unit 14 will select the road marking lines excluding the colored lines as the control target boundary, and more specifically, it will select the road marking lines (lane markings) that are white or yellow lines inside the colored lines as the control target boundary.

[0036] Furthermore, the road configuration determination unit 13 may also determine whether a guide line for vehicle guidance is drawn on the road, near the center of the vehicle's lane. In the image shown in Figure 7, boundary lines 81 and 82 made of white lines are drawn as the left and right boundaries of the vehicle's lane 51, and a guide line 83 made of a colored line is drawn between the left and right boundary lines 81 and 82 and approximately in the center of the vehicle's lane 51. The guide line 83 is, for example, a solid green line.

[0037] In this case, the boundary selection unit 14 selects the boundary lines 81 and 82 as the control target boundary from among the boundary lines 81 and 82 on the left and right sides of the vehicle's lane and the guide line 83 approximately in the center of the vehicle's lane.

[0038] Furthermore, the road configuration determination unit 13 may also determine whether a bicycle lane is set up in a part of the vehicle's lane to allow bicycles to travel. In the road shown in Figure 8, both the left and right sides of the vehicle's lane 91 are demarcated by boundary lines 92 and 93. A shoulder 94 is provided on the left side of the vehicle's lane 91, with the boundary line 92 in between, and a sidewalk 96 is provided on the left side of the shoulder 94, with a three-dimensional structure 95 such as a curb or guardrail in between. For example, the boundary line 92 on the left side of the vehicle's lane is a solid white line, and the boundary line 93 on the right side of the vehicle's lane is a dashed white line. Within the vehicle's lane 91, a blue line 97 is drawn at a position slightly to the left in the left-right direction, and the space between the boundary line 92 on the left side of the vehicle's lane and the blue line 97 is the bicycle lane 98. The blue line 97 is either a dashed or solid line and is the boundary line of the bicycle lane 98 within the vehicle's lane 91.

[0039] In this case, when the vehicle CA is traveling in its own lane 91, the boundary selection unit 14 switches and selects the controlled boundary depending on whether or not there are bicycles traveling in the bicycle lane 98 within the own lane 91. Specifically, if there are no bicycles traveling in the bicycle lane 98, the boundary selection unit 14 selects the boundary lines 92 and 93 on the left and right sides of the own lane as the controlled boundary. If there are bicycles traveling in the bicycle lane 98, the boundary selection unit 14 selects the blue line 97 on the left side of the own lane and the boundary line 93 on the right side of the own lane as the controlled boundary.

[0040] Incidentally, unlike in Figure 8, if a bicycle lane 98 is provided outside the vehicle's own lane 91 (outside the boundary line 92 on the left side of the vehicle's own lane), then regardless of whether or not there are bicycles in the bicycle lane 98, the boundary lines 92 and 93 on the left and right sides of the vehicle's own lane should be selected as the controlled boundary.

[0041] The driving control unit 15 uses the boundary selected by the boundary selection unit 14 (control target boundary) to perform lane departure suppression control (LKA control) by preventing the vehicle from getting too close to or crossing the boundary.

[0042] Figures 9 and 10 are flowcharts showing the processing procedure for driving assistance control. This process is repeatedly executed by the ECU 10 at predetermined intervals.

[0043] In Figure 9, step S101 acquires the image captured by camera 21. Step S102 acquires map information and the vehicle's current position. This allows the vehicle's position to be recognized (self-position recognition unit). At this time, the ECU 10 may acquire map information used in, for example, an in-vehicle navigation system. The navigation system has a function to acquire map information, which is information about roads the vehicle can travel on, and a function to recognize the vehicle's current position. Self-position information is acquired, for example, by a vehicle position sensor using GPS or GNSS (Global Navigation Satellite System).

[0044] In step S103, edge points are extracted from the captured image based on changes in brightness values, and road markings are recognized based on the extracted edge points. This allows for the recognition of boundary lines, which are white or yellow lines, and colored lines other than white and yellow lines on the road surface.

[0045] In step S104, the road configuration of the road on which the vehicle is traveling is determined from the captured image. At this time, based on learning by deep learning, the features of the boundary structure on both the left and right sides of the vehicle's lane are extracted from the entire captured image, and the road configuration is determined from the features of that boundary structure. In step S104, as part of the road configuration determination process, a determination process is performed to determine whether the road is a temporarily shared section, and a determination process is performed to determine whether the road markings include colored lines and where and how the colored lines are drawn.

[0046] Subsequently, in step S105, based on the road configuration determination result in step S104, it is determined whether the road configuration of the current lane is a provisional shared section. If it is a provisional shared section, the process proceeds to step S106. In step S106, the boundary on the side of the current lane's left and right boundaries that is different from the provisional boundary is selected as the control target boundary. When driving through the provisional shared section shown in Figures 3 to 5, the left lane marking 53 is selected as the control target boundary among the left and right lane boundaries.

[0047] Subsequently, in step S107, lane departure suppression control is performed using the boundary (control target boundary) selected in step S106, ensuring that the vehicle does not get too close to or cross the boundary.

[0048] In step S108, information regarding the road configuration at the current location is stored in the memory of the ECU10 as attribute information of the current location in the map information (information storage unit). If it is determined that the location is a provisional access section, attribute information indicating that it is a provisional access section is stored in the memory of the ECU10. After the attribute information of the current location is stored in the map information, it is preferable that the road configuration is determined using the attribute information in step S104.

[0049] On the other hand, if it is determined in step S105 that the section is not a provisional shared section, the process proceeds to step S111 in Figure 10. In step S111, it is determined whether the road markings of the current lane include colored lines other than white and yellow lines. If colored lines are included, the process proceeds to step S112.

[0050] In step S112, it is determined whether or not colored lines are included in the road markings on both the left and right sides of the current lane. If colored lines are included in the road markings on both the left and right sides of the current lane, the process proceeds to step S113. If colored lines are not included in the road markings on both the left and right sides of the current lane, the process proceeds to step S121.

[0051] Step S113 determines whether the boundary between the left and right sides of the current lane is a composite line consisting of multiple road marking lines. Step S113 also determines whether the road marking lines on the road surface are recognized as composite lines. Therefore, even if the road marking lines are actually drawn as composite lines on the road surface, if the white line portion is not recognized as a road marking line, it is determined that the road marking lines consist only of colored lines.

[0052] If it is determined to be a composite line, the process proceeds to step S114, where the road marking lines excluding the colored lines are selected as the control target boundary. More specifically, the road marking lines (lane lines) that are white or yellow inside the colored lines are selected as the control target boundary.

[0053] Furthermore, if it is determined that the line is not a composite line, the process proceeds to step S115, where the colored line is selected as the control target boundary, and the lane boundary position is set at a position offset from the colored line by the line width to the outside of the lane. At this time, assuming that a boundary line exists outside the colored line, it is advisable to set the virtual boundary by offsetting a predetermined line width dimension (approximately 150 mm) from the recognition position of the colored line. For example, even if a composite line consisting of a boundary line and a colored line is actually drawn on the road, if the boundary line cannot be correctly recognized, the position of the virtual boundary is estimated based on the colored line. It is also possible to determine whether the left and right boundaries of the own lane are composite lines and set virtual boundaries individually according to the determination results for each left and right boundary.

[0054] Next, it is determined whether the recognized lane boundary is a Keep Green Line. A Keep Green Line is a lane boundary in the form shown in Figure 6, for example. If it is determined to be a Keep Green Line, the process proceeds to step S117, where a notification is issued to the driver that changing lanes is not recommended in order to restrict the vehicle from changing lanes. This notification to the driver may be made by voice or display. If the ECU 10 performs lane change support control to assist steering during a lane change after the driver operates the turn signal, the permission flag that allows the implementation of lane change support control should be turned off in step S117.

[0055] Subsequently, the process proceeds to step S107 in Figure 9, where lane departure suppression control is performed using the boundary (control target boundary) selected in steps S114 and S115. In the following step S108, the information regarding the lane boundary determined in steps S111 to S113 is stored in the memory of the ECU 10 as attribute information of the current self-position in the map information.

[0056] In step S121, it is determined whether the road has a bicycle lane within its own lane. Specifically, it is determined whether the road has a bicycle lane in the form shown in Figure 8. If step S121 is affirmed, the process proceeds to step S122. In step S122, it is determined whether or not a bicycle is traveling in the bicycle lane. If no bicycle is traveling in the bicycle lane, the process proceeds to step S123, where both the left and right boundary lines of the own lane are selected as the control target boundaries.

[0057] Furthermore, if a bicycle is traveling in the bicycle lane, the system proceeds to step S124, where the boundary line of the bicycle lane within the current lane is selected as the control target boundary. In the following step S125, since a bicycle is traveling in the bicycle lane within the current lane, the system notifies the driver to avoid contact with the bicycle.

[0058] As explained in Figure 8, if no bicycles are traveling in the bicycle lane 98, the boundary lines 92 and 93 on the left and right sides of the vehicle's own lane are selected as the controlled boundaries. If bicycles are traveling in the bicycle lane 98, the blue line 97 on the left side of the vehicle's own lane and the boundary line 93 on the right side of the vehicle's own lane are selected as the controlled boundaries.

[0059] Furthermore, a situation in which step S121 is denied is when the road markings in the current lane include a colored line, but that colored line is not one of the road markings to the left or right of the current lane, nor is it a bicycle lane (a situation in which step S111 is YES, and steps S112 and S121 are NO). Under these circumstances, it is considered that within the current lane, a guide line is drawn as a road marking that is a colored line to guide the vehicle (see Figure 7).

[0060] It should be noted that while both the case where a colored line (blue line 97 in Figure 8) is drawn within the vehicle's lane to create a bicycle lane and the case where a guide line (guide line 83 in Figure 7) is drawn within the vehicle's lane are the same in that a colored line is drawn within the vehicle's lane, the position of the colored line for creating a bicycle lane and the colored line for use as a guide line differ in the lateral direction within the vehicle's lane, so it is possible to distinguish between the two types of colored lines. The colored line for creating a bicycle lane is placed close to the left-hand lane marking, at a predetermined distance (for example, about 1m) from that left-hand lane marking. In contrast, the colored line for use as a guide line is placed near the center of the vehicle's lane in the lateral direction, slightly to the right. In other words, unlike the boundary line of the bicycle lane (blue line 97), the guide line is intended to be driven over by the vehicle, and the vehicle's passing line is defined so that the driver in the vehicle passes directly over or near directly over the guide line.

[0061] If step S121 is rejected, the process proceeds to step S126. In step S126, the driver is notified to drive their vehicle along the guide line. This notification may be given to the driver via voice or display. Subsequently, in step S127, both left and right boundary lines of the vehicle's lane are selected as the control target boundaries. In particular, the ECU 10 selects the left and right road marking lines of the vehicle's lane as the control target boundaries, provided that the vehicle crosses the guide line.

[0062] Furthermore, the situation in which step S111 is rejected is when the road is not a temporarily shared section and the road markings on the own lane do not include colored lines (situations in which steps S105 and S111 are NO). Under these circumstances, the road shape of the road is not special, and it is assumed that general white or yellow boundary lines are drawn as the boundaries on the left and right sides of the own lane. Therefore, if step S111 is rejected, the process proceeds to step S127. In step S127, both the left and right boundary lines of the own lane are selected as the boundaries to be controlled.

[0063] After steps S123, S125, and S127, the process proceeds to step S107 in Figure 9, where lane departure suppression control is performed using the boundary (control target boundary) selected in steps S123, S124, and S127. In the following step S108, information regarding bicycle lanes and guide lines determined in step S121, etc., is stored in the memory of the ECU 10 as attribute information of the current self-position in the map information.

[0064] According to the embodiment described in detail above, the following excellent effects can be obtained.

[0065] The road configurations on which vehicles travel are diverse, and depending on the road configuration, it may be difficult to correctly recognize boundary lane lines using the edge point sequence extracted from the image captured by camera 21, raising concerns that this could hinder driving assistance control. In light of this, the system recognizes road markings on both sides of the vehicle's lane based on edge points extracted from the image captured by camera 21. Furthermore, it determines the road configuration of the road the vehicle is traveling on from the image captured by camera 21, and based on that road configuration, selects the control target boundary to be used for driving assistance control from among the road markings on both sides of the vehicle's lane recognized from the edge points. In this case, by determining the road configuration from the image captured by camera 21 (vehicle front image) and then selecting the road markings to be used for driving assistance control from among the road markings on both sides of the vehicle's lane, it is possible to appropriately determine which road markings should be used for driving assistance control. As a result, the lane boundaries of the road can be properly recognized, and consequently, appropriate driving assistance can be provided.

[0066] Furthermore, because the system is configured to select the control target boundary based on road markings recognized by edge point sequences and the road shape determination result from the entire image, even if the accuracy of road marking recognition by edge point sequences decreases due to adverse environments, the control target boundary can still be appropriately selected based on the road shape determination result. This improves the accuracy of driving assistance control.

[0067] In a temporarily shared section, a decrease in boundary recognition accuracy may occur on the side of the temporary boundary between the left and right boundaries of the vehicle's lane. Considering this, when the road is determined to be a temporarily shared section, the system selects the road marking line on the side of the lane that is different from the temporary section boundary as the control target boundary. This allows for appropriate LKA control to be performed while taking into account the difference in recognition accuracy between the left and right boundaries when driving in a temporarily shared section.

[0068] When it is determined that the boundary between the left and right sides of the current lane is a composite line and that the composite line includes a colored line as a road marking, the system now selects the road marking excluding the colored line as the control target boundary. More specifically, when it is determined that the composite line includes a colored line, the system now selects the road marking outside the colored line, which is either a white or yellow line, as the control target boundary. This allows the system to properly determine lane boundaries even on roads where composite lines including colored lines are drawn.

[0069] When road markings include colored lines other than white and yellow lines, these colored lines may be located inside the lane boundary of the current lane. Considering this, when a road marking is determined to be a colored line, the colored line is selected as the control target boundary, and the position of the lane boundary is offset from the colored line in the direction of the lane width. This allows for proper identification of lane boundaries on roads with composite lines including colored lines.

[0070] When it is determined that a guide line is drawn near the center of the vehicle's lane on the road, the system is configured to select the road markings on either side of the vehicle's lane as the control target boundary, based on the condition that the vehicle crosses the guide line. In this case, LKA control and other functions can be properly executed while distinguishing between the boundary lines and guide lines on the road.

[0071] The system recognizes the vehicle's own position and stores information about the road configuration at that location as attribute information for the map. This allows the vehicle to accurately understand the road configuration based on the attribute information when passing through the same location in the future, and consequently, to perform appropriate driving assistance control.

[0072] (Other embodiments) The above embodiment may be modified as follows, for example.

[0073] The road configuration determination unit 13 may determine, as part of the road configuration determination, that one of the boundaries on the left and right sides of the vehicle's lane is a first boundary with a single road marking line, and the other is a second boundary with a composite line consisting of multiple road marking lines. For example, as shown in Figure 3, when the vehicle is traveling through a provisional opening section, the road configuration of the travel path will be such that the boundary on the left side of the vehicle's lane is the first boundary (a single road marking line), and the boundary on the right side of the vehicle's lane is the second boundary (a composite line consisting of multiple road marking lines). Note that even outside of provisional opening sections, the boundaries on the left and right sides of the vehicle's lane may be a combination of the first and second boundaries.

[0074] If one of the left and right boundaries of the vehicle's lane is the first boundary and the other is the second boundary, and the extraction of edge point sequences is partially interrupted at each road marking line constituting the composite line at the second boundary, then adjacent road marking lines will be alternately recognized as lane boundaries. In this case, there is a concern that vehicle swaying may occur in LKA control. In consideration of this, the boundary selection unit 14, when the road shape determination unit 13 determines that one of the left and right boundaries of the vehicle's lane is the first boundary and the other is the second boundary, selects the road marking line recognized by the road marking line recognition unit 12 at the first boundary as the boundary to be controlled.

[0075] Specifically, ECU10 should execute the process shown in Figure 11. This process replaces the process shown in Figure 9. In Figure 11, the process related to determining the road configuration is designated as steps S201 to S203, which differs from the process in Figure 9. Processes common to both Figure 9 and Figure 11 are given the same step numbers and their explanations are omitted.

[0076] In Figure 11, after steps S101 to S103, the process proceeds to step S201, where the road configuration of the road on which the vehicle is traveling is determined from the captured image. In step S201, as part of the road configuration determination process, a determination process is executed to determine whether one of the boundaries of the vehicle's lane is a first boundary (a single road marking line) and the other boundary is a second boundary (a composite line consisting of multiple road marking lines). It is also desirable that the determination process described in step S104 of Figure 9 be executed in the same way in step S201.

[0077] Subsequently, in step S202, based on the road configuration determination result in step S201, it is determined whether the road configuration of the current lane has a first boundary and a second boundary. If step S202 is affirmed, the process proceeds to step S203. For example, step S202 is affirmed when the road is a provisional section. In step S203, the provisional boundary and the first boundary are selected as the control target boundary from the left and right boundaries of the current lane. When driving on the roads shown in Figures 3 to 5, the left lane marking 53 is selected as the control target boundary from the left and right boundaries of the current lane.

[0078] Subsequently, in step S107, lane departure suppression control is performed using the boundary (control target boundary) selected in step S203. In step S108, information regarding the left and right boundaries of the vehicle's lane is stored in the memory of the ECU10 as attribute information of the vehicle's current position in the map information.

[0079] The above configuration prevents the problem of multiple adjacent road markings being alternately recognized as lane boundaries, which can cause vehicle swaying during LKA control.

[0080] The road shape determination unit 13 may determine whether the road is an S-shaped curve. Figure 12 is a photographed image of a road where the road is an S-shaped curve. On an S-shaped curve, the lane 101 undulates from side to side, which can cause changes in the line width of the road markings 102 and 103 on the image, or cause parts of the road markings 102 and 103 to be irregularly hidden and become invisible due to preceding vehicles, guardrails, etc., which can reduce the reliability of recognizing the road markings.

[0081] Therefore, when the boundary selection unit 14 determines that the road is an S-shaped curve, it determines the reliability of the road markings on the left and right sides of the lane, and based on that reliability, selects a control target boundary from the road markings recognized by the road marking recognition unit 12. The reliability of the road markings can be determined, for example, based on the magnitude of the edge intensity and the continuity of the edge points when extracting the road markings from the captured image. In this case, if the reliability index calculated from the magnitude of the edge intensity and the continuity of the edge points is greater than or equal to a predetermined value, it is determined that the reliability is high (the reliability of the road markings is greater than or equal to a predetermined value), and if the reliability index is less than a predetermined value, it is determined that the reliability is low (the reliability of the road markings is less than a predetermined value).

[0082] Specifically, ECU10 should execute the process shown in Figure 13. This process replaces the process shown in Figure 9. In Figure 13, the process related to determining the road configuration is designated as steps S301 to S303, which differs from the process in Figure 9. Processes common to both Figure 9 and Figure 13 are given the same step numbers and their explanations are omitted.

[0083] In Figure 13, step S301 performs a determination process to determine if the road is an S-shaped curve. It is also preferable that step S301 performs the same determination process as described in step S104 in Figure 9. If the road is an S-shaped curve, step S302 is affirmed and the process proceeds to step S303. In step S303, based on the reliability of the road markings on both sides of the lane, the control target boundary is selected from the road markings recognized by the road marking recognition unit 12. At this time, the road marking with the highest reliability is selected as the control target boundary.

[0084] This allows for proper driving assistance even when the road markings on one of the boundaries of the lane are difficult to recognize while driving on an S-shaped curve. In step S301 of Figure 13, if it is determined that the road is a provisional section, that the road markings include colored lines, and that the road is an S-shaped curve, the determination that it is a provisional section should be given the highest priority when selecting the control target boundary. If it is not a provisional section, the control target boundary should be selected based on whether the road markings include colored lines or whether the road is an S-shaped curve.

[0085] In the above embodiment, the road configuration is determined by extracting feature points through deep learning, but this configuration may be changed. For example, image patterns for a large number of road configurations can be assumed and registered in memory in advance. Then, the road configuration of the road can be determined by comparing the captured images acquired when the vehicle is driving with the pre-prepared image patterns.

[0086] The control devices and methods described herein may be implemented by a dedicated computer provided by configuring a processor and memory programmed to perform one or more functions embodied by a computer program. Alternatively, the control devices and methods described herein may be implemented by a dedicated computer provided by configuring a processor by one or more dedicated hardware logic circuits. Alternatively, the control devices and methods described herein may be implemented by one or more dedicated computers configured by a combination of a processor and memory programmed to perform one or more functions and a processor configured by one or more hardware logic circuits. Furthermore, the computer program may be stored as instructions executed by the computer on a computer-readable non-transitional tangible recording medium. [Explanation of Symbols]

[0087] 10...ECU, 12...Road marking recognition unit, 13...Road shape determination unit, 14...Boundary selection unit, 21...Camera.

Claims

1. A driving support device (10) is applied to a vehicle equipped with a camera (21) that photographs the area in front of the vehicle, and which performs driving support control of the vehicle based on the left and right boundaries of the lane in which the vehicle is traveling, A road marking recognition unit (12) extracts edge points from the image captured by the aforementioned camera based on brightness changes, and recognizes the road marking lines of the current lane based on the extracted edge points, A road shape determination unit (13) determines the road shape of the road on which the vehicle is traveling from the image captured by the camera, A boundary selection unit (14) selects a control target boundary to be used for the driving support control from the road marking lines recognized by the road marking line recognition unit, based on the road shape determined by the road shape determination unit, A driving assistance system equipped with the following features.

2. The road configuration determination unit determines that the road configuration is a provisional shared section. The driving support device according to claim 1, wherein the boundary selection unit, when the road shape determination unit determines that the driving road is a temporary shared section, selects the road marking line on the left and right of the lane recognized by the road marking line recognition unit, which is different from the temporary section boundary, as the control target boundary.

3. The road configuration determination unit determines, as part of the determination of the road configuration, that one of the left and right boundaries of the lane is a first boundary with a single road marking line drawn on it, and the other is a second boundary with a composite line consisting of multiple road marking lines drawn on it. The driving support device according to claim 1, wherein the boundary selection unit, when the road shape determination unit determines that one of the left and right boundaries of the own lane is the first boundary and the other is the second boundary, selects the road marking line recognized by the road marking line recognition unit at the first boundary as the control target boundary.

4. The road form determination unit determines, as the determination of the road form, that the boundary is a composite line consisting of a plurality of road marking lines, and that the road marking lines of the composite line include colored lines other than white and yellow. The driving support device according to claim 1, wherein the boundary selection unit selects the road marking lines excluding the color lines as the controlled boundary when the road shape determination unit determines that the boundary is the composite line and that the color lines are included in the road marking lines of the composite line.

5. The driving support device according to claim 4, wherein the boundary selection unit, when the road shape determination unit determines that the boundary is the composite line and that the color line is included in the road marking line of the composite line, selects a road marking line that is outside the color line and is a white or yellow line as the control target boundary.

6. The road shape determination unit determines, as part of the determination of the road shape, that the road marking lines of the boundary are colored lines other than white and yellow. The driving support device according to claim 1, wherein the boundary selection unit, when the road shape determination unit determines that the road surface marking line is the color line, selects the color line as the controlled boundary and sets the position of the boundary to a position offset from the color line in the lane width direction.

7. The road configuration determination unit determines, as part of the road configuration, whether a guide line for vehicle guidance is drawn on the road, near the center of the lane. The driving support device according to claim 1, wherein the boundary selection unit, when the road shape determination unit determines that the guide line is drawn, selects the road marking lines to the left and right of the vehicle's lane, recognized by the road marking line recognition unit, as the control target boundary, on the condition that the vehicle straddles the guide line.

8. A self-position recognition unit that recognizes the vehicle's own position, An information storage unit stores information about the road configuration determined by the road configuration determination unit as attribute information of the self-position in the map information, A driving support device according to any one of claims 1 to 7, comprising:

9. A program applied to a vehicle equipped with a camera (21) that photographs the area in front of the vehicle, and which is executed by a computer to perform driving assistance control of the vehicle based on the left and right boundaries of the lane in which the vehicle is traveling, From the image captured by the aforementioned camera, edge points are extracted based on brightness changes, and a road marking recognition process is performed to recognize the road markings of the current lane based on the extracted edge points. A road configuration determination process that determines the road configuration of the road on which the vehicle is traveling from the image captured by the aforementioned camera, A boundary selection process is performed to select a control target boundary to be used for the driving support control from the road marking lines recognized by the road marking line recognition process, based on the road shape determined by the road shape determination process. A program that includes this.

Images