Driving assistance device and program

Abstract

To provide a technique for appropriately and quickly executing white line recognition in a driving support device for controlling a vehicle on the basis of white line recognition information.SOLUTION: A driving support device includes a white line recognition unit 12 for recognizing a white line around a self vehicle using an image and a state amount, and calculating white line recognition information, and an alarm / vehicle control unit 13 for performing alarm and / or control of the vehicle on the basis of the white line recognition information. The white line recognition unit includes a line candidate extraction unit 17 for extracting a line candidate that becomes a candidate of a boundary line of a travel road of the vehicle, a boundary line selection unit 18 for selecting a boundary line from the line candidate, a boundary line recognition unit 20 for recognizing the boundary line on the basis of learning by deep learning, and a class identification unit 21 for imparting class identification. The boundary line selection unit includes a feature calculation unit 33 for calculating a feature amount for selecting a boundary line from the line candidate. The feature calculation unit calculates a percentage ratio that is a ratio of a feature amount identified as a road surface class or a white line class in the line candidate, and the boundary line selection unit 18 determines whether or not the line candidate is a noise line on the basis of the percentage content.SELECTED DRAWING: Figure 7

Description

[Technical field]

[0001] Driving support device that controls the vehicle based on white line recognition and Programs Regarding. [Background technology]

[0002] Patent Document 1 describes a driving support device that combines the recognition result of adjacent lanes with map information to determine whether to change lanes. In this driving support device, the reliability of the recognition of adjacent lanes is determined by the consistency with the map information. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2016-16853 A Summary of the Invention [Problem to be solved by the invention]

[0004] In Patent Document 1, when a map cannot be used, there is a possibility that adjacent lanes may be misidentified. When recognizing adjacent lanes during lane change, conventionally, the width of the own lane before the lane change is maintained with a certain tolerance and line candidates that will become adjacent lanes after the lane change are searched for. When recognizing adjacent lanes captured on the edge of an image by a camera, this method can provide a certain degree of robustness when applied to expressways in that it suppresses the misrecognition of adjacent lanes with distortion or tolerance of the lens, obstruction by a moving object traveling in the adjacent lane, and noise lines such as curbs and dirt lines.

[0005] However, in recent years, there has been a demand for technology that can stably recognize white lines of adjacent lanes, etc., even in environments worse than those of expressways, such as urban areas and mountain roads. In poor environments, there is a concern that conventional methods will have difficulty recognizing white lines, due to factors such as the significant difference in lane width of adjacent lanes before and after lane changes. In order to appropriately control the vehicle, such as by changing lanes, based on white line recognition information obtained by white line recognition, it is necessary to recognize white lines appropriately and quickly.

[0006] In view of the above, the present invention provides a driving assistance device that controls a vehicle based on white line recognition information. and Programs The present invention aims to provide a technology for performing white line recognition appropriately and quickly in the above-mentioned vehicle. [Means for solving the problem]

[0007] The present invention provides first and second driving assistance devices comprising an image capturing unit that acquires an image captured by an on-board camera, a vehicle state quantity sensor unit that acquires a state quantity of the vehicle measured by a sensor, a white line recognition unit that calculates white line recognition information by recognizing white lines around the vehicle using the image and the state quantity, and an alarm / vehicle control unit that issues an alarm and controls the vehicle based on the white line recognition information.

[0008] In the first driving assistance device, the white line recognition unit includes a line candidate extraction unit that extracts line candidates that are candidates for boundary lines of the road on which the vehicle is traveling, a boundary selection unit that selects the boundary line from the line candidates extracted by the line candidate extraction unit, a boundary recognition unit that recognizes the boundary line based on learning by deep learning, and a class identification unit that assigns a predetermined class identification to each pixel of the image. The boundary selection unit includes a feature calculation unit that calculates a feature amount for selecting the boundary line from the line candidates extracted by the line candidate extraction unit. The feature calculation unit calculates an inclusion rate, which is the proportion of the feature amount identified by the class identification unit as a road surface class or a white line class in the line candidates extracted by the line candidate extraction unit, and the boundary selection unit determines whether the line candidate is a noise line based on the inclusion rate.

[0009] According to the first driving assistance device, the boundary selection unit includes a feature calculation unit that calculates an inclusion rate, which is a ratio of feature amounts that are identified as a road surface class or a white line class in the line candidate. The feature calculation unit can further determine a noise line based on the inclusion rate, and therefore can prevent the noise line from being mistaken for a boundary line. As a result, the white line recognition can be performed appropriately and quickly to select a boundary line.

[0010] In the second driving assistance device, the white line recognition unit includes a line candidate extraction unit that extracts line candidates that are candidates for boundary lines of the road on which the vehicle is traveling, a boundary line selection unit that selects the boundary line from the line candidates extracted by the line candidate extraction unit, and a boundary recognition unit that recognizes the boundary line based on learning by deep learning. The boundary line selection unit includes a lane change determination unit that determines whether the vehicle has changed lanes based on the selected boundary line and the position of the vehicle, and when determining that the vehicle has changed lanes, selects the boundary line based on the position of the boundary line recognized by the boundary recognition unit when selecting a lane boundary line from the candidate lines extracted by the line candidate extraction unit.

[0011] According to the second driving support device, when it is determined that the vehicle has changed lanes, the boundary line selection unit selects the lane boundary line from the candidate lines extracted by the line candidate extraction unit based on the position of the boundary line recognized by the boundary recognition unit. For example, even if the lane width before and after the lane change is different, the boundary line can be selected by appropriately and quickly performing white line recognition. [Brief description of the drawings]

[0012] [Figure 1] 1 is a diagram showing a vehicle equipped with a driving assistance device according to a first embodiment, and an on-board camera installed in the vehicle. [Diagram 2] 1 is a block diagram showing a configuration of a driving assistance device according to a first embodiment. [Diagram 3] FIG. 2 is a block diagram showing the configuration of a white line detection unit according to the first embodiment. [Figure 4]4 is a flowchart of a white line recognition process executed by a white line recognition unit shown in FIG. 3 . [Diagram 5] 5 is a flowchart of a process for selecting lane boundary lines shown in FIG. 4; [Figure 6] 5A and 5B are diagrams for explaining lane recognition when changing lanes; [Figure 7] FIG. 11 is a block diagram showing the configuration of a white line recognition unit according to a second embodiment. [Figure 8] 8 is a flowchart of a white line recognition process executed by a white line recognition unit shown in FIG. 7 . [Figure 9] 9 is a flowchart of a process for selecting a lane boundary line shown in FIG. 8 . [Figure 10] 11A to 11C are diagrams for explaining erroneous recognition of line types due to zebra zones. [Figure 11] 11A to 11C are diagrams for explaining erroneous recognition of the line type of an auxiliary line. [Figure 12] FIG. 4 is a diagram for explaining noise lines. [Figure 13] 10 is a flowchart of a process for selecting a lane boundary line shown in FIG. 9; DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] (First embodiment) The driving assistance device according to the first embodiment is mounted on a vehicle having an on-board camera 2 as shown in Fig. 1, and includes an image capturing unit 10, a vehicle state quantity sensor unit 11, a white line recognition unit 12, and an alarm / vehicle control unit 13 as shown in Fig. 2. The on-board camera 2 is a camera formed of a well-known CCD camera or the like, and is mounted near the upper part of the windshield inside the vehicle 1 as shown in Fig. 1, and can capture an image of a capturing range 3. The on-board camera 2 captures (i.e., captures) an image ahead of the vehicle 1, and generates image data for the captured image of the front. The image captured by the on-board camera 2 includes the road surface ahead of the vehicle 1.

[0014] The vehicle-mounted camera 2 repeatedly captures images at preset time intervals to obtain images of the road ahead. The vehicle-mounted camera 2 is capable of capturing color images. The image capturing unit 10 acquires image data of the images captured by the vehicle-mounted camera 2 and outputs the image data to the white line recognition unit 12.

[0015] The vehicle state quantity sensor unit 11 acquires state quantities of the vehicle 1 from various sensors that measure the state quantities of the vehicle 1, and outputs the state quantities to the white line recognition unit 12. The sensors include, for example, a vehicle speed sensor and a yaw rate sensor. In addition, for example, a pitch angle sensor and a steering angle sensor may be included. The sensors repeatedly measure state quantities indicating the state of the vehicle 1 at preset time intervals, and the measurement results are acquired by the vehicle state quantity sensor unit 11. The on-board sensor may include a well-known navigation device, and the navigation device may store map information including information on road shapes and the like.

[0016] The white line recognition unit 12 is mainly configured with a well-known microcomputer equipped with a CPU, a ROM, a RAM, and a semiconductor memory such as a flash memory. The white line recognition unit 12 executes various functions as shown in Figures 3 and 4 by the CPU executing a program stored in a non-transitory tangible recording medium. In this example, the semiconductor memory corresponds to the non-transitory tangible recording medium storing the program.

[0017] The white line recognition unit 12 may include one or more microcomputers. The various functions realized by the white line recognition unit 12 are not limited to those realized by a CPU executing a program, and some or all of the functions may be realized by using one or more pieces of hardware.

[0018] The alarm / vehicle control unit 13 is mainly configured with a well-known microcomputer equipped with a CPU, a ROM, a RAM, and semiconductor memories such as flash memory. The alarm / vehicle control unit 13 performs alarm output control for outputting an alarm and driving control for controlling the driving state of the host vehicle 1 based on white line recognition information obtained by white line recognition in the white line recognition unit 12. The alarm output control includes control for outputting an alarm when the host vehicle 1 is about to deviate from a lane (i.e., the host vehicle's lane). The driving control includes steering control and braking control that are performed when the host vehicle 1 is about to deviate from a lane.

[0019] As shown in Fig. 3, the white line recognition unit 12 includes an edge extraction unit 14, a white line paint extraction unit 15, a lane paint determination unit 16, a line candidate extraction unit 17, a boundary selection unit 18, a road shape estimation unit 19, and a deep learning (DNN) boundary recognition unit 20. The boundary selection unit 18 includes a lane change determination unit 31 and a position learning unit 32. Hereinafter, machine learning using deep learning may be simply referred to as machine learning. The white line paint is paint that constitutes lines such as white lines and yellow lines formed as dashed or solid lines on the road along the direction in which the road extends in order to divide areas in the width direction of the road.

[0020] The edge extraction unit 14 extracts edge points, which are pixels with large changes in brightness value, from the image captured by the vehicle-mounted camera 2. A specific method for extracting edge points from a captured image is well known, so a detailed explanation will be omitted here and only an outline will be given. The edge extraction unit 14 searches for pixels with a brightness value change amount equal to or greater than a threshold while scanning the image horizontally from the left end to the right end, and extracts up-edge points and down-edge points.

[0021] An up-edge point is a rising point of a luminance value changing from a low luminance value to a high luminance value, more specifically, a point where the amount of change in the luminance value from a low luminance value to a high luminance value is equal to or greater than a luminance threshold. A down-edge point is a falling point of a luminance value changing from a high luminance value to a low luminance value, more specifically, a point where the amount of change in the luminance value from a high luminance value to a low luminance value is equal to or greater than a luminance threshold. The edge extraction unit 14 repeats the extraction of edge points by scanning in the horizontal direction while shifting the position in the vertical direction of the image, i.e., in the depth direction of the image, to extract edge points from almost the entire region of the image.

[0022] The white line paint extraction unit 15 extracts white line paint, which is a block of paint that constitutes a demarcation line, by connecting the edge points extracted by the edge extraction unit 14. Specifically, from each of the extracted edge points, a substantially rectangular area formed by a plurality of edge points is extracted as the white line paint. More specifically, an up edge point is paired with a down edge point located to the right of the up edge point and closest to the up edge point. Then, a substantially rectangular area in which the pairs are arranged at intervals within a preset interval threshold in the traveling direction of the vehicle 1 is extracted as the white line paint.

[0023] The own lane paint determination unit 16 determines whether the white line paint extracted by the white line paint extraction unit 15 is a white line paint (own lane paint) indicating the boundary line of the own lane. The white line paint determined to be the own lane paint is used by the line candidate extraction unit 17 when extracting candidates for a division line.

[0024] The line candidate extraction unit 17 extracts line candidates that are candidates for boundary lines of the road on which the vehicle 1 is traveling. The line candidate extraction unit 17 extracts lines that exist so as to extend along the traveling direction of the vehicle 1. That is, the line candidate extraction unit 17 extracts demarcation lines that exist so as to extend along the traveling direction of the vehicle 1 by connecting white line paints that have been determined to be the vehicle's lane paint in the traveling direction.

[0025] The boundary line selection unit 18 selects a boundary line from the line candidates extracted by the line candidate extraction unit 17. Specifically, from the multiple demarcation lines, i.e., demarcation line candidates, extracted by the line candidate extraction unit 17, the boundary line selection unit 18 selects demarcation lines suitable as a pair of left and right boundary lines that define the left and right boundaries of a lane (e.g., the own lane), and sets the selected demarcation lines as the boundary lines.

[0026] For example, among the multiple marking lines extracted by the line candidate extraction unit 17, a pair of marking lines that are closest to the vehicle 1 in the left and right directions of the vehicle 1 and have an interval of a predetermined distance or more can be selected as the boundary line of the vehicle lane. Note that various other selection conditions may be taken into consideration depending on the road conditions.

[0027] The boundary line selection unit 18 includes a lane change determination unit 31 and a position learning unit 32.

[0028] The lane change determination unit 31 determines whether the vehicle 1 has changed lanes based on the selected boundary line and the position of the vehicle 1. When the lane change determination unit 31 determines that the vehicle 1 has changed lanes, the boundary line selection unit 18 selects a boundary line based on the position of the boundary line recognized by the DNN boundary recognition unit 20 when selecting a lane boundary line from the candidate lines extracted by the line candidate extraction unit 17.

[0029] The position learning unit 32 learns adjacent lane positions in a time series. The boundary line selection unit 18 can select a boundary line from among the candidate lines extracted by the line candidate extraction unit 17 based on the adjacent lane positions recognized by the DNN boundary recognition unit 20 and learned by the position learning unit 32. If the host vehicle 1 changes lanes in a state in which the adjacent lane positions recognized by the DNN boundary recognition unit 20 and learned by the position learning unit 32 have not been calculated, the boundary line selection unit 18 selects a boundary line using the lane width recognized before the host vehicle 1 changes lanes as a reference position.

[0030] The road shape estimation unit 19 estimates the shape of the road on which the vehicle is currently traveling, including the vehicle's own lane, based on, for example, a pair of currently selected left and right boundary lines. Specifically, the road shape estimation unit 19 estimates various road parameters, such as lane position, lane inclination, lane curvature, lane width, etc., as road parameters indicating the characteristics of the shape of the road on which the vehicle is currently traveling.

[0031] Here, the lane width is the distance between a pair of left and right boundary lines. The lane position is the central position on the lane width based on the center of the vehicle 1, and can be expressed as the distance from the center of the vehicle 1 to that central position. The lane inclination is the inclination of the tangent at the central position of a virtual lane center that passes through the center of the left and right boundary lines with respect to the vehicle travel direction. The lane curvature is the curvature of the lane center. The road shape estimated by the road shape estimation unit 19, i.e., various road parameters, are output to the warning / vehicle control unit 13.

[0032] The DNN boundary recognition unit 20 recognizes boundary lines based on learning by deep learning. Specifically, the DNN boundary recognition unit 20 recognizes the region of the own lane ahead of the own vehicle 1 using the result of machine learning by deep learning of the image ahead of the own vehicle 1 for the image ahead of the own vehicle 1 acquired by the image capturing unit 10. The DNN boundary recognition unit 20 is preferably configured to recognize at least one of the own lane, adjacent lanes, and road edges of the road on which the own vehicle 1 is traveling.

[0033] Machine learning by deep learning for these images is well known and will be described briefly. For example, a large number of images are acquired by photographing the area in front of the vehicle 1. Then, for the large number of images, the objects and the like captured are divided into, for example, a road area, an area of ​​other vehicles, and an area other than other vehicles and roads, and the road area is further divided into an area of ​​the vehicle's lane and other areas. Then, learning by deep learning is performed on the large number of images divided into a plurality of areas in this way.

[0034] The number of images, models, and the number of layers of the models used in deep learning can be appropriately selected based on known techniques. A neural network can be used as a model for this deep learning. The neural network can be configured with multiple layers of interconnected nodes. This network can have tens to hundreds of hidden layers.

[0035] Such a deep learning model is called a deep neural network, and can learn using large-scale labeled data and the structure of a neural network. Note that a convolutional neural network can be used as a deep neural network.

[0036] Using the results of learning as described above, the DNN boundary recognition unit 20 can divide, for example, the latest image acquired this time, into an area of ​​the own lane, an area of ​​roads other than the own lane, an area other than roads, and an area of ​​other vehicles in front of the own vehicle 1. In other words, the DNN boundary recognition unit 20 obtains a recognition result of the own lane area indicating which part of the image acquired this time is the own lane area.

[0037] With the above configurations, the white line recognition unit 12 can execute a learning recognition process to recognize white lines using the results obtained by the processing of the DNN boundary recognition unit 20. The white line recognition unit 12 can also execute a non-learning process to acquire information on the own lane ahead of the own vehicle 1 based on the image ahead of the own vehicle 1 acquired by the image capture unit 10, without using the recognition results by the DNN boundary recognition unit 20.

[0038] Fig. 4 shows a flowchart of the white line recognition process executed by the white line recognition unit 12. The process shown in Fig. 4 is repeatedly executed at a predetermined cycle.

[0039] In step S101, substantially the entire area of ​​the image captured by the vehicle-mounted camera 2 is scanned, and edge points, which are pixels with large changes in luminance value, are extracted. Then, the process proceeds to step S102.

[0040] In step S102, the edge points extracted in step S101 are connected to each other to extract white line paint, which is a block of paint that constitutes the demarcation line. Specifically, from each of the extracted edge points, a substantially rectangular area formed by a plurality of edge points is extracted as the white line paint. Then, the process proceeds to step S103.

[0041] In step S103, it is determined whether or not each of the white line paints extracted in step S102 is a current lane paint, and then the process proceeds to step S104.

[0042] In step S104, line candidates are extracted by connecting the white line paints determined to be the own vehicle lane paint in the traveling direction to extract the lane markings that exist so as to extend along the traveling direction of the own vehicle 1. Then, the process proceeds to step S105.

[0043] In step S105, a boundary line is selected from the line candidates extracted in step S104. Specifically, from the multiple lane lines extracted in step S104, i.e., from the lane line candidates, a pair of lane lines suitable as a pair of left and right boundary lines defining the left and right boundaries of a lane (e.g., the vehicle's lane) is selected, and the selected lane lines are recognized as the boundary lines. Then, the process proceeds to step S107.

[0044] In step S107, the process of selecting lane boundary lines shown in Fig. 5 is executed. First, in step S201, the position of the adjacent lane recognized by deep learning is learned. Then, the process proceeds to step S202.

[0045] For example, as shown in Fig. 6, on a road divided by a right white solid line 40 and a left white solid line 43, lanes 44L, 44C, and 44R are divided by white dashed lines 41 and 42, and when the vehicle 1 is traveling in the right lane 44R, the lane width w1 of the lane 44R, which is the vehicle's lane, can be represented by the distance between the lateral position x0 of the left white solid line 43 and the lateral position x1 of the white dashed line 41. The lane width w2 of the lane 44C adjacent to the vehicle's lane can be represented by the distance between the lateral position x1 of the white dashed line 41 and the lateral position x2b of the white dashed line 42. The lane width w2 is narrower than the lane width w1, and the lateral position x2a at a distance w1 from the lateral position x1 of the white dashed line 41 is shifted to the left of the lateral position x2b. When the vehicle 1 is traveling in lane 44R, the adjacent lane position estimated based on the lane width w1 of the vehicle's lane is lateral position x2a, and the adjacent lane position recognized by deep learning is lateral position x2b.

[0046] In step S202, it is determined whether the host vehicle 1 has changed lanes. For example, in Fig. 6, if the host vehicle 1 has changed lanes from lane 44R to lane 44C, it is determined that the host vehicle 1 has changed lanes, and the process proceeds to step S203. If the host vehicle 1 has not changed lanes, the process proceeds to step S205.

[0047] In step S203, it is determined whether or not the position learning for the adjacent lane position recognized by deep learning executed in step S201 is completed. If it is completed, proceed to step S204. If it is not completed, proceed to step S205.

[0048] In step S204, the boundary line of the own lane is selected from the candidate lines extracted in step S104 based on the lateral position x2b, which is the adjacent lane position recognized by deep learning. In step S205, the boundary line of the own lane is selected from the candidate lines extracted in step S104, using the lane width w1 recognized before the lane change of the own vehicle 1 as a reference position. In other words, the boundary line of the own lane is selected from the candidate lines extracted in step S104 based on the lateral position x2a.

[0049] As described above, according to the first embodiment, even if the lane width before and after the lane change is different, the adjacent lane position by deep learning can be recognized as the boundary line of the own lane after the lane change. Therefore, as shown in Fig. 6, even if the lane width before and after the lane change is different, the lane boundary line can be selected by appropriately and quickly performing white line recognition.

[0050] Second embodiment The driving support device according to the second embodiment differs from the driving support device according to the first embodiment in the configuration of the white line recognition unit 12a. Fig. 7 shows the white line recognition unit 12a according to the second embodiment. The white line recognition unit 12a differs from the white line recognition unit 12 shown in Fig. 3 in that it includes a class identification unit 21 and that the boundary line selection unit 18 includes a feature calculation unit 33, a line type determination unit 34, and a noise line determination unit 35. Other configurations are generally similar to those of the white line recognition unit 12, so description thereof will be omitted.

[0051] The class identification unit 21 assigns a predetermined class identification to each pixel of the image. The class identification unit 21 is preferably configured to identify at least one of the following classes as a class: road surface, own lane white line paint, own lane gap, other line white line paint, zebra, solid object, and moving object. Note that the own lane white line paint class, own lane gap class, other line white line paint class, and zebra class are classes obtained by further subdividing the white line class. The class identification assigned by the class identification unit 21 is output to the own lane paint determination unit 16 and the boundary line selection unit 18.

[0052] The feature calculation unit 33 calculates feature amounts for selecting boundary lines from the line candidates extracted by the line candidate extraction unit 17. The feature calculation unit 33 calculates the inclusion rate for each class identification unit 21 assigned to the white line paint in the line candidates extracted by the line candidate extraction unit 17. For example, the feature calculation unit 33 calculates the inclusion rate that is the proportion of feature amounts identified as the road surface class or the white line class.

[0053] The line type determination unit 34 determines the line type of the line candidate based on the content rate of each class calculated by the feature calculation unit 33. For example, the line type determination unit 34 determines that the line type of the line candidate is a solid line based on at least one of the content rate of the zebra class calculated by the feature calculation unit 33, the line type of the own vehicle lane (DNN own vehicle lane) recognized by the DNN boundary recognition unit 20, and the consistency between the own vehicle lane extracted by the line candidate extraction unit 17 and the own vehicle lane recognized by the DNN boundary recognition unit 20. Also, for example, the line type determination unit 34 determines that the line type of the line candidate is an auxiliary line in a compound line based on at least one of the content rate of the other white line paint class calculated by the feature calculation unit 33, the line type of the own vehicle lane (DNN own vehicle lane) recognized by the DNN boundary recognition unit 20, and the consistency between the own vehicle lane extracted by the line candidate extraction unit 17 and the own vehicle lane recognized by the DNN boundary recognition unit 20.

[0054] The noise line determination unit 35 determines whether or not a line candidate is a noise line based on the inclusion rate for each class calculated by the feature calculation unit 33. The feature calculation unit 33 calculates an inclusion rate, which is the proportion of features classified as the road surface class or white line class by the class identification unit 21, in the line candidates extracted by the line candidate extraction unit 17, and the noise line determination unit 35 determines whether or not a line candidate is a noise line based on this inclusion rate.

[0055] A flowchart of the white line recognition process executed by the white line recognition unit 12a is shown in Fig. 8. The process shown in Fig. 8 is repeatedly executed at a predetermined cycle.

[0056] In step S301, an edge extraction process is executed, and in step S302, a white line paint extraction process is executed. The processes shown in S301 and S302 are similar to the processes shown in steps S101 and S102, and therefore a description thereof will be omitted.

[0057] In step S303, it is determined whether each of the white line paints extracted in step S302 is the own lane paint. In this determination, the class classification given to the white line paint may be taken into consideration. Then, the process proceeds to step S304.

[0058] In step S304, line candidates are extracted, and in step S305, boundary recognition is performed. The processes shown in S304 and S305 are similar to the processes shown in steps S104 and S105, and therefore description thereof will be omitted.

[0059] In step S306, a predetermined class is assigned to each pixel of the image, and at least one of the following is identified as a class: road surface, lane paint, lane gap, other lane paint, zebra, solid object, and moving object.

[0060] In step S307, the process of selecting lane boundary lines shown in Fig. 9 is executed. First, in step S401, feature amounts for selecting boundary lines are calculated from the line candidates extracted in step S304. A content rate, which is the ratio of feature amounts classified as the road surface class or the white line class, in the line candidates extracted in step S304 is calculated. Then, the process proceeds to step S402.

[0061] In step S402, the type of line is determined based on the content rate of each class calculated in step S401, the boundary recognition result by deep learning, and the line candidates extracted in step S304.

[0062] For example, the line type determination of the white solid line 45 shown in Fig. 10(a) will be described as an example. In Fig. 10(a), a white solid line 45, a yellow solid line 46, and a white solid line 47 are arranged in this order on the right side of the vehicle's lane, with a diagonal line 45a extending from the white solid line 45 toward the yellow solid line 46 and a diagonal line 47a extending from the white solid line 47 toward the yellow solid line 46, forming a zebra zone with the white solid lines 45, 47 and the diagonal lines 45a and 47a.

[0063] When recognizing the white solid line 45, as shown in FIG. 10(b), the white solid line 45 may be recognized as being interrupted at the connection portion between the white solid line 45 and the diagonal line 45a, and may be erroneously recognized as a white dashed line 45e.

[0064] In contrast, if all three conditions are met: (condition 1-1) the white solid line 45 is determined to be a zebra class in class identification, (condition 1-2) the white solid line 45 is determined to be a solid line as a result of boundary recognition by deep learning, and (condition 1-3) the own lane extracted by line candidate extraction matches the own lane recognized by deep learning, then the line type of the white solid line 45 can be determined to be a solid line.

[0065] In addition, for example, as shown in FIG. 11, when the boundary line of the vehicle's lane is constituted by solid lines 48, 50 on the outside and auxiliary lines 49, 51 on the inside, the gap length and line length of the auxiliary lines 49, 51 are determined not to satisfy the conditions for determining that the line type is an auxiliary line, and the auxiliary lines 49, 51 may be mistaken for white dashed lines.

[0066] In contrast, when both of the following two conditions are satisfied: (Condition 2-1) the proportion of the auxiliary lines 49, 51 determined to be in the other white line class in class identification is equal to or greater than a predetermined first proportion threshold; and (Condition 2-2) the scene is determined to be a composite line scene in scene identification as a result of boundary recognition by deep learning, the line type of the auxiliary lines 49, 51 can be determined to be a composite line. Alternatively, by adding the following condition to the gap length and line length conditions: (Condition 2-3) the proportion of the auxiliary lines 49, 51 determined to be in the other white line class in class identification is equal to or greater than a predetermined second proportion threshold, the line type of the auxiliary lines 49, 51 can be determined to be an auxiliary line in a composite line. Note that the first proportion threshold is a value indicating a proportion greater than the second proportion threshold.

[0067] After step S402, the process proceeds to step S404. In step S404, a noise line determination is performed based on the class inclusion rate calculated in step S401. For example, it is determined whether or not a line candidate is a noise line based on the inclusion rate, which is the ratio of features identified as the white line class.

[0068] As shown in FIG. 12, a road is divided by a right white solid line 53 and an auxiliary line 54 and a left white solid line 57 and an auxiliary line 58, and lanes 59L, 59R are divided by a white dashed line 55 and an auxiliary line 56. A case where the vehicle 1 is traveling in the left lane 59L is described as an example. As shown in FIG. 12, if a coal tar line 60 exists inside the white dashed line 55 in the lane 59L that is the vehicle's lane, the coal tar line 60 may be erroneously recognized as an auxiliary line. In this case, the coal tar line 60 can be determined to be a noise line when both of the following two conditions are satisfied: (Condition 3-1) the proportion of the coal tar line 60 determined to be in the other white line class in the class identification is less than a predetermined third proportion threshold, and (Condition 3-2) the proportion of the coal tar line 60 determined to be in the vehicle's lane white line class in the class identification is less than a predetermined fourth proportion threshold. The third proportion threshold is a value indicating a proportion greater than the fourth proportion threshold.

[0069] After step S404, the process proceeds to step S405. The process of selecting lane boundary lines shown in FIG. 13 is executed. In step S501, it is determined whether a predetermined line candidate has been recognized with high reliability. The reliability is determined comprehensively based on the brightness of the image, the total extension distance of the line candidate, the number of edges, etc. If it is determined that the reliability is high, the process proceeds to step S502. If it is determined that the reliability is not high, the process ends.

[0070] In step S502, it is determined whether the travel distance of the vehicle 1 is equal to or greater than L1 under the condition that the reliability is determined to be high. If the travel distance is equal to or greater than L1, the process proceeds to step S503. If the travel distance is less than L1, the process ends.

[0071] In step S503, it is determined whether or not the line candidate is closest to the vehicle 1. If it is closest, the process proceeds to step S504. If it is not closest, the process ends.

[0072] In step S504, it is determined whether the line candidate is a noise line. If it is not a noise line, the process proceeds to step S506, where the line candidate is selected as a lane boundary line. If it is determined to be a noise line, the process proceeds to step S505. If it is determined in step S404 that the line candidate is a noise line such as the coal tar line 60, a negative determination is made in step S504, and the process proceeds to step S505.

[0073] In step S505, in a state in which the line candidate is determined to be a noise line, it is determined whether or not the travel distance of the vehicle 1 is less than L2. If it is less than L2, the process proceeds to step S506, where the line candidate is selected as a lane boundary line. If it is equal to or greater than L2, the process ends. When the process shown in FIG. 13 ends, the process shown in FIG. 9 also ends.

[0074] As described above, according to the white line recognition unit 12a of the second embodiment, in addition to boundary line recognition by the DNN boundary recognition unit 20, a boundary line can be selected based on class classification by the class classification unit 21 and the content rate of each class by the feature calculation unit 33 using the class classification, etc., so that erroneous recognition in boundary line recognition for white line paint accompanied by zebra scenes as shown in FIG. 10, erroneous recognition in boundary line recognition for white line paint accompanied by auxiliary lines as shown in FIG. 11, and erroneous recognition in boundary line recognition due to noise lines such as coal tar lines as shown in FIG. 12 can be suppressed.

[0075] According to each of the above embodiments, the following effects can be obtained.

[0076] The driving assistance device comprises an image capturing unit 10 that acquires images captured by an on-board camera 2, a vehicle state quantity sensor unit 11 that acquires state quantities of the vehicle 1 measured by a sensor, a white line recognition unit 12, 12a that calculates white line recognition information by recognizing white lines around the vehicle 1 using the images and state quantities, and an alarm / vehicle control unit 13 that issues an alarm and controls the vehicle 1 based on the white line recognition information.

[0077] In the driving assistance device according to the first embodiment, the white line recognition unit 12 includes a line candidate extraction unit 17 that extracts line candidates that are candidates for the boundary line of the road on which the vehicle 1 is traveling, a boundary line selection unit 18 that selects a boundary line from the line candidates extracted by the line candidate extraction unit 17, and a DNN boundary recognition unit 20 that recognizes the boundary line based on learning by deep learning. The boundary line selection unit 18 includes a lane change determination unit 31 that determines whether the vehicle 1 has changed lanes based on the selected boundary line and the position of the vehicle 1, and when determining that the vehicle 1 has changed lanes, selects the boundary line based on the position of the boundary line recognized by the DNN boundary recognition unit 20 when selecting the lane boundary line from the candidate lines extracted by the line candidate extraction unit 17.

[0078] According to the white line recognition unit 12 according to the first embodiment, when determining that the vehicle 1 has changed lanes, the boundary line selection unit 18 selects a lane boundary line from the candidate lines extracted by the line candidate extraction unit 17 based on the position of the boundary line recognized by the DNN boundary recognition unit 20. Therefore, even if the lane width before and after the lane change is different, for example, the boundary line can be appropriately and quickly recognized and a boundary line can be selected.

[0079] The DNN boundary recognition unit 20 is preferably configured to recognize at least one of the own lane, adjacent lanes, and road edges of the road on which the own vehicle 1 is traveling.

[0080] The boundary line selection unit 18 includes a position learning unit 32 that learns adjacent lane positions, which are positions of adjacent lanes, in a time series. Furthermore, the boundary line selection unit 18 is configured to select a boundary line from among the candidate lines extracted by the line candidate extraction unit 17 based on the adjacent lane positions recognized by the DNN boundary recognition unit 20 and learned by the position learning unit 32. This allows the boundary line to be selected more appropriately and quickly by performing white line recognition. Furthermore, when the vehicle 1 changes lanes in a state in which the adjacent lane positions recognized by the DNN boundary recognition unit 20 and learned by the position learning unit 32 have not been calculated, the boundary line selection unit 18 is configured to select a boundary line using the lane width recognized before the vehicle 1 changes lanes as a reference position. Therefore, even if the calculation of the adjacent lane positions is not completed in time, the boundary line can be selected appropriately to some extent.

[0081] In the driving assistance device according to the second embodiment, the white line recognition unit 12a includes a line candidate extraction unit 17 that extracts line candidates that are candidates for the boundary line of the road on which the vehicle 1 is traveling, a boundary selection unit 18 that selects a boundary line from the line candidates extracted by the line candidate extraction unit 17, a DNN boundary recognition unit 20 that recognizes the boundary line based on learning by deep learning, and a class identification unit 21 that assigns a predetermined class identification to each pixel of the image. The boundary selection unit 18 includes a feature calculation unit 33 that calculates a feature amount for selecting a boundary line from the line candidates extracted by the line candidate extraction unit 17. The feature calculation unit 33 calculates an inclusion rate, which is the ratio of the feature amount identified as the road surface class or the white line class by the class identification unit 21 in the line candidates extracted by the line candidate extraction unit 17, and the boundary selection unit 18 judges whether the line candidate is a noise line or not based on the inclusion rate.

[0082] According to the white line recognition unit 12a of the second embodiment, the boundary line selection unit 18 includes a feature calculation unit 33 that calculates an inclusion rate, which is a ratio of feature amounts that are classified as the road surface class or the white line class in the line candidate. The feature calculation unit 33 can further determine a noise line based on the inclusion rate, and therefore can prevent a noise line from being mistaken for a boundary line. As a result, it is possible to appropriately and quickly perform white line recognition and select a boundary line.

[0083] The class identification unit 21 is preferably configured to identify at least one of the following classes for each pixel of the image: road surface, lane white paint, lane gap, other line white paint, zebra, three-dimensional object, and moving object.

[0084] The feature calculation unit 33 is preferably configured to calculate the content rate of each class based on the class identification given to the white line paint in the line candidate. In this case, the boundary selection unit 18 is preferably configured to determine whether or not the line candidate is a noise line based on the content rate of each class calculated by the feature calculation unit 33. Since a noise line can be determined based on the content rate of each class, it is possible to more effectively prevent a noise line from being mistaken for a boundary line.

[0085] The boundary line selection unit 18 may include a line type determination unit 34 that determines the line type of the line candidate based on the content rate of each class calculated by the feature calculation unit 33. In this case, the line type determination unit 34 may be configured to determine that the line type of the line candidate is a solid line based on at least one of the content rate of the zebra class calculated by the feature calculation unit 33, the line type of the own lane recognized by the DNN boundary recognition unit 20, and the consistency between the own lane recognized by the line candidate and the own lane recognized by the DNN boundary recognition unit 20. In white line paint including zebra, it is possible to effectively suppress erroneous recognition of a solid line as a broken line.

[0086] The line type determination unit 34 may be configured to determine that the line type of the line candidate is an auxiliary line based on at least one of the content rate of the other white line paint class calculated by the feature calculation unit 33, the line type of the own lane recognized by the DNN boundary recognition unit 20, and the consistency between the own lane recognized by the line candidate and the own lane recognized by the DNN boundary recognition unit 20. This can effectively prevent the auxiliary line from being erroneously recognized as a broken line.

[0087] The components of the white line recognition section 12 according to the first embodiment and the components of the white line recognition section 12a according to the second embodiment can be used in one white line recognition section.

[0088] The control unit and the method described in the present disclosure may be realized by a special-purpose computer provided by configuring a processor and a memory programmed to execute one or more functions embodied in a computer program. Alternatively, the control unit and the method described in the present disclosure may be realized by a special-purpose computer provided by configuring a processor with one or more dedicated hardware logic circuits. Alternatively, the control unit and the method described in the present disclosure may be realized by one or more special-purpose computers configured by a combination of a processor and a memory programmed to execute one or more functions and a processor configured with one or more hardware logic circuits. In addition, the computer program may be stored in a computer-readable non-transitory tangible recording medium as instructions executed by a computer. [Explanation of symbols]

[0089] 10: image capturing unit, 11: vehicle state quantity sensor unit, 12, 12a: white line recognition unit, 13: warning / vehicle recognition unit, 17: line candidate extraction unit, 18: boundary line selection unit, 20: DNN boundary recognition unit, 31: lane change determination unit

Claims

1. An image capturing unit (10) for acquiring an image captured by an on-board camera; A vehicle state quantity sensor unit (11) that acquires a state quantity of the vehicle measured by a sensor; a white line recognition unit (12a) that uses the image and the state quantity to recognize white lines around the vehicle and calculates white line recognition information; a warning / vehicle control unit (13) that issues a warning and controls the vehicle based on the white line recognition information, The white line recognition unit is a line candidate extraction unit (17) that extracts line candidates that are candidates for boundary lines of a road on which the vehicle is traveling; a boundary selection unit (18) that selects the boundary line from the line candidates extracted by the line candidate extraction unit; A boundary recognition unit (20) that recognizes the boundary line based on learning by deep learning; A class identification unit (21) that assigns a predetermined class identification to each pixel of the image, the boundary selection unit includes a feature calculation unit (33) that calculates a feature amount for selecting the boundary line from the line candidates extracted by the line candidate extraction unit, the feature calculation unit calculates a content rate, which is a ratio of the feature amount classified as a road surface class or a white line class by the class classification unit, in the line candidates extracted by the line candidate extraction unit; The boundary selection unit determines whether the line candidate is a noise line based on the content rate.

2. The driving assistance device according to claim 1 , wherein the class identification unit identifies at least one of a road surface, a lane marking, a lane gap, a marking on other lines, a zebra, a three-dimensional object, and a moving object as a class for each pixel of the image.

3. The driving support device according to claim 1 , wherein the feature calculation unit calculates the content rate for each class based on a class identification given to the white line paint in the line candidate.

4. The driving support device according to claim 3 , wherein the boundary selection unit determines whether the line candidate is a noise line based on the inclusion rate for each of the classes calculated by the feature calculation unit.

5. The driving support device according to claim 3 , wherein the boundary selection unit includes a line type determination unit that determines a line type of the line candidate based on the content rate for each class calculated by the feature calculation unit.

6. 6. The driving assistance device according to claim 5, wherein the line type determination unit determines that the line type of the line candidate is a solid line based on at least one of the content rate of zebra classes calculated by the feature calculation unit, the line type of the own lane recognized by the boundary recognition unit, and the consistency between the own lane recognized by the line candidate and the own lane recognized by the boundary recognition unit.

7. 6. The driving assistance device according to claim 5, wherein the line type determination unit determines that the line type of the line candidate is an auxiliary line based on at least one of the content rate of the other line white line paint class calculated by the feature calculation unit, the line type of the own lane recognized by the boundary recognition unit, and the consistency between the own lane recognized by the line candidate and the own lane recognized by the boundary recognition unit.

8. An image capturing unit (10) for acquiring an image captured by an on-board camera; The present invention is applied to a driving assistance device including a vehicle state quantity sensor unit (11) that acquires a state quantity of the vehicle measured by a sensor, On the computer, a white line recognition process for calculating white line recognition information by recognizing white lines around the vehicle using the image and the state amount; a warning / vehicle control process for issuing a warning and controlling the host vehicle based on the white line recognition information; The white line recognition process includes: A line candidate extraction process for extracting line candidates that are candidates for boundary lines of a road on which the vehicle is traveling; a boundary line selection process for selecting the boundary line from the line candidates extracted by the line candidate extraction process; A boundary recognition process for recognizing the boundary line based on learning by deep learning; A class classification process for assigning a predetermined class classification to each pixel of the image, the boundary line selection process includes a feature calculation process of calculating a feature amount for selecting the boundary line from the line candidates extracted by the line candidate extraction process, a content rate, which is a ratio of the feature amount classified as a road surface class or a white line class by the class classification process, in the line candidates extracted by the line candidate extraction process in the feature calculation process; a program for determining whether the line candidate is a noise line based on the content rate in the boundary line selection process;

Images