Driving assistance device and driving assistance method

Abstract

To track a compartment line in high precision.SOLUTION: A driving support device for recognizing a compartment line around the own vehicle using at least one sensor, comprises: a compartment line detecting unit that detects information on the compartment line from data acquired by the sensor; a compartment line sort candidate estimating unit that recognizes a compartment line sort from the detected compartment line information and calculates a likelihood for each compartment line sort; a compartment line sort branch point estimating unit that detects a compartment line sort branch point, which is a point at which the likelihood of the compartment line sort changes with respect to a traveling direction of the own vehicle, and distinguishes areas before and after the compartment line sort branch point into a first area and a second area; and a compartment line information integrating unit that integrates the compartment line of the first area and the compartment line of the second area into a continuously recognizable compartment line when the compartment line of the first area and the compartment line of the second area are of the same compartment line sort.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a driving assistance device. [Background technology]

[0002] Currently, various driving assistance systems are being developed and commercialized. In order to realize applications such as lane keeping systems and lane change systems, it is necessary to use sensors mounted on the vehicle to detect and track the lane markings around the vehicle with high accuracy.

[0003] A Kalman filter is generally used to track lane markings. This method can predict the position of an object at the next time based on the object's position up to the previous time. However, if the white line being tracked cannot be correctly matched to the obtained observation (detected lane marking position), the accuracy of the prediction will decrease. The position, shape, and type of lane markings are used to match lane markings. However, in existing methods, one lane marking type is generally assigned to an observed lane marking, but in the case of sensors with a wide observation range, multiple lane marking types may be included. For this reason, in places where the lane marking type changes frequently, such as merging lanes on expressways and near intersections, it is difficult to correctly distinguish the lane marking type, and lane marking tracking may fail.

[0004] The following prior art is included as background art in this technical field. Patent Document 1 (JP Patent Publication 2010-244456A) describes a boundary recognition device in which a boundary candidate extraction unit extracts boundary candidate for lanes on a road from an image captured by an on-board camera using known image processing such as pattern matching and Hough transform, one or more types of boundary feature calculation units calculate the certainty of the extracted boundary candidate as a boundary in the form of a likelihood, a boundary feature integration unit multiplies and integrates the calculated likelihoods to output a likelihood indicating the boundary in the form of a likelihood, a boundary candidate having the maximum likelihood among the output likelihoods is selected as a boundary on the road by a boundary selection unit, and the boundary feature calculation unit calculates the likelihood of the boundary candidate using brightness variance, internal edge amount, etc., and changes and outputs the likelihood of the boundary candidate according to the feature amount detected by a road surface feature extraction unit or the like. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2010-244456 A Summary of the Invention [Problem to be solved by the invention]

[0006] Systems that use information on surrounding lane lines, such as lane keeping systems and lane change systems, require stable lane line information over a long period of time. In order to obtain stable lane line information over a long period of time, it is necessary to associate the recognition results of each frame and integrate the observation results of several frames, since the observation results of only one frame may contain momentary non-detection or false detection. When applying the technology disclosed in Patent Document 1, only one lane line type can be assigned to the observation results, and therefore even if the lane line types change frequently, the one lane line type that is most frequently included in the observation results is assigned, which may result in failure to associate the lane lines. In addition, since the lane line positions that are output depend on the lane line types, it is desirable to assign lane line types to the observation results for each area, rather than one lane line type.

[0007] Therefore, an object of the present invention is to provide a driving assistance device that tracks lane markings with high accuracy by using a sensor attached to a vehicle. [Means for solving the problem]

[0008] A representative example of the invention disclosed in the present application is as follows: A driving support device that recognizes marking lines around a vehicle using at least one sensor, and calculates marking line information from data acquired by the sensor. detection A marking line detection unit for detecting a marking line; detection The type of lane markings is determined from the lane markings information. Estimation The likelihood of each lane marking type is calculated as Estimation a lane marking type candidate estimating unit for estimating the likelihood of the lane marking type in the traveling direction of the host vehicle; Based on A lane marking type branch point is detected, and an area before and after the lane marking type branch point is divided into a first area and a second area. Split a lane marking type branch point estimating unit for estimating a lane marking in the first area and a lane marking in the second area; Based on the result of the integration decision, A driving assistance device comprising: a lane marking information integrating unit that integrates the lane markings of the first area and the lane markings of the second area into a lane marking that can be continuously recognized. Effect of the Invention

[0009] According to one aspect of the present invention, it is possible to track lane lines with high accuracy by using a sensor attached to a vehicle. Problems, configurations and effects other than those described above will become apparent from the following description of the embodiments. [Brief description of the drawings]

[0010] [Figure 1] 1 is a diagram illustrating a configuration of a driving assistance device according to a first embodiment. [Diagram 2] 11 is a flowchart of a process executed by a lane marking detection unit according to the first embodiment. [Diagram 3] FIG. 4 is a diagram illustrating an example of a method for extracting a lane marking candidate according to the first embodiment. [Figure 4] FIG. 4 is a diagram illustrating an example of a method for extracting a lane marking candidate according to the first embodiment. [Diagram 5] FIG. 2 is a diagram showing an example of a method for detecting a lane marking having a certain width according to the first embodiment; [Figure 6] 11 is a flowchart of a process executed by a lane marking type candidate estimation unit according to the first embodiment. [Figure 7] FIG. 2 is a diagram illustrating an example of a lane marking type candidate estimation method according to the first embodiment. [Figure 8] FIG. 4 is a diagram illustrating an example of lane marking information according to the first embodiment. [Figure 9] 11 is a flowchart of a process executed by a lane marking type integrating unit according to the first embodiment. [Figure 10] FIG. 11 is a diagram showing an example of integrated lane line information according to the first embodiment. [Figure 11] 11 is a flowchart of a process executed by a lane marking type branch point estimation unit according to the first embodiment. [Figure 12] FIG. 2 is a diagram illustrating an example of a lane marking type branch point estimation method according to the first embodiment. [Figure 13] 11 is a flowchart of a process executed by a lane line integration necessity determination unit according to the first embodiment. [Figure 14] 11 is a flowchart of a process executed by a lane line reliability estimation unit according to the first embodiment. [Figure 15] 11 is a flowchart of a process executed by a lane marking information integrating unit according to the first embodiment. [Figure 16] FIG. 2 is a diagram illustrating an example of a lane marking information integration method according to the first embodiment. [Figure 17] FIG. 2 is a diagram illustrating an example of a lane marking information integration method according to the first embodiment. [Figure 18] FIG. 11 is a diagram illustrating a configuration of a driving assistance device according to a second embodiment. [Figure 19] 13 is a flowchart of a process executed by a scene understanding unit according to the second embodiment. [Figure 20] FIG. 11 is a diagram illustrating an example of a scene understanding method according to a second embodiment. [Figure 21] 13 is a flowchart of a process executed by a lane marking type integrating unit according to the second embodiment. [Figure 22] FIG. 13 is a diagram illustrating an example of a list used in the lane marking type integrating unit of the second embodiment. [Diagram 23] 13 is a flowchart of a process executed by a lane marking type integrating unit according to the second embodiment. [Figure 24] FIG. 11 is a diagram illustrating a configuration of a driving assistance device according to a third embodiment. [Diagram 25] 13 is a flowchart of a process executed by a lane marking type candidate estimation unit according to a third embodiment. [Figure 26] FIG. 13 is a diagram illustrating an example of a lane marking candidate type estimating method according to a third embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] <Example 1> FIG. 1 is a diagram showing a configuration of a driving support device 100 according to a first embodiment of the present invention. The driving support device 100 shown in FIG. 1 is mounted on a moving body such as a passenger car, and is used in combination with an external system 300. In the following description, the moving body on which the driving support device 100 is mounted is referred to as the "own vehicle". The driving support device 100 operates repeatedly at a predetermined timing (for example, every 50 milliseconds). The driving support device 100 has a function of acquiring data from various sensors 110 such as a camera and LiDAR mounted on the own vehicle, and a function of acquiring vehicle information 120 such as speed, steering wheel turning angle, shift brake, and blinker state from the vehicle.

[0012] As shown in FIG. 1, the driving assistance device 100 has, as its functions, a lane line detection unit 200, a lane line type candidate estimation unit 210, a lane line type integration unit 220, a lane line type branch point estimation unit 230, a lane line integration necessity determination unit 240, a lane line reliability estimation unit 250, and a lane line information integration unit 260, and outputs the results of calculations to an external system 300. The driving assistance device 100 has, for example, a CPU or GPU as a calculation device, and a RAM or ROM as a storage device, and can realize these functions by expanding a predetermined program stored in the ROM into the RAM and executing it by the CPU. Note that some or all of the functions of the driving assistance device 100 may be realized using a hardware calculation device such as an FPGA or ASIC.

[0013] The driving support device 100 receives data acquired by a sensor 110 attached to the vehicle and vehicle information 120 acquired from the vehicle via a network such as CAN. The driving support device 100 sequentially acquires data from the sensor 110 (e.g., a camera, LiDAR) mounted on the vehicle at a predetermined frame rate. The vehicle information 120 refers to information related to the movement of the vehicle acquired from the vehicle, and information equivalent thereto, such as vehicle speed, steering angle, vehicle turning radius, accelerator and brake states, and shift lever state. The vehicle information 120 is also sequentially acquired at a predetermined frame rate, similar to the information acquired from the sensor 110.

[0014] The lane marking detection unit 200 detects lane marks from data obtained from the sensor 110. It is assumed that the lane marks are detected by a known method, for example, a Line Segment Detector (LSD) or a Hough transform when extracting lane marks from an image. The lane marking detection unit 200 outputs position information of the detected lane marks to the lane marking type candidate estimation unit 210.

[0015] The lane line type candidate estimation unit 210 estimates the lane line type to which each lane line detected by the lane line detection unit 200 is likely to belong. The lane line type can be estimated using a known method, such as pattern matching that uses patterns of lane line types, or a lane line model that models the periodicity of each lane line type. The lane line type candidate estimation unit 210 outputs the lane line type determination result to the lane line type integration unit 220.

[0016] The lane marking type integrating unit 220 integrates, for each lane marking, the information on the type of each lane marking estimated by the lane marking type candidate estimation unit 210. As a result of the integration, the lane marking type with a likelihood equal to or greater than a certain level is assigned as the lane marking type. The lane marking type integrating unit 220 outputs the assigned lane marking type to the lane marking type branch point estimation unit 230.

[0017] The lane line type branching point estimation unit 230 detects a point where the lane line type changes when the lane line type integrating unit 220 integrates lane lines and multiple lane line types are assigned to one integrated lane line. The point where the lane line type changes can be detected by using the proportion of lane line types contained in the integrated lane lines. When a change point where the lane line type changes is found, the lane line areas before and after the change point are divided into areas and lane line types are assigned to the divided areas. The lane line type branching point estimation unit 230 outputs the lane line types assigned to each divided area to the lane line integration necessity determination unit 240.

[0018] The lane line merging necessity determination unit 240 determines whether or not it is necessary to merge the divided lane lines. The result of the determination is used by the lane line information integrating unit 260. Note that the lane line merging necessity determination unit 240 is effective when the road structure is complex and there are many diagonal white lines, so it does not have to be provided when the road structure is simple.

[0019] The lane line reliability estimation unit 250 assigns a reliability to the detected lane lines. For example, the reliability is generally determined by lowering the reliability of a lane line detected by a camera the farther it is from the vehicle, and by increasing the reliability when a lane line is detected in the same location by multiple sensors 110. The lane line reliability estimation unit 250 outputs the estimated reliability to the lane line information integrating unit 260.

[0020] The lane line information integration unit 260 associates the tracked lane lines with newly observed lane lines, integrates the information of the associated lane lines, and determines whether the lane lines are to be output. For example, the lane lines are associated with each other using the shape, position, type, etc. of the lane lines. In addition, unnecessary lane lines may be excluded from the output targets by referring to the lane widths defined in the Road Structure Act. The lane line information integration unit 260 outputs information on the lane lines determined to be output targets to the external system 300.

[0021] The above-mentioned functional blocks may detect the lane markings on the left and right sides of the vehicle separately. The consistency of the lane markings detected on both sides of the vehicle may be determined by referring to rules defined in the road structure example.

[0022] The external system 300 may be, for example, a system that uses marking line information to recognize lanes in which the vehicle can travel, such as a lane keeping system or a lane change system, or an application that senses the surrounding environment of the route traveled by the vehicle and generates a map.

[0023] Next, the process executed by the driving assistance device 100 will be described with reference to a flowchart in the case where at least one of a camera and a LiDAR is used as the sensor 110.

[0024] FIG. 2 is a flowchart of the process executed by the lane marking detection unit 200.

[0025] First, in S201, the lane line detection unit 200 extracts line segments that are candidates for lane lines from the acquired data. The candidates for lane lines include lane lines that are generally made of white or orange lines, as well as lane lines that are made of Bott's dots or cat's eyes. FIG. 3 shows an example of a method for extracting lane line candidates. FIG. 3 shows an overhead image converted from an image acquired by a camera attached to the front of the vehicle. It is assumed that lane line candidates are detected by a known method, and lane lines can be extracted from an image using, for example, a Line Segment Detector (LSD) or a Hough transform. In addition, lane line candidates can be detected from a sequence of points acquired by LiDAR using a method of performing road surface estimation based on the vehicle attitude and extracting a sequence of points on the road surface that have a reflection intensity of a certain level or more.

[0026] Next, in S202, the lane marking detection unit 200 filters the lane marking candidates extracted in S201 to remove noise. This is because when lines are extracted using a method such as LSD, road markings are also extracted as shown in FIG. 3(b) in addition to cracks on the road surface and shadows of utility poles and buildings. For example, there is a method of filtering using the direction and length of the line segment relative to the vehicle. Specifically, as shown in FIG. 4, the inclination of the line segment is calculated based on the direction of the vehicle, and a histogram is created in which the length of the line segment is used as a weight and the range from -90° to +90° is divided into predetermined angle intervals (for example, 10° increments). Then, as shown in FIG. 3(c), only the range of the largest value ±10° is left out of the created histogram, and the rest are discarded. The above-mentioned histogram is created by dividing the histogram into 10° increments, using the length of the line segment as a weight, and only the result of the largest value ±10° is left out, but various methods can be adopted for dividing the histogram, weighting the line segment, and determining which line segments to leave, and the method is not limited to this. These values ​​should be determined based on the resolution of the sensor 110. In the above description, the line segments to be left are determined by generating a histogram, but a method of leaving line segments that are close to the orientation of the host vehicle may be used.

[0027] Next, in S203, the lane marking detection unit 200 groups the line segments based on the result of S202. That is, when a lane marking is divided into multiple line segments due to a broken line or rubbing of a white line, the line segments are grouped by determining whether the multiple line segments belong to the same group. For example, the grouping of the line segments is performed by using the position and angle of the lane marking candidate from the vehicle and determining that the line marks within a predetermined threshold value are the lane marking candidate that divides the same lane. The position of the lane marking candidate may be calculated from the angle of the lane marking, which is calculated from the position of the lane marking candidate in the lateral direction of the vehicle and the inclination of the lane marking candidate with respect to the vehicle traveling direction, among the parameters obtained by approximating each detected lane marking candidate with a straight line. In this embodiment, the line segments whose position difference between the lane marking candidates is within 0.5 m when viewed in the lateral direction of the vehicle and whose angle error with respect to the lane marking is within 10° are associated with each other.

[0028] Next, in S204, the lane marking detection unit 200 extracts lane marks from the lane marking candidates. Here, a lane marking refers to a line with a certain width. FIG. 5 shows an example of a method for detecting lane marks with a certain width. In FIG. 5, a sampling interval is set to a certain width in the traveling direction of the vehicle, and the set sampling line is scanned to detect a set of points (edges of lane marks) where the brightness changes from dark to light and from light to dark. Among the sets of detected points, those with a certain width are extracted as lane marks. In this case, the certain width is set to a width of 0.15 to 0.45 m, which is defined by the Road Structure Act as a road marking. However, this width may be set wider or narrower in consideration of the target to be detected.

[0029] In addition, when the lane marking types include Bott's dots, cat's eyes, and other markings that are short in the direction of vehicle travel, the lane markings may not be detected if the sampling interval described above is long. For this reason, if Bott's dots or cat's eyes are detected by a camera or LiDAR, the method of this embodiment can also be applied to Bott's dots and cat's eyes.

[0030] Next, in S205, the lane line detection unit 200 calculates the lane line position to be used in subsequent processing based on the lane line position extracted in S204. The lane line position is determined to be the intersection point closer to the vehicle among the intersection points of the sampling lines and the lane line set at regular intervals. In this way, the detected lane line is represented as a sequence of points. In this example, the lane line position is represented as a sequence of points sampled at regular intervals, but it may also be handled as a line.

[0031] Next, the lane marking type candidate estimation unit 210 will be described with reference to Fig. 6. First, in S211, the lane marking type candidate estimation unit 210 sets a sampling interval that is a fixed interval from the vehicle, and extracts the brightness of the corresponding lane marking on the sampling line (S211). In this example, the sampling interval is assumed to be 1.5 m, but other intervals may be used.

[0032] Next, in S212, the lane marking type candidate estimation unit 210 compares the graph shape in which the luminance of the lane marking on the road surface is plotted for each sampling interval extracted in S211 with a model of the lane marking shape in which the luminance is modeled in advance by a known method such as template matching to obtain a degree of agreement. The lane marking shape model is a model in which the luminance is plotted on the vertical axis and the lateral position from the vehicle is plotted on the horizontal axis as shown in FIG. 7. For example, in the case of a single white line as shown in FIG. 7(a), a lane marking shape model appears every cycle. In the case of a zebra pattern as shown in FIG. 7(b), a lane marking shape model in which the position and thickness of the peak change appears. In the case of a double line consisting of an orange line and a single white line as shown in FIG. 7(c), a lane marking shape model in which the peak on the orange line side is slightly lower appears every cycle. In the case of a lane marking consisting of a white dashed line and a white single lane marking as shown in FIG. 7(d), a lane marking shape model with a different number of peaks appears at a constant cycle. In this way, the lane marking to be detected is modeled in advance for each lane marking type. The graph shape of the brightness detected for each sampled line is matched with the lane line shape model by selecting the one that matches the most as a result of template matching. Although there is a misalignment between the graph shape of the detected brightness and the lane line shape model in the horizontal position, this misalignment can be eliminated by shifting the lane line by the vehicle's horizontal position. The final calculation for each lane line type is determined based on the extent to which each lane line shape graph appears within a specified area, for example, within a range of 50 m ahead of the vehicle. The threshold value for the percentage of each lane line shape model that appears for each lane line type differs for each sampling interval, and is therefore calculated in advance at the set sampling interval. In this example, the detection range was set to 50 m, lane line types were found within a range of 10 m, and the likelihood of the corresponding lane line type was calculated by adding 0.2 to the likelihood of each lane line type. In this example, the likelihood was calculated in this manner, but the likelihood of the lane line type may be calculated by Bayesian estimation or the like based on the tracked lane line information.

[0033] Next, in S213, the lane mark type candidate estimation unit 210 registers the likelihood of the lane mark type for each lane mark estimated in S212 in a list as shown in Fig. 8. As shown in Fig. 8, this list registers the type of sensor that detected the lane mark, the lane mark ID, the likelihood of each lane mark type, and the coordinates of the sequence of points representing the lane mark.

[0034] Next, the lane marking type integrating unit 220 will be described with reference to FIG. 9. First, in S221, the lane marking type integrating unit 220 converts the coordinates of data acquired by the multiple sensors 110 to integrate the coordinate system. For example, a coordinate system centered on the vehicle is defined as having the center of the tire at the rear of the vehicle as the origin, with the front as +x, the left as +y, the vertical upward direction as +z, and the rotation around each axis counterclockwise as +. At this time, the data acquired from each sensor 110 has an offset from this coordinate system by the mounting position and attitude of the sensor 110. Therefore, by converting each data so as to eliminate this offset, all the data can be converted into the same coordinate system. Note that if the acquired data is already expressed in the same coordinate system, the process of S221 is not necessary.

[0035] Next, in S222, the lane marking type integration unit 220 associates the lane marks acquired by each sensor 110. The lane marks within a certain range are associated using the distance between approximation parameters calculated from the point sequence indicating the lane marking shape. The parameters approximating the point sequence are obtained by respectively obtaining approximation parameters for a straight line, a quadratic curve, and a circle, and the distance between the approximation parameters and the point sequence is calculated, and the most suitable approximation parameter is determined to be the one with the smallest distance. The method of calculating the distance between approximation parameters and the method of calculating the distance between the point sequence and the approximation parameters are well known, so their explanation is omitted. Furthermore, the lane marks being tracked at this time are also associated at the same time. The method described above is also used for association with the tracked lane marks. Furthermore, although the association was performed using approximation parameters this time, the association method is not limited to this.

[0036] Next, in S223, the lane marking type integrating unit 220 integrates the likelihoods of the lane marking types associated in S222. The likelihoods are integrated using the following formula.

[0037]

number

[0038] In formula (1), Ti indicates that each lane mark type is correct, Sfc indicates the lane mark type determination result of the front camera, Sl indicates the lane mark type determination result of the LiDAR, P(Sfc|Ti) indicates the likelihood that the result recognized by the front camera is each lane mark type, P(Sl|Ti) indicates the likelihood that the result recognized by the LiDAR is each lane mark type, P(Ti) is the prior probability, and P(Ti|Sfc,Sl) is the posterior probability. Here, the prior probability refers to each likelihood registered for the corresponding tracked lane mark. In addition, although the likelihood integration method using Bayesian estimation has been shown here, it is also possible to take the average of the likelihoods of the lane mark types estimated by each sensor 110, or to take a weighted average by adding weights for each sensor type.

[0039] Next, in S224, the lane marking type integration unit 220 determines the type of lane marking based on the likelihood integrated in S223. This time, the lane marking type was assigned to the lane marking with a likelihood of 0.4 or more. Here, if the likelihoods of multiple lane markings are high, multiple lane marking types are assigned. Also, although the likelihood threshold was set to 0.4 this time, this value may be determined arbitrarily.

[0040] Next, in S225, the lane marking type integrating unit 220 registers the lane marking information in a list (FIG. 10) in memory. The lane marking ID and the likelihood of each lane marking type are registered in the list.

[0041] Next, the lane line type branch point estimation unit 230 will be described with reference to FIG. 11. First, in S231, the lane line type branch point estimation unit 230 sets a lane line branch point candidate. The branch point candidate for the lane line type is set by calculating a ratio based on the lane line type set in FIG. 10 and the likelihood of each type, and setting a branch point candidate from the ratio. An example of setting a branch point candidate is shown in FIG. 12(a). This example shows the lane line with lane line ID 3 in FIG. 10. From the list in FIG. 10, the types of the corresponding lane line are assigned to single lane line and zebra zone. The likelihood of the single lane line is 0.4, and the likelihood of the zebra zone is 0.4. From this, a candidate point is determined by two methods. One is the ratio of the likelihood between types. Since the likelihood ratio is 1:1, a candidate point is set at the center of the detection distance R shown in FIG. 12(a). The other is the ratio with respect to the detection area. In this case, the likelihood of both lane marking types is 0.4, so they are set at a location that is detection distance R × 0.4 from the host vehicle side and detection distance R × 0.4 from the limit side of the detection distance (the side farther from the host vehicle). When set using this method, three candidate points are set, as shown at P in Figure 12(a). Note that this process is only performed when multiple lane marking types are assigned to one lane marking. Also, the method of setting the candidate points is not limited to this method.

[0042] Next, in S232, the lane marking type branch point estimation unit 230 estimates the lane marking branch point from the branch point candidates set in S231. The lane marking branch point is estimated by dividing the lane marking at the position of the branch point candidate set, and then calculating the likelihood of each lane marking type for each divided area (S212). For example, when dividing at the center point of the point P shown in FIG. 12(a), the lane marking is divided into areas R1 and R2 as shown in FIG. 12(b). The process of S212 is executed for each divided area. In this example, the point P at which the likelihood of the single lane marking and the zebra zone are the highest in each corresponding area is obtained. In the illustrated example, the center point P at R1 where the likelihood of the zebra zone and the single lane marking is the highest at R2 is the change point. In this example, the point at which the likelihood of the lane marking type for each area is high is set as the change point, but the lane marking branch point may be obtained by using a lane marking model of the branch point or template matching with the image. Note that this process is performed only when multiple lane marking types are assigned to one lane marking.

[0043] If there is a lane marking on which a branch point has been detected in the processing up to this point (yes in S233), the processing proceeds to S234; if there is no branch point (no in S233), the processing of the lane marking type branch point estimation unit 230 ends.

[0044] In S234, the lane marking type branch point estimation unit 230 divides the area based on the detected branch point as shown in Fig. 12(b) . The area is divided into a side near the vehicle and a side far from the vehicle based on the branch point.

[0045] In S235, the lane marking type and branch point estimation unit 230 re-registers the lane marking information for each divided area in the list. In S235, not only the lane marking type but also the point sequence representing the lane markings belonging to each area is divided. In addition, branch point information (position and which lane markings were divided into which lane markings) is also registered.

[0046] Next, the lane line integration necessity determination unit 240 will be described with reference to Fig. 13. The lane line integration necessity determination unit 240 determines whether or not to recombine the divided lane line information. If a lane line type branch point was not detected in the previous processing, the processing of the lane line integration necessity determination unit 240 is skipped. First, in S241, the lane line integration necessity determination unit 240 reads branch point information.

[0047] Next, in S242, the lane marking integration necessity determination unit 240 calculates the distance between the point sequence positions belonging to each area using the branch point information read in S241. This is intended to divide lane markings that are far apart, such as when point sequences are obtained before and after an intersection. In this case, it was decided to divide when the distance between the point sequences is 3m or more, based on the dashed line threshold. In other cases, the process proceeds to S243.

[0048] Next, in S243, the lane marking integration necessity determination unit 240 calculates approximation parameters using the point sequence for each divided area. The approximation parameters have already been explained, so explanation is omitted. Here, the angle between the point sequence is calculated using the calculated approximation parameters. Here, if the angle is greater than a predetermined threshold, it is considered that the situation is one in which the number of lanes is increasing, such as in an area where lanes are increasing, so it is decided to divide the area.

[0049] Next, in S244, the lane marking integration necessity determination unit 240 registers information on whether the lane markings near the branch point are to be integrated or divided. The registered information is used in the subsequent processes.

[0050] Next, the lane line reliability estimation unit 250 will be described with reference to Fig. 14. The lane line reliability estimation unit 250 assigns reliability to the lane line information obtained up to this point. First, in S251, the lane line reliability estimation unit 250 converts the coordinates of the lane line information and integrates them into the same coordinate system. The method of coordinate conversion has already been described, so description will be omitted.

[0051] Next, in S252, the lane line reliability estimation unit 250 associates the lane lines with each other using the lane lines that have been subjected to coordinate transformation. The lane lines are associated with each other in units of divided lane lines. Similarly, the lane lines are associated with lane lines that have been tracked in the past. The method of associating the lane lines has already been described, and therefore will not be described here.

[0052] Next, in S253, the lane line reliability estimation unit 250 estimates the reliability of each lane line. The reliability is estimated based on whether or not the same lane line can be detected by each sensor 110 in the overlapping area of ​​the observation range of the sensors 110. For example, when the camera and LiDAR have an overlapping area in front of the vehicle, the reliability of the lane line detected by only one sensor 110 is set to 0.5, and the reliability of the lane line detected by both sensors 110 is set to 1.0. An example of setting the reliability based on the number of sensors 110 that can detect the lane line has been described. The probability of the existence of the lane line may be calculated by Bayesian estimation from the number of times tracking was performed or whether tracking was possible, and these may be used as the reliability.

[0053] Next, the lane line information integrating unit 260 will be described with reference to Fig. 15. The lane line information integrating unit 260 integrates detected lane line information. First, in S261, the lane line information integrating unit 260 converts the coordinates of the lane line information. The coordinate conversion process has already been described, so a description thereof will be omitted.

[0054] Next, in S262, the lane line information integration unit 260 matches the lane lines. Here, matching using approximation parameters and matching using lane line types are performed. Matching using approximation parameters has already been described, so a description is omitted. Matching using lane line types involves two processes. One is a validity determination using lane line types. For example, in the case of a road with two lanes on each side and a roadway width a of 6 m or more as shown in FIG. 16, the center line is a solid white line. In this way, it is possible to estimate road conditions from lane line information. For this reason, the road width is estimated using the lane line type, using the estimated lane line position with high reliability. At this time, if a lane line with low reliability, for example a lane line with a broken line type is detected near the center line in the situation of FIG. 16, it is excluded from the matching and tracking targets. The other is matching for each lane line area. The detected lane line information is managed for each lane line type. This is because the sequence of points representing the detected lane line positions is on the inside D of the lane line T as shown in Figure 17. For this reason, approximation parameters are estimated for each lane line region, and the distance between lane lines of the same lane line type is calculated to match the lane lines.

[0055] Next, in S263, the lane line information integration unit 260 integrates the information of the lane line areas associated in S262. Here, since it is assumed that the lane line information matches, the point sequence parameters representing the positions of the lane lines are integrated. In this way, by managing the point sequences representing the lane lines for each lane line type, the lane lines are tracked stably. Also, here, the lane line areas are managed separately, but the connection relationships for each lane line area are maintained at the lane line type branch point, so this does not pose a problem in tracking the lane lines.

[0056] After the above series of processes are performed, the lane marking information is output to an external application that uses the information.

[0057] <Example 2> In the second embodiment, a scene understanding unit 270 is added to the driving assistance device 100. By adding the scene understanding unit 270, the processes of the lane marking type integrating unit 220 and the lane marking reliability estimating unit 250 are changed.

[0058] In order to accurately recognize lane markings, it is necessary to prioritize the sensor information acquired by the sensor 110 by recognizing the environment around the vehicle and taking into consideration scenes in which the lighting environment changes significantly, such as backlighting or tunnel entrances and exits, and weather conditions, such as rain and fog. In terms of lane marking types, dotted lines tend to appear in scenes with downward curves, and zebra zones and lane marks with complex shapes tend to appear near intersections. The scene understanding unit 270 of the second embodiment is assumed to have a function of recognizing these external factors.

[0059] Fig. 18 is a diagram showing the configuration of a driving assistance device 100 according to a second embodiment of the present invention. As shown in Fig. 18, the driving assistance device 100 has, as its functions, a lane line detection unit 200, a lane line type candidate estimation unit 210, a lane line type integrating unit 220, a lane line type branch point estimation unit 230, a lane line integration necessity determination unit 240, a lane line reliability estimation unit 250, a lane line information integrating unit 260, and a scene understanding unit 270, and outputs the results of calculations to an external system 300.

[0060] First, the scene understanding unit 270 will be described. The scene understanding unit 270 recognizes two types of scenes. One is a scene where the recognition performance of the sensor 110 is significantly degraded. This includes, for example, a scene where the lighting conditions change significantly, such as backlighting or the entrance and exit of a tunnel, a scene where the sensing range is blocked by water or dirt adhering to the windshield or camera lens, and weather factors such as rain, fog, and snow. The other is a scene where a specific lane marking type, such as a curve on a downhill road or a road marking, is likely to appear. For this reason, map information 400 is input to the scene understanding unit 270.

[0061] Next, the scene understanding unit 270 will be described with reference to Fig. 19. First, in S271, the scene understanding unit 270 converts the coordinates of data acquired by the various sensors 110. This time, the coordinate conversion of the data is performed by projecting distance information acquired by the LiDAR onto an image. The coordinates of the data can be converted using the relative position and orientation between the sensors 110. The coordinate conversion using the relative position and orientation between the sensors 110 is well known, so a description thereof will be omitted.

[0062] Next, in S272, the scene understanding unit 270 recognizes a scene in which the recognition performance of the sensor 110 is significantly degraded. The scene in which the recognition performance of the sensor 110 is significantly degraded depends on the sensor 110 mounted on the vehicle, so a scene in which the performance is degraded is set in advance for each configuration of the sensor 110. The set scene may be, for example, a scene in which backlight or a tunnel entrance / exit is determined by detecting a scene in which the luminance in the image changes significantly, or an object attached to the lens or windshield is determined by an object recognized in the same place for a certain period of time. The scene may also be determined by machine learning that has previously learned the target scene.

[0063] Next, in S273, if a scene in which the recognition performance is significantly degraded is recognized in S272, the scene understanding unit 270 estimates the sensor 110 and the sensing area affected in S274. The estimation of the sensor 110 and the sensing area affected is performed in the following procedure. First, in a scene as shown in FIG. 20(a), for example, if a performance degradation factor e (backlight) of the sensor 110 is recognized from an image in S272, the direction of the disturbance factor from the vehicle is estimated based on the position on the image. The direction of the disturbance factor can be easily estimated if the mounting position of the sensor 110 with respect to the vehicle is known. After that, taking into consideration the direction, the range of influence, and the type of sensor that acquired the data, for example, in the situation shown in FIG. 20(b), the sensor 110 and the sensing area affected by the performance degradation factor e are estimated based on the detection result within the range divided by a1 to a2. As a result, it is estimated that the detection results of the front camera and part of the right side of F and the detection results of the right side camera are affected by R, and it is estimated that the detection results of the front camera and part of the left side of F and the detection results of the left side camera are not affected. Here, the reason why the sensor type is taken into consideration is that, for example, even in a scene where sensor 110 is affected by backlight, the results of detecting the lane lines by the camera are affected, but the results of detecting the lane lines by LiDAR are not affected, so only the detection results of the affected sensor 110 are targeted.

[0064] Next, in S274, the scene understanding unit 270 estimates the estimated disturbance, the sensor 110 affected by the disturbance, and information on the lane markings affected by the disturbance, and in S275, registers the estimated information in the list shown in Fig. 8. At this time, the disturbance factor is registered in the list based on the information on the affected sensor 110 and the lane markings.

[0065] Next, in S276, the scene understanding unit 270 refers to the map information 400 to recognize road conditions in which a specific lane line type is likely to appear. In the case of an environment in which there is a bias in the frequency of appearance of lane lines, such as a downhill curve or near a highway junction, the surrounding conditions as shown in FIG. 22(b) and lane line likelihood initial values ​​indicating the probability of appearance of lane line types that are biased in frequency of appearance (i.e., are likely to appear) in that condition are read from a list and set as the initial value of the likelihood of each lane line. The road conditions to be estimated vary depending on the lane line type to be identified, and are not limited. As with disturbance estimation, the road conditions can be recognized using machine learning that has previously learned the environment to be identified, or template matching of road markings.

[0066] Next, if a specific road condition is recognized in S276 (yes in S277), the scene understanding unit 270 registers the road condition in memory in S278.

[0067] Next, referring to FIG. 21, the lane marking type integration unit 220, whose processing content changes due to the addition of the scene understanding unit 270, will be described. In FIG. 21, the processing of S221, S222, S224, and S225 is the same as that in FIG. 9, and therefore the description will be omitted. In S223′, in order to use the influence of the disturbance factor recognized by the scene understanding unit 270 on the sensor 110 for lane marking type integration, the reliability of each recognition result due to the sensor type and disturbance influence, which are set in advance as shown in FIG. 22(a), is read and used. Here, the sensor type and disturbance are described in the list of FIG. 8, and the list of FIG. 22(a) is searched based on the information to set the reliability of the lane marking. At this time, the formula for likelihood integration using reliability is calculated by substituting the weight reflected in P(Sfc│Ti) shown in formula (1) as shown in the following formulas (2) and (3). Here, 0 indicates the lowest reliability and 1 indicates the highest reliability.

[0068]

number

number

[0069] Here, Si is the sensor type, T0 is the probability of a correct answer, T1 is the probability of an incorrect answer, and R is the reliability.

[0070] Next, with reference to FIG. 23(a), the lane line reliability estimation unit 250, whose processing content changes due to the addition of the scene understanding unit 270, will be described. In FIG. 23(a), the processing of S251 and S252 is the same as that in FIG. 14, and therefore the description will be omitted. In S253', similar to S223', in order to utilize the influence of the disturbance factor recognized by the scene understanding unit 270 on the sensor 110 for lane line type integration, the reliability of each recognition result due to the sensor type and disturbance influence shown in FIG. 23(b) that have been set in advance is read to set the reliability of the lane line. Here, the sensor type and disturbance are described in the list of FIG. 8, and the list of FIG. 23(b) is searched based on the information to obtain the reliability of the lane line. The reliability of the lane line is assigned to each area of ​​the lane line type. Note that, when one lane line is detected by two sensors 110, it is preferable to use the detection result of the sensor 110 with the higher reliability. Here, the reliability of the lane line is registered in advance in the list based on the sensor type, disturbance, and lane line type. This is because the orange line is particularly difficult to recognize in situations such as backlighting, twilight, etc. However, since it may not be possible to set the reliability due to memory capacity or other factors, the method of setting the reliability is not limited to this.

[0071] <Example 3> An embodiment in which map information 400 is combined will be described with reference to FIG. 24. The map information 400 may be either a high-precision map in which detailed information such as road width, lane lines, and connections between lanes is described, or a navigation map including intersection information. By using information such as lane number and road width registered in the map information 400, it becomes possible to determine the lane line type, estimate lane line branching points, and integrate lane line information with higher accuracy. The map information 400 is used for the processing of the lane line type candidate estimation unit 210, lane line type branching point estimation unit 230, and lane line information integration unit 260 in FIG. 24. For this reason, the processing of each functional block 210, 230, and 260 in which the processing content is changed will be described.

[0072] Next, the lane line type candidate estimation unit 210 will be described with reference to Fig. 25. The lane line type candidate estimation unit 210 estimates lane line type candidates in consideration of the map information 400. First, in S221', the lane line type candidate estimation unit 210 calculates the position of the host vehicle on the map to be used in order to read the map information 400. For example, the latitude, longitude, and altitude acquired by the GNSS are used to calculate the position of the host vehicle on the map, and information about the surroundings of the host vehicle is read from the map information 400. At this time, since the information acquired from the GNSS may contain momentary errors due to the effects of multipath, etc., a value estimated from several frames of data using a time series filter, etc. may be used.

[0073] Next, in S222', the lane marking type candidate estimation unit 210 reads surrounding information of the vehicle from the map information 400. The surrounding information is limited by the map information 400 used. This time, assuming that a high-precision map is used, the lane marking type, lane marking position, number of lanes, road width, and lane connection information are read. The range for reading the map information 400 is information within a predetermined distance from the vehicle.

[0074] Next, in S223', the lane marking type candidate estimation unit 210 converts the information read in S222' into a coordinate system centered on the vehicle. The coordinate conversion is performed by a known method such as the Hubeny formula using the latitude and longitude acquired from the GNSS information.

[0075] Next, in S224', the lane line type candidate estimation unit 210 associates the map information 400 whose coordinates have been converted in S223' with the lane line information detected by the vehicle. Since the lane line position information obtained from the map information 400 is a sequence of points, this association is performed in the same manner as for matching lane lines with each other. Even if the lane line information obtained from the map information 400 is expressed as a function rather than a sequence of points, the sequence of points information can be obtained by sampling at a specified interval.

[0076] Next, in S225', the lane line type candidate estimation unit 210 determines the lane line type. The lane line type can be determined by assigning the lane line type of the map information 400 that corresponds to the lane line information detected by the vehicle. If the lane line detected by the vehicle cannot be associated with the lane line information acquired from the map information 400, it is determined to be an erroneous detection, and the detected lane line information is deleted. By assuming that the map information 400 is the latest, the lane line information that cannot be associated is deleted. For this reason, if there is an intention to update the map, the lane line type may be estimated using the method of the first embodiment with the lane line type obtained from the map information 400 as the initial value. In addition, when using a map in which lane line types, such as a navigation map, are not registered, the initial value for determining the lane line type is changed using information such as before an intersection or before a curve, and the map information 400 is used.

[0077] The lane marking type branch point estimation unit 230 uses the result of associating the detected lane marking with the lane marking information read from the map information 400 by the lane marking type candidate estimation unit 210 to set a branch point of the lane marking type at the lane marking with which a corresponding lane marking has been detected. If a branch point of the lane marking type is not registered in the map information 400, the branch point is estimated by the method of the first embodiment.

[0078] The lane line information integration unit 260 integrates the lane lines using the number of lanes, road width, and lane line type registered in the map information 400. The lane line information integration process is basically the same as that shown in FIG. 15, and uses the map information 400 in the lane line integration in S263. For example, when the map information 400 shown in FIG. 26(a) is obtained, the observation result shown in FIG. 26(b) is obtained. Here, b in the figure indicates a lane line that is determined to be a broken line due to fading of the solid line, c in the figure indicates a solid line that is erroneously detected as a structure such as a guardrail, and d in the figure indicates the lane line information being tracked. Here, first, the lane number and road width registered in the map information 400 are used to remove any lane lines that are considered to be erroneously detected from the lane line information observed by the vehicle. At this time, the distance a between each lane line is calculated as shown in FIG. 26(a) using the positions of each lane line observed. The absolute value of the difference between the distance calculated from the combination of each lane line and the distance a between each lane line is set as a score, and a matching method such as the Hungarian method is used to find the combination with the smallest score. At this time, the combination to be found is set to the maximum number of lanes shown in map information 400. This process removes false positives such as c in the figure. Using the selected observation values, this process matches the next lane line being tracked with the newly observed lane line. The method of matching has already been explained, so an explanation will be omitted. Finally, the lane line type of the matched lane line is corrected. The correction is performed by comparing the lane line information that differs with map information 400 as an erroneous judgment result and overwriting it with map information 400, as shown in b in the figure.

[0079] The lane marking information estimated by this series of processes is output to the external system 300.

[0080] <Example 4> In the fourth embodiment, an embodiment will be described in which lane line information estimated using the driving support device 100 is registered as map information 400. The map information 400 is registered in a server or a storage device in the driving support device. When registering in the map information 400, the external system 300 in Fig. 24 becomes the map information 400. The driving support device itself has already been described in the first to third embodiments, so a description thereof will be omitted.

[0081] The registration of the lane marking information in the map information 400 is intended to generate a map showing the lane markings and obstacle positions of the surrounding environment used for automatic driving, and to assume a case where the lane marking information may differ from the registered information due to construction or other reasons. The map may be generated or updated when the system recognizes that there is no surrounding map, or may be generated or updated when the user changes to the map generation / update mode. The map information 400 is converted from a coordinate system centered on the vehicle to a coordinate system that can be expressed in absolute values ​​using information such as GNSS, and then recorded. This makes it possible to share maps between vehicles. The information registered in the map includes the lane marking position, lane marking type, lane marking branch point, and time. The registration of the lane marking branch point can be used to estimate the vehicle's longitudinal position, which is difficult to estimate using the map information 400, and to switch processing when entering an intersection.

[0082] As described above, the driving assistance device 100 of an embodiment of the present invention is equipped with a lane line detection unit 200 that detects lane line information from data acquired by sensor 110, a lane line type candidate estimation unit 210 that recognizes the type of lane line from the detected lane line information and calculates the likelihood of each type, a lane line type branching point estimation unit 230 that detects a lane line type branching point, which is a point where the likelihood of the lane line type changes in the direction of travel of the vehicle, and distinguishes the areas before and after the lane line type branching point into a first area and a second area, and a lane line information integration unit 260 that integrates the lane lines in the first area and the lane lines in the second area into lane lines that can be continuously recognized, thereby allowing the lane lines to be tracked to be continuously recognized and the lane lines to be tracked with high accuracy.

[0083] The system also includes a lane line integration necessity determination unit 240 that determines whether the lane lines of the first area and the lane lines of the second area need to be integrated. If the lane line integration necessity determination unit 240 determines that integration is necessary, the lane line information integration unit 260 integrates the lane lines of the first area and the lane lines of the second area into a lane line that can be continuously recognized, so that the lane lines can be integrated even when the road structure is complex and there are many diagonal white lines.

[0084] The vehicle also includes a scene understanding unit 270 that estimates the situation around the vehicle from data acquired by the sensor 110, and a lane line reliability estimation unit 250 that estimates the reliability of the lane line type determined for each sensor 110 based on the output of the scene understanding unit 270 and determines the reliability of the lane line for each sensor 110.The lane line type candidate estimation unit 210 refers to the estimated reliability and uses at least one of the luminance and reflection intensity acquired by the sensor 110 to determine lane line information of the lane line type candidate for each sensor 110 based on the edges of the lane line, so that the lane lines can be correctly tracked even if the information acquired by the sensor 110 is affected by external disturbances.

[0085] In addition, the lane line type candidate estimation unit 210 calculates the likelihood of each lane line type by comparing the brightness information of the lane lines sampled at regular intervals in the direction of travel of the vehicle with a lane line shape model that models the brightness information, so that lane lines can be tracked using the same logic regardless of the type of sensor 110 (camera, LiDAR, etc.).

[0086] In addition, the lane line information integration unit 260 uses the lane line type to match the lane lines and integrate the lane line information, so that lane lines can be tracked with higher accuracy than simply looking at their positions, and erroneous detection of lane lines due to incorrect matching can be reduced.

[0087] In addition, the lane marking type branch point estimation unit 230 divides the lane markings into areas before and after the lane marking type branch point and manages the lane marking information for each lane marking type, so that the lane marking type before and after the branch point can be accurately determined.

[0088] In addition, when multiple lane line types are assigned, the lane line type branching point estimation unit 230 detects lane line branching points based on the likelihood ratio of the lane line types, thereby eliminating the need for complex processing and enabling the lane line type to be determined quickly and with low load.

[0089] In addition, the scene understanding unit 270 sets the reliability of the lane markings based on the disturbance factors, the sensor type, and the lane marking type. Since the appearance of the lane markings changes depending on the disturbances, the scene understanding unit 270 can set weights in detail according to the disturbance factors, thereby enabling the lane markings to be correctly recognized regardless of the influence of the disturbances.

[0090] In addition, the lane marking information integration unit 260 integrates lane marking types whose likelihoods have been calculated for each sensor 110 using the likelihood of the lane marking types that are likely to appear in the conditions around the vehicle estimated by the scene understanding unit 270, thereby enabling the lane marking type to be recognized quickly and accurately.

[0091] In addition, the lane line type candidate estimation unit 210 recognizes the type of lane line from the detected lane line information and map information 400, and the lane line type branching point estimation unit 230 uses the map information 400 to detect lane line type branching points, which are points where the likelihood of the lane line type changes in the direction of travel of the vehicle, so that the lane line type and changing points can be recognized quickly and accurately.

[0092] In addition, the lane marking information integration unit 260 integrates the lane marks of the first area with the lane marks of the second area using the number of lanes and road width obtained from the map information 400, thereby preventing the misidentification of structures such as guardrails as lane marks.

[0093] In addition, the lane line information integration unit 260 integrates lane lines of a type whose likelihood has been calculated for each sensor 110 using the likelihood of the lane line type that is likely to appear at the location estimated from the map information 400, thereby enabling the lane line type to be recognized quickly and accurately.

[0094] Furthermore, the lane line information integrating unit 260 registers information on the integrated lane line positions, types, and branch points in the map information, so that the latest information can be reflected on the map and the map information can be updated.

[0095] The present invention is not limited to the above-described embodiments, and includes various modified examples and equivalent configurations within the spirit of the appended claims. For example, the above-described embodiments have been described in detail to clearly explain the present invention, and the present invention is not necessarily limited to those including all of the configurations described. Furthermore, a part of the configuration of one embodiment may be replaced with the configuration of another embodiment. Furthermore, the configuration of another embodiment may be added to the configuration of one embodiment. Furthermore, a part of the configuration of each embodiment may be added, deleted, or replaced with another configuration.

[0096] In addition, each of the above-mentioned configurations, functions, processing units, processing means, etc. may be realized in hardware, for example by designing some or all of them as an integrated circuit, or may be realized in software by a processor interpreting and executing a program that realizes each function.

[0097] Information such as programs, tables, and files that realize each function can be stored in a storage device such as a memory, a hard disk, or an SSD (Solid State Drive), or in a recording medium such as an IC card, an SD card, or a DVD.

[0098] In addition, the control lines and information lines shown are those considered necessary for the explanation, and do not necessarily show all the control lines and information lines necessary for implementation. In reality, it can be considered that almost all components are connected to each other. [Explanation of symbols]

[0099] 100 Driving support device 110 External Recognition Sensor 120 Vehicle Information 130 GNSS receiver 200 Lane line detection unit 210 Lane type candidate estimation section 220 Lotteries 230 Lane line type branch point estimation section 240 Determination of necessity of lane line integration 250 Lane line reliability estimation unit 260 Lane Line Information Integration Section 270 Scene Understanding Department 300 External Systems 400 Map Information

Claims

1. A driving assistance device that recognizes marking lines around a vehicle using at least one sensor, A lane marking detection unit that detects lane marking information from the data acquired by the sensor; a lane marking type candidate estimation unit that estimates a lane marking type from the detected lane marking information and estimates a likelihood of each lane marking type; a lane marking type branch point estimation unit that detects a lane marking type branch point in a traveling direction of the vehicle based on the likelihood of the lane marking type and divides an area before and after the lane marking type branch point into a first area and a second area; A driving assistance device comprising a lane line information integration unit that integrates the corresponding lane lines of the first area and the second area into a continuously recognizable lane line based on the determination result of whether or not the lane lines of the first area and the second area need to be integrated.

2. The driving assistance device according to claim 1, a demarcation line integration necessity determination unit that calculates a distance between demarcation lines for each of the demarcation line regions using approximate parameters calculated from the shapes of each of the demarcation lines, and determines that the demarcation lines of the first region and the demarcation lines of the second region need to be integrated when the approximate parameters calculated from the shapes of each of the demarcation lines of the first region and the demarcation lines of the second region are within a predetermined range; The lane marking information integration unit is a driving assistance device that, when the lane marking integration necessity determination unit determines that integration is necessary, integrates the lane markings of the first area and the lane markings of the second area into a continuously recognizable lane marking.

3. The driving assistance device according to claim 1, a scene understanding unit that estimates a situation around the vehicle from the data acquired by the sensor; a lane line reliability estimation unit that estimates a reliability of the lane line type determined for each of the sensors based on an output of the scene understanding unit, and determines a reliability of the lane line for each of the sensors; The lane line type candidate estimation unit is a driving assistance device that determines lane line information of the lane line type candidate for each sensor based on the edges of the lane line, using at least one of the brightness and reflection intensity acquired by the sensor and referring to the estimated reliability.

4. The driving assistance device according to claim 1, The lane marking type candidate estimation unit is a driving assistance device that calculates the likelihood of each lane marking type by comparing brightness information of lane marks sampled at regular intervals in the direction of vehicle travel with a lane marking shape model that models the brightness information.

5. The driving assistance device according to claim 1, The lane marking information integration unit is a driving assistance device that associates the lane marks using lane marking types and integrates lane marking information.

6. The driving assistance device according to claim 1, The lane marking type branch point estimation unit is a driving assistance device that divides lane marks into areas before and after the lane marking type branch point and manages lane marking information for each lane marking type.

7. The driving assistance device according to claim 1, The lane marking type branch point estimation unit is a driving assistance device that detects a lane marking type branch point based on a likelihood ratio of the lane marking types when a plurality of lane marking types are assigned.

8. The driving assistance device according to claim 3, The scene understanding unit is a driving assistance device that sets the reliability of a lane marking based on a disturbance factor, a sensor type, and a lane marking type.

9. The driving assistance device according to claim 3, The lane marking information integrating unit is a driving assistance device that integrates lane markings of a type for which the likelihood is calculated for each sensor, using the reliability of each sensor based on the situation around the vehicle estimated by the scene understanding unit.

10. The driving assistance device according to claim 3, The lane marking information integration unit is a driving assistance device that integrates lane markings of a type whose likelihood is calculated for each sensor using the likelihood of the lane marking type that is likely to appear in the situation around the vehicle estimated by the scene understanding unit.

11. The driving assistance device according to claim 1, the lane marking type candidate estimation unit recognizes a type of lane marking from the detected lane marking information and map information; The lane marking type branch point estimation unit is a driving assistance device that uses the map information to detect a lane marking type branch point, which is a point at which the likelihood of the lane marking type changes in the driving direction of the vehicle.

12. The driving assistance device according to claim 1, The lane marking information integration unit is a driving assistance device that integrates the lane markings using the number of lanes and road width obtained from map information.

13. The driving assistance device according to claim 1, The lane marking information integrating unit is a driving assistance device that integrates lane markings of a type whose likelihood is calculated for each sensor, using the likelihood of the lane marking type that is likely to appear at a position estimated from map information.

14. The driving assistance device according to claim 1, The lane marking information integration unit is a driving assistance device that registers information on the integrated lane marking positions, types, and branch points in map information.

15. A driving assistance method executed by a driving assistance device that recognizes marking lines around a vehicle using at least one sensor, comprising: The driving assistance device includes a calculation device that executes a predetermined process and a storage device that is connected to the calculation device, The driving assistance method includes: a lane marking detection step in which the arithmetic device detects lane marking information from the data acquired by the sensor; a lane line type candidate estimating step in which the arithmetic device estimates a lane line type from the detected lane line information and estimates a likelihood for each lane line type; a lane line type branching point estimation step in which the calculation device detects a lane line type branching point based on the likelihood of the lane line type in the traveling direction of the vehicle, and divides an area before and after the lane line type branching point into a first area and a second area; A driving assistance method comprising a lane line information integration procedure in which the calculation device integrates the corresponding lane lines of the first area and the lane lines of the second area into a continuously recognizable lane line based on the result of a determination of whether or not the lane lines of the first area and the lane lines of the second area need to be integrated.

Images