Road lane line recognition device and road lane line recognition program

Abstract

To realize stable road lane line recognition operation even under adverse conditions such as nighttime and bad weather.SOLUTION: A road lane line recognition device (5) for recognizing road lane lines from a captured image includes: a plurality of line type candidate determination units (7) for performing line type candidate determination, which is line type determination in lane line candidates, based on line information, which is information on shape characteristics in the lane line candidates to be candidates for the road lane lines, obtained from the captured image; and a line type determination selection unit (8) for performing line type determination as to which line type of road lane line the lane line candidate corresponds to, by selecting one of the line type candidate determination results by the plurality of line type candidate determination units or each of the plurality of line type candidate determination units. The plurality of line type candidate determination units have different determination algorithms.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to a road-dividing line recognition device and a road-dividing line recognition program for recognizing road-dividing lines from a captured image. [Background technology]

[0002] Various devices are known that recognize road dividing lines such as white lines on a road on which a vehicle is traveling for driving support and the like. For example, Patent Document 1 discloses a technology that distinguishes between white lines and noise such as erased line marks based on the contrast ratio between white line candidates, thereby suppressing erroneous recognition. An erased line mark refers to a white line that has been erased but the mark remains. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2011-186722 A Summary of the Invention [Problem to be solved by the invention]

[0004] In this type of device, it is required to realize a stable road marking line recognition operation even in a situation where road surface recognition conditions are poor, such as at night or in bad weather. The present invention has been made in consideration of the above-mentioned circumstances. [Means for solving the problem]

[0005] The road dividing line recognition device (5) described in claim 1 is configured to recognize a road dividing line from a captured image. This road marking recognition device: a plurality of line type candidate determination units (7) that perform line type candidate determination, which is a determination of the line type of the lane marking candidate, based on line information that is information regarding shape characteristics of the lane marking candidate that is a candidate for a road lane marking obtained from the captured image; a line type determination selection unit (8) that performs line type determination as to which line type of the road-dividing line candidate corresponds to the lane-dividing line candidate by selecting one of the line type candidate determination units or one of the line type candidate determination results by each of the line type candidate determination units; Equipped with The line type candidate determining units have different determination algorithms. The road lane line recognition program according to claim 18 is a computer program executed by the road lane line recognition device, The process executed by the road division line recognition device is as follows: a plurality of line type candidate determination processes each having a different determination algorithm, which perform line type candidate determination for determining a line type of a lane marking candidate based on line information, which is information regarding shape characteristics of a lane marking candidate that is a candidate for a road lane marking, obtained from the captured image; a line type determination selection process for selecting one of the line type candidate determination results obtained by the plurality of line type candidate determination processes, thereby executing a line type determination as to which line type of the road-dividing line the lane marking candidate corresponds to; Includes.

[0006] In addition, in each section of the application documents, each element may be given a reference number in parentheses. However, such a reference number merely indicates an example of the correspondence between the element and the specific means described in the embodiment described later. Therefore, the present invention is not limited in any way by the description of the reference numbers above. [Brief description of the drawings]

[0007] [Figure 1A] 1 is a side view of a vehicle equipped with an in-vehicle system including an image recognition device according to an embodiment of the present invention. [Figure 1B] 1 is a plan view of a vehicle equipped with an in-vehicle system including an image recognition device according to an embodiment of the present invention. [Diagram 2] FIG. 2 is a block diagram showing a schematic configuration of the in-vehicle system shown in FIG. [Diagram 3]3 is a block diagram showing a schematic functional configuration realized by the image recognition device shown in FIG. 2. [Figure 4] 4 is a block diagram showing a schematic functional configuration realized in a first embodiment of a selection calculation unit shown in FIG. 3. FIG. [Diagram 5] 5 is a table showing an example of setting or calculation of weights by a weight calculation unit shown in FIG. 4. [Figure 6] 5 is a table showing an example of setting or calculation of weights by a weight calculation unit shown in FIG. 4. [Figure 7] 5 is a table showing an example of effects achieved by the first embodiment of the selection calculation unit shown in FIG. 4. [Figure 8] 5 is a flowchart showing an example of the operation of a line type determination and selection unit according to the first embodiment shown in FIGS. 3 and 4. [Figure 9] 5 is a flowchart showing an example of the operation of a line type determination and selection unit according to the first embodiment shown in FIGS. 3 and 4. [Figure 10] 5 is a flowchart showing an example of the operation of a line type determination and selection unit according to the first embodiment shown in FIGS. 3 and 4. [Figure 11] 5 is a flowchart showing an example of the operation of a line type determination and selection unit according to the first embodiment shown in FIGS. 3 and 4. [Figure 12] 4 is a block diagram showing a schematic functional configuration realized in a second embodiment of the selection calculation unit shown in FIG. 3. FIG. [Figure 13] 13 is a flowchart showing an example of the operation of a line type determination and selection unit according to the second embodiment shown in FIGS. 3 and 12. [Figure 14] 4 is a block diagram showing a schematic functional configuration realized in a third embodiment of the selection calculation unit shown in FIG. 3. FIG. [Figure 15] 15 is a flowchart showing an example of the operation of a line type determination and selection unit according to the third embodiment shown in FIGS. 3 and 14. [Figure 16] 15 is a flowchart showing an example of the operation of a line type determination and selection unit according to the third embodiment shown in FIGS. 3 and 14. [Figure 17]7 is a block diagram showing a schematic functional configuration realized in a fourth embodiment of the selection calculation unit shown in FIG. 3. FIG. [Figure 18] 18 is a flowchart showing an example of the operation of a line type determination and selection unit according to the fourth embodiment shown in FIGS. 3 and 17. [Figure 19] 4 is a block diagram showing a schematic functional configuration realized in an embodiment of a first additional candidate determination unit shown in FIG. 3. FIG. [Figure 20] 20 is a conceptual diagram showing an outline of the operation of a first additional candidate determination unit shown in FIG. 19. FIG. [Figure 21] 20 is a diagram equivalent to a time chart showing an outline of the operation of the first additional candidate determination unit shown in FIG. 19. [Figure 22] 20 is a flowchart showing an example of an operation of a first additional candidate determination unit shown in FIG. 19 . [Diagram 23] 20 is a flowchart showing another operation example of the first additional candidate determination unit shown in FIG. 19 . [Figure 24] 20 is a flowchart showing still another operation example of the first additional candidate determination unit shown in FIG. 19 . [Diagram 25] 20 is a conceptual diagram showing another example of the outline of the operation of the first additional candidate determination unit shown in FIG. 19. FIG. [Figure 26] 26 is a flowchart showing an example of an operation of a first additional candidate determination unit corresponding to the operation outline shown in FIG. 25. [Figure 27] 4 is a block diagram showing a schematic functional configuration realized in another embodiment of the first additional candidate determination unit shown in FIG. 3. FIG. [Figure 28] 28 is a flowchart showing an example of an operation of a first additional candidate determination unit shown in FIG. 27. [Figure 29] 28 is a flowchart showing another operation example of the first additional candidate determination unit shown in FIG. 27. [Diagram 30] 4 is a block diagram showing a schematic functional configuration realized in an embodiment of a second additional candidate determination unit shown in FIG. 3. FIG. [Diagram 31] 31 is a conceptual diagram showing an outline of the operation of a second additional candidate determination unit shown in FIG. 30. FIG. [Diagram 32] 31 is a flowchart showing an outline of road shape estimation using a result of line type determination by a second additional candidate determination unit shown in FIG. 30. [Diagram 33] 31 is a flowchart showing an example of an operation of a second additional candidate determination unit shown in FIG. 30. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] (Embodiment) Hereinafter, an embodiment of the present invention will be described with reference to the drawings. Note that various modified examples applicable to one embodiment may be hindered from understanding the embodiment if they are inserted in the middle of a series of explanations related to the embodiment. For this reason, the modified examples are described together after the series of explanations related to the embodiment.

[0009] (In-vehicle system configuration) Referring to FIG. 1A and FIG. 1B, an in-vehicle system 1 is mounted on a vehicle C, which is an automobile traveling on a road. The in-vehicle system 1 is configured to be able to execute various operations, such as driving control operations of the vehicle C and warning operations for occupants, using image recognition results based on images captured by a camera 2. That is, the in-vehicle system 1 has a configuration as a so-called driving automation system (i.e., an automatic driving system or a driving assistance system). "Autonomous driving" refers to a driving automation level in which a driving automation system is responsible for, i.e., executes, all dynamic driving tasks, which corresponds to levels 3 to 5 in the standard "SAE J3016" published by SAE International. SAE is an abbreviation for Society of Automotive Engineers. "Dynamic driving tasks" are all operational and tactical functions that need to be performed in real time when operating a vehicle in road traffic, excluding strategic functions. "Strategic functions" include itinerary planning, route selection, etc., and specifically include deciding or selecting "whether to go or not, when, where and how to go." "Driving assistance" refers to a driving automation level corresponding to levels 1 to 2 in "SAE J3016," where a driving automation system continuously executes a longitudinal vehicle motion control subtask and / or a lateral vehicle motion control subtask of a dynamic driving task in a specific limited area. The longitudinal vehicle motion control subtask and / or the lateral vehicle motion control subtask include, for example, starting, steering, accelerating, decelerating, braking, stopping, shift range change, etc. Vehicle driving control operations in autonomous driving, vehicle driving control operations in driving assistance, and the associated guidance, notification, or warning operations may be collectively referred to as "driving automation related operations" below.

[0010] The camera 2 is a so-called forward camera, and is fixed on the body or inside the vehicle cabin of the host vehicle C so as to capture an image of the area ahead of the host vehicle C. That is, the camera 2 is provided so as to be able to capture images of road markings such as road dividing lines provided on the road surface R ahead of the host vehicle C, and objects such as obstacles present ahead of the host vehicle C. A road surface photographing range R1, which is an area indicated by diagonal hatching in FIG. 1B, is the photographing range of the road surface R by the camera 2. As shown in FIG. 1B, the edge of the road surface photographing range R1 closest to the host vehicle C, i.e., the "near side," is referred to as a proximal end boundary R2.

[0011] Referring to FIG. 2, the in-vehicle system 1 includes a camera 2, a host vehicle state detection unit 3, a recognition result utilization unit 4, and an image recognition device 5. The host vehicle state detection unit 3 is provided to detect various information indicating the driving state of the host vehicle C (for example, vehicle speed, yaw rate, etc.). That is, the host vehicle state detection unit 3 is a collective name for well-known sensors necessary for vehicle driving control, such as a vehicle speed sensor and a gyro sensor, for the sake of simplicity of illustration and explanation. The recognition result utilization unit 4 is configured to execute various operations, such as a driving control operation of the host vehicle C and an alert operation for the occupants, based on the result of image recognition of the image captured by the camera 2 by the image recognition device 5. That is, the recognition result utilization unit 4 has a configuration capable of realizing driving automation-related operations, such as lane departure warning, lane keeping assistance, lane change assistance, automatic lane change, and automatic overtaking, based on the recognition result of the road dividing line based on the image captured by the camera 2.

[0012] The image recognition device 5 is configured to detect, i.e., recognize, road markings and objects present in front of the vehicle C by performing image recognition based on input information from the camera 2 and the vehicle state detection unit 3. That is, the image recognition device 5 as a road dividing line recognition device in the present invention is configured to recognize road dividing lines from an image captured by the camera 2. In this embodiment, the image recognition device 5 has a configuration as an on-board computer including a processor 5a and a memory 5b. The processor 5a is configured by a CPU, an MPU, or the like. The memory 5b includes at least a ROM or a nonvolatile rewritable memory and a RAM among various non-transient substantial storage media such as a ROM, a RAM, a nonvolatile rewritable memory, and the like. The nonvolatile rewritable memory is a storage device that can rewrite information while the power is on, but holds information in an unrewritable manner while the power is off, and is, for example, a flash ROM, a hard disk, or the like. The image recognition device 5 is configured to perform an image recognition operation by reading and executing a computer program stored in the ROM or the nonvolatile rewritable memory.

[0013] (Image Recognition Device) Fig. 3 shows a part of the functional configuration realized on the in-vehicle microcomputer by the processor 5a reading and executing a program from the memory 5b. As shown in Fig. 3, the image recognition device 5 includes an information acquisition unit 6, a plurality of line type candidate determination units 7, a line type determination and selection unit 8, and a road shape estimation unit 9 as functional configurations realized on the in-vehicle microcomputer.

[0014] The information acquisition unit 6 is provided to acquire output information from the camera 2 and the vehicle state detection unit 3, and to hold the output information in a chronological order. Each of the multiple line type candidate determination units 7 is provided to execute a line type candidate determination, which is a determination of the line type of the lane line candidate, based on line information, which is information on the shape characteristics of the lane line candidate that is a candidate for the road lane line, acquired from the captured image. The "line type" includes, for example, a straight line, a dashed line, a double line consisting of a straight line and a dashed line, a double line consisting of a straight line and a straight line, and the like. The line type determination selection unit 8 is provided to execute a line type determination as to which line type of the road lane line the recognized lane line candidate corresponds to by selecting one of the line type candidate determination results by the multiple line type candidate determination units 7 or each of the multiple line type candidate determination units 7. The road shape estimation unit 9 is provided to estimate the road model shape using a Kalman filter or the like, based on the edge points, which are the line information. The "edge point" refers to a point, i.e., a pixel, where the change in pixel parameters, such as brightness, from the surroundings is greater than or equal to a predetermined value. In addition, the road dividing line to be recognized and its candidate dividing line candidates may be collectively referred to as "boundary lines" hereinafter.

[0015] The image recognition device 5 has a plurality of line type candidate determination units 7 having different determination algorithms. Fig. 3 shows an example in which three line type candidate determination units 7 are provided as the plurality of line type candidate determination units 7: a basic candidate determination unit 70, a first additional candidate determination unit 71 as a first line type candidate determination unit, and a second additional candidate determination unit 72 as a second line type candidate determination unit.

[0016] The basic candidate determination unit 70 is configured to determine the line type candidate by using template matching based on the paint length and gap length, which are line information. The "paint length" is the length of the paint portion, which is a portion painted in white, orange, or the like on the road surface R, in the front-rear direction, i.e., in the traveling direction of the vehicle C, and may also be called the "marker length." The traveling direction of the vehicle C is parallel to the vehicle overall length direction of the vehicle C while traveling straight, but may be a curved direction along the curve while traveling around a curve. The "gap length" is the length of the gap portion in the broken line in the front-rear direction. The gap portion is a portion between multiple paint portions lined up in the front-rear direction in the broken line. That is, the recognition method of template matching based on the detected paint length and gap length by the basic candidate determination unit 70 is already publicly known or well known at the time of filing the present application, so further detailed explanation will be omitted.

[0017] The first additional candidate determination unit 71 is configured to determine a line type candidate based on time-series changes in the position of a specific edge point (e.g., a near edge point) in the paint portion, which is line information, in the front-rear direction. The "near edge point" is an edge point located on the "near side" closest to the vehicle C in the paint portion of the lane line candidate in the captured image or in a predetermined area in the captured image, for example, an edge point at a position corresponding to the near end boundary R2 shown in FIG. 1B. In contrast, an edge point located on the "far side" farthest from the vehicle C in the paint portion of the lane line candidate in the captured image or in a predetermined area in the captured image is referred to as a "far edge point." The first additional candidate determination unit 71 and a line type candidate determination method therefor will be described in detail later.

[0018] The second additional candidate determination unit 72 is configured to determine a line type candidate based on a class determination result by semantic segmentation, which is line information. Semantic segmentation is an algorithm that uses machine learning (e.g., DNN, CNN, etc.) to associate a label or category of a "class" such as "plant," "animal," "person," "road," "sky," "sea," "car," etc., with each pixel in an image. DNN stands for deep neural network. CNN stands for convolutional neural network. Specifically, the second additional candidate determination unit 72 determines a line type candidate based on a class rate, which is a probability that a pixel corresponds to a predetermined class (e.g., a paint part or a gap part). The second additional candidate determination unit 72 and a line type candidate determination method therefor will be described in detail later.

[0019] The line type determination selection unit 8 has a selection calculation unit 81 and a selection result determination unit 82. The selection calculation unit 81 is provided to execute necessary calculation processing for selecting one of the multiple line type candidate determination units 7 or the line type candidate determination results by each of the multiple line type candidate determination units 7. The selection result determination unit 82 is provided to select one line type candidate determination unit 7 or the line type candidate determination result by this based on the calculation result by the selection calculation unit 81, thereby determining the final line type recognition result.

[0020] (Selection Calculation Unit: First Embodiment) In this embodiment, the line type determination selection unit 8 is configured to select the line type candidate determination unit or the line type candidate determination result according to the establishment status of the scene corresponding to the shooting situation that affects the line type candidate determination. The "shooting situation that affects the line type candidate determination" includes not only the shooting environment such as the natural environment and the presence of other vehicles, but also the situation of the shooting subject such as whether the shooting subject is a compound line or a blurred line. A "blurred line" refers to a road dividing line in which the paint part is blurred, i.e., partially missing. In other words, the "scene" is, for example, "nighttime," "rain," "obstructed by a preceding vehicle," "compound line," "blurred line," etc. "Obstructed by a preceding vehicle" refers to a state in which a part of the road dividing line is obstructed by another vehicle preceding the host vehicle C or its shadow.

[0021] FIG. 4 shows an example of the functional configuration of the selection calculation unit 81 for the final line type determination by the line type determination selection unit 8 in this embodiment. Hereinafter, the functional configuration of the line type determination selection unit 8 in this embodiment will be described with reference to FIG. 3 and FIG. 4. As shown in FIG. 4, the selection calculation unit 81 has a scene determination unit 83 and a weight value calculation unit 84. The scene determination unit 83 is provided to determine the establishment status of a scene. Specifically, the scene determination unit 83 is configured to determine the establishment status of a scene based on an image captured by the camera 2. The weight value calculation unit 84 is provided to set a weight value, which is a weighting value for each of the multiple line type candidate determination units 7, for each of the different scenes. In addition, the weight value calculation unit 84 is provided to calculate a weight value for each of the multiple line type candidate determination units 7 based on the determination result of the establishment status of the scene by the scene determination unit 83. Specifically, the weight value calculation unit 84 is configured to calculate a total value of weight values ​​corresponding to the established scenes for each of the multiple line type candidate determination units 7 when multiple scenes are established. The line type determination selection unit 8, i.e., the selection result determination unit 82 shown in Figure 3, is configured to select the line type candidate determination unit 7 or the line type candidate determination result based on the weighted value corresponding to the scene determined to be established by the scene determination unit 83.

[0022] Below, an overview of the operation of the image recognition device 5 according to this embodiment or the image recognition program (i.e., road dividing line recognition program) which is a computer program executed by the image recognition device 5 will be described together with the effects achieved by the image recognition device 5. Hereinafter, the image recognition device 5 according to this embodiment and the image recognition program executed by the image recognition device 5 may be collectively referred to as "this embodiment."

[0023] FIG. 5 shows a specific example of the accuracy rate of the road dividing line recognition result for each scene in each of the different determination algorithms. In FIG. 5, "basic algorithm" indicates the determination algorithm by the basic candidate determination unit 70, that is, the line type determination algorithm using template matching based on the paint length and the gap length. "additional algorithm 1" indicates the determination algorithm by the first additional candidate determination unit 71, that is, the line type determination algorithm based on the time series change in the position in the forward and backward direction of a specific edge point. "additional algorithm 2" indicates the determination algorithm by the second additional candidate determination unit 72, that is, the line type determination algorithm based on the class determination result by semantic segmentation. A to E in "scene" are the above-mentioned nighttime, preceding vehicle obstruction, etc. The numerical values ​​in FIG. 5 indicate the accuracy rate in percentage.

[0024] Automated driving operations, such as lane-related control such as lane keeping and automatic driving control, tend to be applied not only to expressways and other motorways but also to general roads. In particular, in lane change assistance and automatic lane change, it is required to accurately determine the type of road dividing line as an index for determining whether control is possible (for example, lane change is not possible when the line is a solid yellow or white line). Here, in general roads, particularly in urban areas and mountain roads, the road surface recognition conditions are worse than those of motorways. In addition, in the conventionally known line type determination method using template matching, in adverse conditions such as blurring or preceding vehicle obstruction, or in special patterns that do not fit the template, the paint length and gap length become unclear, and there is a concern that the accuracy of line type determination decreases. For this reason, it is required to realize a stable road dividing line recognition operation even in adverse conditions for road surface recognition, such as nighttime, bad weather, blurring, etc.

[0025] In this regard, as shown in FIG. 5, the determination algorithm with the highest accuracy rate differs for each scene. That is, each determination algorithm has different scenes that are "good" at determining the scene. Specifically, for example, a basic algorithm using template matching based on paint length and gap length is highly robust under good conditions. On the other hand, additional algorithm 1 based on the time series change in the position of a specific edge point in the forward and backward directions has high determination accuracy in scenes B and C, which are poor conditions. In addition, additional algorithm 2 based on the class determination result by semantic segmentation has high determination accuracy in scenes C, D, and E, which are poor conditions. Therefore, the inventor has found a method that can cover scenes that are difficult to determine using algorithms using template matching that are well known in the art.

[0026] That is, in this embodiment, one of the multiple line type candidate determination units 7 or the line type candidate determination result thereof is selected according to the determination result of the scene establishment status. Specifically, in this embodiment, for example, when there is one established scene, one of the multiple line type candidate determination units 7 corresponding to the established scene is selected, and line type candidate determination is performed only by the selected line type candidate determination unit 7. Alternatively, in this embodiment, for example, when there is one established scene, a final line type determination result is selected from the line type candidate determination results of each of the multiple line type candidate determination units 7 according to the established scene. Alternatively, in this embodiment, for example, when there is multiple established scenes, a final line type determination result is selected from the line type candidate determination results of each of the multiple line type candidate determination units 7 according to the established scene. In this way, this embodiment uses multiple line type determination methods depending on the scene. Therefore, according to this embodiment, it is possible to realize a stable road division line recognition operation even under adverse conditions such as nighttime or bad weather.

[0027] FIG. 6 shows the result of setting the weight value based on the accuracy rate shown in FIG. 5. In this specific example, the weight value is set to "0" when the accuracy rate is less than 90%, and is set to the "1st digit" of the two-digit number obtained by rounding down the decimal point of the accuracy rate number when the accuracy rate is 90% or more. For example, when the accuracy rate is "91.7%, the weight value is "1". In this embodiment, the line type determination selection unit 8 selects the line type candidate determination unit 7 or the line type candidate determination result thereof based on the weight value corresponding to the scene determined to be established by the scene determination unit 83. Specifically, the line type determination selection unit 8 selects the line type candidate determination unit 7 or the line type candidate determination result thereof that has the largest weight value corresponding to the established scene. For example, in the case of scene A, the weight value in the basic algorithm is "3", the weight value in the additional algorithm 1 is "0", and the weight value in the additional algorithm 2 is "1". Therefore, the selection result determination unit 82 determines the final line type recognition result by selecting the basic candidate determination unit 70 corresponding to the basic algorithm with the largest weighted value, or the line type candidate determination result obtained by this. In the case of scene C, the weighted values ​​for additional algorithm 1 and additional algorithm 2 are the same and the maximum value. However, even in such a case, a weighted value of "1" or more is premised on an accuracy rate of 90% or more, so there is a high probability that the line type candidate determination results of both will match. Therefore, even if there are multiple determination algorithms with the same and maximum weighted value, any one of them can be selected arbitrarily.

[0028] FIG. 7 is a table of FIG. 5 to which the effects of this embodiment have been added. In FIG. 7, "Current Algorithm" indicates the line type determination algorithm of this embodiment, which uses a plurality of determination algorithms depending on the scene. "Comparison with Conventional Algorithm" indicates the comparison result of the accuracy rate, i.e., the deviation, with the case where only the basic algorithm corresponding to the conventional line type determination algorithm is used. As shown in FIG. 7, according to the line type determination algorithm of this embodiment, by selecting the optimal line type determination method for each scene, a good accuracy rate of 90% or more can be achieved in any scene. In particular, according to this embodiment, it is possible to perform accurate line type determination even under adverse conditions such as scenes D and E, where the accuracy rate was relatively low with the conventional line type determination algorithm.

[0029] When the scene determination unit 83 determines that a plurality of scenes have been established, the weight value calculation unit 84 sets a weight value corresponding to each of the plurality of line type candidate determination units 7 for each established scene. Furthermore, the weight value calculation unit 84 calculates a weight value for each of the plurality of line type candidate determination units 7 based on the result of the scene establishment status determination by the scene determination unit 83. That is, the weight value calculation unit 84 adds up the weight values ​​corresponding to the established scenes for each of the plurality of line type candidate determination units 7. Then, the selection result determination unit 82 selects the line type candidate determination unit 7 corresponding to the maximum sum value, which is the result of the addition, or the line type candidate determination result obtained thereby.

[0030] Specifically, for example, when scene A is established, the weight value calculation unit 84 sets the weight value in the basic algorithm to "3", the weight value in the additional algorithm 1 to "0", and the weight value in the additional algorithm 2 to "1". For example, when scene B is established, the weight value calculation unit 84 sets the weight value in the basic algorithm to "6", the weight value in the additional algorithm 1 to "8", and the weight value in the additional algorithm 2 to "7". Then, when the established scenes are scenes A and B, the weight value calculation unit 85 adds up the weight values ​​of both. Then, the weight value in the basic algorithm becomes "9", the weight value in the additional algorithm 1 becomes "8", and the weight value in the additional algorithm 2 becomes "8". Therefore, the selection result determination unit 82 selects the basic candidate determination unit 70 corresponding to the basic algorithm with the largest added value "9", or the line type candidate determination result obtained thereby, to determine the final line type determination result. That is, when only scene B occurs, the selection result determination unit 82 selects additional algorithm 1 that is strong against scene B, whereas when not only scene B but also scene A occurs, the selection result determination unit 82 selects the basic algorithm by comprehensively taking into account the strengths of multiple algorithms. Thus, according to this embodiment, good line type determination is possible even in a situation where multiple scenes occur.

[0031] 8 to 11 are flowcharts showing an example of an operation of the image recognition device 5 according to this embodiment. In the flowcharts in the figures, "S" is an abbreviation of "step". The same applies to other flowcharts from FIG. 9 onwards. Referring to FIG. 8, in step 810, the processor 5a acquires a frame image. Next, in step 821, the processor 5a executes line type candidate determination by the first line type candidate determination unit 7 (for example, the basic candidate determination unit 70 in the example of FIG. 3). Similarly, in step 822, the processor 5a executes line type candidate determination by the second line type candidate determination unit 7 (for example, the first additional candidate determination unit 71 in the example of FIG. 3). In this way, the processor 5a executes steps 821 to 82n, which are line type candidate determination processes by the 1st to nth line type candidate determination units 7, and then proceeds to step 830 and step 840.

[0032] In step 830, the processor 5a executes a line type determination selection process. That is, the processor 5a executes a line type determination as to which line type of the road-dividing line the lane marking candidate corresponds to, by selecting one of the line type candidate determination results by each of the multiple line type candidate determination units 7. Next, in step 840, the processor 5a determines whether or not the line type determination process is completed, that is, whether or not there is a next frame image for which the line type determination process should be performed. If there is a next frame image (i.e., step 840=NO), the processor 5a proceeds to the process for the next frame image. In this case, the processor 5a returns the process to step 810. On the other hand, if there is not a next frame image (i.e., step 840=YES), the processor 5a temporarily ends the line type determination process.

[0033] Fig. 9 shows details of the process content in step 830. That is, the processor 5a executes the processes of steps 901 to 904 in the line type determination and selection process in step 830 shown in Fig. 8. Specifically, as shown in Fig. 9, first, in step 901, the processor 5a acquires image data captured by the camera 2. Next, in step 902, the processor 5a determines whether a scene exists. Subsequently, in step 903, the processor 5a sets a weight value, which is a weighting value for each line type candidate determination unit 7, according to the determination result of the scene existence in step 902. Then, in step 904, the processor 5a selects a line type candidate determination result based on the weight value corresponding to the scene determined to exist.

[0034] Fig. 10 shows details of the processing content in step 903. That is, the processor 5a executes the processing from step 1001 onwards in the weight setting processing in step 903 shown in Fig. 9. Note that in the following explanation, for the sake of simplicity, the explanation will be based on an example configuration where n=3 in Fig. 8, that is, as shown in Fig. 3, three line type candidate determination units 7, namely, a basic candidate determination unit 70, a first additional candidate determination unit 71 and a second additional candidate determination unit 72, are provided as the multiple line type candidate determination units 7.

[0035] Specifically, as shown in Fig. 10, first, in step 1001, the processor 5a initializes the setting state of the weight value for each line type candidate determination unit 7. Next, in step 1002, the processor 5a acquires the determination result of the scene establishment status in step 902 shown in Fig. 9. Then, in step 1003 and thereafter, the processor 5a sets a weight value for each line type candidate determination unit 7 corresponding to the scene that has been determined to be established.

[0036] First, in step 1003, the processor 5a determines whether or not scene A exists. If scene A exists (i.e., step 1003=YES), the processor 5a executes the process of step 1004 and then advances the process to step 1005. In step 1004, the processor 5a sets the weight values ​​when scene A exists to "3", "0", and "1" for each of the multiple line type candidate determination units 7, i.e., the basic algorithm, the additional algorithm 1, and the additional algorithm 2, respectively. If scene A does not exist (i.e., step 1003=NO), the processor 5a skips the process of step 1004 and advances the process to step 1005.

[0037] In step 1005, the processor 5a determines whether or not scene B has occurred. If the heavy rain scene has occurred (i.e., step 1005=YES), the processor 5a executes the process of step 1006 and then advances the process to step 1007. In step 1006, the processor 5a sets the weight values ​​for the basic algorithm, additional algorithm 1, and additional algorithm 2 when scene B has occurred to "6," "8," and "7," respectively. If scene B has not occurred (i.e., step 1005=NO), the processor 5a skips the process of step 1006 and advances the process to step 1007.

[0038] In step 1007, the processor 5a determines whether or not scene C has occurred. If scene C has occurred (i.e., step 1007=YES), the processor 5a executes the process of step 1008 and then advances the process to step 1009. In step 1008, the processor 5a sets the weight values ​​for the basic algorithm, additional algorithm 1, and additional algorithm 2 when scene C has occurred to "6", "8", and "8", respectively. If scene C has not occurred (i.e., step 1007=NO), the processor 5a skips the process of step 1008 and advances the process to step 1009.

[0039] In step 1009, the processor 5a determines whether or not scene D has occurred. If scene D has occurred (i.e., step 1009=YES), the processor 5a executes the process of step 1010 and then advances the process to step 1011. In step 1010, the processor 5a sets the weight values ​​for the basic algorithm, additional algorithm 1, and additional algorithm 2 when scene D has occurred to "0", "1", and "8", respectively. If scene D has not occurred (i.e., step 1009=NO), the processor 5a skips the process of step 1010 and advances the process to step 1011.

[0040] In step 1011, the processor 5a determines whether or not scene E has occurred. If scene E has occurred (i.e., step 1011=YES), the processor 5a executes the process of step 1012 and then advances the process to step 1013. In step 1012, the processor 5a sets the weight values ​​for the basic algorithm, additional algorithm 1, and additional algorithm 2 when scene E has occurred to "0", "0", and "8", respectively. If scene E has not occurred (i.e., step 1011=NO), the processor 5a skips the process of step 1012 and advances the process to step 1013.

[0041] In step 1013, the processor 5a calculates a final weight value corresponding to each of the multiple line type candidate determination units 7 (i.e., each algorithm). That is, for example, if there is one established scene, the processor 5a sets the weight value set in any one of the executed steps among step 1004, step 1006, step 1008, step 1010, and step 1012 as the final weight value. On the other hand, if there are multiple established scenes, the processor 5a calculates the final weight value by adding up the weight values ​​set in the multiple executed steps among step 1004, step 1006, step 1008, step 1010, and step 1012.

[0042] FIG. 11 summarizes the processing of steps 903 to 904 in FIG. 9. Referring to FIG. 11, first, in step 1101, the processor 5a acquires a weight value for each scene. That is, the processing of step 1101 corresponds to the processing of steps 1001 to 1012 in the flowchart of FIG. 10. Next, in step 1102, the processor 5a calculates a final weight value for the basic algorithm, that is, the basic candidate determination unit 70. The final weight value for the basic algorithm obtained in step 1102 is set as A0. Similarly, in step 1103, the processor 5a calculates a final weight value for the additional algorithm 1, that is, the first additional candidate determination unit 71. The final weight value for the additional algorithm 1 obtained in step 1103 is set as A1. Furthermore, in step 1104, the processor 5a calculates a final weight value for the additional algorithm 2, that is, the second additional candidate determination unit 72. The final weight value for the additional algorithm 2 obtained in step 1104 is set as A2. That is, the processes in steps 1102 to 1104 correspond to the process in step 1013 in the flowchart of Fig. 10. Then, the processor 5a advances the process to step 1105 and onwards in order to select one of the line type candidate determination results by the multiple line type candidate determination units 7 or each of the multiple line type candidate determination units 7.

[0043] In step 1105, the processor 5a compares the value of A0 with the value of A1. If A0≥A1 (i.e., step 1105 = YES), the processor 5a proceeds with the process to step 1106. In step 1106, the processor 5a compares the value of A0 with the value of A2. If A0≥A2 (i.e., step 1106 = YES), the processor 5a proceeds with the process to step 1107. In step 1107, the processor 5a selects the basic algorithm, i.e., the basic candidate determination unit 70, or the line type candidate determination result therefrom, and determines the final line type recognition result. If A0 < A2 (i.e., step 1106 = NO), the processor 5a proceeds with the process to step 1108. In step 1108, the processor 5a selects the additional algorithm 2, i.e., the second additional candidate determination unit 72, or the line type candidate determination result therefrom, and determines the final line type recognition result. If A0 < A1 (i.e., step 1105 = NO), the processor 5a proceeds with the process to step 1109. In step 1109, the processor 5a compares the value of A1 with the value of A2. If A1≥A2 (i.e., step 1109 = YES), the processor 5a proceeds with the process to step 1110. In step 1110, the processor 5a selects the additional algorithm 1, i.e., the first additional candidate determination unit 71, or the line type candidate determination result therefrom, and determines the final line type recognition result. If A1 < A2 (i.e., step 1109 = NO), the processor 5a proceeds with the process to step 1108.

[0044] (Selection operation unit: Second embodiment) Hereinafter, another example of the selection calculation unit 81, that is, a second embodiment, will be described. In the description of this embodiment, differences from the first embodiment will be mainly described. In addition, the same reference numerals are used for parts that are the same or equivalent in the first and second embodiments. Therefore, in the following description of this embodiment, the description of the first embodiment may be appropriately used for components having the same reference numerals as those in the first embodiment, unless there is a technical contradiction or special additional explanation. The same applies to other embodiments described later, and the description of the preceding embodiment may be appropriately used for components having the same reference numerals as those in the preceding embodiment, unless there is a technical contradiction or special additional explanation.

[0045] With reference to Fig. 3, in this embodiment, the line type determination selection unit 8 is configured to execute line type determination based on a result of comparing the determination results when there are multiple line type candidate determination units 7 or their determination results corresponding to the largest weighting value. Fig. 12 shows an example of the functional configuration of a selection calculation unit 81 for final line type determination in this embodiment. That is, the configuration shown in Fig. 12 is a partial modification of the configuration shown in Fig. 4. Fig. 13 is a flowchart showing an example of operation corresponding to this embodiment. The contents of this embodiment will be described below with reference to Figs. 3, 12, and 13.

[0046] In this embodiment, the selection calculation unit 81 has a scene determination unit 83, a weighted value calculation unit 84, and a determination result comparison unit 86. When there are multiple line type candidate determination units 7 or their determination results corresponding to the largest weighted value, the determination result comparison unit 86 compares the determination results with each other and performs a final line type determination based on the comparison result.

[0047] Fig. 13 shows an overview of the operation of the judgment result comparison unit 86. Referring to Fig. 13 below, first, in step 1300, the processor 5a acquires the weighting values ​​set for each of the multiple line type candidate judgment units 7 in the current frame image. Next, in step 1301, the processor 5a judges whether or not the maximum value of the weighting values ​​acquired in step 1300 is equal to or greater than a threshold value. If the maximum value is equal to or greater than the threshold value (i.e., step 1301 = YES), the processor 5a advances the process to step 1302. In step 1302, the processor 5a judges whether or not the maximum value of the weighting values ​​acquired in step 1300 is one, that is, whether or not there is only one line type candidate judgment unit 7 with the maximum weighting value.

[0048] If there is one maximum value (i.e., step 1302=YES), the processor 5a advances the process to step 1303. In step 1303, the processor 5a selects the line type candidate determination unit 7 with the maximum weighted value, or the line type candidate determination result obtained thereby, as in the first embodiment. Next, in step 1304, the processor 5a outputs one line type candidate determination result based on the selection result in step 1303 as the final line type determination result. On the other hand, if there are multiple maximum values ​​(i.e., step 1302=NO), the processor 5a advances the process to step 1305. In step 1305, the processor 5a determines whether or not the line type candidate determination results from the multiple line type candidate determination units 7 with the maximum weighted values ​​are the same. As described above, the line type candidate determination results from the multiple line type candidate determination units 7 with the maximum weighted values ​​usually match. Therefore, if the line type candidate determination results are the same (i.e., step 1305=YES), the processor 5a advances the process to step 1304. On the other hand, if the line type candidate determination results are different (i.e., step 1305=NO), which occurs very rarely, the processor 5a advances the process to step 1306 in order to avoid problems due to erroneous determination of the line type as much as possible. Also, if the processor 5a determines in step 1301 that the maximum value is less than the threshold value (i.e., step 1301=NO), the processor 5a also advances the process to step 1306. In step 1306, the processor 5a determines that the final line type determination result is unknown.

[0049] (Selection Calculation Unit: Third Embodiment) Hereinafter, another example of the selection calculation unit 81 will be described. There is a possibility that momentary erroneous determination may occur in scene determination. For this reason, this embodiment has a configuration in which the reliability of the scene determination result is calculated, and the line type candidate determination unit 7 or the line type candidate determination result is selected based on the reliability. That is, as shown in FIG. 14, in this embodiment, the selection calculation unit 81 has a scene determination unit 83, a weighted value calculation unit 84, and a scene reliability calculation unit 87. The scene reliability calculation unit 87 is provided to calculate a scene determination reliability, which is the reliability of the scene determination result by the scene determination unit 83. The scene determination reliability can be calculated, for example, by statistically processing the scene determination result by the scene determination unit 83 over time. That is, for example, if the number of frames or the number of consecutive frames in which the same scene is determined in the most recent chronologically consecutive N frames is p, the scene determination reliability can be expressed as p / N, where the maximum value is 1 and the minimum value is 0. Specifically, if heavy rain is judged consecutively in 3 frames out of the most recent 10 frames, p = 3 and N = 10, and the scene judgment reliability (rain) = 0.3. In this way, the scene judgment reliability can be calculated based on the scene judgment result by the scene judgment unit 83 or its continuation state.

[0050] 3 is capable of selecting the line type candidate determination unit 7 or its line type candidate determination result having the largest weighting value in the scene having the highest scene determination reliability calculated by the selection calculation unit 81, i.e., the scene reliability calculation unit 87. Alternatively, the selection result determination unit 82 is capable of selecting the line type candidate determination unit 7 or its line type candidate determination result based on the weighting value corrected by the scene determination reliability calculated by the selection calculation unit 81, i.e., the scene reliability calculation unit 87.

[0051] FIG. 15 shows an overview of an operation example according to this embodiment. As shown in FIG. 15, in this embodiment, first, in step 1501, the processor 5a acquires image data captured by the camera 2. Next, in step 1502, the processor 5a judges whether a scene exists. Then, in step 1503, the processor 5a calculates a scene determination reliability based on time-series data of the scene existence determination result in step 1502. Also, in step 1504, the processor 5a sets a weight value, which is a weighting value for each line type candidate determination unit 7, according to the scene existence determination result in step 1502. Then, in step 1505, the processor 5a integrates, i.e., corrects, the weight value using the scene determination reliability. Specifically, for example, if a heavy rain scene and a preceding vehicle obstruction scene exist and the scene determination reliability (rain) = 0.3 and the scene determination reliability (obscuration) = 0.2, each weighted value corresponding to the heavy rain scene is multiplied by a correction value of 0.3, and each weighted value corresponding to the preceding vehicle obstruction scene is multiplied by a correction value of 0.2. Finally, in step 1506, the processor 5a selects the line type candidate determination unit 7 or the line type candidate determination result based on the weighted values ​​integrated in step 1505.

[0052] FIG. 16 shows an outline of another mode of correction of weight value by scene determination reliability. This specific example shows the case of a heavy rain scene. First, the processor 5a executes the processes of steps 1601 to 1606 in order. In step 1601, the processor 5a initializes the setting state of the weight value for each line type candidate determination unit 7. In step 1602, the processor 5a acquires the determination result of the scene establishment status in the frame image of the current line type determination target. In step 1603, the processor 5a acquires the execution result of step 1602 in the previous time, that is, the determination result of the scene establishment status in the frame image of the previous line type determination target. In step 1604, the processor 5a sets a weight value, which is a weighting value for each line type candidate determination unit 7, according to the determination result of the scene establishment status in step 1602. In step 1605, the processor 5a calculates a weight value corresponding to each of the multiple line type candidate determination units 7 (i.e., each algorithm) based on the weight value set in step 1604. In step 1606, the processor 5a determines whether the heavy rain scene continues.

[0053] If the heavy rain scene continues (i.e., step 1606=YES), the processor 5a advances the process to step 1607. In step 1607, the processor 5a determines whether or not the number of times the heavy rain scene continues is less than a predetermined maximum threshold. If the number of times the heavy rain scene continues is less than the predetermined maximum threshold (i.e., step 1607=YES), the processor 5a advances the process to step 1608. In step 1608, the processor 5a adds 1 to the number of times the heavy rain scene continues. If the number of times the heavy rain scene continues is equal to or greater than the predetermined maximum threshold (i.e., step 1607=NO), the processor 5a advances the process to step 1609. In step 1609, the processor 5a sets the number of times the heavy rain scene continues to a predetermined maximum threshold. That is, the processor 5a limits the upper limit of the number of times the heavy rain scene continues to the predetermined maximum threshold.

[0054] If the heavy rain scene is not continuing (i.e., step 1606=NO), the processor 5a advances the process to step 1610. In step 1610, the processor 5a determines whether or not the number of continuous heavy rain scenes is greater than a predetermined minimum threshold. If the number of continuous heavy rain scenes is greater than the predetermined minimum threshold (i.e., step 1610=YES), the processor 5a advances the process to step 1611. In step 1611, the processor 5a adds -1 to the number of continuous heavy rain scenes. If the number of continuous heavy rain scenes is equal to or less than the predetermined minimum threshold (i.e., step 1610=NO), the processor 5a advances the process to step 1612. In step 1612, the processor 5a sets the number of continuous heavy rain scenes to a predetermined minimum threshold. That is, the processor 5a protects the lower limit of the number of continuous heavy rain scenes to the predetermined minimum threshold.

[0055] After the processing of step 1608, step 1609, step 1611, or step 1612, the processor 5a advances the processing to step 1613 and step 1614. In step 1613, the processor 5a calculates the scene determination reliability of the heavy rain scene by dividing the continuation count by a predetermined maximum threshold. In step 1614, the processor 5a corrects the weighted value calculated in step 1605 by multiplying the scene determination reliability.

[0056] (Selection Calculation Unit: Fourth Embodiment) Hereinafter, another example of the selection calculation unit 81 will be described. As described above, in a scene where a preceding vehicle is obscured, the line type determination by the basic algorithm based on the paint length and the gap length is less accurate than the line type determination by the other additional algorithms 1 and 2. In addition, when comparing the line type determination by the additional algorithm 1 based on the time series change in the position of a specific edge point in the longitudinal direction with the line type determination by the additional algorithm 2 based on the class determination result by semantic segmentation, the latter has higher accuracy. This is because, when the degree of obscuration by the preceding vehicle is large, the accuracy of the line type candidate determination based on the time series change in the position of a specific edge point in the longitudinal direction may be slightly reduced, whereas semantic segmentation enables good recognition by the class of "obscuration". For this reason, when the degree of obscuration by the preceding vehicle is large, the line type candidate determination by the additional algorithm 2 based on the class determination result by semantic segmentation has the highest accuracy, i.e., the highest accuracy rate.

[0057] 3 and 17, in this embodiment, the line type determination and selection unit 8, i.e., the selection calculation unit 81, has a class acquisition unit 88 and a blocking rate calculation unit 89. The class acquisition unit 88 is configured to acquire class determination results by semantic segmentation of all edge points in the lane line candidate. The blocking rate calculation unit 89 is configured to calculate a blocking rate indicating a blocking state of the lane line candidate by other vehicles. Then, when the blocking rate is less than the selection determination threshold, the selection result decision unit 82 selects the first additional candidate determination unit 71 or its line type candidate determination result, and when the blocking rate is equal to or greater than the selection determination threshold, the selection result decision unit 82 selects the second additional candidate determination unit 72 or its line type candidate determination result.

[0058] FIG. 18 shows an overview of an operation example according to this embodiment. As shown in FIG. 18, in this embodiment, first, the processor 5a executes the processes of steps 1801 to 1804 in order. In step 1801, the processor 5a acquires all edge points in a frame image. In step 1802, the processor 5a acquires a class determination result by semantic segmentation of all edge points acquired in step 1801. In step 1803, the processor 5a calculates an occlusion ratio based on the class determination result acquired in step 1802. In step 1804, the processor 5a determines whether the occlusion ratio is equal to or greater than a threshold value.

[0059] If the occlusion ratio is equal to or greater than the threshold (i.e., step 1804=YES), the processor 5a advances the process to step 1805. In step 1805, the processor 5a selects the additional algorithm 2, i.e., a line type determination method using a determination algorithm based on a class determination result by semantic segmentation. In other words, the processor 5a selects the second additional candidate determination unit 72 or its line type candidate determination result for determining the final line type determination result. If the occlusion ratio is less than the threshold (i.e., step 1804=NO), the processor 5a advances the process to step 1806. In step 1806, the processor 5a selects the additional algorithm 1, i.e., a line type determination method using a determination algorithm based on a time-series change in the position of a specific edge point in the forward and backward directions. In other words, the processor 5a selects the first additional candidate determination unit 71 or its line type candidate determination result for determining the final line type determination result. Following the process of step 1805 or step 1806, the processor 5a advances to the process for the next frame image.

[0060] (Additional Algorithm 1) Below, we will explain the details of addition algorithm 1, which is a line type determination algorithm based on time-series changes in the position of a specific edge point in the forward / backward direction, i.e., the first additional candidate determination unit 71. As shown in Fig. 19, the first additional candidate determination unit 71 has a first boundary line information detection unit 711, an edge point selection unit 712, a time-series change acquisition unit 713, and a first additional candidate determination unit 714.

[0061] The first boundary line information detection unit 711 is adapted to acquire, i.e., calculate, line information such as information on edge points, paint length, gap length, etc. The edge point selection unit 712 is adapted to select a specific edge point in a lane line candidate in the captured image, i.e., an edge point (e.g., a near edge point) to be processed by the time series change acquisition unit 713. The time series change acquisition unit 713 is adapted to acquire, i.e., calculate, a time series change in the position of the edge point selected by the edge point selection unit 712. The first additional candidate determination unit 714 is adapted to determine a line type candidate based on the time series change in the position in the forward and backward directions of the specific edge point acquired by the time series change acquisition unit 713.

[0062] The processing contents of the first additional candidate determination unit 71 using the above-mentioned addition algorithm 1 will be described below together with the effects achieved by this. In the line type determination, in the conventionally known template matching method, in adverse conditions such as blurring or obstruction by a preceding vehicle, or in the case of a special pattern that does not match the template, the paint length and gap length become unclear, and there is a concern that the accuracy of the line type determination will decrease. In this regard, the inventors have found, as a result of extensive research, that it is possible to realize a stable line type determination by detecting edge endpoints on the paint part of the boundary line candidate and determining the line type from the time-series change in the position information.

[0063] FIG. 20 shows the change in the position of edge points in the front-rear direction between two frame images captured consecutively in time series, that is, between a frame image captured at time t0 and the next frame image captured at time t1. In the figure, circles indicate edge points. The rectangular range indicated by the dashed line indicates the area in the frame image from which edge points are extracted, and is hereinafter referred to as the "edge point extraction area RZ". Such edge point extraction area RZ is the entire area of ​​the frame image or a part thereof. In other words, the edge point extraction area RZ is not necessarily the entire image area. Specifically, for example, the outer edge of the frame image is affected by lens aberration in the camera 2. In addition, the distance error may be large in the distant area, and the resolution of the edge points may be reduced. Therefore, the proximal end RZ1, which is the edge on the near side of the edge point extraction area RZ, may be set to a position slightly forward of the proximal end boundary R2 shown in FIG. 1B, for example, a position equivalent to about 8 m from the front end of the vehicle C. In addition, the distal end RZ2, which is the rear edge of the edge point extraction region RZ, can be set at a position in the frame image slightly lower than the position corresponding to the vanishing point when the vehicle C is placed on a horizontal surface so that it can run, for example, at a position corresponding to approximately 50 m from the front end of the vehicle C.

[0064] FIG. 20 shows an example in which a solid line candidate for a lane line is present on the left side of the edge point extraction region RZ and a dashed line candidate for a lane line is present on the right side. As shown in FIG. 20, in the solid line, the positions of the far edge points indicated by black circles and the near edge points indicated by diagonal hatching have a small amount of variation in the front-rear direction. In contrast, in the dashed line, the positions of the far edge points and the near edge points have a large amount of variation in the front-rear direction. FIG. 21 shows the change in the amount of positional variation of the near edge points in the front-rear direction over time in the solid line and the dashed line. In FIG. 21, the vertical axis shows the amount of positional variation in meters. The horizontal axis shows the number of frames corresponding to time. The frame interval is 100 ms. That is, FIG. 21 corresponds to the horizontal axis of the time chart converted from time to the number of frames. In this way, based on the amount of variation in the positions of the far edge points and the near edge points in the front-rear direction over time, a stable line type determination can be realized even under adverse conditions.

[0065] Specifically, for example, the edge point selection unit 712 selects a near edge point, which is an edge point closest to the vehicle C, and / or a far edge point, which is an edge point farthest from the vehicle C, in the lane marking candidate within the edge point extraction region RZ. The time series change acquisition unit 713 acquires time series changes in the positions of the edge points selected by the edge point selection unit 712. Then, the first additional candidate determination unit 714 determines a line type candidate based on the time series changes in the positions of the specific edge points in the forward and backward directions acquired by the time series change acquisition unit 713. That is, for example, when the positional changes of the near edge point or the far edge point are large, the line type candidate is determined to be a dashed line. On the other hand, for example, when the positional changes of the near edge point or the far edge point are small, the line type candidate is determined to be a solid line. Note that the edge point selection unit 712 can perform processing at each boundary line, and therefore can be used even when there are multiple lines such as multiple lines. That is, by processing each line, it is possible to finally use it for multiple line determination.

[0066] Fig. 22 is a flowchart corresponding to an aspect of selecting a near edge point and determining a line type candidate. Referring to Fig. 22 below, first, the processor 5a executes the processes of steps 2201 to 2203 in order. Specifically, in step 2201, the processor 5a obtains a near edge point in the currently acquired frame image. In step 2202, the processor 5a obtains, that is, calculates, the amount of positional variation from the near edge point in the previously acquired frame image. In step 2203, the processor 5a determines whether or not the amount of positional variation calculated in step 2202 is less than a threshold value.

[0067] If the positional variation is less than the threshold (i.e., step 2203=YES), the processor 5a advances the process to step 2204 and step 2205. In step 2204, the processor 5a determines that the line type of the lane marking candidate is likely to be a solid line. Therefore, in step 2205, the processor 5a adds the one-frame travel distance as the solid line distance. That is, assuming that the solid line distance is Ds and the one-frame travel distance is ΔDs, the processor 5a sets the value of Ds+ΔDs as a new Ds. Note that the "solid line distance" is the length of the continuous paint portion (i.e., the length on the road surface R) in the lane marking candidate to be determined. Also, the "one-frame travel distance" is the distance traveled by the host vehicle C in one frame interval. On the other hand, if the positional variation is equal to or greater than the threshold (i.e., step 2203=NO), the processor 5a advances the process to step 2206 and step 2207. In step 2206, the processor 5a determines that the line type of the lane marking candidate is likely to be a broken line. Therefore, in step 2207, the processor 5a subtracts the one-frame progress distance as a solid line distance. That is, the processor 5a sets the value of Ds-ΔDs as a new Ds.

[0068] After executing the process of step 2205 or step 2207 depending on the determination result in step 2203, the processor 5a advances the process to step 2208. In step 2208, the processor 5a determines whether the solid line distance is equal to or greater than the threshold. If the solid line distance is equal to or greater than the threshold (i.e., step 2208=YES), the processor 5a advances the process to step 2209. In step 2209, the processor 5a determines that the line type of the lane line candidate is a solid line. In contrast, if the solid line distance is less than the threshold (i.e., step 2208=NO), the processor 5a advances the process to step 2210. In step 2210, the processor 5a determines that the line type of the lane line candidate is a broken line. In this way, after executing the process of step 2209 or step 2210 depending on the determination result in step 2208, the processor 5a advances the process to step 2211. In step 2211, the processor 5a saves the near edge point in the currently acquired frame image as the previous value for the next processing time.

[0069] For example, edge point detection may be continuously difficult on the near end RZ1 side due to obstruction by a preceding vehicle in the own lane or an adjacent lane. Alternatively, for example, depending on the vehicle speed, paint length, or gap length, the accuracy of line type determination may decrease due to edge point position fluctuation. Specifically, for example, in the case of an auxiliary line in which both the paint length and the gap length are small, depending on the vehicle speed, it may always be the case that approximately the same paint position is acquired as the near edge point. In this regard, by monitoring edge point position fluctuation at multiple positions in the forward and backward directions, it is possible to effectively suppress the decrease in line type determination accuracy in the above-mentioned cases.

[0070] Therefore, the first additional candidate determination unit 71 may determine the line type of the lane marking candidate based on the far-side determination result, which is the line type determination result based on the time series change of the position of the far edge point, and the near-side determination result, which is the line type determination result based on the time series change of the position of the near edge point. The line type determination using the far-side determination result and the near-side determination result can use, for example, a logical operation such as AND or OR. FIG. 23 is a flowchart corresponding to such an embodiment. Referring to FIG. 23 below, first, the processor 5a executes the processes of steps 2301 to 2303 in order. After executing the process of step 2303, the processor 5a executes the processes of steps 2304 to 2307 and the processes of steps 2308 to 2311, and then proceeds to the process of step 2312. Then, the processor 5a determines the line type according to the determination result in step 2312.

[0071] Specifically, first, in step 2301, the processor 5a acquires, i.e., detects, edge points on the paint constituting the lane marking candidate in the currently acquired frame image. Next, in step 2302, the processor 5a acquires near edge points from among the edge points acquired in step 2301. Then, in step 2303, the processor 5a acquires far edge points from among the edge points acquired in step 2301.

[0072] In step 2304, the processor 5a calculates the solid line distance on the near edge point side. The "solid line distance on the near edge point side" is the solid line distance when a solid line is assumed to extend from the near edge point as a starting point toward the back side. In step 2305, the processor 5a judges whether the solid line distance on the near edge point side is equal to or greater than a threshold value. If the solid line distance on the near edge point side is equal to or greater than a threshold value (i.e., step 2305=YES), the processor 5a advances the process to step 2306. In step 2306, the processor 5a judges that the near edge point side is a solid line. That is, the processor 5a judges that the line type of the lane marking candidate currently being the line type judgment target is a solid line when viewed from the near edge point side. If the solid line distance on the near edge point side is less than the threshold value (i.e., step 2305=NO), the processor 5a advances the process to step 2307. In step 2307, the processor 5a determines that the side of the near edge point is a dashed line.

[0073] In step 2308, the processor 5a calculates the solid line distance on the far edge point side. The "solid line distance on the far edge point side" is the solid line distance when a solid line is assumed to extend from the far edge point as a starting point toward the near side. In step 2309, the processor 5a judges whether the solid line distance on the far edge point side is equal to or greater than a threshold value. If the solid line distance on the far edge point side is equal to or greater than the threshold value (i.e., step 2309=YES), the processor 5a advances the process to step 2310. In step 2310, the processor 5a judges that the far edge point side is a solid line. If the solid line distance on the far edge point side is less than the threshold value (i.e., step 2309=NO), the processor 5a advances the process to step 2311. In step 2311, the processor 5a judges that the far edge point side is a broken line.

[0074] In step 2312, the processor 5a judges whether the near edge point side or the far edge point side is determined to be a solid line. That is, the processor 5a judges whether the processing of step 2306 or step 2310 has been executed. In other words, the processor 5a judges whether the judgment result in step 2305 or step 2309 is "YES". If the near edge point side or the far edge point side is determined to be a solid line (i.e., step 2312=YES), the processor 5a advances the processing to step 2313. In step 2313, the processor 5a judges the line type of the lane line candidate to be a solid line. In contrast, if the near edge point side and the far edge point side are determined to be a broken line (i.e., step 2312=NO), the processor 5a advances the processing to step 2314. In step 2314, the processor 5a judges the line type of the lane line candidate to be a broken line. Finally, in step 2315, the processor 5a saves the near edge points and far edge points in the currently acquired frame image as previous values ​​for the next processing time.

[0075] The first additional candidate determination unit 71 may determine the line type of the lane line candidate based on the weighted far-side determination result and near-side determination result. That is, the first additional candidate determination unit 71 sets a weighting value for the far-side determination result based on a time series change in the position of the far edge point. The first additional candidate determination unit 71 also sets a weighting value for the near-side determination result based on a time series change in the position of the near edge point. Then, the first additional candidate determination unit 71 determines the line type of the lane line candidate based on the far-side determination result and the near-side determination result and the weighting value for them. The weighting value in this case may be calculated based on the stability (e.g., variance, etc.) of the amount of position change in a certain travel section or travel time. FIG. 24 is a flowchart corresponding to such an embodiment. Referring to FIG. 24 below, first, the processor 5a executes the processes of steps 2401 to 2409 in order. Then, the processor 5a determines the line type according to the determination result in step 2409.

[0076] Specifically, in step 2401, the processor 5a acquires, i.e., detects, edge points on the paint constituting the lane line candidate in the currently acquired frame image. In step 2402, the processor 5a acquires near edge points from among the edge points acquired in step 2401. In step 2403, the processor 5a acquires far edge points from among the edge points acquired in step 2401. In step 2404, the processor 5a calculates the solid line distance on the near edge point side. In step 2405, the processor 5a calculates the solid line distance on the far edge point side. In step 2406, the processor 5a calculates the stability (e.g., variance, etc.) of the amount of position change at the near edge point and the far edge point. In step 2407, the processor 5a calculates a weighting value based on the stability calculated in step 2406. In step 2408, the processor 5a applies (eg, multiplies) the weighting value calculated in step 2408 to the solid-line distance calculated in steps 2404 and 2405, thereby calculating a weighted solid-line distance.

[0077] In step 2409, the processor 5a determines whether the weighted solid line distance on the near edge point side or the far edge point side is equal to or greater than a threshold value. If it is equal to or greater than the threshold value (i.e., step 2409=YES), the processor 5a advances the process to step 2410. In step 2410, the processor 5a determines that the line type of the lane line candidate is a solid line. In contrast, if it is less than the threshold value (i.e., step 2409=NO), the processor 5a advances the process to step 2411. In step 2411, the processor 5a determines that the line type of the lane line candidate is a dashed line. Finally, the processor 5a advances the process to step 2412. In step 2412, the processor 5a saves the near edge points and far edge points in the currently acquired frame image as previous values ​​for the next process.

[0078] Here, as described above, the first additional candidate determination unit 71 may determine the location for monitoring the time-series change of the position of a specific edge point, such as a far edge point and a near edge point, by a predetermined distance. That is, for example, appropriate positions of the proximal end RZ1 and the distal end RZ2 in the edge point extraction region RZ may be set in advance using the results of experiments or computer simulations. Alternatively, the monitoring positions of the far edge point and the near edge point may be set at positions offset by a predetermined distance from the proximal end RZ1 and the distal end RZ2. Alternatively, the edge point positions may also be monitored at intermediate positions between the proximal end RZ1 and the distal end RZ2. FIG. 25 shows an example in which the near monitoring position P1, which is the monitoring position of the near edge point, is offset slightly backward from the proximal end RZ1, the far monitoring position P2, which is the monitoring position of the far edge point, is offset slightly forward from the distal end RZ2, and an intermediate monitoring position P3 is set between the two. In the figure, the edge points shown with cross-hatching indicate detected edge points that correspond to intermediate monitoring position P3, which is located between the far edge point and the near edge point. Such detected edge points are detected at positions closest to intermediate monitoring position P3 in the front-to-rear direction, i.e., the up-down direction in the figure. Note that multiple intermediate monitoring positions P3 may be provided. Positions at which time-series changes in the positions of near edge points, far edge points, etc. are monitored (i.e., near monitoring position P1, etc.) may hereinafter be collectively referred to simply as "monitoring positions."

[0079] FIG. 26 shows an example of operation in the case where an intermediate monitoring position P3 is used in addition to the near monitoring position P1 and the far monitoring position P2. Referring to FIG. 26, first, in step 2601, the processor 5a acquires, i.e., detects, edge points on the paint constituting the lane line candidate in the currently acquired frame image. Next, in step 2602, the processor 5a acquires edge points corresponding to each monitoring position from the edge points acquired in step 2401. Next, in step 2603, the processor 5a calculates the solid line distance (e.g., the solid line distance on the near edge point side) of one of the edge points acquired in step 2602. Then, in step 2604, the processor 5a determines whether or not the calculation of the solid line distance for all edge points is completed. If there is an edge point for which the solid line distance has not been calculated (i.e., step 2604=NO), the processor 5a advances the process to step 2605. In step 2605, the processor 5a advances the process to the next edge point (e.g., a far edge point), and calculates the solid-line distance at that edge point (e.g., the solid-line distance on the far edge point side) in step 2603. When the calculation of the solid-line distance for all edge points is completed in this manner (i.e., step 2604=YES), the processor 5a advances the process to step 2606 and subsequent steps.

[0080] In step 2606, the processor 5a weights the solid line distance based on the stability (e.g., variance) of the amount of position change at a plurality of monitoring positions, as in the above example. Then, in step 2607, the processor 5a determines whether the weighted solid line distance is equal to or greater than a threshold, as in step 2409 above. If it is equal to or greater than the threshold (i.e., step 2607=YES), the processor 5a advances the process to step 2608. In step 2608, the processor 5a determines that the line type of the lane line candidate is a solid line. On the other hand, if it is less than the threshold (i.e., step 2607=NO), the processor 5a advances the process to step 2609. In step 2609, the processor 5a determines that the line type of the lane line candidate is a broken line. Finally, the processor 5a advances the process to step 2610. In step 2610, the processor 5a saves each edge point in the currently acquired frame image as a previous value for the next process.

[0081] The monitoring position may be set according to the shooting conditions and the like. FIG. 27 is a block diagram corresponding to such an embodiment. That is, as shown in FIG. 27, the first additional candidate determination unit 71 has a first boundary information detection unit 711, an edge point selection unit 712, a time series change acquisition unit 713, a first additional candidate determination unit 714, and a monitoring position determination unit 715. The monitoring position determination unit 715 is provided to set the monitoring position according to the shooting conditions and the like. Specifically, for example, the monitoring position determination unit 715 can set the monitoring position based on the paint length and gap length, which are line information. This makes it possible to achieve good line type recognition accuracy even in the case where the paint length and gap length are both small auxiliary lines and the position change due to the line type is unlikely to occur at a specific monitoring position depending on the vehicle speed, as described above.

[0082] FIG. 28 shows an example of operation corresponding to such an aspect. Referring to FIG. 28 below, first, in step 2801, the processor 5a acquires, i.e., detects, edge points on the paint constituting the lane line candidate in the currently acquired frame image. Next, in step 2802, the processor 5a acquires, i.e., calculates, the paint length and the gap length. Subsequently, in step 2803, the processor 5a sets a monitoring position based on the paint length and the gap length acquired in step 2802, and acquires an edge point for determining the line type at the set monitoring position. Furthermore, in step 2804, the processor 5a calculates the solid line distance of one of the edge points acquired in step 2803 (for example, a near edge point). Then, in step 2805, the processor 5a judges whether or not the calculation of the solid line distance for all edge points is completed. If there is an edge point for which the solid line distance has not been calculated (i.e., step 2805=NO), the processor 5a advances the process to step 2806. In step 2806, the processor 5a advances the process to the next edge point (e.g., a far edge point), and calculates the solid-line distance at that edge point in step 2804. When the calculation of the solid-line distance for all edge points is completed in this manner (i.e., step 2805=YES), the processor 5a advances the process to step 2807 and subsequent steps.

[0083] In step 2807, the processor 5a weights the solid line distance based on the stability (e.g., variance) of the amount of position change at a plurality of monitoring positions, as in the above example. Then, in step 2808, the processor 5a determines whether the weighted solid line distance is equal to or greater than a threshold. If it is equal to or greater than the threshold (i.e., step 2808=YES), the processor 5a advances the process to step 2809. In step 2809, the processor 5a determines that the line type of the lane line candidate is a solid line. On the other hand, if it is less than the threshold (i.e., step 2808=NO), the processor 5a advances the process to step 2810. In step 2810, the processor 5a determines that the line type of the lane line candidate is a broken line. Finally, the processor 5a advances the process to step 2811. In step 2812, the processor 5a saves each edge point in the currently acquired frame image as a previous value for the next process.

[0084] Alternatively, for example, when determining the monitoring position, in a situation where an obstruction such as an adjacent vehicle is reflected, an arbitrary monitoring position may not be stable, and the line type determination accuracy may decrease. Specifically, for example, when a cutting-in vehicle is present diagonally ahead of the vehicle C and obstruction occurs due to the cutting-in vehicle, the stability is relatively low at the proximal end RZ1 or its vicinity because it is affected by the obstruction of the object, whereas the stability is relatively high at the distal end RZ2 or its vicinity because the influence of the obstruction of the object is small. Therefore, the monitoring position determination unit 715 can set the monitoring position based on the recognized position of the object in the captured image around the vehicle C. That is, for example, the monitoring position determination unit 715 sets the monitoring position in an area that does not overlap with the rectangular object recognition frame formed at the position of the recognized object. This can effectively suppress the deterioration of the line type determination accuracy caused by obstructions such as adjacent vehicles.

[0085] FIG. 29 shows an example of operation corresponding to such an aspect. Referring to FIG. 29 below, first, in step 2901, the processor 5a acquires and detects edge points on the paint constituting the lane line candidate in the currently acquired frame image. Next, in step 2902, the processor 5a acquires an object recognition result. Subsequently, in step 2903, the processor 5a sets a monitoring position based on the object recognition result acquired in step 2902, and acquires an edge point for line type determination at the set monitoring position. Furthermore, in step 2904, the processor 5a calculates the solid line distance of one of the edge points acquired in step 2903 (for example, a near edge point). Then, in step 2905, the processor 5a judges whether or not the calculation of the solid line distance for all edge points is completed. If there is an edge point for which the solid line distance has not been calculated (i.e., step 2905=NO), the processor 5a advances the process to step 2906. In step 2906, the processor 5a advances the process to the next edge point (e.g., a far edge point), and calculates the solid-line distance at that edge point in step 2904. When the calculation of the solid-line distance for all edge points is completed in this manner (i.e., step 2905=YES), the processor 5a advances the process to step 2907 and subsequent steps. The process contents in step 2907 and subsequent steps are the same as the process contents in step 2807 and subsequent steps.

[0086] (Additional Algorithm 2) Below, we will explain the details of the addition algorithm 2, which is a line type determination algorithm based on the class determination result by semantic segmentation, i.e., the second additional candidate determination unit 72. As shown in Figure 30, the second additional candidate determination unit 72 has a second boundary line information detection unit 721, a class output unit 722, a class rate calculation unit 723, and a second additional candidate determination unit 724.

[0087] The second boundary information detection unit 721 is adapted to detect boundary edge point information by machine learning such as DNN, which is line information, and road shapes (e.g., horizontal position, yaw angle, curvature, etc.) obtained from polynomial fitting. The class output unit 722 is adapted to output a class by semantic segmentation for each pixel in the lane line candidate in the captured image. The class rate calculation unit 723 is adapted to calculate a class rate, which is a probability that a pixel corresponds to a class, based on the output of the class output unit 722. Specifically, in this embodiment, the class rate calculation unit 723 is adapted to calculate the class rate for the entire pixel row arranged in the front-rear direction, i.e., the entire boundary line, without limiting the area of ​​interest. The second additional candidate determination unit 724 is adapted to determine the line type of the lane line candidate based on the class rate calculated by the class rate calculation unit 723.

[0088] The processing contents of the second additional candidate determination unit 72 using the above-mentioned addition algorithm 2 will be described below together with the effects achieved by this. In the line type determination, in the conventionally known template matching method, in adverse conditions such as blurring or obstruction by a preceding vehicle, or in the case of a special pattern that does not match the template, the paint length and gap length become unclear, and there is a concern that the accuracy of the line type determination will decrease. In particular, there may be cases where it is difficult to detect edge points on the paint due to obstruction by a preceding vehicle or shade on the paint, etc.

[0089] In this regard, the inventors have found, as a result of intensive research, that a stable line type determination can be realized by using the class rate (for example, paint rate, gap rate, etc.) of semantic segmentation. FIG. 31 shows an overview of line type determination using class rates. In the figure, white circles indicate pixels of the "paint" class, and black circles indicate pixels of the "gap" class. In the example of FIG. 31, a pixel row in which pixels of the "paint" class are continuously arranged in the front-rear direction exists on the left side, and a pixel row in which pixels of the "paint" class and pixels of the "gap" class are mixed exists on the right side. The pixel row on the left side has a paint rate of 100% and a gap rate of 0%. Therefore, it is possible to estimate that such a pixel row is a solid road dividing line. On the other hand, the pixel row on the right side has a paint rate of 60% and a gap rate of 40%. Therefore, it is possible to estimate that such a pixel row is a broken road dividing line. Thus, according to this embodiment, good line type determination is possible even in a situation in which it is difficult to determine the line type due to the change in the position of the edge points as described above. The class rate is calculated not for a specific area, but for the entire pixel row arranged in the front-rear direction, i.e., for the entire boundary line. Specifically, for example, if a part of the boundary line is blocked by a preceding vehicle, the class rate calculation unit 723 calculates the class rate by capturing the whole part other than the blocked part.

[0090] FIG. 32 shows an overview of road shape estimation using the line type determination result by the second additional candidate determination unit 72. Referring to FIG. 32 below, first, in step 3201, the processor 5a detects the road shape (for example, lateral position, yaw angle, curvature, etc.) obtained from boundary edge point information by machine learning such as DNN or polynomial fitting, which is line information. In addition, in step 3202, the processor 5a outputs classes such as "road surface", "own lane paint", "own lane gap", "other lane paint", "zebra", "solid object", and "moving object" for each pixel by semantic segmentation using DNN. Next, in step 3203, the processor 5a tracks the same line based on the degree of coincidence for each period of the road shape information, and determines the own lane / adjacent lane. In step 3204, the processor 5a determines the line type and line color of each boundary. In step 3205, the processor 5a determines the boundary type (multiple lines / compound lines) from the line type determination result and compound line information by DNN. In step 3206, the processor 5a determines an increase or decrease in the number of lanes (branching or merging) from the boundary edge point sequence and the road shape. In step 3207, the processor 5a estimates a road model shape from the boundary edge point sequence using a Kalman filter or the like.

[0091] FIG. 33 shows an example of line type candidate determination by the second additional candidate determination unit 72. Referring to FIG. 33 below, first, in step 3301, the processor 5a acquires all edge points on the boundary line, i.e., the lane line candidate. Next, in step 3302, the processor 5a acquires semantic segmentation classes for all edge points. Then, in step 3303, the processor 5a calculates the class rate (e.g., paint rate, gap rate, zebra rate, etc.) for each lane line candidate. In the figure, "Pp" indicates the paint rate and "Pg" indicates the gap rate. Then, in step 3304, the processor 5a determines whether the paint rate is equal to or greater than the solid line threshold.

[0092] If the paint ratio is equal to or greater than the threshold for a solid line (i.e., step 3304=YES), the processor 5a advances the process to step 3305. In step 3305, the processor 5a determines that the line type of the lane line candidate is a solid line. In contrast, if the paint ratio is less than the threshold for a solid line (i.e., step 3304=NO), the processor 5a advances the process to step 3306. In step 3306, the processor 5a determines whether or not the paint ratio is equal to or less than the threshold for a broken line. If the paint ratio is equal to or less than the threshold for a broken line (i.e., step 3306=YES), the processor 5a advances the process to step 3307. In step 3307, the processor 5a determines whether or not the gap ratio is equal to or greater than the threshold for a broken line. If the gap ratio is equal to or greater than the threshold for a broken line (i.e., step 3307=YES), the processor 5a advances the process to step 3308. In step 3308, the processor 5a determines that the line type of the lane line candidate is a broken line. If the paint ratio is less than the solid line threshold (i.e., step 3304=NO) and the paint ratio exceeds the dashed line threshold (i.e., step 3306=NO) or the gap ratio is less than the dashed line threshold (i.e., step 3307=NO), the processor 5a advances the process to step 3309. In step 3309, the processor 5a determines that the line type of the partition line candidate is other than a solid line or a dashed line. After the process of step 3305, step 3308, or step 3309, the processor 5a advances the process to the next frame image.

[0093] For example, the second additional candidate determination unit 724 can determine the line type of the lane marking candidate based on the time-series change in the class rate. Specifically, for example, the class rate calculation unit 723 can correct the class rate by multiplying the most recently calculated class rate by a correction coefficient according to the time-series change in the class rate. Such a correction coefficient may also be referred to as the reliability of the class rate. This makes it possible to perform a stable line type determination even when there is a change in the occlusion state due to a preceding vehicle or the like.

[0094] In general, the recognition distance of other lanes, such as adjacent lanes, is longer than that of the own lane. Therefore, the paint rate and gap rate thresholds for determining whether the lane is a solid line or a broken line may be switched depending on whether the lane is the own lane or another lane. In other words, when the second additional candidate determination unit 724 determines the line type of the lane candidate depending on whether the class rate is equal to or greater than the class determination threshold, the second additional candidate determination unit 724 may use different class determination thresholds for the lane candidate in the own lane and the lane candidate in the adjacent lane. This can effectively suppress the variation in recognition accuracy associated with the difference in lateral position, i.e., the position perpendicular to the fore-aft direction.

[0095] (Other variations) The present invention is not limited to the above-mentioned embodiment and modified examples. Therefore, the above-mentioned embodiment and the like can be modified as appropriate. Representative other modified examples will be described below. In the following description of the modified examples, differences from the above-mentioned embodiment and the like will be mainly described. In addition, the same reference numerals are used for parts that are the same or equivalent in the above-mentioned embodiment and the following modified examples. Therefore, in the following description of the modified examples, the description in the above-mentioned embodiment and the like may be used as appropriate for components having the same reference numerals as the above-mentioned embodiment and the like, unless there is a technical contradiction or special additional explanation.

[0096] The present invention is not limited to the specific device configuration shown in the above embodiment. For example, the camera 2 is not limited to a so-called front camera, but may be a full-surround camera. There is also no particular limitation on the number and mounting positions of the cameras 2.

[0097] The image recognition device 5 may be provided in the recognition result utilization unit 4. That is, the image recognition device 5 may be an image recognition unit provided in an automatic driving ECU or a driving assistance ECU. ECU is an abbreviation for Electronic Control Unit.

[0098] All or part of the image recognition device 5 may be configured to include a digital circuit configured to enable the above-mentioned operations, such as an ASIC or FPGA. ASIC is an abbreviation for Application Specific Integrated Circuit. FPGA is an abbreviation for Field Programmable Gate Array. In other words, in the image recognition device 5, an on-board microcomputer portion and a digital circuit portion may coexist.

[0099] The program according to the present invention, which enables the execution of various operations, procedures, or processes described in the above embodiment, can be downloaded or upgraded via V2X communication. V2X is an abbreviation for Vehicle to X. Alternatively, such a program can be downloaded or upgraded via a terminal device provided in a manufacturing plant, a maintenance plant, a dealer, or the like of the vehicle C. Such a program may be stored on a memory card, an optical disk, a magnetic disk, or the like.

[0100] In this way, each of the above functional configurations and processes may be realized by a dedicated computer provided by configuring a processor and a memory programmed to execute one or more functions embodied in a computer program. Alternatively, each of the above functional configurations and processes may be realized by a dedicated computer provided by configuring a processor with one or more dedicated hardware logic circuits. Alternatively, each of the above functional configurations and processes may be realized by one or more dedicated computers configured by combining a processor and a memory programmed to execute one or more functions with a processor configured with one or more hardware logic circuits. Furthermore, the computer program may be stored in a computer-readable non-transient tangible storage medium as instructions executed by the computer. In other words, each of the above functional configurations and processes may be expressed as a computer program including a procedure for realizing it, or as a non-transient tangible storage medium storing the program.

[0101] 3 and the like are merely set for the sake of convenience in order to facilitate understanding of the contents of the present invention. Therefore, even if these functional blocks are not actually realized as subroutines or hardware within the image recognition device 5, the requirements of the present invention can be satisfied as long as the functions or processes of the present invention are realized.

[0102] In FIG. 3, three line type candidate determination units 7, namely, a basic candidate determination unit 70, a first additional candidate determination unit 71, and a second additional candidate determination unit 72, are provided as the multiple line type candidate determination units 7, but the present invention is not limited to this embodiment. Specifically, for example, two line type candidate determination units 7, namely, a basic candidate determination unit 70 and a first additional candidate determination unit 71, may be provided as the multiple line type candidate determination units 7. That is, the second additional candidate determination unit 72 may be omitted. Similarly, the first additional candidate determination unit 71 may be omitted. Alternatively, four or more line type candidate determination units 7 may be provided as the multiple line type candidate determination units 7. Furthermore, in the multiple line type candidate determination units 7, a boundary line information detection unit that acquires line information may be shared.

[0103] The present invention is not limited to the specific operation modes shown in the above embodiment. For example, the application of the present invention is not limited to vehicle driving control such as lane departure warning, lane keeping assistance, lane change assistance, automated lane change, adaptive cruise control, and automatic overtaking. Specifically, the present invention may be applied to, for example, current position estimation of the vehicle C and probe map information. In addition, as an example of use, when a solid line is determined and the number of edge points is small, it is possible to lower the dynamic threshold for edge extraction and provide feedback to ensure the number of edge points. Furthermore, as another example of use, it is possible to dynamically optimize the parameters of the estimation model. Specifically, since road shape estimation uses only a few of the edge points, the selection of the edge points is crucial. When a solid line is determined, the edges are considered to be stably arranged, and the observation error can be set small. When a broken line is determined, the observation error can be set large, taking into consideration that the position may slightly deviate from the boundary line when the edge of the end point of the broken line is selected.

[0104] Regarding the present invention, it is possible to assume an embodiment in which one of the multiple line type candidate determination units 7 is selected according to the scene, and line type determination is performed only by the selected line type candidate determination unit 7, and an embodiment in which one of the line type determination results in each of the multiple line type candidate determination units 7 is selected according to the scene. That is, the present invention is not limited to any of the above embodiments unless there is a restriction due to technical consistency or a special intention. Also, any of the above embodiments may be used depending on the scene establishment status. That is, for example, when only one scene is established, line type determination is performed only by one line type candidate determination unit 7 to reduce the calculation load, while when multiple scenes are established, the line type candidate determination unit 7 with the highest weight value in each established scene may be selected, and the final line type determination may be performed based on the line type determination result of the selected line type candidate determination unit 7. Therefore, for example, in the examples of Figures 5 and 6, when a composite line scene and a heavy rain scene are established, the line type candidate determination by the additional algorithm 2, i.e., the second additional candidate determination unit, may not be performed.

[0105] The scene determination unit 83 may determine whether a scene exists based on the output of various sensors, such as a raindrop sensor and an illuminance sensor, included in the vehicle state detection unit 3. There is no particular limitation on the method of setting the weight value. That is, in the examples of Fig. 5 and Fig. 6, the weight value may be, for example, "0" when the accuracy rate is less than 90%, and may be the "one's digit" of the two-digit number obtained by rounding off the first decimal place of the accuracy rate number when the accuracy rate is 90% or more. Alternatively, the weight value may be, for example, a value obtained by subtracting a predetermined value (for example, 90) from the accuracy rate.

[0106] When determining between numerical values, "XX or more" and "exceeding XX" are interchangeable. Similarly, "XX or less" and "less than XX" are interchangeable. Terms that show similar concepts, such as "detection," "sensing," "calculation," "acquisition," and "computation," are interchangeable as long as they are not technically contradictory.

[0107] The modified examples are not limited to the above examples. In addition, multiple modified examples may be combined with each other. Furthermore, all or part of the above embodiment and all or part of the modified examples may be combined with each other.

[0108] (Perspective) As is apparent from the above description of the configurations and operations according to the embodiments and modified examples, the disclosure of this specification includes at least the following aspects. [Point 1-1] A road dividing line recognition device (5) that recognizes road dividing lines from a captured image, a plurality of line type candidate determination units (7) that perform line type candidate determination, which is a determination of the line type of the lane marking candidate, based on line information that is information regarding shape characteristics of the lane marking candidate that is a candidate for a road lane marking obtained from the captured image; a line type determination selection unit (8) that performs line type determination as to which line type of the road-dividing line candidate corresponds to the lane-dividing line candidate by selecting one of the line type candidate determination units or one of the line type candidate determination results by each of the line type candidate determination units; Equipped with The plurality of line type candidate determination units have different determination algorithms. Road marking recognition device. [Point 1-2] The line type determination and selection unit A scene determination unit (83) is provided for determining a scene establishment status corresponding to a shooting situation that affects the line type candidate determination, selecting the line type candidate determination unit or the line type candidate determination result according to a result of the determination of the scene establishment state by the scene determination unit; A road dividing line recognition device according to aspect 1-1. [Points 1-3] A weight value that is a weighting value for each of the line type candidate determination units is set for each of the different scenes, the line type determination selection unit selects the line type candidate determination unit or the line type candidate determination result based on the weighted value corresponding to the scene determined to be established by the scene determination unit. A road dividing line recognition device according to aspect 1-2. [Points 1-4] the line type determination selection unit, when there are a plurality of the line type candidate determination units or their determination results corresponding to the largest weighting value, performs the line type determination based on a result of comparing the determination results with each other; A road dividing line recognition device according to aspects 1-3. [Points 1-5] The line type determination and selection unit when the scene determination unit determines that a plurality of the scenes have been established, the weighted values ​​corresponding to the established scenes are added for each of the plurality of line type candidate determination units; selecting the line type candidate determination unit or the line type candidate determination result corresponding to the maximum value of the sum resulting from the addition; A road dividing line recognition device according to aspects 1-3. [Points 1-6] The line type determination and selection unit A scene reliability calculation unit (87) is further provided to calculate a reliability of the scene determination result by the scene determination unit, selecting the line type candidate determination unit or the line type candidate determination result based on the weighted value corrected by the reliability; The road lane marking recognition device according to any one of Aspects 1-3 to 1-5. [Point 1-7] The line type determination and selection unit A scene reliability calculation unit (87) is further provided to calculate a reliability of the scene determination result by the scene determination unit, selecting the line type candidate determination unit or the line type candidate determination result based on the reliability; A road dividing line recognition device according to aspect 1-2. [Point 1-8] the scene reliability calculation unit calculates the reliability based on a continuation state of a result of the scene determination by the scene determination unit; A road dividing line recognition device according to aspect 1-6 or 1-7. [Point 1-9] A first line type candidate determination unit (71), which is one of the plurality of line type candidate determination units, an edge point selection unit (712) that selects edge points in the lane marking candidate in the captured image; a time series change acquisition unit (713) that acquires a time series change in the position of the edge point selected by the edge point selection unit; Equipped with the edge point selection unit selects a near edge point, which is the edge point closest to the host vehicle (C), and / or a far edge point, which is the edge point farthest from the host vehicle; the first line type candidate determination unit determines the line type of the lane marking candidate based on a time series change in the position of the near edge point and / or the far edge point acquired by the time series change acquisition unit. The road lane line recognition device according to any one of Aspects 1-1 to 1-8. [Points 1-10] The first line type candidate determination unit determining the line type of the lane marking candidate based on a far-side determination result, which is a determination result of the line type based on a time series change in the positions of the far edge points, and a near-side determination result, which is a determination result of the line type based on a time series change in the positions of the near edge points; A road dividing line recognition device according to aspects 1-9. [Point 1-11] The first line type candidate determination unit setting a weighting value for the far-side determination result based on a time-series change in the position of the far edge point; setting a weighting value for the near side determination result based on a time series change in the position of the near edge point; determining the line type of the lane marking candidate based on the weighted far-side determination result and the near-side determination result; A road dividing line recognition device as described in aspects 1-10. [Point 1-12] the first line type candidate determination unit sets a location for monitoring a time series change in the positions of the far edge point and the near edge point based on a paint length and a gap length which are the line information; A road lane line recognition device according to any one of Aspects 1-9 to 1-11. [Point 1-13] the first line type candidate determination unit sets a location for monitoring a time-series change in the positions of the far edge point and the near edge point based on a recognized position in the captured image of an object around the host vehicle; A road lane line recognition device according to any one of Aspects 1-9 to 1-11. [Point 1-14] A second line type candidate determination unit (72), which is another one of the plurality of line type candidate determination units, a class output unit (722) that outputs a class determined by semantic segmentation for each pixel in the lane marking candidate in the captured image; A class rate calculation unit (723) that calculates a class rate, which is a probability that the class corresponds to the class, based on the output of the class output unit; Equipped with determining the line type of the lane marking candidate based on the class rate calculated by the class rate calculation unit; The road lane line recognition device according to any one of Aspects 1-9 to 1-13. [Point 1-15] the second line type candidate determination unit determines the line type of the lane marking candidate based on a time-series change in the class rate. A road dividing line recognition device as described in aspect 1-14. [Point 1-16] The second line type candidate determination unit determining the line type of the lane marking candidate based on whether the class rate is equal to or greater than a class determination threshold; using different class determination thresholds for the lane marking candidate in the own lane and the lane marking candidate in an adjacent lane; A road dividing line recognition device according to aspect 1-14 or 1-15. [Point 1-17] The line type determination and selection unit A blocking rate calculation unit (89) is provided which calculates a blocking rate indicating a blocking state of the lane marking candidate by another vehicle, When the shielding rate is less than a selection determination threshold, the first line type candidate determination unit or the line type candidate determination result thereof is selected; When the shielding rate is equal to or greater than the selection determination threshold, the second line type candidate determination unit or the line type candidate determination result is selected. The road lane line recognition device according to any one of Aspects 1-14 to 1-16. [Point 2-1] A road dividing line recognition program executed by a road dividing line recognition device (4) that recognizes road dividing lines from a captured image, The process executed by the road division line recognition device is as follows: a plurality of line type candidate determination processes each having a different determination algorithm, which perform line type candidate determination for determining a line type of a lane marking candidate based on line information, which is information regarding shape characteristics of a lane marking candidate that is a candidate for a road lane marking, obtained from the captured image; a line type determination selection process for selecting one of the line type candidate determination results obtained by the plurality of line type candidate determination processes, thereby executing a line type determination as to which line type of the road-dividing line the lane marking candidate corresponds to; Including, Road marking recognition program. [Point 2-2] The line type determination and selection process includes: A scene determination process is included, which determines a scene establishment status corresponding to a shooting situation that affects the line type candidate determination, selecting the line type candidate determination process or the line type candidate determination result according to a determination result of the scene establishment status by the scene determination process; A road dividing line recognition program according to aspect 2-1. [Point 2-3] A weighting value is set for each of the different scenes, the weighting value being a weighting value for each of the line type candidate determination processes; the line type determination selection process selects the line type candidate determination process or the line type candidate determination result based on the weighted value corresponding to the scene determined to be established by the scene determination process; A road dividing line recognition program according to aspect 2-2. [Point 2-4] the line type determination selection process, when there are a plurality of the line type candidate determination processes or their determination results corresponding to the largest weighted value, performs the line type determination based on a result of comparing the determination results; A road dividing line recognition program according to aspect 2-3. [Point 2-5] The line type determination and selection process includes: if the scene determination process determines that a plurality of the scenes have been established, the weighted values ​​corresponding to the established scenes are added for each of the plurality of line type candidate determination processes; selecting the line type candidate determination process or the line type candidate determination result corresponding to the maximum sum value that is the result of the addition; A road dividing line recognition program according to aspect 2-3. [Point 2-6] The line type determination and selection process includes: The method further includes a scene reliability calculation process for calculating a reliability of the scene determination result by the scene determination process, selecting the line type candidate determination process or the line type candidate determination result based on the weighted value corrected by the reliability; A road division line recognition program according to any one of Aspects 2-3 to 2-5. [Point 2-7] The line type determination and selection process includes: The method further includes a scene reliability calculation process for calculating a reliability of the scene determination result by the scene determination process, selecting the line type candidate determination process or the line type candidate determination result based on the reliability; A road dividing line recognition program according to aspect 2-2. [Point 2-8] the scene reliability calculation process calculates the reliability based on a continuation state of a scene determination result by the scene determination process; A road dividing line recognition program according to aspect 2-6 or 2-7. [Point 2-9] A first line type candidate determination process, which is one of the plurality of line type candidate determination processes, includes: an edge point selection process for selecting edge points on the lane marking candidate in the captured image; a time series change acquisition process for acquiring a time series change in the position of the edge point selected by the edge point selection process; Including, The edge point selection process includes selecting a near edge point, which is the edge point closest to the host vehicle (C), and / or a far edge point, which is the edge point farthest from the host vehicle; the first line type candidate determination process determines the line type of the lane marking candidate based on a time series change in the positions of the near edge point and / or the far edge point acquired by the time series change acquisition process; A road division line recognition program according to any one of Aspects 2-1 to 2-8. [Point 2-10] The first line type candidate determination process includes: determining the line type of the lane marking candidate based on a far-side determination result, which is a determination result of the line type based on a time series change in the positions of the far edge points, and a near-side determination result, which is a determination result of the line type based on a time series change in the positions of the near edge points; A road dividing line recognition program according to aspect 2-9. [Point 2-11] The first line type candidate determination process includes: setting a weighting value for the far-side determination result based on a time-series change in the position of the far edge point; setting a weighting value for the near side determination result based on a time series change in the position of the near edge point; determining the line type of the lane marking candidate based on the weighted far-side determination result and the near-side determination result; A road dividing line recognition program according to aspect 2-10. [Point 2-12] the first line type candidate determination process sets a location for monitoring a time series change in the positions of the far edge point and the near edge point based on a paint length and a gap length, which are the line information; A road dividing line recognition program according to any one of aspects 2-9 to 2-11. [Point 2-13] The first line type candidate determination process sets a location for monitoring a time series change in the positions of the far edge point and the near edge point based on a recognized position in the captured image of an object around the host vehicle. A road dividing line recognition program according to any one of aspects 2-9 to 2-11. [Point 2-14] A second line type candidate determination process, which is another one of the plurality of line type candidate determination processes, includes: a class output process for outputting a class determined by semantic segmentation for each pixel of the lane marking candidate in the captured image; A class rate calculation process for calculating a class rate, which is a probability of belonging to the class, based on an output of the class output process; having determining the line type of the lane marking candidate based on the class rate calculated by the class rate calculation process; A road dividing line recognition program according to any one of aspects 2-9 to 2-13. [Point 2-15] the second line type candidate determination process determines the line type of the lane marking candidate based on a time series change in the class rate. A road dividing line recognition program according to aspect 2-14. [Point 2-16] The second line type candidate determination process includes: determining the line type of the lane marking candidate based on whether the class rate is equal to or greater than a class determination threshold; using different class determination thresholds for the lane marking candidate in the own lane and the lane marking candidate in an adjacent lane; A road dividing line recognition program according to aspect 2-14 or 2-15. [Point of View 2-17] The line type determination and selection process includes: A blocking rate calculation process is provided for calculating a blocking rate indicating a blocking state of the lane marking candidate by another vehicle, When the shielding rate is less than a selection determination threshold, the first line type candidate determination process or the line type candidate determination result is selected; When the blocking rate is equal to or greater than the selection determination threshold, the second line type candidate determination process or the result of the second line type candidate determination process is selected. A road division line recognition program according to any one of Aspects 2-14 to 2-16. [Point 3-1] A road dividing line recognition device (5) that recognizes road dividing lines from a captured image, a line type candidate determination unit (71) that executes a line type candidate determination, which is a determination of a line type of the lane line candidate, based on line information that is information on shape characteristics of the lane line candidate that is a candidate for a road lane line obtained from the captured image, The line type candidate determination unit an edge point selection unit (712) that selects edge points in the lane marking candidate in the captured image; a time series change acquisition unit (713) that acquires a time series change in the position of the edge point selected by the edge point selection unit; Equipped with the edge point selection unit selects a near edge point, which is the edge point closest to the host vehicle (C), and / or a far edge point, which is the edge point farthest from the host vehicle; the first line type candidate determination unit determines the line type of the lane marking candidate based on a time series change in the position of the near edge point and / or the far edge point acquired by the time series change acquisition unit. Road marking recognition device. [Point 3-2] The line type candidate determination unit determining the line type of the lane marking candidate based on a far-side determination result, which is a determination result of the line type based on a time series change in the positions of the far edge points, and a near-side determination result, which is a determination result of the line type based on a time series change in the positions of the near edge points; A road dividing line recognition device according to aspect 3-1. [Point 3-3] The line type candidate determination unit setting a weighting value for the far-side determination result based on a time-series change in the position of the far edge point; setting a weighting value for the near side determination result based on a time series change in the position of the near edge point; determining the line type of the lane marking candidate based on the weighted far-side determination result and the near-side determination result; A road dividing line recognition device according to aspect 3-2. [Point 3-4] the line type candidate determination unit sets a location for monitoring a time series change in the positions of the far edge point and the near edge point based on a paint length and a gap length which are the line information; The road lane line recognition device according to any one of Aspects 3-1 to 3-3. [Point 3-5] the line type candidate determination unit sets a location for monitoring a time series change in the positions of the far edge points and the near edge points based on a recognized position in the captured image of an object around the host vehicle; The road lane marking recognition device according to any one of Aspects 3-1 to 3-4. [Point 4-1] A road division line recognition program for recognizing a road division line from a captured image, comprising: a line type candidate determination process (71) for determining a line type of the lane line candidate based on line information, which is information on shape characteristics of the lane line candidate that is a candidate for a road lane line, obtained from the captured image; The line type candidate determination process includes: An edge point selection process (712) for selecting edge points on the lane marking candidate in the captured image; a time series change acquisition process (713) for acquiring a time series change in the position of the edge point selected by the edge point selection process; Including, The edge point selection process includes selecting a near edge point, which is the edge point closest to the host vehicle (C), and / or a far edge point, which is the edge point farthest from the host vehicle; the first line type candidate determination process determines the line type of the lane marking candidate based on a time series change in the positions of the near edge point and / or the far edge point acquired by the time series change acquisition process; Road marking recognition program. [Point 4-2] The line type candidate determination process includes: determining the line type of the lane marking candidate based on a far-side determination result, which is a determination result of the line type based on a time series change in the positions of the far edge points, and a near-side determination result, which is a determination result of the line type based on a time series change in the positions of the near edge points; A road dividing line recognition program according to aspect 4-1. [Point 4-3] The line type candidate determination process includes: setting a weighting value for the far-side determination result based on a time-series change in the position of the far edge point; setting a weighting value for the near side determination result based on a time series change in the position of the near edge point; determining the line type of the lane marking candidate based on the weighted far-side determination result and the near-side determination result; A road dividing line recognition program according to aspect 4-2. [Point 4-4] the line type candidate determination process sets a location for monitoring a time series change in the positions of the far edge point and the near edge point based on a paint length and a gap length, which are the line information; A road division line recognition program according to any one of Aspects 4-1 to 4-3. [Point 4-5] the line type candidate determination process sets a location for monitoring a time series change in the positions of the far edge point and the near edge point based on a recognized position in the captured image of an object around the host vehicle; A road division line recognition program according to any one of Aspects 4-1 to 4-4. [Point 5-1] A road dividing line recognition device (5) that recognizes road dividing lines from a captured image, a line type candidate determination unit (72) that executes a line type candidate determination, which is a determination of a line type of the lane line candidate, based on line information that is information on shape characteristics of the lane line candidate that is a candidate for a road lane line obtained from the captured image, The line type candidate determination unit a class output unit (722) that outputs a class determined by semantic segmentation for each pixel in the lane marking candidate in the captured image; A class rate calculation unit (723) that calculates a class rate, which is a probability that the class corresponds to the class, based on the output of the class output unit; Equipped with determining the line type of the lane marking candidate based on the class rate calculated by the class rate calculation unit; Road marking recognition device. [Point 5-2] the line type candidate determination unit determines the line type of the lane marking candidate based on a time-series change in the class rate. A road dividing line recognition device according to aspect 5-1. [Point 5-3] The line type candidate determination unit determining the line type of the lane marking candidate based on whether the class rate is equal to or greater than a class determination threshold; using different class determination thresholds for the lane marking candidate in the own lane and the lane marking candidate in an adjacent lane; A road dividing line recognition device according to aspect 5-1 or 5-2. [Point 5-4] The line type determination and selection unit A blocking rate calculation unit (89) is provided which calculates a blocking rate indicating a blocking state of the lane marking candidate by another vehicle, When the occlusion rate is equal to or greater than a selection determination threshold, it is determined that the line type candidate determination result of the line type candidate determination unit has a higher determination accuracy than a determination result by another determination algorithm. The road lane line recognition device according to any one of Aspects 5-1 to 5-3. [Point 6-1] A road division line recognition program for recognizing a road division line from a captured image, comprising: a line type candidate determination process for determining a line type of the lane marking candidate based on line information, which is information on shape characteristics of the lane marking candidate that is a candidate for a road lane marking, obtained from the captured image; The line type candidate determination process includes: a class output process for outputting a class determined by semantic segmentation for each pixel of the lane marking candidate in the captured image; A class rate calculation process for calculating a class rate, which is a probability of belonging to the class, based on an output of the class output process; Including, determining the line type of the lane marking candidate based on the class rate calculated by the class rate calculation process; Road marking recognition program. [Point 6-2] the line type candidate determination process determines the line type of the lane marking candidate based on a time-series change in the class rate; A road dividing line recognition program according to aspect 6-1. [Point 6-3] The line type candidate determination process includes: determining the line type of the lane marking candidate based on whether the class rate is equal to or greater than a class determination threshold; using different class determination thresholds for the lane marking candidate in the own lane and the lane marking candidate in an adjacent lane; A road dividing line recognition program according to aspect 6-1 or 6-2. [Point 6-4] The line type determination and selection process includes: A blocking rate calculation process is included, the blocking rate indicating a blocking state of the lane marking candidate by another vehicle is calculated, When the occlusion rate is equal to or greater than a selection determination threshold, it is determined that the line type candidate determination result of the line type candidate determination process has a higher determination accuracy than a determination result by another determination algorithm. A road dividing line recognition program according to any one of Aspects 6-1 to 6-3. [Explanation of symbols]

[0109] 1. In-vehicle systems 2 Camera 5. Image recognition device (road marking recognition device) 7 Line type candidate determination section 71 First additional candidate determination unit (first line type candidate determination unit) 72 Second additional candidate determination unit (second line type candidate determination unit) 8 Line type selection section 83 Scene Judgment Section 87 Scene Reliability Calculation Unit C Vehicle

Claims

1. A road dividing line recognition device (5) that recognizes road dividing lines from a captured image, a plurality of line type candidate determination units (7) for performing line type candidate determination, which is a determination of the line type of the lane marking candidate, based on line information, which is information regarding shape characteristics of the lane marking candidate that is a candidate for a road lane marking, obtained from the captured image; a line type determination selection unit (8) that performs line type determination as to which line type of the road-dividing line candidate corresponds to the lane-dividing line candidate by selecting one of the line type candidate determination units or one of the line type candidate determination results by each of the line type candidate determination units; Equipped with The plurality of line type candidate determination units have different determination algorithms. Road marking recognition device.

2. The line type determination and selection unit A scene determination unit (83) is provided for determining a scene establishment status corresponding to a shooting situation that affects the line type candidate determination, selecting the line type candidate determination unit or the line type candidate determination result according to a result of the determination of the scene establishment state by the scene determination unit; The road division line recognition device according to claim 1.

3. A weight value that is a weighting value for each of the line type candidate determination units is set for each of the different scenes, the line type determination selection unit selects the line type candidate determination unit or the line type candidate determination result based on the weighted value corresponding to the scene determined to be established by the scene determination unit. The road division line recognition device according to claim 2.

4. the line type determination selection unit, when there are a plurality of the line type candidate determination units or their determination results corresponding to the largest weighting value, performs the line type determination based on a result of comparing the determination results with each other; The road division line recognition device according to claim 3.

5. The line type determination and selection unit when the scene determination unit determines that a plurality of the scenes have been established, the weighted values ​​corresponding to the established scenes are added for each of the plurality of line type candidate determination units; selecting the line type candidate determination unit or the line type candidate determination result corresponding to the maximum value of the sum resulting from the addition; The road division line recognition device according to claim 3.

6. The line type determination and selection unit A scene reliability calculation unit (87) is further provided to calculate a reliability of the scene determination result by the scene determination unit, selecting the line type candidate determination unit or the line type candidate determination result based on the weighted value corrected by the reliability; The road division line recognition device according to claim 3.

7. The line type determination and selection unit A scene reliability calculation unit (87) is further provided to calculate a reliability of the scene determination result by the scene determination unit, selecting the line type candidate determination unit or the line type candidate determination result based on the reliability; The road division line recognition device according to claim 2.

8. the scene reliability calculation unit calculates the reliability based on a continuation state of a result of the scene determination by the scene determination unit; 8. The road division line recognition device according to claim 6 or 7.

9. A first line type candidate determination unit (71), which is one of the plurality of line type candidate determination units, an edge point selection unit (712) that selects edge points in the lane marking candidate in the captured image; a time series change acquisition unit (713) that acquires a time series change in the position of the edge point selected by the edge point selection unit; Equipped with the edge point selection unit selects a near edge point, which is the edge point closest to the host vehicle (C), and / or a far edge point, which is the edge point farthest from the host vehicle; the first line type candidate determination unit determines the line type of the lane marking candidate based on a time series change in the position of the near edge point and / or the far edge point acquired by the time series change acquisition unit. The road division line recognition device according to claim 1.

10. The first line type candidate determination unit determining the line type of the lane marking candidate based on a far-side determination result, which is a determination result of the line type based on a time series change in the positions of the far edge points, and a near-side determination result, which is a determination result of the line type based on a time series change in the positions of the near edge points; The road division line recognition device according to claim 9.

11. The first line type candidate determination unit setting a weighting value for the far-side determination result based on a time-series change in the position of the far edge point; setting a weighting value for the near side determination result based on a time series change in the position of the near edge point; determining the line type of the lane marking candidate based on the weighted far-side determination result and the near-side determination result; The road division line recognition device according to claim 10.

12. the first line type candidate determination unit sets a location for monitoring a time series change in the positions of the far edge point and the near edge point based on a paint length and a gap length which are the line information; The road division line recognition device according to claim 9.

13. the first line type candidate determination unit sets a location for monitoring a time-series change in the positions of the far edge point and the near edge point based on a recognized position in the captured image of an object around the host vehicle; The road division line recognition device according to claim 9.

14. A second line type candidate determination unit (72), which is one of the plurality of line type candidate determination units, A class output unit (722) that outputs a class determined by semantic segmentation for each pixel in the lane marking candidate in the captured image; A class rate calculation unit (723) that calculates a class rate, which is a probability of belonging to the class, based on an output of the class output unit; Equipped with determining the line type of the lane marking candidate based on the class rate calculated by the class rate calculation unit; The road division line recognition device according to claim 1 or 9.

15. the second line type candidate determination unit determines the line type of the lane marking candidate based on a time-series change in the class rate. The road division line recognition device according to claim 14.

16. The second line type candidate determination unit determining the line type of the lane marking candidate based on whether the class rate is equal to or greater than a class determination threshold; using different class determination thresholds for the lane marking candidate in the own lane and the lane marking candidate in an adjacent lane; The road division line recognition device according to claim 14.

17. A first line type candidate determination unit (71), which is one of the plurality of line type candidate determination units, an edge point selection unit (712) that selects edge points in the lane marking candidate in the captured image; a time series change acquisition unit (713) that acquires a time series change in the position of the edge point selected by the edge point selection unit; Equipped with the edge point selection unit selects a near edge point, which is the edge point closest to the host vehicle (C), and / or a far edge point, which is the edge point farthest from the host vehicle; the first line type candidate determination unit determines the line type of the lane marking candidate based on a time series change in the position of the near edge point and / or the far edge point acquired by the time series change acquisition unit; A second line type candidate determination unit (72), which is another one of the plurality of line type candidate determination units, A class output unit (722) that outputs a class determined by semantic segmentation for each pixel in the lane marking candidate in the captured image; A class rate calculation unit (723) that calculates a class rate, which is a probability of belonging to the class, based on an output of the class output unit; Equipped with determining the line type of the lane marking candidate based on the class rate calculated by the class rate calculation unit; The road division line recognition device according to claim 1.

18. The line type determination and selection unit A blocking rate calculation unit (89) is provided which calculates a blocking rate indicating a blocking state of the lane marking candidate by another vehicle, When the shielding rate is less than a selection determination threshold, the first line type candidate determination unit or the line type candidate determination result thereof is selected; When the shielding rate is equal to or greater than the selection determination threshold, the second line type candidate determination unit or the line type candidate determination result is selected. The road dividing line recognition device according to claim 17.

19. A road dividing line recognition program executed by a road dividing line recognition device (4) that recognizes road dividing lines from a captured image, comprising: The process executed by the road division line recognition device is as follows: a plurality of line type candidate determination processes each having a different determination algorithm, which perform line type candidate determination for determining a line type of a lane marking candidate based on line information, which is information regarding shape characteristics of a lane marking candidate that is a candidate for a road lane marking, obtained from the captured image; a line type determination selection process for selecting one of the line type candidate determination results obtained by the plurality of line type candidate determination processes, thereby executing a line type determination as to which line type of the road-dividing line the lane marking candidate corresponds to; Including, Road marking recognition program.

Images