External environment recognition system, vehicle control device, roadway recognition method, and program

Abstract

To provide an external field recognition system that can recognize an existence of a section line of the own vehicle even if the section line is not visible in a forward image.SOLUTION: A travel path recognition device 5 of an external field recognition system comprises: a class classification unit 50 that classifies the subject of a forward image of the own vehicle into a plurality of classes and generates an edge image; a parallel edge extraction unit 51 that extracts the own vehicle section line edge and a plurality of parallel edges for each class from the edge image; a characteristic ratio parameter calculating unit 52 which calculates a characteristic ratio parameter value that is correlated with a characteristic ratio between a comparison distance between a reference edge and a comparison edge along a lateral reference line extending along a width direction in the edge image, and an own vehicle section line distance between the reference edge and the own vehicle section line edge; and an own vehicle section line estimating unit 54 which estimates a position of the own vehicle section line at a current time instance based on positions of the reference edge and the comparison edge at the current time instance and a value of a characteristic ratio parameter at a time instance earlier than the current time instance.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to an external environment recognition system, a vehicle control device, a roadway recognition method, and a program. More particularly, the present invention relates to an external environment recognition system, a vehicle control device, a roadway recognition method, and a program that recognize a roadway based on an image ahead of a host vehicle. [Background technology]

[0002] In recent years, efforts to provide access to sustainable transport systems that take into consideration vulnerable transport participants have become more active. To achieve this, we are focusing on research and development into preventive safety technologies to further improve road safety and convenience.

[0003] For example, Patent Document 1 describes a preventive safety technology that recognizes white lines, road edges, and the like (hereinafter also referred to as "vehicle lane markings") that demarcate the vehicle's lane based on an image of the road ahead acquired by an on-board camera. When recognizing vehicle lane markings based on an image in this way, recognition accuracy may be reduced due to various factors, such as discontinuities in the road edge, backlight, blurred white lines, and the presence of foreign objects such as puddles. In response to this, the technology described in Patent Document 1 estimates the direction of the lane based on the direction of linear features included in an image acquired by an imaging unit and the direction of the edge of a surface area included in a distance image containing three-dimensional distance information generated based on the image acquired by the imaging unit or distance information detected by a distance detection unit. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2016-099650 Summary of the Invention [Problem to be solved by the invention]

[0005] The technology described in Patent Document 1 is useful when a portion of a vehicle's lane marking that should actually exist cannot be properly recognized due to backlighting, blurring, or other reasons, as described above. However, in places where no vehicle's lane marking exists, such as in the center of an intersection, the technology described in Patent Document 1 cannot recognize the vehicle's lane marking. In other words, the technology described in Patent Document 1 cannot recognize the vehicle's lane marking if no vehicle's lane marking is captured in the image acquired by the onboard camera.

[0006] The present invention aims to provide an external environment recognition system, a vehicle control device, a roadway recognition method, and a program that can recognize the presence of lane markings even when the lane markings are not visible in the forward image, and ultimately to contribute to the development of sustainable transportation systems. [Means for solving the problem]

[0007] (1) The external environment recognition system according to the present invention comprises a front image acquisition means for acquiring an image of the front side as seen from the host vehicle as a front image, and a roadway recognition means for recognizing the roadway of the host vehicle based on the front image, wherein the roadway recognition means performs a segmentation process on the front image to classify the objects in the front image into a plurality of classes including the host vehicle lane markings that divide the roadway, and generates an edge image by extracting edges of the classes; and a classification means for extracting, from the edge image, host vehicle lane marking edges that are edges of the host vehicle lane markings and a plurality of parallel edges that are parallel to the host vehicle lane marking edges when the field of view of the front image acquisition means is viewed in a plane, for each class, and selecting one of the plurality of parallel edges as a reference edge. a feature ratio parameter calculation means for calculating a value of a feature ratio parameter that is correlated with a feature ratio between a first distance between the reference edge and the comparison edge along a horizontal reference line that extends in the width direction in the edge image and a second distance between the reference edge and the vehicle lane marking edge; a storage means for storing the value of the feature ratio parameter calculated by the feature ratio parameter calculation means at a time prior to the current time; and a vehicle lane marking estimation means for estimating the position of the vehicle lane marking at the current time based on the positions of the reference edge and the comparison edge at the current time and the value of the feature ratio parameter stored in the storage means.

[0008] (2) In this case, it is preferable that the parallel edge extraction means extracts, as the parallel edges, edges that extend toward a vanishing area centered on a common vanishing point in the edge image.

[0009] (3) In this case, it is preferable that the parallel edge extraction means virtually divides the edges present in the edge image into a plurality of edge segments by one or more dividing lines parallel to the horizontal reference line, and extracts, from among the plurality of edges included in the edge image, those that have a plurality of edge segments extending toward the vanishing ranges defined for each divided section virtually divided by the dividing lines as the parallel edges.

[0010] (4) In this case, it is preferable that the feature ratio parameter calculation means calculates the value of the feature ratio parameter for each divided section, the storage means stores the value of the feature ratio parameter calculated by the feature ratio parameter calculation means for each divided section, and the vehicle lane marking line estimation means estimates the position of the vehicle lane marking line at the current time for each divided section based on the value of the feature ratio parameter stored for each divided section in the storage means.

[0011] (5) In this case, the roadway recognition means preferably further includes a reliability calculation means for calculating a reliability value for each comparison edge based on a comparison between the position of the vehicle lane marking edge extracted by the parallel edge extraction means and the position of the vehicle lane marking estimated by the vehicle lane marking estimation means, the storage means preferably stores the reliability value calculated by the reliability calculation means at a time prior to the current time, and the vehicle lane marking estimation means preferably estimates the position of the vehicle lane marking at the current time based on the positions of the reference edge and the comparison edge at the current time and the feature ratio parameter and the reliability value stored in the storage means.

[0012] (6) In this case, it is preferable that the vehicle lane marking estimation means estimates the position of the vehicle lane marking at the current time so that the value of the feature ratio at the current time is equal to the value of the feature ratio calculated based on the value of the feature ratio parameter stored in the memory means.

[0013] (7) In this case, it is preferable that the vehicle lane marking line estimation means includes a model estimation means that estimates the position of the vehicle lane marking line at the current time by using a vehicle lane marking line position estimation model that associates the positions of the reference edge and the comparison edge with the position of the vehicle lane marking line, and a model update means that updates the vehicle lane marking line position estimation model based on the value of the feature ratio parameter stored in the memory means.

[0014] (8) The vehicle control device of the present invention comprises the external environment recognition system and a control device that controls at least one of the steering mechanism, the driving device, and the braking device of the vehicle based on the recognition results by the external environment recognition system, and is characterized in that when the forward image does not show an object classified as the vehicle lane marking, the control device controls at least one of the steering mechanism, the driving device, and the braking device based on the position of the vehicle lane marking estimated by the vehicle lane marking estimation means.

[0015] (9) A roadway recognition method according to the present invention is a method for recognizing a roadway of a vehicle by a computer based on a forward image acquired by a forward image acquisition means for capturing an image of the roadway as viewed from the vehicle, the method comprising the steps of: applying a segmentation process to the forward image to classify the objects in the forward image into a plurality of classes including the vehicle lane markings that divide the roadway; and generating an edge image by extracting edges of the classes; and extracting, for each class, vehicle lane marking edge edges, which are edges of the vehicle lane markings, and a plurality of parallel edges that are parallel to the vehicle lane marking edges when the field of view of the forward image acquisition means is viewed in a plane; and extracting, from the plurality of parallel edges, a vehicle lane marking edge that is an edge of the vehicle lane marking edge, when the field of view of the forward image acquisition means is viewed in a plane. the step of selecting one of the parallel edges as a reference edge; the step of selecting the parallel edges other than the reference edge as comparison edges from among the plurality of parallel edges, and the step of calculating the value of a feature ratio parameter that is correlated with a feature ratio between a first distance between the reference edge and the comparison edge along a horizontal reference line extending in the width direction in the edge image and a second distance between the reference edge and the vehicle lane marking edge; the step of storing the value of the feature ratio parameter at a time prior to the current time in a storage means; and the step of estimating the position of the vehicle lane marking at the current time based on the positions of the reference edge and the comparison edge at the current time and the value of the feature ratio parameter stored in the storage means.

[0016] (10) A program according to the present invention causes a computer to recognize a roadway on which a vehicle is traveling based on a forward image acquired by a forward image acquisition means for capturing an image of the roadway ahead as viewed from the vehicle, and includes the steps of: causing the computer to perform segmentation processing on the forward image to classify objects in the forward image into a plurality of classes including the vehicle lane markings that divide the roadway, and generating an edge image in which edges of the classes are extracted; and extracting, for each class, vehicle lane marking edge edges, which are edges of the vehicle lane markings, and a plurality of parallel edges that are parallel to the vehicle lane marking edges when the field of view of the forward image acquisition means is viewed in a plane, from the edge image. selecting one of the parallel edges as a reference edge; selecting the parallel edges other than the reference edge as comparison edges from among the plurality of parallel edges; calculating the value of a feature ratio parameter that is correlated with a feature ratio between a first distance between the reference edge and the comparison edge along a horizontal reference line extending in the width direction in the edge image and a second distance between the reference edge and the vehicle lane marking edge; storing the value of the feature ratio parameter at a time prior to the current time in a storage means; and estimating the position of the vehicle lane marking at the current time based on the positions of the reference edge and the comparison edge at the current time and the value of the feature ratio parameter stored in the storage means. [Effects of the Invention]

[0017] (1) In the present invention, the classification means performs segmentation processing on a forward image of the host vehicle to classify objects in the forward image into multiple classes (including the host vehicle's lane markings) and generate an edge image by extracting edges for each class. The parallel edge extraction means extracts the host vehicle's lane marking edge and multiple parallel edges that are parallel to the host vehicle's lane marking edge in a planar view from the edge image and selects one of the multiple parallel edges as a reference edge. The feature ratio parameter calculation means calculates a feature ratio parameter value that correlates with a feature ratio between a first distance (the distance between the reference edge and the comparison edge) and a second distance (the distance between the reference edge and the host vehicle's lane marking edge) along the lateral reference line. The storage means stores the feature ratio parameter value calculated by the feature ratio parameter calculation means at a time prior to the current time. As will be described later with reference to FIG. 5, a geometric relationship is established such that the feature ratio, which is the ratio between the first distance and the second distance, is constant regardless of the position of the lateral reference line in the depth direction on the forward image. Therefore, the vehicle lane marking estimation means utilizes the geometric relationship of such feature ratios to estimate the position of the vehicle lane marking at the current time based on the positions of the reference edge and comparison edge at the current time and the value of the feature ratio parameter at a time prior to the current time that is stored in the storage means. Therefore, according to the present invention, even if the vehicle lane marking is not visible in the forward image, its presence can be recognized, thereby contributing to the development of a sustainable transportation system.

[0018] (2) The parallel edge extraction means extracts, as parallel edges, edges that extend toward a vanishing area centered on a common vanishing point in the edge image, thereby easily extracting multiple parallel edges.

[0019] (3) As will be explained later with reference to Figure 9, when the road on which the vehicle is traveling curves to the left or right, the lane marking edges and parallel edges parallel to them in a planar view appear as curves in the edge image, making it more difficult to extract parallel edges than when the road is straight. Therefore, the parallel edge extraction means virtually divides the edges in the edge image into multiple edge segments by one or more lane markings parallel to the horizontal reference line, and extracts, from the multiple edges included in the edge image, those that have multiple edge segments extending toward a vanishing range defined for each virtual divided lane marked by the lane markings as parallel edges. Therefore, according to the present invention, multiple parallel edges can be easily extracted even when the road is curved.

[0020] (4) The feature ratio parameter calculation means calculates the value of the feature ratio parameter for each divided section, the storage means stores the value of the feature ratio parameter calculated by the feature ratio parameter calculation means for each divided section, and the vehicle lane marking estimation means estimates the position of the vehicle lane marking at the current time for each divided section based on the value of the feature ratio parameter stored for each divided section in the storage means. Thus, according to the present invention, the position of the vehicle lane marking can be easily estimated even when the road is curved.

[0021] (5) The feature ratio and the feature ratio parameter value change depending on the position of a comparison edge selected from among multiple parallel edges. Therefore, if the reliability of a comparison edge is low, the reliability of the vehicle lane marking position estimated based on the position of that comparison edge also decreases. Therefore, the reliability calculation means calculates a reliability value for each comparison edge based on a comparison between the position of the vehicle lane marking edge extracted by the parallel edge extraction means and the position of the vehicle lane marking estimated by the vehicle lane marking estimation means. The storage means stores the reliability values ​​calculated by the reliability calculation means at times prior to the current time. The vehicle lane marking estimation means estimates the vehicle lane marking position at the current time based on the positions of the reference edge and comparison edge at the current time and the feature ratio parameter and reliability values ​​stored in the storage means. Therefore, according to the present invention, the vehicle lane marking position can be estimated taking into account the reliability of the comparison edge.

[0022] (6) The vehicle lane marking estimation means estimates the position of the vehicle lane marking at the current time so that the value of the feature ratio at the current time is equal to the value of the feature ratio calculated based on the value of the feature ratio parameter stored in the storage means. Therefore, according to the present invention, the position of the vehicle lane marking can be estimated by a simple calculation in the vehicle lane marking estimation means.

[0023] (7) The model estimation means estimates the position of the vehicle lane marking at the current time by using a vehicle lane marking position estimation model that associates the positions of the reference edge and comparison edge with the position of the vehicle lane marking, and the model update means updates the vehicle lane marking position estimation model based on the value of the feature ratio parameter stored in the storage means. Therefore, according to this invention, the input / output relationship of the vehicle lane marking position estimation model can be updated in response to changes in the configuration of the road on which the vehicle is traveling (such as the number of lanes and lane width), making it possible to appropriately estimate the position of the vehicle lane marking in response to changes in the configuration of the road.

[0024] (8) A vehicle control device according to the present invention includes an external environment recognition system as described above and a control device that controls at least one of the steering mechanism, driving device, and braking device of the vehicle based on the recognition results of the external environment recognition system. Furthermore, if an object classified as a vehicle lane marking is not captured in the forward image, the control device controls at least one of the steering mechanism, driving device, and braking device based on the position of the vehicle lane marking estimated by the vehicle lane marking estimation means of the external environment recognition system. Therefore, according to the present invention, control by the control device can continue even if the vehicle lane marking disappears while the vehicle is traveling.

[0025] (9) The roadway recognition method according to the present invention has the same effects as the external environment recognition system described above.

[0026] (10) The program according to the present invention has the same effects as the external environment recognition system described above. [Brief explanation of the drawings]

[0027] [Figure 1] 1 is a diagram schematically illustrating a configuration of a vehicle equipped with a vehicle control device and an external environment recognition system according to a first embodiment of the present invention. [Figure 2] FIG. 2 is a functional block diagram of the roadway recognition device. [Figure 3A] 10 is an example of a diagram in which multiple classes extracted from a certain forward image are color-coded. [Figure 3B] 10 is an example of an edge image generated from the same forward image. [Figure 4A] This is a diagram obtained by converting each pixel constituting the edge image shown in FIG. 3B into a line in parameter space. [Figure 4B] FIG. 4B is a diagram highlighting a portion of a line extracted from the parameter space shown in FIG. 4A. [Figure 5] 10 is a diagram illustrating the vehicle lane marking edge, the reference edge, and the comparison edge extracted by the parallel edge extraction unit superimposed on the forward image. FIG. [Figure 6] FIG. 10 is a diagram illustrating an example of time-series data of feature ratio parameter values ​​stored in a storage unit. [Figure 7] 10 is a flowchart showing a specific procedure of a roadway recognition process. [Figure 8] FIG. 4 is a functional block diagram of a roadway recognition device of an external environment recognition system according to a second embodiment of the present invention. [Figure 9] FIG. 10 is a diagram illustrating an example of an edge image generated by processing by a classifying unit. [Figure 10] FIG. 10 is a diagram showing four divided sections obtained by dividing the edge image shown in FIG. 9 by four division lines. [Figure 11] 10A and 10B are diagrams for explaining a procedure for extracting a plurality of parallel edges in a parallel edge extraction unit. [Figure 12] 11 is a diagram showing an example of the results of evaluation of a plurality of edge line segments included in the four divided sections shown in FIG. 10 by a parallel edge extraction unit. FIG. [Figure 13] FIG. 10 is a diagram showing time-series data of feature ratio parameter values ​​stored in a storage unit. [Figure 14]10A and 10B are diagrams illustrating a procedure for extracting a plurality of parallel edges from an edge image by a parallel edge extraction unit according to a modified example. [Figure 15] FIG. 10 is a functional block diagram of a roadway recognition device of an external environment recognition system according to a third embodiment of the present invention. [Figure 16] FIG. 10 is a functional block diagram of a roadway recognition device of an external environment recognition system according to a fourth embodiment of the present invention. [Figure 17] 10A and 10B are diagrams for explaining a procedure for calculating a reliability value for a comparison edge in a reliability calculation unit. [Figure 18] 10 is a diagram for explaining a procedure for estimating the position of the vehicle lane marking based on a reliability value in the vehicle lane marking estimation unit. FIG. DETAILED DESCRIPTION OF THE INVENTION

[0028] First Embodiment Hereinafter, a vehicle control device and an external environment recognition system according to a first embodiment of the present invention will be described with reference to the drawings.

[0029] FIG. 1 is a diagram schematically illustrating the configuration of a vehicle V equipped with a vehicle control device 1 and an external environment recognition system 3 according to this embodiment. The upper part of FIG. 1 shows a plan view of the vehicle V, and the lower part of FIG. 1 shows a side view. Note that the following description will be given assuming that the vehicle V is a so-called right-hand drive four-wheel vehicle in which the driver's seat is located on the right side in the vehicle width direction as viewed along the direction of travel, but the present invention is not limited to this. The vehicle V may also be a so-called left-hand drive four-wheel vehicle in which the driver's seat is located on the left side in the vehicle width direction as viewed along the direction of travel.

[0030] The vehicle V is equipped with an electric power steering device 9 as a steering mechanism that steers the left and right front wheels Wf, a power plant 8 as a driving device that generates a driving force to rotate the front wheels Wf, which are the driving wheels of the vehicle V, a braking device 7 that generates a braking force to stop the rotation of the front wheels Wf and rear wheels Wr, and a vehicle control device 1 that controls the electric power steering device 9, the power plant 8, and the braking device 7.

[0031] The electric power steering device 9 includes a gearbox 93 that connects a pinion shaft 92 extending from a steering wheel 91 that receives steering operations by the driver to the left and right front wheels Wf, an electric motor 94 provided in the gearbox 93, and a steering sensor 95 that detects the steering angle of the steering wheel 91.

[0032] The gearbox 93 includes a rack shaft extending along the vehicle width direction and meshing with the pinion shaft 92, tie rods connecting both ends of the rack shaft to the left and right front wheels Wf, and converts the rotational movement of the steering wheel 91 caused by the driver's steering operation into movement along the vehicle width direction, thereby steering the left and right front wheels Wf in the direction of travel. The electric motor 94 rotates in response to a control signal output from the vehicle control device 1, and generates a driving force to assist the driver's steering operation or to automatically steer the front wheels Wf without the driver's steering operation. The steering sensor 95 detects the steering angle of the steering wheel 91 and sends a signal corresponding to the detected value to the vehicle control device 1.

[0033] The power plant 8 is a driving force generating source that generates a driving force for rotating the front wheels Wf to move the vehicle V forward or backward along the traveling direction in response to acceleration / deceleration operation of an accelerator pedal (not shown) by the driver and control signals output from the vehicle control device 1. In the following, a case will be described in which the power plant 8 is a drive motor that generates driving force by consuming electric power supplied from a high-voltage battery, a fuel cell stack, or the like (not shown), but the present invention is not limited to this. The power plant 8 may also be an engine that generates driving force by consuming fuel stored in a fuel tank (not shown), or a transmission that changes the speed of the engine output and transmits it to the front wheels Wf.

[0034] The braking device 7 includes a disc brake device that generates braking force to slow down or stop the rotation of each wheel Wf, Wr by tightening discs provided on the axles of each wheel Wf, Wr mainly when driving, based on braking operation of the brake pedal (not shown) by the driver or control signals output from the vehicle control device 1, and a parking brake that generates braking force to keep the rotation of each wheel Wr, Wf stopped mainly when parking.

[0035] The vehicle control device 1 includes an external environment recognition system 3 that recognizes the external situation of the vehicle V by acquiring external environment information about the surroundings of the vehicle V (such as the positions, distances, relative speeds, and traveling directions of obstacles and other vehicles, and the road on which the vehicle V is traveling), and a control device 2 that controls at least one of an electric power steering device 9, a power plant 8, and a braking device 7 based on the recognition results of the external environment recognition system 3. Note that, hereinafter, only the configuration of the external environment recognition system 3 that has the above-described external environment recognition function, mainly related to the function of recognizing the road on which the vehicle V is traveling (hereinafter also referred to as the "roadway recognition function") will be described in detail.

[0036] The external environment recognition system 3 includes an in-vehicle front camera 4 that captures an image of the front side as seen from the vehicle V as a forward image, and a roadway recognition device 5 that recognizes the roadway the vehicle V is traveling on based on the forward image captured by the in-vehicle front camera 4.

[0037] The in-vehicle front camera 4 is a camera that captures images of the area ahead in the traveling direction of the vehicle V. The in-vehicle front camera 4 is attached, for example, to a position on the inside of the cabin of the roof of the vehicle V, closer to the front window. The forward images captured by the in-vehicle front camera 4 are sequentially transmitted to the roadway recognition device 5.

[0038] While the vehicle V is traveling, the roadway recognition device 5 performs the following calculations on the forward images successively acquired by the on-board front camera 4 to recognize the positions of the roadway of the vehicle V, i.e., the left and right vehicle lane marks on both the left and right sides of the roadway along the vehicle width direction (hereinafter, these are also collectively referred to as "vehicle lane marks"). The roadway recognition device 5 transmits information regarding the recognition results of the vehicle lane marks to the control device 2.

[0039] The roadway recognition device 5 is a computer configured with hardware such as a processing means such as a CPU, auxiliary storage means such as an HDD or SSD that stores a program that causes the processing means to execute the roadway recognition processing described below, and a main storage means such as a RAM for storing data that is temporarily required for the processing means to execute the program.

[0040] 2 is a functional block diagram of the roadway recognition device 5. The roadway recognition device 5 has the above-described hardware configuration, which includes a class classification unit 50, a parallel edge extraction unit 51, a feature ratio parameter calculation unit 52, a storage unit 53, and a vehicle lane marking estimation unit 54.

[0041] The classifying unit 50 performs a known segmentation process (for example, semantic segmentation process) on the forward image acquired by the in-vehicle front camera 4 to classify the objects in the forward image into multiple classes including the vehicle's lane markings, and generates an edge image by extracting the edges of each class. The classifying unit 50 transmits the generated edge image and information about the multiple classes included in this edge image to the parallel edge extracting unit 51.

[0042] Figure 3A is an example of a diagram in which multiple classes extracted from a forward image by processing in the class classification unit 50 are color-coded, and Figure 3B is an example of an edge image generated from the same forward image by processing in the class classification unit 50.

[0043] As shown in Fig. 3A, the classification unit 50 can classify objects shown in the forward image into a plurality of classes, such as the vehicle's lane markings, adjacent lane markings, median strips, white lines, curbs, sidewalks, fences, houses, utility poles, buildings, plants, automobiles, bicycles, pedestrians, and structures. As shown in Fig. 3B, the classification unit 50 can generate an edge image made up of edges of each class included in the forward image. Hereinafter, the edges of the classes classified as the vehicle's lane markings by the classification unit 50 are also referred to as "vehicle's lane marking edges."

[0044] 2, the parallel edge extraction unit 51 extracts, from the edge image generated by the classification unit 50, edges of the class classified as the host vehicle lane marking by the classification unit 50 (hereinafter also referred to as "host vehicle lane marking edge") and multiple edges (hereinafter also referred to as "parallel edges") that are parallel to the host vehicle lane marking edge when the field of view of the in-vehicle front camera 4 is viewed in a plane, for each class extracted from the forward image by the classification unit 50. In other words, the parallel edge extraction unit 51 extracts, in the edge image, multiple edges that extend toward a vanishing range centered on the vanishing point common to the host vehicle lane marking edge as parallel edges.

[0045] The following describes the procedure for extracting a plurality of parallel edges that are parallel to the vehicle lane marking edge in a plan view from the edge image in the parallel edge extraction unit 51, with reference to FIGS. 4A and 4B.

[0046] First, the parallel edge extraction unit 51 performs a known Hough transform process on the edge image generated by the classification unit 50 to extract straight line elements from among the multiple edges that make up the edge image.

[0047] Figure 4A is a diagram obtained by converting each pixel constituting the edge image shown in Figure 3B into a line in a parameter space with the horizontal axis representing the angle (θ) and the vertical axis representing the radius (ρ). In the Hough transform process, each point in the image space represented by coordinates (x, y) is projected onto a sinusoidal line in the parameter space as shown in Figure 4A. Therefore, a straight line element in the image space appears in the parameter space as a point where many lines overlap (in the example of Figure 4A, these are points with high brightness, indicated by square points 41A to 48A).

[0048] FIG. 4B is a diagram highlighting some of the straight lines extracted from the parameter space shown in FIG. 4A. In FIG. 4B, only eight straight lines are shown in bold. In the example of FIG. 4B, eight points 41A, 42A, 43A, 44A, 45A, 46A, 47A, and 48A in the parameter space are shown as lines 41B, 42B, 43B, 44B, 45B, 46B, 47B, and 48B, respectively. In FIG. 4B, line 41B is the left vehicle lane marking, and line 42B is the right vehicle lane marking. In the following description referring to FIGS. 4A and 4B, for ease of understanding, the edge image shown in FIG. 3B has been trimmed below line 3L so that the vehicle lane marks 41B and 42B appear as straight lines in the edge image.

[0049] As shown in Figure 4B, of the eight lines 41B-48B, four lines 41B-44B are parallel to the vehicle's lane markings 41B and 42B in a planar view, and four lines 45B-48B are non-parallel to the vehicle's lane markings 41B and 42B in a planar view. As shown in Figure 4B, the four lines 41B-44B that are parallel to the vehicle's lane marking edges in a planar view extend toward a vanishing point common to the vehicle's lane marking edges in the edge image. Therefore, the four points 41A-44A in the parameter space that represent these four lines 41B-44B appear on a single sine wave (see line 49A in Figure 4A) that passes through two points that represent the vehicle's lane marking edges in the parameter space. On the other hand, in plan view, four points 45A to 48A in the parameter space that represent lines 45B to 48B that are not parallel to the vehicle's lane marking edge appear at positions away from line 49A, which is the sine wave, in the parameter space.

[0050] Because the Hough transform has the above-described properties, the parallel edge extraction unit 51 extracts, from among the multiple edges constituting the edge image, edges whose peaks appear within a predetermined width centered on a single sine wave (see line 49A in FIG. 4A) that passes through two peaks representing the vehicle lane marking edge on the Hough plane as parallel edges. This allows the parallel edge extraction unit 51 to extract, from the edge image, multiple edges that extend toward a vanishing range centered on a vanishing point common to the vehicle lane marking edge as parallel edges.

[0051] Returning to FIG. 2, the parallel edge extraction unit 51 extracts the vehicle lane marking edge and multiple parallel edges relative to the vehicle lane marking edge from the edge image using the procedure described above, and then selects one of the multiple extracted parallel edges as a reference edge. More specifically, the parallel edge extraction unit 51 preferably selects, from the multiple parallel edges, a parallel edge of a class that is recognized by the classification unit 50 continuously for as long as possible while the vehicle V is traveling, such as a median strip or curb, as the reference edge. In the following, the parallel edges other than the reference edge among the multiple parallel edges extracted by the parallel edge extraction unit 51 are referred to as comparison edges.

[0052] The parallel edge extraction unit 51 also transmits information (hereinafter also referred to as “edge position information”) regarding the position, orientation, length of the line segments, etc. on the edge image of the vehicle lane marking edge, reference edge, and multiple comparison edges extracted by the above-mentioned procedure to the feature ratio parameter calculation unit 52.

[0053] Based on the edge position information of the vehicle lane marking edge, the reference edge, and the multiple comparison edges extracted by the parallel edge extraction unit 51, the feature ratio parameter calculation unit 52 calculates the value of the feature ratio defined as follows between the vehicle lane marking edge, the reference edge, and the multiple comparison edges.

[0054] Figure 5 is a diagram showing the left vehicle lane marking edge LL, the right vehicle lane marking edge LR, the reference edge LB, the first comparison edge L1, the second comparison edge L2, and the third comparison edge L3 extracted by the parallel edge extraction unit 51, superimposed on the forward image from which these edges were extracted.

[0055] Figure 5 shows a case where vehicle V is traveling in the second lane from the left in an area where driving on the left is permitted. Figure 5 also shows a case where the edge of the curb that exists to the left of vehicle V from the left vehicle lane marking edge LL is set as reference edge LB, the edge of the buildings that line the sidewalk that extends to the left of reference edge LB is set as first comparison edge L1, the edge of the marking line that marks the adjacent lane to the right of vehicle V's path is set as second comparison edge L2, and the edge formed by a line of one or more vehicles traveling side by side in the adjacent lane to the right is set as third comparison edge L3.

[0056] 5, the feature ratio parameter calculation unit 52 virtually sets a horizontal reference line VL extending in the width direction below the common vanishing point P of the edges LL, LR, LB, and L1-L3 in the forward image, and calculates the distance along the horizontal reference line VL from the reference edge LB to the host lane marking edges LL and LB and the comparison edges L1-L3. More specifically, the feature ratio parameter calculation unit 52 calculates the distance along the horizontal reference line VL between the reference edge LB and the host lane marking edges LL and LR (hereinafter referred to as the "host lane marking distance"), the distance along the horizontal reference line VL between the reference edge LB and the comparison edges L1, L2, and L3 (hereinafter referred to as the "comparison distance"), and a feature ratio between these comparison distances and the host lane marking distance (= host lane marking distance / comparison distance). Note that in the present invention, the feature ratio defined above and parameters correlated with this feature ratio (i.e., the combination of the host lane marking distance and the comparison distance) are referred to as feature ratio parameters.

[0057] 5, a horizontal reference line VL' parallel to the horizontal reference line VL is set between the horizontal reference line VL and the vanishing point P, that is, at a position farther from the vehicle V than the horizontal reference line VL in the depth direction in the forward image. In this case, a similarity relationship is established between a triangle having the vanishing point P as a vertex and the intersection of the multiple edges LL, LR, LB, L1 to L3 with the horizontal reference line VL, and a triangle having the vanishing point P as a vertex and the intersection of the multiple edges LL, LR, LB, L1 to L3 with the horizontal reference line VL'. Therefore, for example, the feature ratio (D1 / D2) between the comparison distance ("D2" in FIG. 5) along the horizontal reference line VL between the reference edge LB and the third comparison edge L3 and the host vehicle lane line distance ("D1" in FIG. 5) along the horizontal reference line VL between the reference edge LB and the left host vehicle lane line LL is equal to the feature ratio (D1' / D2') between the comparison distance ("D2'" in FIG. 5) along the horizontal reference line VL' between the reference edge LB and the third comparison edge L3 and the host vehicle lane line distance ("D1'" in FIG. 5) along the horizontal reference line VL' between the reference edge LB and the left host vehicle lane line LL. Note that a similar relationship also holds for the feature ratio defined for the right host vehicle lane line LB. In other words, a geometric relationship holds such that the feature ratio, which is the ratio of the comparison distance to the host vehicle lane line distance defined as above, is constant regardless of the position of the horizontal reference line in the depth direction on the forward image. This means that even if the presence of the vehicle lane markings cannot be recognized from a forward image acquired at a specific time, the position of the vehicle lane markings at that specific time can be estimated from the positions of the reference edge and at least one comparison edge at that specific time and the value of the feature ratio at a time prior to that specific time.

[0058] 2 , the storage unit 53 stores time-series data of the values ​​of the feature ratio parameters (i.e., the feature ratio, the combination of the comparison distance and the vehicle lane marking distance, etc.) calculated for each forward image through the above-described procedures by the classifying unit 50, the parallel edge extracting unit 51, and the feature ratio parameter calculating unit 52. This allows the vehicle lane marking estimation unit 54, which will be described later, to obtain the values ​​of the feature ratio parameters at times prior to a specific time by reading them from the storage unit 53.

[0059] The vehicle lane marking estimation unit 54 estimates the position of the vehicle lane marking at the current time based on the position information of the reference edge and comparison edge extracted by the parallel edge extraction unit 51 at the current time and the value of the feature ratio parameter at a time earlier than the current time stored in the memory unit 53. More specifically, the vehicle lane marking estimation unit 54 estimates the position of the vehicle lane marking at the current time so that the feature ratio value at the current time is equal to the feature ratio value at a time earlier than the current time stored in the memory unit 53. In other words, if the distance along the horizontal reference line between the reference edge and the comparison edge at the current time is "a" and the feature ratio value at a time earlier than the current time is "r", the vehicle lane marking at the current time is estimated to be located at a position a distance "a × r" along the horizontal reference line from the reference edge at the current time.

[0060] Fig. 6 is a diagram showing an example of time-series data of feature ratio parameter values ​​stored in storage unit 53. Fig. 6 shows an example of time-series data of feature ratio parameter values ​​in a case where multiple edges LL, LR, LB, L1 to L3 described with reference to Fig. 5 have been recognized by parallel edge extraction unit 51. Fig. 6 also shows a case where, assuming that the current time is "n," feature ratio parameter values ​​at five points in time prior to the current time, namely "n-1," "n-2," "n-3," "n-4," and "n-5," are stored in storage unit 53. Fig. 6 also shows a case where the distance from a reference edge to each of edges LL, LR, LB, L1 to L3 is stored as a feature ratio parameter. Therefore, for example, if the second comparison edge at time "n-5" is used, the feature ratio for the left vehicle lane marking edge at time "n-5" will be "D_LL(n-5) / D_L2(n-5)", and the feature ratio for the right vehicle lane marking edge at time "n-5" will be "D_LR(n-5) / D_L2(n-5)".

[0061] Figure 6 also shows a case where, at time "n-2," the first comparison edge can no longer be recognized, and at the current time "n," the left vehicle lane marking edge, right vehicle lane marking edge, first comparison edge, and third comparison edge cannot be recognized.

[0062] In the example shown in Figure 6, the vehicle lane marking estimation unit 54 can estimate the position of the left vehicle lane marking edge and the right vehicle lane marking edge based on the distance "D_L2(n-1)" along the horizontal reference line between the reference edge and the second comparison edge at the current time "n" and the values ​​of the feature ratio parameter at times "n-1," "n-2," "n-3," "n-4," and "n-5" prior to the current time stored in the memory unit 53. More specifically, when the vehicle lane marking line estimation unit 54 uses the value of the feature ratio parameter at time "n-1," for example, it can estimate that the left vehicle lane marking line edge is located at a position "D_L2(n)×D_LL(n-1) / D_L2(n-1)" away from the reference edge at the current time along the horizontal reference line, and that the right vehicle lane marking line edge is located at a position "D_L2(n)×D_LR(n-1) / D_L2(n-1)" away from the reference edge at the current time along the horizontal reference line. Furthermore, when using the value of the feature ratio parameter at time "n-5," for example, the vehicle lane marking estimation unit 54 can estimate that the left vehicle lane marking edge is located along the horizontal reference line at a distance "D_L2(n)×D_LL(n-5) / D_L2(n-5)" from the reference edge at the current time, and the right vehicle lane marking edge is located along the horizontal reference line at a distance "D_L2(n)×D_LR(n-5) / D_L2(n-5)" from the reference edge at the current time. The vehicle lane marking estimation unit 54 may also estimate the position of the vehicle lane marking by averaging the distances estimated based on the values ​​of the feature ratio parameter at times "n-1," "n-2," "n-3," "n-4," and "n-5."

[0063] 6 illustrates a case in which feature ratio parameter calculation unit 52 sets one horizontal reference line for one forward image (or edge image), calculates the vehicle lane marking distance and comparison distance along this horizontal reference line as feature ratio parameters, and stores these in storage unit 53. However, the present invention is not limited to this. Feature ratio parameter calculation unit 52 may set multiple horizontal reference lines for one forward image (or edge image), and calculate the average of the vehicle lane marking distance and the average of the comparison distance along these multiple horizontal reference lines as feature ratio parameters, and store these in storage unit 53.

[0064] Fig. 7 is a flowchart showing a specific procedure of a roadway recognition process for recognizing the roadway of the vehicle V by the external environment recognition system 3 described above. The roadway recognition process shown in Fig. 7 is repeatedly executed in the roadway recognition device 5 under a predetermined control cycle while the vehicle V is traveling.

[0065] First, in step ST1, the roadway recognition device 5 acquires an image of the front of the vehicle V captured by the in-vehicle front camera 4, and the image is displayed in step ST2.

[0066] Next, in step ST2, the roadway recognition device 5 performs segmentation processing on the forward image acquired in step ST1 to classify the subjects in the forward image into multiple classes including the vehicle's lane markings, and generates an edge image by extracting the edges of each class, and then proceeds to step ST3.

[0067] Next, in step ST3, the roadway recognition device 5 extracts the vehicle lane marking edge and multiple parallel edges that are parallel to the vehicle lane marking edge in a planar view from the edge image generated in step ST2, for each class, and then proceeds to step ST4.

[0068] Next, in step ST4, the roadway recognition device 5 selects one of the parallel edges extracted in step ST3 as a reference edge, and proceeds to step ST5. Note that, hereinafter, the parallel edges other than the reference edge are referred to as comparison edges.

[0069] Next, in step ST5, the roadway recognition device 5 uses the position information of the vehicle lane marking edge, the reference edge, and multiple comparison edges extracted in steps ST3 to ST4 to calculate the value of a feature ratio parameter for the vehicle lane marking edge (i.e., the comparison distance along the horizontal reference line (distance between the reference edge and the comparison edge) and the vehicle lane marking distance (distance between the reference edge and the vehicle lane marking edge), or the feature ratio between these comparison distances and the vehicle lane marking distance), and then proceeds to step ST6.

[0070] In step ST6, the roadway recognition device 5 stores the value of the feature ratio parameter calculated in step ST5 in the storage unit 53, and then proceeds to step ST7.

[0071] In step ST7, the roadway recognition device 5 determines whether or not an object classified as the vehicle's lane marking is captured in the forward image acquired in the current control cycle (i.e., the current time), in other words, whether or not the vehicle's lane marking edge was able to be extracted from the edge image in step ST3 executed in the current control cycle.

[0072] If the determination result in step ST7 is YES, that is, if the lane marking can be recognized from the forward image acquired in the current control cycle, the lane recognition device 5 proceeds to step ST8. In step ST8, the lane recognition device 5 recognizes the position of the lane marking based on the position information of the lane marking edge extracted from the edge image in step ST3 executed in the current control cycle, and ends the lane recognition process in FIG.

[0073] On the other hand, if the determination result in step ST7 is NO, that is, if the lane recognition device 5 was unable to recognize the vehicle lane marking from the forward image acquired in the current control cycle, the lane recognition device 5 proceeds to step ST9. In step ST9, the lane recognition device 5 estimates the position of the vehicle lane marking based on the position information of the reference edge and comparison edge extracted from the edge image in steps ST3 and ST4 executed in the current control cycle and the value of the feature ratio parameter stored in the memory unit 53 in step ST6 executed in the control cycle before the previous one, and ends the lane recognition process of FIG.

[0074] 1, the control device 2 is a computer configured with hardware such as a processing means such as a CPU, auxiliary storage means such as an HDD or SSD storing various programs, and main storage means such as a RAM for storing data temporarily required for the processing means to execute the programs. The control device 2 controls at least one of the electric power steering device 9, the power plant 8, and the braking device 7 based on the recognition results of the external environment recognition system 3 regarding the road on which the vehicle V is traveling.

[0075] More specifically, when a subject classified as a vehicle lane marking is captured in the forward image, i.e., when the judgment result of the processing of step ST7 in Figure 7 is YES, the control device 2 controls at least one of the electric power steering device 9, the power plant 8, and the braking device 7 based on the position information of the vehicle lane marking extracted by the parallel edge extraction unit 51 (i.e., the recognition result by the processing of step ST8).

[0076] Furthermore, if the forward image does not show any subject classified as the vehicle's lane marking, i.e., if the judgment result of the processing of step ST7 in Figure 7 is NO, the control device 2 controls at least one of the electric power steering device 9, the power plant 8, and the braking device 7 based on the position information of the vehicle's lane marking estimated by the vehicle's lane marking estimation unit 54 (i.e., the estimation result by the processing of step ST9).

[0077] The external environment recognition system 3 and the vehicle control device 1 according to this embodiment have the following advantages. (1) The classification unit 50 performs segmentation processing on the forward image of the vehicle V to classify the objects in the forward image into multiple classes (including the vehicle's lane markings) and generate an edge image by extracting the edges of each class. The parallel edge extraction unit 51 extracts the vehicle's lane marking edge and multiple parallel edges that are parallel to the vehicle's lane marking edge in a planar view from the edge image and selects one of the multiple parallel edges as a reference edge. The feature ratio parameter calculation unit 52 calculates the value of a feature ratio parameter that correlates with the feature ratio between the comparison distance along the horizontal reference line (the distance between the reference edge and the comparison edge) and the vehicle's lane marking distance (the distance between the reference edge and the vehicle's lane marking edge). The memory unit 53 stores the value of the feature ratio parameter calculated by the feature ratio parameter calculation unit 52 at a time earlier than the current time. 5, and estimates the position of the vehicle lane marking at the current time based on the position information of the reference edge and comparison edge at the current time and the value of the feature ratio parameter at a time earlier than the current time that is stored in the memory unit 53. Therefore, the external environment recognition system 3 can recognize the presence of the vehicle lane marking even when it is not visible in the forward image, thereby contributing to the development of a sustainable transportation system.

[0078] (2) The parallel edge extraction unit 51 extracts, as parallel edges, edges that extend toward a vanishing area centered on a common vanishing point in the edge image, thereby easily extracting multiple parallel edges.

[0079] (3) The vehicle lane marking estimation unit 54 estimates the position of the vehicle lane marking at the current time so that the value of the feature ratio at the current time is equal to the value of the feature ratio calculated based on the value of the feature ratio parameter stored in the memory unit 53. Therefore, according to the external environment recognition system 3, the position of the vehicle lane marking can be estimated by a simple calculation in the vehicle lane marking estimation unit 54.

[0080] (4) The vehicle control device 1 includes the external environment recognition system 3 as described above, and a control device 2 that controls at least one of the electric power steering device 9, the power plant 8, and the braking device 7 of the vehicle V based on the recognition results of the external environment recognition system 3. Furthermore, when a subject classified as a vehicle lane marking is not captured in the forward image, the control device 2 controls at least one of the electric power steering device 9, the power plant 8, and the braking device 7 based on the position of the vehicle lane marking estimated by the vehicle lane marking estimation unit 54 of the external environment recognition system 3. Therefore, the vehicle control device 1 allows the control device 2 to continue control even if the vehicle lane marking no longer exists while the vehicle is traveling.

[0081] Second Embodiment Next, an external environment recognition system according to a second embodiment of the present invention will be described with reference to the drawings. In the following description, the same components as those in the external environment recognition system 3 according to the first embodiment will be denoted by the same reference numerals, and detailed description thereof will be omitted.

[0082] FIG. 8 is a functional block diagram of a roadway recognition device 5A of an external environment recognition system 3A according to this embodiment.

[0083] FIG. 9 is a diagram showing an example of an edge image generated by processing by the classifying unit 50 of the roadway recognition device 5A according to this embodiment. Note that FIG. 9 shows objects recognized as different classes by the classifying unit 50 in different colors. Below, a procedure for estimating the position of a vehicle lane marking by the roadway recognition device 5A will be described using an example in which the roadway curves to the left as shown in FIG. 9A. Note that the following illustrates a case in which the classifying unit 50 classifies class 9a as a left vehicle lane marking, class 9b as a right vehicle lane marking (or a median strip), class 9c as a sidewalk extending along the roadway, class 9d as a fence extending along the sidewalk, class 9e as a roadway marking, class 9f as a sidewalk extending along the roadway, class 9g as a vacant lot, class 9h as a house, and class 9i as a utility pole.

[0084] In an edge image such as that shown in FIG. 9, when the field of view of the vehicle-mounted front camera is viewed in a plan view, the edges of classes 9a and 9b, which are the vehicle's lane markings, are parallel to the edges of the other classes 9c to 9g. However, although the edges of classes 9a to 9g are parallel in a plan view, they curve to the left as the vehicle moves forward in the direction of travel. For this reason, using the processing described in the first embodiment, it is difficult to extract the edges of classes 9c to 9g as parallel edges that are parallel to the vehicle's lane markings. In contrast, the parallel edge extraction unit 51A according to this embodiment processes the edge image according to the procedure described below, thereby extracting multiple parallel edges that are parallel to the curved vehicle's lane markings in a plan view, as shown in FIG. 9.

[0085] First, the parallel edge extraction unit 51A virtually divides the edge image generated by the classification unit 50 and the edges present in this edge image into a plurality of division sections and edge line segments using one or more division lines parallel to the horizontal reference line that is set when the feature ratio parameter calculation unit 52A calculates the value of the feature ratio parameter.

[0086] 10 is a diagram showing four divided sections, a first divided section 101, a second divided section 102, a third divided section 103, and a fourth divided section 104, obtained by dividing the edge image shown in FIG. 9 by four divided sections: a first divided section LD1, a second divided section LD2, a third divided section LD3, and a fourth divided section LD4. The following describes a case in which the parallel edge extraction unit 51A sets the spacing between the four divided sections LD1 to LD4 irregularly as shown in FIG. 10, but the present invention is not limited to this. The number of divided sections may be one or more. The spacing between the multiple divided sections may be equal, or may be set to decrease as the distance from the vehicle V increases in the depth direction.

[0087] Next, the parallel edge extraction unit 51A extracts as parallel edges those edges included in the original edge image that have a predetermined number (for example, "3" in the example shown in Figure 12 described below) or more of edge segments extending toward the lost areas 101P, 102P, 103P, and 104P defined for each of the divided sections 101 to 104 divided as described above.

[0088] 11 is a diagram illustrating a procedure for extracting multiple parallel edges in the parallel edge extraction unit 51A. For ease of understanding, only the fourth divided section 104 and the third divided section 103 are illustrated in FIG. 11. In FIG. 11, edge line segments of class 9i classified as utility poles are indicated by symbols Ea and Eh, edge line segments of class 9h classified as houses are indicated by symbol Eb, edge line segments of class 9c classified as sidewalks are indicated by symbol Ec, edge line segments of class 9a classified as left vehicle lane markings are indicated by symbol Ed, edge line segments of class 9b classified as right vehicle lane markings are indicated by symbol Ee, edge line segments of class 9e classified as opposite lane markings are indicated by symbol Ef, edge line segments of class 9f classified as sidewalks are indicated by symbol Eg, and edge line segments of class 9g classified as vacant lots are indicated by symbol Ei.

[0089] 11, in the fourth divided section 104, three edge line segments Ea, Eh, and Ei do not extend toward the lost area 104P. Furthermore, in this fourth divided section 104, six edge line segments Eb, Ec, Ed, Ee, Ef, and Eg extend toward the lost area 104P. On the other hand, the five edge line segments Ea, Eb, Eg, Eh, and Ei that existed in the fourth divided section 104 do not exist in the second divided section 102. Furthermore, in this second divided section 102, four edge line segments Ec, Ed, Ee, and Ef extend toward the lost area 102P.

[0090] Fig. 12 is a diagram showing an example of the results of evaluation by the parallel edge extraction unit 51A of multiple edge segments included in the four divided sections 101 to 104 as shown in Fig. 10. In Fig. 12, edge segments that do not exist in the target divided section are marked with an "x" mark, edge segments that do not extend into the vanishing range defined for each divided section are marked with a "△" mark, and edge segments that extend into the vanishing range defined for each divided section are marked with a "◯" mark.

[0091] As shown in Fig. 12, edge Ec has four edge line segments extending toward the disappearance area defined for each divided section, edge Ef has three edge line segments extending toward the disappearance area, edges Eb and Eg have one edge line segment extending toward the disappearance area, and edges Ea, Eh, and Ei have no edge line segments extending toward the disappearance area. Therefore, in the example shown in Fig. 12, when the setting number is set to "3," the parallel edge extraction unit 51A extracts two edges Ec and Ef as parallel edges. Note that the procedure for extracting edge line segments extending from each divided section 101 to 104 toward its respective disappearance area is the same as in the first embodiment, so a detailed description will be omitted.

[0092] After extracting multiple parallel edges from the edge image using the above procedure, the parallel edge extraction unit 51A selects one of the multiple parallel edges as a reference edge using the same procedure as in the first embodiment, and sets the others as comparison edges.

[0093] Returning to FIG. 8, the feature ratio parameter calculation unit 52A uses the position information of the vehicle lane marking edge, the reference edge, and the comparison edge extracted by the parallel edge extraction unit 51A in the above-described procedure to calculate the value of the feature ratio parameter defined in the same manner as in the first embodiment for each divided section.

[0094] 11, for example, the ratio of the distance along the horizontal reference line between edges Ec and Ed to the distance along the horizontal reference line between edges Ed and Ee in the fourth divided section 104 is different from the ratio of the distance along the horizontal reference line between edges Ec and Ed to the distance along the horizontal reference line between edges Ed and Ee in the second divided section 102. In other words, when the road is curved, the geometric relationship for the feature ratio described in the first embodiment generally holds within a common divided section, but does not hold between different divided sections. Therefore, the feature ratio parameter calculation unit 52A calculates the value of the feature ratio parameter for each divided section as described above.

[0095] The storage unit 53A stores, for each divided section, time-series data of the values ​​of the feature ratio parameters calculated for each forward image by the classifying unit 50, the parallel edge extracting unit 51A, and the feature ratio parameter calculating unit 52A according to the above-described procedures, as shown in FIG. 13.

[0096] The vehicle lane marking estimation unit 54A estimates the position of the vehicle lane marking at the current time for each divided section based on the value of the feature ratio parameter stored for each divided section in the memory unit 53A. Note that the procedure for estimating the position of the vehicle lane marking in each divided section is the same as in the first embodiment, and therefore a detailed description thereof will be omitted.

[0097] In addition to the effects (1) to (4) above, the external environment recognition system 3A according to this embodiment has the following effects. (5) The parallel edge extraction unit 51A virtually divides the edges in the edge image into multiple edge segments by one or more lane markings parallel to the horizontal reference line, and extracts, from the multiple edges included in the edge image, those that have a predetermined number or more of edge segments extending toward a vanishing range defined for each virtual lane marking. Thus, the external environment recognition system 3A can easily extract multiple parallel edges even when the road is curved.

[0098] (6) The feature ratio parameter calculation unit 52A calculates the value of the feature ratio parameter for each divided section, the memory unit 53A stores the value of the feature ratio parameter calculated by the feature ratio parameter calculation unit 52A for each divided section, and the vehicle lane marking estimation unit 54A estimates the position of the vehicle lane marking at the current time for each divided section based on the value of the feature ratio parameter stored for each divided section in the memory unit 53A. Therefore, the external environment recognition system 3A can easily estimate the position of the vehicle lane marking even when the road is curved.

[0099] In the above second embodiment, the parallel edge extraction unit 51A extracts a plurality of parallel edges from the edge image by virtually dividing the edge image by one or more division lines, but the present invention is not limited to this.

[0100] 14 is a diagram showing a procedure for extracting multiple parallel edges from an edge image by a parallel edge extraction unit according to a modified example. As will be described below, the algorithm for extracting multiple parallel edges from an edge image by a parallel edge extraction unit according to a modified example corresponds to an algorithm in which the intervals between the multiple dividing lines determined by the parallel edge extraction unit 51A according to the second embodiment are shortened to the minimum.

[0101] The parallel edge extraction unit according to the modified example expresses the widthwise position x of multiple edges included in the edge image as a function of the longitudinal position y. Note that FIG. 14 illustrates only two edges Ec and Ed. In the following, the functions of these edges Ec and Ed are referred to as fc(y) and fd(y), respectively. In this case, as shown by the solid lines in FIG. 14, the slopes of the tangents of the edges Ec and Ed at the longitudinal position y equal to h are expressed by the differential values ​​fc′(h) and fd′(h) of the functions fc(y) and fd(y). Furthermore, as shown in FIG. 14, the tangents of multiple edges that are parallel to each other in planar view intersect within a common vanishing range Py, which differs for each longitudinal position y. Therefore, when multiple edges included in the edge image are expressed by the above-described functions, by evaluating these functions and their differential values, it is possible to continuously evaluate along the longitudinal direction whether a target edge is parallel to the vehicle lane marking edge in planar view. Therefore, the parallel edge extraction unit according to the modified example extracts multiple parallel edges that are parallel to the vehicle lane markings in a planar view by evaluating functions of multiple edges included in the edge image and their differential values, etc.

[0102] Third Embodiment Next, an external environment recognition system according to a third embodiment of the present invention will be described with reference to the drawings. In the following description, the same components as those in the external environment recognition system 3 according to the first embodiment will be denoted by the same reference numerals, and detailed description thereof will be omitted.

[0103] 15 is a functional block diagram of a roadway recognition device 5B of an external environment recognition system 3B according to this embodiment. The roadway recognition device 5B according to this embodiment differs from the roadway recognition device 5 according to the first embodiment in the configuration of a vehicle lane marking estimation unit 54B.

[0104] The vehicle lane marking estimation unit 54B includes a model estimation unit 541 and a model update unit 542, and estimates the position of the vehicle lane marking by using these units.

[0105] The model estimation unit 541 estimates the position of the host lane marking at the current time by using a host lane marking position estimation model that associates the position information of the reference edge and the comparison edge with the position information of the host lane marking. More specifically, the model estimation unit 541 estimates the position of the host lane marking at the current time by using, as the host lane marking position estimation model, a regression model that uses the position information of the reference edge and the comparison edge at the current time as an explanatory variable and the position information of the host lane marking at the current time as a target variable, or a neural network that uses the position information of the reference edge and the comparison edge at the current time as an input and outputs the position information of the host lane marking at the current time.

[0106] The model update unit 542 updates the vehicle lane marking line position estimation model of the model estimation unit 541 based on the value of the feature ratio parameter at a time earlier than the current time that is stored in the memory unit 53. More specifically, the model update unit 542 updates the vehicle lane marking line position estimation model by using, as learning data, the time-series data of the feature ratio parameter values ​​that is stored in the memory unit 53 by the parallel edge extraction unit 51 and that is calculated while all of the vehicle lane marking line edges, reference edges, and comparison edges are being extracted.

[0107] In addition to the effects (1) to (4) above, the external environment recognition system 3B according to this embodiment has the following effects. (7) The model estimation unit 541 estimates the position of the vehicle lane marking at the current time by using a vehicle lane marking position estimation model that associates the position information of the reference edge and the comparison edge with the position information of the vehicle lane marking, and the model update unit 542 updates the vehicle lane marking position estimation model based on the value of the feature ratio parameter stored in the storage unit 53. Therefore, the external environment recognition system 3B can update the input / output relationship of the vehicle lane marking position estimation model in response to changes in the configuration of the road on which the vehicle is traveling (such as the number of lanes and lane width), and therefore can appropriately estimate the position of the vehicle lane marking in response to changes in the configuration of the road.

[0108] <Fourth embodiment> Next, an external environment recognition system according to a fourth embodiment of the present invention will be described with reference to the drawings. In the following description, the same components as those in the external environment recognition system 3 according to the first embodiment will be denoted by the same reference numerals, and detailed description thereof will be omitted.

[0109] 16 is a functional block diagram of a roadway recognition device 5C of an external environment recognition system 3C according to this embodiment. The roadway recognition device 5C according to this embodiment differs from the roadway recognition device 5 according to the first embodiment in that it further includes a reliability calculation unit 55 and in the configurations of a storage unit 53C and a vehicle lane marking estimation unit 54C.

[0110] As described above, the value of the feature ratio, which is defined by the ratio between the comparison distance and the vehicle lane marking distance, varies depending on the position of the comparison edge. Therefore, if the reliability of the comparison edge extracted by the parallel edge extraction unit 51 is low, the reliability of the feature ratio value calculated based on the position information of this comparison edge also decreases, and the reliability of the vehicle lane marking estimated based on this feature ratio value also decreases. Therefore, the reliability calculation unit 55 calculates the reliability value for the comparison edge extracted by the parallel edge extraction unit 51 using the procedure described below.

[0111] FIG. 17 is a diagram illustrating the procedure for calculating the reliability value for the comparison edge in the reliability calculation unit 55. FIG. 17 schematically illustrates the widthwise positions of multiple edges extracted by the parallel edge extraction unit 51 at times "n," "n-1," ..., "n-5," where "n" represents the current time. In FIG. 17, the widthwise positions of the reference edge, left vehicle lane marking edge, right vehicle lane marking edge, first comparison edge, and second comparison edge are indicated by circles, starting from the left. FIG. 17 also illustrates a case in which the parallel edge extraction unit 51 extracts the reference edge, left vehicle lane marking edge, right vehicle lane marking edge, and second comparison edge at approximately the same positions at all times, but extracts the first comparison edge at time "n" at a position slightly closer to the second comparison edge than at past times "n-1" to "n-5."

[0112] Here, the position of the right vehicle lane marking edge at time "n" is compared between when it is estimated based on the position information of the first comparison edge at time "n" and when it is estimated based on the position information of the second comparison edge at time "n".

[0113] As shown by the square marks in FIG. 17, when the position of the right host vehicle lane marking edge at time "n" is determined using the position information of the second comparison edge at time "n" and the feature ratio parameter values ​​at times "n-1" to "n-5," the host vehicle lane marking estimation unit 54 can estimate the right host vehicle lane marking edge to be in a generally appropriate position. In contrast, as shown by the triangle marks in FIG. 17, when the position of the right host vehicle lane marking edge at time "n" is determined using the position information of the first comparison edge at time "n" and the feature ratio parameter values ​​at times "n-1" to "n-5," the host vehicle lane marking estimation unit 54 cannot estimate the right host vehicle lane marking edge to be in an appropriate position. Therefore, in the example shown in FIG. 17, the first comparison edge can be said to have a lower reliability than the second comparison edge.

[0114] Therefore, the reliability calculation unit 55 calculates a reliability value for each comparison edge based on a comparison between the position of the vehicle lane marking edge extracted by the parallel edge extraction unit 51 and the position of the vehicle lane marking edge estimated by the vehicle lane marking estimation unit 54. More specifically, the reliability calculation unit 55 calculates the reliability value of the target comparison edge so that the value decreases as the position of the vehicle lane marking edge estimated by the vehicle lane marking estimation unit 54 based on the position information of the target comparison edge becomes farther from the position of the vehicle lane marking edge extracted by the parallel edge extraction unit 51. Note that the reliability calculation unit 55 preferably sets the maximum value to "1" and the minimum value to "0" and calculates the reliability value of each comparison edge between these maximum and minimum values. Furthermore, in this case, the reliability calculation unit 55 preferably calculates the reliability value for each comparison edge so that the sum of the reliability values ​​of all comparison edges recognized at that time becomes the maximum value.

[0115] 16 , the memory unit 53C stores time-series data of the feature ratio parameter values ​​calculated for each forward image by the classifying unit 50, the parallel edge extracting unit 51, and the feature ratio parameter calculating unit 52, and time-series data of the reliability value calculated for each comparison edge by the reliability calculating unit 55. This allows the vehicle lane marking estimation unit 54C, which will be described later, to obtain the feature ratio parameter and reliability values ​​at times prior to a specific time by reading them from the memory unit 53C.

[0116] The vehicle lane marking estimation unit 54C estimates the position of the vehicle lane marking at the current time based on the position information of the reference edge and comparison edge extracted by the parallel edge extraction unit 51 at the current time and the feature ratio parameter and reliability values ​​at a time prior to the current time stored in the memory unit 53C.

[0117] FIG. 18 is a diagram illustrating the procedure for estimating the position of the vehicle lane marking line in the vehicle lane marking line estimation unit 54C based on the reliability value. FIG. 18 schematically illustrates the widthwise positions of multiple edges extracted by the parallel edge extraction unit 51 at times “n,” “n-1,” …, “n-5,” where “n” represents the current time. In FIG. 18, the widthwise positions of the reference edge, left vehicle lane marking line edge, right vehicle lane marking line edge, first comparison edge, and second comparison edge are indicated by circles, starting from the left. FIG. 18 also illustrates a case in which the parallel edge extraction unit 51 was unable to extract the position of the right vehicle lane marking line edge at time “n,” and therefore the position of the right vehicle lane marking line edge is estimated by the vehicle lane marking line estimation unit 54C. In the following description, the reliability a1 for the first comparison edge at time “n-1” is set to “0.75,” and the reliability a2 for the second comparison edge at time “n-1” is set to “0.75.”

[0118] In such a case, the vehicle lane marking estimation unit 54C estimates the position of the right vehicle lane marking edge by weighting, based on the reliability value for each comparison edge, the position of the right vehicle lane marking (see the triangle mark in FIG. 18) calculated based on the position information of the first comparison edge at time "n" and the value of the feature ratio parameter between times "n-1" and "n-5" and the position of the right vehicle lane marking (see the square mark in FIG. 18) calculated based on the position information of the second comparison edge at time "n" and the value of the feature ratio parameter between times "n-1" and "n-5." That is, in the example shown in FIG. 18, the vehicle lane marking estimation unit 54C estimates that the right vehicle lane marking edge is located at a position that divides the space between the square marking and the triangle marking at a ratio of 1:3 (see the black circle mark in FIG. 18).

[0119] As described above, the vehicle lane marking estimation unit 54C estimates the position of the vehicle lane marking by averaging the positions of the vehicle lane marking estimated by a procedure similar to that of the first embodiment based on the position information of each comparison edge at time "n" with a weight corresponding to the reliability value for each comparison edge.

[0120] In addition to the effects (1) to (4) above, the external environment recognition system 3C according to this embodiment has the following effects. (8) The reliability calculation unit 55 calculates a reliability value for each comparison edge based on a comparison between the position of the host lane marking edge extracted by the parallel edge extraction unit 51 and the position of the host lane marking estimated by the host lane marking estimation unit 54C. The memory unit 53C stores the reliability values ​​calculated by the reliability calculation unit 55 at times prior to the current time. The host lane marking estimation unit 54C estimates the position of the host lane marking at the current time based on the position information of the reference edge and comparison edge at the current time and the feature ratio parameter and reliability values ​​stored in the memory unit 53C. Therefore, the external environment recognition system 3C can estimate the position of the host lane marking while taking the reliability of the comparison edge into consideration.

[0121] Although one embodiment of the present invention has been described above, the present invention is not limited to this, and the detailed configuration may be modified as appropriate within the scope of the spirit of the present invention. [Explanation of symbols]

[0122] V...Vehicle 1...Vehicle control device 2...Control device 3,3A,3B,3C…External world recognition system 4...In-vehicle front camera (means for acquiring forward images) 5,5A,5B,5C...Traveling route recognition device (travelling route recognition means) 50...Classification unit (classification means) 51, 51A...Parallel edge extraction unit (parallel edge extraction means) 52, 52A...Feature ratio parameter calculation unit (feature ratio parameter calculation means) 53,53A,53C...Storage unit (storage means) 54, 54A, 54B, 54C... Vehicle lane line estimation unit (vehicle lane line estimation means) 541...Model estimation unit (model estimation means) 542...Model update unit (model update means) 55... Reliability calculation unit (reliability calculation means) 7...Brake device 8...Power plant (driving device) 9...Electric power steering device (steering mechanism)

Claims

1. a forward image acquisition means for acquiring an image of the forward side as viewed from the host vehicle as a forward image; a travel path recognition means for recognizing a travel path of the host vehicle based on the forward image, The roadway recognition means a classification means for performing a segmentation process on the forward image to classify the objects in the forward image into a plurality of classes including the vehicle lane markings that divide the road, and for generating an edge image by extracting edges of the classes; a parallel edge extraction means for extracting, for each class, from the edge image, a vehicle lane marking edge that is an edge of the vehicle lane marking, and a plurality of parallel edges that are parallel to the vehicle lane marking edge when the field of view of the forward image acquisition means is viewed in a plane, and selecting one of the plurality of parallel edges as a reference edge; a feature ratio parameter calculation means for calculating a value of a feature ratio parameter that is correlated with a feature ratio between a first distance between the reference edge and the comparison edge along a horizontal reference line that extends in the width direction of the edge image and a second distance between the reference edge and the vehicle lane marking edge, and a storage means for storing the value of the feature ratio parameter calculated by the feature ratio parameter calculation means at a time earlier than the current time; an own vehicle lane marking estimation means for estimating the position of the own vehicle lane marking at the current time based on the positions of the reference edge and the comparison edge at the current time and the value of the feature ratio parameter stored in the storage means.

2. 2. The external environment recognition system according to claim 1, wherein the parallel edge extraction means extracts, as the parallel edges, edges that extend toward a vanishing area centered on a common vanishing point in the edge image.

3. The parallel edge extraction means virtually dividing an edge present in the edge image into a plurality of edge segments by one or more division lines parallel to the horizontal reference line; The external environment recognition system according to claim 2, characterized in that, of the plurality of edges included in the edge image, edges having a plurality of edge line segments extending toward the loss range defined for each division section virtually divided by the division lines are extracted as the parallel edges.

4. the feature ratio parameter calculation means calculates a value of the feature ratio parameter for each of the divided sections; the storage means stores the value of the feature ratio parameter calculated by the feature ratio parameter calculation means for each of the divided sections; The external environment recognition system according to claim 3, characterized in that the vehicle lane marking estimation means estimates the position of the vehicle lane marking at the current time for each divided section based on the value of the feature ratio parameter stored for each divided section in the storage means.

5. the lane recognition means further includes a reliability calculation means for calculating a reliability value for each comparison edge based on a comparison between the position of the vehicle lane marking edge extracted by the parallel edge extraction means and the position of the vehicle lane marking estimated by the vehicle lane marking estimation means, the storage means stores the reliability value calculated by the reliability calculation means at a time earlier than the current time; 2. The external environment recognition system according to claim 1, wherein the vehicle lane marking estimation means estimates the position of the vehicle lane marking at the current time based on the positions of the reference edge and the comparison edge at the current time and the feature ratio parameter and the reliability value stored in the storage means.

6. 6. The external environment recognition system according to claim 1, wherein the vehicle lane marking estimation means estimates the position of the vehicle lane marking at the current time so that the value of the feature ratio at the current time is equal to the value of the feature ratio calculated based on the value of the feature ratio parameter stored in the storage means.

7. The vehicle lane marking estimation means a model estimation means for estimating the position of the vehicle lane marking at the current time by using a vehicle lane marking position estimation model that associates the positions of the reference edge and the comparison edge with the position of the vehicle lane marking; 6. The external environment recognition system according to claim 1, further comprising: a model updating means for updating the vehicle lane marking line position estimation model based on the value of the feature ratio parameter stored in the storage means.

8. An external environment recognition system according to any one of claims 1 to 5; A vehicle control device comprising: a control device that controls at least one of a steering mechanism, a traveling drive device, and a braking device of the vehicle based on a recognition result by the external environment recognition system, The vehicle control device is characterized in that, when the forward image does not show an object classified as the vehicle lane marking, the control device controls at least one of the steering mechanism, the driving drive device, and the braking device based on the position of the vehicle lane marking estimated by the vehicle lane marking estimation means.

9. A travel path recognition method for recognizing a travel path of a vehicle by a computer based on a forward image acquired by a forward image acquisition means that captures an image of a forward side as viewed from the vehicle, the method comprising: performing a segmentation process on the forward image to classify objects in the forward image into a plurality of classes including lane markings that demarcate the roadway, and generating an edge image by extracting edges of the classes; extracting, for each class, from the edge image, a vehicle lane marking edge that is an edge of the vehicle lane marking, and a plurality of parallel edges that are parallel to the vehicle lane marking edge when the field of view of the forward image acquisition means is viewed in a plane, and selecting one of the plurality of parallel edges as a reference edge; a step of setting the parallel edges other than the reference edge as comparison edges among the plurality of parallel edges, and calculating a value of a feature ratio parameter correlated with a feature ratio between a first distance between the reference edge and the comparison edge along a lateral reference line extending in the width direction in the edge image and a second distance between the reference edge and the vehicle lane marking edge; storing in a storage means the value of the feature ratio parameter at a time earlier than the current time; and estimating the position of the vehicle lane marking at the current time based on the positions of the reference edge and the comparison edge at the current time and the value of the feature ratio parameter stored in the storage means.

10. A program for causing a computer to recognize a traveling path of a vehicle based on a forward image acquired by a forward image acquisition means that captures an image of a forward side as viewed from the vehicle, The computer, performing a segmentation process on the forward image to classify objects in the forward image into a plurality of classes including lane markings that demarcate the roadway, and generating an edge image by extracting edges of the classes; extracting, for each class, from the edge image, a vehicle lane marking edge that is an edge of the vehicle lane marking, and a plurality of parallel edges that are parallel to the vehicle lane marking edge when the field of view of the forward image acquisition means is viewed in a plane, and selecting one of the plurality of parallel edges as a reference edge; selecting one of the parallel edges as a reference edge; a step of setting the parallel edges other than the reference edge as comparison edges among the plurality of parallel edges, and calculating a value of a feature ratio parameter correlated with a feature ratio between a first distance between the reference edge and the comparison edge along a lateral reference line extending in the width direction in the edge image and a second distance between the reference edge and the vehicle lane marking edge; storing in a storage means the value of the feature ratio parameter at a time earlier than the current time; and estimating the position of the vehicle lane marking at the current time based on the positions of the reference edge and the comparison edge at the current time and the value of the feature ratio parameter stored in the storage means.