Estimation device, estimation method, and estimation program

Abstract

To provide an estimation device capable of accurately estimating three-dimensional coordinates in front of a lane, and to provide an estimation method and an estimation program therefor.SOLUTION: An estimation device acquires a runway image as an image of a runway on which a vehicle imaged by an imaging device installed in the vehicle travels; extracts lane boundaries based on the runway image; selects two points at which inclinations of the corresponding two points on the right and left lane boundaries on the runway image satisfies a predetermined condition; and derives at least one of three-dimensional coordinates of the lane boundaries corresponding to the selected two points, and the three-dimensional coordinates of a lane center, based on coordinates on the runway image at each of the selected two points.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present disclosure relates to an estimation device, an estimation method, and an estimation program. [Background technology]

[0002] Patent Document 1 discloses a technique for extracting left and right lane boundaries from an image of the area ahead of the vehicle, determining a vanishing line from the lane boundaries, and calculating a bank angle at a position ahead of the vehicle from the vanishing line. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-096427 Summary of the Invention [Problem to be solved by the invention]

[0004] Driver assistance systems and automated driving systems require a function for performing steering control so that the vehicle travels along a driving lane detected by a sensor, and the three-dimensional coordinates of the vehicle ahead in the driving lane are used as the target value for steering control. The technology described in Patent Document 1 can estimate the bank angle and the distance in the depth direction of the image, but cannot estimate the three-dimensional coordinates of the vehicle ahead in the driving lane. Furthermore, the technology described in Patent Document 1 cannot estimate the bank angle and the distance in the depth direction of the image in scenes where the vanishing line cannot be observed because the distant lane boundary is obscured by a preceding vehicle, a side wall, or the like.

[0005] The present disclosure has been made in consideration of the above circumstances, and aims to provide an estimation device, an estimation method, and an estimation program that can accurately estimate the three-dimensional coordinates of the driving lane ahead. [Means for solving the problem]

[0006] The estimation device of the first aspect includes an acquisition unit that acquires a road image, which is an image of the road on which the vehicle is traveling, captured by a camera mounted on the vehicle; an extraction unit that extracts lane boundaries based on the road image; a selection unit that selects two points on the road image where the slopes of two corresponding points on the boundary between the left and right lanes satisfy a predetermined condition; and a derivation unit that derives at least one of the three-dimensional coordinates of the lane boundary corresponding to the two selected points and the three-dimensional coordinates of the center of the lane based on the coordinates on the road image of the two points selected by the selection unit.

[0007] According to the estimation device of the first aspect, it is possible to accurately estimate the three-dimensional coordinates of the driving lane ahead.

[0008] In the second aspect of the estimation device, in the estimation device of the first aspect, when the radius of curvature of the road is equal to or greater than a threshold, the selection unit selects two corresponding points on the boundary of the lane in the road image, which have the same coordinates on the vertical axis of the road image.

[0009] According to the estimation device of the second aspect, the three-dimensional coordinates of the driving lane ahead can be estimated with high accuracy by simple processing.

[0010] The estimation device of the third aspect is the estimation device of the first aspect, wherein when the radius of curvature of the road is less than a threshold, the selection unit selects two points whose slopes at two corresponding points on the boundary of the lane in the road image correspond to a ratio obtained by dividing the radius of curvature by the height of the imaging device.

[0011] According to the estimation device of the third aspect, it is possible to estimate the three-dimensional coordinates of the driving lane ahead with higher accuracy.

[0012] In the estimation device of the fourth aspect, in the estimation device of the first aspect, when the absolute value of the yaw rate while traveling on the road is equal to or greater than a threshold value, the selection unit selects two points on the boundary of the lane in the road image whose slopes correspond to a ratio obtained by dividing the vehicle speed by the height and yaw rate of the imaging device.

[0013] According to the estimation device of the fourth aspect, it is possible to estimate the three-dimensional coordinates of the driving lane ahead with higher accuracy.

[0014] The estimation device of the fifth aspect is an estimation device of any one of the first to fourth aspects, further including a correction unit that estimates second three-dimensional coordinates corresponding to one of the two points at a second point in time that is later than the first point in time based on first three-dimensional coordinates derived by the derivation unit corresponding to one of the two points selected by the selection unit from the road image captured at a first point in time and measurement values ​​by a sensor mounted on the vehicle, derives a correction amount for the coordinates on the road image of one of the two points based on the difference between the first three-dimensional coordinates and the second three-dimensional coordinates at the second point in time, and corrects the coordinates on the road image of the one of the two points using the correction amount, and the derivation unit derives at least one of the three-dimensional coordinates of the lane boundary and the three-dimensional coordinates of the lane center corresponding to the two points based on the coordinates of the two points on the road image after correction by the correction unit.

[0015] According to the estimation device of the fifth aspect, it is possible to estimate the three-dimensional coordinates of the driving lane ahead with higher accuracy.

[0016] The estimation method of the sixth aspect involves a computer performing a process in which it acquires a road image, which is an image of the road on which the vehicle is traveling, photographed by a photographing device mounted on the vehicle, extracts lane boundaries based on the road image, selects two points on the boundary between the left and right lanes in the road image whose slopes satisfy a predetermined condition, and derives at least one of the three-dimensional coordinates of the lane boundary corresponding to the two selected points and the three-dimensional coordinates of the center of the lane based on the coordinates of the two selected points on the road image.

[0017] The estimation program of the seventh aspect causes a computer to execute a process of acquiring a road image, which is an image of the road on which the vehicle is traveling, photographed by a photographing device mounted on the vehicle, extracting lane boundaries based on the road image, selecting two points on the boundary between the left and right lanes in the road image whose slopes satisfy a predetermined condition, and deriving at least one of the three-dimensional coordinates of the lane boundary corresponding to the two selected points and the three-dimensional coordinates of the center of the lane based on the coordinates of the two selected points on the road image. [Effects of the Invention]

[0018] According to the present disclosure, it is possible to accurately estimate the three-dimensional coordinates of the driving lane ahead. [Brief explanation of the drawings]

[0019] [Figure 1] 1 is a schematic side view showing an example of a configuration of a vehicle. [Figure 2] FIG. 2 is a block diagram illustrating an example of a hardware configuration of the estimation device. [Figure 3] 1 is a block diagram showing an example of a functional configuration of an estimation device according to a first embodiment. [Figure 4] FIG. 2 is a diagram showing an example of two corresponding points on left and right white lines. [Figure 5] FIG. 10 is a diagram for explaining a process of selecting two corresponding points on the left and right white lines. [Figure 6] FIG. 10 is a diagram for explaining a process of selecting two corresponding points on the left and right white lines. [Figure 7] 5 is a flowchart showing an example of an estimation process according to the first embodiment. [Figure 8] FIG. 10 is a diagram for explaining a case where the i and y coordinates of two corresponding points on the left and right white lines are different. [Figure 9] FIG. 10 is a diagram for explaining a difference in three-dimensional coordinates. [Figure 10] FIG. 10 is a diagram for explaining a difference in the X-axis direction after a predetermined time has elapsed. [Figure 11] FIG. 10 is a diagram for explaining a difference on an image after a predetermined time has elapsed. [Figure 12] FIG. 10 is a block diagram showing an example of the functional configuration of an estimation device according to a second embodiment. [Figure 13] 10 is a flowchart illustrating an example of a correction amount derivation process according to the second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0020] Hereinafter, examples of embodiments for carrying out the technology of the present disclosure will be described in detail with reference to the drawings.

[0021] [First embodiment] First, with reference to FIG. 1, the configuration of a vehicle 10 according to this embodiment will be described. As shown in FIG. 1, the vehicle 10 includes a camera 12 as an example of an imaging device mounted on the vehicle, and an estimation device 14. Below, an example will be described in which an orthogonal coordinate system fixed to the camera 12 (hereinafter referred to as a "camera-fixed coordinate system") is applied as an example of a preset coordinate system. The camera-fixed coordinate system according to this embodiment is configured with three axes, with the X axis representing the width direction of the vehicle 10 (in this embodiment, to the right), the Y axis representing the direction of gravity, and the Z axis representing the traveling direction of the vehicle 10, and the camera 12 is set as the origin. The estimation device 14 according to this embodiment is applied to an autonomous vehicle that allows switching between autonomous driving and manual driving, and a vehicle equipped with a driving assistance device.

[0022] The camera 12 is a monocular camera equipped with an image sensor such as a CCD (Charge Coupled Device) image sensor or a CMOS (Complementary Metal Oxide Semiconductor) image sensor. The camera 12 is installed facing forward at the rearview mirror position inside the vehicle cabin, captures images of the area ahead of the vehicle 10, and outputs images of the road along which the vehicle 10 is traveling (hereinafter referred to as "road image") to the estimation device 14. The camera 12 captures the road image at a preset frame rate. In the following, the coordinate system of the image captured by the camera 12 (hereinafter referred to as "image coordinate system") will be described using an orthogonal coordinate system consisting of two axes, with the vertical direction of the image being the i-axis and the horizontal direction of the image being the ix-axis.

[0023] Next, a hardware configuration of the estimation device 14 according to this embodiment will be described with reference to Fig. 2. As shown in Fig. 2, the estimation device 14 includes a CPU (Central Processing Unit) 20, a ROM (Read Only Memory) 21, a RAM (Random Access Memory) 22, and an in-vehicle communication I / F (Interface) 23. The CPU 20, the ROM 21, the RAM 22, and the in-vehicle communication I / F 23 are connected to each other so as to be able to communicate with each other via a bus 27. An example of the estimation device 14 is a computer such as an ECU (Electronic Control Unit).

[0024] The CPU 20 is an example of a processor, and executes various programs and controls various components. That is, the CPU 20 reads programs from the ROM 21 and executes the programs using the RAM 22 as a work area. The ROM 21, which serves as a storage unit, stores an estimation program 30.

[0025] The RAM 22 temporarily stores programs and data as a working area. The in-vehicle communication I / F 23 is an interface for connecting to the camera 12 and the sensors 16. This interface uses a communication standard based on the CAN (Controller Area Network) protocol. The sensors 16 include various sensors mounted on the vehicle 10. The sensors 16 include, for example, a vehicle speed sensor that measures the vehicle speed and a yaw rate sensor that measures the yaw rate.

[0026] Next, the functional configuration of the estimation device 14 according to this embodiment will be described with reference to Fig. 3. As shown in Fig. 3, the estimation device 14 includes an acquisition unit 40, an extraction unit 42, a selection unit 44, and a derivation unit 46. The CPU 20 executes the estimation program 30, thereby functioning as the acquisition unit 40, the extraction unit 42, the selection unit 44, and the derivation unit 46. In this embodiment, a case will be described in which the vehicle 10 is traveling on a curve during a steady circular turn with a bank angle. In addition, in this embodiment, a case will be described in which white lines are used as lane marks indicating the boundaries of the lanes.

[0027] The acquisition unit 40 acquires road images captured by the camera 12 via the in-vehicle communication I / F 23.

[0028] The extraction unit 42 extracts the white lines based on the road image acquired by the acquisition unit 40 using known processing. Specifically, the extraction unit 42 extracts the coordinates (ix l ,iy l ) and the coordinates of the points on the right white line (ix r ,iy r ) is obtained. For example, the extraction unit 42 extracts white lines from the road image using an edge extraction filter such as a Sobel filter, assuming that there is a difference in brightness on the image at the boundary between the road surface and the white lines. Note that the extraction unit 42 may also extract white lines using a Hough transform or the like, assuming that the white lines are straight lines within a small region in the image. The extraction unit 42 may also extract white lines by inputting the road image into a trained model obtained by deep learning. The extraction unit 42 may also derive a function that takes ix coordinates as input and outputs the iy coordinates of points on the white lines corresponding to the input ix coordinates, by fitting the point cloud on the extracted white lines with a B-spline function or the like.

[0029] As an example, as shown in FIG. 4, the selection unit 44 selects two points on the left and right white lines in the road image extracted by the extraction unit 42, the slopes of which satisfy a predetermined condition. In the example of FIG. 4, the coordinates of the point on the left white line (hereinafter referred to as the "first point") of the two selected points are (ix l ,iy l ), and the coordinates of the point on the right white line (hereinafter referred to as the "second point") are (ix r ,iy r ) is shown. Also, the α shown in the example of Figure 4 l represents the slope of the tangent to the first point of the left white line, and α r represents the slope of the tangent to the right white line at the second point.

[0030] In this embodiment, the selection unit 44 selects two points that satisfy the condition that a line connecting the two points is parallel to the horizontal direction of the image. That is, the selection unit 44 selects two points on the left and right white lines in the road image that have the same i and y coordinates, which are coordinates of the vertical axis of the road image. In the example of FIG. 4, l =iy r The two points are selected.

[0031] Here, the selection of two corresponding points on the left and right white lines will be described in detail. As an example, as shown in Fig. 5, the left and right white lines of the road on which the vehicle 10 is traveling can be represented by arcs of different heights in the Y-axis direction cut by a plane parallel to the base of a cone. The upper side of Fig. 5 shows a plan view, and the lower side of Fig. 5 shows a cross-sectional view. Point P1 in Fig. 5 indicates a point on the left white line, and point P2 indicates a point on the right white line. Furthermore, r in Fig. 5 indicates the radius of curvature.

[0032] If the vehicle body is placed perpendicular to the road surface by the bank angle ψ, the vehicle body and camera 12 will also be tilted relative to the road surface by the bank angle ψ. That is, consider the projection from the X'-Y'-Z coordinate system shown in the cross section of FIG. 5 to the ix-iy coordinate system shown in FIG. 6. Imagine a line P passing through the apex of the cone in FIG. 5, points P1, and P2. Since linearity is preserved in perspective transformation, the line P in FIG. 5 will also be a straight line in the ix-iy coordinate system, as shown in FIG. 6. The slope α of the line P in FIG. 6 s If we know the distance, we can determine the correspondence between the left and right lane width segments in the road image.

[0033] Although it is omitted in Fig. 6 because it is outside the frame, the coordinates of the point (ix v ,iy v ) is expressed by the following equations (1) and (2). In equations (1) and (2), ix0 and iy0 represent the coordinates of the center of the camera 12, and e represents the distance [m] along the X-axis direction between the center of the camera 12 and the center of the lane at Z=0. In addition, in equations (1) and (2), f represents the focal length [m], and r h , r vrepresents the horizontal pixel resolution [m / pixel] and the vertical pixel resolution [m / pixel]. Also, h in equation (2) represents the height [m] of the camera 12 from the road surface at Z=0. h is assumed to be known by measuring it in advance or by obtaining a measurement value from a vehicle height sensor.

[0034]

number

[0035]

number

[0036] The coordinates of point P1 in Figure 6 (ix l ,iy l ) to the coordinates of the vertex of the cone (ix v ,iy v ) the slope of the line drawn to s is expressed by the following equation (3): Here, the point P1 will be explained, but the same applies to the point P2.

[0037]

number

[0038] Substituting equations (1) and (2) into equation (3), we obtain the following equation (4).

[0039]

number

[0040] Multiplying the numerator and denominator of the right-hand side of equation (4) by Z and rearranging it, we obtain the following equation (5).

[0041]

number

[0042] Here, since the apex of the cone is directly to the side when viewed from the camera 12, Z=0 can be set in equation (5), and the following equation (6) is obtained.

[0043]

number

[0044] Assuming that |r|>>|e| and ignoring e in equation (6), we obtain one term on the right-hand side of equation (7). In this case, the yaw rate ω around the Y axis and the yaw rate ω measured by the on-board gyro sensor that measures the yaw rate around the vertical axis (Y' axis) of the vehicle that is perpendicular to the road surface tilted at a bank angle ψ are obs The relationship between ω obs =ωcosψ, and if the vehicle speed is v, then the two terms on the right side of equation (7) can be obtained from the known knowledge that ω=v / r. Note that ω is a positive value when the lane curves to the right, and a negative value when the lane curves to the left.

[0045]

number

[0046] Similarly, when the lane curves to the right, the following equation (8) is obtained.

[0047]

number

[0048] The line P connecting the apex of the cone and point P1 on the left white line also passes through point P2 on the right white line, which corresponds to the end point of the lane width segment. s In other words, if the lane curves to the left, the slope α is calculated according to equation (7), and if the lane curves to the right, the slope α is calculated according to equation (8). s By searching the road image from a point on one white line in the direction of the arrow, it is possible to extract the corresponding point on the other white line.

[0049] For example, r=100[m], h=1.6[m], ψ=20[deg], r h =r v In this case, when substituted into equation (7), α s =-66.5, which translates to an angle of -89.1 degrees relative to the i and y axes. In other words, if the radius of curvature is relatively large, two corresponding points on the left and right white lines can be approximated by two points with the same i and y coordinates.

[0050] If the radius of curvature of the road is equal to or greater than a threshold, the selection unit 44 may select two corresponding points on a white line in the road image that have the same coordinates on the vertical axis of the road image. Also, if the radius of curvature of the road is less than a threshold, the selection unit 44 may select two corresponding points on a white line in the road image that have the same coordinates on the vertical axis of the road image. s As shown in the first item on the right side of equations (7) and (8), the slope α s is a slope according to a ratio obtained by dividing the radius of curvature r by the height h of the camera 12. The selection unit 44 may obtain the radius of curvature r in these cases from an infrastructure system that provides map information or road information, or may obtain the radius of curvature r from the steering angle, vehicle speed v, and yaw rate ω obtained from the sensor 16. obs The threshold value in these cases may be determined in advance based on, for example, experiments. As shown in the two items on the right side of the equations (7) and (8), the selection unit 44 determines the yaw rate ω obs If the absolute value of is greater than or equal to the threshold, the slope of the corresponding two points on the white line in the road image is calculated based on the vehicle speed v, the height h of the camera 12, and the yaw rate ω obs Alternatively, two points may be selected whose slopes correspond to the ratio obtained by dividing by .

[0051] The derivation unit 46 derives the three-dimensional coordinates of the white line and the three-dimensional coordinates of the center of the lane in the width direction corresponding to the two selected points based on the coordinates of the two points on the road image selected by the selection unit 44. Note that the derivation unit 46 may derive either the three-dimensional coordinates of the white line or the three-dimensional coordinates of the center of the lane in the width direction. A specific example of the process performed by the derivation unit 46 to derive the three-dimensional coordinates of the white line and the three-dimensional coordinates of the center of the lane in the width direction corresponding to the two selected points will be described below.

[0052] As shown in the example of FIG. 4, in the road image, the coordinates of the first point on the left white line of the two points selected by the selection unit 44 are (ix l ,iy l ) and the coordinates of the second point on the right white line are (ix r ,iy r ) In addition, in the road image, the gradient of the tangent line at the first point of the left white line is expressed as α l The slope of the tangent at the second point of the right white line is α r This is expressed as follows.

[0053] The derivation unit 46 derives the three-dimensional coordinates (x, y, z) of the center of the lane width direction in the camera fixed coordinate system, and the azimuth angle θ, depression angle φ, and bank angle ψ that represent the amount of rotation of the vehicle body, according to the following equations (9) to (19). Note that ix0 and iy0 represent the coordinates of the center of the camera 12 in the image coordinate system, and ix c and iy c is (ix l ,iy l ) and (ix r ,iy r ) represents the coordinates in the image coordinate system of the intersection point of the tangent lines (i.e., the vanishing point) that touch each of the points. c and iy c is (ix l ,iy l ), (ix r ,iy r ), α l , and α r It can be calculated using known knowledge. Also, f represents the focal length [m], and r h , r vrepresents the horizontal pixel resolution [m / pixel] and the vertical pixel resolution [m / pixel], and w represents the lane width [m]. w may be calculated from the nearest left and right white lines using a known method based on a local plane assumption, or may use values ​​registered in map information, a database, or the like. In this embodiment, w is assumed to be constant from the nearest point of the vehicle 10 to the target distance.

[0054]

number

[0055]

number

[0056]

number

[0057]

number

[0058]

number

[0059]

number

[0060]

number

[0061]

number

[0062]

number

[0063]

number

[0064]

number

[0065]

number

[0066] Furthermore, the derivation unit 46 derives the three-dimensional coordinates of the white lines corresponding to the two points selected by the selection unit 44 according to the following equations (20) to (22): (x l ,y l ,z l ) represents the three-dimensional coordinates of the first point in the camera-fixed coordinate system, and (x r ,y r ,z r ) represents the 3D coordinates of the second point in the camera-fixed coordinate system.

[0067]

number

[0068]

number

[0069]

number

[0070] The selection unit 44 may select multiple pairs of corresponding two points on the left and right white lines in the road image extracted by the extraction unit 42. In this case, the derivation unit 46 may derive, for each of the multiple pairs of two points, the three-dimensional coordinates of the white line corresponding to the two points and the three-dimensional coordinates of the center in the width direction of the lane. Furthermore, in this case, the derivation unit 46 may derive the three-dimensional coordinates of the white line corresponding to the two points at a target distance ahead (i.e., the target Z coordinate) and the three-dimensional coordinates of the center in the width direction of the lane by interpolation or filtering using the three-dimensional coordinates derived for each of the multiple pairs of two points. Furthermore, when selecting the two corresponding points on the left and right white lines, the selection unit 44 may select the iy coordinates corresponding to the target distance based on the longitudinal alignment data of the road. l and iy r may be set.

[0071] The three-dimensional coordinates of the two corresponding points on the left and right white lines and the three-dimensional coordinates of the center of the lane width derived in this manner are used to control automated driving and driving assistance systems.

[0072] Next, the operation of the estimation device 14 according to this embodiment will be described with reference to Fig. 7. The estimation process shown in Fig. 7 is executed, for example, when the operation mode of the vehicle 10 is set to the autonomous driving mode, when an execution instruction is input by the driver, etc.

[0073] 7, the acquisition unit 40 acquires the road image captured by the camera 12 via the in-vehicle communication I / F 23. In step S12, the extraction unit 42 extracts white lines based on the road image acquired in step S10, as described above.

[0074] In step S14, the selection unit 44 selects two points on the left and right white lines in the road image extracted in step S12, where the slopes of the two corresponding points satisfy a predetermined condition, as described above. In step S16, the derivation unit 46 derives the three-dimensional coordinates of the white lines corresponding to the selected two points and the three-dimensional coordinates of the center of the lane in the width direction, based on the coordinates of the two points on the road image selected in step S14, as described above. When the processing of step S16 ends, the estimation processing ends.

[0075] As described above, according to this embodiment, the three-dimensional coordinates of the vehicle ahead in the driving lane can be estimated with high accuracy.

[0076] [Second embodiment] A second embodiment of the disclosed technology will be described. Note that the configuration of a vehicle 10 (see FIG. 1) and the hardware configuration of an estimation device 14 (see FIG. 2) according to the second embodiment are the same as those of the first embodiment, and therefore descriptions thereof will be omitted.

[0077] In the first embodiment, an example was described in which the selection unit 44 selects two points with the same i and y coordinates as two corresponding points on the left and right white lines. However, as shown in Fig. 8, the i and y coordinates may be changed depending on the roll angle generated with respect to the road surface directly below the vehicle 10 due to centrifugal force applied by the vehicle motion, and the difference between the bank angle of the road surface directly below the vehicle 10 and the bank angle ahead of the target distance, which occurs when the vehicle 10 travels on a road where the bank angle gradually changes, such as when traveling on a transition curve. l ≠iy r This may be the case.

[0078] Therefore, when the selection unit 44 selects two points with equal i and y coordinates as two corresponding points on the left and right white lines, and the derivation unit 46 derives three-dimensional coordinates based on the coordinates on the road image of the two points selected by the selection unit 44, errors may occur in the derived three-dimensional coordinates.

[0079] As shown in Figure 9, when two corresponding points on the left and right white lines contain errors, deriving three-dimensional coordinates based on the coordinates of those two points on the road image results in a difference ΔZ in the Z-axis direction and a difference ΔX in the X-axis direction with respect to the true white line. These differences occur in the depth direction on the road image, so they cannot be observed at the time the road image is captured. However, as shown in Figures 10 and 11, when the points are tracked after a certain time has passed, they deviate from the white line observed on the road image, resulting in a difference ΔX obs is observed.

[0080] Therefore, the estimation device 14 according to this embodiment calculates the difference ΔX based on road images captured at multiple points in time. obs Estimate the difference ΔX obs It has the function of deriving corrected 3D coordinates based on the above.

[0081] The functional configuration of the estimation device 14 according to this embodiment will be described with reference to Fig. 12. As shown in Fig. 12, the estimation device 14 further includes a correction unit 48 in addition to the acquisition unit 40, extraction unit 42, selection unit 44, and derivation unit 46 according to the first embodiment. When the CPU 20 executes the estimation program 30, the estimation device 14 functions as the acquisition unit 40, extraction unit 42, selection unit 44, derivation unit 46, and correction unit 48.

[0082] The acquisition unit 40 according to this embodiment acquires a road image captured by the camera 12 at a first time point (hereinafter referred to as a "first road image") via the in-vehicle communication I / F 23. The acquisition unit 40 also acquires a road image captured by the camera 12 at a second time point later than the first time point (hereinafter referred to as a "second road image") via the in-vehicle communication I / F 23. The time difference between the first time point and the second time point is experimentally determined according to the frame rate of the camera 12, such as a period of five frames.

[0083] The three-dimensional coordinates in the camera-fixed coordinate system of two corresponding points on the left and right white lines in the first road image are calculated using the above-mentioned equations (9) to (22). These three-dimensional coordinates may contain the above-mentioned errors. Here, we will explain an example in which we assume that the vehicle 10 is traveling on a road that curves to the left, fix a point on the left white line, and calculate the correction amount for correcting the three-dimensional coordinates of a point on the right white line.

[0084] The correction unit 48 receives input of the three-dimensional coordinates (hereinafter referred to as "first three-dimensional coordinates") derived by the derivation unit 46, which correspond to the point on the right white line of the two points selected by the selection unit 44 from the first road image. These three-dimensional coordinates are updated as history up to the second point in time. In the following, the first three-dimensional coordinates are defined as (x r ,y r ,z r) Since the vehicle 10 is moving, in the camera fixed coordinate system, this point on the right white line moves relatively from the first time point to the second time point.

[0085] The correction unit 48 estimates the three-dimensional coordinates (hereinafter referred to as "second three-dimensional coordinates") of the point on the first three-dimensional coordinates at a second time point by utilizing known knowledge of vehicle motion based on the first three-dimensional coordinates and vehicle speed and yaw rate as examples of measurement values ​​from the sensor 16.

[0086] The correction unit 48 calculates a correction amount Δiy for the point on the right white line of the two points selected by the selection unit 44 based on the difference between the first three-dimensional coordinates and the second three-dimensional coordinates at the second time point. r The correction amount Δiy of the coordinates on the road image r Hereinafter, the correction amount Δiy r A specific example of the derivation process of the first three-dimensional coordinate (x r ,y r ,z r ) The difference ΔZ in the Z-axis direction and the difference ΔX in the X-axis direction, which occur due to an incorrect correspondence between the two corresponding points on the left and right white lines, can be linearly approximated as follows:

[0087]

number

[0088] tanθ in equation (23) a and tanθ b are expressed by the following equations (24) and (25).

[0089]

number

[0090]

number

[0091] Also, ΔX and ΔXobs The relationship between θ and θ is expressed by the following equation (26): traj represents a relative movement angle that indicates the amount of relative rotation between a point in the first three-dimensional coordinate system and a point in the second three-dimensional coordinate system (see FIG. 10). traj can be calculated by integrating the yaw rate over time.

[0092]

number

[0093] z expressed by Eq. (22) r iy r By partially differentiating with respect to, the following equation (27) is obtained.

[0094]

number

[0095] ∂z in equation (27) r / ∂iy r is expressed by the following equation (28).

[0096]

number

[0097] According to equations (23) to (28), ΔX obs iy, which is the iy coordinate of the point on the right white line of the road image, r Correction amount Δiy r can be calculated using the following equation (29).

[0098]

number

[0099] The correction unit 48 calculates iy coordinates of a point on the right white line of the road image by the following equation (30): r The correction amount Δiy rThis correction yields the combination of two points shown in FIG. 11b, which is closer to the correct answer than the combination of two points shown in FIG. 11a.

[0100]

number

[0101] The derivation unit 46 calculates the coordinates of the two points on the road image after correction by the correction unit 48 (ix l ,iy l ) and (ix r ,iy r ) based on the above, the three-dimensional coordinates of the white line corresponding to the two points and the three-dimensional coordinates of the center of the lane in the width direction are derived. The process of deriving these three-dimensional coordinates is the same as in the first embodiment.

[0102] The above description has been given using an example in which the vehicle 10 is traveling on a road that curves to the left, but the same applies when the vehicle 10 is traveling on a road that curves to the right. In this case, the point on the right white line is fixed, and a correction amount for correcting the coordinates of the point on the left white line is calculated. In this case, the following equation (31) is used instead of equation (28).

[0103]

number

[0104] The correction unit 48 calculates the correction amount Δiy r The correction unit 48 may repeatedly perform the derivation process of the correction amount Δiy for a plurality of times, for example, for every predetermined number of frames. r The derivation process of the correction amount Δiy is performed in the same manner for multiple pairs of corresponding two points on the left and right white lines with different distance Z or iy coordinates, and the correction amount Δiy obtained for each pair is r The final correction amount Δiy r may be calculated.

[0105] Next, the operation of the estimation device 14 according to this embodiment will be described with reference to Fig. 13. The correction amount derivation process shown in Fig. 13 is executed, for example, when the operation mode of the vehicle 10 is set to the autonomous driving mode, when an execution instruction is input by the driver, etc.

[0106] In step S20 of FIG. 13, the acquisition unit 40 acquires a road image captured by the camera 12 at a first time point via the in-vehicle communication I / F 23. The acquisition unit 40 also acquires a road image captured by the camera 12 at a second time point via the in-vehicle communication I / F 23. In step S22, the extraction unit 42 extracts white lines based on the road image captured at the first time point acquired in step S20, as described above. The extraction unit 42 also extracts white lines based on the road image captured at the second time point acquired in step S20, as described above.

[0107] In step S24, the selection unit 44 selects two points on the left and right white lines in the road image extracted in step S22, the slopes of which satisfy a predetermined condition, as described above. In step S26, the derivation unit 46 derives first three-dimensional coordinates based on the coordinates of the two points on the road image selected in step S24, as described above.

[0108] In step S28, as described above, the correction unit 48 estimates the second three-dimensional coordinates based on the first three-dimensional coordinates and the vehicle speed and yaw rate as examples of measured values ​​by the sensor 16. In step S30, as described above, the correction unit 48 calculates the correction amount Δiy of the coordinates on the road image of the two points selected in step S24 based on the difference between the first three-dimensional coordinates and the second three-dimensional coordinates at the second time point. r is derived.

[0109] In step S32, the correction unit 48 calculates the iy coordinate of the point on the right white line of the road image, as described above. r The correction amount Δiy rIn step S34, as described above, the derivation unit 46 derives the three-dimensional coordinates of the white line corresponding to the two points and the three-dimensional coordinates of the center of the lane in the width direction based on the coordinates of the two points on the road image after correction in step S32. When the processing of step S34 ends, the correction amount derivation processing ends.

[0110] As described above, according to this embodiment, the three-dimensional coordinates of the vehicle ahead in the driving lane can be estimated with high accuracy.

[0111] In the above embodiment, a case has been described in which a Cartesian coordinate system fixed to the camera 12 is applied as the preset coordinate system, but the disclosed technology is not limited to this. For example, a Cartesian coordinate system fixed to the road surface on which the vehicle 10 travels may be applied as the preset coordinate system.

[0112] In the above embodiment, the functional units that execute various processes of the estimation device 14 may be implemented by a processor other than a CPU. Examples of processors in this case include programmable logic devices (PLDs) such as field-programmable gate arrays (FPGAs), whose circuit configuration can be changed after manufacture, and dedicated electrical circuits such as application-specific integrated circuits (ASICs), which are processors having a circuit configuration specifically designed to execute specific processes. Each functional unit of the estimation device 14 may be implemented by one of these various processors, or by a combination of two or more processors of the same or different types (e.g., multiple FPGAs, or a combination of a CPU and an FPGA). The hardware structure of these various processors is, more specifically, an electrical circuit that combines circuit elements such as semiconductor elements.

[0113] In the above embodiment, the estimation program 30 is pre-stored (installed) in the ROM 21, but the present invention is not limited to this. The estimation program 30 may be provided in a form recorded on a recording medium such as a CD-ROM (Compact Disc Read Only Memory), a DVD-ROM (Digital Versatile Disc Read Only Memory), or a USB (Universal Serial Bus) memory. The estimation program 30 may also be downloaded from an external device via a network. [Explanation of symbols]

[0114] 10 vehicles 12 Camera 14 Estimation device 16 sensors 20 CPU 30 Estimation Program 40 Acquisition Department 42 Extraction part 44 Selection section 46 Derivation part 48 Correction section

Claims

1. an acquisition unit that acquires a road image, which is an image of the road along which the vehicle is traveling, captured by a photographing device mounted on the vehicle; an extraction unit that extracts lane boundaries based on the road image; a selection unit that selects two points on the boundary between the left and right lanes in the road image, the gradients of which satisfy a predetermined condition; a derivation unit that derives at least one of three-dimensional coordinates of a lane boundary and three-dimensional coordinates of a lane center corresponding to the two selected points based on the coordinates of the two points on the road image selected by the selection unit; An estimation device comprising:

2. When the radius of curvature of the road is equal to or greater than a threshold, the selection unit selects two corresponding points on a boundary of a lane in the road image, the two points having the same coordinates on an axis in the up-down direction of the road image. The estimation device according to claim 1 .

3. When the radius of curvature of the road is less than a threshold, the selection unit selects two points on the road image corresponding to a lane boundary, the slopes of which correspond to a ratio obtained by dividing the radius of curvature by the height of the image capturing device. The estimation device according to claim 1 .

4. When the absolute value of the yaw rate during travel on the road is equal to or greater than a threshold value, the selection unit selects two points on the road image corresponding to a lane boundary, the slope of which corresponds to a ratio obtained by dividing the vehicle speed by the height and yaw rate of the image capturing device. The estimation device according to claim 1 .

5. a correction unit that estimates second three-dimensional coordinates corresponding to one of the two points at a second time point after the first time point based on first three-dimensional coordinates derived by the derivation unit that correspond to one of the two points selected by the selection unit from the road image captured at a first time point and measurement values ​​by a sensor mounted on the vehicle, derives a correction amount for the coordinates on the road image of one of the two points based on a difference between the first three-dimensional coordinates and the second three-dimensional coordinates at the second time point, and corrects the coordinates on the road image of the one of the two points using the correction amount, The derivation unit derives at least one of three-dimensional coordinates of a lane boundary and three-dimensional coordinates of a lane center corresponding to the two points based on the coordinates of the two points on the road image after correction by the correction unit. The estimation device according to any one of claims 1 to 4.

6. Acquire a road image, which is an image of the road along which the vehicle is traveling, captured by a photographing device mounted on the vehicle; Extracting lane boundaries based on the road image; Two points are selected on the boundary between the left and right lanes in the road image, the gradients of which satisfy a predetermined condition; Based on the coordinates of the two selected points on the road image, at least one of the three-dimensional coordinates of the boundary of the lane and the three-dimensional coordinates of the center of the lane corresponding to the two selected points is derived. A method of estimating that the processing is performed by a computer.

7. Acquire a road image, which is an image of the road along which the vehicle is traveling, captured by a photographing device mounted on the vehicle; Extracting lane boundaries based on the road image; Two points are selected on the boundary between the left and right lanes in the road image, the gradients of which satisfy a predetermined condition; Based on the coordinates of the two selected points on the road image, at least one of the three-dimensional coordinates of the boundary of the lane and the three-dimensional coordinates of the center of the lane corresponding to the two selected points is derived. An estimation program that causes a computer to execute the processing.

Images