Display device

Abstract

To provide a display device capable of suppressing positional deviation of a road shape image in a display image obtained by image recognition using an on-vehicle camera.SOLUTION: The display device 100 is a device that displays a lane marking of a lane in which a vehicle travels on the display 3. The display apparatus 100 includes the recognition generation unit 11 that recognizes the lane marking based on the image of the 1a captured by the camera in the vehicle and generates the display image including the road shape image arranged along the lane marking, and the display control unit 12 that displays the display image on the display 3. The display control unit 12 estimates the cant angle of the lane based on the detection result of the internal sensor 2 that detects the traveling state of the vehicle, and corrects the position of the road shape image in the display image using the cant angle.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a display device. [Background technology]

[0002] Patent Document 1 describes a vehicle display control device that uses millimeter-wave radar to identify the three-dimensional position of an object relative to the vehicle, uses a high-precision map to identify the three-dimensional position of the object relative to the vehicle, and superimposes a virtual image that emphasizes the object according to the identified three-dimensional position. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2020-16541 Summary of the Invention [Problem to be solved by the invention]

[0004] For example, image recognition using an on-board camera used for vehicle driving assistance or autonomous driving, etc., can involve recognition errors. For example, in situations where the recognition errors are noticeable, the positions of lane markings and the like in the display image resulting from the image recognition may be displaced due to the influence of the recognition errors. When using high-precision maps to more accurately display objects, as in the above-described conventional technology, there are issues such as increased costs for preparing high-precision maps by actual measurements and room for improvement in the immediacy of updating high-precision maps. [Means for solving the problem]

[0005] A display device according to one aspect of the present invention is a display device that displays the lane markings on an in-vehicle display along which a vehicle is traveling, and includes a recognition and generation unit that recognizes the lane markings based on an image captured by the vehicle's on-board camera and generates a display image including a road shape image arranged along the lane markings, and a display control unit that displays the display image on the vehicle's display unit.The display control unit estimates the cant angle of the lane based on the detection results of an internal sensor that detects the vehicle's traveling state, and corrects the position of the road shape image in the display image using the cant angle.

[0006] In a display device according to one aspect of the present invention, lane markings are recognized based on an image captured by an onboard camera of a vehicle. In the display image, the position of a road shape image extending along the recognized lane markings is corrected using a cant angle of the lane estimated based on the detection results of an internal sensor that detects the vehicle's driving state. This makes it possible to suppress positional deviations of the road shape image in the display image obtained by image recognition using an onboard camera while suppressing increases in cost compared to, for example, using a high-precision map.

[0007] In one embodiment, the display control unit may use the vehicle speed, lateral acceleration, and yaw rate of the vehicle as the vehicle's running conditions to estimate the cant angle of the lane at the vehicle's position based on an equation of motion that represents the balance between the lateral force acting on the vehicle due to the cant of the lane and the lateral force corresponding to the lateral acceleration of the vehicle, and the lateral force associated with the turning of the vehicle. In this case, the position of the road shape image in the display image can be corrected using the cant angle at the vehicle's position estimated using the vehicle speed, lateral acceleration, and yaw rate of the vehicle.

[0008] In one embodiment, the display control unit may estimate the cant angle so that the cant angle gradually increases with increasing distance forward from the vehicle position when the estimated cant angle at the vehicle position is smaller than a predetermined cant angle threshold and the increase rate of the lane marking curvature in a predetermined section ahead of the vehicle is equal to or greater than a predetermined increase rate threshold, or may estimate the cant angle so that the cant angle gradually decreases with increasing distance forward from the vehicle position when the estimated cant angle at the vehicle position is equal to or greater than the cant angle threshold and the decrease rate of the lane marking curvature in the predetermined section is equal to or greater than a predetermined decrease rate threshold. In this case, for example, when the vehicle is approaching a curve from a straight line, the cant angle can be estimated to be larger in advance to match the curve with gradually increasing curvature. On the other hand, when the vehicle is approaching a curve from a straight line, the cant angle can be estimated to be smaller in advance to match the curve with gradually decreasing curvature. [Effects of the Invention]

[0009] According to the present invention, it is possible to suppress deviation in the position of a road shape image in a display image obtained by image recognition using an on-board camera. [Brief explanation of the drawings]

[0010] [Figure 1] 1 is a block diagram showing an example of a configuration of a display device according to an embodiment. [Figure 2] 10A and 10B are diagrams for explaining positional deviation of lane lines in a display image; [Figure 3] 10A and 10B are diagrams illustrating an example of correction of the position of a lane marking in a display image. [Figure 4] 10 is a flowchart illustrating an example of processing performed by the display device according to the embodiment. [Figure 5] 5 is a flowchart showing an example of a process for estimating a cant angle in FIG. 4. DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0012] Fig. 1 is a block diagram showing a display device 100 according to an embodiment. The display device 100 shown in Fig. 1 is mounted on a vehicle such as a passenger car or a freight vehicle, and displays images according to the surrounding conditions of the vehicle. The display device 100 displays the lane markings on an in-vehicle display.

[0013] The lane markings displayed on the in-vehicle display are used, for example, for lane tracing control (LKA [Lane Keeping Assist]) that prompts the driver to steer the vehicle so that it does not deviate from the lane it is traveling in, and control (LDW [Lane Departure Warning]) that alerts the driver that the vehicle is at risk of deviating from the lane. The vehicle may be configured to be able to execute driving assistance functions such as the LKA and LDW, and an ADAS [Advanced Drive Assistance System] such as an autonomous driving function including vehicle speed control.

[0014] As shown in FIG. 1, the display device 100 includes a display control ECU (Electronic Control Unit) 10 that performs overall management of the device. The display control ECU 10 is an electronic control unit having a CPU (Central Processing Unit) and a storage unit. The storage unit is composed of, for example, a ROM (Read Only Memory), a RAM (Random Access Memory), and an EEPROM (Electrically Erasable Programmable Read-Only Memory). The display control ECU 10 realizes various functions by, for example, causing the CPU to execute programs stored in the storage unit. An external sensor 1 and a display 3 (display unit) are connected to the display control ECU 10. Note that the display control ECU 10 may be composed of multiple electronic units.

[0015] The external sensor 1 is a detection device that detects the situation around the vehicle. The external sensor 1 includes a camera (on-board camera) 1a that captures an image in front of the vehicle. The camera 1a is an imaging device that captures an image in front of the vehicle. The camera 1a is provided, for example, behind the windshield of the vehicle and captures an image in front of the vehicle. Types of the camera 1a include, for example, a telephoto camera, a wide-angle camera, a monocular camera, and a stereo camera. A stereo camera has two imaging units arranged to reproduce binocular parallax. The imaging information of the stereo camera also includes information in the depth direction. The camera 1a transmits information about the captured image to the display control ECU 10.

[0016] The external sensor 1 may include a radar sensor. A radar sensor is a detection device that detects objects around the vehicle using radio waves (e.g., millimeter waves) or light. A radar sensor detects objects by transmitting radio waves or light to the vicinity of the vehicle and receiving the radio waves or light reflected by the objects. Examples of radar sensors include a LiDAR (Light Detection And Ranging) 1b and a millimeter-wave radar 1c. The radar sensor may acquire distance measurements to objects around the vehicle (other vehicles, etc.). The radar sensor transmits information about detected objects and acquired distance measurements to the display control ECU 10. In the following description, at least one of the LiDAR 1b and the millimeter-wave radar 1c will be collectively referred to simply as "radar sensors 1b, 1c."

[0017] The internal sensor 2 is a detection device that detects the running state of the vehicle. The internal sensor 2 includes a vehicle speed sensor, an acceleration sensor, and a yaw rate sensor. The vehicle speed sensor is a detector that detects the speed of the vehicle. The vehicle speed sensor transmits the detected vehicle speed information (wheel speed information) to the display control ECU 10.

[0018] The acceleration sensor is a detector that detects the acceleration of the vehicle. The acceleration sensor includes at least a lateral acceleration sensor that detects the lateral acceleration, which is the acceleration in the vehicle width direction. The acceleration sensor transmits the detected acceleration information to the display control ECU 10.

[0019] The yaw rate sensor is a detector that detects the yaw rate (angular velocity) around the vertical axis of the center of gravity of the vehicle. For example, a gyro sensor can be used as the yaw rate sensor. The yaw rate sensor transmits the detected yaw rate information of the vehicle to the display control ECU 10.

[0020] The display 3 is an in-vehicle display provided in the cabin of the vehicle. An example of the display 3 is a head-up display (HUD). The display 3 may be configured as an AR-HUD that projects an image onto the display surface of the front windshield using, for example, augmented reality (AR) technology. The display area of ​​the head-up display is a pre-set area on the front windshield, and is the range where a virtual image is projected and superimposed. The display 3 is controlled to display various information in response to a control signal from the display control ECU 10. The display 3 displays a virtual image including a vehicle image representing another vehicle superimposed on an image showing the situation ahead of the vehicle. Note that the display 3 is not limited to a HUD. The display 3 may also be a liquid crystal display provided in front of the driver on an instrument panel or a liquid crystal display of a navigation system.

[0021] Next, a description will be given of the functional configuration of the display control ECU 10. The display control ECU 10 includes a recognition generation unit 11 and a display control unit 12.

[0022] The recognition generation unit 11 recognizes marking lines based on the image captured by the camera 1a of the vehicle. As an example of marking lines, the recognition generation unit 11 recognizes a pair of left and right white lines extending to separate the lane on which the vehicle is traveling (the vehicle's own lane). Note that the marking lines recognized by the recognition generation unit 11 are not limited to the pair of left and right white lines, but may be one of these, or another marking line extending along the vehicle's own lane. The recognition generation unit 11 can recognize marking lines by performing known image recognition on the image captured by the camera 1a, such as deep learning.

[0023] The recognition generation unit 11 recognizes the position of the lane marking line based on the image captured by the vehicle's camera 1a. The lane marking line position is the position of the lane marking line in the display image for representing the lane marking line in the display image. The lane marking line position may be, for example, the x, y coordinates of a point corresponding to the position of the lane marking line in the x, y coordinate system of the display image. The lane marking line position may also be a series of coordinates corresponding to the respective positions of the point cloud that constitutes the lane marking line in the display image.

[0024] The recognition generation unit 11 recognizes, for example, the reference lane line position as the lane line position. The reference lane line position is, for example, the position of the road shape image corresponding to a predetermined plane on which the vehicle exists in the display image. The predetermined plane means a plane that serves as a reference for displaying the image, and may be assumed to be, for example, a horizontal plane (a plane with no gradient) on which the vehicle exists. In other words, the road surface of the current lane may not be parallel to the predetermined plane, for example, when the road surface of the current lane has a gradient (cant) in the lane width direction.

[0025] The recognition generation unit 11 generates a display image. The display image is an image to be displayed on the display 3. The display image may be an image corresponding to the entire display area of ​​the display 3, such as an image showing the scenery ahead as viewed by the driver through the windshield of the vehicle. If the display 3 is a HUD, the display image may be an image of a virtual image to be superimposed.

[0026] The display image includes a road shape image arranged along the lane markings. The road shape image is an image for conceptually representing the road shape extending along the lane markings in the display image. The road shape image can be, for example, a line segment arranged along the lane markings representing the road surface of the current lane in the display image, or an icon image of multiple polygons or circles. The road shape image can be, for example, a straight line or a curve extending along the lane markings representing the road surface of the current lane in the display image. The road shape image can be drawn along the recognized series of lane marking positions. The recognition generation unit 11 generates a road shape image based on the recognized lane marking positions.

[0027] FIG. 2 is a diagram illustrating positional deviation of lane markings in a display image. FIG. 2 shows a display image 20 representing a vehicle traveling around a left curve on a highway as an example of a scene ahead of the vehicle. In FIG. 2, the display image 20 is a virtual image projected onto the vehicle's windshield and is superimposed on the scene ahead of the vehicle as viewed by the driver through the windshield. A pair of lane markings 21 exist in the actual scene ahead of the vehicle. Therefore, when a road shape image is included in the display image 20 and superimposed on the actual scene, the display error of the road shape image (the error in the lane marking position) corresponds to the amount of deviation based on the actual lane markings 21 in FIG. 2.

[0028] In FIG. 2, multiple LKA indicators 22 (road shape images) arranged along the vehicle's lane are included in the display image 20 and superimposed on the scenery ahead, and are displayed for controlling the LKA. The LKA indicators 22 may be, for example, a triangle oriented with one vertex pointing in the vehicle's traveling direction. The multiple LKA indicators 22 may be arranged at predetermined intervals along a virtual line indicating the target trajectory. The recognition / generation unit 11 can calculate the trajectories of the multiple LKA indicators 22 using a known road shape model. The road shape model may be, for example, a clothoid model. Based on the recognized lane lines and the positions of the reference lane lines, the recognition / generation unit 11 can calculate the curvature of the lane lines, the amount of change in the curvature of the lane lines, the offset (lateral position) of the lane lines from a predetermined reference point (e.g., the origin) in the display image 20, the relative angle (yaw angle) between the lane lines and the vehicle, and so on, and use these to calculate the trajectory of the LKA indicators 22. In other words, the LKA indicators 22 are calculated based on the positions of the reference lane lines. The LKA display 22 may be a figure other than a triangle, or may be an image of a continuous or intermittent line.

[0029] The recognition generation unit 11 can calculate the positions of a pair of LDW displays 23 (road shape images) using a method similar to that used to calculate the trajectory of the LKA display 22. The LDW display 23 is an image included in the display image 20 and superimposed on the forward scenery, and is an image displayed for the above-mentioned LDW control so as to correspond to the position of a lane marking. The LDW display 23 is also calculated based on the reference lane marking position. For example, the LDW display 23 can be an image of a line segment approximately parallel to the tangent direction of the lane marking at a position corresponding to a predetermined distance ahead of the vehicle. In the example of FIG. 2, the LDW display 23 is displayed as a pair so as to correspond to the positions of the lane markings on both the left and right ends of the lane. The LDW display 23 may also be displayed so as to correspond to the position of the lane marking on either the left or right side of the lane (for example, the side where the vehicle is about to deviate from the lane).

[0030] Here, the left curve in FIG. 2 has a cant such that the road surface of the vehicle's lane is lower on the left side and higher on the right side in the lane width direction. In a curve with a cant, the actual lane marking 21 may not exist on a predetermined plane (the horizontal plane on which the vehicle exists). However, the reference lane marking position recognized by the recognition generation unit 11 is recognized as a corresponding position on the predetermined plane. Therefore, the actual lane marking 21 on the right side may be erroneously recognized as being farther away from the vehicle than the actual distance, and the actual lane marking 21 on the left side may be erroneously recognized as being closer to the vehicle than the actual distance.

[0031] Therefore, if the reference lane marking position recognized from the captured image is simply used, the LKA display 22 and the LDW display 23 will be deviated in the lane width direction from the actual lane marking 21 in Figure 2. The LKA display 22 becomes more displaced from the actual lane marking 21 the farther away it is from the vehicle. The LDW display 23 is displaced in the lane width direction from the actual lane marking 21. This type of deviation in the LKA display 22 and the LDW display 23 from the actual lane marking 21 can be considered to be the result of a recognition error by the camera 1a on a curve with a cant.

[0032] Therefore, the display control unit 12 estimates the cant angle of the lane based on the detection result of the internal sensor 2 that detects the running state of the vehicle, and uses the cant angle to correct the position of the road shape image in the display image 20. As an example of estimating the cant angle, the display control unit 12 uses the vehicle speed, lateral acceleration, and yaw rate of the vehicle as the running state of the vehicle, and estimates the cant angle θ of the lane at the position of the vehicle based on an equation of motion that represents the balance between the lateral force caused by the turning of the vehicle and the lateral force acting on the vehicle due to the cant of the lane.

[0033] The lateral force Fy1 that accompanies a vehicle turning is the product of the vehicle's mass m and the vehicle's centrifugal acceleration a. Since the centrifugal acceleration a is equal to the product of the vehicle's speed v and the vehicle's angular velocity ω, the lateral force Fy1 is equal to the product of the vehicle's mass m, the vehicle's speed v, and the vehicle's angular velocity ω. The subscript y represents the y-axis in the vehicle-based coordinate system, which is the width direction of the vehicle.

[0034] The lateral force Fy2 acting on a vehicle due to lane cant can be calculated by approximating the state in which the vehicle is positioned tangentially on a curve with a cant angle θ to the decomposition of gravity on a simple slope. In the width direction of the vehicle, Fy2 is equal to the product of the vehicle mass m, the gravitational acceleration g, and sin θ.

[0035] By formulating an equation of motion that expresses the balance between the lateral force Fy, the lateral force Fy1, and the lateral force Fy2 corresponding to the lateral acceleration ay detected by the vehicle's internal sensor 2, substituting that the lateral force Fy is equal to the product of the vehicle's mass m and the lateral acceleration ay, and rearranging the equation for the cant angle θ, the following equation (1) is obtained. This cant angle θ is based on the vehicle's running state, and therefore represents the cant angle at the vehicle's position. Therefore, using the following equation (1), the cant angle θ at the vehicle's position can be estimated based on the vehicle's speed v, angular velocity ω, and lateral acceleration ay.

number

[0036] For example, the display control unit 12 uses the estimated cant angle θ to calculate clothoid parameters so as to simulate a change in the attitude of the camera 1a in the roll direction according to the cant angle θ, assuming that the reference lane marking position has been recognized in an image captured by the camera 1a in a horizontal attitude, and corrects the position of the road shape image in the display image 20. In the example of FIG. 2, the position of the road shape image refers to the positions of the LKA display 22 and the LDW display 23. For an image of a plurality of discretely arranged LKA displays 22, a predetermined position (e.g., the coordinates of one vertex) of the triangle image of each LKA display 22 may be used as the position of the road shape image. For an image of a line segment of the LDW display 23, a predetermined position (e.g., the coordinates of the center of the line segment) of the LDW display 23 may be used as the position of the road shape image.

[0037] The display control unit 12 may convert the reference lane marking position on the horizontal plane on which the vehicle is located into an estimated lane marking position on a plane with a cant angle θ by matrix calculation using the following formula (2): In formula (2), X, Y, and Z are the reference lane marking position (world coordinate system) on the horizontal plane on which the vehicle is located, Ex, Ey, and Ez are translation offset amounts indicating the offset of camera 1a from a predetermined reference position on the vehicle layout, a, b, c, ∼h, and i are rotational components (corresponding to the roll amount) of camera 1a calculated from the cant angle θ, ox and oy are the x and y coordinates of the center of display image 20, f is the focal length of camera 1a, kx and ky are the size of one pixel, and x, y, and z are the estimated lane marking position (world coordinate system) on the plane with a cant angle θ.

number

[0038] The display control unit 12 displays a display image on the display 3 of the vehicle. FIG. 3 is a diagram showing an example of correcting the position of lane markings in a display image. FIG. 3 shows a display image 30 representing a vehicle traveling around a right curve in a tunnel on a highway, as another example of a scene ahead of the vehicle. In FIG. 3, the display image 30 including the LKA display 31 (road shape image) is also a virtual image projected onto the windshield of the vehicle, and is superimposed on the scene ahead of the vehicle as viewed by the driver through the windshield. A pair of actual lane markings 32 exists in the actual scene ahead of the vehicle. The LKA display 31 is, for example, a triangular image similar to the LKA display 22 in FIG. 2, and is shown by a solid line in the example of FIG. 3.

[0039] In the example of FIG. 3 , the position of the solid-line LKA indicator 31 is a position corrected by the display control unit 12 using the estimated cant angle θ as described above. On the other hand, the position of the LKA indicator 33 is not corrected by the display control unit 12 but is a position calculated using the reference lane marking position recognized from the captured image. The LKA indicator 33 is, for example, a triangular image similar to the LKA indicator 22 in FIG. 2 and is indicated by a dashed line in the example of FIG. 3 . As shown in FIG. 3 , correcting the position of the LKA indicator 31 using the estimated cant angle θ reduces the amount of deviation from the position of the LKA indicator 33 relative to a pair of actual lane markings 32. Note that the dashed-dotted LKA indicator 33 is shown in FIG. 3 for comparison with the solid-line LKA indicator 31. The LKA indicator 33 does not necessarily have to be included in the actual display image 30.

[0040] Incidentally, in a situation where the vehicle approaches a curve from a straight line, the cant angle may be estimated in advance to be larger in accordance with the gradually increasing curvature of the curve. When the cant angle estimated at the vehicle position is smaller than a predetermined cant angle threshold and the rate of increase in the curvature of the lane marking in a predetermined section ahead of the vehicle is equal to or greater than a predetermined increase rate threshold, the display control unit 12 may estimate the cant angle so that the cant angle gradually increases with increasing distance forward from the vehicle position. The cant angle threshold is a cant angle threshold for determining whether the road surface at the vehicle position is straight or curved. The increase rate threshold is a curvature increase rate (positive curvature change rate) threshold for determining whether the road surface changes from straight to curved ahead of the vehicle position. The cant angle threshold and the increase rate threshold may be set in advance according to, for example, predetermined road design standards, etc.

[0041] For example, the display control unit 12 may estimate the cant angle so that the cant angle gradually increases with increasing distance forward from the vehicle position by adding a predetermined additional cant angle for each predetermined distance forward from the vehicle position based on the cant angle θ estimated at the vehicle position. The predetermined distance and the additional cant angle are correction parameters for estimating the cant angle in accordance with a curve whose curvature gradually increases. The predetermined distance and the additional cant angle can be set in advance in an actual vehicle test, a simulation, or the like, depending on the timing at which the condition for estimating the cant angle so as to gradually increase is satisfied and the change in the curvature of the curve over time after that timing.

[0042] Similarly, in a situation where the vehicle is approaching a straight road from a curve, the cant angle may be estimated in advance to be smaller in accordance with the gradually decreasing curvature of the curve. When the cant angle estimated at the vehicle position is equal to or greater than a cant angle threshold and the rate of decrease in the curvature of the lane marking in a predetermined section is equal to or greater than a predetermined decrease rate threshold, the display control unit 12 may estimate the cant angle so that the cant angle gradually decreases with increasing distance from the vehicle position forward. The decrease rate threshold is a threshold for the curvature decrease rate (negative curvature change rate) for determining whether the road surface changes from a curve to a straight road ahead of the vehicle position. The decrease rate threshold may be set in advance according to, for example, predetermined road design standards, etc.

[0043] For example, the display control unit 12 may estimate the cant angle so that the cant angle gradually decreases with increasing distance forward from the vehicle position by subtracting a predetermined subtraction cant angle from the cant angle estimated at the vehicle position for each predetermined distance forward from the vehicle position. The predetermined distance and the subtraction cant angle are correction parameters for estimating the cant angle in accordance with a curve whose curvature gradually decreases. The predetermined distance and the subtraction cant angle can be set in advance in an actual vehicle test, a simulation, or the like, depending on the timing at which the conditions for estimating the cant angle so that the cant angle gradually decreases are satisfied and the change in the curvature of the curve after that timing.

[0044] In addition, in a situation where the vehicle is neither approaching a curve from a straight road nor approaching a straight road from a curve as described above, the cant angle θ estimated at the vehicle position may be used as is.

[0045] [Display control ECU processing] Next, an example of the processing of the display control ECU 10 will be described with reference to Fig. 4 and Fig. 5. Fig. 4 is a flowchart showing an example of the processing of the display device according to the embodiment. Fig. 5 is a flowchart showing an example of the processing of estimating the cant angle in Fig. 4. The processing shown in Fig. 4 and Fig. 5 is performed repeatedly at a predetermined interval during operation of the display control ECU 10, for example.

[0046] As shown in FIG. 4, in S01, the display control ECU 10 causes the recognition generation unit 11 to recognize the lane markings and their positions based on the captured image. The recognition generation unit 11 recognizes the lane markings based on the captured image from the camera 1a. The recognition generation unit 11 recognizes the lane marking positions of the lane markings based on the captured image from the vehicle's camera 1a. The recognition generation unit 11 recognizes, for example, the reference lane marking position as the lane marking position.

[0047] In S02, the display control ECU 10 generates a display image using the recognition generation unit 11. The recognition generation unit 11 generates a display image including a road shape image representing the lane markings, for example, using the reference lane marking positions.

[0048] In S03, the display control ECU 10 acquires the vehicle running state using the display control unit 12. The display control unit 12 acquires, for example, the vehicle speed, the vehicle lateral acceleration, and the vehicle yaw rate as the vehicle running state based on the detection result of the internal sensor 2.

[0049] In S04, the display control ECU 10 estimates the cant angle using the display control unit 12. The display control ECU 10 performs the process of estimating the cant angle shown in FIG.

[0050] 5, in S11, the display control ECU 10 estimates the cant angle at the vehicle position using the display control unit 12. The display control unit 12 estimates the cant angle at the vehicle position according to, for example, the above equation (1).

[0051] In S12, the display control ECU 10 determines whether the cant angle estimated at the vehicle position by the display control unit 12 is smaller than the cant angle threshold value. If it is determined that the cant angle estimated at the vehicle position by the display control unit 12 is smaller than the cant angle threshold value (S12: YES), the display control ECU 10 proceeds to S13.

[0052] In S13, the display control ECU 10 determines whether the rate of increase in the curvature of the lane markings in a predetermined section ahead of the vehicle is equal to or greater than the increase rate threshold value, using the display control unit 12. If the display control unit 12 determines that the rate of increase in the curvature of the lane markings in a predetermined section ahead of the vehicle is equal to or greater than the increase rate threshold value (S13: YES), the display control ECU 10 proceeds to S14.

[0053] In S14, the display control ECU 10 causes the display control unit 12 to estimate the cant angle so that the cant angle gradually increases as the vehicle moves further forward from the vehicle position. For example, the display control unit 12 estimates the cant angle so that the cant angle gradually increases as the vehicle moves further forward from the vehicle position by adding a predetermined additional cant angle each time the vehicle moves a predetermined distance forward from the vehicle position, based on the cant angle estimated at the vehicle position. Thereafter, the display control ECU 10 ends the processing of FIG. 5 and returns to the processing of FIG. 4.

[0054] On the other hand, if the display control unit 12 determines that the rate of increase in the curvature of the lane markings in a predetermined section ahead of the vehicle is smaller than the increase rate threshold (S13: NO), the display control ECU 10 terminates the processing of Figure 5 and returns to the processing of Figure 4.

[0055] On the other hand, if it is determined that the cant angle estimated by the display control unit 12 at the vehicle position is equal to or greater than the cant angle threshold value (S12: NO), the display control ECU 10 proceeds to S15.

[0056] In S15, the display control ECU 10 determines whether the rate of decrease in the curvature of the lane markings in a predetermined section ahead of the vehicle is equal to or greater than the decrease rate threshold value, using the display control unit 12. If the display control unit 12 determines that the rate of decrease in the curvature of the lane markings in a predetermined section ahead of the vehicle is equal to or greater than the decrease rate threshold value (S15: YES), the display control ECU 10 proceeds to S16.

[0057] In S16, the display control ECU 10 causes the display control unit 12 to estimate the cant angle so that the cant angle gradually decreases as the vehicle moves further forward from the vehicle position. For example, the display control unit 12 estimates the cant angle so that the cant angle gradually decreases as the vehicle moves further forward from the vehicle position by subtracting a predetermined subtraction cant angle from the cant angle estimated at the vehicle position every time the vehicle moves further forward by a predetermined distance. Thereafter, the display control ECU 10 ends the processing of FIG. 5 and returns to the processing of FIG. 4.

[0058] On the other hand, if the display control unit 12 determines that the rate of decrease in the curvature of the lane marking in a predetermined section ahead of the vehicle is smaller than the rate of decrease threshold (S15: NO), the display control ECU 10 terminates the processing of Figure 5 and returns to the processing of Figure 4.

[0059] As described above, the display device 100 recognizes the lane markings 21 based on the image captured by the camera 1a of the vehicle. In the display image 20, the positions of the LKA indicator 22 and the LDW indicator 23 (road shape image) arranged along the recognized lane markings 21 are corrected using the cant angle θ of the lane estimated based on the detection results of the internal sensor 2 that detects the vehicle's traveling state. This makes it possible to suppress a cost increase compared to when a high-precision map is used, for example, while also suppressing positional deviations of the road shape image in the display image 20 obtained by image recognition using the camera 1a.

[0060] In the display device 100, the display control unit 12 uses the vehicle speed v, the lateral acceleration ay, and the angular velocity ω of the vehicle as the vehicle's running state, and estimates the cant angle θ of the lane at the vehicle's position based on an equation of motion that represents the balance between the lateral force Fy1 caused by the vehicle turning, the lateral force Fy2 acting on the vehicle due to the cant of the lane, and the lateral force Fy corresponding to the lateral acceleration of the vehicle, for example, by the above equation (1). As a result, the positions of the LKA display 22 and the LDW display 23 on the display image 20 can be corrected using the cant angle θ at the vehicle's position estimated using the vehicle speed v, the lateral acceleration ay, and the angular velocity ω of the vehicle.

[0061] In the display device 100, if the cant angle θ estimated at the vehicle position is smaller than a predetermined cant angle threshold and the rate of increase in the curvature of the lane marking 21 in a predetermined section ahead of the vehicle is equal to or greater than a predetermined increase rate threshold, the display control unit 12 estimates the cant angle so that the cant angle gradually increases with increasing distance forward from the vehicle position. As a result, for example, in a situation where the vehicle is approaching a curve from a straight line, the cant angle can be estimated to be larger in advance to match the curve with gradually increasing curvature. On the other hand, if the cant angle θ estimated at the vehicle position is equal to or greater than the cant angle threshold and the rate of decrease in the curvature of the lane marking 21 in the predetermined section is equal to or greater than a predetermined decrease rate threshold, the display control unit 12 estimates the cant angle so that the cant angle gradually decreases with increasing distance forward from the vehicle position. As a result, for example, in a situation where the vehicle is approaching a straight line from a curve, the cant angle can be estimated to be smaller in advance to match the curve with gradually decreasing curvature.

[0062] Although the embodiments of the present invention have been described above, the present invention is not limited to the above-described embodiments.

[0063] In the above embodiment, when the vehicle approaches a curve from a straight line, the display control unit 12 estimates the cant angle to be large in advance to match the curve's gradually increasing curvature, but this processing is not essential. When the vehicle approaches a curve from a straight line, the display control unit 12 estimates the cant angle to be small in advance to match the curve's gradually decreasing curvature, but this processing is not essential. For example, the cant angle may be uniformly the cant angle θ estimated at the vehicle position.

[0064] In the above embodiment, the display control unit 12 estimates the cant angle θ of the lane at the vehicle position, for example, using the above equation (1), but this example is not essential. The cant angle of the lane at the vehicle position may also be estimated using an equation other than the above equation (1). Furthermore, it is not essential that the cant angle θ be estimated based on an equation of motion that uses the vehicle speed v, the vehicle lateral acceleration ay, and the vehicle angular velocity ω as the vehicle's running state and expresses the balance between the lateral force associated with the turning of the vehicle and the lateral force acting on the vehicle due to the cant of the lane. For example, one or two of the vehicle speed, the vehicle lateral acceleration, and the vehicle yaw rate may be used as the vehicle's running state, or other physical quantities that express the vehicle's running state may be used. [Explanation of symbols]

[0065] 1a...camera (on-board camera), 2...internal sensor, 3...display (display unit), 11...recognition generation unit, 12...display control unit, 100...display device, ay...lateral acceleration, Fy, Fy1, Fy2...lateral force, v...vehicle speed, θ...cant angle.

Claims

1. A display device that displays lane markings on an in-vehicle display, a recognition and generation unit that recognizes the lane markings based on an image captured by an on-board camera of the vehicle and generates a display image including a road shape image arranged along the lane markings; a display control unit that displays the display image on a display unit of the vehicle, The display control unit estimates a cant angle of the lane based on a detection result of an internal sensor that detects the driving state of the vehicle, and corrects the position of the road shape image in the display image using the cant angle.

2. 2. The display device according to claim 1, wherein the display control unit uses the vehicle speed, the lateral acceleration, and the yaw rate of the vehicle as driving conditions of the vehicle, and estimates the cant angle of the lane at the position of the vehicle based on an equation of motion that represents a balance between a lateral force caused by turning of the vehicle, a lateral force acting on the vehicle due to cant of the lane, and a lateral force corresponding to the lateral acceleration of the vehicle.

3. The display control unit when the cant angle estimated at the position of the vehicle is smaller than a predetermined cant angle threshold and the rate of increase in the curvature of the lane marking in a predetermined section ahead of the vehicle is equal to or greater than a predetermined increase rate threshold, estimating the cant angle so that the cant angle gradually increases with increasing distance forward from the position of the vehicle, 3. The display device according to claim 2, wherein, when the cant angle estimated at the vehicle position is equal to or greater than the cant angle threshold and the rate of decrease in the curvature of the lane marking in the specified section is equal to or greater than a specified decrease rate threshold, the cant angle is estimated so that the cant angle gradually decreases as the distance from the vehicle position increases forward.

Images