Driving assistance device

Abstract

Provided is a driving assistance device that enables a user to intuitively grasp that a driving operation for avoiding an obstacle or a zone line was performed by automatic driving assistance. For example, the driving assistance device is configured to: display, on a liquid crystal display 4, a vehicle periphery image showing the periphery of the vehicle 2; and display a shield image 75 between the vehicle and an obstacle or division line to be avoided in the vehicle periphery image, in connection with a driving operation for, by automatic control, preventing the obstacle or division line in the periphery of the vehicle from being approached.

Description

Driving assistance devices

[0001] The present invention relates to a driving assistance device that assists driving of a vehicle.

[0002] In recent years, in addition to manual driving, in which a vehicle is driven based on a user's driving operation, new automated driving assistance systems have been proposed that assist a user in driving a vehicle by having the vehicle perform some or all of the user's driving operations.The automated driving assistance system, for example, constantly detects the current position of the vehicle, the lane in which the vehicle is traveling, and the positions of other vehicles in the vicinity, and automatically controls the vehicle, including steering, drive sources, and braking, so that the vehicle travels along a predetermined route.

[0003] Here, when avoiding an obstacle during manual driving, the user recognizes the obstacle and selects the operation to avoid the obstacle and performs the steering or braking operation themselves, so that steering or braking (deceleration) that the user does not expect is basically not performed. On the other hand, when driving with the above-mentioned automatic driving assistance, the steering or braking operation to avoid the obstacle is performed automatically, so that the steering or braking operation that the user does not expect may be performed, particularly in situations where the user is not paying much attention to the surroundings, which may confuse the user. As a means to solve this, for example, as disclosed in Japanese Patent Application Laid-Open No. 2004-240480, a method is considered in which an image of a three-dimensional object is superimposed on the position of the obstacle included in the surroundings image displayed on the in-vehicle monitor, thereby allowing the user to be aware of the presence of the obstacle in advance.

[0004] JP 2004-240480 A (paragraphs 0079-0087)

[0005] The aforementioned Patent Document 1 also discloses changing the display color of a three-dimensional object or flashing it when the vehicle approaches an obstacle, with the primary purpose of allowing the user to recognize the presence of the obstacle without overlooking it. However, while changing the display color or flashing the display can make it easier to recognize the presence of an obstacle, they do not provide guidance that suggests a steering or braking operation has been performed. Therefore, even if the autonomous driving assistance system automatically performs steering or braking operations to avoid the obstacle while the three-dimensional object shown in Patent Document 1 is displayed, there is still a problem of confusing the user due to unexpected steering or braking operations. Furthermore, when driving with autonomous driving assistance, the vehicle basically stays within the lane markings in sections with lane markings. Therefore, at forks and curves, for example, steering and braking operations relative to the lane markings are automatically performed to ensure driving along the lane markings. Therefore, similar problems have arisen not only with obstacles but also with lane markings.

[0006] The present invention has been made to solve the above-mentioned problems in the conventional art, and aims to provide a driving assistance device that allows the user to intuitively understand that a driving operation to avoid an obstacle or marking line has been performed through automatic driving assistance by displaying an image of an obstruction between the vehicle and the obstacle or marking line in connection with the driving operation to avoid the obstacle or marking line through automatic driving assistance.

[0007] To achieve the above object, the driving assistance device of the present invention automatically controls at least a portion of a vehicle's driving operation. The device displays a vehicle surroundings image showing the vehicle's surroundings on a display device, and, in association with a driving operation to avoid approaching an obstacle or lane marking around the vehicle through the automatic control, displays an obstruction image between the vehicle and the obstacle or lane marking to be avoided in the vehicle surroundings image. Note that "automated driving assistance" refers to a function that performs or assists at least a portion of the driver's driving operation of the vehicle on behalf of the driver. The "vehicle surroundings image" may be a real image of the vehicle's surroundings captured by an imaging device such as a camera, or a virtual image reproducing the vehicle's surroundings using CG (computer graphics). Furthermore, if the image is a real image, it may be the captured image itself, or an image obtained by processing the captured image. For example, it may be an image obtained by combining images captured by multiple cameras or by converting the viewpoint. Furthermore, "displaying an obstruction image in connection with a driving operation to avoid approaching an obstacle or a dividing line around the vehicle" may mean displaying an obstruction image at the time a driving operation to avoid approaching an obstacle or a dividing line is performed, or displaying an obstruction image just before a driving operation to avoid approaching an obstacle or a dividing line is performed, or displaying an obstruction image after a driving operation to avoid approaching an obstacle or a dividing line is performed.

[0008] According to the driving assistance device of the present invention having the above configuration, by displaying an obstruction image between the vehicle and the obstacle or lane marking in association with the driving operation to avoid the obstacle or lane marking through automated driving assistance, the user can intuitively understand that the driving operation to avoid the obstacle or lane marking has been performed through automated driving assistance. As a result, the user will not be confused even if the driving operation to avoid the obstacle or lane marking has been performed automatically.

[0009] 16 is a schematic configuration diagram of a vehicle according to a first embodiment. FIG. 17 is a block diagram showing the configuration of a driving assistance device according to the first embodiment. FIG. 18 is a flowchart of a driving assistance processing program according to the first embodiment. FIG. 19 is a diagram explaining a method of converting a captured image into a bird's-eye image. FIG. 20 is a diagram explaining a method of generating a bird's-eye image. FIG. 21 is a diagram showing an example of a support image displayed on a liquid crystal display. FIG. 22 is a diagram showing an example of an object image arrangement relative to a detected obstacle. FIG. 23 is a diagram showing a location where an object image is arranged in a bird's-eye image. FIG. 24 is a diagram showing the relationship between an object image and a virtual light source. FIG. 25 is a diagram showing a modified example of an object image. FIG. 26 is a diagram showing an example of a support image displayed on a liquid crystal display when a steering operation to avoid an obstacle is not performed. FIG. 27 is a diagram explaining a method of setting a range for deforming the appearance of an object image. FIG. 28 is a diagram showing an example of a support image displayed on a liquid crystal display when a steering operation to avoid an obstacle is performed. FIG. 29 is a flowchart of a driving assistance processing program according to a second embodiment. FIG. 29 is a diagram explaining a virtual viewpoint of a bird's-eye image. FIG. 21 is a diagram explaining a virtual viewpoint of a bird's-eye image after movement. FIG. 22 is a diagram explaining a field-of-view radius and a field-of-view angle of a virtual viewpoint. 10 is a diagram showing an example of a bird's-eye view image displayed after changing the field of view radius and field of view angle; FIG. 11 is a diagram showing an example of a bird's-eye view image displayed on a liquid crystal display when a driving operation is performed to avoid approaching an obstacle; FIG. 12 is a diagram explaining a method for generating an obstruction image; FIG. 13 is a diagram showing an example of a bird's-eye view image displayed when the shape of the obstruction image is not drawn to follow the shape of the host vehicle image; FIG. 14 is a diagram showing an example of a bird's-eye view image displayed when the shape of the obstruction image is drawn to follow the shape of the host vehicle image; FIG. 15 is an example of a display of an obstruction image that is taller than the host vehicle image; FIG. 16 is an example of a display of an obstruction image that is shorter than the host vehicle image; FIG. 17 is a diagram explaining a modified example; FIG. 18 is a diagram explaining a modified example;

[0010] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS A first embodiment and a second embodiment of a driving assistance device according to the present invention will be described in detail below with reference to the drawings.

[0011] First Embodiment First, a vehicle 2 equipped with a driving assistance device 1 according to a first embodiment will be described below. Fig. 1 is a schematic diagram of a vehicle 2 according to the first embodiment.

[0012] Here, the vehicle 2 may be, for example, an automobile (internal combustion engine automobile) that uses an internal combustion engine (engine, etc.) as a drive source, an automobile (electric automobile, fuel cell automobile, etc.) that uses an electric motor (motor, etc.) as a drive source, or an automobile that uses both of these as drive sources (hybrid automobile). Furthermore, the vehicle type is not limited, and may be a standard car, a large commercial truck, a bus, construction machinery, or the like. Furthermore, although the following description will be of a four-wheeled automobile, it may also be a two-wheeled or three-wheeled vehicle.

[0013] However, vehicle 2 is a vehicle capable of manual driving, in which the vehicle drives based on the user's driving operation, as well as assisted driving using automatic driving assistance, in which the vehicle drives automatically without the user's driving operation.

[0014] Furthermore, the autonomous driving assistance may be performed only under specific circumstances, such as when parking or leaving a parking lot, or may be performed for all road sections, or may be performed only while the vehicle is traveling on a specific road section (for example, a highway with a gate (manned or unmanned, toll or free) at the boundary). In the following description, the autonomous driving section in which the autonomous driving assistance of the vehicle is performed includes all road sections, including general roads and highways, as well as parking lots. Furthermore, the autonomous driving assistance is performed only when the user selects to perform the autonomous driving assistance (for example, by turning on the autonomous driving start button) and it is determined that autonomous driving assistance is possible. On the other hand, the vehicle 2 may be a vehicle that is only capable of assisted driving with autonomous driving assistance. Alternatively, when an obstacle (for example, a person, a bicycle, another vehicle, a wall, etc.) is detected in an assistance target area set in the space ahead of the vehicle 2, autonomous driving assistance such as steering or braking to avoid the obstacle may be performed as assistance (safety device) to the driver regarding the obstacle, regardless of whether it is triggered by a user selection.

[0015] In the vehicle control in the automated driving assistance of the first embodiment, for example, the current position of the vehicle, the lane the vehicle is traveling in, and the positions of surrounding obstacles are detected as needed, and vehicle control, such as steering, drive source, and braking, is automatically performed so that the vehicle travels along the generated travel trajectory at a speed in accordance with the generated speed plan. However, only the steering operation may be performed automatically, and the drive source and brakes may be controlled manually. Furthermore, the automated driving assistance for the vehicle of the first embodiment also includes lane departure prevention assistance. When lane departure prevention assistance is performed, if lane departure lines (e.g., roadway centerlines, lane boundaries, roadway outer lines, parking space lines, etc.) are detected around the vehicle 2, automated driving assistance such as steering and braking is also performed to avoid departure from the lane departure lines. Furthermore, when the automated driving assistance is performed, a scene around the vehicle (which may be a real scene or a virtual CG scene) is displayed on an in-vehicle display, as described below. If an obstacle is present around the vehicle, an object image indicating the presence of the obstacle is superimposed on the obstacle's position within the scene.

[0016] 1, the vehicle 2 has an operation unit 3 that accepts operations from the occupant, a liquid crystal display 4 that displays bird's-eye and overhead images of the vehicle's surroundings and other information related to driving assistance to the occupant, a speaker 5 that outputs audio guidance related to driving assistance, a front camera 6, a rear camera 7, and side cameras 8A and 8B for capturing images of the vehicle's surroundings, ultrasonic sensors 9A to 9L that detect obstacles around the vehicle, and a driving assistance ECU (electronic control unit) 10 that performs various calculations based on input information. The driving assistance ECU 10 is collectively referred to as the driving assistance device 1.

[0017] Each component of the vehicle 2 will be described below. First, the operation unit 3 is arranged, for example, on the front of the steering wheel (also called the steering wheel), and includes an operation button that is operated when starting automatic driving assistance. By operating the operation unit 3, the user can switch between manual driving, in which the vehicle drives based on the user's driving operation, and automatic driving assistance, in which the vehicle drives automatically without the user's driving operation. The operation unit 3 may have a touch panel provided on the front of the liquid crystal display 4. It may also have a microphone and a voice recognition device.

[0018] The liquid crystal display 4 is a type of display device and is attached to the instrument panel of the vehicle 2. During autonomous driving assistance, the liquid crystal display 4 displays bird's-eye and overhead images of the vehicle's surroundings, which are generated by performing viewpoint conversion and synthesis processing on images captured by the front camera 6, rear camera 7, and side cameras 8A and 8B. If there is an obstacle around the vehicle 2, an object image showing a three-dimensional shape of the obstacle at the position in the bird's-eye image is also displayed. Details of the object image will be described later. The liquid crystal display 4 may also be used for a navigation device.

[0019] The speaker 5 is mounted on the instrument panel of the vehicle 2 and outputs voice guidance and warning sounds related to driving assistance. The speaker 5 may also be used for the navigation device.

[0020] The forward camera 6 is an imaging device having a camera using a solid-state imaging element such as a CCD, and is installed, for example, above the front bumper of the vehicle 2 or behind the rearview mirror, with its optical axis facing forward in the direction of travel of the vehicle.

[0021] The rear camera 7 is an imaging device having a camera that also uses a solid-state imaging element such as a CCD, and is installed, for example, near the upper center of the license plate attached to the rear of the vehicle 2, with its optical axis facing toward the rear of the vehicle.

[0022] Furthermore, the side cameras 8A and 8B are imaging devices having cameras that similarly use solid-state imaging elements such as CCDs, and are installed, for example, on the left and right side mirrors of the vehicle 2 with their optical axes facing out to the sides of the vehicle.

[0023] The driving assistance ECU 10 generates bird's-eye and overhead images of the surroundings of the vehicle by performing viewpoint conversion and synthesis processing on the images captured by the front camera 6, rear camera 7, and side cameras 8A and 8B. During execution of autonomous driving assistance, the driving assistance ECU 10 also performs image recognition processing on the captured images to detect lane markings and obstacles (other vehicles, pedestrians, bicycles, walls, guardrails, and other structures) around the vehicle, and executes autonomous driving assistance based on the detection results.

[0024] The ultrasonic sensors 9A-9L are positioned at predetermined intervals on the front, rear, and sides of the vehicle 2. They transmit ultrasonic waves as search waves around the vehicle 2 and detect objects around the vehicle by receiving reflected waves from the search waves. Specifically, they are a type of distance measurement sensor that can detect the distance (measured distance) to the object that reflected the search wave by measuring the time from transmission to reception. The ultrasonic sensors 9A-9L are also configured to generate output signals (including the distance to the detected object) corresponding to the reception results of the received waves and output them to the control unit. Examples of objects that can be detected by the ultrasonic sensors 9A-9L include obstacles that the vehicle 2 must avoid while traveling, such as people, bicycles, other vehicles, and walls. Instead of ultrasonic sensors, millimeter-wave sensors or laser sensors may also be used as distance measurement sensors. Furthermore, in the first embodiment, as described above, obstacles can also be detected by performing image recognition on the images captured by the front camera 6, rear camera 7, and side cameras 8A and 8B. Therefore, if obstacle detection is performed by the above cameras, the ultrasonic sensors 9A to 9L may be omitted.

[0025] Meanwhile, the driving assistance ECU 10 is an electronic control unit that performs various processes related to autonomous driving assistance. For example, it constantly detects the vehicle's current position, the lane the vehicle is traveling on, and the positions of surrounding obstacles, and controls the vehicle, such as steering, drive source, and braking, to travel along the generated travel trajectory at a speed according to the generated speed plan. The LCD display 4 also displays a scene around the vehicle (which may be a real scene or a virtual CG scene), and if an obstacle is present around the vehicle, it also displays an object image indicating the presence of the obstacle superimposed on the scene. The driving assistance ECU 10 is connected to the operation unit 3, LCD display 4, speaker 5, front camera 6, rear camera 7, side cameras 8A and 8B, and ultrasonic sensors 9A to 9L via an in-vehicle network such as a CAN. It is also connected to various sensors installed in the vehicle 2, such as a vehicle speed sensor, acceleration sensor, and steering sensor, as well as an in-vehicle navigation system. The detailed configuration of the driving assistance ECU 10 will be described later.

[0026] In addition, vehicle 2 has basic components as vehicle 2 in addition to the components shown in Figure 1, but we will only explain the configuration related to the control of automatic driving assistance and the control related to that configuration.

[0027] Next, a detailed description will be given of the driving assistance ECU 10, in particular, of the driving assistance device 1 provided in the vehicle 2. Fig. 2 is a block diagram showing the configuration of the driving assistance device 1 according to the first embodiment.

[0028] As shown in FIG. 2 , the driving assistance ECU (electronic control unit) 10 is an electronic control unit that controls the entire driving assistance device 1. The ECU 10 includes internal storage devices such as a CPU 31 that functions as a calculation device and a control device, a RAM 32 that is used as a working memory when the CPU 31 performs various calculation processes and that stores route data and the like used when a route is searched, a ROM 33 that stores control programs as well as a driving assistance processing program (see FIG. 3 ) described below, and a flash memory 34 that stores programs read from the ROM 33. The driving assistance ECU 10 has various functions as processing algorithms, such as a function to detect obstacles around the vehicle, a function to display an object image superimposed around an obstacle detected around the vehicle, a function to automatically control at least a part of vehicle operation, a function to change the shape of an object image displayed around an obstacle to be avoided when the vehicle is operated to avoid the obstacle by an automatic driving control means, and a function to determine the range of the outer periphery to be deformed for the displayed object image.

[0029] The driving assistance ECU 10 is also connected to various sensors 36 for detecting vehicle behavior, such as a vehicle speed sensor, an acceleration sensor, and a steering sensor, as well as various vehicle drive units 37, such as the steering, brakes, and accelerator, and detects the current vehicle behavior based on the detection results of these sensors 36, while controlling the various drive units 37 to provide automatic driving assistance for the vehicle 2. Specific details of the automatic driving assistance include, for example, detecting the current vehicle position, the lane the vehicle is traveling on, and the positions of surrounding obstacles at any time, and controlling the steering, drive source, brakes, and other vehicle components so that the vehicle travels along the generated travel trajectory at a speed in accordance with the generated speed plan. However, it is also possible to automatically perform only the steering operation, while manually controlling the drive source and brakes.

[0030] The flash memory 34 also includes a vehicle information DB 35, which stores various information related to the vehicle 2. For example, the vehicle information DB 35 stores the installation positions (height from the ground and left-right position) of the cameras and ultrasonic sensors 9A to 9L installed on the vehicle 2, the detection axis X, the overall length, the vehicle width, the wheelbase, the minimum turning radius, etc. This information is input in advance by the occupant or a person from the vehicle manufacturer.

[0031] Next, a driving assistance processing program executed by the driving assistance ECU 10 in the driving assistance device 1 having the above configuration will be described with reference to Fig. 3. Fig. 3 is a flowchart of the driving assistance processing program according to the first embodiment. Here, the driving assistance processing program is executed after the ACC (accessory power supply) of the vehicle 2 is turned on, and is a program that provides assistance to the user using bird's-eye images and overhead images of the vehicle surroundings while automatic driving assistance is being performed for the vehicle. The program shown in the flowchart in Fig. 3 below is stored in the RAM 32 and ROM 33 provided in the driving assistance device 1, and is executed by the CPU 31.

[0032] First, in step (hereinafter abbreviated as S) 1, the CPU 31 determines whether assisted driving by automatic driving assistance is being performed. In the first embodiment, assisted driving by automatic driving assistance is performed when the user selects to perform automatic driving assistance by operating the operation unit 3 and it is determined that driving by automatic driving assistance is possible. The automatic driving assistance includes detecting the current position of the vehicle, the lane in which the vehicle is traveling, and the positions of surrounding lane markings and obstacles as needed, and automatically controlling the vehicle, including the steering, drive source, and brakes, so that the vehicle travels along the generated travel trajectory at a speed in accordance with the generated speed plan. However, it is also possible to automatically perform only the steering operation and manually control the drive source and brakes.

[0033] If it is determined that assisted driving by the automated driving assistance is being performed (S1: YES), the process proceeds to S2. On the other hand, if it is determined that assisted driving by the automated driving assistance is not being performed (S1: NO), the driving assistance processing program is terminated.

[0034] In S2, the CPU 31 generates a bird's-eye image looking diagonally downward from the sky and a bird's-eye image looking vertically downward from the sky based on real-time images captured by the front camera 6, the rear camera 7, and the side cameras 8A and 8B. The bird's-eye image generation method will be described below as an example. As shown in FIG. 4 , the real-time images captured by each camera are projected onto a virtual projection plane, which is a horizontal plane corresponding to the height of the ground surface. The projected images on the virtual projection plane are then converted into images viewed from a virtual viewpoint looking vertically downward from above the vehicle 2, thereby generating the bird's-eye image of each camera. The conversion to the image viewed from the virtual viewpoint (viewpoint conversion) is performed by first converting the coordinates of the captured image coordinate system, which is set along a plane perpendicular to the optical axis of the camera, into coordinates of a ground coordinate system, which is set along the ground surface, and then converting them into coordinates of the bird's-eye image coordinate system. The conversion formulas used for each coordinate conversion are already known, so a description thereof will be omitted. 5, a bird's-eye view image 41 obtained by viewpoint-converting an image captured by the front camera 6, a bird's-eye view image 42 obtained by viewpoint-converting an image captured by the rear camera 7, a bird's-eye view image 43 obtained by viewpoint-converting an image captured by the side camera 8A, and a bird's-eye view image 44 obtained by viewpoint-converting an image captured by the side camera 8B are synthesized (spliced ​​together), and a host vehicle image 45, which schematically shows the appearance of the host vehicle, is inserted between the bird's-eye view images 41 to 44 to generate a bird's-eye view image. Note that the process for generating the bird's-eye view image is essentially the same as that for generating the bird's-eye view image, except that the angle of the line of sight during viewpoint conversion is different, and therefore a description thereof will be omitted. Furthermore, if the bird's-eye view image is to be a bird's-eye view image looking diagonally down ahead in the direction of travel, the bird's-eye view image 42 obtained by viewpoint-converting an image captured by the rear camera 7 may be excluded from the synthesis target (i.e., an image of only the area ahead in the direction of travel from the vehicle position may be used). Conversely, if a bird's-eye view image looking diagonally down behind the vehicle when reversing is to be obtained, the bird's-eye view image 41 obtained by converting the viewpoint of the image captured by the front camera 6 may be excluded from the synthesis target (i.e., the image may be only behind the vehicle position).

[0035] Thereafter, in S3, the CPU 31 displays the real-time bird's-eye view image and overhead view image, which are generated in S2 and show the current environment around the vehicle, on the liquid crystal display 4. In the first embodiment, both the bird's-eye view image and the overhead view image are displayed simultaneously on the liquid crystal display 4, but it is also possible to display only one of them, or to display them in a switchable manner by a user operation.

[0036] FIG. 6 shows an example of the support image 51 displayed on the LCD display 4 in S3. As shown in FIG. 6, the support image 51 is divided into two screens, with a bird's-eye view image 52 displayed on the left and a bird's-eye view image 53 displayed on the right. Note that FIG. 6 particularly illustrates a case in which the vehicle is moving forward, and the bird's-eye view image 52 displays a bird's-eye view of the area ahead in the direction of travel, viewed diagonally downward. Furthermore, the bird's-eye view image 52 and the bird's-eye view image 53 also display a vehicle image 45 that schematically illustrates the vehicle. The vehicle image 45 indicates the current position of the vehicle in the bird's-eye view image 52 or the bird's-eye view image 53. As a result, the user can clearly understand the environment around the vehicle, including areas that are difficult to see directly, by clarifying the relative position of the vehicle. The CPU 31 may also calculate the vehicle's future movement trajectory based on detection values ​​from a vehicle speed sensor, a steering sensor, etc., and superimpose the movement trajectory on the bird's-eye view image 52 or the bird's-eye view image 53. Thereafter, the support image 51 continues to be displayed until the assisted driving by the automatic driving assistance is ended (S10: YES).

[0037] Next, in S4, the CPU 31 detects obstacles around the vehicle. If an obstacle is present, the CPU 31 also detects the position and area of ​​the obstacle in the bird's-eye view image and the overhead view image generated in S2. Note that an obstacle here refers to an object that the vehicle must avoid while traveling, such as another vehicle, a pedestrian, a bicycle, a wall, a guardrail, or other structure. Passable steps and the like are excluded. Obstacles may be detected by performing image recognition on the images captured by the front camera 6, the rear camera 7, and the side cameras 8A and 8B, or by ultrasonic sensors 9A to 9L, or by a combination of both. Note that obstacles can be detected from the captured images by, for example, performing brightness correction based on the brightness difference between the road surface and obstacles on the road surface, followed by binarization processing to separate the obstacle from the image, geometric processing to correct distortion, and smoothing processing to remove noise from the image, thereby detecting the boundary between the road surface and the obstacle. The type of obstacle may also be detected using a known template matching process, feature point detection process, etc. Furthermore, the image recognition process for the captured image is not limited to the above example, and may be performed using machine learning, for example. If no obstacle is detected in S4, the process from S5 onwards is not executed, and the process proceeds to S10.

[0038] Next, in S5, the CPU 31 determines the location, color, shape, and size of an object image to be superimposed on the bird's-eye view image and overhead view image generated in S2, particularly the bird's-eye view image. Here, the object image is an image that indicates the position of an obstacle in the bird's-eye view image, and specifically, an image of a three-dimensional (3D) pole standing on the road surface. Note that in the first embodiment, the object image is superimposed on only the bird's-eye view image, but the object image may also be superimposed on the overhead view image.

[0039] First, the "arrangement of the object image" is determined based on the position of the obstacle in the bird's-eye image detected in S4. Specifically, as shown in FIG. 7A, the object image 61 may be arranged so that an edge of the object image 61 is located relative to the position of the obstacle 60 detected in the bird's-eye image in S4, and the object image 61 is located on the vehicle's side relative to the obstacle 60. Alternatively, as shown in FIG. 7B, the object image 61 may be arranged so that the edge of the object image 61 is located relative to the position of the obstacle 60 detected in the bird's-eye image in S4, and the object image 61 is located on the opposite side of the obstacle 60 from the vehicle's position, as shown in FIG. 7C. However, in order to make the occupant more aware of the obstacle 60, the mode (a) is preferable.

[0040] In the following, an example of the (a) embodiment will be described. The area near the boundary between the area in the bird's-eye view image occupied by the obstacle detected in S4 and the area in which the vehicle is traveling (the road or passage) is determined as the location where the object image should be placed. For example, in the bird's-eye view image 52 shown in Fig. 8, a situation in which a vehicle is traveling down a passage in a parking lot includes two parked vehicles on the left and right front of the vehicle as obstacles. Therefore, line segment 55, which is the boundary between the parked vehicle on the left front and the passage, and line segment 56, which is the boundary between the parked vehicle on the right front and the passage, are determined as the locations where the object images should be placed. The object images are placed on the determined line segments so as to surround the obstacles at regular intervals (e.g., 30 cm intervals).

[0041] Next, regarding the "color of the object image," for example, all may be determined to be the same color (e.g., green), but it is preferable to change the color of the object image depending on the distance from the vehicle. For example, object images within a predetermined distance from the vehicle indicate obstacles that the user should pay particular attention to, so they are determined to be red or a flashing color, while other object images are determined to be green. In addition, in the first embodiment, as described below, a process (S7) is performed to change the shape of the object image displayed around the obstacle to be avoided. Alternatively, object images whose shape has changed may be determined to be red or a flashing color, while other object images may be determined to be green. Transparency is also determined, and to prevent the obstacle from being obscured by the object image, the transparency is set to semi-transparent (e.g., 50%). However, because object images within a predetermined distance from the vehicle and object images whose shape has changed indicate obstacles that the user should pay particular attention to, the transparency may be lowered (e.g., 20%).

[0042] Furthermore, for object images, textures are applied to the surfaces to express the texture of the object surface. However, the position of the virtual light source that determines the brightness of the texture surface is not set to a single common light source for all object images, but rather a different light source is set for each object image. For example, when object images 61A to 61C are arranged side by side as shown in FIG. 9, virtual light source 62A is set for object image 61A, virtual light source 62B is set for object image 61B, and virtual light source 62C is set for object image 61C. Basically, these are set so that light hits them from the side. Here, if only one virtual light source is set for multiple object images 61A to 61C, for example, when multiple object images 61A to 61C are arranged side by side as shown in FIG. 9, a problem occurs in which some object images are poorly illuminated and the texture is not expressed. However, in the first embodiment, a virtual light source is set for each object image, thereby preventing this problem from occurring.

[0043] The light intensity of the virtual light source may also be changed over time. For example, it is possible to display a scene that alternates between a realistic representation with a strong light source and an abstract representation with a weak light source at regular intervals. This prevents users from mistaking the object image for an actual structure.

[0044] Next, the "shape of the object image" is basically a pole (cylinder) shape as shown in Fig. 9. However, the shape of the object image is not limited to a pole shape, and may be, for example, a rectangular parallelepiped shape or a cone shape. Furthermore, in the first embodiment, the object image is a plurality of object images 61A to 61C arranged at regular intervals (for example, 30 cm intervals) so as to surround the periphery of the obstacle, but it may also be, for example, a single continuous wall-shaped object image 61D as shown in Fig. 10. Note that if the object image 61D is a wall-shaped object image, the object image 61D is arranged so as to surround the periphery of the obstacle.

[0045] Finally, the "size of the object image" is set appropriately within a range that does not obstruct the user's field of view. For example, if the object image is in the shape of a pole (cylinder) as shown in Figure 9, the pole will be 30 cm in diameter and 1 m in height (not the size on the screen but the size in the actual scene). However, although the size of the object images is the same, the size of the object images placed on the near side (closer to the vehicle) on the screen is determined to be larger. In other words, the size on the screen of the object images displayed in the bird's-eye view image is determined depending on the position where they are placed.

[0046] Furthermore, if the type of obstacle can be detected, the color, shape, size, etc. of the object image around the obstacle may be changed depending on the type of obstacle. For example, for a more dangerous obstacle (such as a pedestrian), the object image around the obstacle may be made bolder or flashing to make it stand out.

[0047] Thereafter, in S6, the CPU 31 acquires the control content (the type of automatic driving assistance performed under what circumstances) executed in the automatic driving assistance process performed in parallel with this program, and determines, in particular, whether a steering operation for avoiding an obstacle was performed by the automatic driving assistance. Note that, regarding the steering operation, a condition may be that a steering operation of a threshold angle or more or a steering operation of a threshold speed or more was performed.

[0048] If it is determined that a steering operation to avoid an obstacle has been performed (S6: YES), the process proceeds to S7. On the other hand, if it is determined that a steering operation to avoid an obstacle has not been performed (S6: NO), the process proceeds to S9.

[0049] In S9, which is executed when it is determined that a steering operation to avoid an obstacle has not been performed, the CPU 31 superimposes and displays an object image with the arrangement, color, shape, and size determined in S5 on the bird's-eye view image 52 displayed on the LCD display 4 in S3. As a result, for example, in the bird's-eye view image 52 shown in FIG. 11, a situation in which a vehicle is traveling down an aisle in a parking lot includes parked vehicles to the left and right of the vehicle as obstacles. Therefore, an object image 61 is superimposed and displayed at the boundary between the parked vehicle to the left and the aisle, and another object image 61 is superimposed and displayed at the boundary between the parked vehicle to the right and the aisle. Note that the object images 61 are arranged at regular intervals (e.g., 30 cm intervals) to surround the obstacle in the bird's-eye view image 52. As a result, a user who views the support image 51 can clearly grasp the presence of the obstacle, including its position relative to the vehicle. 11, the object image 61 is displayed superimposed only on the bird's-eye view image 52, but the object image may also be displayed superimposed on the overhead view image 53. Then, the process proceeds to S10.

[0050] Meanwhile, in S7, the CPU 31 changes the shape of the object image displayed around the obstacle to be avoided by steering. Specifically, as shown in FIG. 12 , the shape is changed by recessing a portion of the outer periphery of the object image 61, which represents a three-dimensional shape. The width and depth of the recessed area may be fixed or may be varied depending on the steering angle and steering speed. For example, the larger the steering angle or the faster the steering speed to avoid the obstacle, the wider and deeper the recess. When recessing the object image, it is desirable to first thicken the object image (e.g., 1.2 times the diameter) to prevent the object image from becoming too thin. When recessing the object image, it is desirable to gradually change the recession rather than switching from a non-recessed state to a recessed state all at once. The extent of the recessed area (the direction of the recession) of the outer periphery of the object image is adjusted in S8, described later, and is therefore fixed in S7.

[0051] Next, in S8, the CPU 31 rotates the object image displayed around the obstacle to be avoided by the steering operation around the central axis, thereby adjusting the range of the outer peripheral surface of the object image to be recessed (the direction of recession) (i.e., the range of the outer peripheral surface to be deformed is determined by the extent of the rotation).

[0052] In the first embodiment, the range of the outer peripheral surface to be recessed is changed in accordance with a change in the positional relationship between the vehicle and the object image. Specifically, the criteria for determining the range of the outer peripheral surface to be recessed differ between an object image located on the forward side of the current position of the vehicle and an object image located on the reverse side of the current position of the vehicle.

[0053] As an example, in the example shown in FIG. 13 , for object images 61A to 61C located further in the traveling direction than the current position of the vehicle (more precisely, the front end of the vehicle when moving forward, and the rear end when moving backward), the surface facing the vehicle (more precisely, the center front end of the vehicle when moving forward, and the center rear end when moving backward) is determined to be recessed, and the object image is rotated according to the determination. Therefore, as the position of the vehicle changes, the recessed range of object images 61A to 61C also changes accordingly. On the other hand, for object images 61D to 61F located in the opposite direction of the traveling direction than the current position of the vehicle, a fixed range (for example, the surface opposite an obstacle) is determined to be recessed regardless of the vehicle position, and the object image is rotated according to the determination. Note that FIG. 13 shows the case where the vehicle is moving forward.

[0054] However, the example shown in FIG. 13 is merely an example, and the criteria for determining the range of depression in the object image can be changed as appropriate. For example, a fixed range (e.g., the surface opposite the obstacle) may be determined to be depressed for all object images on the screen regardless of the current position of the vehicle (deformation pattern 1). Alternatively, a surface facing the vehicle (more precisely, the center front end of the vehicle when moving forward, and the center rear end when moving backward) may be determined to be depressed for each object image on the screen (deformation pattern 2). Alternatively, a surface facing the closest part of the vehicle (closest point) may be determined to be depressed for each object image on the screen (deformation pattern 3). Alternatively, deformation pattern 3 may be used for object images located further in the traveling direction than the current position of the vehicle (more precisely, the front end of the vehicle when moving forward, and the rear end when moving backward), and deformation pattern 1 may be used for object images located in the opposite direction from the current position of the vehicle.

[0055] Then, in S9, the CPU 31 superimposes and displays an object image having the arrangement, color, shape, and size determined in S5 on the bird's-eye view image 52 displayed on the liquid crystal display 4 in S3. Furthermore, the object image displayed around an obstacle to be avoided by steering is superimposed and displayed as the object image deformed in S7 and rotated in S8. As a result, for example, in the bird's-eye view image 52 shown in FIG. 14, a situation in which a vehicle is traveling down an aisle in a parking lot includes parked vehicles to the left and right of the vehicle as obstacles. Therefore, an object image 61 is superimposed and displayed on the boundary between the parked vehicle to the left and the aisle, and a similar object image 61 is superimposed and displayed on the boundary between the parked vehicle to the right and the aisle. Furthermore, particularly in the case of autonomous driving assistance, when a steering operation is performed to avoid a parked vehicle to the left and the aisle, the object image 61 at the boundary between the parked vehicle to the left and the aisle is displayed as a recessed image, as shown in FIG. 14. By denting the object image 61, the user can intuitively understand that the most recent automatic steering operation to turn right was a steering operation to avoid a parked vehicle ahead on the left. Note that the dent in the object image 61 that occurs in conjunction with the steering operation may be returned to a non-dented state after a certain period of time has elapsed, or may remain dented until it disappears off the screen.

[0056] Thereafter, in S10, the CPU 31 determines whether or not to end assisted driving by the automatic driving assistance. Here, the assisted driving by the automatic driving assistance may be ended, for example, on the condition that the user performs a predetermined ending operation on the operation unit 3, or on the condition that the shift position is shifted to "P" or the engine is turned off. Alternatively, the end condition may be on the condition that the automatic driving assistance cannot be continued.

[0057] If it is determined that the assisted driving by the autonomous driving assistance has ended (S10: YES), the driving assistance processing program is terminated. On the other hand, if it is determined that the assisted driving by the autonomous driving assistance has not ended (S10: NO), the process returns to S2, and the display of the assistance image on the liquid crystal display 4 continues.

[0058] In the driving assistance processing program according to the first embodiment, when an obstacle is detected around the vehicle, an object image 61 is displayed around the obstacle (S9). However, when a marking line (e.g., a roadway center line, a lane boundary line, a roadway outer boundary line, a parking space line, etc.) is detected instead of an obstacle, the object image 61 may be similarly displayed around the detected marking line. Furthermore, in the automated driving assistance, a steering operation is performed to avoid deviation from a marking line by lane departure prevention assistance. In S6, it may be determined whether a steering operation to avoid deviation from a marking line has been performed, and if it is determined that a steering operation to avoid deviation from a marking line has been performed, the shape of the object image 61 displayed around the marking line that is the target of avoidance by the steering operation may be changed (S7). Since the processing is similar, except that the target is replaced by a marking line instead of an obstacle, a detailed description thereof will be omitted.

[0059] As described above in detail, the driving assistance device 1 and the computer program executed by the driving assistance device 1 according to the first embodiment detect obstacles in the vicinity of the vehicle 2 (S4) and display an object image superimposed around the detected obstacle in the vicinity of the vehicle 2 (S9). At the same time, the driving assistance device 1 is capable of driving with automatic driving assistance that automatically controls at least a portion of the vehicle operation of the vehicle 2. When a vehicle operation to avoid an obstacle is performed through the automatic driving assistance, the shape of the object image displayed around the obstacle to be avoided is changed (S7, S8). This allows the user to intuitively understand that a vehicle operation has been performed based on the change in the shape of the object image. As a result, the user is not confused even when a vehicle operation to avoid an obstacle is performed automatically. Furthermore, the object image is an image of an object that represents a three-dimensional shape, and the shape of the object image is changed by depressing or expanding at least a portion of the outer periphery of the object that represents the three-dimensional shape (S7). This allows the user to intuitively understand that a vehicle operation has been performed based on the change in the shape of the object image. Furthermore, the object image is an image of an object that represents a three-dimensional shape, and the shape of the object image is changed by deforming a portion of the outer peripheral surface of the object that represents the three-dimensional shape, and the range of the outer peripheral surface to be deformed is changed in accordance with a change in the positional relationship between the vehicle and the object image (S8). Therefore, by changing the shape in accordance with a change in the position of the vehicle, it is possible to give the impression that the object image has been deformed due to the vehicle's movement, and it is possible to more intuitively grasp that a vehicle operation has been performed. Furthermore, since the criteria for determining the range of the outer peripheral surface to be deformed are different for an object image located further in the traveling direction than the current position of the vehicle and an object image located in the opposite direction from the current position of the vehicle (S8), when the user visually recognizes the deforming object image, it is possible to give the user a more natural impression that the object image has been deformed due to the vehicle's movement.

[0060] [Summary of First Embodiment] The first embodiment preferably includes at least the following components: obstacle detection means (31) for detecting an obstacle (60) in the vicinity of a vehicle (2), image display means (31) for displaying an object image (61) superimposed around the obstacle detected in the vicinity of the vehicle, automatic driving control means (31) for automatically controlling at least a part of the vehicle operation, and display control means (31) for changing the shape of the object image displayed around the obstacle to be avoided when the automatic driving control means performs a vehicle operation to avoid the obstacle.

[0061] This configuration allows the user to intuitively understand that a vehicle operation has been performed based on a change in the shape of the object image, and as a result, the user will not be confused even if the vehicle operation to avoid an obstacle is performed automatically.

[0062] In addition, in the first embodiment, it is preferable that the object image (61) is an image of an object showing a three-dimensional shape, and the display control means changes the shape of the object image by concaving or expanding at least a portion of the outer surface of the object showing the three-dimensional shape.

[0063] According to this configuration, it is possible for the user to intuitively understand that a vehicle operation has been performed based on the change in the shape of the object image.

[0064] In addition, in the first embodiment, the object image (61) is an image of an object showing a three-dimensional shape, and the display control means changes the shape of the object image by deforming a part of the outer surface of the object showing the three-dimensional shape, and preferably changes the range of the outer surface to be deformed in accordance with a change in the positional relationship between the vehicle and the object image.

[0065] According to this configuration, by changing the shape in accordance with changes in the vehicle's position, it is possible to give the impression that the object image has been deformed as the vehicle is moving, allowing the user to more intuitively understand that a vehicle operation has been performed.

[0066] Furthermore, the first embodiment has a range determination means (31) for determining the range of the outer peripheral surface to be deformed for the displayed object image (61), and the display control means deforms the outer peripheral surface of the displayed object image within the range determined by the range determination means, and it is preferable that the criteria for determining the range of the outer peripheral surface to be deformed by the range determination means differ between the object image located on the side of the current position of the vehicle (2) in the direction of travel and the object image located on the side of the current position of the vehicle in the opposite direction to the direction of travel.

[0067] According to this configuration, when the user visually recognizes the deforming object image, it is possible to give the user the impression that the object image has been deformed more naturally as the vehicle is traveling.

[0068] Second Embodiment Next, a driving assistance device according to a second embodiment will be described with reference to Figures 15 to 27. In the following description, the same reference numerals as those in the configuration of the driving assistance device 1 according to the first embodiment shown in Figures 1 to 14 indicate the same or equivalent parts as those in the configuration of the driving assistance device 1 according to the first embodiment.

[0069] The schematic configuration of the driving assistance device according to the second embodiment is substantially the same as that of the driving assistance device 1 according to the first embodiment. Furthermore, various control processes are also substantially the same as those of the driving assistance device 1 according to the first embodiment. However, the driving assistance device 1 according to the first embodiment is different in that, while the driving assistance device 1 according to the first embodiment displays an object image 61 superimposed around an obstacle or a lane marking detected in the vicinity of the vehicle 2, the driving assistance device 1 according to the second embodiment displays an object image superimposed between the vehicle and an obstacle or a lane marking detected in the vicinity of the vehicle 2. Furthermore, the driving assistance device 1 according to the first embodiment displays the object image 61 regardless of whether an avoidance operation has been performed to avoid approaching the obstacle or lane marking, whereas the driving assistance device 1 according to the second embodiment displays the object image in association with an avoidance operation to avoid approaching the obstacle or lane marking (which may be at the timing when the avoidance operation has been performed, or after or immediately before the avoidance operation has been performed).

[0070] Next, a driving assistance processing program according to the second embodiment will be described with reference to Fig. 15. Fig. 15 is a flowchart of the driving assistance processing program according to the second embodiment. Here, the driving assistance processing program is executed after the ACC power (accessory power supply) of the vehicle 2 is turned on, and is a program that provides assistance to the user using bird's-eye images and overhead images of the area around the vehicle while automatic driving assistance is being performed for the vehicle. The program shown in the flowchart in Fig. 15 below is stored in the RAM 32 and ROM 33 provided in the driving assistance device 1, and is executed by the CPU 31.

[0071] First, in S11, the CPU 31 determines whether assisted driving by automatic driving assistance is being performed. In the second embodiment, assisted driving by automatic driving assistance is performed when the user operates the operation unit 3 to select automatic driving assistance and it is determined that driving by automatic driving assistance is possible. The automatic driving assistance includes detecting the current position of the vehicle, the lane in which the vehicle is traveling, and the positions of surrounding lane markings and obstacles as needed, and automatically controlling the vehicle, such as steering, drive source, and brakes, so that the vehicle travels along the generated travel trajectory at a speed in accordance with the generated speed plan. However, it is also possible to automatically perform only the steering operation and manually control the drive source and brakes.

[0072] If it is determined that assisted driving by the automated driving assistance is being performed (S11: YES), the process proceeds to S12. On the other hand, if it is determined that assisted driving by the automated driving assistance is not being performed (S11: NO), the driving assistance processing program is terminated.

[0073] In S12, the CPU 31 generates a bird's-eye view image 52 looking down diagonally from the sky around the vehicle and an overhead view image 53 looking down vertically from the sky around the vehicle based on real-time images captured by the front camera 6, the rear camera 7, and the side cameras 8A and 8B. Details of the method for generating each image are the same as those in S2, and therefore will not be described here.

[0074] The virtual viewpoint of the bird's-eye view image 52 generated in S12 is a virtual viewpoint 71 that looks down diagonally from the sky around the vehicle as shown in Fig. 16. However, the position of this virtual viewpoint 71 is not always fixed with respect to the vehicle 2, and may change its relative position with respect to the vehicle 2 when a driving operation is performed to avoid approaching an obstacle, as will be described later. The field of view radius and field of view angle may also change in the same way. Details will be described later.

[0075] Thereafter, in S13, the CPU 31 displays the real-time bird's-eye image 52 and the overhead image 53, which show the current environment around the vehicle and were generated in S12, as support images on the liquid crystal display 4. In the second embodiment, both the bird's-eye image 52 and the overhead image 53 are simultaneously displayed on the liquid crystal display 4, but only one of them may be displayed, or they may be displayed in a switchable manner by a user operation. Details are the same as in S3, and therefore will not be described here. Thereafter, the bird's-eye image 52 and the overhead image 53 are continuously displayed until assisted driving by the autonomous driving assistance is terminated (S20: YES).

[0076] 17, the bird's-eye view image 52 displayed in S13 includes a host vehicle image 45 showing the vehicle's exterior at its current position, and it is desirable to display the host vehicle image 45 with semi-transparency (e.g., 30% transparency) rather than opaque. This allows the bird's-eye view image 52 to be more easily recognized as the surrounding environment. On the other hand, when a driving operation is performed to avoid approaching an obstacle (S15: YES), as described below, the transparency of the host vehicle image 45 may be reduced to clarify the positional relationship between the vehicle and the obstacle.

[0077] Next, in S14, the CPU 31 detects obstacles around the vehicle. If an obstacle is present, the CPU 31 also detects the position of the obstacle and the area occupied by the obstacle in the bird's-eye view image and the overhead view image generated in S12. Note that an obstacle here refers to an object that the vehicle must avoid when traveling, such as another vehicle, a pedestrian, a bicycle, a wall, a guardrail, or other structure. On the other hand, passable steps and the like are excluded. Note that the details are the same as in S4, so a description thereof will be omitted. If no obstacle is detected in S14, the processing from S15 onwards is not executed, and the processing proceeds to S20.

[0078] Then, in S15, the CPU 31 acquires the control content (the type of automatic driving assistance performed under what circumstances) executed in the automatic driving assistance process performed in parallel with this program, and determines whether a steering operation, which is a driving operation for avoiding approaching an obstacle by automatic driving assistance, has been performed. Note that the steering operation may be determined to be a condition that the steering angle is equal to or greater than a threshold value or that the steering speed is equal to or greater than a threshold value.

[0079] If it is determined that a steering operation has been performed to avoid approaching an obstacle (S15: YES), the process proceeds to S16. On the other hand, if it is determined that a steering operation has not been performed to avoid approaching an obstacle (S15: NO), the process proceeds to S20 without displaying an obstruction image 75 (described later).

[0080] Next, in S16, the CPU 31 determines whether the vehicle 2 is traveling on a road where there are multiple factors that hinder the vehicle's travel in the direction of travel, based on the results of image recognition of real-time images captured by the front camera 6, the rear camera 7, and the side cameras 8A and 8B. Factors that hinder the vehicle's travel include, for example, parked vehicles on the side of the road, oncoming vehicles, utility poles, road signs, billboards, etc. For example, if any of these obstacles exists in the direction of travel of the vehicle 2 and is within a predetermined distance from the vehicle's future travel trajectory, it is determined that the vehicle 2 is traveling on a road where there are multiple factors that hinder the vehicle's travel in the direction of travel.

[0081] On narrow roads such as narrow streets, there is a high possibility that there are multiple factors that could impede the travel of the vehicle 2 in the direction of travel. Therefore, in S16, it may be determined whether the road on which the vehicle is traveling is a narrow road such as a narrow street (for example, a road width of less than 4 m). Also, on roads that are wider than narrow streets and allow two-way traffic, but do not have a center line, oncoming vehicles can be a factor that impedes the travel of the vehicle 2 in the direction of travel. Therefore, in S16, it may be determined whether the road on which the vehicle is traveling does not have a center line. In such cases, the determination in S16 may be made based on map information and the current position of the vehicle.

[0082] If it is determined that the vehicle 2 is not traveling on a road where there are multiple factors that hinder its traveling in the direction of travel (S16: NO), the process proceeds to S17. On the other hand, if it is determined that the vehicle 2 is traveling on a road where there are multiple factors that hinder its traveling in the direction of travel (S16: YES), the process proceeds to S18.

[0083] In S17, the CPU 31 moves the position of the virtual viewpoint 71 of the bird's-eye image 52 displayed on the liquid crystal display 4. Specifically, the position of the virtual viewpoint 71 is moved toward the vehicle and toward the location of the obstacle to be avoided. The movement of the virtual viewpoint 71 is performed under the condition that at least both the host vehicle image 45 and the obstacle 76 to be avoided (however, at least a portion of each may be included) are maintained in the bird's-eye image 52. Note that the movement of the position of the virtual viewpoint 71 in S17 includes not only movement up and down, left and right, and forward and backward, but also changing the angle of the viewpoint (line of sight). Here, FIG. 18 shows an example of the position of the virtual viewpoint 71 after the movement, particularly showing the moved position of the virtual viewpoint 71 when an obstacle is detected forward and to the right of the vehicle 2. As shown in FIG. 18, when an obstacle is detected forward and to the right of the vehicle 2, the position of the virtual viewpoint 71 is moved toward the vehicle 2 and to the right of the detected obstacle so that the situation on the right side of the vehicle 2 can be confirmed in detail. Furthermore, the line of sight may be changed to the left front, facing the direction of the vehicle 2.

[0084] Furthermore, as a result of changing the virtual viewpoint 71 in S17, the bird's-eye image 52 displayed on the liquid crystal display 4 also changes. FIG. 19 is a diagram showing the bird's-eye image 52 displayed when the position of the virtual viewpoint 71 is changed to the position shown in FIG. 18. That is, as a result of moving the virtual viewpoint 71 in S17, the bird's-eye image 52 displayed on the liquid crystal display 4 changes from the bird's-eye image 52 shown in FIG. 17 to the bird's-eye image 52 shown in FIG. 19. In the bird's-eye image 52 shown in FIG. 19, the right-hand side of the host vehicle image 45 is enlarged as the position of the virtual viewpoint 71 is changed, making it possible to check the situation on the right side of the vehicle in particular in detail. Note that, for ease of comparison, the bird's-eye image 52 shown in FIGS. 17 and 19 omits everything except the host vehicle image 45, but in reality, obstacles around the vehicle are also displayed.

[0085] 18 and 19, the left and right sides are reversed, and the position of the virtual viewpoint 71 is moved closer to the vehicle 2 and to the left side where the obstacle is detected, so that the situation on the left side of the vehicle 2 can be confirmed in detail. Furthermore, the line of sight may also be changed to the right front, facing the direction of the vehicle 2.

[0086] The change in the display image accompanying the movement of the virtual viewpoint 71 may be animated so that the virtual viewpoint 71 moves over time, may be instantaneous switching, or the screen may be split to simultaneously display the virtual viewpoint 71 after the movement and the virtual viewpoint 71 before the movement. The virtual viewpoint 71 may be moved when a driving operation to avoid approaching an obstacle is performed (at the start of the driving operation), immediately before the driving operation to avoid approaching an obstacle is performed, or after the driving operation to avoid approaching an obstacle is performed (a predetermined time after the start of the driving operation). Then, the process proceeds to S19.

[0087] Meanwhile, in S18, the CPU 31 performs processing to change the field of view radius and field of view angle of the virtual viewpoint 71 without changing the position of the virtual viewpoint 71. Figure 20 is a diagram showing the field of view radius and field of view angle of the virtual viewpoint 71. Points P0 and P1 correspond to the virtual viewpoint and the virtual point of gaze, respectively, in the virtual space. The distance D between points P0 and P1 corresponds to the viewpoint radius, and the angle θ corresponds to the field of view angle described above.

[0088] In S18, the CPU 31 changes the coordinate parameters and angle θ of points P0 and P1 to increase the viewpoint radius and decrease the field of view angle relative to the current state. Figure 21 shows an example of a bird's-eye view image 52 generated by changing the field of view radius and field of view angle of the virtual viewpoint 71 in S18. As shown in Figure 21, the bird's-eye view image 52 generated by increasing the viewpoint radius of the virtual viewpoint 71 and decreasing the field of view angle has a wider display range of the right and left sides of the host vehicle image 45 compared to the bird's-eye view image 52 shown in Figure 17 before the change, resulting in a bird's-eye view image that makes it easier to view the areas around the right and left sides of the host vehicle image 45. As a result, it is possible to provide a display that more easily conveys to the user the positional relationship between the vehicle and obstacles on the left and right sides. Furthermore, although the bird's-eye view image 52 shown in FIG. 21 omits everything other than the image of the vehicle itself 45, in reality, obstacles around the vehicle are also displayed, and when the field-of-view radius and field-of-view angle of the virtual viewpoint 71 are changed in S18, the field-of-view radius and field-of-view angle are changed on the condition that at least both the image of the vehicle itself 45 and the obstacle 76 to be avoided (however, it is sufficient that at least a portion of each is included) are maintained in the bird's-eye view image 52.

[0089] In S17, when a steering operation is performed to avoid approaching an obstacle, the position of the virtual viewpoint is changed to display a positional relationship between the vehicle and the obstacle that is easier to convey to the user. However, when the vehicle 2 is traveling on a road with multiple obstacles in the vehicle's forward direction, it is expected that steering operations to avoid approaching an obstacle will occur frequently. Therefore, if the virtual support is moved as in S17, the virtual viewpoint will continuously move, making the screen difficult for the user to see and causing motion sickness. Therefore, when the vehicle 2 is traveling on a road with multiple obstacles in the vehicle's forward direction, the position of the virtual viewpoint 71 is not changed, but the field of view radius and field of view angle of the virtual viewpoint 71 are changed in S18 to solve the above problem. However, the processes of S16 and S18 are not required. The position of the virtual viewpoint 71 may always be changed (S17) regardless of whether the vehicle 2 is traveling on a road with multiple obstacles in the vehicle's forward direction.

[0090] Although only the viewpoint radius and field of view angle are changed in S18, the virtual viewpoint position and angle may also be adjusted to the extent that the aforementioned effect is not impaired. The viewpoint radius and field of view angle may be animated so that they move over time, or they may be switched immediately. The field of view radius and field of view angle may be changed when a driving operation to avoid approaching an obstacle is performed (at the start of the driving operation), or just before a driving operation to avoid approaching an obstacle is performed, or after a driving operation to avoid approaching an obstacle is performed (a predetermined time after the start of the driving operation). Then, the process proceeds to S19.

[0091] In S19, the CPU 31 displays an object image, particularly on the bird's-eye view image 52, out of the bird's-eye view image 52 and the overhead view image 53 displayed on the liquid crystal display 4 in S13, by superimposing it on the bird's-eye view image 52. In the following description, the object image displayed in the second embodiment will be referred to as an obstruction image 75. The obstruction image 75 is an image displayed in the bird's-eye view image 52 between the vehicle and an obstacle that has been avoided by steering. Specifically, the obstruction image 75 is an image of a three-dimensional (3D) wall that stands on the road surface and has a thickness on the vehicle side. In the second embodiment, the obstruction image 75 is displayed by superimposing it on only the bird's-eye view image 52, but the obstruction image 75 may also be displayed by superimposing it on the overhead view image 53.

[0092] Fig. 22 is an example of the bird's-eye view image 52 displayed on the liquid crystal display 4 when an obstruction image 75 is displayed. As shown in Fig. 22, the obstruction image 75 is displayed between the host vehicle image 45 and an obstacle 76 that has been selected as an obstacle to be avoided by steering. In the example shown in Fig. 22, object images 61 are also displayed at regular intervals to surround the obstacle 76, as in the first embodiment. However, in the second embodiment, the object images 61 may not be displayed. When the object images 61 are displayed, it is desirable to display the object images 61 at regular intervals to surround the detected obstacle at the time the obstacle is detected, regardless of whether a driving operation is being performed to avoid approaching the obstacle, as in the first embodiment.

[0093] In the vehicle 2, a driving operation is performed to avoid approaching the obstacle (S15: YES), and therefore, in the bird's-eye view image 52 subsequently displayed on the liquid crystal display 4, the host vehicle image 45 moves closer to the obstacle 76 as the vehicle 2 moves, and then the host vehicle image 45 comes into contact with the obstruction image 75, or the host vehicle image 45 turns and moves away from the obstruction image 75 just before coming into contact. As a result, in the bird's-eye view image 52 displayed on the liquid crystal display 4, the obstruction image 75 displayed between the obstacle 76 and the host vehicle image 45 appears as if it has bounced off the host vehicle image 45, and the user can intuitively understand that control to avoid the obstacle 76 has been performed by the autonomous driving assistance.

[0094] The obstruction image 75 is displayed in association with a driving operation to avoid approaching an obstacle in the vicinity of the vehicle, and the specific timing of display may be when the driving operation to avoid approaching the obstacle is performed (at the start of the driving operation), or just before the driving operation to avoid approaching the obstacle is performed, or after the driving operation to avoid approaching the obstacle is performed (a predetermined time after the start of the driving operation).

[0095] Next, an example of a method for drawing the obstacle image 75 in S19 will be described with reference to FIG. 23 . The CPU 31 measures the distance from the vehicle 2 to the detected obstacle 76 at predetermined intervals using ultrasonic sensors 9A-9L, a camera, or the like. In the example shown in FIG. 23 , the distance from the vehicle 2 to the obstacle 76 is detected for each detection point 81. The CPU 31 arranges the shape of the obstacle image 75 previously stored in the flash memory 34 for each detection point 81, calculates an approximate curve 82 from each detection point 81, and calculates a tangent 83 to the approximate curve at each detection point. The CPU 31 then arranges the obstacle image 75 in accordance with the angle of the tangent 83, thereby drawing the opposing surface of the obstacle image 75 so that it faces the vehicle 2. As a result, the obstacle images 75 are connected together to draw a single obstacle image 75. However, the method for drawing the obstacle image 75 shown in FIG. 23 is merely an example, and methods other than the above may also be used to draw the obstacle image 75.

[0096] Furthermore, when the host vehicle image 45 comes into contact with the obstruction image 75, the outer periphery of the obstruction image 75 is drawn to have an appearance that follows the shape of the host vehicle image 45 in the area where the obstruction image 75 and the host vehicle image 45 come into contact. Here, Fig. 24 shows, as a comparative example, a bird's-eye view image 52 that is displayed when the shape of the obstruction image 75 is not drawn to follow the shape of the host vehicle image 45. On the other hand, Fig. 25 shows a bird's-eye view image 52 that is displayed when the shape of the obstruction image 75 is drawn to follow the shape of the host vehicle image 45.

[0097] 24 and 25 , in the bird's-eye view image 52 shown in Fig. 24 , when the host vehicle image 45 comes into contact with the obstruction image 75, the host vehicle image 45 becomes embedded in the obstruction image 75, and it is difficult to give the user the feeling that the obstruction image 75 has bounced off the host vehicle image 45 even if the host vehicle image 45 subsequently moves away from the obstruction image 75. On the other hand, in the bird's-eye view image 52 shown in Fig. 25 , when the host vehicle image 45 comes into contact with the obstruction image 75, the outer peripheral surface of the obstruction image 75 is deformed to have an appearance that follows the shape of the host vehicle image 45, so that it is possible to give the user the impression that the host vehicle image 45 is embedded in the obstruction image 75 and to give the user the impression that the obstruction image 75 has bounced off the host vehicle image 45. Furthermore, the change in shape of the obstruction image 75 caused by contact between the obstruction image 75 and the vehicle image 45 is returned to its pre-deformation state when the vehicle image 45 subsequently moves away from the obstruction image 75.

[0098] In addition, in the second embodiment, as a method for drawing the shape of the obstruction image 75 so that it conforms to the shape of the host vehicle image 45, the host vehicle image 45 and the obstruction image 75 are drawn in that order, and the distances of the polygons from the virtual viewpoint (hereinafter referred to as depth values) are compared to draw the polygons with smaller depth values, so that if an object is in the foreground, the object behind it is not displayed. That is, by comparing the depth values ​​of the polygons of the host vehicle image 45 and the obstruction image 75, only the host vehicle image 45 located in the foreground is displayed in the area where the host vehicle image 45 overlaps with the obstruction image 75. Note that as a method for drawing the shape of the obstruction image 75 so that it conforms to the shape of the host vehicle image 45, other conventional techniques, general-purpose techniques, or standard techniques may be used, such as changing the shape of the obstruction image 75 itself to a concave shape to match the shape of the host vehicle image 45.

[0099] Furthermore, in the second embodiment, the shape of the obstruction image 75 pre-stored in the flash memory 34 is described as a cube. However, the shape of the obstruction image 75 is not limited to this and may be other three-dimensional shapes, lines, and surfaces. Furthermore, it is desirable to render the height of the obstruction image 75 lower than that of the vehicle 2. This makes it easier to grasp that the host vehicle image 45 and the obstruction image 75 are in contact with each other. For example, FIG. 26 illustrates a bird's-eye view image 52 in which the obstruction image 75 is displayed higher than the host vehicle image 45, and FIG. 27 illustrates a bird's-eye view image 52 in which the obstruction image 75 is displayed lower than the host vehicle image 45. Comparing FIG. 26 and FIG. 27 , the bird's-eye view image 52 in FIG. 27 displays the obstruction image 75 shorter than the host vehicle image 45, making it easier to grasp the boundary where the host vehicle image 45 and the obstruction image 75 are in contact, i.e., how the shape of the obstruction image 75 is deformed. Therefore, the image can be easily grasped as it makes it easier to grasp the contact between the host vehicle image 45 and the obstruction image 75. On the other hand, the bird's-eye view image 52 shown in Figure 26 displays an obstruction image 75 that is taller than the vehicle image 45, making it difficult to grasp the boundary where the vehicle image 45 and the obstruction image 75 meet.

[0100] Furthermore, the shape of the obstruction image 75 can be changed depending on the type of obstacle being avoided by steering. For example, if the obstacle is a human, the obstruction image 75 can be made thicker, and if the obstacle is an obstacle such as a traffic cone (registered trademark) that may intentionally cause the vehicle to approach, the obstruction image 75 can be made thinner. The thickness of the obstruction image 75 may also be changed depending on whether the obstacle is moving or not. The thickness of the obstruction image 75 may also be changed based on information obtained from the front camera 6, rear camera 7, side cameras 8A and 8B, ultrasonic sensors 9A-9L, and other sensors 36. In addition, the shape of the obstruction image 75 may be changed depending on the distance between the vehicle 2 and the obstacle. For example, the closer the distance between the vehicle 2 and the obstacle, the thicker the obstruction image 75 may be drawn. Furthermore, not only the thickness but also the height of the obstruction image 75 may be changed. Instead of changing the shape of the obstruction image 75 depending on the type of obstacle, the color of the obstruction image 75 may also be changed. For example, obstacles that pose a greater risk (such as pedestrians) can be displayed in a more visible color (such as red) or flashed to make them more noticeable.

[0101] Furthermore, the obstruction image 75 may be displayed semi-transparently (e.g., with a transmittance of 30%) rather than opaquely. This allows the bird's-eye view image 52 to more easily recognize the surrounding environment. Alternatively, the distance between the vehicle 2 and the obstacle may be measured, and the obstruction image 75 may be rendered with a gradually changing transmittance such that the closer the distance between the vehicle 2 and the obstacle, the lower the transmittance, and the farther the distance between the vehicle 2 and the obstacle, the higher the transmittance. This reduces the transmittance of the obstruction image 75 as the host vehicle image 45 sinks into the obstruction image 75, making it appear as if the obstruction image 75 is being repelled more strongly. Furthermore, the transmittance of the obstruction image 75 may be varied depending on the time elapsed since the driver initiated a driving operation to avoid approaching the obstacle or the amount of driving operation. For example, the transmittance may be gradually reduced as time elapses since the driver initiated a driving operation to avoid approaching the obstacle. Alternatively, the transmittance may be changed according to the amount of change in the steering angle to avoid approaching an obstacle, and more specifically, the transmittance may be changed so as to decrease as the steering angle increases.

[0102] As described above, when a driving operation is performed to avoid approaching an obstacle, the transmittance of the host vehicle image 45 included in the bird's-eye view image 52 is reduced to clarify the positional relationship between the vehicle and the obstacle, but the change in the transmittance of the host vehicle image 45 may be performed simultaneously with the display of the obstruction image 75, simultaneously with the virtual viewpoint processing in S17 or S18, or simultaneously with the driving operation by the driving assistance device. Also, the host vehicle image 45 may be switched to non-transparent (transmittance 0%) all at once, or the transmittance may be reduced in stages.

[0103] Furthermore, when the host vehicle image 45 comes into contact with the obstruction image 75, a gap may be provided between the host vehicle image 45 and the obstruction image 75. One method of providing the gap is to overlay a transparent host vehicle image that is larger than the host vehicle image 45 on the host vehicle image 45, so that the screen display always appears to show a gap between the obstruction image 75 and the host vehicle image 45. By providing the gap, the screen display does not appear to show that the host vehicle has come into contact with an obstacle, thereby reducing the user's misunderstanding and anxiety. Note that other conventional, general-purpose, or standard techniques may be used to provide a gap between the host vehicle image 45 and the obstruction image 75, such as by changing the shape of the obstacle itself.

[0104] Furthermore, the obstruction image 75 drawn in S19 is gradually hidden as the driving operation to avoid approaching the obstacle 76 is completed. For example, after avoiding an obstacle, control to return the steering wheel toward the neutral position is initiated, and the transparency of the obstruction image 75 can be gradually increased to hide the obstruction image 75 (transmittance 100%) near the neutral position. Alternatively, the obstruction image 75 may be continuously displayed until the steering wheel is in the neutral position, and then hidden in a single step when the steering wheel is in the neutral position. Alternatively, the obstruction image 75 may be hidden after a predetermined time has elapsed since the display of the obstruction image 75 began, or after the vehicle has moved a predetermined distance. Furthermore, the position of the virtual viewpoint changed in S17 and the field of view radius and field of view angle changed in S18 are also returned to their pre-change states in conjunction with the hiding of the obstruction image 75. The position of the virtual viewpoint, the field of view radius, and the field of view angle may also be gradually returned to their pre-change states in the same manner as the hiding of the obstruction image 75, or may be returned in a single step at a specific timing after avoiding the obstacle.

[0105] Thereafter, in S20, the CPU 31 determines whether or not to end assisted driving by the automatic driving assistance. Here, the assisted driving by the automatic driving assistance may be ended, for example, on the condition that the user performs a predetermined ending operation on the operation unit 3, or on the condition that the shift position is shifted to "P" or the engine is turned off. Alternatively, the end condition may be on the condition that the automatic driving assistance cannot be continued.

[0106] If it is determined that the assisted driving by the automatic driving assistance has ended (S20: YES), the driving assistance processing program is terminated. On the other hand, if it is determined that the assisted driving by the automatic driving assistance has not ended (S20: NO), the process returns to S12, and the display of the assistance image on the liquid crystal display 4 continues.

[0107] In the automated driving assistance, lane departure prevention assistance not only performs steering control to avoid the obstacle, but also performs steering operations to avoid deviation from a lane marking. In the driving assistance processing program according to the second embodiment, when a driving operation is performed to avoid approaching an obstacle, an obstruction image 75 is displayed between the vehicle and the obstacle to be avoided (S19). However, even when a driving operation is performed to avoid approaching a lane marking (e.g., a roadway center line, a lane boundary line, a roadway outer edge line, a parking space line, etc.) instead of an obstacle, the obstruction image 75 may be displayed between the vehicle and the lane marking (e.g., a roadway center line, a lane boundary line, a roadway outer edge line, a parking space line, etc.). The obstruction image 75 may be displayed when a driving operation to avoid approaching the lane marking is performed, immediately before a driving operation to avoid approaching the lane marking is performed, or after a driving operation to avoid approaching the lane marking is performed. In terms of details, the processing is similar, except that the target is replaced by a lane marking. For example, the thickness of the obstruction image 75 may be changed depending on the type of dividing line to be avoided (e.g., roadway center line, lane boundary line, roadway outer line, parking space line, etc.). In other words, the more dangerous a dividing line to deviate from, such as a roadway center line, the thicker the obstruction image 75 is displayed.

[0108] As described above in detail, the driving assistance device 1 and the computer program executed by the driving assistance device 1 according to the second embodiment display a vehicle surroundings image showing the periphery of the vehicle 2 on the liquid crystal display 4 (S13), and display an obstruction image 75 between the vehicle and an obstacle or lane marking to be avoided in the vehicle surroundings image in association with a driving operation to avoid approaching the obstacle or lane marking around the vehicle by automatic control (S19). This allows the user to intuitively understand that a driving operation to avoid the obstacle or lane marking has been performed by automatic driving assistance. As a result, the user is not confused even when a driving operation to avoid the obstacle or lane marking is automatically performed. Furthermore, the vehicle surroundings image is an image of the periphery of the vehicle viewed from a virtual viewpoint, and the position of the virtual viewpoint is changed to display the vehicle surroundings image in association with a driving operation to avoid approaching the obstacle or lane marking around the vehicle by automatic control (S17). Therefore, by changing the position of the virtual viewpoint, the vehicle surroundings image can be generated in a way that makes it easier to understand the relationship between the vehicle and the obstacle or lane marking. Furthermore, when the obstruction image 75 is displayed, the obstruction image 75 is displayed as an image of a three-dimensional object with a thickness on the vehicle side. Therefore, when a driving operation is performed to avoid approaching an obstacle or a lane marking, the user can feel as if the displayed obstruction image 75 is bouncing off the vehicle. This allows the user to intuitively understand that a driving operation to avoid an obstacle or a lane marking has been performed. Furthermore, the vehicle surroundings image displays the host vehicle image 45, which shows the vehicle exterior, at the current position of the vehicle. When the obstruction image 75 is displayed, the outer periphery of the obstruction image 75 is displayed with an appearance that follows the shape of the host vehicle image 45 within the range where the obstruction image 75 and the host vehicle image 45 meet in the vehicle surroundings image. This makes it possible to create an appearance in which the host vehicle image 45 is embedded in the obstruction image 75 and to give the user the feeling that the obstruction image 75 is bouncing off the host vehicle image 45.

[0109] Summary of the Second Embodiment The second embodiment includes at least the following configuration: A driving assistance device (1) that automatically controls at least a part of the driving operation of a vehicle (2), wherein a vehicle surroundings image (52) showing the surroundings of the vehicle is displayed on a display device (4), and preferably, in association with a driving operation for avoiding approaching an obstacle (76) or a marking line around the vehicle by the automatic control, an obstruction image (75) is displayed in the vehicle surroundings image between the vehicle and the obstacle or marking line to be avoided.

[0110] According to this configuration, by displaying an obstruction image between the vehicle and the obstacle or lane marking in association with the driving operation to avoid the obstacle or lane marking by the automated driving assistance, the user can intuitively understand that the driving operation to avoid the obstacle or lane marking has been performed by the automated driving assistance. As a result, the user will not be confused even if the driving operation to avoid the obstacle or lane marking has been performed automatically.

[0111] In addition, in the second embodiment, the vehicle surroundings image (52) is an image of the surroundings of the vehicle (2) viewed from a virtual viewpoint (71), and it is preferable that the position of the virtual viewpoint is changed to display the vehicle surroundings image in association with a driving operation for avoiding approaching an obstacle (76) or a dividing line around the vehicle by the automatic control.

[0112] According to this configuration, by changing the position of the virtual viewpoint, it is possible to obtain a vehicle surroundings image that allows the driver to more easily grasp the relationship between the vehicle and obstacles or lane markings.

[0113] In the second embodiment, it is preferable that the obstruction image (75) is displayed as an image of a three-dimensional object having a thickness on the vehicle side.

[0114] With this configuration, when a driving maneuver is performed to avoid approaching an obstacle or a marking line, the user can feel as if the displayed image of the obstruction is bouncing off the vehicle, thereby enabling the user to intuitively understand that a driving maneuver has been performed to avoid the obstacle or the marking line.

[0115] In the first embodiment, the vehicle surroundings image (52) displays a host vehicle image (45) showing the vehicle exterior at the current position of the vehicle (2), and it is preferable that the obstruction image (75) is displayed with an appearance that follows the shape of the host vehicle image in the area where the obstruction image and the host vehicle image meet within the vehicle surroundings image.

[0116] According to this configuration, it is possible to express that the image of the host vehicle is embedded in the image of the obstructing object, and it is also possible to give the user the sensation that the image of the obstructing object is bouncing off the image of the host vehicle.

[0117] [Note] The above-described embodiments also disclose the following inventions. In the following description, the names and expressions of corresponding components in the embodiments and the symbols used in the drawings are added in parentheses for reference. However, the components of each invention are not limited to these notes.

[0118] (Invention A) The driving assistance device (1) according to claim 3, wherein the thickness of the obstruction image (75) is changed depending on the type of the obstacle (76) or the lane marking to be avoided.

[0119] This makes it possible to suggest the degree of danger of obstacles or lane lines to be avoided from the shape of the obstruction image.

[0120] (Invention B) The vehicle surroundings image (52) displays a host vehicle image (45) showing the vehicle exterior at the current position of the vehicle (2), and when displaying the host vehicle image in the vehicle surroundings image, the transparency of the host vehicle image is changed to a lower transparency in association with a driving operation to avoid approaching an obstacle or a dividing line around the vehicle by the automatic control.

[0121] This makes it possible to clarify the positional relationship between the vehicle and the obstacle when a driving operation is performed to avoid approaching an obstacle or a marking line.

[0122] (Invention C) The driving assistance device (1) according to claim 1, wherein the display of the obstruction image (75) changes the transparency of the obstruction image depending on at least one of the distance from the vehicle (2) to the obstacle (76) or the lane marking, the elapsed time since the driving operation was started, and the amount of the driving operation.

[0123] According to this, by changing the transmittance of the obstruction image, the transmittance can be increased in situations where it is not necessary to make the obstruction image clearer, making it easier to recognize the surrounding environment, while the transmittance can be decreased in situations where it is necessary to make the obstruction image clearer, making it possible to clarify the positional relationship between the vehicle and the obstacle.

[0124] (Invention D) The driving assistance device (1) according to claim 2, wherein when a driving operation is performed by the automatic control to avoid approaching an obstacle (76) or a marking line around the vehicle (2), the position of the virtual viewpoint (71) is moved in the direction of the obstacle or marking line that is approaching the vehicle and is being avoided, and the image of the vehicle's surroundings is displayed.

[0125] This makes it possible to change the position of the virtual viewpoint to create an image of the vehicle's surroundings that allows the driver to grasp in more detail the relationship between the vehicle and the obstacle or lane marking that needs to be avoided.

[0126] (Invention E) The vehicle surroundings image (52) displays a host vehicle image (45) showing the vehicle exterior at the current position of the vehicle (2), and when the vehicle is traveling on a road where there are multiple factors that hinder travel in the vehicle's direction of travel, even if the automatic control performs a driving operation to avoid approaching an obstacle or a dividing line around the vehicle, the driving assistance device (1) as described in claim 2 displays the vehicle surroundings image by changing the field of view radius and field of view angle without changing the position of the virtual viewpoint (71).

[0127] This makes it possible to display the positional relationship between the vehicle and obstacles more easily to the user by changing the field of view radius and field of view angle without moving the virtual viewpoint in situations where the virtual viewpoint is constantly shifting.

[0128] The present invention is not limited to the above-described embodiments, and various improvements and modifications are possible without departing from the spirit and scope of the present invention. For example, in the first and second embodiments, an image of the scenery around the vehicle captured by a camera is displayed on the liquid crystal display 4 inside the vehicle, and an object image or an obstruction image is also displayed on the liquid crystal display 4, thereby superimposing the object image or obstruction image on the scenery around the vehicle. However, a head-up display system (HUD) may be used as a means for displaying an image to be superimposed on the scenery around the vehicle. In the head-up display system, for example, the object image or obstruction image displayed on the display inside the vehicle is reflected on the windshield so that the user can view it, thereby allowing the user to view the object image or obstruction image (more precisely, a virtual image of the object image or obstruction image) superimposed on the actual scenery viewed through the windshield.

[0129] Furthermore, in the first and second embodiments, bird's-eye images and overhead images generated from images captured by the front camera 6, rear camera 7, and side cameras 8A and 8B are displayed on the liquid crystal display 4 as landscape images around the vehicle. However, it is also possible to display the image captured by the front camera 6 or rear camera 7 as is on the liquid crystal display 4, rather than a processed image, and superimpose an object image or an obstruction image on that image.

[0130] Furthermore, the scenery around the vehicle displayed on the liquid crystal display 4 may not be an image captured by a camera, but may be a virtual scenery image (e.g., a three-dimensional map image) that reproduces the surroundings of the vehicle using CG (computer graphics). For example, a three-dimensional map image around the current location may be acquired or generated and displayed on the liquid crystal display 4, and an object image or an obstruction image may be displayed on the displayed three-dimensional map image. Even in this case, the same effect can be achieved. Furthermore, as shown in FIG. 28 , the assistance image displayed on the liquid crystal display 4 may be an image of the vehicle itself 45 and an object image 61 or an obstruction image 75, without displaying the scenery around the vehicle.

[0131] In the first embodiment, the shape of the object image is changed when a steering operation is performed to avoid an obstacle (S7, S8). However, as an autonomous driving assistance for avoiding an obstacle, avoidance by braking (deceleration) is also conceivable in addition to avoidance by steering operation. Therefore, the shape of the object image may be changed when a braking operation is performed to avoid an obstacle. Furthermore, the shape of the object image may be changed in a similar manner when other vehicle operations are performed to avoid an obstacle, not limited to steering operation or braking operation.

[0132] In the second embodiment, the obstruction image 75 is displayed (S19) when a steering operation is performed to avoid approaching an obstacle. However, as an automatic driving assistance for avoiding approaching an obstacle, avoidance by braking (deceleration) is also considered in addition to avoidance by steering operation. Therefore, the obstruction image 75 may be displayed when a braking operation is performed to avoid approaching an obstacle. Furthermore, the obstruction image 75 may be displayed in the same way when any other vehicle operation is performed to avoid approaching an obstacle, not limited to steering operation or braking operation.

[0133] In the first and second embodiments, it is assumed that the support images shown in Fig. 14 and Fig. 22 are displayed while the vehicle is being driven with automatic driving assistance, but the support images shown in Fig. 14 and Fig. 22 may also be displayed while the vehicle is being driven manually. In that case, when a steering operation or braking operation is performed to avoid an obstacle during manual driving, the shape of the object image is changed (S7, S8), or an obstruction image 75 is displayed (S19).

[0134] In the first and second embodiments, the object image 61 and the obstruction image 75 are displayed only in the bird's-eye view image 52 out of the bird's-eye view image 52 and the overhead view image 53 displayed on the LCD display 4. However, the object image 61 and the obstruction image 75 may also be displayed in the overhead view image 53. As shown in FIG. 29 , instead of the object image and the obstruction image 75, icons 85 may be displayed around the host vehicle image 45 in the overhead view image 53. The icons 85 consist of a pair of icons, one on the left and one on the right. For example, when a steering operation is performed to avoid an obstacle, the icon 85 on the side where the obstacle to be avoided is located flashes or changes color. Furthermore, in addition to the display on the LCD display 4, a sound may be output from the speaker 5 to notify the driver of the approaching or avoiding of an obstacle.

[0135] In the first embodiment, the shape of an object image is changed by recessing a portion of the outer periphery of the object (FIG. 12), but the shape of the object image may be changed by expanding a portion of the outer periphery of the object. Also, the area to be recessed or expanded may be the entire outer periphery of the object, rather than just a portion of it.

[0136] In the first and second embodiments, the driving assistance processing program (FIGS. 3 and 15) is executed by the driving assistance ECU 10 of the driving assistance device 1, but the executing entity can be changed as appropriate. For example, the program may be executed by a control unit of the liquid crystal display 4, a vehicle control ECU, a control unit of a navigation device, or other in-vehicle device.

[0137] DESCRIPTION OF SYMBOLS 1... Driving assistance device, 2... Vehicle, 4... Liquid crystal display (display device), 6... Front camera, 7... Rear camera, 8A, 8B... Side cameras, 10... Driving assistance ECU, 52... Bird's-eye view image (image of vehicle surroundings), 53... Bird's-eye view image, 61... Object image, 71... Virtual viewpoint, 75... Obstruction image, 76... Obstacle

Claims

1. A driving assistance device that automatically controls at least a portion of a vehicle's driving operations, which displays a vehicle surroundings image showing the surroundings of the vehicle on a display device, and in association with driving operations to avoid approaching an obstacle or lane marking around the vehicle by the automatic control, displays an image of an obstruction between the vehicle and the obstacle or lane marking to be avoided in the vehicle surroundings image.

2. A driving assistance device as described in claim 1, wherein the vehicle surroundings image is an image of the surroundings of the vehicle viewed from a virtual viewpoint, and the position of the virtual viewpoint is changed to display the vehicle surroundings image in association with driving operations for avoiding approaching obstacles or lane markings around the vehicle by the automatic control.

3. A driving assistance device according to claim 1, wherein the obstruction image is displayed as an image of a three-dimensional object having a thickness on the vehicle side.

4. A driving assistance device as described in claim 3, wherein the vehicle surroundings image displays an image of the vehicle's own vehicle showing the vehicle's exterior at the vehicle's current position, and when displaying the obstruction image, the outer periphery of the obstruction image is displayed with an appearance that follows the shape of the vehicle image in the area where the obstruction image and the vehicle image contact within the vehicle surroundings image.