Driving assistance systems

Abstract

This driver assistance system provides a way for the user to intuitively understand that automatic control has been performed to avoid approaching obstacles or lane markings around the vehicle. [Solution] A vehicle surrounding image showing the area around the vehicle 2 and a predicted path image 51 showing the expected future path of the vehicle 2 in the vehicle surrounding image are displayed on the liquid crystal display 4. In connection with automatic control driving operations to avoid approaching obstacles or lane markings around the vehicle 2, the display form of at least the side of the predicted path image 51 that is close to the obstacle or lane marking to be avoided is changed.

Description

Technical Field

[0001] The present invention relates to a driving support device that performs driving support for a vehicle.

Background Art

[0002] In recent years, as a driving mode of a vehicle, in addition to manual driving that travels based on a user's driving operation, an automatic driving support system that assists the user in driving the vehicle by executing part or all of the user's driving operation on the vehicle side has been newly proposed. In the automatic driving support system, for example, the current position of the vehicle, the lane in which the vehicle travels, and the positions of other surrounding vehicles are detected at any time, and vehicle control such as steering, drive source, and brake is automatically performed so as to travel along a preset route.

[0003] Here, when avoiding an obstacle during manual driving, the user recognizes the obstacle and selects an operation for avoiding the obstacle by the user himself / herself to perform a steering operation or a brake operation. Therefore, basically, unexpected steering operations or brake operations (deceleration) are not performed for the user. On the other hand, when traveling by the above-described automatic driving support, since a steering operation or a brake operation for avoiding an obstacle is automatically performed, particularly in a situation where the user is not paying much attention to the surroundings, an unexpected steering operation or brake operation may be performed for the user, which may confuse the user. As a means for solving this, for example, as disclosed in Japanese Patent Application Laid-Open No. 2004-240480, it is conceivable to make the user grasp the presence of an obstacle in advance by superimposing and displaying an image of a three-dimensional object on the position of the obstacle included in the surrounding image displayed on the in-vehicle monitor.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

[0005] Patent Document 1 discloses that when a vehicle approaches an obstacle, the display color of a 3D object can be changed or the display can be made to flash. These are basically intended to ensure that the user is aware of the presence of the obstacle without overlooking it. However, while changing the display color or flashing the display can make it easier to be aware of the presence of an obstacle, it does not provide guidance that makes the user imagine that steering or braking operations have been performed. Therefore, even if steering and braking operations to avoid the obstacle are automatically performed by the automated driving assistance system while the 3D object shown in Patent Document 1 is displayed, there is still a problem of confusing the user with steering or braking operations that the user did not expect. In addition, when driving with automated driving assistance, the vehicle basically maintains its position within the lane markings in sections with lane markings. For example, at junctions and curves, steering and braking operations are automatically performed in relation to the lane markings in order to drive along the lane markings. Therefore, the same problem occurred not only with obstacles but also with lane markings.

[0006] The present invention was made to solve the aforementioned problems of the conventional system, and aims to provide a driving assistance device that allows the user to intuitively understand that automatic control has been performed to avoid approaching obstacles or lane markings around the vehicle, by changing the display format of a predicted path image that shows the expected future path of the vehicle in relation to automatic control driving operations to avoid approaching obstacles or lane markings around the vehicle. [Means for solving the problem]

[0007] To achieve the above objective, the driver assistance device according to the present invention is a driver assistance device that automatically controls at least a part of the driving operation of a vehicle, and displays a vehicle surrounding image showing the area around the vehicle and a predicted path image showing the expected future path of the vehicle in the vehicle surrounding image on a display device, and changes the display form of at least the side of the predicted path image that is close to the obstacle or lane marking to be avoided in relation to the automatically controlled driving operation to avoid approaching an obstacle or lane marking in the area around the vehicle. Furthermore, the "vehicle surroundings image" may be a real-world image of the area around the vehicle captured by an imaging device such as a camera, or it may be a virtual image of the area around the vehicle recreated using computer graphics (CG). In addition, if it is a real-world image, it may be the captured image itself, or an image that has been processed from the captured image. For example, it may be an image created by combining images taken with multiple cameras, or an image with a transformed viewpoint. Furthermore, "the expected future path of the vehicle" may refer to either the expected path of the vehicle if it were to move while maintaining the steering angle of the vehicle at the time of display, or the expected path of the vehicle if it were to move according to automatic control. Furthermore, the "predicted path image" can be multiple lines, a single line, or a polygonal image rather than a linear one. It can also simply be an area distinguished by color, or a three-dimensional image rather than a two-dimensional one. Furthermore, "changing the display format in connection with the automatic control driving operation to avoid approaching an obstacle or lane marking in the vicinity of the vehicle" means that the display format may be changed at the time the automatic control driving operation to avoid approaching an obstacle or lane marking is performed, or immediately before the automatic control driving operation to avoid approaching an obstacle or lane marking is performed, or after the automatic control driving operation to avoid approaching an obstacle or lane marking has been performed. [Effects of the Invention]

[0008] According to the driver assistance device of the present invention having the above configuration, by changing the display format of the predicted path image showing the expected future path of the vehicle in relation to the driving operation by automatic control to avoid approaching obstacles or lane markings around the vehicle, it is possible to make the user intuitively understand that automatic control to avoid approaching obstacles or lane markings around the vehicle has been performed. As a result, even when driving operations to avoid obstacles or lane markings are performed automatically, the user will not be confused. [Brief explanation of the drawing]

[0009] [Figure 1] This is a schematic diagram of the vehicle according to this embodiment. [Figure 2] This is a block diagram showing the configuration of the driver assistance system according to this embodiment. [Figure 3] This is a flowchart of the driver assistance processing program according to this embodiment. [Figure 4] This diagram illustrates the method for converting captured images into bird's-eye view images. [Figure 5] This diagram illustrates the method for generating a bird's-eye view image. [Figure 6] This figure shows an example of a predicted path image generated when the vehicle's future path is straight. [Figure 7] This figure shows an example of a predicted path image generated when the vehicle's future path is a left turn. [Figure 8] This figure shows an example of a predicted path image generated when the vehicle's future path is a left turn. [Figure 9] This figure shows an example of how a liquid crystal display (LCD) can display a predicted trajectory image when the display format of the image is changed. [Figure 10] This figure illustrates an example of gradually changing the display format of the predicted path image. [Figure 11] This figure shows an example of how a liquid crystal display (LCD) can display a predicted trajectory image when the display format of the image is changed. [Figure 12]This figure shows an example of how a liquid crystal display (LCD) can display a predicted trajectory image when the display format of the image is changed. [Modes for carrying out the invention]

[0010] Hereinafter, one embodiment of the driver assistance device according to the present invention will be described in detail with reference to the drawings. First, the vehicle 2 equipped with the driver assistance device 1 according to this embodiment will be described below. Figure 1 is a schematic diagram of the vehicle 2 according to this embodiment.

[0011] Here, Vehicle 2 may be, for example, an automobile powered by an internal combustion engine (internal combustion engine vehicle), an automobile powered by an electric motor (electric vehicle, fuel cell vehicle, etc.), or an automobile powered by both (hybrid vehicle). Furthermore, there is no restriction on the type of vehicle; it may be a regular passenger car, a large commercial truck, a bus, construction machinery, etc. Also, although the following explanation will refer to it as a four-wheeled vehicle, it may also be a two-wheeled or three-wheeled vehicle.

[0012] However, Vehicle 2 shall be a vehicle capable of not only manual driving based on the user's driving operations, but also assisted driving through automated driving assistance, in which the vehicle drives automatically without user operation.

[0013] In addition, the automatic driving support may be performed only under specific situations such as when parking or leaving the garage, or it may be performed for all road sections, or it may be configured to be performed only while the vehicle is traveling on a specific road section (for example, a highway where a gate (regardless of manned or unmanned, toll or free) is provided at the boundary). In the following description, the automatic driving sections where the automatic driving support of the vehicle is performed include all road sections including general roads and highways, as well as parking lots. Furthermore, it is performed only when the user selects to perform automatic driving support (for example, turns on the automatic driving start button) and it is determined that it is possible to perform driving by automatic driving support. On the other hand, the vehicle 2 may be a vehicle that can only perform assisted driving by automatic driving support. Or, without being triggered by the user's selection, when it is detected that an obstacle (for example, a person, a bicycle, another vehicle, a wall, etc.) is located in the support target area set in the front space of the vehicle 2, automatic driving support such as a steering operation or a braking operation for avoiding the obstacle may be performed as support for the driver with respect to the obstacle.

[0014] In the vehicle control in the automatic driving support of this embodiment, for example, the current position of the vehicle, the lane in which the vehicle is traveling, and the positions of surrounding obstacles are detected at any time, and along the generated travel trajectory, the vehicle control such as steering, drive source, and brake is automatically performed at a speed according to the also generated speed plan. However, it is also possible to automatically perform only the steering operation and perform the control of the drive source and the brake based on manual operation. In addition, the automatic driving support of the vehicle in this embodiment includes lane departure prevention support. When it is detected that a dividing line (for example, a center line of the lane, a lane boundary line, an outer line of the lane, a parking dividing line, etc.) is located around the vehicle 2, automatic driving support such as a steering operation or a braking operation is also performed to avoid deviating from the dividing line. Furthermore, when performing the above automatic driving support, as will be described later, the scenery around the vehicle (which may be a real scene or a CG virtual scenery) is displayed on the in-vehicle display, and an expected route image indicating the future expected route of the vehicle 2 is also displayed for the displayed scenery around the vehicle.

[0015] Further, as shown in FIG. 1, the vehicle 2 includes an operation unit 3 that receives operations from the occupant, a liquid crystal display 4 that displays an aerial image, an overhead image, or other driving support-related information around the vehicle to the occupant, a speaker 5 that outputs voice guidance related to driving support, a front camera 6, a rear camera 7, side cameras 8A and 8B for imaging the periphery of the vehicle, ultrasonic sensors 9A to 9L that detect obstacles around the vehicle, and a driving support ECU (Electronic Control Unit) 10 that performs various arithmetic processes based on the input information. Note that the driving support device 1 includes the driving support ECU 10.

[0016] Hereinafter, each component included in the vehicle 2 will be described. First, the operation unit 3 is disposed, for example, on the front surface of the steering wheel (also referred to as a steering wheel), and includes operation buttons and the like that are operated when starting automatic driving support. By operating the operation unit 3, the user can switch between manual driving in which the vehicle travels based on the user's driving operation and support driving by automatic driving support in which the vehicle automatically travels without depending on the user's driving operation. Note that the operation unit 3 may have a touch panel provided on the front surface of the liquid crystal display 4. Further, it may have a microphone and a voice recognition device.

[0017] The liquid crystal display 4 is a type of display device, is provided on the instrument panel of the vehicle 2, and displays an aerial image or an overhead image around the vehicle generated by subjecting the captured images captured by the front camera 6, the rear camera 7, and the side cameras 8A and 8B to viewpoint conversion and compositing processing during the execution of automatic driving support. Further, for the aerial image and the overhead image, a predicted route image indicating the predicted route of the vehicle 2 in the future is also displayed. Note that the liquid crystal display 4 may be used in common with a navigation device.

[0018] Also, the speaker 5 is provided on the instrument panel of the vehicle 2 and outputs guidance voice, warning sound, and the like related to driving support. Note that the speaker 5 may be used in common with a navigation device.

[0019] Furthermore, the front camera 6 is an imaging device that has a camera using a solid-state image sensor such as a CCD, and is installed, for example, above the front bumper of the vehicle 2 or behind the rearview mirror, with the optical axis facing forward in the direction of travel of the vehicle.

[0020] The rear camera 7 is an imaging device that also has a camera using a solid-state image sensor such as a CCD, and is mounted, for example, near the center above the license plate attached to the rear of the vehicle 2, with the optical axis facing the rear of the vehicle.

[0021] Furthermore, the side cameras 8A and 8B are imaging devices that also have cameras using solid-state image sensors such as CCDs, and are mounted, for example, on the left and right side mirrors of vehicle 2, with the optical axis facing the side of the vehicle.

[0022] The driver assistance ECU 10 generates bird's-eye and overhead views of the vehicle's surroundings by performing viewpoint transformation and synthesis processing on the images captured by the front camera 6, rear camera 7, and side cameras 8A and 8B. Furthermore, during automated driving assistance, it performs image recognition processing on the captured images to detect lane markings and obstacles (other vehicles, pedestrians, bicycles, walls, guardrails, and other structures) around the vehicle, and performs automated driving assistance based on the detection results.

[0023] The ultrasonic sensors 9A to 9L are positioned at predetermined intervals on the front, rear, and sides of the vehicle, respectively. They transmit ultrasonic waves as probe waves around the vehicle 2 and detect objects that reflected the probe waves by receiving reflected waves from objects around the vehicle. Specifically, they are a type of distance measuring sensor capable of detecting the distance (measured distance value) to the object that reflected the probe waves by measuring the time from transmission to reception. Furthermore, the ultrasonic sensors 9A to 9L are configured to generate an output signal (including the distance to the detected object) corresponding to the reception result of the received wave and output it to the control unit. The objects to be detected by the ultrasonic sensors 9A to 9L include, for example, people, bicycles, other vehicles, walls, and other obstacles that the vehicle 2 needs to avoid while driving. In addition, millimeter-wave sensors or laser sensors may be used as distance measuring sensors instead of ultrasonic sensors. Furthermore, in this embodiment, as described above, it is also possible to detect obstacles by performing image recognition on the images captured by the front camera 6, rear camera 7, and side cameras 8A and 8B. Therefore, if obstacle detection is performed using the above cameras, the ultrasonic sensors 9A to 9L may be omitted.

[0024] On the other hand, the driver assistance ECU 10 is an electronic control unit that performs various processes related to automated driving assistance. For example, it constantly detects the vehicle's current position, the lane it is traveling in, and the positions of surrounding obstacles, and controls the vehicle, such as steering, drivetrain, and brakes, to ensure that the vehicle travels along a generated trajectory at a speed according to a similarly generated speed plan. The liquid crystal display 4 also displays the scenery around the vehicle (which may be a real-world scene or a CG virtual scene), and further displays a predicted path image showing the expected future path of the vehicle 2 within the scenery. Furthermore, if an automated driving operation is performed to avoid approaching an obstacle or lane marking around the vehicle, the display format of at least the side of the predicted path image that is close to the obstacle or lane marking to be avoided is changed. The driver assistance ECU 10 is connected to the aforementioned control unit 3, liquid crystal display 4, speaker 5, front camera 6, rear camera 7, side cameras 8A, 8B, and ultrasonic sensors 9A~9L via an in-vehicle network such as CAN. It is also connected to various sensors mounted on the vehicle 2, such as a vehicle speed sensor, acceleration sensor, and steering sensor, as well as in-vehicle devices such as a navigation system. The detailed configuration of the driver assistance ECU10 will be described later.

[0025] In addition to the components shown in Figure 1, Vehicle 2 also has other basic components as Vehicle 2, but only the configuration related to the control of the automated driving assistance system and the control related to said configuration will be explained.

[0026] Next, we will describe in detail the driver assistance ECU 10, which is part of the driver assistance system 1 provided by the vehicle 2 described above. Figure 2 is a block diagram showing the configuration of the driver assistance system 1 according to this embodiment.

[0027] As shown in Figure 2, the driver assistance ECU (Electronic Control Unit) 10 is an electronic control unit that controls the entire driver assistance system 1, and includes a CPU 31 as a calculation device and control device, a RAM 32 which is used as working memory when the CPU 31 performs various calculations and stores generated driving trajectory data, a ROM 33 which stores control programs as well as driver assistance processing programs (see Figure 3) described later, and a flash memory 34 which stores programs read from the ROM 33. The driver assistance ECU 10 also has various functions as a processing algorithm. For example, it has a function to display a vehicle surrounding image showing the area around the vehicle 2 and a predicted path image showing the expected future path of the vehicle 2 in the vehicle surrounding image on the liquid crystal display 4, and a function to change the display form of at least the side of the predicted path image that is close to the obstacle or lane marking to be avoided in relation to automatic control driving operations to avoid approaching obstacles or lane markings around the vehicle 2.

[0028] Furthermore, the driver assistance ECU 10 is connected to various sensors 36 for detecting the vehicle's behavior, such as a vehicle speed sensor, acceleration sensor, and steering sensor, as well as to various drive units 37 of the vehicle, such as the steering, brakes, and accelerator. Based on the detection results of these sensors 36, it detects the vehicle's current behavior and controls each drive unit 37 to provide automatic driving assistance for the vehicle 2. Specifically, the automatic driving assistance includes, for example, continuously detecting the vehicle's current position, the lane it is traveling in, and the positions of surrounding obstacles, and controlling the vehicle, such as the steering, drive source, and brakes, to travel along the generated driving trajectory at a speed according to the generated speed plan. However, it is also possible to perform only the steering operation automatically, while controlling the drive source and brakes is done manually.

[0029] Furthermore, ROM33 includes vehicle information DB35, which stores various information about vehicle 2. For example, it stores the installation positions (height from the ground, left-right position) of cameras and ultrasonic sensors 9A-9L installed on vehicle 2, as well as the detection axis X, overall length, vehicle width, wheelbase, and minimum turning radius. This information is entered in advance by the occupants or personnel from the vehicle manufacturer.

[0030] Next, the driver assistance processing program executed by the driver assistance ECU 10 in the driver assistance device 1 having the above configuration will be explained with reference to Figure 3. Figure 3 is a flowchart of the driver assistance processing program according to this embodiment. Here, the driver assistance processing program is executed after the ACC power supply (accessory power supply) of the vehicle 2 is turned ON, and is a program that provides user assistance using bird's-eye view images and overhead view images of the area around the vehicle while the vehicle's automatic driving assistance is being performed. The program shown in the flowchart in Figure 3 below is stored in the RAM 32 and ROM 33 of the driver assistance device 1 and executed by the CPU 31.

[0031] First, in step 1 (hereinafter abbreviated as S), the CPU 31 determines whether or not assisted driving by automated driving assistance is being performed. In this embodiment, assisted driving by automated driving assistance is performed when the user selects to perform automated driving assistance by operating the control unit 3, and it is determined that it is possible to perform driving with automated driving assistance. Furthermore, the content of the automated driving assistance includes detecting the vehicle's current position, the lane the vehicle is traveling in, and the positions of surrounding lane markings and obstacles, and automatically controlling the vehicle, such as steering, drive source, and brakes, so that the vehicle travels along the generated driving trajectory at a speed according to the generated speed plan. However, it is also possible to perform only the steering operation automatically, and control of the drive source and brakes based on manual operation.

[0032] If it is determined that assisted driving is being performed by the automated driving assistance system (S1:YES), the process proceeds to S2. Conversely, if it is determined that assisted driving is not being performed by the automated driving assistance system (S1:NO), the driving assistance processing program is terminated.

[0033] In S2, the CPU 31 generates a bird's-eye view image of the area around the vehicle, looking diagonally downwards from above, based on real-time images captured by the front camera 6, rear camera 7, and side cameras 8A and 8B. The method for generating the bird's-eye view image 40 looking diagonally downwards from the front of the vehicle is described below. As shown in Figure 4, the real-time images captured by each camera are projected onto a virtual projection plane, which is a horizontal plane corresponding to the height of the ground surface. The images projected onto the virtual projection plane are then converted into images viewed from a virtual viewpoint that looks diagonally downwards from above the vehicle 2 in the direction of travel (however, if the vehicle is reversing, it will be the rear of the vehicle), thereby generating the bird's-eye view images for each camera. The conversion to an image viewed from a virtual viewpoint (viewpoint conversion) is performed by first converting each coordinate of the image coordinate system, which is set along a plane perpendicular to the optical axis of the camera, to each coordinate of the ground coordinate system, which is set along the ground surface, and then to each coordinate of the bird's-eye view image coordinate system. The conversion formulas used for each coordinate conversion are already publicly known, so their explanation is omitted. Then, as shown in Figure 5, the bird's-eye view image is generated by combining (stitching together) the bird's-eye view image 41 obtained by changing the viewpoint of the image captured by the front camera 6, the bird's-eye view image 42 obtained by changing the viewpoint of the image captured by the rear camera 7, the bird's-eye view image 43 obtained by changing the viewpoint of the image captured by the side camera 8A, and the bird's-eye view image 44 obtained by changing the viewpoint of the image captured by the side camera 8B. Furthermore, the bird's-eye view image 45, which schematically shows the vehicle, is inserted between each of the bird's-eye view images 41 to 44. Note that for the bird's-eye view image 40, if it is to be a bird's-eye view image looking diagonally downwards in the direction of travel, the bird's-eye view image 42 obtained by changing the viewpoint of the image captured by the rear camera 7 may be excluded from the combination (i.e., it may be an image of only the area in front of the vehicle in the direction of travel). Conversely, if it is to be a bird's-eye view image looking diagonally downwards behind the vehicle when reversing, the bird's-eye view image 41 obtained by changing the viewpoint of the image captured by the front camera 6 may be excluded from the combination (i.e., it may be an image of only the area behind the vehicle).

[0034] Alternatively, instead of a bird's-eye view image, a bird's-eye view image showing the area around the vehicle from above, vertically downwards, may be generated. Or, both bird's-eye view and bird's-eye view images may be generated. The process for generating the bird's-eye view image is basically the same as generating the bird's-eye view image, with only the angle of the line of sight direction differing during viewpoint conversion, so an explanation will be omitted.

[0035] Subsequently, in S3, the CPU 31 obtains the predicted future path of the vehicle. The predicted future path of the vehicle may be (A) the path expected if the vehicle moves while maintaining its current steering angle, or (B) the path expected if the vehicle moves according to the automatic control provided by the currently running automated driving assistance system.

[0036] The path described in (A) above can be predicted by obtaining the current steering angle from the steering sensor on vehicle 2 via CAN. On the other hand, the path described in (B) can be predicted by obtaining the driving trajectory that has been pre-generated by the currently running automated driving assistance system, i.e., the future driving trajectory of the vehicle. Note that the path described in (A) will be changed accordingly if the steering angle changes, but the path described in (B) will basically remain fixed even if the steering angle changes, unless there are circumstances such as the detection of a new obstacle that needs to be avoided.

[0037] Next, in S4, the CPU 31 generates a predicted path image 51. The predicted path image 51 is an image showing the expected future path of the vehicle in the vehicle surrounding image. The expected future path of the vehicle is one of the paths (A) or (B) predicted in S3. Here, Figure 6 shows an example of the predicted path image 51 generated in S4. As shown in Figure 6, the predicted path image 51 consists of two line segments corresponding to the left and right ends of the vehicle, respectively, arranged along the expected future path of the vehicle. More specifically, it has a predicted path image 51A corresponding to the left end of the vehicle and a predicted path image 51B corresponding to the right end of the vehicle. The length L1 between the predicted path image 51A and the predicted path image 51B is arranged to maintain a distance equal to or greater than the vehicle width (vehicle width + α). Figure 6 specifically shows the predicted path image 51 generated when the vehicle's expected future path is straight. If the vehicle's expected future path is to turn right or left, the generated predicted path image 51 will also be a curved line segment to the right or left, as shown in Figure 7.

[0038] In the example above, the predicted path image 51 represents the vehicle's path using two line segments, but it is not limited to this; for example, as shown in Figure 8, it may be a single line segment with a width along the vehicle's path. In that case, the width L2 of the line segment in the predicted path image 51 should be the same as or greater than the vehicle width (vehicle width + α). Furthermore, it is desirable that the predicted path image 51, as shown in Figure 8, be a semi-transparent image. This prevents a decrease in the visibility of the vehicle's surrounding image when the predicted path image 51 is superimposed on the vehicle's surrounding image, as will be described later.

[0039] Subsequently, in S5, the CPU 31 displays a real-time bird's-eye view image 40, which represents the current environment around the vehicle and was generated in S2, on the liquid crystal display 4 as a vehicle surroundings image. Furthermore, the predicted path image 51, generated in S4, is superimposed on the bird's-eye view image 40 and displayed on the liquid crystal display 4. In this embodiment, only the bird's-eye view image 40 is displayed on the liquid crystal display 4, but the overhead view image may also be displayed, or the bird's-eye view image and the overhead view image may be displayed in a way that allows the user to switch between them.

[0040] Specific examples of the display on the LCD screen 4 are shown in Figures 6 to 8 above. By viewing the bird's-eye view image 40, the user can clarify the vehicle's future path, including areas that are difficult to see directly, and understand the environment around the vehicle. The bird's-eye view image 40 and the predicted path image 51 will continue to be displayed until the automated driving assistance is terminated (S11: YES). In addition, if the vehicle's path is updated, the shape of the predicted path image 51 will change accordingly.

[0041] Furthermore, the bird's-eye view image 40 displayed in S5 includes a vehicle image 45 showing the vehicle's appearance at its current position. It is preferable that the vehicle image 45 be semi-transparent (for example, with a transparency of 30%) rather than opaque. This makes it possible to create a bird's-eye view image 40 that makes the surrounding environment easier to perceive. On the other hand, if driving maneuvers are performed to avoid approaching an obstacle (S7:YES), as described later, the transparency of the vehicle image 45 may be reduced to clarify the positional relationship between the vehicle and the obstacle.

[0042] Next, in S6, the CPU 31 determines whether or not an obstacle has been detected around the vehicle. If an obstacle is detected, the CPU 31 also detects the position of the obstacle and the area it occupies in the bird's-eye view image 40 generated in S2. Here, an obstacle is something that the vehicle must avoid while driving, such as other vehicles, pedestrians, bicycles, walls, guardrails, and other structures. On the other hand, obstacles that can be passed over, such as steps, are excluded. Obstacle detection may be performed by image recognition on the images captured by the front camera 6, rear camera 7, and side cameras 8A and 8B, or by ultrasonic sensors 9A to 9L, or by using both in combination. As for the process of detecting obstacles from the captured image, for example, brightness correction is performed based on the brightness difference between the road surface and the obstacle on the road surface, then binarization is performed to separate the obstacle from the image, geometric processing is performed to correct distortion, and smoothing is performed to remove noise from the image, making it possible to detect the boundary line between the road surface and the obstacle. Furthermore, the type of obstacle may also be detected using known template matching or feature point detection processes. Additionally, the image recognition processing on the captured image is not limited to the above examples; for example, machine learning may be used.

[0043] If an obstacle is detected around the vehicle (S6:YES), the process proceeds to S7. Conversely, if no obstacle is detected around the vehicle (S6:NO), the process proceeds to S11.

[0044] In S7, CPU31 acquires the control details (what kind of automatic driving assistance was performed and under what circumstances) of the automatic driving assistance process that runs in parallel with this program, and determines whether or not steering operations, which are driving operations performed by automatic control to avoid approaching obstacles through automatic driving assistance, were performed. Note that the conditions for steering operations may also be that steering operations exceeding a threshold angle or a threshold speed were performed.

[0045] If it is determined that automatic steering control has been performed to avoid approaching an obstacle (S7: YES), the process proceeds to S8. Conversely, if it is determined that automatic steering control has not been performed to avoid approaching an obstacle (S7: NO), the process proceeds to S11.

[0046] Next, in S8, the CPU 31 determines whether there are obstacles to be avoided by the automatically controlled steering operation located on both sides of the vehicle 2 (to the right and left of the vehicle's direction of travel). For example, if there are obstacles on both the right and left sides of the vehicle 2's direction of travel, and steering operations are performed to pass between each obstacle, the determination is YES. Furthermore, in S8, not only obstacles currently being avoided by the automatic control may be included, but obstacles that may be avoided in the future. For example, if the vehicle 2 avoids an obstacle on its left side and then avoids an obstacle on its right side, the determination is YES.

[0047] Furthermore, on narrow roads such as side streets, there is a high probability that there will be obstacles on both sides of vehicle 2 that need to be avoided by steering. Therefore, in S8, it is also possible to determine whether the road on which the vehicle is traveling is a narrow road such as a side street (for example, a road less than 4m wide). In addition, even if the road is wider than a side street and allows two-way traffic, if there is no center line, oncoming vehicles will be obstacles that need to be avoided by steering. Therefore, in S8, it is also possible to determine whether the road on which the vehicle is traveling is a road without a center line. In that case, the determination in S8 can also be made from map information and the vehicle's current position.

[0048] If the automatic steering control determines that the obstacle to be avoided is located on both sides of vehicle 2 (S8: YES), the system proceeds to S10. Conversely, if the automatic steering control determines that the obstacle to be avoided is located only on either the right or left side of vehicle 2 (S8: NO), the system proceeds to S9.

[0049] In S9, the CPU 31 changes only the display form of the predicted path image 51 displayed on the liquid crystal display 4 that is close to the obstacle to be avoided by automatic control. In this embodiment, the external shape of the predicted path image 51 is changed so that the height of at least a portion of the area of ​​the predicted path image 51 that is close to the obstacle is increased.

[0050] The details of the S9 process will be explained below with specific examples. For example, Figure 9 shows an example of a change in display format when the predicted path image 51 is displayed using two line segments: a predicted path image 51A corresponding to the left end of the vehicle and a predicted path image 51B corresponding to the right end of the vehicle. In the example shown in Figure 9, the obstacle 55 located to the right front of the vehicle's path is the obstacle to be avoided, so the display format is changed for predicted path image 51B, which is the line segment on the side closer to the obstacle 55, out of predicted path image 51A and predicted path image 51B. More specifically, the CPU 31 changes the external shape of predicted path image 51B so that its height increases. As a result, as shown in Figure 9, the predicted path image 51B changes from a two-dimensional line drawn on the road surface to a surface with height. This makes it possible to display a new wall between the vehicle image 45 and the obstacle 55, and to make it appear as if the vehicle image 45 is traveling along the wall in accordance with the vehicle 2's movement. Furthermore, the height of the wall may be fixed, or it may be changed depending on the type and location of the obstacle to be avoided. For example, if the type of obstacle 55 is one that requires particular attention from the vehicle (e.g., a pedestrian), or if the obstacle 55 approaches the vehicle within a predetermined distance, the height of the wall can be changed to become higher.

[0051] Alternatively, the height of the wall may be set according to the height of the obstacle 55 to be avoided. Specifically, it is possible to set the wall height as high as possible within the range in which the obstacle 55 is visible without being hidden by the wall in the bird's-eye view image 40.

[0052] Furthermore, the timing for changing the display format of the predicted path image 51 may be the timing when driving operations to avoid approaching an obstacle are performed (start of driving operations), immediately before driving operations to avoid approaching an obstacle are performed, or after driving operations to avoid approaching an obstacle are performed (after a predetermined time has elapsed from the start of driving operations).

[0053] Furthermore, the display format of the predicted path image 51 may be changed all at once or in stages. In particular, when changing in stages, the wall may be depicted as gradually rising from the ground, as shown in Figure 10. This makes it possible to give the user a sense of security, as if they were protected by a hidden wall even before the wall appears. Moreover, the change in display format may not only be a change in the external shape, but also a change in the display color, a flashing display, a combination of these, or all of these simultaneously. In particular, when the condition is met that the obstacle 55 is approaching the vehicle within a predetermined distance, the display color may be changed or a flashing display may be made in addition to the change in the external shape. Alternatively, when the condition is met that the type of obstacle 55 is an obstacle that requires particular attention from the vehicle (for example, a pedestrian), the display color may be changed or a flashing display may be made in addition to the change in the external shape.

[0054] On the other hand, in S9, the external shape of the predicted path image 51 may be changed, and only the display color may be changed or the image may be made to flash. For example, as shown in Figure 9, if there is an obstacle 55 to be avoided to the right front of the vehicle's path, the display color of the predicted path image 51B, which is the line segment on the side adjacent to the obstacle 55, can be changed to red or made to flash.

[0055] Furthermore, in S9, in addition to the change in the display format of the predicted path image 51, the speaker 5 may also provide voice or sound effects to indicate that an automatic control driving operation has been performed to avoid approaching an obstacle.

[0056] Figure 9 illustrates the case where the predicted path image 51 is displayed using two line segments: a predicted path image 51A corresponding to the left end of the vehicle and a predicted path image 51B corresponding to the right end of the vehicle. However, when the predicted path image 51 is displayed using a single line segment with a width along the vehicle's path, the display format is changed as shown in Figure 11.

[0057] In the example shown in Figure 11, the obstacle 55 located to the right and in front of the vehicle's path is the obstacle to be avoided. Therefore, the display format is changed for a predetermined width W area of ​​the predicted path image 51, starting from at least the end closest to the obstacle 55 (the right end in Figure 11). More specifically, the CPU 31 changes the external shape of the predicted path image 51 to increase its height in the corresponding area. As a result, as shown in Figure 11, a portion of the predicted path image 51 changes from a two-dimensional line drawn on the road surface to a surface with height. This creates the appearance of a new wall between the vehicle image 45 and the obstacle 55, and makes it possible to display the vehicle image 45 as if it were traveling along the wall in accordance with the vehicle's movement.

[0058] Furthermore, by changing the display mode of the predicted path image 51 shown in Figures 9 and 11, the image of the wall that appears may be displayed as semi-transparent (for example, with a transparency of 30%) instead of opaque. Also, the transparency of the wall image may be changed according to its height from the ground, with higher locations having higher transparency. This makes it possible to create a bird's-eye view image 40 that makes the surrounding environment easier to perceive. On the other hand, the distance between the vehicle 2 and the obstacle may be measured, and the transparency of the wall image may be changed according to the measured distance. For example, the transparency may be gradually changed so that the transparency decreases as the distance between the vehicle 2 and the obstacle decreases, and increases as the distance between the vehicle 2 and the obstacle increases. Furthermore, the transparency of the wall image may be changed according to the elapsed time since the start of driving operations to avoid approaching the obstacle, or the amount of driving operations. For example, the transparency may be gradually decreased as time elapses since the start of driving operations to avoid approaching the obstacle. Alternatively, the transmittance may be changed according to the amount of change in the steering angle to avoid approaching an obstacle, or more specifically, the transmittance may be changed to decrease as the steering angle increases.

[0059] Furthermore, the predicted path image 51, whose display format was changed in S9, is gradually returned to its original display format as the driving operation to avoid approaching the obstacle is completed. For example, after control to return the steering to the neutral position is started after avoiding the obstacle, it is possible to gradually lower the height of the appearing wall and return it to the original line segment near the neutral position. Alternatively, the display format change may continue until the steering reaches the neutral position, and the wall height may be switched in one step at the moment the steering reaches the neutral position. Alternatively, the display format of the predicted path image 51 may be changed and then returned to the original display format after a predetermined time has elapsed or after moving a predetermined distance. After that, the process proceeds to S11.

[0060] Meanwhile, in S10, the CPU 31 changes the display format of the predicted path image 51 displayed on the liquid crystal display 4, specifically the left and right sides. For example, Figure 12 shows an example of a change in display format when the predicted path image 51 is displayed using two line segments: a predicted path image 51A corresponding to the left end of the vehicle and a predicted path image 51B corresponding to the right end of the vehicle.

[0061] In the example shown in Figure 12, obstacles 55 to the right front and obstacles 56 to the left front of the vehicle's path become obstacles to be avoided by automatic steering control. In the example shown in Figure 12, the display format is changed for both predicted path image 51A and predicted path image 51B. More specifically, the CPU 31 changes the external shape of predicted path image 51A to increase its height, and also changes the external shape of predicted path image 51B to increase its height. As a result, as shown in Figure 12, predicted path image 51A and predicted path image 51B change from two-dimensional lines drawn on the road surface to surfaces with height. This causes a new wall to appear between the vehicle image 45 and obstacles 55 and 56, and makes it possible to display the vehicle image 45 as if it is traveling along the wall in accordance with the vehicle 2's movement. The height of the wall may be fixed, or it may be changed according to the type and position of the obstacles to be avoided. For example, if the types of obstacles 55 and 56 are obstacles that require particular attention from the vehicle (e.g., pedestrians), or if obstacles 55 and 56 approach the vehicle within a predetermined distance, the height of the wall can be changed to increase.

[0062] Subsequently, in S11, the CPU 31 determines whether or not to terminate the assisted driving provided by the automated driving support system. Here, the termination of the assisted driving provided by the automated driving support system may be conditional on, for example, the user performing a predetermined termination operation on the control unit 3, or on the shift position being moved to "P" or the engine being turned off. Alternatively, the termination may be conditional on the situation becoming such that it is no longer possible to continue the automated driving support system.

[0063] Then, if it is determined that the assisted driving by the automated driving support system has ended (S11:YES), the driving support processing program is terminated. On the other hand, if it is determined that the assisted driving by the automated driving support system has not ended (S11:NO), the program returns to S2, and the display of the bird's-eye view image 40 and the predicted path image 51 on the LCD display 4 continues.

[0064] In addition, in the automated driving assistance system, lane departure prevention assistance is provided, which includes steering control to avoid the aforementioned obstacles, as well as steering operations to avoid deviating from lane markings. In the driving assistance processing program according to this embodiment, when an automated driving operation is performed to avoid approaching an obstacle, the display format of the predicted path image 51 on the side approaching the obstacle to be avoided is changed (S9, S10). However, even when a driving operation is performed to avoid approaching a lane marking (e.g., road center line, lane boundary line, road outer line, parking space line, etc.) instead of an obstacle, the display format of the predicted path image 51 on the side approaching the lane marking to be avoided may also be changed. The timing of changing the display format of the predicted path image 51 may be when a driving operation to avoid approaching a lane marking is performed, just before a driving operation to avoid approaching a lane marking is performed, or after a driving operation to avoid approaching a lane marking is performed. In detail, only the target has been replaced from an obstacle to a lane marking, and the processing content is the same.

[0065] As described in detail above, according to the driving support device 1 and the computer program executed by the driving support device 1 according to this embodiment, a vehicle surrounding image showing the area around the vehicle 2 and a predicted path image 51 showing the expected future path of the vehicle 2 in the vehicle surrounding image are displayed on the liquid crystal display 4 (S5). In addition, in connection with the driving operation by automatic control to avoid approaching obstacles or lane markings around the vehicle 2, the display form of at least the side of the predicted path image 51 that is close to the obstacle or lane marking to be avoided is changed (S9, S10). As a result, it is possible to intuitively understand to the user that automatic control has been performed to avoid approaching obstacles or lane markings around the vehicle. Furthermore, when changing the display format of the predicted path image 51, the external shape of the predicted path image is changed so that the height of at least a portion of the predicted path image 51 that is close to an obstacle or lane marking is increased. This makes it possible to display the vehicle as if a new wall were appearing between the vehicle and the obstacle or lane marking, and as if the vehicle were traveling along the wall. Furthermore, when changing the display format of the predicted path image 51, the display color of the side of the predicted path image 51 that is close to an obstacle or lane marking is changed, making it possible to visually indicate to the user that automatic control has been performed to avoid approaching an obstacle or lane marking around the vehicle. As a result, even if driving operations to avoid an obstacle or lane marking are performed automatically, the user will not be confused. Furthermore, when changing the display format of the predicted path image 51, the side of the predicted path image 51 that is close to an obstacle or lane marking will be displayed in a flashing light. This makes it possible to visually indicate to the user that automatic control has been performed to avoid approaching an obstacle or lane marking around the vehicle. As a result, even if driving operations to avoid an obstacle or lane marking are performed automatically, the user will not be confused.

[0066] It should be noted that the present invention is not limited to the embodiments described above, and various improvements and modifications are possible without departing from the spirit of the invention. For example, in this embodiment, a landscape image of the area around the vehicle captured by a camera is displayed on the in-vehicle liquid crystal display 4, and a predicted path image 51 is further displayed on the liquid crystal display 4, thereby superimposing the predicted path image 51 onto the landscape around the vehicle. However, a head-up display system (HUD) may also be used as a means to display the image to be superimposed onto the landscape around the vehicle. In a head-up display system, for example, the predicted path image 51 displayed on the in-vehicle display is reflected off the windshield for the user to see, thereby allowing the user to see the predicted path image 51 (more precisely, a virtual image of the predicted path image 51) superimposed onto the actual landscape seen through the windshield.

[0067] In this embodiment, the bird's-eye view image 40, generated from images captured by the front camera 6, rear camera 7, and side cameras 8A and 8B, is displayed on the liquid crystal display 4 as the scenery image around the vehicle. However, the overhead view image may also be displayed on the liquid crystal display 4 simultaneously with the bird's-eye view image 40. For example, the screen of the liquid crystal display 4 can be divided into two screens, left and right, with the bird's-eye view image displayed on one screen and the overhead view image on the other. The predicted path image 51 may also be superimposed on the overhead view image displayed on the liquid crystal display 4. Alternatively, instead of processed images, the raw images captured by the front camera 6 or rear camera 7 may be displayed on the liquid crystal display 4 as the scenery image around the vehicle, and the predicted path image 51 may be superimposed on those images.

[0068] Furthermore, the scenery around the vehicle displayed on the liquid crystal display 4 may be a virtual scenery image (for example, a 3D map image) that recreates the area around the vehicle using computer graphics (CG), rather than an image captured by a camera. For example, a 3D map image of the area around the current position can be acquired or generated and displayed on the liquid crystal display 4, and the predicted path image 51 can be displayed on the displayed 3D map image. In this case as well, the same effect will be achieved.

[0069] Furthermore, in this embodiment, the display format of the predicted path image 51 is changed when steering operations are performed to avoid an obstacle (S9, S10). However, in addition to avoidance by steering operations, avoidance by braking operations (deceleration) can also be considered as an automated driving support method for avoiding obstacles. Therefore, the display format of the predicted path image 51 may also be changed when braking operations are performed to avoid an obstacle. Moreover, the display format of the predicted path image 51 may also be changed in the same way when other vehicle operations are performed to avoid obstacles, not limited to steering operations or braking operations.

[0070] Furthermore, in this embodiment, the bird's-eye view images 40 shown in Figures 9 and 12 are displayed on the premise that driving is being performed with automated driving assistance. However, the bird's-eye view images 40 shown in Figures 9 and 12 may also be displayed while driving manually. In that case, when steering or braking operations are performed to avoid obstacles during manual driving, the display form of the predicted path image 51 will be changed (S9, S10).

[0071] Furthermore, the execution order of each step in the driver assistance processing program shown in Figure 3 is just an example, and the execution order can be changed as appropriate. For example, although the vehicle surrounding image was generated in S2 and the predicted path image was generated in S5, the vehicle surrounding image could be generated after the predicted path image. Also, some steps included in the driver assistance processing program can be omitted; for example, steps S8 and S10 may be omitted.

[0072] Furthermore, in this embodiment, the display format of the predicted path image 51 is changed simultaneously with the implementation of an automatic control driving operation to avoid approaching an obstacle or lane marking (S9, S10). However, the display format may be changed on the condition that an automatic control driving operation to avoid approaching an obstacle or lane marking is performed and the vehicle's steering angle exceeds a threshold. Alternatively, the display format may be changed when an automatic control driving operation to avoid approaching an obstacle or lane marking is performed and a predetermined time has elapsed since the driving operation was performed, or on the condition that a steering operation exceeding a threshold speed is performed, or on the condition that the vehicle body of vehicle 2 has turned by a predetermined amount or more.

[0073] [Note] The embodiments described above also illustrate the following inventions. In the following description, the names and expressions of corresponding components in the embodiments, as well as the reference numerals used in the drawings, are indicated in parentheses for reference. However, the components of each invention are not limited to these indications.

[0074] (Invention A) In addition to changing the display format of the predicted path image (51), the driving support device (1) according to claim 1 provides voice notification that an automatic control driving operation has been performed to avoid approaching the obstacles (55, 56) or lane markings.

[0075] This makes it possible to intuitively understand, through both visual and auditory means, that an automated driving operation has been performed to avoid an obstacle or lane marking by automatic control.

[0076] (Invention B) The driving support device (1) according to claim 1, wherein if the obstacles (55, 56) or lane markings that are the subject of the automatic control driving operation are located on both sides of the vehicle (2), the display format of the predicted path image (51) on both sides is changed.

[0077] According to this, when there are obstacles or lane markings to avoid on both the left and right sides of the vehicle, it is possible to inform the user that the driver assistance system has appropriately recognized them and is taking automatic control, thereby providing the user with greater peace of mind.

[0078] (Invention C) The vehicle surrounding image (40) is an image captured by an imaging device (6, 7, 8A, 8B) provided by the vehicle (2), or an image obtained by viewpoint transformation and synthesis of captured images, according to claim 1, the driving support device (1).

[0079] According to this, it is possible to display an image of the vehicle's surroundings that accurately reflects the vehicle's environment, including areas that are blind spots from the driver's perspective, thereby improving the ease of understanding the automatic control system.

[0080] (Invention D) An aerial view of the area around the vehicle, showing the area from above, is displayed on the display device simultaneously with the vehicle surroundings image (40). The driver assistance device (1) according to claim 1, wherein the predicted path image (51) is also displayed in relation to the overhead view image of the area around the vehicle.

[0081] According to this, it becomes easier to understand the positional relationship between the vehicle and obstacles, and the content of the automatic control becomes easier to understand intuitively.

[0082] (Invention E) The driving support device according to claim 2, wherein the external shape of the predicted path image (51) is changed in stages.

[0083] According to this, it is possible not only to make the user intuitively feel that their vehicle is protected by driver assistance systems, but also to give the user a sense of security as if they were protected by a hidden wall, even before the external shape is changed.

[0084] (Invention F) The driving assistance device (1) according to claim 1, wherein the automatic control driving operation is performed to avoid approaching obstacles (55, 56) or lane markings in the vicinity of the vehicle (2), and the display form is changed at the timing when the steering angle of the vehicle exceeds a threshold.

[0085] According to this, the condition for a change in the display format of the predicted path image includes the steering angle exceeding a threshold. Therefore, when there are multiple obstacles to avoid, it becomes possible to prevent the user from becoming confused by the continuous changes in the display format.

[0086] (Invention G) The driver assistance device (1) according to claim 1, wherein the predicted path image (51) shows either the predicted path of the vehicle (2) when it moves while maintaining the steering angle of the vehicle (2) at the time of display, or the predicted path of the vehicle when it moves in accordance with the automatic control.

[0087] According to this, the predicted path image can display the path that the vehicle is expected to travel, making it possible to convey to the user the safety of the automated driving control system.

[0088] (Invention H) The predicted path image (51) is an image of a line segment arranged along the path of the vehicle (2) and having a width equal to or greater than the width of the vehicle (2), The driving support device (1) according to claim 1, wherein when changing the display format of the predicted path image, the display format is changed for a predetermined width area of ​​the predicted path image, starting from at least the end of the image that is close to the obstacle (55, 56) or lane markings that are to be avoided.

[0089] According to this, users can grasp the expected path of the vehicle at a glance, improving convenience.

[0090] (Invention I) The predicted path image (51) is an image of two line segments arranged along the path of the vehicle (2) and corresponding to the left and right ends of the vehicle (2), The driving support device (1) according to claim 1, wherein when changing the display format of the predicted path image (51), the display format is changed with respect to at least the line segment of the two line segments that is close to the obstacle (55, 56) or lane marking that is to be avoided.

[0091] According to this, both ends of the predicted path of vehicle 2 are clearly represented, allowing the user to clearly understand the predicted path. [Explanation of Symbols]

[0092] 1…Driver assistance system, 2…Vehicle, 4…LCD display (display device), 6…Front camera, 7…Rear camera, 8A,8B…Side cameras, 10…Driver assistance ECU, 31…CPU, 40…Bird's-eye view image (image of the area around the vehicle), 51…Predicted path image, 55,56…Obstacles

Claims

1. A driver assistance device that automatically controls at least a part of the driving operations of a vehicle, The display device displays a vehicle surroundings image showing the area around the vehicle, and a predicted path image showing the expected future path of the vehicle in the vehicle surroundings image. A driving assistance device that, in connection with the automatic control driving operation for avoiding approaching obstacles or lane markings in the vicinity of the vehicle, changes the display format of the predicted path image, at least on the side that is close to the obstacle or lane marking to be avoided.

2. The driving assistance device according to claim 1, wherein changing the display form of the predicted path image means changing the external shape of the predicted path image so that the height of at least a portion of the predicted path image that is close to the obstacle or lane markings increases.

3. The driving support device according to claim 1 or claim 2, wherein changing the display format of the predicted path image means changing the display color of the side of the predicted path image that is close to the obstacle or lane marking.

4. The driving assistance device according to claim 1 or claim 2, wherein changing the display format of the predicted path image means flashing the side of the predicted path image that is close to the obstacle or lane marking.

Images