Driving assistance device, driving assistance method, and program

Abstract

To more properly assist in driving in accordance with surrounding states.SOLUTION: A driving assistance device comprises a recognition unit that recognizes the surrounding state of a vehicle, a display control unit that causes a display device to show an image for assisting a crew who drives the vehicle in a plurality of preset display modes, and a driving control unit that controls the driving operation of the vehicle by the crew or the traveling of the vehicle. The plurality of display modes include a display mode that corresponds to a mode in which at least the vehicle travels down a narrow road. The display control unit causes transition to one of the plurality of display modes on the basis of the surrounding state recognized by the recognition unit. When showing an image in the display mode that corresponds to the mode of traveling down a narrow road, the display control unit shows an image that is simulated to be the vehicle; when there exists an object in the surrounding of the vehicle, the display control unit shows a direction in which the vehicle should proceed, and shows an image that prompts the crew to steer the vehicle.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to a driving assistance device, a driving assistance method, and a program. [Background technology]

[0002] In recent years, efforts to provide access to sustainable transportation systems that take into consideration vulnerable traffic participants have been gaining momentum. To achieve this, efforts are being focused on research and development to further improve traffic safety and convenience through research and development of driving assistance technologies. In this regard, there is known a technology that determines whether to permit passing assistance for a vehicle on a narrow road based on the speed of an oncoming vehicle when the vehicle passes another vehicle, and displays an image captured by a side camera on a display if it is determined that passing assistance is permitted (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2020-149204 Summary of the Invention [Problem to be solved by the invention]

[0004] However, in driving assistance technology, there are cases where it is not possible to display an appropriate image according to various other surrounding conditions.

[0005] The present invention has been made in consideration of the above circumstances, and aims to provide a driving assistance device, a driving assistance method, and a program that can provide more appropriate driving assistance in accordance with the surrounding conditions, thereby contributing to the development of sustainable transportation systems. [Means for solving the problem]

[0006] A driving assistance device, a driving assistance method, and a program according to the present invention employ the following configuration. (1): A driving assistance device according to one aspect of the present invention includes a recognition unit that recognizes the surrounding conditions of a vehicle; a display control unit that causes a display device to display images to assist a driver driving the vehicle in a plurality of preset display modes; and a driving control unit that controls the driving operation of the vehicle by the driver or the traveling of the vehicle, wherein the plurality of display modes include display modes corresponding to at least a first mode in which the vehicle travels along the direction of extension of a road, a second mode in which the vehicle travels through an intersection, a third mode in which the vehicle travels through a merging section, a fourth mode in which the vehicle travels through a narrow road, and a fifth mode in which the vehicle travels while avoiding contact with objects, and the display control unit transitions to one of the plurality of display modes based on the surrounding conditions recognized by the recognition unit, and when displaying an image in the display mode corresponding to the fourth mode, the display control unit displays an image that resembles the vehicle, and further, when an object is present around the vehicle, displays the direction in which the vehicle should travel and an image that prompts the driver to perform a steering operation.

[0007] (2): In the above aspect (1), the display control unit selects and displays an image to be displayed based on a display mode transitioned based on the surrounding conditions recognized by the recognition unit, from among at least an image that resembles the vehicle, an image showing the position of an object around the vehicle, an image showing the direction in which the vehicle should proceed, a target position to which the vehicle will change lanes, an image that prompts the vehicle to accelerate or decelerate, and an image that prompts the occupant to steer.

[0008] (3): In the above aspect (2), the display control unit displays the image simulating the vehicle in the third mode and the fourth mode as an image of the vehicle viewed from above, and displays the image simulating the vehicle M in the first mode, the second mode, and the fifth mode as an image of the vehicle M viewed from behind.

[0009] (4): In any one of the above aspects (1) to (3), when the display control unit displays an image in a display mode corresponding to the fourth mode, the image resembling the vehicle is displayed larger than the image resembling the vehicle displayed in the third mode.

[0010] (5): In any one of the above aspects (1) to (4), the display control unit displays an image in a display mode corresponding to the fourth mode when the width of the road on which the vehicle is traveling recognized by the recognition unit is less than a predetermined value, or when the vehicle is traveling while avoiding an object in front of the vehicle or when the vehicle is passing an object traveling in the opposite direction, and the distance to the object is less than a predetermined distance and the contact time between the vehicle and the object is equal to or greater than a threshold value.

[0011] (6) In the above aspect (5), the driving control unit performs control to guide the occupant to perform a steering operation to avoid contact between the vehicle and the object.

[0012] (7) In the above aspect (6), the driving control unit performs an operation to guide the steering by the occupant by using a reaction force against the steering wheel operated by the occupant.

[0013] (8): In any one of the above aspects (1) to (7), when the recognition unit recognizes another vehicle stopped in front of the vehicle waiting to turn right or left, the display control unit displays an image in a display mode corresponding to the fourth mode.

[0014] (9): A driving assistance method according to one aspect of the present invention includes a computer recognizing a surrounding situation of a vehicle, displaying images on a display device in a plurality of preset display modes to assist a driver driving the vehicle, and executing driving control to control the driving operation of the vehicle by the driver or the traveling of the vehicle, the plurality of display modes including display modes corresponding to at least a first mode in which the vehicle travels along the direction of extension of a road, a second mode in which the vehicle travels through an intersection, a third mode in which the vehicle travels through a merging section, a fourth mode in which the vehicle travels through a narrow road, and a fifth mode in which the vehicle travels while avoiding contact with objects, the computer transitions to one of the plurality of display modes based on the recognized surrounding situation, and when an image is displayed in the display mode corresponding to the fourth mode, an image simulating the vehicle is displayed, and further, when an object is present around the vehicle, the computer displays the direction in which the vehicle should travel and an image prompting the driver to perform a steering operation.

[0015] (10): A program according to one aspect of the present invention causes a computer to recognize the surrounding conditions of a vehicle, display on a display device images to assist a driver of the vehicle in a plurality of preset display modes, and execute driving control to control the driving operation of the vehicle by the driver or the traveling of the vehicle, the plurality of display modes including display modes corresponding to at least a first mode in which the vehicle travels along the direction of a road, a second mode in which the vehicle travels through an intersection, a third mode in which the vehicle travels through a merging section, a fourth mode in which the vehicle travels through a narrow road, and a fifth mode in which the vehicle travels while avoiding contact with objects, and transitions to one of the plurality of display modes based on the recognized surrounding conditions, and when an image is displayed in the display mode corresponding to the fourth mode, displays an image simulating the vehicle, and further, when an object is present around the vehicle, displays the direction in which the vehicle should travel and an image prompting the driver to perform a steering operation. [Effects of the Invention]

[0016] According to the above aspects (1) to (10), more appropriate driving assistance can be provided in accordance with the surrounding circumstances. [Brief explanation of the drawings]

[0017] [Figure 1] 1 is a configuration diagram of a vehicle system 1 that uses a driving assistance device according to an embodiment. [Figure 2] FIG. 2 is a diagram for explaining the function of a determination unit 120. [Figure 3] FIG. 2 is a diagram illustrating a first display mode in a normal mode. [Figure 4] FIG. 10 is a diagram illustrating a second display mode in the normal mode. [Figure 5] FIG. 10 is a diagram illustrating a third display mode in the normal mode. [Figure 6] FIG. 10 is a diagram illustrating a fourth display mode in the normal mode. [Figure 7] FIG. 10 is a diagram illustrating a fifth display mode in the normal mode. [Figure 8] FIG. 10 is a diagram illustrating a sixth display mode in the normal mode. [Figure 9] FIG. 10 is a diagram for explaining a first display mode in an intersection mode. [Figure 10] FIG. 10 is a diagram for explaining a second display mode in an intersection mode. [Figure 11] FIG. 10 is a diagram illustrating a first display mode in a merging mode. [Figure 12] FIG. 10 is a diagram illustrating a second display mode in the merging mode. [Figure 13] FIG. 10 is a diagram illustrating a third display mode in the merging mode. [Figure 14] FIG. 10 is a diagram illustrating a fourth display mode in the merging mode. [Figure 15] FIG. 10 is a diagram for explaining a first display mode in a narrow passage passing mode. [Figure 16]FIG. 10 is a diagram for explaining a second display mode in a narrow passage passing mode. [Figure 17] FIG. 10 is a diagram for explaining a third display mode in a narrow passage passing mode. [Figure 18] FIG. 10 is a diagram for explaining a first display mode in the emergency avoidance mode. [Figure 19] 3 is a flowchart showing an example of a flow of processing executed by the driving assistance device 100 of the embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0018] Hereinafter, a description will be given of an embodiment of a driving assistance device, a driving assistance method, and a program according to the present invention with reference to the drawings. Note that the following description will be given for a case where a law stipulating driving on the left side applies, but if a law stipulating driving on the right side applies, the left and right should be read in reverse.

[0019] [Overall configuration] FIG. 1 is a configuration diagram of a vehicle system 1 that uses a driving assistance device according to an embodiment. A vehicle (hereinafter, referred to as the host vehicle M) on which the vehicle system 1 is mounted may be, for example, a two-wheeled, three-wheeled, or four-wheeled vehicle, and its drive source may be an internal combustion engine such as a diesel engine or a gasoline engine, an electric motor, or a combination thereof. The electric motor operates using power generated by a generator connected to the internal combustion engine, or discharge power from a secondary battery or a fuel cell. In the following, as an example, the driving assistance device may be applied to an autonomous vehicle. Autonomous driving refers to, for example, automatically controlling one or both of the steering and acceleration / deceleration of the host vehicle M to perform driving control. The driving control of the host vehicle M may include various driving assistance functions, such as adaptive cruise control (ACC), auto lane changing (ALC), lane keeping assistance system (LKAS), forward collision warning (FCW), and collision mitigation braking system (CMBS). An autonomous vehicle may be partially or entirely controlled by a passenger (driver) through manual driving.

[0020] The vehicle system 1 includes, for example, a camera (an example of an imaging unit) 10, a radar device 12, a LIDAR (Light Detection and Ranging) 14, an object recognition device 16, a communication device 20, an HMI (Human Machine Interface) 30, vehicle sensors 40, a navigation device 50, a driver monitor camera 70, a driving operator 80, a driving assistance device 100, a driving force output device 200, a braking device 210, and a steering device 220. These devices and equipment are connected to each other via multiplexed communication lines such as a CAN (Controller Area Network) communication line, serial communication lines, a wireless communication network, etc. Note that the configuration shown in FIG. 1 is merely an example, and some of the configuration may be omitted, or other configurations may be added. A combination of the camera 10, the radar device 12, and the LIDAR 14 is an example of an "external sensor ES." The external sensor ES may include other detection units (e.g., sonar) that recognize the surrounding conditions of the vehicle, or may include the object recognition device 16. Furthermore, the external sensor ES may have a simple configuration such as only the camera 10, or only the camera 10 and the radar device 12. The HMI 30 is an example of an "output device."

[0021] The camera 10 is a digital camera that uses a solid-state imaging element such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor). The camera 10 is attached to any location of the host vehicle M. For example, when capturing an image in front of the host vehicle M, the camera 10 is attached to the top of the front windshield or the back of the rearview mirror. When capturing an image behind the host vehicle M, the camera 10 is attached to the top of the rear windshield or the back door. When capturing an image of the sides and rear of the host vehicle M, the camera 10 is attached to a door mirror or the like. The camera 10 periodically and repeatedly captures images of the surroundings of the host vehicle M, for example. The camera 10 may be a stereo camera.

[0022] The radar device 12 emits radio waves such as millimeter waves around the vehicle M and detects radio waves reflected by an object (reflected waves) to detect at least the position (distance and direction) of the object. The radar device 12 is attached to any location on the vehicle M. The radar device 12 may detect the position and speed of an object using an FM-CW (Frequency Modulated Continuous Wave) method.

[0023] The LIDAR 14 irradiates the surroundings of the vehicle M with light (or electromagnetic waves with wavelengths similar to light) and measures the scattered light. The LIDAR 14 detects the distance to the target based on the time between light emission and light reception. The irradiated light is, for example, pulsed laser light. The LIDAR 14 is attached to any location on the vehicle M.

[0024] The object recognition device 16 performs sensor fusion processing on the detection results from some or all of the camera 10, radar device 12, and LIDAR 14 included in the external sensors ES to recognize the position, type, speed, etc. of the object. The object recognition device 16 outputs the recognition results to the driving assistance device 100. The object recognition device 16 may output the detection results from the camera 10, radar device 12, and LIDAR 14 directly to the driving assistance device 100. The object recognition device 16 may be omitted from the vehicle system 1.

[0025] The communication device 20 communicates with other vehicles in the vicinity of the vehicle M, for example, using a cellular network, a Wi-Fi network, Bluetooth (registered trademark), DSRC (Dedicated Short Range Communication), etc., or communicates with various server devices via a wireless base station.

[0026] The HMI 30 presents various information to an occupant of the vehicle M under the control of the HMI control unit 140 and accepts input operations from the occupant. The HMI 30 includes, for example, a display device 32. The display device 32 is, for example, a liquid crystal display (LCD) or an organic electroluminescence (EL) display device. The display device 32 is provided, for example, in front of the driver's seat (the seat closest to the steering wheel) on the instrument panel, at a position where the occupant can see it through the gap in the steering wheel or over the steering wheel. The display device 32 may also be provided in the center of the instrument panel. The display device 32 may also be a head-up display (HUD). The HUD projects an image onto a portion of the front windshield in front of the driver's seat, allowing the occupant sitting in the driver's seat to see a virtual image. The display device 32 displays an image generated by the HMI control unit 140, which will be described later. The HMI 30 may also include, for example, a speaker, switches, a microphone, a buzzer, a touch panel, keys, etc. The HMI 30 may also include a driving changeover switch that switches between automatic driving and manual driving by the occupant. The switch may include, for example, a turn signal switch (directional indicator). The turn signal switch may be provided, for example, on the steering column or steering wheel. The turn signal switch is an example of an operation unit that accepts, for example, an instruction from the occupant to change lanes of the vehicle M. The switch may also include a switch for adjusting the set speed of the vehicle M.

[0027] The vehicle sensors 40 include a vehicle speed sensor that detects the speed of the host vehicle M, an acceleration sensor that detects acceleration, a yaw rate sensor that detects the angular velocity around a vertical axis, and a direction sensor that detects the orientation of the host vehicle M. The vehicle sensors 40 may also include a steering angle sensor that detects the steering angle of the host vehicle M (which may be the angle of the steering wheels or the operating angle of the steering wheel). The vehicle sensors 40 may also include a position sensor that acquires the position of the host vehicle M. The position sensor is, for example, a sensor that acquires position information (longitude and latitude information) from a GPS (Global Positioning System) device. The position sensor may also be a sensor that acquires position information using a GNSS (Global Navigation Satellite System) receiver 51 of the navigation device 50.

[0028] The navigation device 50 includes, for example, a GNSS (Global Navigation Satellite System) receiver 51, a navigation HMI 52, and a route determination unit 53. The navigation device 50 stores map information 54 in a storage device such as a hard disk drive (HDD) or flash memory. The GNSS receiver 51 identifies the position of the vehicle M based on signals received from GNSS satellites. The position of the vehicle M may be identified or supplemented by an inertial navigation system (INS) that uses the output of the vehicle sensors 40. The navigation HMI 52 includes a display device, a speaker, a touch panel, keys, etc. The navigation HMI 52 may share some or all of the components with the HMI 30 described above. The route determination unit 53 determines a route (hereinafter, a route on a map) from the position of the vehicle M identified by the GNSS receiver 51 (or any input position) to a destination input by the occupant using the navigation HMI 52, by referring to the map information 54. The map information 54 is information that represents road shapes using, for example, links indicating roads and nodes connected by the links. The map information 54 may also include information on road curvature and POI (Point of Interest) information. The map information 54 may also include, for example, information on the center of lanes or information on lane boundaries, as well as road information, traffic regulation information, address information (address and postal code), facility information, telephone number information, and the like. The map information 54 may be updated as needed by the communication device 20 communicating with other devices.

[0029] The navigation device 50 may provide route guidance using the navigation HMI 52 based on the route on the map. The navigation device 50 may be realized, for example, by the functions of a terminal device such as a smartphone or tablet device carried by the occupant. The navigation device 50 may transmit the current position and destination to a navigation server via the communication device 20 and obtain a route equivalent to the route on the map from the navigation server.

[0030] The driver monitor camera 70 is, for example, a digital camera that uses a solid-state imaging element such as a CCD or CMOS. The driver monitor camera 70 is attached to any location on the vehicle M in a position and orientation that allows it to capture an image of the head of an occupant (hereinafter, driver) seated in the driver's seat of the vehicle M from the front (in an orientation that captures the face). For example, the driver monitor camera 70 is attached to the top of a display device provided in the center of the instrument panel of the vehicle M. The driver monitor camera 70 outputs an image of the interior of the vehicle, including the driver of the vehicle M, captured from its installed position to the driving assistance device 100.

[0031] The driving operators 80 include, for example, a steering wheel, an accelerator pedal, a brake pedal, a shift lever, and other operators. The driving operators 80 are equipped with sensors that detect the amount of operation or the presence or absence of operation, and the detection results are output to the driving assistance device 100 or some or all of the driving force output device 200, the brake device 210, and the steering device 220. The steering wheel is an example of an "operator that accepts steering operation by the driver." The operator does not necessarily have to be annular and may be in the form of an irregular steering wheel, a joystick, a button, or the like. A steering grip sensor 84 is attached to the steering wheel 82. The steering grip sensor 84 is realized by a capacitance sensor or the like, and outputs a signal to the driving assistance device 100 that can detect whether the driver is gripping the steering wheel 82 (meaning that the driver is in contact with the steering wheel in a state where force can be applied). The steering wheel may also be provided with a mechanism that, under the control of the driving assistance device 100, applies a reaction force to the occupant (driver) to steer in a predetermined direction (or not to steer in a predetermined direction).

[0032] The driving assistance device 100 includes, for example, a recognition unit 110, a determination unit 120, a driving control unit 130, an HMI control unit 140, and a storage unit 150. The recognition unit 110, the determination unit 120, the driving control unit 130, and the HMI control unit 140 are each realized by a hardware processor such as a CPU (Central Processing Unit) executing a program (software). Some or all of these components may be realized by hardware (including circuitry) such as an LSI (Large Scale Integration), an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or a GPU (Graphics Processing Unit), or may be realized by a combination of software and hardware. The program may be stored in advance in a storage device (a storage device having a non-transitory storage medium) such as an HDD or flash memory of the driving assistance device 100, or may be stored in a removable storage medium such as a DVD or CD-ROM, and installed in the HDD or flash memory of the driving assistance device 100 by inserting the storage medium (non-transitory storage medium) into a drive device. The HMI control unit 140 is an example of a "display control unit."

[0033] The storage unit 150 may be realized by the various storage devices described above, or a solid state drive (SSD), an electrically erasable programmable read only memory (EEPROM), a read only memory (ROM), or a random access memory (RAM). The storage unit 150 stores, for example, programs and various other information. The storage unit 150 may also store, for example, map information 54.

[0034] The recognition unit 110 recognizes the position, speed, acceleration, and other status of objects present around the vehicle M (within a predetermined distance from the vehicle M) based on information input from the external sensor ES. The object may be, for example, another vehicle, a bicycle, a pedestrian, or other traffic participant. The position of the object is recognized as a position on an absolute coordinate system with a representative point of the vehicle M (such as the center of gravity or the center of the drive shaft) as the origin, and is used for control. The position of the object may be represented by a representative point such as the center of gravity or a corner of the object, or by an area. The "state" of the object may include the acceleration or jerk of the object, or its "behavior state" (for example, whether the object is changing lanes or about to change lanes). The recognition unit 110 may also recognize the type of object (another vehicle, bicycle, pedestrian), etc., based on characteristic information such as the size, shape, and color of the object.

[0035] Furthermore, the recognition unit 110 recognizes, for example, the lane in which the host vehicle M is traveling (driving lane). For example, the recognition unit 110 recognizes the dividing lines on the left and right sides of the host vehicle M from the camera image captured by the camera 10, and recognizes the driving lane based on the positions of the recognized dividing lines. Note that the recognition unit 110 may recognize the driving lane by recognizing landmarks (road boundaries, road boundaries) that can identify the lane position, including not only dividing lines but also shoulders, curbs, medians, guardrails, fences, walls, etc. In this recognition, the position of the host vehicle M obtained from the navigation device 50 and the processing results by the INS may be taken into consideration. The recognition unit 110 may also recognize stop lines, obstacles, red lights, toll booths, and other road phenomena.

[0036] When recognizing the driving lane, the recognition unit 110 recognizes the position and attitude of the host vehicle M with respect to the driving lane. For example, the recognition unit 110 may recognize the deviation of the reference point of the host vehicle M from the center of the lane and the angle it forms with a line connecting the centers of the lanes in the traveling direction of the host vehicle M as the relative position and attitude of the host vehicle M with respect to the driving lane. Alternatively, the recognition unit 110 may recognize the position of the reference point of the host vehicle M with respect to one of the side edges of the driving lane (a dividing line or a road boundary) as the relative position of the host vehicle M with respect to the driving lane. Note that the recognition of the driving lane by the recognition unit 110 and the recognition of the position and attitude of the host vehicle M with respect to the driving lane may be performed by the identification unit 153, which will be described later.

[0037] Furthermore, the recognition unit 110 may, for example, implement a function based on AI (Artificial Intelligence) and a function based on a pre-given model in parallel. For example, the function of "recognizing an intersection" may be implemented by executing in parallel recognition of an intersection based on deep learning or the like and recognition based on pre-given conditions (such as the presence of traffic lights and road signs that can be pattern-matched), and by scoring both and comprehensively evaluating them.

[0038] Furthermore, the recognition unit 110 analyzes the image captured by the driver monitor camera 70, and recognizes the line of sight of the occupants (particularly the driver) of the vehicle M based on the analysis results.

[0039] The determination unit 120 determines the driving state (driving mode) of the host vehicle M that the driving control unit 130 should execute based on the recognition result by the recognition unit 110. The driving modes of the host vehicle M include, for example, a road-following driving mode (an example of a first mode) in which the host vehicle M travels at least along the extension direction of the road (a travelable direction), an intersection mode (an example of a second mode) in which the host vehicle M travels through an intersection, a merging mode (an example of a third mode) in which the host vehicle M travels through a merging section, a narrow road passing mode (an example of a fourth mode) in which the host vehicle M travels through a road whose width is less than a predetermined value, and an emergency avoidance mode (an example of a fifth mode) in which the host vehicle M avoids contact between the host vehicle M and an object. The road-following driving mode is an example of a "normal mode." The merging mode may also include a mode (lane change mode) in which the host vehicle M changes lanes from the driving lane to an adjacent lane. The determination unit 120 determines the driving mode of the host vehicle M based on the positional relationship between the host vehicle M and surrounding objects recognized by the recognition unit 110 and road information around the host vehicle M obtained from the map information 54. The determination unit 120 may also determine a display mode associated with the driving mode of the host vehicle M. The determination unit may also determine whether the host vehicle M will come into contact with a surrounding object or whether the degree of proximity is equal to or greater than a threshold based on the relative distance, relative speed, movement direction, etc. between the host vehicle M and the surrounding object. The function of the determination unit 120 will be described in detail later.

[0040] The driving control unit 130 controls the driving operation of the host vehicle M by the occupant of the host vehicle M or the traveling of the host vehicle M. For example, the driving control unit 130 executes driving assistance corresponding to the traveling state of the host vehicle M determined by the determination unit 120. The driving assistance may include, for example, control to apply a predetermined torque reaction force to the steering wheel of the driving operator 80 to cause the occupant to perform a steering operation in a predetermined direction, control to avoid contact between the host vehicle M and an object, control to change the host vehicle M from the traveling lane to an adjacent lane, etc. Furthermore, the driving control unit 130 may cause the HMI control unit 140 to output information corresponding to the traveling state of the host vehicle M from the HMI 30 so that the occupant (particularly the driver) of the host vehicle M can drive the host vehicle M appropriately according to the traveling state of the host vehicle M.

[0041] The HMI control unit 140 notifies the occupant of predetermined information via the HMI 30 and acquires the details of the occupant's operations performed via the HMI 30. The predetermined information includes, for example, information related to the driving of the vehicle M, such as information about the state of the vehicle M and information about driving control. The information about the state of the vehicle M includes, for example, the speed of the vehicle M, engine speed, and shift position. The information about driving control includes, for example, an inquiry about whether to change lanes, whether or not to execute each of the driving modes (first mode to fifth mode), information about changing the driving mode, and information imposed on the occupant for switching the driving mode (task request information for the occupant). The predetermined information may also include information unrelated to the driving control of the vehicle M, such as television programs, content (for example, movies) stored on a storage medium such as a DVD, etc. In addition, the specified information may include, for example, information regarding the current position and destination of the vehicle M, information regarding the remaining fuel, information indicating whether the lane in which the vehicle M is traveling has been identified, the remaining distance until the traveling mode is switched, the direction in which the lanes are increased or decreased, the number of lanes to be increased or decreased, the number of lanes running parallel to the traveling lane (number of parallel lanes), etc.

[0042] For example, the HMI control unit 140 may generate an image including the predetermined information described above and display the generated image on the display device 32 of the HMI 30, or may generate sound indicating the predetermined information and output the generated sound from a speaker of the HMI 30. For example, the HMI control unit 140 causes the display device 32 to display an image in a display mode corresponding to one of a plurality of modes based on the surrounding situation recognized by the recognition unit 110. For example, the HMI control unit 140 selects and generates an image to be displayed based on the display mode transitioned based on the surrounding situation recognized by the recognition unit from at least an image simulating the host vehicle M, an image indicating the positions of objects around the host vehicle M, an image indicating the direction in which the host vehicle M should travel, a target position to which the host vehicle M will change lanes, an image indicating the direction in which the occupant should gaze, an image prompting acceleration or deceleration, and an image prompting the occupant to perform a steering operation, and causes the display device 32 to display the generated image. The direction in which the vehicle M should travel and the direction in which the occupants of the vehicle M should look may be obtained, for example, based on the position of an object recognized by a predetermined recognition model relative to the vehicle M, or by inputting the current surrounding conditions of the vehicle M into a learned model that has been learned from data such as the driving details and line of sight of an experienced driver according to the surrounding conditions included in past driving history, etc. Furthermore, the HMI control unit 140 may output the information received by the HMI 30 to the communication device 20, the navigation device 50, the driving control unit 130, etc.

[0043] The driving force output device 200 outputs a driving force (torque) for the vehicle to travel to the driving wheels. The driving force output device 200 includes, for example, a combination of an internal combustion engine, an electric motor, a transmission, etc., and an ECU (Electronic Control Unit) that controls these. The ECU controls the above components according to information input from the driving assistance device 100 or information input from the driving operator 80.

[0044] Brake device 210 includes, for example, a brake caliper, a cylinder that transmits hydraulic pressure to the brake caliper, an electric motor that generates hydraulic pressure in the cylinder, and a brake ECU. The brake ECU controls the electric motor according to information input from driving assistance device 100 or information input from driving operator 80, so that brake torque corresponding to the braking operation is output to each wheel. Brake device 210 may include a backup mechanism that transmits hydraulic pressure generated by operation of a brake pedal included in driving operator 80 to the cylinder via a master cylinder. Note that brake device 210 is not limited to the configuration described above, and may also be an electronically controlled hydraulic brake device that controls an actuator according to information input from driving assistance device 100 to transmit hydraulic pressure from a master cylinder to the cylinder.

[0045] The steering device 220 includes, for example, a steering ECU and an electric motor. The electric motor applies a force to, for example, a rack and pinion mechanism to change the direction of the steered wheels. The steering ECU drives the electric motor to change the direction of the steered wheels in accordance with information input from the driving assistance device 100 or information input from the driving operator 80. Furthermore, the steering device 220 may apply a torque reaction force under the control of the driving assistance device 100 so as to rotate the steering wheel in a predetermined direction (or not to rotate it in a predetermined direction) in response to the driving operation of the occupant.

[0046] [Judgment section] Next, the details of the function of the determination unit 120 will be described. FIG. 2 is a diagram for explaining the function of the determination unit 120. In the example of FIG. 2, five modes (normal mode (first mode), intersection mode (second mode), merging mode (third mode), narrow road passing mode (fourth mode), and emergency avoidance mode (fifth mode)) indicating the driving state of the host vehicle M described above are shown. For example, the determination unit 120 determines to transition to a mode corresponding to the condition when a predetermined transition condition is satisfied based on the normal mode (road following driving mode), and determines to transition to the normal mode when a predetermined return condition is satisfied in the mode to which the transition has been made. When it is determined that the driving state has transitioned to each mode, the driving control unit 130 causes the HMI control unit 140 to generate an image associated with a display mode based on the determination result, and causes the generated image to be displayed on the display device 32. The transition conditions and return conditions to each mode shown in FIG. 2 will be described below. In the example of Figure 2, transition conditions from normal mode to other modes and return conditions from other modes to normal mode are set, but transition conditions and return conditions between each mode may be set without going through normal mode.

[0047] <Normal mode (road following mode)> The normal mode is a driving mode when the shape of the road (lane) on which the host vehicle M is traveling is not a predetermined road shape such as an intersection, a merging lane, or a narrow road, and it is determined that there is little possibility of the host vehicle M coming into contact with another vehicle. The normal mode may also be a driving mode when a transition condition to another mode is not satisfied, or when a return condition for another mode is satisfied. The determination unit 120 may, for example, acquire the shape of the road on which the host vehicle M is traveling by referring to map information (map information 54) based on the position information of the host vehicle M, or may acquire the shape based on the detection results of the external sensor ES.

[0048] When executing the normal mode, the HMI control unit 140 displays an image that resembles the vehicle M, and further displays an image indicating the direction in which the occupant should look when there is an object whose degree of proximity to the vehicle M (an index value indicating the likelihood of contact) is above a threshold (the likelihood of contact is low but the relative distance is less than a predetermined distance), and displays an image indicating the relative position of the object with respect to the vehicle M when there is an object whose degree of proximity to the vehicle M is below a threshold (the likelihood of contact is low but the relative distance is greater than a predetermined distance).

[0049] For example, as shown on road RD1 in Fig. 2, when another vehicle m1 is present ahead of the host vehicle M, the driving control unit 130 may cause the display device 32 to display an image showing the position of the other vehicle m1 relative to the position of the host vehicle M, or may generate a route that avoids contact with the other vehicle m1 and display it on the display device 32. Furthermore, as shown on road RD2 in Fig. 2, when an object (traffic participant) OB1 such as a bicycle is present that is overtaking the host vehicle m1, the driving control unit 130 may cause the display device 32 to display an image showing the position and direction of the other vehicle M or the object OB1 relative to the position of the host vehicle M, or may cause the display device 32 to display an image showing a route for overtaking the host vehicle m1 or the object OB1 without contacting them. The driving control unit 130 may control one or both of the steering and speed of the host vehicle M to perform driving control so that the host vehicle M does not come into contact with the object.

[0050] <Intersection mode> A condition (transition condition A) for transitioning from the normal mode to the intersection mode is, for example, that map information (map information 54) is referenced based on the position of the host vehicle M, and an intersection is found to exist in the traveling direction of the host vehicle M within a predetermined distance from the host vehicle M. In addition to the above conditions, transition condition A may also be a case where the turn signal switch of the HMI 30 is activated by an occupant's operation, causing either the left or right turn signal to light up (blink). Instead of (or in addition to) referring to map information, the determination unit 120 may determine whether the host vehicle M is traveling through an intersection based on the detection result of the external sensor ES. When the determination unit 120 determines to transition to the intersection mode, the driving control unit 130 transitions the driving mode of the host vehicle M to the intersection mode.

[0051] When transitioning to intersection mode, the HMI control unit 140 generates an image that resembles the vehicle M, and if there is an object in the intersection that may be brought into contact with the vehicle M, it generates an image that indicates the relative position of the object with respect to the vehicle M, and if the object is located in the direction of travel of the vehicle M, it generates an image that indicates the direction in which the occupant should look, and displays the generated image on the display device 32.

[0052] For example, the HMI control unit 140 causes the display device 32 to display an image indicating the position of another object (traffic participant) OB1 such as a bicycle passing through the intersection when the host vehicle M turns right, as shown on road RD3 in Figure 2, or causes the display device 32 to display an image indicating the position of another object (traffic participant) OB2 such as a bicycle passing near the intersection when the host vehicle M turns left, as shown on road RD4.

[0053] Furthermore, a condition for returning from intersection mode to normal mode (return condition A) is, for example, that the position of the host vehicle M is a predetermined distance or more away from the intersection. Furthermore, return condition A may include a case where the host vehicle M is not traveling in a merging section or a narrow road, and there is not a high possibility of contact with another object. The driving control unit 130 may control one or both of the steering and speed of the host vehicle M to perform driving control so that the host vehicle M does not contact an object.

[0054] <Merge mode> The condition for transitioning from the normal mode to the merging mode (transition condition B) may be, for example, when the host vehicle M is traveling along a road including multiple lanes that allow travel in the same direction, or when the host vehicle M recognizes that the currently traveling lane will merge into another lane, and when the occupant operates the turn signal switch of the HMI 30 to turn on (blink) either the left or right turn signal. In addition to the above conditions, transition condition B may also include a case where the road on which the host vehicle M is traveling is not near an intersection. Furthermore, the determination unit 120 may refer to map information based on the position of the host vehicle M and further subdivide the determination into the merging mode if a merging section exists within a predetermined distance from the host vehicle M in the traveling direction of the host vehicle M, and the lane change mode if a merging section does not exist within the predetermined distance. Furthermore, the determination unit 120 may determine whether the host vehicle M is traveling through a merging section based on the detection results of the external sensor ES, instead of (or in addition to) referring to map information.

[0055] When transitioning to the merging mode, the HMI control unit 140 displays an image that resembles the host vehicle M, and further displays other vehicles traveling in the lane to which the lane is to be changed in an image of a first predetermined color, and displays an image indicating the target position of the lane to which the lane is to be changed in an image of a second predetermined color. Furthermore, the HMI control unit 140 displays an image of the first predetermined color and an image of the second predetermined color side by side on the side of the image that resembles the host vehicle M where the host vehicle M is to change lanes. Furthermore, the HMI control unit 140 may display an image that accelerates or decelerates the host vehicle M based on the positional relationship between the host vehicle M and the target position.

[0056] For example, the HMI control unit 140 displays on the display device 32 an image to assist driving when changing lanes from a merging lane to a main lane or from a driving lane to an adjacent lane so as to avoid contact with other vehicles m1 and m2 around the vehicle M, as shown on roads RD5 and RD6 in Figure 2.

[0057] The driving control unit 130 may control one or both of the steering and the speed of the vehicle M to perform driving control so that the vehicle M does not come into contact with an object. The driving control unit 130 may also perform control to guide the occupant to perform a steering operation to cause the vehicle M to change lanes. In this case, the driving control unit 130 performs an operation to guide the occupant to perform a steering operation to cause the occupant to change lanes by using a reaction force against the steering wheel operated by the occupant.

[0058] Furthermore, a condition for returning from the merging mode to the normal mode (returning condition B) is, for example, when the turn signals have stopped lighting up. Another condition may be that the position of the vehicle M is a predetermined distance or more away from the merging section. Furthermore, the returning condition B may include a case where the vehicle M is not traveling near an intersection or on a narrow road, and there is no possibility of contact with other objects.

[0059] <Narrow Passage Mode> A condition for transitioning from the normal mode to the narrow road passing mode (transition condition C) is, for example, when the width of the road on which the host vehicle M is traveling is less than a predetermined value. Furthermore, transition condition C is, for example, when the host vehicle M travels while avoiding an object ahead of the host vehicle M or when the host vehicle M passes an object traveling on the opposite side, as shown on roads RD7 and RD8 in FIG. 2, when the distance to the object is less than a predetermined distance and the time to collision (TTC) between the host vehicle M and the object is equal to or greater than a threshold. Further, transition condition C may include a case where the minimum width of an area recognized as a drivable area for the host vehicle M is less than a predetermined value (for example, when the horizontal distance from the left edge of the road to the leftmost edge of a vehicle waiting to turn right is less than a predetermined value). The time to collision (TTC) is, for example, a value calculated by dividing the relative distance between the host vehicle M and the object by the relative speed. The threshold may be, for example, a fixed value or a variable value set depending on the speed of the host vehicle M, the speed of the object, road conditions, etc. Furthermore, transition condition C may include a case where the speed of the host vehicle M is less than a predetermined speed.

[0060] For example, the determination unit 120 determines to transition to the narrow road passing mode when another vehicle m1 is recognized ahead of the host vehicle M and is stopped waiting to turn right or left, as shown on road RD8 in Fig. 2, and there is no adjacent lane to the left of the driving lane of the host vehicle M (or the other vehicle m1). Note that the determination unit 120 also determines to transition to the narrow road passing mode when the other vehicle m1 is stopped waiting to turn left and there is no adjacent lane to the right of the driving lane of the host vehicle M, or when there is an adjacent lane but it is under construction or there is no space for the host vehicle M to enter the adjacent lane due to a line of vehicles or the like.

[0061] When the mode is switched to the narrow passage mode, the HMI control unit 140 displays an image simulating the host vehicle M, and further displays the direction in which the host vehicle M should travel if an object is present around the host vehicle M, and also displays an image prompting the occupant to perform a steering operation. For example, in the scene shown on road RD7 or road RD8, the HMI control unit 140 displays an image on the display device 32 to assist driving through the narrow passage so as to avoid contact with other vehicles m1 and m2 present around the host vehicle M. The driving control unit 130 may execute steering control and speed control to avoid contact with the other vehicles m1 and m2. The driving control unit 130 may also perform control to guide the occupant to perform a steering operation to avoid contact between the host vehicle M and the object. In this case, the driving control unit 130 performs an operation to guide the occupant to perform a steering operation by using a reaction force against the steering wheel operated by the occupant.

[0062] Furthermore, a condition for returning from the narrow road passing mode to the normal mode (return condition C) is, for example, when the width of the road on which the host vehicle M is traveling is equal to or greater than a predetermined value. The return condition C may also include when the host vehicle M and the other vehicle m1 are separated by a predetermined distance or greater after passing each other, or when the speed of the host vehicle M is equal to or greater than a predetermined speed. Furthermore, the return condition C may also include when the minimum width of the area recognized as the drivable area of ​​the host vehicle M is equal to or greater than a predetermined value (for example, when the narrow road passing mode is activated in an attempt to pass beside a vehicle waiting to turn right, the vehicle waiting to turn right completes a right turn, increasing the drivable area and causing the minimum width to be equal to or greater than a predetermined value).

[0063] <Emergency Avoidance Mode> A condition for transitioning from the normal mode to the emergency avoidance mode (transition condition D) is, for example, when the contact margin time TTC between the host vehicle M and an object present in the vicinity is less than a threshold value. Note that the transition condition D may include the execution of driving assistance such as a forward contact warning (FCW) when the contact margin time TTC between the host vehicle M and an object present in the vicinity is less than a first threshold value, or the execution of driving assistance such as contact mitigation braking (CMBS) when the contact margin time TTC is less than a second threshold value that is smaller than the first threshold value.

[0064] When transitioning to emergency avoidance mode, the HMI control unit 140 displays an image that resembles the vehicle M, and further displays an image on the pseudo image of the vehicle M indicating the direction in which the vehicle should proceed, and displays the area around the image that resembles the vehicle M other than the direction in which the vehicle M should proceed in a background color that is different from the background color displayed in the first to fourth modes.

[0065] 2, the HMI control unit 140 causes the display device 32 to display an image for assisting driving to avoid contact between the host vehicle M and objects OB1 and OB2, such as bicycles, approaching in front of the host vehicle M. The driving control unit 130 may control one or both of the steering and speed of the host vehicle M to perform driving control so that the host vehicle M does not come into contact with the object.

[0066] Furthermore, a condition for returning from the emergency avoidance mode to the normal mode (return condition D) is, for example, when the time to contact TTC between the vehicle M and an object is equal to or greater than a threshold value. Furthermore, the return condition D may include a case where the vehicle M is not traveling through an intersection, a merging section, a narrow road, or the like.

[0067] [Images displayed in each mode] Next, an example of an image displayed in each of the above modes will be described. <Normal mode: First display mode> Fig. 3 is a diagram for explaining a first display mode in normal mode. Fig. 3 shows an image IM10A generated by the HMI control unit 140 and displayed on the display device 32 when the normal mode is executed, and the traveling state of the host vehicle M in which the content shown in image IM10A is displayed. Note that the display content and layout included in image IM10A are not limited to this. The same applies to the explanation of the following drawings.

[0068] 3, the host vehicle M is traveling along a road RD11 including lanes L1 and L2 and sidewalks SW1 and SW2 in the direction of the road. An area VA1 indicates the line of sight of the occupant (driver) of the host vehicle M recognized by the recognition unit 110.

[0069] When the normal mode is being executed, the HMI control unit 140 generates an image IM10A including a first image IM11 simulating the host vehicle M and second images (azimuth images) IM12-1 to IM12-16 obtained by dividing the area around the host vehicle M (360 degrees) into 16 parts at predetermined intervals (for example, equal intervals). The first image IM11 in the normal mode is an image simulating the host vehicle M as seen from behind. Each of the second images IM12-1 to 12-16 has an area AR1 closest to the host vehicle M, an area AR3 farthest from the host vehicle M, and an area AR2 between areas AR1 and AR3. The second images IM12-1 to 12-16 are displayed in a predetermined background color (first background color).

[0070] 3, when the vehicle is traveling, the HMI control unit 140 displays the second images IM12-1, IM12-2, and IM12-16, which correspond to the directions in which the vehicle M can travel without coming into contact with an object based on the surrounding conditions recognized by the recognition unit 110, in a color (first highlight color, for example, blue) different from that of the other second images IM12-3 to IM12-15. This makes it easier for the occupant (driver) to recognize the directions in which the vehicle M is unlikely to come into contact with an object.

[0071] <Normal mode: Second display mode> FIG. 4 is a diagram illustrating a second display mode in the normal mode. The example of FIG. 4 differs from the driving state shown in FIG. 3 in that an object (e.g., a pedestrian) OB1 is present on the sidewalk SW1. The object OB1 is recognized as a pedestrian by the recognition unit 110, and the pedestrian is unlikely to enter the lane L1 from the sidewalk SW1. The relative distance between the host vehicle M and the object OB1 is equal to or greater than a predetermined distance. Therefore, the object OB1 is determined to be an object whose proximity to the host vehicle M is less than a threshold. In this case, based on the relative position of the object OB1 as seen from the host vehicle M and the direction in which the object OB1 exists, the HMI control unit 140 generates an image IM10B that displays, among the second images IM12-1 to IM12-16, a second image IM12-14 in the direction in which the object OB1 exists, and that displays, among the regions AR1 to AR3 of the second image IM12-14, a region AR2 corresponding to the distance from the host vehicle M, in a color (a second highlight color, for example, green) that can be distinguished from other second images and other regions. In the example of Figure 4, since there is a low possibility that pedestrian OB1 will enter lane L1 from sidewalk SW1 or come into contact with vehicle M, an image showing the position of object OB1 is displayed in a second highlight color that is less emphasized than the first highlight color, corresponding to the relative position of object OB1 with respect to vehicle M.

[0072] <Normal mode: Third display mode> Fig. 5 is a diagram for explaining a third display mode in the normal mode. The example of Fig. 5 shows a display mode in a scene where another vehicle m1 is present ahead of the host vehicle M on the lane L1 on which the host vehicle M is traveling, and where the distance between the host vehicle M and the other vehicle m1 is equal to or greater than a predetermined distance, as compared to the traveling state shown in Fig. 3. Note that the example of Fig. 5 shows a scene in which the recognition result of the recognition unit 110 recognizes another vehicle present in the direction of the second image IM12-16.

[0073] In this case, the HMI control unit 140 generates an image IM10C in which, among the second images IM12-1 to IM12-16, the second images IM12-1 and IM12-2 indicating the direction in which the host vehicle M is unlikely to come into contact with an object are displayed in a first accent color, and, when the occupant's line of sight is directed toward the other vehicle m1, two areas AR2 and AR3 of the areas AR1 to AR3 of the second image IM12-16 corresponding to the direction of the other vehicle m1 are displayed in a color (a third accent color: for example, orange) different from the colors of the second images IM12-1 to IM12-15. Note that the other vehicle m1 is located on the lane L1 in which the host vehicle M is traveling, and therefore there is a possibility of contact with the host vehicle M. Therefore, by displaying the two areas in the third accent color, it is easier to draw the occupant's attention compared to displaying one area. In addition, in the third display mode, if the occupant's line of sight is not directed toward the other vehicle m1, the HMI control unit 140 displays the second image IM12 corresponding to the direction of the other vehicle m1 in a fourth highlight color (e.g., red), as shown in the sixth display mode of the normal mode described below.

[0074] <Normal mode: 4th display mode> FIG. 6 is a diagram illustrating a fourth display mode in the normal mode. Compared with the driving state shown in FIG. 5, the example of FIG. 6 illustrates a situation in which an instruction is displayed to instruct the host vehicle M to tilt to the right with respect to the direction of extension of the road RD11 in order to overtake the other vehicle m1. In the example of FIG. 6, the line of sight direction VA1 of the occupant of the host vehicle M is assumed to be directed in a direction that does not include the other vehicle m1. In addition, in the situation shown in FIG. 6, it is assumed that the degree of proximity between the host vehicle and the other vehicle m1 is equal to or greater than a threshold. In this case, the HMI control unit 140 displays, among the second images IM12-1 to IM12-16, the second image IM12-2 corresponding to the course direction in which the host vehicle M should head, in a first highlight color. The course direction in which the host vehicle M should head includes a direction in which the host vehicle M can travel with a possibility of contact with an object (contact risk) less than a threshold. Furthermore, the HMI control unit 140 generates an image IM10D that displays two areas AR2 and AR3 of the areas AR1 to AR3 of the second image IM12-16 and IM12-1, which correspond to the direction of the other vehicle m1 as seen from the host vehicle M, in a third accent color, as an image indicating the direction in which the occupant should gaze. By increasing the area displayed in the third accent color compared to the third display mode, the occupant can be more clearly notified that the other vehicle m1 is approaching the host vehicle M and the direction in which the occupant should gaze. Note that the scene shown in FIG. 6 is a screen after the occupant has once recognized the other vehicle m1, so the above-mentioned "direction in which the occupant should gaze" may be interpreted as "the direction in which the occupant should pay attention."

[0075] <Normal mode: 5th display mode> Fig. 7 is a diagram for explaining a fifth display mode in the normal mode. In comparison with the traveling state shown in Fig. 6, the example of Fig. 7 shows a scene in which the host vehicle M has overtaken another vehicle m1 and is instructed to move in a direction to the center of the lane L1. Note that in the example of Fig. 7, the line of sight direction VA1 of the occupant of the host vehicle M is directed in a direction that does not include the other vehicle m1.

[0076] In this case, the HMI control unit 140 generates an image IM10E in which, among the second images IM12-1 to IM12-16, the second image IM12-16 corresponding to the direction in which the host vehicle M should travel is displayed in a first accent color, and among the regions AR1 to AR3 of the second image IM12-11 corresponding to the direction in which the other vehicle m1 is located as seen from the host vehicle M, one region AR2 set according to the distance between the host vehicle M and the other vehicle m1 is displayed in a third accent color. In the fifth display mode, the host vehicle M has already overtaken the other vehicle m1, so the possibility of contact between the host vehicle M and the other vehicle m1 is low (the degree of proximity is less than the threshold). Therefore, the HMI control unit 140 displays, among the regions AR1 to AR3 of the second image IM12-11, only one region AR2 corresponding to the position of the other vehicle m1 in the third accent color. This makes it easier for the occupant to recognize that the other vehicle m1 is near the host vehicle M, even though the possibility of contact with the other vehicle m1 is low.

[0077] <Normal mode: 6th display mode> Fig. 8 is a diagram for explaining a sixth display mode in normal mode. The example of Fig. 8 shows a scene in which other objects OB1 and OB2 are present near the traveling direction (line of sight direction VA1) of the host vehicle M. The other objects OB1 and OB2 are, for example, traffic participants such as pedestrians and bicycles. Furthermore, the other objects OB1 and OB2 are moving on the lane L1 on which the host vehicle M is traveling and are present within a predetermined distance from the host vehicle M. Therefore, the driving control unit 130 determines that the degree of proximity between the other objects OB1 and OB2 and the host vehicle M is equal to or greater than a threshold.

[0078] In this case, the HMI control unit 140 generates an image IM10F that displays, as an image indicating the direction in which the occupant should look among the second images IM12-1 to IM12-16, the areas AR2 and AR3 displayed in the two areas of the host vehicle M and the other objects OB1 and OB2 among the areas AR1 to AR3 in each of the second images IM12-1, IM12-2, and IM12-16, in a color different from that used in the other second images (a fourth highlight color: for example, red). The fourth highlight color is a color that is more highlighted than the first to third highlight colors. This makes it possible to notify the occupant of the direction in which the objects OB1 and OB2, which have a high possibility of coming into contact with the host vehicle M, are located and which the occupant should look at.

[0079] 8, in addition to displaying the image 10F described above, the driving control unit 130 may also execute speed control such as slowing down or stopping the host vehicle M. In this case, the driving control unit 130 may cancel the deceleration or stop control when the distance between the host vehicle M and the objects OB1 and OB2 is greater than or equal to a predetermined distance, or when the objects OB1 and OB2 move to positions other than the lane L1.

[0080] <Intersection mode: First display mode> Fig. 9 is a diagram for explaining a first display mode in the intersection mode. The example of Fig. 9 shows a scene in which the host vehicle M is traveling on a road RD12 near an intersection where lanes L1, L2 and lanes L3, L4 intersect, and the host vehicle M is turning right from lane L1 to lane L3. In the example of Fig. 9, it is assumed that an object OB1 present near the intersection is not included in the line of sight direction VA of the occupant, and that the determination unit 120 has determined that there is a possibility of contact between the host vehicle M and the object OB1 based on their relative distance, relative speed, and moving direction.

[0081] When traveling near an intersection in intersection mode, the HMI control unit 140 generates an image IM10G in which the second image IM12 corresponding to the line of sight VA1 is not highlighted, and the second image IM12-4 corresponding to the position where the object OB1 is located relative to the host vehicle M is displayed in a fourth highlight color, in order to make it particularly easy to recognize the object OB1 present in the vicinity of the host vehicle M. In the example of FIG. 9 , only the area AR2 of the second image IM12-4 is displayed in the fourth highlight color based on the relative distance between the host vehicle M and the object OB1. However, the areas AR1 and AR3 may also be displayed in the fourth highlight color depending on the relative distance. This notifies the occupant that an object is present near the direction of travel, thereby guiding the occupant's gaze in the direction in which the occupant should focus, making it easier for the occupant to notice the object OB1.

[0082] <Intersection mode: Second display mode> Fig. 10 is a diagram illustrating a second display mode in the intersection mode. The example of Fig. 10 differs from the first display mode shown in Fig. 9 in that an object OB1 is included in the line of sight of the occupant. In this case, the HMI control unit 140 generates an image IM10H that displays a second image IM12-3 corresponding to the position where the object is located in a third highlight color that is less emphasized than the fourth highlight color.

[0083] In the example of Figure 10, based on the relative distance between the vehicle M and the object OB1, only the area AR2 of the second image IM12-3 corresponding to the position of the object OB1 as seen from the vehicle M is displayed in the third highlight color.In addition, the HMI control unit 140 may display the areas AR1 and AR3 of the second image IM12-3 in the third highlight color when the relative distance is less than a predetermined distance.

[0084] <Merge mode: First display mode> Fig. 11 is a diagram for explaining a first display mode in the merging mode. The example of Fig. 11 shows a scene in which a host vehicle M changes lanes from lane L1 to lane L2 on a road RD13 that includes lanes L1 and L2 and is accessible in the traveling direction. In the example of Fig. 11, other vehicles m1 and m2 are traveling on lane L2, into which the host vehicle M will change lanes.

[0085] In this case, the driving control unit 130 derives the relative speed and relative position between the host vehicle M and each of the other vehicles m1 and m2, and sets a target position (target position) to which the host vehicle M will change lanes based on the derived relative speed and relative position. The driving control unit 130 also causes the HMI control unit 140 to generate an image for driving the host vehicle M to move to the target position without coming into contact with the other vehicles m1 and m2.

[0086] In the first display mode of the merging mode, the HMI control unit 140 generates an image IM20A including a third image IM21 simulating the host vehicle M, fourth images (azimuth images) IM22-1 to IM22-12 obtained by dividing the area around the host vehicle M (360 degrees) into 12 regions at predetermined intervals (for example, equal intervals), fifth images IM23-1 and IM23-2 indicating the position of the other vehicle M, and a sixth image IM24 indicating the target position TA1. The third image IM12 is an image simulating the host vehicle M as seen from above. Each of the fourth images IM22-1 to IM22-12 has an area AR1 closest to the host vehicle M, an area AR3 farthest from the host vehicle M, and an area AR2 between areas AR1 and AR3.

[0087] The fifth image IM23 and the sixth image IM24 may be displayed in a manner that allows each image to be distinguished. The fifth image IM23 is an example of an "image of a first predetermined color," and the sixth image IM24 is an example of an "image of a second predetermined color." The fifth image IM23 and the sixth image IM24 may be displayed side by side on the lane-changing side of the host vehicle M relative to the third image that resembles the host vehicle M, and at least a portion of the fifth image IM23 and the sixth image IM24 may be displayed superimposed on the fourth image IM22. In the merging mode, the fourth image IM22 includes information indicating acceleration or deceleration of the host vehicle M.

[0088] In the merging mode, fifth images IM23-1 and IM23-2 indicating the position of other vehicles M and sixth image IM24 indicating target position TA1 are displayed based on the position of third image IM21 simulating the vehicle M, making it easier for the occupants to recognize the current target position TA1 relative to the vehicle M.

[0089] In the example of FIG. 11, the position of the host vehicle M is located behind the target position TA1 by a predetermined distance or more. Therefore, the HMI control unit 140 generates an image that prompts the occupant to accelerate the host vehicle M. Specifically, among the fourth images IM22-1 to IM22-12, the fourth image IM22-1 showing the area ahead of the host vehicle M is displayed in a color different from the other fourth images IM22-2 to IM22-12 to prompt the occupant to accelerate the host vehicle M. Note that in the example of FIG. 11, since the area ahead is located by a predetermined distance or more, all of the areas AR1 to AR3 of the fourth image IM22-1 are displayed in different colors to prompt the occupant to accelerate the vehicle M more rapidly. In the example of FIG. 11, the area AR1 is displayed in the darkest color, and the areas AR2 and AR3 are displayed in gradually lighter colors, but the display manner is not limited to this.

[0090] <Merge mode: Second display mode> FIG. 12 is a diagram illustrating a second display mode in the merging mode. In the example of FIG. 12, the position of the host vehicle M and the target position are closer than in the example of FIG. 11. In this case, the target position TA1 is still located ahead of the host vehicle M. Therefore, the HMI control unit 140 generates an image IM20B that displays only an area AR1 included in the fourth image IM22-1 showing the area ahead of the host vehicle M in a color different from the other fourth images IM22-2 to IM22-12. This makes it easier to understand that only a slight acceleration is required compared to the first display mode of FIG. 11, and that the host vehicle M is approaching the target position TA1.

[0091] <Merge mode: third display mode> FIG. 13 is a diagram illustrating a third display mode in the merging mode. In the example of FIG. 13, compared to the example of FIG. 12, a target position exists beside the host vehicle M. In this case, further acceleration of the host vehicle M is not necessary, so the HMI control unit 140 ends displaying the fourth image IM22-1 differently from the other images IM22-2 to IM22-12. This allows the occupant to easily understand that the host vehicle M is positioned beside the target position TA. Therefore, the occupant can position the host vehicle M at the target position by steering the host vehicle M and changing lanes from lane L1 to lane L2. Furthermore, when the third display mode shown in FIG. 13 is reached (i.e., when the host vehicle M is positioned beside the target position TA1), the HMI control unit 140 may cause the HMI 30 to output information (e.g., an image or a sound) instructing the occupant to change lanes from lane L1 to lane L2. In addition, the driving control unit 130 may apply a torque reaction force that causes the steering wheel operated by the occupant of the host vehicle M to rotate to the right (not to rotate to the left).

[0092] <Merge mode: 4th display mode> FIG. 14 is a diagram illustrating a fourth display mode in the merging mode. Compared with the example of FIG. 11, the example of FIG. 14 illustrates a case where a target position TA1 relative to the host vehicle M is located a predetermined distance or more behind the host vehicle M. In this case, the HMI control unit 140 generates an image IM20D, which prompts the occupant to slow down the host vehicle M, by displaying, among the fourth images IM22-1 to IM22-12, the regions AR1 to AR3 of the fourth image IM22-7 positioned behind the third image IM21 representing the host vehicle M in a color different from that of the other fourth images. Furthermore, the HMI control unit 140 associates the fifth images IM23-1 and IM23-2 and the sixth image IM24 with the relative positions of the other vehicles m1 and m2 and the target position TA1 relative to the host vehicle M, and displays them side by side at positions on the lane-changing side of the third image IM21 representing the host vehicle M and at positions corresponding to the relative distance from the host vehicle M.

[0093] This allows the occupant to easily understand from the image IM20D that the host vehicle M needs to decelerate in order to change lanes. Note that the HMI control unit 140 may reduce the number of areas among the areas AR1 to AR3 that are highlighted in a predetermined color as the relative distance between the host vehicle M and the target position TA1 becomes shorter. This makes it easier for the occupant to understand how much the host vehicle M needs to decelerate from its current speed (deceleration rate).

[0094] <Narrow Passage Mode: First Display Mode> Fig. 15 is a diagram for explaining a first display mode in the narrow road passing mode. The example of Fig. 15 shows a host vehicle M traveling on a road RD14 whose road width W1 is less than a predetermined width and which is determined to be a narrow road, and another vehicle m1 traveling on the road RD14 opposite the host vehicle M. In addition, the example of Fig. 15 assumes that there are walls (examples of obstacles) WL1 and WL2 on the outside of both ends of the road RD14 that the vehicle cannot pass through.

[0095] In the first display mode of the narrow passage mode, the HMI control unit 140 generates an image IM30A including a seventh image IM31 simulating the host vehicle M, eighth images (azimuth images) IM32-1 to IM32-12 obtained by dividing the surroundings (360 degrees) of the host vehicle M into 12 parts at a predetermined standard (for example, equal intervals), and an image IM40A including images IM41 and IM42 simulating a steering wheel. The image IM40A is an example of an image that prompts the occupant to perform a steering operation. The seventh image IM31, like the third image IM31, is an image of the host vehicle M viewed from above, but is larger than the third image IM21. In the narrow passage mode, the vehicle travels at a low speed, so the situation farther from the host vehicle does not need to be given importance, and the vehicle travels through a narrower area, so there is a high possibility of contact with an object. Therefore, by displaying the image (7th image IM31) of the vehicle M in the display mode for passing through narrow roads larger than the image (3rd image IM21) of the vehicle M in the merging mode, the position of nearby objects relative to the vehicle M and the direction in which to proceed can be grasped more accurately, and the occupants can be alerted that more precise operation is required in close range compared to other modes.

[0096] Each of the eighth images IM32-1 to IM32-12 has the three areas AR1 to AR3 described above. Furthermore, image IM40A includes a first steering image IM41 displayed in accordance with the current steering amount and steering direction of the host vehicle M, and a second steering image IM42 displayed in accordance with the steering amount and steering direction in accordance with the direction in which the host vehicle M should move. Note that the HMI control unit 140 may display images IM30A and IM40A within a predetermined range, or may display images IM30A and IM40A on different display devices if the host vehicle M has multiple display devices 32.

[0097] 15, it is determined that the host vehicle M and the other vehicle m1 are approaching each other and may come into contact with each other. Therefore, the HMI control unit 140 displays the entire areas AR1 to AR3 of the eighth image IM32-12, which indicates the direction in which the host vehicle M should travel, in a color different from that of the other eighth images IM32-1 to IM32-12, to encourage the occupant to move the host vehicle M to the left.

[0098] Furthermore, the HMI control unit 140 displays the second steering image IM42 rotated further to the left than the first steering image IM41 in the image IM40A, prompting the occupant to rotate the steering wheel of the host vehicle M to the left. This allows the occupant to easily understand that the steering wheel should be rotated to the left and the amount of rotation (steering amount). Note that in the narrow passage mode, the driving control unit 130 may perform control to guide the occupant to perform a steering operation to avoid contact between the host vehicle M and another vehicle m1. In this case, the driving control unit 130 performs an operation to guide the occupant to steer the host vehicle M so that the host vehicle M moves in the desired direction by using a reaction force against the steering wheel operated by the occupant.

[0099] <Narrow Passage Mode: Second Display Mode> FIG. 16 is a diagram illustrating a second display mode in the narrow passage mode. The example of FIG. 16 differs from the scene shown in FIG. 15 in that the vehicle M is moving to the left due to the occupant's steering operation. In this case, the actual steering amount and steering direction of the steering wheel of the vehicle M match the steering amount and steering direction of the steering wheel associated with the direction in which the vehicle M should travel. Therefore, the image IM40B generated by the HMI control unit 140 displays the first steering image IM41 superimposed on the second steering image IM42, and therefore displays only the first steering image IM41. Furthermore, the image IM30B generated by the HMI control unit 140 displays the entire regions AR1 to AR3 of the eighth image IM32-1 corresponding to the occupant's line of sight (the front of the vehicle M) in a color different from that of the other eighth images IM32-2 to IM32-12.

[0100] This makes it easier for the occupants to understand that the host vehicle M is traveling in a direction that will avoid contact with the other vehicle m1.

[0101] <Narrow Passage Mode: Third Display Mode> 17 is a diagram illustrating a third display mode in the narrow road passing mode. The screen shown in FIG. 17 illustrates a scene in which the host vehicle M has moved to the left and is able to pass another vehicle m1, but is approaching a wall WL1 located outside the edge of the road RD14 (the distance between the host vehicle M and the wall WL1 is less than a predetermined distance). In this case, the HMI control unit 140 generates an image IM30C in which, as an image indicating the direction in which the occupant should pay attention, the HMI control unit 140 displays, in a different color, an area AR1 that is closer to the host vehicle M among the areas of the eighth image IM32-12 corresponding to the direction in which the wall is located as seen from the host vehicle M, and displays, in a different color from the other areas, all areas AR1 to AR3 of the eighth image IM32-2 that indicate the direction in which the host vehicle M should travel (specifically, the direction in which the host vehicle M should avoid contact with the wall WL1). In addition, the HMI control unit 140 generates an image IM40C that includes a first steering image IM41 based on the overall steering wheel position and a second steering image 42 for rotating the steering wheel in the direction in which the vehicle M should move.

[0102] In this way, by displaying images IM30C and IM40C on the display device 32, it is possible to notify the driver that the vehicle M is approaching the wall WL1 and to prompt the driver to steer the vehicle M.

[0103] <Emergency Avoidance Mode: First Display Mode> Fig. 18 is a diagram for explaining a first display mode in the emergency avoidance mode. Fig. 18 shows a host vehicle M traveling in lane L1 of road RD15 which includes lanes L1 and L2 and sidewalks SW1 and SW2, and an object (bicycle) OB1 entering lane L1 from the sidewalk SW1. In the example of Fig. 18, it is assumed that a transition condition D to the emergency avoidance mode is met.

[0104] In the first display mode of the emergency avoidance mode, the HMI control unit 140 generates an image IM50A that includes a ninth image IM51 that simulates the host vehicle M and tenth images (azimuth images) IM52-1 to IM52-16 that divide the surroundings (360 degrees) of the host vehicle M into 16 parts at predetermined intervals (for example, equal intervals). The ninth image IM51 in the normal mode is an image that simulates the host vehicle M as seen from behind, similar to the first image IM11. Furthermore, each of the second images IM12-1 to IM12-16 has three areas AR1 to AR3 as described above.

[0105] In the emergency avoidance mode, the HMI control unit 140 highlights the entire area AR1 to AR3 of the ninth image IM52-4, which indicates the direction in which contact between the host vehicle M and the object OB1 should be avoided, in a color different from that of the other ninth images IM52-1 to IM52-3 and IM52-5 to IM52-16. The HMI control unit 140 also displays the ninth images IM52-1 to IM52-3 and IM52-5 to IM52-16 in a second background color different from the first background color. The second background color is a more emphasized color than the first background color. As shown in FIG. 18, displaying the entire periphery of the host vehicle M in the second background color makes it easier for the occupant to recognize the presence of an object with a high probability of contact.

[0106] In the emergency avoidance mode, the HMI control unit 140 does not display the image IM40 relating to the steering amount and steering direction of the steering wheel, as in the narrow passage mode. This prevents the occupants from being distracted from driving due to the display of multiple images.

[0107] [Processing flow] Fig. 19 is a flowchart showing an example of the flow of processing executed by the driving assistance device 100 of the embodiment. In the processing of Fig. 19, of the various processing executed by the driving assistance device 100, the image display processing according to the mode will be mainly described. Note that the processing of Fig. 7 may be repeatedly executed at a predetermined cycle or at a predetermined timing. Also, the initial mode of the processing of Fig. 19 is assumed to be the normal mode (road-following mode).

[0108] In the example of FIG. 10, the recognition unit 110 recognizes the surrounding conditions of the host vehicle M (step S100). Next, the determination unit 120 performs a process of determining the mode of the driving state of the host vehicle M based on the recognition result by the recognition unit 110 (step S102). For example, the determination unit 120 determines whether any of the transition conditions (e.g., transition conditions A to D) associated with a plurality of preset modes is satisfied (step S104). If any of the transition conditions is satisfied, the driving control unit transitions to the mode associated with the transition condition (step S106). Next, the HMI control unit 140 generates an image associated with the transitioned mode (step S108). The generated image is displayed on the display device 32 to notify the occupant (step S110).

[0109] If it is determined in the process of step S104 that any of the transition conditions is not satisfied, the HMI control unit 140 generates an image associated with the normal mode (step S112) and notifies the occupant by displaying the generated image on the display device 32 (step S114). This ends the process of this flowchart.

[0110] 19, when a transition condition is satisfied from the normal mode, the mode transitions to a mode corresponding to the transition condition, and then when a predetermined return condition is satisfied, the mode returns to the normal mode. In this case, the HMI control unit 140 generates an image in a display mode corresponding to the normal mode and displays it on the display device 32 to notify the occupant.

[0111] <Modification> When displaying each of the images described above, the HMI control unit 140 may highlight the image by adjusting gradation, patterns, blinking, brightness, brightness, or the like, instead of highlighting the image by color.

[0112] According to the embodiment described above, the driving assistance device 100 includes a recognition unit 110 that recognizes the surrounding situation of the vehicle, an HMI control unit (an example of a display control unit) 140 that displays images on a display device to assist a driver driving the vehicle in a plurality of preset display modes, and a driving control unit 130 that controls the driving operation of the vehicle by the driver or the traveling of the vehicle, and the plurality of display modes include at least a first mode in which the vehicle travels along an extension direction of a road, a second mode in which the vehicle travels through an intersection, a third mode in which the vehicle travels through a merging section, and a fourth mode in which the vehicle travels through a narrow road. The display control unit transitions to one of the plurality of display modes based on the surrounding conditions recognized by the recognition unit, and when displaying an image in the display mode corresponding to the fourth mode, the display control unit displays an image that resembles the vehicle, and when an object is present around the vehicle, the display control unit displays an image that indicates the direction in which the vehicle should travel and prompts the occupant to steer, thereby enabling more appropriate driving assistance to be provided according to the surrounding conditions.

[0113] The above-described embodiment can be expressed as follows. a storage device storing a program; a hardware processor; The hardware processor executes the program, Recognizes the surrounding situation of the vehicle, displaying images on a display device to assist a driver driving the vehicle in a plurality of preset display modes; Execute driving control to control the driving operation of the vehicle by the occupant or the running of the vehicle; the plurality of display modes include display modes corresponding to at least a first mode in which the vehicle runs along an extension direction of a road, a second mode in which the vehicle runs through an intersection, a third mode in which the vehicle runs through a merging section, a fourth mode in which the vehicle runs through a narrow road, and a fifth mode in which the vehicle runs while avoiding contact with an object, transitioning the display mode to one of the plurality of display modes based on the recognized surrounding situation; When an image is displayed in a display mode corresponding to the fourth mode, an image simulating the vehicle is displayed; Furthermore, when an object is present around the vehicle, the direction in which the vehicle should proceed is displayed, and an image is displayed to prompt the occupant to steer the vehicle. The driving assistance device is configured as follows.

[0114] The above describes the form for carrying out the present invention using an embodiment, but the present invention is not limited to such an embodiment, and various modifications and substitutions can be made within the scope that does not deviate from the gist of the present invention. [Explanation of symbols]

[0115] 1...vehicle system, 10...camera, 12...radar device, 14...LIDAR, 16...object recognition device, 20...communication device, 30...HMI, 32...display device, 40...vehicle sensor, 50...navigation device, 70...driver monitor camera, 80...driving operator, 100...driving assistance device, 110...recognition unit, 120...determination unit, 130...driving control unit, 140...HMI control unit, 150...memory unit, 200...driving drive force output device, 210...brake device, 220...steering device

Claims

1. a recognition unit that recognizes the surrounding situation of the vehicle; a display control unit that displays, on a display device, images for assisting a driver who drives the vehicle in a plurality of preset display modes; a driving control unit that controls the driving operation of the vehicle by the occupant or the traveling of the vehicle, the plurality of display modes include display modes corresponding to at least a first mode in which the vehicle runs along an extension direction of a road, a second mode in which the vehicle runs through an intersection, a third mode in which the vehicle runs through a merging section, a fourth mode in which the vehicle runs through a narrow road, and a fifth mode in which the vehicle runs while avoiding contact with an object, the display control unit transitions the display mode to one of the plurality of display modes based on the surrounding situation recognized by the recognition unit; the display control unit displays an image simulating the vehicle when displaying an image in a display mode corresponding to the fourth mode, Furthermore, when an object is present around the vehicle, the direction in which the vehicle should proceed is displayed, and an image is displayed that prompts the occupant to steer the vehicle. the display control unit displays an image in a display mode corresponding to the fourth mode when the vehicle is traveling while avoiding an object present in front of the vehicle or when the vehicle is passing an object traveling on the opposite side, and the distance to the object is less than a predetermined distance and the contact margin time between the vehicle and the object is equal to or greater than a threshold. Driving assistance device.

2. a recognition unit that recognizes the surrounding situation of the vehicle; a display control unit that displays, on a display device, images for assisting a driver who drives the vehicle in a plurality of preset display modes; a driving control unit that controls the driving operation of the vehicle by the occupant or the traveling of the vehicle, the plurality of display modes include display modes corresponding to at least a first mode in which the vehicle runs along an extension direction of a road, a second mode in which the vehicle runs through an intersection, a third mode in which the vehicle runs through a merging section, a fourth mode in which the vehicle runs through a narrow road, and a fifth mode in which the vehicle runs while avoiding contact with an object, the display control unit transitions the display mode to one of the plurality of display modes based on the surrounding situation recognized by the recognition unit; the display control unit displays an image simulating the vehicle when displaying an image in a display mode corresponding to the fourth mode, Furthermore, when an object is present around the vehicle, the direction in which the vehicle should proceed is displayed, and an image is displayed that prompts the occupant to steer the vehicle. the display control unit, when the recognition unit recognizes another vehicle that is stopped in front of the vehicle and waiting to turn right or left, displays an image in a display mode corresponding to the fourth mode. Driving assistance device.

3. a recognition unit that recognizes the surrounding situation of the vehicle; a display control unit that displays, on a display device, images for assisting a driver who drives the vehicle in a plurality of preset display modes; a driving control unit that controls the driving operation of the vehicle by the occupant or the traveling of the vehicle, the plurality of display modes include display modes corresponding to at least a first mode in which the vehicle runs along an extension direction of a road, a second mode in which the vehicle runs through an intersection, a third mode in which the vehicle runs through a merging section, a fourth mode in which the vehicle runs through a narrow road, and a fifth mode in which the vehicle runs while avoiding contact with an object, the display control unit transitions the display mode to one of the plurality of display modes based on the surrounding situation recognized by the recognition unit; the display control unit displays an image simulating the vehicle when displaying an image in a display mode corresponding to the fourth mode, Furthermore, when an object is present around the vehicle, the direction in which the vehicle should proceed is displayed, and an image is displayed that prompts the occupant to steer the vehicle. when displaying an image in a display mode corresponding to the fourth mode, the display control unit displays a first image simulating the vehicle and a second image obtained by dividing a periphery of the first image according to a predetermined criterion, each of the divided second images is displayed in a plurality of regions according to a distance from the vehicle; the display control unit, when the vehicle and another vehicle traveling opposite to the vehicle pass each other, displays, based on the surrounding situation, an area of ​​the divided area of ​​the second image that indicates a direction in which the vehicle should travel, in a color different from that of other divided areas. Driving assistance device.

4. The display control unit selects and displays an image to be displayed based on a display mode transitioned based on the surrounding situation recognized by the recognition unit, from among at least an image simulating the vehicle, an image showing the position of an object around the vehicle, an image showing the direction in which the vehicle should travel, a target position to which the vehicle will change lanes, an image encouraging acceleration or deceleration, and an image encouraging the occupant to perform a steering operation. The driving assistance device according to claim 1 or 2.

5. the display control unit causes the image simulating the vehicle in the third mode and the fourth mode to be displayed as an image of the vehicle seen from above, and causes the image simulating the vehicle in the first mode, the second mode, and the fifth mode to be displayed as an image of the vehicle seen from behind. The driving assistance device according to claim 4.

6. When displaying an image in a display mode corresponding to the fourth mode, the display control unit displays the image simulating the vehicle larger than the image simulating the vehicle displayed in the third mode. The driving assistance device according to claim 1 or 2.

7. The driving control unit performs control to guide the occupant to perform a steering operation to avoid contact between the vehicle and the object. The driving assistance device according to claim 1 .

8. The driving control unit performs an operation to guide the steering by the occupant by using a reaction force against a steering wheel operated by the occupant. The driving assistance device according to claim 7.

9. The computer Recognizes the vehicle's surroundings, displaying images on a display device to assist a driver driving the vehicle in a plurality of preset display modes; Execute driving control to control the driving operation of the vehicle by the occupant or the running of the vehicle; the plurality of display modes include display modes corresponding to at least a first mode in which the vehicle runs along an extension direction of a road, a second mode in which the vehicle runs through an intersection, a third mode in which the vehicle runs through a merging section, a fourth mode in which the vehicle runs through a narrow road, and a fifth mode in which the vehicle runs while avoiding contact with an object, transitioning the display mode to one of the plurality of display modes based on the recognized surrounding situation; When an image is displayed in a display mode corresponding to the fourth mode, an image simulating the vehicle is displayed; Furthermore, when an object is present around the vehicle, the direction in which the vehicle should proceed is displayed, and an image is displayed that prompts the occupant to steer the vehicle. When the vehicle is traveling while avoiding an object present in front of the vehicle or when the vehicle is passing an object traveling on the opposite side, and the distance to the object is less than a predetermined distance and the contact margin time between the vehicle and the object is equal to or greater than a threshold, an image is displayed in a display mode corresponding to the fourth mode. Driving assistance methods.

10. The computer Recognizes the vehicle's surroundings, displaying images on a display device to assist a driver driving the vehicle in a plurality of preset display modes; Execute driving control to control the driving operation of the vehicle by the occupant or the running of the vehicle; the plurality of display modes include display modes corresponding to at least a first mode in which the vehicle runs along an extension direction of a road, a second mode in which the vehicle runs through an intersection, a third mode in which the vehicle runs through a merging section, a fourth mode in which the vehicle runs through a narrow road, and a fifth mode in which the vehicle runs while avoiding contact with an object, transitioning the display mode to one of the plurality of display modes based on the recognized surrounding situation; When an image is displayed in a display mode corresponding to the fourth mode, an image simulating the vehicle is displayed; Furthermore, when an object is present around the vehicle, the direction in which the vehicle should proceed is displayed, and an image is displayed that prompts the occupant to steer the vehicle. When another vehicle that is stopped in front of the vehicle and waiting to turn right or left is recognized, an image is displayed in a display mode corresponding to the fourth mode. Driving assistance methods.

11. The computer Recognizes the vehicle's surroundings, displaying images on a display device to assist a driver driving the vehicle in a plurality of preset display modes; Execute driving control to control the driving operation of the vehicle by the occupant or the running of the vehicle; the plurality of display modes include display modes corresponding to at least a first mode in which the vehicle runs along an extension direction of a road, a second mode in which the vehicle runs through an intersection, a third mode in which the vehicle runs through a merging section, a fourth mode in which the vehicle runs through a narrow road, and a fifth mode in which the vehicle runs while avoiding contact with an object, transitioning the display mode to one of the plurality of display modes based on the recognized surrounding situation; When an image is displayed in a display mode corresponding to the fourth mode, an image simulating the vehicle is displayed; Furthermore, when an object is present around the vehicle, the direction in which the vehicle should proceed is displayed, and an image is displayed that prompts the occupant to steer the vehicle. When an image is displayed in a display mode corresponding to the fourth mode, a first image simulating the vehicle and a second image obtained by dividing the periphery of the first image according to a predetermined standard are displayed, each of the divided second images is displayed in a plurality of regions according to a distance from the vehicle; When the vehicle and another vehicle traveling in the opposite direction to the vehicle pass each other, a region of the divided region of the second image that indicates a direction in which the vehicle should travel is displayed in a color different from that of other divided regions based on the surrounding situation. Driving assistance methods.

12. On the computer, Recognize the vehicle's surroundings, displaying images on a display device to assist a driver driving the vehicle in a plurality of preset display modes; Execute driving control to control the driving operation of the vehicle by the occupant or the running of the vehicle; the plurality of display modes include display modes corresponding to at least a first mode in which the vehicle runs along an extension direction of a road, a second mode in which the vehicle runs through an intersection, a third mode in which the vehicle runs through a merging section, a fourth mode in which the vehicle runs through a narrow road, and a fifth mode in which the vehicle runs while avoiding contact with an object, transitioning the display mode to one of the plurality of display modes based on the recognized surrounding situation; When an image is displayed in a display mode corresponding to the fourth mode, an image simulating the vehicle is displayed; Furthermore, when an object is present around the vehicle, the direction in which the vehicle should proceed is displayed, and an image is displayed that prompts the occupant to steer the vehicle. When the vehicle is traveling while avoiding an object present in front of the vehicle or when the vehicle is passing an object traveling on the opposite side, and the distance to the object is less than a predetermined distance and the contact margin time between the vehicle and the object is equal to or greater than a threshold, an image is displayed in a display mode corresponding to the fourth mode. program.

13. On the computer, Recognize the vehicle's surroundings, displaying images on a display device to assist a driver driving the vehicle in a plurality of preset display modes; Execute driving control to control the driving operation of the vehicle by the occupant or the running of the vehicle; the plurality of display modes include display modes corresponding to at least a first mode in which the vehicle runs along an extension direction of a road, a second mode in which the vehicle runs through an intersection, a third mode in which the vehicle runs through a merging section, a fourth mode in which the vehicle runs through a narrow road, and a fifth mode in which the vehicle runs while avoiding contact with an object, transitioning the display mode to one of the plurality of display modes based on the recognized surrounding situation; When an image is displayed in a display mode corresponding to the fourth mode, an image simulating the vehicle is displayed; Furthermore, when an object is present around the vehicle, the direction in which the vehicle should proceed is displayed, and an image is displayed that prompts the occupant to steer the vehicle. When another vehicle that is stopped in front of the vehicle and waiting to turn right or left is recognized, an image is displayed in a display mode corresponding to the fourth mode. program.

14. On the computer, Recognize the vehicle's surroundings, displaying images on a display device to assist a driver driving the vehicle in a plurality of preset display modes; Execute driving control to control the driving operation of the vehicle by the occupant or the running of the vehicle; the plurality of display modes include display modes corresponding to at least a first mode in which the vehicle runs along an extension direction of a road, a second mode in which the vehicle runs through an intersection, a third mode in which the vehicle runs through a merging section, a fourth mode in which the vehicle runs through a narrow road, and a fifth mode in which the vehicle runs while avoiding contact with an object, transitioning the display mode to one of the plurality of display modes based on the recognized surrounding situation; When an image is displayed in a display mode corresponding to the fourth mode, an image simulating the vehicle is displayed; Furthermore, when an object is present around the vehicle, the direction in which the vehicle should proceed is displayed, and an image is displayed that prompts the occupant to steer the vehicle. When an image is displayed in a display mode corresponding to the fourth mode, a first image simulating the vehicle and a second image obtained by dividing the periphery of the first image according to a predetermined standard are displayed, each of the divided second images is displayed in a plurality of regions according to a distance from the vehicle; When the vehicle and another vehicle traveling in the opposite direction to the vehicle pass each other, a region of the divided region of the second image that indicates a direction in which the vehicle should travel is displayed in a color different from that of other divided regions based on the surrounding situation. program.

Images