Display control device and display control method

Abstract

A processor provided in an operation system determines, on the basis of map data or the like, whether or not a host vehicle is traveling in a region in front of a gate, the region being within a predetermined distance of the gate. When the processor is traveling in the region in front of the gate, the processor changes the configuration of a route image according to an automation level and a road structure in the vicinity of the gate. For example, the combination of objects (in other words, display items) arranged in the route image is changed according to the automation level.

Description

Display control device and display control method CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application is based on Patent Application No. 2024-041617 filed in Japan on March 15, 2024, and the contents of the original application are incorporated by reference in their entirety.

[0002] The present disclosure relates to a technology for controlling the presentation of information to a driver when passing through a gate.

[0003] Patent Literature 1 discloses a vehicle control device that automatically drives a vehicle through a toll road gate. The vehicle control device can change the gate that the vehicle is scheduled to pass through depending on whether a card for paying the toll is inserted in the vehicle.

[0004] Patent No. 6692935

[0005] Other vehicles are more likely to cut in around the gate, making control more difficult than when driving on a straight road. For this reason, drivers tend to feel uneasy about the operation of the driving system when driving around the gate.

[0006] One of the objects of the present disclosure is to provide a display control technology that allows a driver to easily understand the planned vehicle behavior.

[0007] The display control device disclosed herein is a display control device used in a vehicle equipped with multiple operating modes with different levels of automation of driving operations, and is equipped with a control unit that controls the display of a route image, which is an image showing the vehicle's driving route, and a communication circuit that allows the control unit to communicate with other devices.The control unit performs the following operations: acquires related road information, which is information about roads around a gate through which the vehicle is scheduled to pass, from the other device via the communication circuit; acquires information indicating the automation level of the currently applied operating mode; and changes the configuration of the pre-gate image, which is a route image when traveling in the area on the entrance side of the gate, based on the automation level information or the related road information.

[0008] In addition, the display control method disclosed herein is a display control method executed by a processor installed in a vehicle having multiple operating modes with different levels of automation of driving operations, and includes obtaining related road information, which is information regarding roads near a gate through which the vehicle is scheduled to pass, from another device, obtaining information indicating a current mode, which is the currently applied operating mode, and changing the configuration of a pre-gate image, which is an image showing the vehicle's driving route near the gate, based on the information in the current mode or the related road information.

[0009] According to the above-described device / method, the configuration of the route image is changed near the gate based on the current mode or information about the road near the gate. This allows the vehicle behavior near the gate to be presented in a format that corresponds to the current mode and road structure. This makes it easier for the driver to understand the vehicle behavior near the gate.

[0010] Note that the symbols in parentheses in the claims indicate a correspondence with the specific means described in the embodiments described below as one aspect, and do not limit the technical scope of the present disclosure.

[0011] 1 is a diagram illustrating the hardware configuration of a vehicle system. FIG. 2 is a functional block diagram of a driving system. FIG. 3 is a diagram illustrating the flow of information between functional blocks. FIG. 4 is a diagram illustrating an example of a route image. FIG. 5 is a diagram illustrating another example of a route image. FIG. 6 is a diagram illustrating another example of a route image. FIG. 7 is a diagram for explaining the number of entry lanes and the number of exit lanes. FIG. 8 is a flowchart illustrating the operation of a processor related to display control of a route image. FIG. 9 is a diagram illustrating an example of a route image in a gate-oriented format. FIG. 10 is a diagram illustrating an example of a route image in a lane-oriented format. FIG. 11 is a flowchart illustrating an example of control of a route image based on an automation level. FIG. 12 is a flowchart illustrating an example of control of a route image based on an automation level and a gate recognition state. FIG. 13 is a flowchart illustrating an example of control of a notification based on a difference in the number of lanes. FIG. 14 is a flowchart illustrating viewpoint adjustment according to the congestion level. FIG. 15 is a diagram illustrating an example of virtual viewpoint control. FIG. 16 is a diagram illustrating another example of virtual viewpoint control. FIG. 17 is a flowchart illustrating a process for setting a target gate according to a toll settlement method. FIG. 18 is a flowchart illustrating a notification control process according to the type of target gate. FIG. 19 is a flowchart illustrating a notification control process according to the driver's state. FIG. 19 is a diagram illustrating an example of a display screen when the set speed is automatically changed. FIG. 19 is a flowchart illustrating a process for determining a system response to an emergency vehicle according to the vehicle's position.

[0012] <Preface> One embodiment of the present disclosure will be described below with reference to the drawings. The present disclosure is not limited to the following embodiment, and may be implemented with various modifications other than those described below without departing from the spirit of the present disclosure. The various supplements and modifications described below may be implemented in appropriate combinations as long as no technical contradictions arise. Components having the same function may be given the same reference numerals, and their description may be omitted. Furthermore, components having the same function may be given the same or similar names, and they may be distinguished by reference numerals. When only a part of a configuration is mentioned, the previous description may apply to the other parts.

[0013] FIG. 1 is a diagram illustrating an example of a schematic configuration of a vehicle system VS according to the present disclosure. The vehicle system VS is a system installed in a vehicle Hv. Hereinafter, the vehicle Hv equipped with the vehicle system VS may also be referred to as the host vehicle. In the present disclosure, the term "host vehicle lane" refers to the lane in which the host vehicle is traveling among multiple lanes on a road. The host vehicle lane can also be called an ego lane. An adjacent lane is a lane adjacent to the host vehicle lane.

[0014] In this disclosure, a preceding vehicle basically refers to a vehicle that is in front of the host vehicle, traveling in the same lane as the host vehicle, and is closest to the host vehicle. A following vehicle refers to another vehicle traveling behind the host vehicle in the host vehicle's lane. A preceding vehicle also includes another vehicle traveling in front of the host vehicle in an adjacent lane, that is, a vehicle traveling diagonally in front. Similarly, a following vehicle also includes not only a following vehicle, but also a vehicle traveling diagonally behind the host vehicle.

[0015] In this disclosure, a driver refers to a person sitting in the driver's seat, that is, a driver's seat occupant, regardless of whether or not they are actually driving. In one aspect, a driver may be understood as a person who should receive the authority and responsibility for driving operations from the vehicle system VS when autonomous driving ends. The term "driver" may be replaced with a driver's seat occupant or a vehicle user. The vehicle may be a remotely controlled vehicle that is remotely controlled by an operator located outside the vehicle. The person who takes over driving operations from the vehicle system VS may be an operator located outside the vehicle. The operator is a person who has the authority to control the vehicle remotely from outside the vehicle. The operator may also be included in the concept of a driver.

[0016] The host vehicle is a vehicle equipped with a so-called automated driving function. The automated driving function allows the vehicle to autonomously navigate along a predetermined route. As defined by the Society of Automotive Engineers (SAE International), there can be multiple levels of automation for driving operations (hereinafter referred to as automation levels). The automation levels can be divided into six stages, for example, levels 0 to 5. In explaining the automation levels, the term "system" primarily refers to the driving system 30.

[0017] Level 0 is equivalent to fully manual driving, in which the driver performs all driving tasks without system intervention. Driving tasks include steering, acceleration / deceleration, and surrounding monitoring. Driving tasks can be rephrased as dynamic driving tasks. Level 1 is a level in which the system supports either steering or acceleration / deceleration. Level 1 includes cases in which only adaptive cruise control (ACC) is performed.

[0018] Level 2 is a level at which both longitudinal and lateral vehicle motion control subtasks are performed in a limited area. Level 2 refers to a level at which the system performs both speed adjustment and steering control. Level 2 may be a level at which the system essentially controls the vehicle behavior, although the driver is required to monitor the surroundings (so-called eyes-on). Steering control at level 2 may be control equivalent to LC (Lane Centering). LC is a function that automatically controls steering so that the host vehicle stays in the center of the lane. Steering assistance at level 2 may be limited steering assistance that is performed only when the host vehicle is about to deviate from the lane. Steering control at level 2 may be LKA (Lane Keeping Assist).

[0019] Level 2 may be divided into Level 2.0 and Level 2.5. Level 2.0 may be a level at which ACC and LKA operate. Level 2.0 is a level at which the system provides partial steering assistance and the driver essentially performs steering. Level 2.5 is a level at which the driving system 30 essentially performs steering, although the driver still needs to monitor the surroundings. Level 2.5 may be a state in which ACC and LC operate. Level 2.5 may also be referred to as Level 2+, Advanced Level 2, Hands-Off Level 2, etc. In the present disclosure, vehicle control corresponding to Level 2.5 is also referred to as automated driving with a periphery monitoring obligation or semi-automated driving.

[0020] Level 3 refers to a level where the system performs all driving tasks within the operational design domain (ODD), while transferring operational authority to the driver in an emergency. ODD specifies the conditions under which automated driving can be performed. Level 4 is a level where the system performs all driving tasks except under specific circumstances such as certain inoperable roads or extreme environments. Level 5 is a level where the system performs all driving tasks in all environments. Automation levels 3 to 5 are automation levels where the driver does not need to monitor the surroundings, in other words, levels corresponding to automated driving. In this disclosure, vehicle control equivalent to level 3 or higher is also referred to as automated driving without the obligation to monitor the surroundings, or autonomous driving control.

[0021] The host vehicle may be equipped with an autonomous driving function equivalent to, for example, level 4. Of course, the host vehicle may also be equipped with an autonomous driving function up to level 3 or 2.5. The configuration of the vehicle system VS disclosed below may be modified as appropriate to conform to the laws and customs of the region in which the vehicle system VS is used, the characteristics / equipment of the vehicle, etc.

[0022] <Overall Configuration of Vehicle System VS> The vehicle system VS includes, as an example, multiple devices shown in FIG. 1 . Specifically, the vehicle system VS includes an environmental sensor 11, a vehicle state sensor 12, a locator 13, a map storage unit 14, a wireless communication device 15, an occupant state sensor 16, a body ECU 17, an exterior display device 18, and a motion actuator 19. The vehicle system VS also includes an HMI system 20 and a driving system 30. Note that ECU stands for Electronic Control Unit, which means an electronic control device. HMI stands for Human Machine Interface. The term "device" may include sensors and circuits. Some devices may be configured as subsystems. That is, a device may be a subsystem.

[0023] The driving system 30 is connected to other devices such as the environmental sensor 11 so as to be able to communicate with each other via an in-vehicle network IvN. The in-vehicle network IvN is a communication network established within the vehicle. The standard of the in-vehicle network IvN may be any standard such as Controller Area Network (hereinafter referred to as CAN: registered trademark) or Ethernet (registered trademark). Some devices may be directly connected to the driving system 30 by dedicated signal lines. The connection form between devices may be changed as appropriate.

[0024] The environmental sensor 11 is a device that senses the surrounding environment of the vehicle. The environmental sensor 11 may be a sensor (a so-called autonomous sensor) that detects objects present within a detection range. The environmental sensor may also be referred to as an object detection sensor. The environmental sensor 11 may include multiple sensors. The vehicle system VS may include, for example, a camera 111, a millimeter-wave radar 112, and a LiDAR 113 as the environmental sensor 11.

[0025] The camera 111 may be an optical camera positioned to capture images of the outside of the vehicle (e.g., the front) at a predetermined angle of view. The camera 111 may be positioned on the upper edge of the windshield, the front grille, the rooftop, etc. The camera 111 may include a camera ECU in addition to a camera main body that generates image frames. The camera main body includes at least an image sensor and a lens. The camera ECU includes a processor and a memory. The processor may be a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit). The camera ECU detects a predetermined detection target by performing recognition processing on the image frames. The camera ECU detects and identifies objects registered as detection targets using, for example, a classifier that applies deep learning. The camera ECU may also calculate the relative position coordinates of the detected object relative to the vehicle from position information (e.g., pixel coordinates) of the detected object in the image frame.

[0026] The camera 111 is configured to detect other moving objects, such as pedestrians, cyclists, and automobiles. A moving object may be referred to as a road user or a traffic participant. The camera 111 may also be configured to detect features such as road edges, road markings, and structures installed along the road. The road markings may include at least one of lane marks indicating lane boundaries, crosswalks, stop lines, guidance strips, and traffic control arrows. The structures installed along the road may include at least one of road signs, guardrails, traffic lights, utility poles, curbs, and commercial signs. The camera 111 may also be configured to detect the lighting status of lighting devices of other vehicles, such as hazard lights and turn signals (so-called blinkers), and the lighting status of traffic lights.

[0027] The camera 111 may include multiple cameras. The vehicle system VS may include, as the cameras 111, a camera that captures images in front of the vehicle Hv, a rear camera that captures images behind the vehicle Hv, a right camera that captures images on the right side, and a left camera that captures images on the left side. The function of analyzing the camera images and detecting the detection target may be provided by another device, such as the driving system 30. The camera 111 transmits data indicating the detection results to the driving system 30 via the in-vehicle network IvN.

[0028] The millimeter-wave radar 112 is a device that detects the relative position and relative speed of an object by transmitting and receiving search waves such as millimeter waves or quasi-millimeter waves in a predetermined direction. The vehicle system VS may be equipped with multiple millimeter-wave radars 112. The multiple millimeter-wave radars 112 may include a forward radar and a rearward radar. The forward radar is a millimeter-wave radar 112 that transmits search waves toward the front of the vehicle. The rearward radar is a millimeter-wave radar 112 that transmits search waves toward the rear of the vehicle. The millimeter-wave radar 112 generates data indicating the relative position and relative speed of the detected object and outputs data indicating the detection result to the driving system 30. Objects detected by the millimeter-wave radar 112 may include other moving objects, manholes (iron plates), three-dimensional structures serving as landmarks, etc.

[0029] The LiDAR 113 is a device that generates three-dimensional point cloud data indicating the positions of reflection points for each detection direction by irradiating laser light. LiDAR is an abbreviation for Light Detection and Ranging or Laser Imaging Detection and Ranging. The LiDAR 113 may be a Time of Flight (ToF) camera that generates an image indicating the distance to an object (a so-called distance image). The LiDAR 113 also outputs data indicating the detection results to the driving system 30. In the present disclosure, data indicating the detection results of the environmental sensors 11, such as the camera 111, the millimeter-wave radar 112, and the LiDAR 113, is also referred to as sensor data.

[0030] The environmental sensor 11 may include sonar, etc. The environmental sensor 11 may include a rain sensor, an illuminance sensor, a temperature sensor, etc. The rain sensor is a sensor that detects rain. The illuminance sensor is a sensor that detects the brightness outside the vehicle. The temperature sensor is a sensor that detects the air temperature outside the vehicle. The environmental sensor 11 may include an acoustic sensor. The acoustic sensor generates an electrical signal corresponding to an external sound that is a sound outside the vehicle. The acoustic sensor may be, for example, a condenser microphone. The output signal of the acoustic sensor can be used to detect emergency vehicles. The combination of sensors included in the environmental sensor 11 may be changed as appropriate.

[0031] The vehicle state sensor 12 is a sensor that outputs information regarding the state of the host vehicle. The vehicle state sensor 12 includes at least one of a vehicle speed sensor, a steering angle sensor, an acceleration sensor, a yaw rate sensor, and an accelerator pedal sensor. The vehicle speed sensor is a sensor that detects the traveling speed of the host vehicle. The steering angle sensor is a sensor that detects the steering angle. The acceleration sensor is a sensor that detects acceleration acting in the longitudinal direction of the host vehicle, lateral acceleration acting in the lateral direction, etc. The yaw rate sensor is a sensor that detects the angular velocity of the host vehicle. The accelerator pedal sensor is a sensor that detects the depression amount / depression force of the accelerator pedal. The brake pedal sensor is a sensor that detects the depression amount / depression force of the brake pedal. The vehicle state sensor 12 outputs data indicative of the detection results to the in-vehicle network IvN. The output data of the vehicle state sensor 12 may also be included in the sensor data.

[0032] The locator 13 is a device that generates and outputs position information of the vehicle (more precisely, the locator 13) using navigation signals transmitted from positioning satellites that constitute the Global Navigation Satellite System (GNSS). The locator 13 includes a GNSS receiver, an inertial sensor, and the like. The locator 13 may determine the position and traveling direction of the vehicle by combining the navigation signals received by the GNSS receiver, the measurement results of the inertial sensor, and vehicle speed information transmitted through the in-vehicle network IvN. In the present disclosure, data indicating the position of the vehicle generated and output by the locator 13 is also referred to as vehicle position data. The vehicle position may be expressed using latitude and longitude, for example. The vehicle position data may also be considered an example of sensor data. The locator 13 outputs the vehicle position data to the driving system 30. The locator 13 may have a function of reading map data around the vehicle position from the map storage unit 14 and providing the map data to the driving system 30.

[0033] The map storage unit 14 is a storage device that stores map data. The map data stored in the map storage unit 14 may be so-called HD (High Definition) map data. The map data stored in the map storage unit 14 includes data such as the three-dimensional shape of roads, the positions of road markings (e.g., lane marks), and the positions of traffic signs, with the accuracy required for autonomous driving. The map storage unit 14 may also store a navigation map that shows the connection relationships between roads. The navigation map may be used to search for a route from the current location to a destination.

[0034] The map data stored in the map memory unit 14 includes gate point data for each gate point. A gate point is a point on a toll road where a gate for collecting tolls is installed. A gate may be a facility for recording entry and exit. A gate may not have a bar. A gate may be a structure in which a camera or wireless communication device for recording entry and exit is installed. The expression "gate point" / "gate" in this disclosure can be read as "toll booth." Gate point data is data that indicates the structure of a gate point. At one gate point, multiple gates may be installed side by side in the width direction of the road.

[0035] The gate location data includes data related to the representative location coordinates, the number of gates installed, the detailed location of each gate, and the settlement method for each gate. The representative location coordinates are location coordinates that roughly indicate the location of the gate location. The representative location coordinates may be, for example, the location coordinates of the gate located in the middle, on the right end, or on the left end of multiple gates lined up horizontally (hereinafter referred to as the representative gate). The number of gates installed can be rephrased as the number of lanes. Each gate provides one lane (passageway). The detailed location data of the gates may be coordinate data such as latitude and longitude. The detailed location of the gates may be expressed as a number, with the right-most or left-most gate being number 1.

[0036] The settlement method data indicates the toll settlement (payment) method. The settlement method can be categorized into manual settlement and automatic settlement. In the manual settlement method, the driver pays the toll by handing cash or a credit card to the gate staff or by inserting it into a settlement machine installed at the gate. In the automatic settlement method, a payment is made according to the vehicle type and the travel section through wireless communication between a wireless communication device installed in the vehicle (known as an on-board device) and wireless communication equipment installed at the gate (known as a roadside device). In Japan, the manual settlement method is sometimes called "general" and the automatic settlement method is sometimes called "ETC (registered trademark)." ETC stands for Electronic Toll Collection.

[0037] In this disclosure, a road section within a predetermined distance before or after a gate point represented by the representative position coordinates is also referred to as a gate area. The start and end points of the gate area may be registered in map data. The gate area may include sections without lane marks (hereinafter referred to as laneless sections) located before and after the gate. The gate area may also include sections where the road width is expanded relative to the connecting road.

[0038] The gate area may be divided into a pre-gate area and a post-gate area. The pre-gate area refers to the area of ​​the gate area located on the entrance side of the gate (in other words, in front of the gate). The post-gate area refers to the area of ​​the gate area located on the exit side of the gate (in other words, behind the gate). If there is a branch point behind the gate, the processor 31 may consider the area up to the branch point to be the post-gate area. The gate area may be divided into three areas: a pre-gate area, a post-gate area, and an inside-gate area. The inside-gate area is the section sandwiched between the equipment that makes up the gate. Passing through a gate corresponds to a state in which the vehicle Hv is in the inside-gate area.

[0039] The map data stored in the map storage unit 14 may be updated by data received by the wireless communication device 15 from a map server or the like. The map storage unit 14 may be a storage device that temporarily stores the map data received by the wireless communication device 15 from the map server until the validity period of the data expires. The map data stored in the map storage unit 14 may be navigation map data, which is map data for navigation, as long as it includes gate point data. In other words, the map storage unit 14 does not necessarily have to store a high-precision map.

[0040] The wireless communication device 15 is a device that enables the vehicle to perform wireless communication with an external device. The external device may include at least one of another vehicle, a server, a traffic information center, a roadside device, and a mobile device (e.g., a smartphone). The wireless communication device 15 is configured to be capable of performing cellular communication. Cellular communication refers to wireless communication compliant with LTE (Long Term Evolution), 4G, 5G, or the like. The wireless communication device 15 may be configured to be capable of performing cellular V2X (PC5 / SideLink / Uu).

[0041] The wireless communication device 15 is also configured to be capable of short-range communication. In this disclosure, short-range communication refers to wireless communication in which the communication distance is limited to within several hundred meters. The short-range communication method used may be DSRC (Dedicated Short Range Communications) compatible with IEEE 802.11p, Wi-Fi (registered trademark), or Bluetooth (registered trademark) Low Energy. The short-range communication method may be the aforementioned cellular V2X, for example, communication using a PC5 interface. The wireless communication device 15 may be configured to be capable of performing data communication related to toll payment with a roadside device installed at a gate when passing through the gate. For example, the wireless communication device 15 may be an on-board device compatible with ETC 2.0.

[0042] The wireless communication device 15 may receive information about gate locations from an external device. For example, the wireless communication device 15 may receive location information about gate locations, information about passable gates, and information about closed gates from a predetermined server. The wireless communication device 15 may receive vehicle data from surrounding vehicles via vehicle-to-vehicle communication. The vehicle data may include speed, current location, turn signal operation status, acceleration, and movement trajectory. The surrounding vehicles here refer to vehicles present within a range where vehicle-to-vehicle communication is possible. The vehicle data transmitted and received via vehicle-to-vehicle communication may be referred to as a vehicle status message. The vehicle status message may be a Cooperative Awareness Message (CAM) defined in ETSI TS 102 637-2 or a Basic Safety Message (BSM) defined in SAE J 2735.

[0043] The occupant status sensor 16 is a sensor that detects the driver's status. The occupant status sensor 16 may be, for example, a driver status monitor (hereinafter referred to as DSM). The DSM is a sensor that detects the driver's facial orientation, gaze direction, eyelid openness, etc. based on a facial image of the driver. The occupant status sensor 16 transmits driver status data indicating the driver's facial orientation, gaze direction, eyelid openness, etc. to the driving system 30. The driver status data may also be considered a type of sensor data. The occupant status sensor 16 may be a pulse sensor, a thermal camera, etc. The occupant status sensor 16 may include a steering wheel sensor. The steering wheel sensor is a sensor that detects whether the driver is gripping the steering wheel. The steering wheel sensor may be a touch sensor provided on the steering wheel.

[0044] The body ECU 17 is an ECU that comprehensively controls the body-related devices mounted on the vehicle. The body-related devices include lighting devices, a horn, door lock motors, etc. The lighting devices include headlights, hazard lights, turn signals, backlights, welcome lights, etc. The body ECU 17 turns on the hazard lights and turn signals in accordance with instructions from the driving system 30.

[0045] The exterior display device 18 is a device that displays an image toward the outside of the vehicle. The exterior display device 18 may be a liquid crystal display or the like. The exterior display device 18 may display an image for communicating with drivers of other vehicles based on an input signal from the driving system 30 or the HMI system 20. For example, the exterior display device 18 displays an image indicating the direction of travel of the vehicle or an image requesting a vehicle traveling in an adjacent lane to yield the right of way (in other words, permission to cut in). The exterior display device 18 may project an image onto the rear window, side window, or road surface around the vehicle. Headlights or taillights may be configured to function as the exterior display device 18. The exterior display device 18 may be considered one of the body-related devices.

[0046] The motion actuator 19 is an actuator for the motion (driving) of the vehicle Hv. The motion actuator 19 controls the motion of the vehicle Hv based on an input control signal. The motion actuator 19 includes a power train including at least one of an engine and a drive motor. The motion actuator 19 also includes a brake actuator and a steering actuator. The steering actuator may be an EPS (Electric Power Steering) motor. Other ECUs, such as a steering ECU that performs steering control, a power unit control ECU that performs acceleration / deceleration control, and a brake ECU, may be interposed between the driving system 30 and the motion actuator 19.

[0047] The HMI system 20 is a subsystem for exchanging information between the occupant and the vehicle system VS. The HMI system 20 includes a display 21 and a speaker 22 as notification devices for notifying the driver of information. The HMI system 20 also includes an input device 23 as an input interface for receiving operations from the occupant on the vehicle system VS (in other words, the vehicle Hv).

[0048] The display 21 may include one or more of a meter display, a center display, and a head-up display (HUD). The meter display is a display arranged in an area of ​​the instrument panel located in front of the driver's seat. The center display is a display provided in the center of the instrument panel in the vehicle width direction. The HUD is a device that projects image light onto a predetermined area of ​​the windshield. The meter display and the center display may be liquid crystal displays or organic EL displays. The display 21 displays an image corresponding to a signal input from the driving system 30. The speaker 22 is a device that outputs sound corresponding to a signal input from the driving system 30. In this disclosure, the term "sound" includes notification sounds, voices, music, etc.

[0049] The vehicle system VS may include a vibrator, an ambient light, or the like as a notification device. The ambient light is a lighting device that uses multiple light emitting diodes (LEDs) and is capable of adjusting the light emission color and light emission intensity. The ambient light may be provided on the instrument panel, the steering wheel, the A-pillar, or the like. The ambient light may be a device that notifies the driver of the operating status of the driving system 30 and the risk of an external situation, etc., by changing the light emission color.

[0050] The input device 23 may include at least one of a mechanical switch, a touch panel, an operating lever, and a pedal. The mechanical switch may be a steering wheel switch, a switch arranged on the instrument panel, or a switch provided on the center console. The steering wheel switch is a switch provided on the spoke portion of the steering wheel. The steering wheel switch may include a switch for switching the automation level, such as a switch for inputting the start / end of autonomous driving. The operating lever may include an operating lever (e.g., a turn signal lever) provided on the steering column. The operating lever may include a shift lever. The touch panel may be a touch panel stacked on the center display. The pedals may include an accelerator pedal and a brake pedal. The steering wheel also corresponds to one type of input device 23.

[0051] The input device 23 outputs an operation signal, which is an electrical signal corresponding to a driver's operation, to the driving system 30. The operation signal includes information indicating the driver's operation. The vehicle system VS accepts braking, accelerator, and steering operations via the input device 23. The vehicle system VS also accepts instructions related to changing the operation mode via the input device 23. The instructions related to changing the operation mode include instructions related to starting and ending autonomous driving. The vehicle system VS may be configured to acquire various instructions from the driver through voice recognition. A device related to voice input, such as a microphone, may also be included in the input device 23. Note that, for example, an HCU (HMI Control Unit) may be interposed between the HMI system 20 and the driving system 30. The HCU is a device that comprehensively controls information output (in other words, notifications) to the driver.

[0052] The driving system 30 is a subsystem that controls the motion actuators 19 based on the detection results of the environmental sensors 11 to perform some or all of the driving operations on behalf of the driver. The driving system 30 may be an automated driving system (ADS). The driving system 30 may also be realized in the form of a device (i.e., an automatic driving device). The term "system" may be replaced with "device" as appropriate.

[0053] The driving system 30 may be realized using one or more computers. One or more computers constituting the driving system 30 correspond to a display control device. The driving system 30 includes a processor 31, a memory 32, a storage 33, a communication unit 34, and a bus connecting these. The processor 31 corresponds to a control unit. The memory 32 is a rewritable volatile storage medium. The memory 32 is, for example, a RAM (Random Access Memory). The storage 33 is, for example, a rewritable non-volatile storage medium such as a flash memory. The storage 33 stores a vehicle control program, which is a program executed by the processor 31. The vehicle control program also includes a notification control program for controlling notification to the driver regarding gate passage. Execution of the notification control program by the processor 31 corresponds to execution of a notification control method. The notification control method may include a display control method. The notification control program may include a display control program.

[0054] The communication unit 34 is an interface through which the processor 31 communicates with other devices constituting the vehicle system VS, such as the environmental sensor 11. The communication unit 34 may include a circuit compatible with a communication method with other devices. The communication unit 34 may be an input / output circuit or an input / output port. The communication unit 34 corresponds to the communication circuit. The communication unit 34 may support any type of wired or wireless communication. Part or all of the wireless communication device 15 may be included in the communication unit 34. Digital data corresponding to a signal received by the communication unit 34 may be temporarily stored in the memory 32. The communication unit 34 receives information required for performing vehicle control such as autonomous driving and driving assistance. "Receiving" may be interchangeably referred to as "acquiring." The communication unit 34 acquires sensor data (i.e., detection results) from the environmental sensor 11. The sensor data includes data on objects present around the vehicle, such as moving bodies, features, and obstacles. The data on detected objects may include the position, moving speed, and type or size of the detected object.

[0055] The sensor data related to features may include data on lane marks and road edges. The lane mark data may include not only position data but also line type data. The line type may be expressed as a continuous line (solid line) or a dashed line. The line type data may include information on the color of the line (yellow or white, etc.). The sensor data may include data indicating the lane mark recognition status, such as whether the lane marks are recognized, and the road edge recognition status, such as whether the road edge is recognized.

[0056] The communication unit 34 also acquires sensor data related to the state of the vehicle, such as the vehicle's traveling speed, acceleration, yaw rate, and external illuminance, from the vehicle state sensor 12. Furthermore, the communication unit 34 acquires vehicle position data from the locator 13. The communication unit 34 may acquire map data of the area around the vehicle by referring to the map storage unit 14.

[0057] The communication unit 34 may acquire data transmitted from an external device in cooperation with the wireless communication device 15. For example, the communication unit 34 may acquire vehicle data transmitted from a preceding vehicle via vehicle-to-vehicle communication. The communication unit 34 also acquires dynamic map data for a road section that the vehicle is scheduled to pass through within a predetermined time in cooperation with the wireless communication device 15. The dynamic map data here includes congestion information, merging vehicle information, and the like.

[0058] The communication unit 34 also acquires information indicating the driver's operation of the vehicle system VS based on a signal from the input device 23. For example, the communication unit 34 acquires instructions related to the start and end of autonomous driving from the input device 23. The communication unit 34 may acquire information indicating the operating status of devices connected to the driving system 30, such as whether the environmental sensor 11 is operating normally. The communication unit 34 may acquire driver status data indicating the eye opening degree and line of sight from the occupant status sensor 16.

[0059] Various data sequentially acquired by the communication unit 34 is stored in a temporary storage medium such as the memory 32 and is used by the environment recognition unit F1, the mode management unit F2, etc. Data may be discarded after a certain time has passed since acquisition. Various data (information) may be acquired by generation, conversion, determination, or calculation based on signals received from other devices. The communication unit 34 or the processor 31 may have a function to generate other data (also referred to as secondary data) based on raw data (also referred to as secondary data) received from other devices.

[0060] The driving system 30 has multiple operating modes with different automation levels. Each operating mode has a different range of driving tasks for which the driver is responsible, in other words, a different range of driving tasks in which the driving system 30 intervenes. The operating mode can be referred to as the driving mode. Here, as an example, the driving system 30 is configured to be switchable among multiple operating modes, including a manual driving mode, a hands-on level 2 mode, a hands-off level 2 mode, a level 3 mode, and a level 4 mode.

[0061] The manual driving mode is an operating mode in which the driver performs all driving tasks. Even in the manual driving mode, the driving system 30 may perform processes to mitigate collision damage or avoid a collision, such as advanced emergency braking (AEB) or advanced emergency steering (AES). The manual driving mode may also be a mode in which the driving system 30 stops operating (a so-called stop mode). Even in the manual driving mode, the driving system 30 may continue to perform driving environment recognition processing in the background (in other words, potentially) so that automatic driving or driving assistance can be started promptly in response to a request from the driver.

[0062] The hands-on Level 2 mode is a mode in which the driver must hold the steering wheel. The hands-on Level 2 mode is an operating mode corresponding to automation Level 2.0. The hands-off Level 2 mode is an operating mode in which the driver does not need to hold the steering wheel, in other words, an operating mode in which hands-off is permitted. The hands-off Level 2 mode is an operating mode corresponding to automation Level 2.5. In other embodiments, the driving system 30 may be implemented with only one of the hands-off Level 2 mode and the hands-on Level 2 mode as the Level 2 mode.

[0063] In this disclosure, hands-on means holding the steering wheel. Hands-off means taking your hands off the steering wheel. Eyes-on means monitoring the area outside the vehicle (mainly ahead) related to the direction of movement of the host vehicle. Eyes-off means taking your eyes off the area outside the vehicle related to the direction of movement of the host vehicle. Hands-off Level 2 mode is an operating mode that performs automated driving control with the obligation to monitor the surroundings. Hands-off Level 2 mode can be called semi-automated driving mode or eyes-on AD mode. In this disclosure, AD means automated / autonomous driving.

[0064] Level 3 mode is an operating mode that executes autonomous driving control without the obligation to monitor surroundings, i.e., vehicle control equivalent to automation level 3. Level 4 mode is an operating mode that executes autonomous driving control equivalent to automation level 4. Level 4 mode is an operating mode in which the driver is allowed to sleep. In this disclosure, the term "autonomous driving" refers to autonomous driving control at level 3 or higher, unless otherwise noted (basically). In this disclosure, the operating mode at level 3 or higher may also be referred to as autonomous driving mode or AD mode.

[0065] The driving system 30 may include multiple processors 31. The processor that executes autonomous driving control at level 3 or higher may be provided separately from the processor that executes vehicle control at level 2 or lower. The driving system 30 may include an autonomous driving ECU, which is an ECU for autonomous driving. The driving system 30 may also include an ADAS (advanced driver-assistance system)-ECU, which is an ECU for driving assistance, separate from the autonomous driving ECU.

[0066] While in the autonomous driving mode, the driving system 30 automatically steers, accelerates, decelerates (i.e., brakes), and the like of the vehicle so that the vehicle travels along a planned route toward a destination set by the driver. Even if a destination is not set, the driving system 30 may select a route to continue traveling / traveling within an area that satisfies the ODD and continue autonomous driving. The conditions for determining whether autonomous driving is possible / impossible, in other words, the detailed conditions that define the ODD, can be changed as appropriate.

[0067] The driving system 30 may perform control to drive the vehicle substantially autonomously even in hands-off level 2 mode. That is, the driving system 30 performs driving environment recognition, driving trajectory planning, and motion control even in hands-off level 2 mode. Motion control includes speed adjustment through acceleration and deceleration, steering control, and the like. The term "autonomous driving" below may be replaced with "quasi-autonomous driving equivalent to level 2.5" unless technical contradictions arise. The autonomous driving mode is terminated due to steering / pedal operation by the driver (so-called override), system limitations, exiting the ODD, and the like. The driving system 30 may have a function to determine whether the vehicle Hv is present in the ODD.

[0068] <Functions of the Driving System> The driving system 30 includes the functional units shown in Fig. 2 as functional units realized by executing an automatic driving program. That is, the driving system 30 includes an environment recognition unit F1, a mode management unit F2, a planning unit F3, a motion control unit F4, and an HMI control unit F5.

[0069] The environment recognition unit F1 recognizes the driving environment of the vehicle based on various data acquired by the communication unit 34. The environment recognition unit F1 may recognize the driving environment of the vehicle by a sensor fusion process that combines the detection results of multiple environmental sensors 11, such as the camera 111 and the millimeter-wave radar 112.

[0070] The environment recognition unit F1 acquires information related to the structure (i.e., configuration) of roads located within a predetermined distance ahead of the vehicle based on at least one of the detection results of the environment sensor 11, signals received from an external device, and map data. The road structure may include the location of gate points, the location of branching roads, the number of lanes, the number of lane marks, the road width, and the curvature of the road. The software / hardware module responsible for the process of recognizing the road structure corresponds to the road recognition unit F11. The environment recognition unit F1 manages whether the environment sensor 11 can recognize lane marks around the vehicle (i.e., the recognition status of the lane marks). The environment recognition unit F1 provides information indicating the recognition status of the lane marks in the environment sensor 11 to the HMI control unit F5 as lane recognition information M4.

[0071] The environment recognition unit F1 also acquires information about the next gate point through which the host vehicle is scheduled to pass. The information about the gate point may be acquired based on map data, or may be identified based on sensor data from the environment sensor 11. The information about the gate point includes the remaining distance to the gate point. The remaining distance to the gate point may also be acquired based on map data, or may be identified based on data of guide signs detected by the camera 111. The environment recognition unit F1 may identify the information about the gate point based on vehicle data received from a vehicle ahead.

[0072] The environment recognition unit F1 may acquire the number of gates and the payment method for each gate from map data or the driving trajectory of the preceding vehicle. The environment recognition unit F1 may consider a gate that requires stopping to pass as a gate requiring manual payment, and a gate that the preceding vehicle passes through without stopping as a gate requiring automatic payment. The functional unit in the environment recognition unit F1 that acquires information related to the gate location corresponds to the gate recognition unit F12. When the environment recognition unit F1 acquires based on the map data that the remaining distance to the gate location is less than a predetermined value, it may determine whether the environment sensor 11 has detected (in other words, recognized) the gate. The environment recognition unit F1 provides information indicating the gate recognition status by the environment sensor 11 to the HMI control unit F5 as gate recognition information M5.

[0073] The driving environment may include the vehicle lane number, weather, road surface conditions, etc. The vehicle lane number indicates the position of the vehicle lane on the road and is determined based on the left road edge. The vehicle lane number directly or indirectly indicates the number of lanes existing to the left of the vehicle lane. The vehicle lane number may also be assigned based on the right road edge. The vehicle lane number may be determined using the distance from the road edge to the vehicle, the number of lane marks detected on the left and right, and some or all of the map data. The vehicle lane number may also be determined using map data and vehicle position data. The weather and road surface conditions may be determined by combining the recognition results of the camera 111 with weather information acquired by the communication unit 34.

[0074] The driving environment includes the positions and types of objects present around the vehicle. The environment recognition unit F1 may acquire the moving speed and moving direction of detected moving objects. The environment recognition unit F1 recognizes the positions and behaviors of other vehicles based on various data acquired by the communication unit 34. A software / hardware module responsible for the process of recognizing other vehicles corresponds to the other vehicle recognition unit F13. The environment recognition unit F1 as the other vehicle recognition unit F13 may calculate a collision risk for each detected other vehicle. The collision risk may be, for example, TTC (Time-To-Collision) or MTC (Margin-To-Collision). For example, the environment recognition unit F1 calculates the TTC for each other vehicle. TTC and MTC are parameters in which the smaller the value, the greater the collision risk.

[0075] In addition to the above-described information, the environment recognition unit F1 may acquire data indicating the vehicle exterior environment related to the ODD. The environment recognition unit F1 may generate an environment model, which is a three-dimensional model that reproduces (represents) the driving environment of the vehicle Hv, as data indicating the vehicle exterior environment. The environment model may also be called a world model. The environment model may be a model in which objects detected by the environment sensor 11, such as moving objects such as other vehicles, lane markers, road edges, traffic lights, etc., are arranged in a three-dimensional space based on the vehicle itself. The environment recognition unit F1 may be understood as a configuration that manages data related to the driving environment. Here, data management may include data acquisition (generation) and updating. In addition, the environment recognition unit F1 may acquire information indicating the vehicle interior environment, such as vehicle interior temperature and driver status data.

[0076] The mode management unit F2 manages the operation mode of the driving system 30 based on information acquired by the communication unit 34. The management of the operation mode may include management of switching between manual driving and automated driving, i.e., management of the transfer of authority between the user and the driving system 30, in other words, management of the takeover of driving. The management of the operation mode corresponds to management of the automation level. The mode management unit F2 estimates the operation mode (i.e., the automation level) intended by the driver based on the operation signal input from the input device 23. The mode management unit F2 switches the operation mode based on the estimation result. For convenience in this disclosure, the currently applied operation mode is also referred to as the current mode.

[0077] When the driving environment satisfies the ODD requirement, the mode management unit F2 switches the operation mode to the automatic driving mode upon receiving an instruction signal to start automatic driving from the input device 23. Furthermore, when the mode management unit F2 predicts that the driving environment recognized by the environment recognition unit F1 will no longer satisfy the ODD requirement during the automatic driving mode, the mode management unit F2 may decide to transition to the manual driving mode and notify the planning unit F3 of this.

[0078] The mode management unit F2 may switch to manual driving mode when an override operation by the driver is detected in the autonomous driving mode or hands-off level 2 mode. An override operation refers to an occupant's operation of driving operation members such as the steering wheel, brake pedal, and accelerator pedal. An override operation may also be referred to as a takeover operation. When the driving system 30 detects that an override operation by the driver has been performed, it promptly transfers driving authority to the driver and notifies the driver by audio output or the like that driving has been switched to manual driving. The mode management unit F2 may acquire data such as the operating status (on / off) of the ACC function and whether or not a preceding vehicle is recognized, and store the data in the memory 32.

[0079] The planning unit F3 is configured to create a driving plan based on data of the driving environment (e.g., an environmental model) managed by the environment recognition unit F1. While in the AD mode or the hands-off level 2 mode, the planning unit F3 generates driving plan data for autonomous driving based on the recognition result of the driving environment by the environment recognition unit F1. The driving plan may be called a control plan or a driving plan.

[0080] The driving plan data may include route data, trajectory data, and motion plan data. The route data is data indicating a comprehensive (long-term) driving plan, such as a route to a destination. The planning unit F3 may generate the route data based on map data indicating road connections, such as map data for navigation.

[0081] Trajectory data is data indicating a relatively local (short-term) driving trajectory. The trajectory plan may include data such as the lane on which the vehicle Hv is traveling, the driving position within the lane, and lane change points. Generating trajectory data may also be called trajectory planning or path planning. The planning unit F3 may generate trajectory data based on route data and driving environment data. The motion plan data is data indicating target speed, steering angle, acceleration, etc. for each time. The motion plan data may be generated based on the trajectory data. In this way, the driving plan data may include schedule information for acceleration / deceleration for speed adjustment on the set route / trajectory and schedule information for steering amount. The driving plan created by the planning unit F3 is input to the motion control unit F4.

[0082] The planning unit F3 also performs gate passage planning processing as processing related to passing through a gate point. The gate passage planning processing includes setting a target gate, generating a travel trajectory to the target gate, and generating a trajectory after passing through the gate. The target gate is a gate through which the vehicle will pass among multiple gates provided at the gate point. A method for setting a target gate will be described separately below.

[0083] In addition to control plans directly related to vehicle driving, the planning unit F3 also generates plans for notifying the driver using notification devices such as the display 21. For example, the planning unit F3 plans the timing of issuing notifications / requests to the driver, such as behavior notification, mode change notification, eyes-on request, hands-on request, TOR (takeover request), and TOR notification. Behavior notification is a process of notifying the driver of planned vehicle behavior, such as lane change, overtaking, deceleration, etc. Mode change notification is a process of notifying the driver that the operation mode will be changed or that the operation mode is scheduled to be changed.

[0084] The eyes-on request is a process that requests the driver to monitor the surroundings as a precaution during the autonomous driving mode. The hands-on request is a process that requests the driver to lightly grip the steering wheel during the autonomous driving mode or the hands-off level 2 mode. TOR is a process in which the driving system 30 requests the driver to take over driving operations at its discretion. TOR may also be referred to as a takeover request, intervention request, or handover request. TOR includes displaying an image on the display 21 requesting the takeover of driving operations. TOR may also include outputting an audio message or warning sound from the speaker 22 requesting the takeover. TOR warning is a process that notifies the driver that the possibility of TOR is increasing.

[0085] Various notifications, including advance notices, suggestions, and requests, include displaying an icon image corresponding to the content on the display 21. A notification is outputting information to the driver, and may be referred to as an alert. Depending on the importance and urgency of the notification, various notifications may involve outputting a notification sound, outputting a voice message, flashing ambient lights, and / or vibrating a vibrator. The process of alerting the driver to the information may involve generating a data signal for driving an alert device and outputting it to the alert device. The planning unit F3 creates a notification request M1, which is a data set indicating the content of the notification and the timing of the notification, and transmits the request to the HMI control unit F5.

[0086] The motion control unit F4 generates control commands for the motion actuator 19 based on the control plan formulated by the planning unit F3. Then, the motion control unit F4 outputs the generated control commands to the motion actuator 19. The motion control unit F4 also controls the lighting state of the turn signals, headlights, hazard lights, etc. according to the driving plan and the driving environment based on the plan of the planning unit F3 and the external environment.

[0087] The planning unit F3 and the motion control unit F4 form an ACC system, which is a subsystem for executing ACC. When the ACC system recognizes a preceding vehicle, it controls the vehicle speed so as to maintain a constant inter-vehicle distance / inter-vehicle time within a set vehicle speed range. When the ACC system does not recognize a preceding vehicle or when the speed of the preceding vehicle exceeds the set vehicle speed, it adjusts the vehicle speed to maintain the set vehicle speed. The ACC system may provide data indicating the recognition status of the preceding vehicle and the implementation status of ACC to the HMI control unit F5.

[0088] The HMI control unit F5 notifies the driver using notification devices such as the display 21 and the speaker 22. Various notifications are provided by displaying an image on the display 21 and / or outputting a voice message or notification sound from the speaker 22. The notification sound may be a warning sound. The notification to the driver may involve turning on an ambient light or activating a vibrator. The HMI control unit F5 executes various notifications based on the plan of the planning unit F3. In other words, the HMI control unit F5 executes notifications to the driver based on a request from the planning unit F3. As shown in FIG. 3, the HMI control unit F5 executes the requested notification based on receiving a notification request M1 (NOTIF_REQ in the figure) from the planning unit F3. The notification request M1 may include a type code that specifies the type of notification, timing data that specifies the timing of notification execution, a combination of devices to be used for notification, etc. The type of notification may be a behavior notice, a mode change notice, an eyes-on request, a hands-on request, a TOR (takeover request), a TOR notice, or the like.

[0089] The HMI control unit F5 also performs processing to display a route image Im, which is an image showing the vehicle's travel route, on the display 21 (e.g., a meter display). The route image Im may be a bird's-eye view image of the area around the vehicle viewed from above. The viewpoint of the bird's-eye view image (a so-called virtual viewpoint) may be located a predetermined distance behind the vehicle Hv from directly above it. For example, the virtual viewpoint may be located 10 m above the road surface and 5 m behind the vehicle Hv. The HMI control unit F5 generates a CG (Computer Graphics) image as the route image Im based on at least one of map data and sensor data, and displays it on the display 21. CG may also be called CGI (Computer Generated Imagery). As described below, the route image Im is also an image showing the vehicle's travel environment. The route image Im may also be referred to as an environmental image.

[0090] As shown in FIG. 4, the route image Im when the host vehicle is traveling in the area in front of the gate may include a host vehicle image G1, a lane mark image G2, a trajectory image G3, an other vehicle image G4, and a gate image G5. In this disclosure, the phrase "the host vehicle is traveling in the area in front of the gate" may be interpreted as a state in which the remaining distance to the gate point is less than a predetermined value. The host vehicle image G1 is an image representing the host vehicle. The host vehicle image G1 may be drawn based on a 3D model representing the host vehicle that has been prepared in advance.

[0091] The lane mark image G2 is an image showing lane marks. In the example of FIG. 4 , lane mark images G2 for all lane marks within a predetermined range from the vehicle are displayed based on map data. In another display setting, the HMI control unit F5 may be configured to display lane mark images G2 only for lane marks detected by the environmental sensor 11, as shown in FIG. 5 . The HMI control unit F5 may also be configured to display detected marks, among the lane marks shown in the map data, in a different manner from undetected marks. The detected marks are lane marks detected by the environmental sensor 11. The undetected marks are lane marks not detected by the environmental sensor 11. As shown in FIG. 6 , the HMI control unit F5 may display undetected marks in a relatively inconspicuous manner, such as gray or with a dashed line, and detected marks in white or with a thick solid line. The lane mark image G2 showing detected marks is also referred to as a detected mark image G21. The lane mark image G2 for undetected marks is also referred to as an undetected mark image G22. In this way, the HMI control unit F5 may display the undetected mark and the detected mark in different display modes.

[0092] The trajectory image G3 is an image showing the planned driving trajectory of the host vehicle. The trajectory image G3 may be an arrow, for example. The other vehicle image G4 is an image showing other vehicles detected by the environmental sensor 11. The other vehicle image G4 may have different designs / shapes depending on the type of other vehicle. The design of the other vehicle image G4 may be different for trucks and passenger cars. If no other vehicles are present (not detected), the route image Im may not include the other vehicle image G4. Furthermore, among the other vehicles detected by the environmental sensor 11, a target vehicle that is set as an ACC target may be displayed in a different manner from other vehicles. For example, while other vehicles that are not target vehicles are displayed in white, light purple, or light blue, the target vehicle may be displayed in orange, yellow, dark blue, etc. By displaying the target vehicle in a relatively conspicuous manner, it is easier for the driver to understand which vehicle the host vehicle is following. Furthermore, the HMI control unit F5 may display an image of a vehicle with a high risk of excessive approach or contact in a different color from the images of other vehicles. A vehicle with a high risk of excessive approach or contact may be a vehicle that is determined by the environment recognition unit F1 to be attempting to cut in, or a vehicle whose TTC is less than a predetermined value.

[0093] The gate image G5 is an image representing a gate ahead of the vehicle. The gate image G5 may be displayed only when the remaining distance to the gate location is less than a predetermined value. In one embodiment, the HMI control unit F5 may be configured to display the gate image G5 only when the environmental sensor 11 can detect the gate. Furthermore, the HMI control unit F5 may display the gate image G5 in a grayed-out or toned-down manner when the environmental sensor 11 cannot detect the gate, and may display the gate image G5 in a bright manner when the environmental sensor 11 can detect the gate. In other words, the HMI control unit F5 may change the display manner of the gate image G5 depending on whether the gate is recognized or not.

[0094] Note that the display elements that can be included in the route image Im are not limited to those described above. The route image Im may also include images of stop lines, crosswalks, traffic lights, railroad crossings, construction sites, regulations, traffic signs, and the like as display elements. When a pedestrian or a cyclist is detected, the HMI control unit F5 may display a route image Im that includes an image of the pedestrian, etc. The HMI control unit F5 may display a target notification image. The target notification image is an image (e.g., a frame image) for highlighting the target vehicle.

[0095] The combination of items that the HMI control unit F5 displays in the route image Im under normal circumstances and the display mode for each item may be registered in storage or the like. Here, "normal circumstances" may be interpreted as a time when the vehicle is traveling at a point away from the gate, for example, a time when the vehicle is traveling on a road section other than the area in front of the gate. The display format of the route image Im under normal circumstances is also referred to as "normal format." The combination of display items included in the route image Im in normal format may be configured so that the driver can register them via a predetermined setting screen. The display items are image elements included in the route image Im, and may simply be referred to as "objects." Hereinafter, the route image Im when the vehicle is traveling in the area in front of the gate will be referred to as "gate front image ImG."

[0096] The HMI control unit F5 of this embodiment is configured to change the display mode of the gate-front image ImG based on at least one of the automation level and the road configuration. The HMI control unit F5 receives automation level information M2 (AD_LV_INFO in the figure) from the mode management unit F2 to adjust the display mode of the gate-front image ImG. The automation level information M2 is information indicating the current automation level setting value. The automation level information M2 may be information indicating the operating mode. The automation level information M2 corresponds to information indicating the current mode.

[0097] The HMI control unit F5 also receives environmental information M3 (ENV_INFO in the figure), lane recognition information M4 (LN_REC_INFO in the figure), and gate recognition information M5 (GT_REC_INFO in the figure) from the environment recognition unit F1. The environmental information M3 is data indicating the driving environment. The environmental information M3 may be an environmental model based on the detection results of the environmental sensor 11. The environmental information M3 may also be map data of the area around the vehicle.

[0098] The environmental information M3 may include vehicle position data. The environmental information M3 may include information that can identify the remaining distance to the gate point. The environmental information M3 may include data indicating whether the vehicle has entered the area in front of the gate, or data for determining this. The environmental information M3 may include gate point data that the vehicle is scheduled to pass next, i.e., data indicating the configuration of the road near the gate. The environmental information M3 may include information indicating whether the area in front of the gate that the vehicle is scheduled to pass through is a laneless section. The environmental information M3 may also include information indicating the number of entry lanes and the number of exit lanes of the gate area that the vehicle is scheduled to pass through. The number of entry lanes is the total number of lanes on the entrance side of the gate area that the vehicle is scheduled to pass through. The number of exit lanes is the total number of lanes on the exit side. For example, in the example shown in FIG. 7, the number of entry lanes is 2 and the number of exit lanes is 4. The number of entry lanes may include the number of lanes of the merging road. The number of exit lanes may include the number of lanes of the branching road. The number of entrance lanes and the number of exit lanes may be determined based on map data or data received from a roadside device. The environmental information M3 corresponds to the related road information.

[0099] The lane recognition information M4 is information about lane marks detected by the environmental sensor 11. The lane recognition information M4 may be interpreted as information indicating the recognition state of the lane marks in the environment recognition unit F1. The lane recognition information M4 may include the position, pattern (line type), color, etc. of the lane marks detected by the environmental sensor 11. If no lane marks are detected by the environmental sensor 11, the lane recognition information M4 may include a lane undetected code. The lane undetected code may be a predetermined code indicating that no lane marks have been detected.

[0100] The gate recognition information M5 is information indicating the recognition status of the gate in the environment recognition unit F1. The gate recognition information M5 may include the position of the gate detected by the environment sensor 11. If the gate is not detected by the environment sensor 11, the gate recognition information M5 may include a gate not detected code. The gate not detected code may be a predetermined code indicating that the gate has not been detected. The lane recognition information M4 and the gate recognition information M5 correspond to recognition status information.

[0101] Transmission and reception of information between functional blocks may be performed via the memory 32. For example, the mode management unit F2 may store automation level information in the memory 32, and the HMI control unit F5 may obtain the latest automation level information by referring to the memory 32. The HMI control unit F5 may also obtain environmental information M3, lane recognition information M4, and gate recognition information M5 by referring to the memory 32.

[0102] <Example (1) of Display Control of Route Image Im> Here, the process related to the display control of the route image Im performed by the processor 31 as the HMI control unit F5 will be described using the flowchart shown in FIG. 8. The flowchart shown in FIG. 8 may be executed in response to a change in the automation level (i.e., the operation mode). The flowchart shown in FIG. 8 may also be executed periodically. The flowchart shown in FIG. 8 includes, as an example, S101 to S109. The term "HMI control unit F5" may be appropriately replaced with "processor 31" or "driving system 30."

[0103] S101 is a step in which the HMI control unit F5 obtains information indicating the current mode from the mode management unit F2. The information indicating the current mode may be automation level information M2. S101 may also be a step in which the latest automation level information M2 stored in the memory 32 is obtained (i.e., read) by referring to the memory 32. S101 may also be a step in which the processor 31 identifies the current mode based on signals from at least one of the environmental sensor 11, the vehicle state sensor 12, the occupant state sensor 16, and the input device 23. When S101 is completed, the processing proceeds to S102.

[0104] S102 is a step in which the HMI control unit F5 acquires information indicating the configuration of roads near the gate through which the vehicle is to pass (i.e., related road information) from the environment recognition unit F1. The related road information may be environmental information M3. The related road information acquired in S102 may be map-based information, may include detection results from the environmental sensor 11, or may be information that combines these. S102 may be a step in which the latest environmental information M3 stored in the memory 32 is acquired (i.e., read) by referring to the memory 32. S102 may be a step in which the processor 31 acquires related road information from at least one of the environmental sensor 11, the locator 13, the wireless communication device 15, etc. After S102 is completed, S103 is executed.

[0105] S103 is a step in which the HMI control unit F5 acquires information indicating the recognition status of lane marks and gates (i.e., recognition status information) from the environment recognition unit F1. The information indicating the recognition status of lane marks may be the lane recognition information M4 described above. The information indicating the recognition status of gates may be gate recognition information M5. S103 may be a step in which the latest lane recognition information M4 and gate recognition information M5 stored in the memory 32 are acquired (i.e., read) by referring to the memory 32. S103 may be a step in which the processor 31 identifies information indicating the recognition status of lane marks and gates based on input signals from the environment sensor 11. When S103 is completed, S104 is executed.

[0106] S104 is a step in which the HMI control unit F5 acquires other information to be used in generating the route image Im. The information used in generating the route image Im includes information for determining the combination of display items for the route image Im. The information used in generating the route image Im may also include information for determining the brightness, color, display format, etc. of the route image Im. For example, the HMI control unit F5 may acquire the brightness outside the vehicle, the positions and vehicle types of other vehicles, track data, etc. When S104 is completed, the process proceeds to S105. The order of execution of S101 to S105 may be reversed as appropriate. S101 to S105 may also be combined into one information acquisition step.

[0107] S105 is a step for determining whether the host vehicle is traveling near a gate. Whether the host vehicle is traveling near a gate may be determined based on the host vehicle position data and the gate location data. The vicinity of the gate may be the area in front of the gate. If the gate is detected by the environmental sensor 11 or the like, the determination in S105 may be made based on the distance to the detected gate.

[0108] If the host vehicle is located within the area in front of the gate, the HMI control unit F5 may determine that the host vehicle is traveling near the gate (i.e., YES). If the host vehicle is located outside the area in front of the gate, the HMI control unit F5 may determine that the host vehicle is not traveling near the gate (i.e., NO). If S105 is determined as YES, the process proceeds to S106. On the other hand, if S105 is determined as NO, the process proceeds to S109.

[0109] S106 is a step for determining whether the area in front of the gate is a lane-free section. Whether the area in front of the gate is a lane-free section may be determined based on map data or the detection results of the environmental sensor 11. If the area in front of the gate is a lane-free section (YES in S106), the process proceeds to S107. On the other hand, if the area in front of the gate is not a lane-free section (NO in S106), the process proceeds to S108.

[0110] S107 is a step in which the HMI control unit F5 generates a gate-oriented route image Im as a gate-front image ImG and displays it on the display 21. The gate-oriented route image Im may be an image that does not include the lane mark image G2, but includes at least the host vehicle image G1, the track image G3, and the gate image G5. In the gate-oriented format, the gate that the host vehicle is scheduled to pass through may be displayed in a different color from the other gates as the gate image G5. The HMI control unit F5 may display the gate that the host vehicle is scheduled to pass through in a predetermined color (e.g., green) and display the other gates in other colors (e.g., white or gray). For emphasis, the color of the gate that the host vehicle is scheduled to pass through may be changed over time.

[0111] FIG. 9 is an example of a gate-focused route image Im. In the example shown in FIG. 9, an emphasis image G7 is added above the gate through which the host vehicle is scheduled to pass (hereinafter also referred to as the target gate). The emphasis image G7 may include text explaining the gate through which the host vehicle is scheduled to pass. The emphasis image G7 may be a frame image surrounding the gate through which the host vehicle is scheduled to pass, or an arrow image. The gate-focused route image Im may also include text or an icon indicating the payment method at the gate through which the host vehicle is scheduled to pass. The gate-focused route image Im may be configured to display a relatively large amount of information about the gate through which the host vehicle is scheduled to pass. Note that while FIG. 9 only shows the host vehicle, if another vehicle is detected by the environmental sensor 11, an other vehicle image G4 may also be displayed.

[0112] The route image Im in the gate-emphasized format may be an image designed to make the gate more prominent than the lane marks. Even in the gate-emphasized format, the lane mark image G2 may be displayed in an inconspicuous manner. In a configuration in which the map data includes virtual lane information for laneless sections, the route image Im generated in S107 may be an image in which the virtual lane marks are displayed in an inconspicuous manner. The inconspicuous manner may be toned down, grayed out, displayed as a dashed line, a relatively thin line, or a combination thereof. As described above, if the environmental sensor 11 is unable to detect the gate itself, the gate image G5 may be displayed in an inconspicuous manner or may be hidden. If the environmental sensor 11 is unable to detect the gate itself, the gate image G5 may be displayed in an inconspicuous manner based on the map data.

[0113] When lane marks are provided on the road surface in the area in front of the gate, the driver can easily verify the validity of the operation of the driving system 30 based on the positional relationship between the lane mark image G2 and the trajectory image G3 displayed on the display 21. On the other hand, when lane marks are not provided, the driver has difficulty verifying the validity of the trajectory set by the driving system 30. To address this issue, the HMI control unit F5 of this embodiment prominently displays the gate image G5 together with the trajectory image G3 when the area in front of the gate is a laneless section. This allows the driver to easily verify the validity of the vehicle's behavior based on the target gate, even if the area in front of the gate is a laneless section. In other words, the driver can easily verify whether the driving system 30 is behaving strangely. Confirming the validity of the operation of the driving system 30 can increase the driver's sense of security. Furthermore, according to the above configuration, when lane marks are not provided in front of the gate, a gate-focused route image Im is displayed. Displaying the gate-focused route image Im makes it easier for the driver to recognize the gate the vehicle is heading toward (i.e., the target gate).

[0114] S108 is a step in which the HMI control unit F5 generates a lane-focused route image Im as a gate-front image ImG and displays it on the display 21. The lane-focused route image Im may be an image designed to make the lane marks more prominent than the gate. The lane-focused route image Im may not include the gate image G5, but may include at least the host vehicle image G1, lane mark image G2, and track image G3. Figure 10 is an example of a lane-focused route image Im. If another vehicle is detected by the environmental sensor 11, the other vehicle image G4 may be displayed.

[0115] In the lane-focused format, the lane area through which the vehicle is scheduled to pass may be displayed in a different color or brightness than the other lane areas. The lane area here may be interpreted as the image area sandwiched between the lane mark images G2. If there are multiple lanes in the area in front of the gate, the lane area through which the vehicle is scheduled to pass may be displayed in a predetermined color (e.g., white), and the other lane areas may be displayed in other colors (e.g., gray). The color of the lane portion may be a color that does not reduce the visibility of the track image G3, or a color that makes the track image G3 stand out (e.g., a complementary color). If the surface of the actual lane is painted blue or red, the color of the lane area may be set to the same or similar color as the actual lane color. Even in the lane-focused format, the gate image G5 may be displayed in an inconspicuous manner.

[0116] According to the above configuration, when lane marks are provided in the area in front of the gate, a lane-focused route image Im is displayed. By displaying the lane-focused route image Im, the driver can easily recognize the correspondence between the vehicle's driving trajectory and the lanes, in other words, the lane the vehicle is scheduled to travel through. Since lanes are associated with gates, displaying the lane the vehicle is scheduled to travel through also makes it easier for the driver to recognize the target gate set by the driving system 30. As a result, the driver can easily verify whether the driving system 30 is behaving strangely. Being able to confirm the validity of the operation of the driving system 30 can increase the driver's sense of security.

[0117] S109 is a process performed while traveling in a section away from the gate. In S109, the HMI control unit F5 operates to generate and display a predetermined normal format route image Im. The items (i.e., image elements) to be displayed in the normal format may be set as appropriate. The normal format route image Im may be an image including a host vehicle image G1, a lane marking image G2, a track image G3, and an other vehicle image G4. The display mode (e.g., color and line type) of road markings and signs included in the route image Im may be changed based on the recognition state of the environmental sensor 11.

[0118] The above describes an example in which the display mode of multiple image elements is changed depending on whether the area near the gate is a lane-free section. Changing the display mode may include switching between display and non-display, changing the color or brightness, changing the size, switching visual effects, and the like. The HMI control unit F5 may change the position of the virtual viewpoint used to generate the route image Im depending on whether the area near the gate is a lane-free section. When the area near the gate is a lane-free section, the HMI control unit F5 may shift the position of the virtual viewpoint higher or further back by a predetermined amount compared to when the area is not a lane-free section. This allows a wider range of road to be displayed. As a result, other vehicles moving toward the target gate of the host vehicle are more easily displayed. The driver can more easily recognize the surrounding traffic conditions from a bird's-eye view. The HMI control unit F5 may change the size of the route image Im depending on whether the area near the gate is a lane-free section. When the area is a lane-free section, the display size may be larger than when the area is not a lane-free section. This may improve the visibility of the route image Im. The HMI control unit F5 may change the position of the virtual viewpoint depending on the remaining distance to the gate. The HMI control unit F5 may bring the virtual viewpoint closer to the vehicle as the remaining distance to the gate decreases. According to this display control, the target gate can be displayed relatively large.

[0119] <Display Control Example (2) of Route Image Im> When the automation level of the current mode is equal to or greater than a predetermined value, the HMI control unit F5 may be configured to include more information about the driving trajectory in the route image Im than when the automation level of the current mode is less than the predetermined value. As shown in FIG. 11 , when the current automation level is equal to or greater than the switching threshold (YES in S111), the HMI control unit F5 displays the path information in detail (S112). On the other hand, when the current automation level is less than the switching threshold (NO in S111), the HMI control unit F5 displays the path information in a basic format (S113). S111 in FIG. 11 is a step of obtaining information indicating the current automation level from the mode management unit F2 and determining whether it is equal to or greater than a predetermined threshold. "AD_LV" in the figure represents the automation level. "ThLV" in the figure represents the switching threshold.

[0120] Path information is information about a driving trajectory. A typical example of an image showing path information is a trajectory image G3. Displaying path information in detail may include, in addition to the trajectory image G3, displaying the planned speed for each point or displaying the route at the next branch point. Displaying path information in detail may include displaying the lateral driving position within the lane, such as the offset from the center of the lane. Near a gate, displaying path information in detail may be displaying the trajectory after passing through the gate. In this disclosure, an image showing the trajectory after passing through the gate is also referred to as a post-gate trajectory image. The pre-gate image ImG displayed when the automation level is below the switching threshold may be an image including a post-gate trajectory image. The post-gate trajectory image may be a line- or band-shaped image element representing the trajectory. The post-gate trajectory image may be an arrow image. Information displayed in addition to the basic format image corresponds to detailed path information.

[0121] The display in the basic format (normal display) may not display detailed information. The display in the basic format may display a track image G3 up to the gate. The route image Im displayed when the automation level is less than the switching threshold may not include a track image after the gate.

[0122] The automation level corresponding to the switching threshold may be 3. According to this configuration, in a driving mode in which the driver is not required to monitor the surroundings, detailed path information (in other words, a path plan) is displayed on the display 21. This configuration can help the driver understand the operation of the system. Furthermore, since detailed information is not displayed in a state in which the driver is required to monitor the surroundings, the risk of the driver continuing to pay attention to the display can be reduced. In another embodiment, the automation level corresponding to the switching threshold may be 2.5.

[0123] The HMI control unit F5 may be configured to output information about the driving trajectory by voice when the automation level when the host vehicle is driving in the area in front of the gate is equal to or higher than the switching threshold. On the other hand, the HMI control unit F5 may be configured not to output information about the driving trajectory by voice when the automation level when the host vehicle is driving in the area in front of the gate is lower than the switching threshold. The HMI control unit F5 may be configured to change the notification mode related to passing through the gate depending on the automation level.

[0124] <Example (3) of Display Control of Route Image Im> The HMI control unit F5 may be configured to include more information about the target gate in the route image Im when the automation level is below a predetermined value than when the automation level is equal to or greater than the predetermined value. As shown in FIG. 12 , when the automation level when the host vehicle is traveling in the area in front of the gate is less than the switching threshold (YES in S121), the HMI control unit F5 displays detailed gate information (S122). On the other hand, when the automation level when the host vehicle is traveling in the area in front of the gate is equal to or greater than the switching threshold (NO in S121), the HMI control unit F5 displays the gate information in a basic format (S123). S122 may be a step of notifying the driver of more information about the target gate than when the automation level is equal to or greater than the predetermined value.

[0125] Gate information is information about a target gate. A typical example of an image showing gate information is a gate image G5. Displaying gate information in detail may include, in addition to the gate image G5, at least one of displaying information indicating the gate's payment method, displaying the target gate number, and displaying the expected speed when passing through the gate. Displaying gate information in detail may also involve displaying, as the gate image G5, an image that more faithfully reproduces the characteristics of the actual gate. Increasing the information about the target gate may involve increasing the display size of the target gate.

[0126] Displaying gate information in detail may include displaying information about other vehicles passing through or about to enter the target gate. This configuration allows the driver to easily recognize the target gate set by the system using other vehicles as landmarks. If there are no other vehicles about to enter the target gate, information indicating that the gate is empty may be displayed. Displaying gate information in detail may involve displaying information that is not displayed in the basic format and / or changing the display image. Displaying gate information in the basic format may involve not displaying the additional information described above and / or not changing the display image. Displaying gate information in the basic format may involve displaying predetermined display items in a predetermined basic format.

[0127] As described above, the automation level corresponding to the switching threshold may be 3. In this case, in a driving mode in which the driver is required to monitor the surroundings, detailed gate information is displayed on the display 21. This makes it easier for the driver to recognize which gate to head to. Furthermore, in a state in which the driver does not need to monitor the surroundings, the display of gate information is simplified. As a result, the driver is relatively more likely to pay attention to the driving trajectory. In another embodiment, the automation level corresponding to the switching threshold may be 2.5.

[0128] The processing content of S122 may additionally or alternatively include displaying a gate image G5 in which the target gate portion of the multiple gates is emphasized. The gate image G5 in which the target gate portion is emphasized may be an image in which the target gate portion is blinking or surrounded by a frame. The processing content of S123 may include displaying a gate image G5 in which the target gate portion is not emphasized.

[0129] The HMI control unit F5 may be configured to output information about the target gate by voice when the automation level when the host vehicle is traveling in the area in front of the gate is less than the switching threshold. On the other hand, the HMI control unit F5 may be configured not to output information about the traveling trajectory by voice when the automation level when the host vehicle is traveling in the area in front of the gate is equal to or greater than the switching threshold. The HMI control unit F5 may be configured to change the combination of devices used to notify information about the target gate depending on the automation level.

[0130] <Display Control Example (4) of Route Image Im> The HMI control unit F5 may change the display mode of the gate image G5 based on the automation level and the recognition state of the gate. For example, the HMI control unit F5 may determine (change) the display mode of the gate image G5 according to the procedure shown in Fig. 13. The flowchart shown in Fig. 13 may be executed periodically while the vehicle is traveling in the area in front of the gate.

[0131] 13, like S111, the HMI control unit F5 acquires information indicating the current automation level and determines whether it is equal to or greater than the automation level switching threshold. If the automation level is equal to or greater than the switching threshold (YES in S131), S132 is executed. On the other hand, if the automation level is equal to or greater than the switching threshold (NO in S131), S135 is executed.

[0132] S132 is a step in which the HMI control unit F5 determines whether the environmental sensor 11 has detected the gate. S132 may be determined based on the gate recognition information M5. If the environmental sensor 11 has detected the gate (YES in S132), S133 is executed. If the environmental sensor 11 has not detected the gate (NO in S132), S134 is executed.

[0133] S133 is a step in which the HMI control unit F5 displays the gate image G5 in a toned-up manner. Tone-up display means displaying the gate image G5 in a bright color such as white (in other words, a color that is easy to see). In contrast, toned-down display means displaying the gate image in a relatively hard-to-see manner (a less noticeable manner), such as in a mode with reduced brightness or in gray. S134 is a step in which the gate image is displayed in a toned-down manner. S135 is a step in which the setting is switched to not display the gate image G5.

[0134] According to the above configuration, the driving system 30 displays the gate image G5 in a toned-up state only when the gate is recognized during automated driving. Therefore, the driver can check the driving system 30's recognition state of the gate from the display state of the gate image G5 during automated driving. Note that the HMI control unit F5 may completely hide the gate image G5 when the automation level is 3 or higher and the gate cannot be recognized. S134 may be a step of hiding the gate image G5, similar to S135.

[0135] In other embodiments, S131 and S135 may be omitted. The HMI control unit F5 may be configured to change the display state of the gate image G5 based on the recognition state of the gate, regardless of the automation level.

[0136] The HMI control unit F5 may be configured to display a gate icon, which is an icon image indicating the recognition status of the gate, on the meter display separately from the route image Im. The HMI control unit F5 may also change the display mode of the gate icon based on the automation level and the recognition status of the gate.

[0137] Alternatively, if a gate is not detected, the HMI control unit F5 may display a route image Im in which a gate image G5 with a question mark added to the gate position indicated by the map data is placed. This configuration also allows the driver to perceive the gate recognition state of the driving system 30.

[0138] <Display Control Example (5) of Route Image Im> The HMI control unit F5 may acquire the difference in the number of lanes based on map data or the like, and change the manner in which information related to the passage gate is notified based on the difference in the number of lanes. The difference in the number of lanes is the difference between the number of entrance lanes and the number of exit lanes. The difference in the number of lanes may be expressed as an absolute value.

[0139] For example, as shown in Fig. 14, the HMI control unit F5 acquires the difference in the number of lanes based on map data or the like (S141). S141 may be executed when the vehicle enters the area in front of the gate or when the remaining distance to the gate point becomes less than a predetermined value. In the figure, "ΔLN" represents the difference in the number of lanes. "ThD" represents the threshold value for the difference in the number of lanes.

[0140] If the lane number difference (ΔLN) is equal to or greater than a predetermined threshold (ThD) (YES in S142), the HMI control unit F5 may be configured to notify the information related to the passage gate in a prominent manner (S143).Also, if the lane number difference is less than the predetermined threshold (NO in S142), the HMI control unit F5 may be configured to notify the information related to the passage gate in a discreet manner (S144).

[0141] The notification of the information related to the passage gate may be a display of a route image Im in the area in front of the gate, or may include outputting a voice message guiding the user to the gate through which the user plans to pass.

[0142] The HMI control unit F5 of this embodiment may be configured to selectively adopt two notification modes for information related to a passage gate: a prominent mode and a discreet mode. The discreet mode refers to a mode in which stimuli such as light and sound are reduced compared to the prominent mode. The discreet mode refers to a notification mode aimed at not bothering the occupants. Notification in a discreet mode refers to a mode in which an image display is the main mode, no vibration is applied to the driver, and the output volume of the notification sound is set to a predetermined value or less. Setting the output volume to a predetermined value or less includes not outputting any sound. The discreet mode can also be referred to as an inconspicuous mode. Notification of information related to a passage gate in a discreet mode may include displaying the route image Im at a normal size or displaying the gate image G5 in a hidden or discreet mode.

[0143] Providing a conspicuous notification means providing a notification in a manner intended to ensure that the driver is clearly aware of the notification content. Providing a conspicuous notification may involve outputting a voice message / sound effect at a volume equal to or greater than a predetermined value. Providing a conspicuous notification may involve applying vibration to the driver. Providing a conspicuous notification corresponds to outputting a stimulus of sufficient intensity to attract the driver's attention. Providing a conspicuous notification of information related to a passage gate may include increasing the display size of the route image Im or flashing the gate image G5.

[0144] The greater the difference in the number of lanes, the more likely it is that the trajectories of vehicles before and after the gate will cross. In other words, the greater the difference in the number of lanes, the greater the difficulty of automated driving. When the difference in the number of lanes is large, safety may be improved if the driver monitors the surroundings. When the difference in the number of lanes is large, by issuing a notice regarding passing through the gate in a conspicuous manner as described above, it is expected that the effect of directing the driver's attention in the direction of the gate is achieved. On the other hand, when the difference in the number of lanes is small, there is less risk of vehicle trajectories crossing than when the number of lanes is large. When the difference in the number of lanes is small, there is less need for a driver using the automated driving function to check the traffic conditions outside the vehicle. When the difference in the number of lanes is less than a predetermined value, issuing a notice regarding passing through the gate in a more discreet manner can reduce the risk of annoyance to the driver.

[0145] The change in the notification mode based on the difference in the number of lanes described above may be performed only when the automation level is equal to or greater than 3. In other embodiments, the change in the notification mode based on the difference in the number of lanes may be performed regardless of the automation level.

[0146] <Example (6) of display control of route image> The processor 31 as the HMI control unit F5 may evaluate the congestion level based on the detection result of the environmental sensor 11, and may change the position of the virtual viewpoint used to generate the route image Im according to the congestion level. The congestion level is a parameter that indicates the degree of traffic congestion in the surrounding area.

[0147] The congestion level may be, for example, the level of congestion in the left-right direction. The congestion level may be evaluated in three levels: level 1, 2, and 3. The higher the congestion level, the more crowded the surrounding area is. A congestion level of 2 or 3 corresponds to a case where the congestion level is higher than a predetermined value. A congestion level of 1 corresponds to a case where the congestion level is lower than a predetermined value.

[0148] The HMI control unit F5 may determine the congestion level as 1 if there is a vacant space of a predetermined size or more on both the left and right sides. The size used in this determination may be a size corresponding to the space required for a lane change. In other words, if a lane change is possible to either the left or right, the congestion level may be determined as level 1. The vertical length of the space required for a lane change may vary depending on the speed. The vertical direction corresponds to the traveling direction set on the road, in other words, the front-to-rear direction of the vehicle. Note that the term "lane change" may be interpreted as a lateral movement of 2.5 m or more in a laneless section. The congestion level may be determined based on the size of the vacant space on the left side of the vehicle and the size of the vacant space on the right side.

[0149] Furthermore, when only one of the left and right lanes is blocked by another vehicle, the processor 31 may determine the congestion level to be 2. When the left side is blocked by another vehicle, this may be interpreted as when there is no free space to change lanes on the left side. When both the left and right sides are blocked by other vehicles, the processor 31 may determine the congestion level to be 3.

[0150] In other embodiments, the congestion level may be determined based on the number of other vehicles ahead of the vehicle or the number of other vehicles behind the vehicle. The congestion level may be divided into two levels or four or more levels instead of three levels. The congestion level may be expressed as a score with a predetermined upper limit of 100 or the like.

[0151] Fig. 15 shows an example of control in which the position of the virtual viewpoint is changed depending on the level of congestion. The process shown in Fig. 15 changes the viewpoint of the route image, in other words, changes the display mode of the route image. S181 shown in Fig. 15 is a step in which the processor 31 evaluates the level of congestion based on the detection results of the environmental sensor 11. Note that the term "detection results of the environmental sensor 11" may be replaced with environmental information generated by the environment recognition unit F1.

[0152] S181 may be executed at regular intervals while the vehicle Hv is in the gate area. For example, S181 may be executed every 2 seconds, 4 seconds, or 6 seconds. If the viewpoint changes multiple times in a short period of time, this may be annoying to the user. Therefore, the execution cycle of S181 may be set to a time that is sufficiently longer than the sensing interval of the environmental sensor 11. Data indicating the congestion level determination result may be stored in memory 32. The congestion level may be managed using a flag or the like.

[0153] In S182 following S181, the processor 31 determines whether the congestion level is 1. If the congestion level is 1 (YES in S182), S183 is executed. In S183, the processor 31 sets the virtual viewpoint used to generate the route image to the first viewpoint position Vb1. Execution of S183 starts generation of a route image seen from the first viewpoint position Vb1.

[0154] The first viewpoint Vb1 is set at a lower position than the other viewpoints described below. For example, the first viewpoint Vb1 may be located at a height of 8 m from the road surface and 1 m behind the vehicle Hv.

[0155] On the other hand, if the congestion level is not 1 (NO in S182), it is determined in S184 whether the congestion level is 2. If the congestion level is 2 (YES in S184), S185 is executed. In S185, the processor 31 sets the virtual viewpoint used to generate the route image to the second viewpoint position Vb2. Execution of S185 starts generation of a route image seen from the second viewpoint position Vb2.

[0156] The second viewpoint Vb2 is set at a position higher than the first viewpoint Vb1 by a predetermined amount, as shown in Fig. 16. For example, the second viewpoint Vb2 may be located at a height of 10 m from the road surface.

[0157] If the congestion level is neither 1 nor 2 (NO in S184), that is, if the congestion level is 3, the processor 31 sets the virtual viewpoint used to generate the route image to the third viewpoint position Vb3 in S186. By executing S186, generation of the route image seen from the third viewpoint position Vb3 begins.

[0158] The third viewpoint Vb3 is set at a position higher than the second viewpoint Vb2 by a predetermined amount, as shown in Fig. 16. For example, the third viewpoint Vb3 may be located at a height of 12 m from the road surface.

[0159] Changing the height of the virtual viewpoint corresponds to changing the height of the bird's-eye view display. The higher the virtual viewpoint, the wider the displayed range. According to the above configuration, the driving system 30 sets the virtual viewpoint higher as the congestion level increases. This allows a wider range of the surrounding environment to be displayed without changing the display size of the route image itself. Therefore, the more congested the surrounding area, the easier it is for the driver to check the surrounding conditions over a wider range. As a result, it becomes easier to determine whether the current trajectory is acceptable or whether it is better to change lanes. Furthermore, when the road is relatively empty, the viewpoint is located closer to the vehicle, making it easier to check the position of the vehicle relative to white lines, etc.

[0160] Note that setting data (hereinafter, viewpoint setting) that specifies the position of a settable virtual viewpoint may be registered in the memory 32. S183 may be interpreted as a step of applying a first viewpoint setting, S185 may be interpreted as a step of applying a second viewpoint setting, and S186 may be interpreted as a step of applying a third viewpoint setting.

[0161] The viewpoint setting may include a position and a line of sight direction as parameters. The viewpoint setting may also include a parameter that defines a field of view φ. The field of view defines the range to be displayed as a route image. The field of view φ may be appropriately designed to include a predetermined range around the vehicle. The field of view may differ depending on the viewpoint position. The viewpoint setting may be configured to be fine-tuned by the user.

[0162] The height and vertical position of each virtual viewpoint shown in FIG. 16 may be different. The vertical position is a position in the front-to-rear direction. The second viewpoint position Vb2 shown in FIG. 16 may be located a predetermined distance behind the first viewpoint position Vb1. The third viewpoint position Vb2 shown in FIG. 16 may be located a predetermined distance behind the second viewpoint position Vb2. The first viewpoint position Vb2 may be located a predetermined distance behind the position directly above the center of the vehicle.

[0163] The setting of the first, second, and third viewpoint positions is not limited to the example shown in FIG. 16 . As shown in FIG. 17 , the first, second, and third viewpoint positions may be the same height but different positions in the front-to-rear direction. The processor 31 may be configured to shift the virtual viewpoint further rearward as the degree of congestion decreases. By shifting the virtual viewpoint further rearward, it becomes easier to check the relative positional relationship between the gate and the vehicle. The line of sight direction for each viewpoint position may be adjusted so that the desired display object is displayed.

[0164] The controller 31 may be configured to execute the above-described process of changing the virtual viewpoint according to the congestion level only when the host vehicle is located within a gate area. In other words, the controller 31 may change the position of the virtual viewpoint used to generate the route image Im related to passing through the gate according to the congestion level. Of course, the controller 31 may be configured to change the virtual viewpoint according to the congestion level not only when the host vehicle is located within the gate area but also when the host vehicle is traveling outside the gate area.

[0165] The processor 31 as the planning unit F3 may be configured to determine whether an automatic settlement function for automatically settling tolls on toll roads for the vehicle is available, and to set a target gate based on the determination result. The automatic settlement function may be available when a dedicated card (so-called ETC card) is inserted into an on-board device (e.g., an ETC on-board device) that wirelessly communicates with an external device for payment.

[0166] If the vehicle is not equipped with an on-board unit for automatic settlement, or if an ETC card is not inserted in the on-board unit, the processor 31 may determine that the automatic settlement function is unavailable. Information on on-board equipment, such as whether or not an on-board unit is installed, may be registered in advance in the storage 33, etc. The processor 31 may determine whether or not an on-board unit for automatic settlement is installed by referring to the storage 33. Furthermore, if the vehicle is equipped with an on-board unit, the processor 31 may obtain whether or not an ETC card is inserted in the on-board unit by communicating with the on-board unit. The processor 31 may obtain whether or not the automatic settlement function is available through an operational input by the driver.

[0167] The processor 31 may set the target gate, for example, according to the procedure shown in Fig. 18. S151 shown in Fig. 18 is a step in which the processor 31 acquires information about the payment method for each gate provided at the gate location ahead, based on at least one of map data, the detection result of the environmental sensor 11, and a wireless signal from a roadside device. When S151 is completed, S152 is executed.

[0168] In step S152, the processor 31 determines whether or not there is a gate that supports the automatic fare adjustment method at the gate point ahead, based on the information acquired in step S151. If there is a gate that supports the automatic fare adjustment method at the gate point ahead (YES in step S152), step S153 is executed. If there is no gate that supports the automatic fare adjustment method at the gate point ahead (NO in step S152), step S156 is executed.

[0169] S153 is a step in which the processor 31 determines whether the automatic payment function is available. If the automatic payment function is available (YES in S152), the processor 31 executes a payment method inquiry process (S154). The payment method inquiry process is a process inquiring of the driver about the toll payment method. The payment method inquiry process may include displaying a predetermined inquiry screen. The inquiry screen may include a first button for the driver to input that the automatic payment method will be used and a second button for the driver to input that the manual payment method will be used. The payment method inquiry process may include outputting a voice message inquiring about the payment method. The processor 31 may obtain the driver's response to the payment method inquiry process based on a signal from the input device 23. The processor 31 may obtain the driver's response by voice input. The processor 31 may obtain the driver's desired payment method, etc., through a conversation with the driver. If the driver's response is obtained in S154, S155 is executed.

[0170] S155 is a step of setting a target gate from among the gates (hereinafter also referred to as candidate gates) corresponding to the payment method selected by the driver. If there are multiple candidate gates, the processor 31 may set the gate among the candidate gates that is closest to the road along which the driver plans to travel after passing through the gate as the target gate. In another example, the processor 31 may set the gate among the candidate gates that is closest to the current traveling position as the target gate. If there is only one candidate gate, the processor 31 may set the candidate gate as the target gate.

[0171] In addition, settlement method setting data may be pre-registered in storage 33. The settlement method setting data is data indicating whether a gate that supports the automatic settlement method is to be selected as the target gate. When the settlement method setting data is stored in storage 33, processor 31 may determine the type of target gate by referencing the settlement method setting data. When the settlement method setting data is stored in storage 33, the settlement method inquiry process may be omitted. S154 may be a step of obtaining the settlement method desired by the driver by referencing the settlement method setting data.

[0172] S156 is a step of setting a target gate from among gates that support the manual payment method. If there are multiple gates that support the manual payment method in S156, processor 31 may set the gate that is closest to the road along which the vehicle will travel after passing through the gate as the target gate. Processor 31 may also set the gate that is closest to the current traveling position as the target gate. Once the target gate has been set, this flow may end.

[0173] The above-described process for setting the target gate may be executed when the remaining distance to the gate point falls below a predetermined value while driving in AD mode or hands-off level 2 mode.

[0174] 19, when the driver is planning to pass through a gate that requires manual fare settlement (YES in S161), the processor 31 as the HMI control unit F5 may execute a process of notifying the driver that the driving mode will be switched to manual driving mode when the driver stops at the gate (S162). A gate that requires manual fare settlement may be considered as a gate that does not require automatic fare settlement.

[0175] 18 may be a step of determining whether the target gate is a gate requiring manual settlement, based on at least one of map data, the detection result of the environmental sensor 11, and a wireless signal from a roadside device. S161 may be executed in response to the vehicle entering the area in front of the gate in the autonomous driving mode.

[0176] In addition to switching to the manual driving mode at the gate location, S162 may include notifying the driver that the vehicle will be passing through a gate that does not support automatic fare settlement. The notice of switching to the manual driving mode at the gate location may be accompanied by output of a voice message. The notice of the vehicle that the vehicle will be passing through a gate that supports manual fare settlement may also include output of a voice message.

[0177] If the target gate is an automatic fare-adjusted gate (NO in S162), the HMI control unit F5 may issue a notification that the gate is approaching (S163). The notification that the gate is approaching may essentially be a notification of information related to the passage gate. The notification in S163 may be prominent or discreet. S163 may be omitted.

[0178] When the vehicle stops at a gate to pay the toll, it is difficult for the driving system 30 to detect that the driver has completed payment and is ready to start. Furthermore, it is expected that there will be a large speed difference between the vehicle that has stopped temporarily at the gate and other vehicles that have passed through the automatic payment system. When entering a gate with a manual payment system, it may be more difficult to control the vehicle after passing through the gate than when entering a gate with an automatic payment system. For these reasons, when the vehicle stops at a gate with a manual payment system, safety can be further improved by temporarily leaving driving operations to the driver.

[0179] <Other Operation Example (2)> While in level 4 mode, the processor 31 as the HMI control unit F5 may determine whether the driver is asleep based on a signal from the occupant state sensor 16. Furthermore, the HMI control unit F5 may change the timing or implementation of the gate-related notification depending on whether the driver is asleep. The gate-related notification may include a gate pre-notification that is performed before passing through the gate, and a gate passing notification that is performed after passing through the gate. The gate pre-notification may be a notification related to the aforementioned traffic gate, or a notification that the gate is approaching. The gate passing notification may be a notification of the toll fee for the toll road.

[0180] The HMI control unit F5 may omit the gate pre-warning when the driver is asleep. This is because there is a risk that the driver will be woken up by the warning. The HMI control unit F5 may be configured to perform the gate pre-warning in a discreet manner when the driver is asleep. The HMI control unit F5 may perform the gate pre-warning in a conspicuous manner when the driver is awake.

[0181] The gate passage notification is a notification regarding fees and is an important notification from a cost perspective. The gate passage notification may be executed without omitting it. However, notification during sleep may lead to a decrease in convenience or comfort for the driver. For example, the HMI control unit F5 may be configured to control the timing of the gate passage notification according to the procedure shown in FIG. 20. S171 shown in FIG. 20 is a step in which the HMI control unit F5 acquires the driver's status based on a signal input from the occupant status sensor 16. S171 may be executed based on receiving a gate passage notification request from the planning unit F3 while driving in level 4 mode. S171 may also be a step in which data indicating the driver's status stored in the memory 32 is read. After S171 is completed, S172 is executed.

[0182] S172 is a step for determining whether the driver is asleep. If the driver is asleep (S172 YES), execution of the gate passage notification is suspended (S173). In S173, the HMI control unit F5 may record the fact that the gate passage notification is suspended using a flag or the like. When S173 is completed, the HMI control unit F5 executes S174. If the driver is not asleep (S172 NO), the HMI control unit F5 may execute S175.

[0183] S174 is a step in which the HMI control unit F5 determines whether the driver has woken up. The determination in S174 may be made based on a signal from the occupant state sensor 16. S174 may be executed periodically (for example, every 10 seconds) until the driver wakes up. If the driver is asleep (S174 NO), S174 may be executed again after a predetermined time. If it is detected that the driver has woken up (S174 YES), the HMI control unit F5 executes S175. S175 is a step in which the HMI control unit F5 executes a gate passage notification using at least one of the display 21 and the speaker 22.

[0184] The HMI control unit F5 may perform the gate passage notification in a conservative manner in S173. Even if the gate passage notification is performed in a conservative manner in S173, the HMI control unit F5 may perform the gate passage notification again upon detecting that the driver is awake.

[0185] <Other Operation Example (3)> Generally, the traveling speed when passing through a gate is limited to a predetermined value or less. Therefore, the processor 31 as the planning unit F3 may automatically change the ACC set speed to a value less than the speed limit near the gate in accordance with traffic rules indicated by speed signs or the like. When the set speed is automatically changed near the gate, the HMI control unit F5 may execute a process to notify the driver that the set speed will be changed, in addition to displaying the normal set speed image Im2, as shown in FIG. 21 .

[0186] FIG. 21 is an example of a display screen of the display 21. FIG. 21 shows an example in which the HMI control unit F5 displays a route image Im, a set speed image Im2, a speed image Im3, a speed sign image Im4, and a deceleration warning image Im5 on the display 21. The set speed image Im2 is an image showing the value of the set speed of the ACC. The set speed of the ACC may be interpreted as a target speed in autonomous driving. In the present disclosure, an operation mode in which the motion actuator 19 is automatically controlled so that the traveling speed of the host vehicle becomes a predetermined set speed (target speed) is also referred to as a speed management mode. A mode in which the ACC operates corresponds to a speed management mode. In this embodiment, a hands-on level 2 mode, a hands-off level 2 mode, a level 3 mode, a level 4 mode, and the like correspond to the speed management mode.

[0187] The speed sign image Im4 is an image that represents the speed limit. The HMI control unit F5 may determine the speed limit from the detection results of the environmental sensor 11 (mainly a camera). The HMI control unit F5 may also determine the speed limit by referring to map data. Note that if a speed sign is not detected by the camera, the speed sign image Im4 does not need to be displayed.

[0188] The deceleration warning image Im5 is an image that warns of deceleration when passing through a gate. The deceleration warning image Im5 corresponds to an image that warns of a change in the set speed. The HMI control unit F5 may display the deceleration warning image Im5 when it detects that the speed limit at the gate location is different from the current speed limit. The HMI control unit F5 may also output from the speaker 22 a voice message indicating that the vehicle will decelerate to pass through the gate, along with displaying the deceleration warning image Im5. The HMI control unit F5 may also implement a warning in front of the gate that includes displaying the deceleration warning image Im5.

[0189] 21 may be a part of the display screen of the display 21. The display screen of the display 21 may display a speedometer image, a meter image of the motor / engine rotation speed, an image indicating the shift position, an image indicating the cruising distance, an image indicating the remaining fuel / battery amount, an image indicating the operation mode (automation level), etc. The background of the display screen may be black, for example.

[0190] <Other operation example (4)> The HMI control unit F5 may be configured to change the display mode of the route image Im after passing through a gate depending on the road configuration of the area after the gate. If the area after the gate is not a lane-free section, the HMI control unit F5 may display the lane mark image G2 in a prominent manner. On the other hand, if the area after the gate is a lane-free section, the HMI control unit F5 may display a route image Im that does not include the lane mark image G2 and that prominently depicts road edge images, which are image elements that indicate road edges.

[0191] The HMI control unit F5 may be configured to change the display mode of the path image Im after passing through the gate depending on the automation level when traveling through the post-gate area. When the automation level when traveling through the post-gate area is equal to or greater than a predetermined value, the HMI control unit F5 may display the track image G3 in a more prominent manner than when the automation level is less than the predetermined value. When the automation level when traveling through the post-gate area is less than the predetermined value, the HMI control unit F5 may not display the track image G3.

[0192] <Other Operation Example (5)> The processor 31 may be configured to detect an emergency vehicle using the environmental sensor 11. Furthermore, the processor 31 may be configured to plan and execute a behavior according to the situation in response to the detection of the emergency vehicle. In response to the detection of the emergency vehicle, the processor 11 plans and executes a basic response, which is an action of yielding the right of way (also called the right of way) to the emergency vehicle.

[0193] The basic response outside an intersection on a general road may be to pull over to the left edge of the road and stop. In some areas, the basic response may be to pull over to the right edge and stop. In an intersection, the basic response may be to pull over to the left (or right) edge of the road and stop after exiting the intersection. The basic response on a highway may differ depending on the number of lanes. If there is only one lane, the vehicle may pull over to the left (right) edge of the road and stop so that there is space for the emergency vehicle to pass. If there are two or more lanes, the vehicle may stay in the driving lane so that the emergency vehicle can pass in the passing lane. The driving lane may be referred to as the first lane. In areas where driving is on the left side, the first lane may be the leftmost lane, and in areas where driving is on the right side, the first lane may be the rightmost lane. When an emergency vehicle approaches during automated driving, the processor 31 may be configured to respond in accordance with the traffic rules of the country or region in which the automated vehicle is used. Furthermore, when an emergency vehicle approaches during autonomous driving, the processor 31 may be configured to react in the same way as other vehicles in front or behind.

[0194] However, the system response when detecting the approach of an emergency vehicle while the vehicle is near a gate has not been studied to date. When a preceding or following vehicle is present, the system may follow the preceding vehicle's behavior. However, there may be cases where neither a preceding nor a following vehicle is present. The road width in the gate area may be widened, and the vehicle may be far from the road edge. Therefore, moving laterally to the road edge is not always an appropriate response. Furthermore, in some areas, emergency vehicles may enter (or exit) expressways using special gates for specific vehicles that are separate from gates for general vehicles. In such areas, emergency vehicles do not use gates for general vehicles. When the vehicle detects the approach of an emergency vehicle while near a gate, stopping within the gate area and waiting for the emergency vehicle to pass may also be an appropriate response.

[0195] For this reason, when an emergency vehicle is detected in the AD mode or the hands-off level 2 mode, the processor 31 may be configured to perform processing according to the control flow shown in Fig. 22. S201 shown in Fig. 22 may be started in response to the detection of an emergency vehicle in the AD mode or the hands-off level 2 mode. The emergency vehicle may be detected using an acoustic sensor, a camera, vehicle-to-vehicle communication, or the like.

[0196] In S201, the processor 31 determines whether the host vehicle is located in the area in front of the gate based on the environmental information. If it is determined that the host vehicle is located in the area in front of the gate (YES in S201), the processor 31 determines in S202 to continue traveling toward the gate. As described above, on general roads, stopping when an emergency vehicle is approaching may be an appropriate behavior. However, in the area in front of the gate, stopping within the gate is less likely to obstruct the passage of the emergency vehicle than stopping away from the gate. If the host vehicle is still located in the area in front of the gate, continuing traveling may contribute to creating a passage space for the emergency vehicle.

[0197] S202 may include the controller 31 reducing the driving speed in response to the presence of an emergency vehicle near the gate. The reduced speed may be a fixed value or may be a predetermined amount lower than the speed that would be applied if no emergency vehicle was detected.

[0198] In S203 following S202, the processor 31 issues a gate approach notification using the display 21 or the like. The gate approach notification is a process of notifying the driver that it is appropriate to continue driving toward the gate. The gate approach notification may include an explanation that continuing driving toward the gate is appropriate behavior. As explained above, on public roads, stopping is a common behavior when an emergency vehicle is approaching, and continuing driving may make the driver feel uncomfortable. By notifying the driver of the appropriateness of continuing driving toward the gate in conjunction with S202, the driver's sense of security may be increased.

[0199] On the other hand, if it is determined that the host vehicle is not located in the area in front of the gate (NO in S201), the processor 31 determines in S204 whether the host vehicle is passing through the gate based on the environmental information. While passing through the gate, the host vehicle may be in the area inside the gate. If it is determined that the host vehicle is located in the area inside the gate (YES in S204), the processor 31 determines in S205 to stop inside the gate. The stopping position inside the gate may be as close to the exit as possible so that following vehicles can also enter the gate. In S206 following S205, the processor 31 issues an inside-gate stop notification using the display 21 or the like. The inside-gate stop notification is a process of notifying the driver that the vehicle will stop inside the gate. The inside-gate stop notification may include an explanation that stopping inside the gate is appropriate behavior that does not obstruct the passage of emergency vehicles.

[0200] Furthermore, if it is determined that the position of the host vehicle is neither in the area in front of the gate nor in the area inside the gate (NO in S204), the processor 31 determines whether the host vehicle is located in the area behind the gate based on the environmental information in S207. If it is determined that the host vehicle is located in the area behind the gate (YES in S207), the processor 31 notifies the driver of the presence of an emergency vehicle and its direction using the display 21 or the like in S208. The direction of the emergency vehicle means the direction in which the emergency vehicle is located. If the direction in which the emergency vehicle is located cannot be determined due to occlusion by other vehicles or sound reverberation, the controller 31 may notify the driver only that an emergency vehicle is present in S208, or may also notify the driver that the direction in which the emergency vehicle is located is undetermined.

[0201] If the direction of the emergency vehicle can be identified, the controller 31 may notify the driver of the direction of the emergency vehicle. If the direction of the emergency vehicle can be identified and the processor 31 detects that the emergency vehicle is ahead of the host vehicle, the processor 31 may decide to continue driving in S208. If the direction of the emergency vehicle can be identified and the processor 31 detects that the emergency vehicle is behind the host vehicle, the processor 31 may decide to stop on the shoulder of the road in S208, as with the basic response. If the direction of the emergency vehicle cannot be identified, the processor 31 may temporarily terminate automated driving after stopping and output TOR to the driver in S208.

[0202] If it is determined that the host vehicle is not in the area in front of the gate, the area inside the gate, or the area after the gate (NO in S207), the processor 31 plans and executes a basic response according to the situation in S209. Furthermore, in S210, the processor 31 notifies the driver of the planned behavior of the host vehicle as the basic response. If it is determined that the host vehicle is not in the area in front of the gate, the area inside the gate, or the area after the gate, this means that the host vehicle is not traveling near the gate, and the basic response should be applied. By adopting the basic response when the host vehicle is not traveling near the gate, it is possible to reduce the risk that the host vehicle will obstruct the passage of an emergency vehicle.

[0203] In this way, when the processor 31 detects an emergency vehicle near the gate, it changes the behavior and notification content according to the position of the host vehicle relative to the gate. This reduces the risk of obstructing the passage of the emergency vehicle. In addition to the above control, the controller 31 also notifies the driver of the host vehicle's planned behavior and its validity using the display 21, etc. This reduces the risk of confusing the driver.

[0204] When an emergency vehicle is detected in the manual steering mode, the processor 31 may execute a process to suggest a driving operation according to the vehicle's position relative to the gate. In the manual steering mode, instead of S202 to S203, the processor 31 may suggest continuing driving toward the gate. In the manual steering mode, instead of S205 to S206, the processor 31 may suggest stopping within the gate. In the manual steering mode, instead of S208, the processor 31 may suggest stopping and monitoring the behavior of the emergency vehicle. The manual steering mode is a mode in which the driver performs steering, and includes a hands-on level 2 mode, a manual driving mode, etc.

[0205] When the processor 31 detects an emergency vehicle while traveling near an access gate, the processor 31 may identify an emergency vehicle access point based on map data. The access gate is a gate for entering an expressway from a general road. The emergency vehicle access point is a point where a road for emergency vehicles connects with the main expressway. The processor 31 may be configured to stop the emergency vehicle behind the emergency vehicle access point when the processor 31 is located behind the emergency vehicle access point.

[0206] When the processor 31 detects an emergency vehicle while traveling near an exit gate, the processor 31 may identify an emergency vehicle exit point based on map data. The exit gate refers to a gate for exiting from an expressway onto a general road. The emergency vehicle exit point is a point where a road for emergency vehicles connects with a general road. When the processor 31 is located behind the emergency vehicle exit point, the processor 31 may control the host vehicle to stop behind the emergency vehicle exit point.

[0207] When the host vehicle is stopped because of an emergency vehicle, the processor 31 may resume driving when it detects that the emergency vehicle is no longer detected or that the emergency vehicle is located ahead of the host vehicle. Furthermore, when the host vehicle is stopped because of an emergency vehicle, the processor 31 may resume driving when it detects that the preceding vehicle has started moving.

[0208] <Regarding System Configuration> The HMI control unit F5 may be located outside the operation system 30. The HMI control unit F5 may be provided in another computer such as an HCU. Some of the data stored in the storage 33 may be stored in an external server or the like.

[0209] <Supplementary Remark (1)> This specification discloses the following technical ideas and their combinations. In addition, vehicle control methods and computer programs corresponding to the following technical ideas are also included within the scope of the present disclosure.

[0210] [Technical Idea 1] A display control device used in a vehicle equipped with multiple operating modes with different levels of automation of driving operations, comprising: a control unit (31) that controls the display of a route image, which is an image showing the vehicle's driving route; and a communication circuit (34) that enables the control unit to communicate with other devices, wherein the control unit: acquires related road information, which is information about roads around a gate through which the vehicle is scheduled to pass, from the other devices via the communication circuit; acquires information indicating the automation level of the operating mode currently applied; and changes the configuration of the pre-gate image, which is the route image when traveling in the area on the entrance side of the gate, based on the automation level information or the related road information.

[0211] [Technical Idea 2] The display control device described in Technical Idea 1, wherein the related road information includes whether or not the area on the entrance side of the gate is a laneless section, which is a section without lane marks, and the control unit is configured to change the configuration of the image in front of the gate depending on whether or not the area on the entrance side of the gate is the laneless section.

[0212] [Technical Idea 3] The display control device according to Technical Idea 1 or 2, wherein the related road information includes whether or not the area on the entrance side of the gate is a laneless section, which is a section without lane marks, and the control unit is configured to: if the area on the entrance side of the gate is not the laneless section, display an image including a trajectory image which is an image element indicating the trajectory of the vehicle and a lane mark image which is an image element indicating lane marks, as the pre-gate image; and if the area on the entrance side of the gate is not the laneless section, display an image including a gate image which is an image element indicating the gate and the trajectory image, without including the lane mark image, as the pre-gate image.

[0213] [Technical Idea 4] A display control device described in any one of Technical Ideas 1 to 3, wherein the control unit is configured to include more information about the vehicle's trajectory in the gate-front image when the automation level is equal to or greater than a predetermined value than when the automation level is less than the predetermined value.

[0214] [Technical Idea 5] A display control device according to Technical Idea 4, wherein the pre-gate image displayed when the automation level is less than the predetermined value is an image that does not include a post-gate track image, which is an image element that shows the trajectory of the vehicle after passing through the gate, and the pre-gate image displayed when the automation level is equal to or greater than the predetermined value is an image that includes the post-gate track image.

[0215] [Technical Idea 6] A display control device described in any one of Technical Ideas 1 to 5, wherein the control unit is configured to include more information about the gate in the pre-gate image when the automation level is less than a predetermined value than when the automation level is equal to or greater than the predetermined value.

[0216] [Technical Idea 7] A display control device described in any one of Technical Ideas 1 to 6, wherein the pre-gate image includes a gate image that is an image showing the gate, and the control unit is configured to display the gate image in a more prominent manner when the automation level is less than a predetermined value than when the automation level is equal to or greater than the predetermined value.

[0217] [Technical Idea 8] A display control device described in any one of Technical Ideas 1 to 7, wherein the control unit is configured to display a gate image, which is an image showing the gate, on the display when the automation level is equal to or greater than a predetermined value, and not to display the gate image on the display when the automation level is less than the predetermined value.

[0218] [Technical Idea 9] The control unit acquires information indicating whether or not a gate through which the vehicle is to pass has been detected by an environmental sensor mounted on the vehicle, based on a signal received by the communication circuit; if the automation level is equal to or greater than a predetermined value and the gate can be detected by the environmental sensor, displays a gate image that is an image of the gate in a predetermined basic manner; if the automation level is equal to or greater than the predetermined value and the gate cannot be detected by the environmental sensor, does not display the gate image or displays it in a manner that is less noticeable than the basic manner; and if the automation level is less than the predetermined value, does not display the gate image. This is a display control device described in any one of Technical Ideas 1 to 7.

[0219] [Technical Idea 10] The display control device according to any one of Technical Ideas 1 to 9, wherein the gate front image is a bird's-eye view image seen from a virtual viewpoint arranged above the vehicle or a predetermined distance behind it.

[0220] [Technical Idea 11] A display control device described in any one of Technical Ideas 1 to 10, wherein the related road information includes a number of entry lanes, which is the total number of lanes existing on the entrance side of a gate area, and a number of exit lanes, which is the total number of lanes existing on the exit side of the gate area, and the control unit is configured to, when the difference between the number of entry lanes and the number of exit lanes is equal to or greater than a predetermined value, implement a notification regarding passage through the gate in a more noticeable manner than when the difference between the number of entry lanes and the number of exit lanes is less than the predetermined value.

[0221] [Technical Idea 12] The display control device described in any one of Technical Ideas 1 to 11, wherein the multiple operating modes include an autonomous driving mode in which the system performs all driving tasks and a manual driving mode in which the driver performs all driving tasks, and the control unit is configured to: determine whether the gate through which the vehicle is scheduled to pass is a manual settlement gate based on a signal received by the communication circuit; and, if the gate through which the vehicle is scheduled to pass is a manual settlement gate and the vehicle is traveling in front of the gate in the autonomous driving mode, execute processing to notify the driver that the vehicle is scheduled to pass through a manual settlement gate and that the driving mode will be switched to the manual driving mode when the vehicle stops at the gate.

[0222] [Technical Idea 13] The control unit is configured to: obtain, via the communication circuit, data indicating whether an automatic settlement function that automatically makes payments for the use of toll roads is available in the vehicle; obtain information regarding payment methods for one or more gates ahead of the vehicle based on the signal received by the communication circuit; and, if the automatic settlement function is available and there are both a gate that does not support automatic settlement and a gate that supports automatic settlement ahead of the vehicle, execute a process to inquire of the driver whether to pass through the gate that supports automatic settlement. This is a display control device described in any one of Technical Ideas 1 to 12.

[0223] [Technical Idea 14] The display control device according to any one of Technical Ideas 1 to 13, wherein the plurality of operation modes include a level 4 mode in which the system executes all driving tasks and the driver is allowed to sleep, and the control unit is configured to: while the operation mode is the level 4 mode, determine whether the driver is sleeping based on a signal received by the communication circuit; if the operation mode when the gate was passed was the level 4 mode and the driver was asleep, suspend notification of information related to passing through the gate; and if the driver wakes up, implement the suspended notification.

[0224] [Technical Idea 15] A display control device according to any one of Technical Ideas 1 to 14, wherein the plurality of operation modes include a speed management mode in which the speed is automatically adjusted so that the vehicle's traveling speed becomes a set speed, and the control unit is configured to, when the operation mode is the speed management mode, display a set speed image which is an image element indicating the set speed, and when the set speed is automatically changed in accordance with traffic rules, in addition to displaying the set speed image, execute processing to notify the driver that the set speed will be changed.

[0225] [Technical Idea 16] The display control device according to any one of Technical Ideas 1 to 15, wherein the control unit is configured to: detect the presence of an emergency vehicle based on a signal received by the communication circuit; upon detecting the emergency vehicle, issue a notification related to the emergency vehicle to the driver; acquire data indicating the relative position of the vehicle with respect to the gate via the communication circuit; and, if the emergency vehicle is detected while the vehicle is traveling around the gate, change the content of the notification depending on the relative position.

[0226] [Technical Idea 17] The display control device according to Technical Idea 16, wherein the control unit is configured to, when detecting the emergency vehicle while the vehicle is traveling in an area on the entrance side of the gate, issue a notification to continue traveling toward the gate.

[0227] [Technical Idea 18] The display control device according to Technical Idea 16 or 17, wherein the control unit is configured to, when detecting the emergency vehicle while the vehicle is passing through the gate, issue a notification that the vehicle will stop inside the gate.

[0228] [Technical Idea 19] The display control device according to any one of Technical Ideas 16 to 18, wherein the control unit is configured to, when the emergency vehicle is detected, further acquire a direction of presence, which is the direction in which the emergency vehicle is located, based on a signal received by the communication circuit, and, when the emergency vehicle is detected while the vehicle is traveling in the area on the exit side of the gate and the direction in which the emergency vehicle is located can be identified, notify the user of the presence of the emergency vehicle and its direction of presence.

[0229] [Technical Idea 20] A display control device according to any one of Technical Ideas 1 to 19, wherein the image in front of the gate is a bird's-eye view image seen from a virtual viewpoint located directly above the vehicle or above the vehicle and a predetermined distance behind the vehicle, and the control unit is configured to: obtain a congestion level indicating the degree of traffic congestion around the gate based on a signal received by the communication circuit; and change the position of the virtual viewpoint according to the congestion level.

[0230] [Technical Idea 21] The display control device according to Technical Idea 20, wherein the control unit is configured to set the virtual viewpoint at a higher position when the congestion degree is equal to or greater than a predetermined value compared to when the congestion degree is less than the predetermined value.

[0231] [Technical Idea 22] The display control device according to Technical Idea 20 or 21, wherein the control unit is configured to set the virtual viewpoint to a position closer to directly above the vehicle when the congestion degree is equal to or greater than a predetermined value, compared to when the congestion degree is less than the predetermined value.

[0232] [Technical Idea A1] A display control device used in a vehicle equipped with multiple operating modes with different levels of automation of driving operations, comprising: a control unit (31) that controls the manner in which information indicating the vehicle's driving route is notified to occupants; and a communication circuit (33) that enables the control unit to communicate with other devices, wherein the control unit performs the following operations based on a signal received via the communication circuit: acquires related road information, which is information regarding roads around a gate through which the vehicle is scheduled to pass; acquires information indicating a current mode, which is the operating mode currently being applied; and changes the manner in which the notification of the vehicle's behavior regarding passing through the gate is performed based on the information in the current mode or the related road information.

[0233] [Technical Idea A2] A display control device used in a vehicle having multiple operating modes with different levels of automation of driving operations, comprising: a control unit (31) that controls notifications for the driver of the vehicle; and a communication circuit (34) for the control unit to communicate with other devices, wherein the control unit: acquires data indicating the relative position of the vehicle with respect to a gate through which the vehicle is scheduled to pass from the other device via the communication circuit; detects the presence of an emergency vehicle based on the signal received by the communication circuit; upon detecting the emergency vehicle, issues a notification related to the emergency vehicle to the driver; and, when the emergency vehicle is detected while the vehicle is traveling around the gate, changes the content of the notification depending on the relative position.

[0234] [Technical Idea A3] A display control device used in a vehicle equipped with multiple operating modes with different levels of automation of driving operations, comprising: a control unit (31) that controls the display of a route image, which is an image showing the driving route of the vehicle; and a communication circuit (34) that enables the control unit to communicate with other devices, wherein the route image is a bird's-eye view image seen from a virtual viewpoint located directly above the vehicle or above the vehicle and a predetermined distance behind the vehicle, and the control unit is configured to: acquire related road information, which is information about roads around a gate through which the vehicle is scheduled to pass, from the other device via the communication circuit; acquire a congestion level that indicates the degree of traffic congestion around the gate based on a signal received by the communication circuit; and change the position of the virtual viewpoint used to generate the route image according to the congestion level.

[0235] <Supplementary Note (2)> The various flowcharts shown in this disclosure are all examples, and the number of steps constituting the flowcharts and the execution order of the processes can be changed as appropriate. The controls shown in each flowchart may be combined / executed in parallel to the extent that there is no contradiction. Expressions such as acquisition, determination, detection, generation, and calculation may be used interchangeably. The acquisition of certain information by a certain device also includes the device generating the information based on a signal input from another device. The case where the vehicle enters a gate area may be replaced with the case where the remaining distance to the gate point is less than a predetermined value.

[0236] The apparatus, system, and method described herein may be implemented by a special-purpose computer including a processor programmed to execute one or more functions embodied in a computer program. The apparatus and method described herein may be implemented using dedicated hardware logic circuits. The apparatus and method described herein may be implemented by one or more special-purpose computers configured by combining a processor that executes a computer program with one or more hardware logic circuits. For example, some or all of the functions of the processor 31 may be implemented in hardware. Implementations of certain functions in hardware include implementations using one or more integrated circuits (ICs). The processor (computing core) may be a CPU, MPU, GPU, or data flow processor (DFP). Some or all of the functions of the processor 31 may be implemented using a system-on-chip (SoC), an integrated circuit (IC), or a field-programmable gate array (FPGA). The computer program may be stored as instructions executed by a computer on a computer-readable non-transitory tangible storage medium. The program storage medium may be a hard-disk drive (HDD), a solid-state drive (SSD), flash memory, etc. The scope of the present disclosure also includes forms such as a program for causing a computer to function as the processor 31 and a non-transitory tangible storage medium such as a semiconductor memory on which the program is stored.

Claims

1. A display control device used in a vehicle equipped with multiple operating modes with different levels of automation of driving operations, comprising: a control unit (31) that controls the display of a route image, which is an image showing the vehicle's driving route; and a communication circuit (34) that enables the control unit to communicate with other devices, wherein the control unit: acquires related road information, which is information about roads around a gate through which the vehicle is scheduled to pass, from the other devices via the communication circuit; acquires information indicating the automation level of the operating mode currently being applied; and changes the configuration of the pre-gate image, which is the route image when the vehicle is driving in the area on the entrance side of the gate, based on the automation level information or the related road information.

2. The display control device described in claim 1, wherein the related road information includes whether or not the area on the entrance side of the gate is a laneless section, which is a section without lane marks, and the control unit is configured to change the configuration of the image in front of the gate depending on whether or not the area on the entrance side of the gate is a laneless section.

3. The display control device of claim 1, wherein the related road information includes whether or not the area on the entrance side of the gate is a laneless section, which is a section without lane marks, and the control unit is configured to: if the area on the entrance side of the gate is not the laneless section, display an image including a trajectory image, which is an image element indicating the trajectory of the vehicle, and a lane mark image, which is an image element indicating lane marks, as the pre-gate image; and if the area on the entrance side of the gate is not the laneless section, display an image including a gate image, which is an image element indicating the gate, and the trajectory image, without including the lane mark image, as the pre-gate image.

4. The display control device described in claim 1, wherein the control unit is configured to include more information regarding the vehicle's trajectory in the gate-front image when the automation level is a predetermined value or higher than when the automation level is less than the predetermined value.

5. A display control device as described in claim 4, wherein the pre-gate image displayed when the automation level is less than the predetermined value is an image that does not include a post-gate track image, which is an image element that shows the trajectory of the vehicle after passing through the gate, and the pre-gate image displayed when the automation level is equal to or greater than the predetermined value is an image that includes the post-gate track image.

6. The display control device described in claim 1, wherein the control unit is configured to include more information about the gate in the pre-gate image when the automation level is less than a predetermined value than when the automation level is equal to or greater than the predetermined value.

7. The display control device described in claim 1, wherein the pre-gate image includes a gate image that is an image showing the gate, and the control unit is configured to display the gate image in a more prominent manner when the automation level is less than a predetermined value than when the automation level is equal to or greater than the predetermined value.

8. The display control device described in claim 1, wherein the control unit is configured to display a gate image, which is an image showing the gate, on the display when the automation level is equal to or greater than a predetermined value, and not to display the gate image on the display when the automation level is less than the predetermined value.

9. The control unit is configured to: obtain information indicating whether or not a gate through which the vehicle is to pass has been detected by an environmental sensor mounted on the vehicle based on a signal received by the communication circuit; display a gate image representing the gate in a predetermined basic manner if the automation level is equal to or greater than a predetermined value and the gate can be detected by the environmental sensor; not display the gate image or display it in a manner less noticeable than the basic manner if the automation level is equal to or greater than the predetermined value and the gate cannot be detected by the environmental sensor; and not display the gate image if the automation level is less than the predetermined value.

10. A display control device as described in claim 1, wherein the gate front image is a bird's-eye view image seen from a virtual viewpoint located directly above the vehicle or above the vehicle and a predetermined distance behind the vehicle.

11. The display control device described in claim 1, wherein the related road information includes the number of entry lanes, which is the total number of lanes existing on the entrance side of the gate area, and the number of exit lanes, which is the total number of lanes existing on the exit side of the gate area, and the control unit is configured to provide a more noticeable notification regarding passage through the gate when the difference between the number of entry lanes and the number of exit lanes is equal to or greater than a predetermined value, compared to when the difference between the number of entry lanes and the number of exit lanes is less than the predetermined value.

12. The display control device described in claim 1, wherein the multiple operating modes include an automated driving mode in which the system performs all driving tasks and a manual driving mode in which the driver performs all driving tasks, and the control unit is configured to determine whether the gate through which the vehicle is scheduled to pass is a manual settlement gate based on the signal received by the communication circuit, and if the gate through which the vehicle is scheduled to pass is a manual settlement gate and the vehicle is traveling in front of the gate in the automated driving mode, to execute processing to notify the driver that the vehicle is scheduled to pass through a manual settlement gate and that the driving mode will be switched to the manual driving mode when the vehicle stops at the gate.

13. The display control device described in claim 1, wherein the control unit is configured to: acquire, via the communication circuit, data indicating whether an automatic settlement function for automatically making payments for the use of toll roads is available in the vehicle; acquire information regarding the payment method for one or more of the gates ahead of the vehicle based on the signal received by the communication circuit; and, if the automatic settlement function is available and there are both a gate that does not support automatic settlement and a gate that supports automatic settlement ahead of the vehicle, execute a process to inquire of the driver whether to pass through the gate that supports automatic settlement.

14. The display control device according to claim 1, wherein the plurality of operating modes include a level 4 mode in which the system executes all driving tasks and the driver is allowed to sleep, and the control unit is configured to: while the operating mode is the level 4 mode, determine whether the driver is sleeping based on a signal received by the communication circuit; if the operating mode when the gate is passed is the level 4 mode and the driver is asleep, to withhold notification of information related to passing through the gate; and to implement the withheld notification when the driver wakes up.

15. A display control device as described in claim 1, wherein the plurality of operating modes include a speed management mode in which the vehicle's traveling speed is automatically adjusted to a set speed, and the control unit is configured to, when the operating mode is the speed management mode, display a set speed image which is an image element indicating the set speed, and when the set speed is automatically changed in accordance with traffic rules, in addition to displaying the set speed image, execute processing to notify the driver that the set speed will be changed.

16. The display control device according to claim 1, wherein the control unit is configured to: detect the presence of an emergency vehicle based on a signal received by the communication circuit; upon detecting the emergency vehicle, issue a notification related to the emergency vehicle to the driver; acquire data indicating the relative position of the vehicle with respect to the gate via the communication circuit; and, if the emergency vehicle is detected while the vehicle is traveling around the gate, change the content of the notification depending on the relative position.

17. The display control device described in claim 16, wherein the control unit is configured to, when the emergency vehicle is detected while the vehicle is traveling in the area on the entrance side of the gate, issue a notification to continue traveling toward the gate.

18. A display control device as described in claim 16 or 17, wherein the control unit is configured to, when the emergency vehicle is detected while the vehicle is passing through the gate, issue a notification that the vehicle will stop inside the gate.

19. The display control device according to claim 16, wherein the control unit is further configured to, when the emergency vehicle is detected, acquire the direction of the emergency vehicle based on the signal received by the communication circuit, and, when the emergency vehicle is detected while the vehicle is traveling in the area on the exit side of the gate and the direction of the emergency vehicle is identified, notify the presence of the emergency vehicle and its direction of presence.

20. The display control device described in claim 1, wherein the image in front of the gate is a bird's-eye view image seen from a virtual viewpoint located directly above the vehicle or above the vehicle and a predetermined distance behind the vehicle, and the control unit is configured to obtain a congestion level indicating the degree of traffic congestion around the gate based on the signal received by the communication circuit, and to change the position of the virtual viewpoint according to the congestion level.

21. A display control device as described in claim 20, wherein the control unit is configured to set the virtual viewpoint at a higher position when the congestion level is equal to or greater than a predetermined value compared to when the congestion level is less than the predetermined value.

22. The display control device described in claim 20, wherein the control unit is configured to set the virtual viewpoint to a position closer to directly above the vehicle when the congestion level is equal to or greater than a predetermined value, compared to when the congestion level is less than the predetermined value.

23. A display control method executed by a processor mounted on a vehicle equipped with multiple operating modes with different levels of automation of driving operations, the display control method including: acquiring related road information, which is information about roads near a gate through which the vehicle is scheduled to pass, from another device; acquiring information indicating a current mode, which is the operating mode currently being applied; and changing the configuration of a pre-gate image, which is an image showing the vehicle's driving route near the gate, based on the information about the current mode or the related road information.

Images