Vehicle control system, vehicle control method, and program

Abstract

A vehicle control system (11) comprises a human-machine interface (31). The human-machine interface (31) performs display control of entertainment content (CT) on the basis of a monitoring result of a passenger (PA).

Description

Technical Field

[0001] The present invention relates to a vehicle control system, a vehicle control method, and a program.

Background Art

[0002] Techniques for displaying information on a windshield or the like to assist a driver in driving are known. Recently, efforts related to autonomous driving have been progressing, and level 3 autonomous driving that releases a driver from driving obligations is also being realized.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] As autonomous driving progresses, the interior of a vehicle can be used as an entertainment space. A passenger can enjoy various entertainment contents presented inside the vehicle. However, in a conventional vehicle control system, no consideration has been given to the convenience of passengers for enjoying entertainment contents.

[0005] Therefore, the present disclosure proposes a vehicle control system, a vehicle control method, and a program capable of enhancing the convenience of passengers for enjoying entertainment contents.

Means for Solving the Problems

[0006] This disclosure provides a vehicle control system having a human-machine interface that controls the display of entertainment content based on passenger monitoring results. Furthermore, this disclosure provides a vehicle control method in which the information processing of the vehicle control system is performed by a computer, and a program that enables the computer to perform the information processing of the vehicle control system. [Brief explanation of the drawing]

[0007] [Figure 1] This diagram illustrates the overview of display control for entertainment content. [Figure 2] This is a block diagram showing an example configuration of a vehicle control system. [Figure 3] This figure shows an example of a sensing area by the external recognition sensor 25. [Figure 4] This diagram shows an example configuration of the in-vehicle sensor and video display unit. [Figure 5] This figure shows an example of how the video content provided will be displayed. [Figure 6] This figure shows another example of the configuration of the video display unit. [Figure 7] This figure shows an example of how the video content provided will be displayed. [Figure 8] This diagram lists the conditions inside and outside the vehicle that can be detected by the sensors. [Figure 9] This figure shows an example of a method for controlling the display of content. [Figure 10] This figure shows an example of the correspondence between the level of information transmission, the surrounding circumstances, and the method of display. [Figure 11] This flowchart shows an example of content display control based on monitoring results inside and outside the vehicle. [Figure 12] This diagram shows the details of the process. [Figure 13] This diagram shows the details of the process. [Figure 14] This diagram shows the details of the process. [Figure 15] This diagram shows the details of the process. [Figure 16] This is a diagram showing the details of the process. [Figure 17] This is a diagram showing an example of event notification. [Figure 18] This is a diagram showing an example of transparent display of an event. [Figure 19] This is a diagram showing an example of control of the content display position based on the brightness distribution. [Figure 20] This is a diagram showing an example of control of the content display position based on the brightness distribution. [Figure 21] This is a diagram showing an example of control of the event notification method based on the information transmission level. [Figure 22] This is a diagram showing an example of control of the content display position based on the event occurrence position. [Figure 23] This is a diagram showing an example of control of the content display position based on the viewing state of the passengers. [Figure 24] This is a diagram showing an example of control of the content display position based on the viewing state of the passengers. [Figure 25] This is a diagram showing an example of display control based on the viewing states of multiple passengers. [Figure 26] This is a diagram showing an example of display control based on the viewing states of multiple passengers. [Figure 27] This is a diagram showing an example of display control based on the viewing states of multiple passengers. [Figure 28] This is a diagram showing an example of display control based on the viewing states of multiple passengers.

Mode for Carrying Out the Invention

[0008] Hereinafter, embodiments of the present disclosure will be described in detail based on the drawings. In the following embodiments, the same parts are denoted by the same reference numerals, and redundant descriptions are omitted.

[0009] The description will be made in the following order. [1. Outline of Display Control of Entertainment Content] [2. Configuration Example of Vehicle Control System] [3. Sensing area of ​​external recognition sensor] [4. Example of configuration of in-vehicle sensors and video display unit] [5. Content display control based on monitoring results inside and outside the vehicle] [5-1. Subjects of Monitoring] [5-2. Content display control according to the level of information transmission] [5-3. Processing Flow] [6. Display Control Examples] [6-1. Event Notifications] [6-2. Event Transparency Display] [6-3. Controlling content display position based on brightness distribution] [6-4. Controlling event notification methods based on information transmission levels] [6-5. Controlling content display position based on event occurrence location] [6. Controlling the display position of content based on the passenger's viewing status] [6-7. Display control based on the viewing status of multiple passengers] [7. Effects]

[0010] [1. Overview of Display Control for Entertainment Content] Figure 1 is a diagram illustrating the overview of display control for entertainment content CT.

[0011] Vehicle 1 has a vehicle control system 11 capable of autonomous driving. Passengers (PA) can use the interior of the vehicle as an entertainment space. Vehicle 1 has a video display unit 93 capable of displaying entertainment content CT. Entertainment content CT is not intended to assist the driver, but rather to entertain passengers (PA). Examples of entertainment content CT include movies, music, games, news, email, maps / tourist information, social media, and chat.

[0012] The video display unit 93 can, for example, switch between a display state and a transparent state. The video display unit 93 is positioned along the inner wall or partition wall of the vehicle 1 so as to surround the interior space. In the display state, the video display unit 93 presents content CT, and in the transparent state, it allows the passenger PA to see outside the vehicle.

[0013] The "display state" includes a semi-transparent state in which information such as images displayed on the scenery outside the vehicle can be overlaid, similar to AR (Augmented Reality). The video display unit 93 can perform AR display by superimposing images onto the scenery outside the vehicle. The video display unit 93 adjusts, for example, the brightness of the display, the transparency of the video display unit 93, or the haze according to the brightness outside the vehicle. This improves the visibility of the content CT.

[0014] The vehicle control system 11 includes a monitor unit MU and a human-machine interface (HMI) 31. The monitor unit MU acquires various information inside and outside the vehicle using various sensors mounted on the vehicle 1. The monitor unit MU outputs the acquired information as monitor information to the HMI 31. The monitor unit MU is composed of, for example, a DMS 30 and a recognition unit 73, which will be described later.

[0015] The monitor information includes information regarding the passenger PA's viewing status (viewing status information VS), information regarding external events (event information ET), information regarding the surrounding environment (environmental information CE), and information regarding the driving status DS. Based on the monitor information, the HMI 31 controls the display of entertainment content CT. The monitor unit MU records, for example, the display position of the content CT and the passenger PA's gaze status towards the content CT as a record of the in-vehicle conditions during driving, and stores this information in the storage unit 28 (see Figure 2). Some parts of the vehicle control system 11, including the storage unit 28, may be implemented as functions on the cloud via the communication unit 22.

[0016] [2. Example of a vehicle control system configuration] Figure 2 is a block diagram showing an example configuration of the vehicle control system 11.

[0017] The vehicle control system 11 is installed in the vehicle 1 and performs processing related to driving assistance and autonomous driving of the vehicle 1.

[0018] The vehicle control system 11 includes a vehicle control ECU (Electronic Control Unit) 21, a communication unit 22, a map information storage unit 23, a location information acquisition unit 24, an external recognition sensor 25, an in-vehicle sensor 26, a vehicle sensor 27, a memory unit 28, a driving support / automatic driving control unit 29, a DMS (Driver Monitoring System) 30, an HMI (Human Machine Interface) 31, and a vehicle control unit 32.

[0019] The vehicle control ECU 21, communication unit 22, map information storage unit 23, location information acquisition unit 24, external recognition sensor 25, in-vehicle sensor 26, vehicle sensor 27, memory unit 28, driving support / autonomous driving control unit 29, driver monitoring system (DMS) 30, human-machine interface (HMI) 31, and vehicle control unit 32 are interconnected and can communicate with each other via a communication network 41. The communication network 41 consists of an in-vehicle communication network or bus that conforms to digital bidirectional communication standards such as CAN (Controller Area Network), LIN (Local Interconnect Network), LAN (Local Area Network), FlexRay (registered trademark), and Ethernet (registered trademark). The communication network 41 may be used differently depending on the type of data being transmitted. For example, CAN may be applied to data related to vehicle control, and Ethernet may be applied to large-capacity data. In addition, the various components of the vehicle control system 11 may be directly connected using wireless communication technologies intended for relatively short-range communication, such as Near Field Communication (NFC) or Bluetooth®, without going through the communication network 41.

[0020] In the following, when each part of the vehicle control system 11 communicates via the communication network 41, the description of the communication network 41 will be omitted. For example, when the vehicle control ECU 21 and the communication unit 22 communicate via the communication network 41, it will simply be described as the vehicle control ECU 21 and the communication unit 22 communicating.

[0021] The vehicle control ECU 21 is composed of various processors, such as a CPU (Central Processing Unit) and an MPU (Micro Processing Unit). The vehicle control ECU 21 controls the functions of the entire vehicle control system 11 or a part of it.

[0022] The communication unit 22 communicates with various devices inside and outside the vehicle, other vehicles, servers, base stations, etc., and transmits and receives various types of data. At this time, the communication unit 22 can communicate using multiple communication methods.

[0023] A brief explanation will be given regarding the external communication capabilities of the communication unit 22. The communication unit 22 communicates with servers located on an external network (hereinafter referred to as "external servers") via a base station or access point using wireless communication methods such as 5G (fifth-generation mobile communication system), LTE (Long Term Evolution), and DSRC (Dedicated Short Range Communications). The external network with which the communication unit 22 communicates is, for example, the internet, a cloud network, or a network specific to a carrier. The communication method used by the communication unit 22 to the external network is not particularly limited, as long as it is a wireless communication method that enables digital two-way communication at a predetermined communication speed and over a predetermined distance.

[0024] Furthermore, for example, the communication unit 22 can communicate with terminals located near the vehicle using P2P (Peer To Peer) technology. Terminals located near the vehicle include, for example, terminals worn by mobile bodies moving at relatively low speeds such as pedestrians and cyclists, terminals installed in fixed locations such as stores, or MTC (Machine Type Communication) terminals. In addition, the communication unit 22 can also perform V2X communication. V2X communication refers to communication between the vehicle and other vehicles, such as vehicle-to-vehicle communication with other vehicles, vehicle-to-infrastructure communication with roadside devices, etc., vehicle-to-home communication with homes, and vehicle-to-pedestrian communication with terminals carried by pedestrians, etc.

[0025] The communication unit 22 can, for example, receive programs from an external source (over the air) to update the software that controls the operation of the vehicle control system 11. The communication unit 22 can also receive map information, traffic information, information about the vehicle 1's surroundings, etc., from an external source. Furthermore, the communication unit 22 can transmit information about the vehicle 1 and information about the vehicle 1's surroundings to an external source. Information about the vehicle 1 that the communication unit 22 transmits to an external source includes, for example, data indicating the status of the vehicle 1 and recognition results from the recognition unit 73. Furthermore, the communication unit 22 can also perform communications corresponding to vehicle emergency notification systems such as e-Call.

[0026] For example, the communication unit 22 receives electromagnetic waves transmitted by road traffic information communication systems (VICS (Vehicle Information and Communication System) (registered trademark)) such as radio beacons, optical beacons, and FM multiplex broadcasting.

[0027] A brief overview of the communication capabilities of the communication unit 22 with the vehicle interior will be provided. The communication unit 22 can communicate with various devices in the vehicle, for example, using wireless communication. The communication unit 22 can communicate wirelessly with devices in the vehicle using communication methods that enable digital bidirectional communication at a predetermined or higher communication speed via wireless communication, such as Wi-Fi, Bluetooth, NFC, and WUSB (Wireless USB). Not limited to these, the communication unit 22 can also communicate with various devices in the vehicle using wired communication. For example, the communication unit 22 can communicate with various devices in the vehicle via wired communication through a cable connected to a connection terminal (not shown). The communication unit 22 can communicate with various devices in the vehicle using communication methods that enable digital bidirectional communication at a predetermined or higher communication speed via wired communication, such as USB (Universal Serial Bus), HDMI (High-Definition Multimedia Interface) (registered trademark), and MHL (Mobile High-definition Link).

[0028] Here, "devices inside the vehicle" refers to, for example, devices inside the vehicle that are not connected to the communication network 41. Examples of devices inside the vehicle include mobile devices and wearable devices carried by passengers such as the driver, and information devices that are brought into the vehicle and temporarily installed.

[0029] The map information storage unit 23 stores either or both maps acquired from external sources and maps created by the vehicle 1. For example, the map information storage unit 23 stores three-dimensional high-precision maps, global maps with lower precision than high-precision maps but covering a wide area, and so on.

[0030] High-precision maps include, for example, dynamic maps, point cloud maps, and vector maps. A dynamic map is, for example, a map consisting of four layers: dynamic information, semi-dynamic information, semi-static information, and static information, and is provided to vehicle 1 from an external server. A point cloud map is a map composed of point clouds (point cloud data). A vector map is, for example, a map that maps traffic information such as the location of lanes and traffic lights to a point cloud map, making it suitable for ADAS (Advanced Driver Assistance System) and AD (Autonomous Driving).

[0031] The point cloud map and vector map may be provided from, for example, an external server, or they may be created in the vehicle 1 as maps for matching with the local map described later, based on sensing results from the camera 51, radar 52, LiDAR 53, etc., and stored in the map information storage unit 23. In addition, if high-precision maps are provided from an external server, in order to reduce communication capacity, map data of, for example, several hundred meters square, relating to the planned route that the vehicle 1 will travel will be acquired from the external server.

[0032] The location information acquisition unit 24 receives GNSS (Global Navigation Satellite System) signals from GNSS satellites and acquires the location information of the vehicle 1. The acquired location information is supplied to the driving support / automatic driving control unit 29. The location information acquisition unit 24 is not limited to using GNSS signals; for example, it may acquire location information using beacons.

[0033] The external recognition sensor 25 is equipped with various sensors used to recognize the external conditions of the vehicle 1, and supplies sensor data from each sensor to various parts of the vehicle control system 11. The types and number of sensors equipped with the external recognition sensor 25 are arbitrary.

[0034] For example, the external recognition sensor 25 includes a camera 51, a radar 52, a LiDAR (Light Detection and Ranging, Laser Imaging Detection and Ranging) 53, and an ultrasonic sensor 54. However, the external recognition sensor 25 may also be configured to include one or more of the cameras 51, radar 52, LiDAR 53, and ultrasonic sensor 54. The number of cameras 51, radar 52, LiDAR 53, and ultrasonic sensor 54 is not particularly limited as long as it is a number that can be realistically installed in the vehicle 1. Furthermore, the types of sensors included in the external recognition sensor 25 are not limited to this example, and the external recognition sensor 25 may include other types of sensors. Examples of the sensing areas of each sensor included in the external recognition sensor 25 will be described later.

[0035] The shooting method of camera 51 is not particularly limited. For example, various types of cameras capable of distance measurement, such as ToF (Time Of Flight) cameras, stereo cameras, monocular cameras, and infrared cameras, can be applied to camera 51 as needed. However, camera 51 may also be used simply for acquiring images, regardless of distance measurement.

[0036] Furthermore, for example, the external recognition sensor 25 may include an environmental sensor for detecting the environment relative to the vehicle 1. The environmental sensor is a sensor for detecting the environment such as weather, climate, and brightness, and may include various sensors such as a raindrop sensor, fog sensor, sunshine sensor, snow sensor, and illuminance sensor.

[0037] Furthermore, for example, the external recognition sensor 25 includes a microphone used for detecting sounds around the vehicle 1 and the location of sound sources.

[0038] The in-vehicle sensor 26 is equipped with various sensors for detecting information inside the vehicle and supplies sensor data from each sensor to various parts of the vehicle control system 11. The types and number of sensors equipped with the in-vehicle sensor 26 are not particularly limited as long as they are types and numbers that can realistically be installed in the vehicle 1.

[0039] For example, the in-vehicle sensor 26 may include one or more sensors from among a camera, radar, seat sensor, steering wheel sensor, microphone, and biosensor. The camera included in the in-vehicle sensor 26 may be a camera with various distance-measuring imaging methods, such as a ToF camera, stereo camera, monocular camera, and infrared camera. However, it is not limited to these; the camera included in the in-vehicle sensor 26 may simply be for acquiring images, regardless of distance measurement. The biosensor included in the in-vehicle sensor 26 may be installed, for example, on a seat or steering wheel, to detect various biometric information of the driver or other passenger.

[0040] The vehicle sensor 27 is equipped with various sensors for detecting the state of the vehicle 1 and supplies sensor data from each sensor to various parts of the vehicle control system 11. The types and number of sensors equipped with the vehicle sensor 27 are not particularly limited as long as they are of a type and number that can be realistically installed on the vehicle 1.

[0041] For example, the vehicle sensor 27 includes a speed sensor, an acceleration sensor, an angular velocity sensor (gyro sensor), and an inertial measurement unit (IMU) that integrates them. For example, the vehicle sensor 27 includes a steering angle sensor for detecting the steering angle of the steering wheel, a yaw rate sensor, an accelerator sensor for detecting the amount of operation of the accelerator pedal, and a brake sensor for detecting the amount of operation of the brake pedal. For example, the vehicle sensor 27 includes a rotation sensor for detecting the rotation speed of the engine or motor, an air pressure sensor for detecting the air pressure of the tires, a slip ratio sensor for detecting the slip ratio of the tires, and a wheel speed sensor for detecting the rotation speed of the wheels. For example, the vehicle sensor 27 includes a battery sensor for detecting the remaining charge and temperature of the battery, and an impact sensor for detecting external impacts. The vehicle sensor 27 includes a sensor for detecting the driving status DS.

[0042] The storage unit 28 includes at least one of a non-volatile storage medium and a volatile storage medium, and stores data and programs. The storage unit 28 can be used as, for example, an EEPROM (Electrically Erasable Programmable Read Only Memory) and a RAM (Random Access Memory), and the storage medium can be a magnetic storage device such as an HDD (Hard Disk Drive), a semiconductor storage device, an optical storage device, or a magneto-optical storage device. The storage unit 28 stores various programs and data used by each part of the vehicle control system 11. For example, the storage unit 28 includes an EDR (Event Data Recorder) and a DSSAD (Data Storage System for Automated Driving), and stores information about the vehicle 1 before and after an event such as an accident, and information acquired by the in-vehicle sensors 26.

[0043] The driving assistance / automatic driving control unit 29 controls the driving assistance and automatic driving of the vehicle 1. For example, the driving assistance / automatic driving control unit 29 includes an analysis unit 61, an action planning unit 62, and an operation control unit 63.

[0044] The analysis unit 61 performs analysis processing on the vehicle 1 and its surroundings. The analysis unit 61 comprises a self-position estimation unit 71, a sensor fusion unit 72, and a recognition unit 73.

[0045] The self-position estimation unit 71 estimates the vehicle's position based on sensor data from the external recognition sensor 25 and a high-precision map stored in the map information storage unit 23. For example, the self-position estimation unit 71 generates a local map based on sensor data from the external recognition sensor 25 and estimates the vehicle's position by matching the local map with the high-precision map. The position of the vehicle 1 is based on, for example, the center of the rear wheel relative to the axle.

[0046] Local maps are, for example, three-dimensional high-precision maps created using technologies such as SLAM (Simultaneous Localization and Mapping), or occupancy grid maps. Three-dimensional high-precision maps are, for example, the point cloud maps mentioned above. Occupancy grid maps divide the three-dimensional or two-dimensional space around vehicle 1 into grids of a predetermined size and show the occupancy status of objects on a grid-by-grid basis. The occupancy status of objects is indicated, for example, by the presence or absence of an object or the probability of its existence. Local maps are also used, for example, in the detection and recognition processing of the external conditions of vehicle 1 by the recognition unit 73.

[0047] The self-position estimation unit 71 may estimate the vehicle 1's own position based on the position information acquired by the position information acquisition unit 24 and the sensor data from the vehicle sensor 27.

[0048] The sensor fusion unit 72 performs sensor fusion processing to obtain new information by combining multiple different types of sensor data (for example, image data supplied from the camera 51 and sensor data supplied from the radar 52). Methods for combining different types of sensor data include integration, fusion, and union.

[0049] The recognition unit 73 performs a detection process to detect the external conditions of the vehicle 1, and a recognition process to recognize the external conditions of the vehicle 1.

[0050] For example, the recognition unit 73 performs detection and recognition processing of the external conditions of the vehicle 1 based on information from the external recognition sensor 25, information from the self-position estimation unit 71, information from the sensor fusion unit 72, etc.

[0051] Specifically, for example, the recognition unit 73 performs detection and recognition processing of objects around the vehicle 1. Object detection processing includes, for example, detecting the presence, size, shape, position, and movement of objects. Object recognition processing includes, for example, recognizing attributes such as the type of object or identifying a specific object. However, detection processing and recognition processing are not necessarily clearly separated and may overlap.

[0052] For example, the recognition unit 73 detects objects around the vehicle 1 by performing clustering, which classifies the point cloud based on sensor data from the radar 52 or LiDAR 53 into clusters of points. This allows the presence, size, shape, and position of objects around the vehicle 1 to be detected.

[0053] For example, the recognition unit 73 detects the movement of objects around the vehicle 1 by performing tracking that follows the movement of clusters of points classified by clustering. This allows the velocity and direction of travel (movement vector) of objects around the vehicle 1 to be detected.

[0054] For example, the recognition unit 73 detects or recognizes vehicles, people, bicycles, obstacles, structures, roads, traffic lights, traffic signs, road markings, etc., based on image data supplied from the camera 51. The recognition unit 73 may also recognize the types of objects around the vehicle 1 by performing recognition processing such as semantic segmentation.

[0055] For example, the recognition unit 73 can perform recognition processing of traffic rules around the vehicle 1 based on the map stored in the map information storage unit 23, the self-position estimation result by the self-position estimation unit 71, and the recognition result of objects around the vehicle 1 by the recognition unit 73. Through this processing, the recognition unit 73 can recognize the location and status of traffic lights, the content of traffic signs and road markings, the content of traffic regulations, and the lanes that can be driven on.

[0056] For example, the recognition unit 73 can perform recognition processing of the environment surrounding the vehicle 1. The surrounding environment that the recognition unit 73 is intended to recognize may include the presence or absence of pedestrians and surrounding vehicles, weather, temperature, humidity, brightness, and road surface conditions.

[0057] The action planning unit 62 creates an action plan for vehicle 1. For example, the action planning unit 62 creates an action plan by performing route planning and route following processes.

[0058] Global path planning is the process of planning the general route from the start to the finish line. This path planning also includes a process called local path planning, which involves generating a track that allows vehicle 1 to move safely and smoothly in its vicinity, taking into account the vehicle's motion characteristics along the planned route.

[0059] Route following is the process of planning actions to safely and accurately travel along the route planned by the route planner within the planned time. The action planning unit 62 can, for example, calculate the target speed and target angular velocity of vehicle 1 based on the results of this route following process.

[0060] The motion control unit 63 controls the operation of the vehicle 1 in order to realize the action plan created by the action planning unit 62.

[0061] For example, the motion control unit 63 controls the steering control unit 81, brake control unit 82, and drive control unit 83, which are included in the vehicle control unit 32 described later, to perform acceleration / deceleration control and direction control so that the vehicle 1 moves along the trajectory calculated by the trajectory plan. For example, the motion control unit 63 performs coordinated control for the purpose of realizing ADAS functions such as collision avoidance or impact mitigation, follow driving, vehicle speed maintenance driving, collision warning of the vehicle, and lane departure warning of the vehicle. For example, the motion control unit 63 performs coordinated control for the purpose of autonomous driving, such as driving autonomously without driver operation.

[0062] The DMS30 performs driver authentication and driver status recognition based on sensor data from the in-vehicle sensors 26 and input data input to the HMI31, which will be described later. The driver status to be recognized may include, for example, physical condition, level of alertness, level of concentration, level of fatigue, gaze direction, level of intoxication, driving operation, and posture.

[0063] Furthermore, the DMS30 may perform authentication processing for passengers other than the driver and recognition processing for the status of said passengers. Also, for example, the DMS30 may perform recognition processing of the conditions inside the vehicle based on sensor data from the in-vehicle sensor 26. Examples of conditions inside the vehicle to be recognized include temperature, humidity, brightness, and odor.

[0064] The HMI31 handles the input of various data and instructions, and presents this data to passengers, including the driver.

[0065] A brief explanation of data input by HMI31 is provided. HMI31 is equipped with an input device for human data input. HMI31 generates input signals based on data and instructions input by the input device and supplies them to each part of the vehicle control system 11. HMI31 is equipped with operators such as a touch panel, buttons, switches, and levers as input devices. However, HMI31 may further be equipped with input devices that allow information to be input by methods other than manual operation, such as voice or gestures. Furthermore, HMI31 may use external connected devices such as a remote control device using infrared or radio waves, or a mobile device or wearable device that corresponds to the operation of the vehicle control system 11, as input devices.

[0066] This section provides a brief overview of how HMI31 presents data. HMI31 generates visual, auditory, and tactile information for the occupant or those outside the vehicle. HMI31 also performs output control, managing the output, content, timing, and method of each generated piece of information. As visual information, HMI31 generates and outputs information indicated by images and light, such as operation screens, vehicle status displays, warning displays, monitor images showing the surroundings of vehicle 1, and entertainment content CTs. HMI31 also generates and outputs auditory information, such as voice guidance, warning sounds, and warning messages. Furthermore, HMI31 generates and outputs tactile information, such as information that is perceived by the occupant's sense of touch through force, vibration, and movement.

[0067] As output devices for visual information output by HMI31, for example, a display device that presents visual information by displaying images itself, or a projector device that presents visual information by projecting images, can be applied. In addition to display devices with ordinary displays, the display device may also be a device that displays visual information within the passenger's field of view, such as a head-up display, a transparent display, or a wearable device with AR (Augmented Reality) functionality. Furthermore, HMI31 can also use display devices such as navigation devices, instrument panels, CMS (Camera Monitoring System), electronic mirrors, and lamps installed in the vehicle 1 as output devices for visual information output.

[0068] For HMI31, output devices that output auditory information can include, for example, audio speakers, headphones, and earphones.

[0069] As an output device for HMI31 to output tactile information, for example, a haptic element using haptic technology can be applied. The haptic element is installed in parts of the vehicle 1 that are in contact with by the occupant, such as the steering wheel and the seat.

[0070] The vehicle control unit 32 controls various parts of the vehicle 1. The vehicle control unit 32 includes a steering control unit 81, a brake control unit 82, a drive control unit 83, a body system control unit 84, a light control unit 85, and a horn control unit 86.

[0071] The steering control unit 81 detects and controls the state of the steering system of the vehicle 1. The steering system includes, for example, a steering mechanism with a steering wheel, an electric power steering system, etc. The steering control unit 81 includes, for example, a steering ECU that controls the steering system, an actuator that drives the steering system, etc.

[0072] The brake control unit 82 detects and controls the state of the brake system of the vehicle 1. The brake system includes, for example, a brake mechanism including a brake pedal, an ABS (Antilock Brake System), a regenerative braking mechanism, etc. The brake control unit 82 includes, for example, a brake ECU that controls the brake system, an actuator that drives the brake system, etc.

[0073] The drive control unit 83 detects and controls the state of the vehicle 1's drive system. The drive system includes, for example, an accelerator pedal, a drive force generating device for generating driving force such as an internal combustion engine or drive motor, and a drive force transmission mechanism for transmitting driving force to the wheels. The drive control unit 83 also includes, for example, a drive ECU for controlling the drive system and actuators for driving the drive system.

[0074] The body system control unit 84 detects and controls the state of the body system of the vehicle 1. The body system includes, for example, a keyless entry system, a smart key system, power window devices, power seats, an air conditioning system, airbags, seat belts, a shift lever, etc. The body system control unit 84 also includes, for example, a body system ECU that controls the body system, actuators that drive the body system, etc.

[0075] The light control unit 85 detects and controls the status of various lights on the vehicle 1. Examples of lights to be controlled include headlights, taillights, fog lights, turn signals, brake lights, projection lights, bumper displays, etc. The light control unit 85 includes a light ECU for controlling the lights, actuators for driving the lights, etc.

[0076] The horn control unit 86 detects and controls the state of the vehicle's car horn. The horn control unit 86 includes, for example, a horn ECU for controlling the car horn, an actuator for driving the car horn, and so on.

[0077] [3. Sensing area of ​​external recognition sensor] Figure 3 shows an example of the sensing area of ​​the external recognition sensor 25, including the camera 51, radar 52, LiDAR 53, and ultrasonic sensor 54. Figure 3 schematically shows the vehicle 1 as viewed from above. When Figure 3 is viewed in an orientation that displays the symbols in the correct direction, the left end is the front end of the vehicle 1, and the right end is the rear end of the vehicle 1.

[0078] Sensing regions 101F and 101B show examples of sensing regions of the ultrasonic sensor 54. Sensing region 101F covers the area around the front end of the vehicle 1 by multiple ultrasonic sensors 54. Sensing region 101B covers the area around the rear end of the vehicle 1 by multiple ultrasonic sensors 54.

[0079] The sensing results in sensing area 101F and sensing area 101B are used, for example, for parking assistance of vehicle 1.

[0080] Sensing areas 102F to 102B show examples of sensing areas for short-range or medium-range radar 52. Sensing area 102F covers a position further in front of vehicle 1 than sensing area 101F. Sensing area 102B covers a position further in rear of vehicle 1 than sensing area 101B. Sensing area 102L covers the rear periphery of the left side of vehicle 1. Sensing area 102R covers the rear periphery of the right side of vehicle 1.

[0081] The sensing results in sensing region 102F are used, for example, to detect vehicles or pedestrians in front of vehicle 1. The sensing results in sensing region 102B are used, for example, to prevent collisions behind vehicle 1. The sensing results in sensing regions 102L and 102R are used, for example, to detect objects in blind spots to the sides of vehicle 1.

[0082] Sensing areas 103F to 103B show examples of sensing areas by camera 51. Sensing area 103F covers a position further in front of vehicle 1 than sensing area 102F. Sensing area 103B covers a position further in rear of vehicle 1 than sensing area 102B. Sensing area 103L covers the periphery of the left side of vehicle 1. Sensing area 103R covers the periphery of the right side of vehicle 1.

[0083] The sensing results in sensing region 103F can be used, for example, for recognition of traffic lights and traffic signs, lane departure prevention support systems, and automatic headlight control systems. The sensing results in sensing region 103B can be used, for example, for parking assistance and surround view systems. The sensing results in sensing regions 103L and 103R can be used, for example, for surround view systems.

[0084] Sensing area 104 shows an example of the sensing area of ​​LiDAR 53. Sensing area 104 covers a position further in front of vehicle 1 than sensing area 103F. On the other hand, sensing area 104 has a narrower range in the left-right direction than sensing area 103F.

[0085] The sensing results in the sensing region 104 can be used, for example, to detect objects such as surrounding vehicles.

[0086] Sensing area 105 shows an example of the sensing area of ​​the long-range radar 52. Sensing area 105 covers a position further in front of vehicle 1 than sensing area 104. On the other hand, sensing area 105 has a narrower range in the left-right direction than sensing area 104.

[0087] The sensing results in sensing area 105 are used, for example, for ACC (Adaptive Cruise Control), emergency braking, collision avoidance, etc.

[0088] Furthermore, the sensing areas of the camera 51, radar 52, LiDAR 53, and ultrasonic sensor 54 included in the external recognition sensor 25 may take various configurations other than those shown in Figure 3. Specifically, the ultrasonic sensor 54 may be configured to sense the sides of the vehicle 1, or the LiDAR 53 may be configured to sense the rear of the vehicle 1. Also, the installation positions of each sensor are not limited to the examples described above. In addition, there may be one or more sensors.

[0089] [4. Example of configuration of in-vehicle sensors and video display unit] Figure 4 shows an example configuration of the in-vehicle sensor 26 and the video display unit 93. Figure 5 shows an example display of the video IM provided as content CT.

[0090] In the examples shown in Figures 4 and 5, the video display unit 93 is positioned to partition the space between the front seats (driver's seat and passenger seat) and the rear seats. A camera CM is installed inside the vehicle to monitor the passengers PA in the rear seats. The camera CM functions as an in-vehicle sensor 26. In the example shown in Figure 4, two camera CMs monitor the entire interior space behind the front seats (rear space).

[0091] The DMS30 acquires viewing status information VS of the passengers PA in the rear space based on the video footage captured by the camera CM. The viewing status information VS includes information such as the number of passengers PA, whether the passengers PA are awake, the passengers PA's field of view, and whether the passengers PA are looking at the content CT. If there are multiple passengers PA, the field of view, whether the passengers PA are awake, and whether the passengers PA are looking at the content CT are determined for each passenger PA. The HMI31 controls the display of the content CT based on the viewing status information VS.

[0092] The video display unit 93 is an optical see-through or video see-through display capable of displaying a transparent background. In this disclosure, as an example, an optical see-through display is used as the video display unit 93. Known transparent displays or technologies such as HUD (Head-Up Display), transparent OLED (Organic Light Emitting Diode), and transparent LED (Light Emitting Diode) can be used as the video display unit 93. In this disclosure, for example, a configuration is adopted in which the video is projected onto a transparent screen SCR as the video display unit 93.

[0093] The video display unit 93 includes, for example, a transparent screen SCR and a projector PJ. The transparent screen SCR has an optical lens film that reflects light projected from the projector PJ. A dimming film capable of electrically controlling the transmittance of ambient light is laminated onto the optical lens film. The HMI 31 adjusts the brightness of the display, or the transparency or haze of the video display unit 93 (transparent screen SCR), based on ambient light information extracted from the environmental information CE. When no content CT is displayed on the video display unit 93, the passenger PA can see the scenery outside the vehicle through the transparent screen SCR from the gap between the driver's seat and the passenger seat.

[0094] Figure 6 shows another example of the configuration of the video display unit 93. Figure 7 shows an example of the display of the video IM provided as content CT.

[0095] In the examples shown in Figures 6 and 7, the video display unit 93 is installed to the side of the rear-row passenger PA. The side windows of the vehicle 1 are replaced by transparent screens SCR. Instead of using the side windows as transparent screens SCR, a film-like transparent screen may be attached to the surface of the side windows or sandwiched inside the side windows. When the video IM is not displayed, the passenger PA can see the scenery outside the vehicle through the transparent screen SCR.

[0096] In the examples in Figures 6 and 7, the rear side windows have the function of a transparent screen (SCR), but the function of a transparent screen (SCR) is not limited to the rear side windows. The windshield, front side windows, or rear window may also have the function of a transparent screen (SCR).

[0097] [5. Content display control based on monitoring results inside and outside the vehicle] [5-1. Subjects of Monitoring] Figure 8 is a diagram listing the conditions inside and outside the vehicle that can be detected by the sensors.

[0098] How to spend time inside the train includes: A1: watching movies, listening to music, playing games; A2: reading news and checking emails; A3: watching VR videos; A4: looking at maps and tourist information; A5: using voice chat; A6: looking at the scenery outside; A7: talking with other passengers; A8: making phone calls on smartphones; and A9: sleeping or just relaxing.

[0099] A1-A5 represent the state of "watching something alone," A4-A7 represent the state of "watching together," A6-A7 represent the state of "not watching anything in particular," A8 represents the state of "using a handheld device," and A9 represents the state of "sleeping." "Watching something alone" and "watching together" represent the state of using the video display unit 93, while "not watching anything in particular," "using a handheld device," and "sleeping" represent the state of not using the video display unit 93. The DMS30 detects the viewing state to determine how the passenger PA is using the video display unit 93.

[0100] Vehicle 1 can be in one of three states: "Manual Driving," "Automatic Driving," or "Fallback." "Automatic Driving" or "Manual Driving" means the vehicle is being driven automatically or manually. "Fallback" means that control is being transferred from automatic driving to manual driving, or a transfer is being requested. "Automatic Driving" is a state where the driver can freely enjoy content CT, while "Manual Driving" and "Fallback" are states where the provision of content CT should be withheld to allow the driver to concentrate on driving. "Fallback" is also a state where it is necessary to inform passengers PA other than the driver that something abnormal has happened to Vehicle 1. The vehicle sensor 27 detects which of the three states Vehicle 1 is currently in as the driving status DS.

[0101] External environmental conditions include "dangers in the surroundings," "sunlight from the setting sun," and "sunny weather." "Dangers in the surroundings" indicates a situation where there is an event that should be notified to the passenger PA, while "sunlight from the setting sun" and "sunny weather" indicate situations where brightness control according to the surrounding environment is required. The recognition unit 73 recognizes the external environment based on the sensor information from the external recognition sensor 25.

[0102] The vehicle control system 11 appropriately controls the display of content CT according to the viewing status of the passenger PA, the driving status DS, external events, and the surrounding environment.

[0103] [5-2. Content display control according to the level of information transmission] Figure 9 shows an example of a method for controlling the display of content CT.

[0104] The HMI31 controls the display of entertainment content CT based on the monitoring results of the passenger PA. For example, the HMI31 has an information transmission method control unit 91, a display method control unit 92, and a video display unit 93.

[0105] The information transmission method control unit 91 determines the information transmission level of an event occurring outside the vehicle based on the monitoring information acquired from the monitor unit MU. The information transmission level indicates the urgency or importance of transmitting information about the event occurring outside the vehicle. Based on the information transmission level, the information transmission method control unit 91 determines the notification method for the event. The determined notification method is the general method of transmitting information about the event. The display method control unit 92 determines the detailed display method (display position and display timing, etc.) of various information, including content CT, based on the monitoring information, in accordance with the notification method.

[0106] Figure 10 shows an example of the correspondence between the level of information transmission, the surrounding circumstances, and the method of display.

[0107] In the example shown in Figure 10, five ranks are defined as information transmission levels, from rank S to rank D. Rank S corresponds to a situation where an accident has occurred ahead. In rank S, viewing of the entertainment content CT displayed in front of the vehicle is stopped. The view of the area outside the vehicle in front is secured, and warning information is displayed in the display area in front.

[0108] Rank A corresponds to a situation where an accident has occurred to the left or right of Vehicle 1. In Rank A, the necessary external view of the side (the side where the accident occurred) is ensured. Entertainment content (CT) that was displayed in the display area on the left or right side where the accident occurred is moved to the display area on the side where the accident did not occur. Warning information is then displayed in the display area on the side where the accident occurred.

[0109] Rank B addresses situations where other vehicles are present at close range. In Rank B, the external field of view is ensured in the necessary direction (the side where the other vehicle is located). Entertainment content (CT) is moved to a non-obtrusive position (for example, a position where it does not overlap with the other vehicle). Warning information is then displayed in the display area on the side where the other vehicle is located.

[0110] Rank C is for situations where other vehicles are present at a distance. In Rank C, a field of view outside the vehicle is secured in the necessary direction (the side where the other vehicle is located). Because the risk of collision with other vehicles is low, the size of the field of view secured is smaller than in Ranks S, A, and B, and no warning information is displayed.

[0111] Rank D corresponds to a situation where there are no other cars around. In Rank D, there are no events that anticipate dangers such as collisions. Therefore, viewing of entertainment content (CT) continues.

[0112] [5-3. Processing Flow] Figure 11 is a flowchart showing an example of content CT display control based on in-vehicle and out-of-vehicle monitoring results. Figures 12 to 16 show the details of the process.

[0113] As shown in Figure 11, the monitoring unit MU acquires information about the interior of the vehicle, information about the surrounding area outside the vehicle, and the driving status DS based on sensor information (steps SA1 to SA3). The monitoring unit MU supplies the acquired information to the HMI31 as monitoring information.

[0114] Information inside the vehicle is acquired, for example, according to the flow shown in Figure 12. First, the monitoring unit MU confirms the number of passengers based on the images captured by the camera CM (step SB1). The monitoring unit MU checks the activity level of each passenger PA (step SB2) and determines whether everyone is asleep or if at least one person is awake. If at least one person is awake, the monitoring unit MU checks the activities of each awake passenger PA (whether or not they are watching content CT) (step SB3).

[0115] Information about the surroundings outside the vehicle is acquired, for example, according to the flow shown in Figure 13. First, the monitoring unit MU checks the weather information based on the sensor information acquired from the external recognition sensor 25 and determines whether visibility is good or not (step SC1). If visibility is good, the monitoring unit MU acquires the direction of the sun using the self-position information and time information detected using SLAM (step SC2). The monitoring unit MU also acquires surrounding accident information and information about surrounding vehicles based on the sensor information acquired from the external recognition sensor 25 (steps SC3-SC4).

[0116] Returning to Figure 11, the information transmission method control unit 91 detects the surrounding conditions of vehicle 1 based on the monitor information. The storage unit 28 stores a table in which definition information for each situation is specified. The table specifies the correspondence between the information transmission level, the surrounding conditions, and the display method as definition information. Based on the table, the information transmission method control unit 91 defines an information transmission level appropriate to the situation (step SA4). The information transmission method control unit 91 selects an information transmission level according to the surrounding conditions and determines the event notification method based on the selected information transmission level (steps SA5-SA6).

[0117] The information transmission level is determined, for example, according to the flow shown in Figure 14. First, the information transmission method control unit 91 acquires accident information and information about surrounding vehicles from the monitor unit MU (steps SD1 to SD2). Based on the acquired information, the information transmission method control unit 91 determines the urgency of information transmission regarding an event that occurred outside the vehicle (step SD3).

[0118] If it is determined that there is an urgency for information transmission, the information transmission method control unit 91 determines the direction in which information regarding the event needs to be displayed based on the location where the event (for example, a threat to vehicle 1) occurred (step SD4). If the threat is ahead, the information transmission method control unit 91 selects rank S as the information transmission level. If the threat is not ahead, the information transmission method control unit 91 selects rank A as the information transmission level.

[0119] If it is determined that there is no urgency for information transmission, the information transmission method control unit 91 determines the degree of danger posed by surrounding vehicles (step SD5). For example, if there is another vehicle nearby, it determines the direction in which information about the other vehicle needs to be displayed (step SD6). If the other vehicle is to the left or right, the information transmission method control unit 91 selects rank B as the information transmission level. If the other vehicle is behind, the information transmission method control unit 91 selects rank C as the information transmission level. If there is no other vehicle nearby, the information transmission method control unit 91 selects rank D as the information transmission level.

[0120] Returning to Figure 11, the display method control unit 92 detects the passenger PA's viewing status information VS based on the monitor information (steps SA7 to SA8). The viewing status information VS includes information about the passenger PA's field of view (field of view information) and information about the passenger PA's state (status information). The status information includes information about whether the passenger PA is awake or not, and information about whether the passenger PA is looking at the content CT or not.

[0121] The viewing status of passenger PAs is detected, for example, according to the flow shown in Figure 15. First, the display method control unit 92 selects all passenger PAs detected based on the monitor information as detection targets (step SE1). The display method control unit 92 defines the status of each passenger PA based on the monitor information and determines whether or not each passenger PA is looking at the content CT (step SE2). For example, if the gaze is directed towards the display area, it is determined that the passenger is looking at the content CT, and if the gaze is not directed towards the display area, it is determined that the passenger is not looking at the content CT.

[0122] The display method control unit 92 determines that all passengers PA who are looking at the content CT will be detected (step SE3). Based on the monitor information, the display method control unit 92 acquires the field of view information of each determined passenger PA (step SE4).

[0123] The display method control unit 92 determines the display area of ​​the content CT based on the passenger PA's field of view information (step SA9). The display method control unit 92 determines the display timing of event information and content CT based on the passenger PA's status information (step SA10). The display method is determined by the display area and display timing. The display method control unit 92 displays the content CT based on the determined display method (step SA11).

[0124] The display method is determined, for example, according to the flow shown in Figure 16. First, the display method control unit 92 calculates the display area where the brightness distribution exceeds the acceptable standard due to ambient light (e.g., sunlight). The acceptable standard is set in advance based on the visibility of the content CT, etc. If the visibility decreases beyond the acceptable limit due to the brightness distribution, it is determined that the acceptable standard has been exceeded. For example, the display method control unit 92 calculates the display area where the sun is reflected based on the direction of the sun. The display method control unit 92 recognizes the calculated display area as the area affected by ambient light (step SF1).

[0125] The display method control unit 92 calculates information about the common field of view of multiple passengers (common field of view information) if there are multiple passengers (PAs) who are looking at the content CT (step SF2). The common field of view refers to the area on the transparent screen SCR where the fields of view of multiple passengers overlap.

[0126] If a common field of view exists, the display method control unit 92 switches the display area to match the common field of view (step SF3). For example, the display method control unit 92 sets the display area of ​​the content CT to an area that is not affected by ambient light and includes the common field of view. If there is no common field of view, the display area is switched to match the field of view of a specific passenger PA with a high degree of attention (step SF4). For example, the display method control unit 92 sets the display area of ​​the content CT to an area that is not affected by ambient light and includes the field of view of a specific passenger PA. Then, the display method control unit 92 displays the content CT according to the defined rank (step SF5).

[0127] [6. Display Control Examples] [6-1. Event Notifications] Below, an example of display control by HMI31 will be explained using Figures 17 to 28. Figure 17 shows an example of event notification.

[0128] When an event occurs outside the vehicle that should notify the occupant PA, the HMI31 displays an event notification (NT) in the display area that overlaps with the occupant PA's field of view (VA) at the same time the event occurs outside the vehicle. Events that should notify the occupant PA include accidents that threaten the safety of the vehicle and the presence of other vehicles engaging in dangerous driving.

[0129] The HMI31 notifies the user of an event NT, for example, by displaying event information ET on the video display unit 93. The event information ET includes information such as the type of event and the location where the event occurred. In the example in Figure 17, when another vehicle approaches the user's vehicle to a predetermined distance, the message "Emergency" is displayed in the lower left of the passenger PA's field of view VA, and an arrow indicating the direction in which the other vehicle is approaching is displayed in the lower right of the field of view VA. "Emergency" indicates the type of event, and the arrow indicates the location where the event occurred.

[0130] Event notifications (NT) are displayed in the portion of the video display unit 93's entire display area (displayable area DA) that overlaps with the passenger's (PA) field of view (VA). Therefore, the passenger (PA) can acquire event information (ET) while viewing the content (CT) without moving their field of view (VA).

[0131] [6-2. Event Transparency Display] Figure 18 shows an example of transparent display of an event.

[0132] The HMI31 makes the display area that overlaps with the passenger's (PA) field of view (VA) transparent when an event occurring outside the vehicle is confirmed within the passenger's (PA) field of view (VA). With this configuration, the status of the event can be viewed through the transparent display area (transparent area TA) without moving the field of view (VA).

[0133] For example, HMI31 moves the display position of the content CT that was displayed in the display area overlapping with the field of view VA, or stops displaying the display area overlapping with the field of view VA. As a result, HMI31 makes the display area overlapping with the field of view VA transparent.

[0134] In the example shown in Figure 18, the display area of ​​the content CT (content display area CDA) has moved from the lower right of the screen, where it overlaps with the field of view VA, to the upper left of the screen, where it does not overlap with the field of view VA. The lower right area of ​​the screen where the content CT was displayed becomes a transparent area TA. The passenger PA can visually observe the situation of events outside the vehicle through the transparent area TA. With this configuration, the continuity of the content CT before and after displaying an event is maintained. Therefore, the passenger PA viewing the content CT is less likely to feel any discomfort. In other words, the passenger PA can visually observe events outside the vehicle and understand the situation, so if they determine that there are no major problems, they can continue to view the content CT. The vehicle control system 11 may also have a means for the user to notify the HMI 31 of their determination of whether there are any problems.

[0135] [6-3. Controlling content display position based on brightness distribution] Figures 19 and 20 show examples of content display position control based on brightness distribution.

[0136] When ambient light causes a brightness distribution in the content display area CDA that exceeds the acceptable standard, the HMI31 moves the display position of the content CT to another display area where the brightness distribution meets the acceptable standard. For example, the HMI31 determines that the brightness distribution exceeds the acceptable standard when the ratio of the brightness of the brightest part to the darkest part within the content display area CDA is greater than a preset threshold.

[0137] In the example shown in Figure 19, the reflection of the sun's SN onto the content display area CDA results in a brightness distribution in the content display area CDA that exceeds the acceptable limit. Therefore, the HMI31 moves the content display area CDA to avoid the reflection of the sun's SN. This suppresses the reduction in the visibility of the content CT due to ambient light.

[0138] When HMI31 moves the display position of the content CT to another display area, it displays guidance information GI to guide the passenger PA's gaze to the other display area. In the example in Figure 19, the guidance information FI is displayed as an arrow indicating the destination of the content display area CDA, along with the message "Display position will be changed" in the lower left corner of the content display area CDA. The guidance information GI allows the passenger PA to know in advance that the content display area CDA will move and in the direction of its movement. Therefore, the display position of the content CT can be moved without causing any discomfort to the passenger PA.

[0139] In the example shown in Figure 20, direct sunlight shines onto a portion of the content display area CDA, resulting in a brightness distribution in the CDA that exceeds the acceptable limit. If the area DL directly exposed to sunlight is significantly brighter than the other areas (the shaded areas), adjusting the transparency and haze of the transparent screen SCR will not sufficiently resolve the brightness distribution. Therefore, after notifying the user via guidance information GI, the HMI31 moves the content display area CDA to a position that avoids the directly exposed area DL.

[0140] Figures 19 and 20 show examples of brightness distribution in the content display area CDA caused by sunlight, but brightness distribution can also be caused by external light other than sunlight. For example, streetlights or city lights reflected in nightscapes can cause brightness distribution in the content display area CDA. In this case as well, the HMI31 moves the content display area CDA to an area with less brightness distribution.

[0141] [6-4. Controlling event notification methods based on information transmission levels] Figure 21 shows an example of how to control event notification methods based on information transmission levels. The passenger PA will use the video display unit 93 located in the center of the vehicle, as shown in Figure 4.

[0142] The HMI31 determines how to notify an event based on the urgency or importance of communicating information about an event occurring outside the vehicle. For example, in the upper example of Figure 21, there are no other vehicles (OT) around the vehicle (OW). Therefore, the HMI31 determines that no event has occurred that should notify the passenger (PA) and does not notify the passenger.

[0143] In the example in the middle of Figure 21, another vehicle (OT) is located a short distance to the right rear of the own vehicle (OW). HMI31 determines that a low-priority event has occurred to the right rear of the own vehicle (OW). HMI31 illuminates the right edge of the content CT in accordance with the direction of the event. This triggers event notification (NT).

[0144] The level of urgency is represented by the intensity of the light. The HMI31 changes the intensity of the light according to the distance between its own OW and the other vehicle's OT. The shorter the distance, the higher the urgency and the greater the light intensity. In the example in the middle of Figure 21, the distance between the own OW and the other vehicle's OT is relatively large. Therefore, the intensity of the light illuminating the right edge of the content CT is relatively low.

[0145] In the lower example of Figure 21, another vehicle (OT) is driving close to the right side of the local vehicle (OW). HMI31 determines that a high-priority event has occurred to the right of the local vehicle (OW). HMI31 notifies the event (NT) by brightly illuminating the right edge of the content (CT).

[0146] [6-5. Controlling content display position based on event occurrence location] Figure 22 shows an example of controlling the content display position based on the location where an event occurs. The passenger PA will use the video display unit 93 in the center of the vehicle as shown in Figure 4.

[0147] HMI31 controls the display position of content CT based on the event occurrence location. For example, in the upper example of Figure 22, there are no other vehicles OT around the own vehicle OW. Therefore, HMI31 determines that no event has occurred that should notify the passenger PA, and does not change the display position of content CT.

[0148] In the example in the middle of Figure 22, an accident (event) has occurred in front of the vehicle OW. HMI31 reduces the display size of the content CT so that the forward view is maintained. HMI31 positions the reduced content CT in a location that does not overlap with the passenger PA's line of sight (for example, at the bottom edge of the transparent screen SCR) so that it does not obstruct the passenger PA's view of the accident site AS.

[0149] In the lower example of Figure 22, the accident occurs to the right of the vehicle's OW. HMI31 reduces the display size of the content CT so that the right-side field of view is maintained. HMI31 positions the reduced content CT in a location that does not overlap with the passenger PA's line of sight (for example, at the left edge of the transparent screen SCR) so as not to obstruct the passenger PA's view of the accident site AS.

[0150] [6. Controlling the display position of content based on the passenger's viewing status] Figures 23 and 24 show examples of controlling the content display position based on the viewing status of the passenger PA. The passenger PA is assumed to be using the video display unit 93 on the side of the vehicle as shown in Figure 6.

[0151] In the example shown in Figure 23, the passenger PA is leaning back to view the content CT. Based on the viewing status information VS, the HMI 31 detects that the passenger PA is intently watching the content CT. The HMI 31 displays the content CT across the entire transparent screen SCR so that the passenger PA can view the content CT on a large screen.

[0152] In the example in Figure 24, the passenger PA is leaning close to the transparent screen SCR and looking at the scenery GA outside the vehicle. Based on the viewing state information VS, the HMI 31 detects that the passenger PA is trying to look at the scenery GA outside the vehicle. The HMI 31 reduces the display size of the content CT so that the lateral view is ensured. The HMI 31 positions the reduced content CT in a location that does not overlap with the passenger PA's line of sight (for example, at the corner of the transparent screen SCR) so as not to obstruct the passenger PA's view of the scenery GA.

[0153] [6-7. Display control based on the viewing status of multiple passengers] Figures 25 to 28 illustrate examples of display control based on the viewing status of multiple passenger PAs. In Figures 25 to 28, individual passenger PAs are distinguished by numbers appended to their symbols, as needed. The method for distinguishing the field of view (VA) of individual passenger PAs is similar.

[0154] Based on viewing status information VS, HMI31 detects the field of view VA of one or more passengers PA who are fixated on the content CT. For each detected field of view VA, HMI31 displays the content CT in the display area that overlaps with the field of view VA.

[0155] In the examples in Figures 25 and 26, passenger PA-2 is looking at the scenery outside the vehicle. Therefore, HMI31 selectively displays content CT in the display area that overlaps with passenger PA-1's field of view VA-1. HMI31 makes the display area that overlaps with passenger PA-2's field of view VA-2 transparent so that passenger PA-2 can see outside the vehicle.

[0156] Based on in-vehicle monitoring results, the HMI31 detects that an occupant PA has started driving and stops displaying the entertainment content CT that the occupant PA is watching. This configuration allows the occupant PA to concentrate on driving, thereby increasing driving safety. Manual driving is detected based on the driving status DS. Which occupant PA has started driving is identified by image analysis of footage from cameras installed in the vehicle.

[0157] Figures 27 and 28 show an example of display control based on the common field of view (CVA) of multiple passenger PAs.

[0158] When a common field of view (CVA) exists in the field of view (VA) of multiple passengers (PA), HMI31 controls the display of event information (ET), etc., based on the location of the common field of view (CVA).

[0159] For example, in the example shown in Figure 27, multiple passengers (PAs) are viewing the same content (CT). A common field of view (CVA) exists on the content (CT). When the approach of another vehicle (OT) is detected (event), the HMI 31 sends a notification (NT) to the display area that overlaps with the common field of view (CVA) to inform the user of the approach of the other vehicle (OT). When the other vehicle (OT) reaches a position where it overlaps with the common field of view (CVA), the HMI 31 moves the content display area (CDA) to a position where it does not overlap with the common field of view (CVA), and makes the display area that overlaps with the common field of view (CVA) transparent.

[0160] In the example shown in Figure 28, the content display area CDA is moved to avoid reflections from the sun (SN). When HMI31 moves the display position of the content CT to another display area, it displays guidance information GI in the display area that overlaps with the common field of view CVA. This allows all passengers PA viewing the common content CT to recognize that the content display area CDA is moving and the direction of its movement.

[0161] [7. Effects] The vehicle control system 11 includes an HMI 31. The HMI 31 controls the display of entertainment content CT based on the monitoring results of the passenger PA. In the vehicle control method of this disclosure, the processing of the vehicle control system 11 is executed by a computer. The program of this disclosure enables the computer to implement the processing of the vehicle control system 11.

[0162] This configuration allows for display control that takes into account the viewing status of the passenger's PA (Personal Audio) system. Therefore, it enhances the convenience of the passenger's PA system for enjoying content CT (Computer Television).

[0163] The content CT is displayed by the video display unit 93. The video display unit 93 can switch between a display state and a transparent state. The HMI 31 adjusts the brightness of the display, or the transparency or haze of the content CT display area (transparent screen SCR), based on the brightness information from outside the vehicle.

[0164] This configuration allows the video display unit 93 to have both a content display function and a window function. By superimposing the video IM onto the scenery outside the vehicle, AR display is also possible. Furthermore, by adjusting the brightness of the display, the transparency of the display area of ​​the content CT, or the haze according to the brightness outside the vehicle, the visibility of the content CT is improved.

[0165] HMI31 determines how to notify an event based on the urgency or importance of communicating information about an event that occurred outside the vehicle.

[0166] This configuration allows users to enjoy content CT while simultaneously being aware of events occurring outside the vehicle, as well as the urgency or importance of those events.

[0167] The vehicle control system 11 includes a monitor unit MU. The monitor unit MU records the display position of the content CT and the gaze status of the passenger PA towards the content CT.

[0168] This configuration makes it easier to determine the cause of an accident if one occurs.

[0169] Furthermore, the effects described herein are merely illustrative and not limiting, and other effects may also occur.

[0170] [Note] Furthermore, this technology can also be configured as follows. (1) A vehicle control system with a human-machine interface that controls the display of entertainment content based on passenger monitoring results. (2) The human-machine interface determines how to notify an event based on the urgency or importance of communicating information about the event that occurred outside the vehicle. The vehicle control system described in (1) above. (3) The aforementioned human-machine interface notifies the passenger of an event in a display area that overlaps with the passenger's field of vision, in accordance with the timing of the event occurring outside the vehicle. The vehicle control system described in (1) or (2) above. (4) The aforementioned human-machine interface makes the display area that overlaps with the passenger's field of view transparent in accordance with the timing at which an event occurring outside the vehicle is observed within the passenger's field of view. A vehicle control system as described in any one of (1) to (3) above. (5) The human-machine interface makes the display area transparent by moving the display position of the content that was displayed in the display area, or by stopping the display of the display area. The vehicle control system described in (4) above. (6) The human-machine interface, when ambient light causes a brightness distribution in the content's display area that exceeds an acceptable standard, moves the display position of the content to another display area where the brightness distribution meets the acceptable standard. A vehicle control system as described in any one of (1) through (5) above. (7) The human-machine interface displays guidance information in the other display area to guide the passenger's gaze when moving the display position of the content to the other display area. The vehicle control system described in (6) above. (8) The aforementioned human-machine interface stops displaying the content that the passenger who has started driving is looking at. A vehicle control system as described in any one of (1) through (7) above. (9) The system includes a monitor unit that records the display position of the content and the passenger's gaze on the content. A vehicle control system as described in any one of (1) through (8) above. (10) The aforementioned human-machine interface adjusts the brightness of the display, or the transparency or haze of the display area of ​​the content, based on the brightness information outside the vehicle. A vehicle control system as described in any one of (1) through (9) above. (11) A computer-based vehicle control method that includes controlling the display of entertainment content based on passenger monitoring results. (12) A program that enables a computer to control the display of entertainment content based on passenger monitoring results. [Explanation of Symbols]

[0171] 11. Vehicle control system 31. HMI (Human-Machine Interface) 93 Video display unit CT Content GI guidance information MU Monitoring Department PA passengers VA field of view

Claims

1. It has a human-machine interface that controls the display of entertainment content based on passenger monitoring results. The human-machine interface determines how to notify an event based on the urgency or importance of communicating information about the event that occurred outside the vehicle. Vehicle control system.

2. The aforementioned human-machine interface notifies the passenger of an event in a display area that overlaps with the passenger's field of vision, in accordance with the timing of the event occurring outside the vehicle. The vehicle control system according to claim 1.

3. The aforementioned human-machine interface makes the display area that overlaps with the passenger's field of view transparent in accordance with the timing at which an event occurring outside the vehicle is observed within the passenger's field of view. The vehicle control system according to claim 1.

4. The human-machine interface makes the display area transparent by moving the display position of the content that was displayed in the display area, or by stopping the display of the display area. The vehicle control system according to claim 3.

5. The human-machine interface, when ambient light causes a brightness distribution in the content's display area that exceeds an acceptable standard, moves the display position of the content to another display area where the brightness distribution meets the acceptable standard. The vehicle control system according to claim 1.

6. The human-machine interface displays guidance information in the other display area to guide the passenger's gaze when moving the display position of the content to the other display area. The vehicle control system according to claim 5.

7. The aforementioned human-machine interface stops displaying the content that the passenger who has started driving is looking at. The vehicle control system according to claim 1.

8. The system includes a monitor unit that records the display position of the content and the passenger's gaze on the content. The vehicle control system according to claim 1.

9. The aforementioned human-machine interface adjusts the brightness of the display, or the transparency or haze of the display area of ​​the content, based on the brightness information outside the vehicle. The vehicle control system according to claim 1.

10. Based on passenger monitoring results, the display of entertainment content is controlled. The method of notifying an event occurs outside the vehicle is determined based on the urgency or importance of communicating information about the event. A vehicle control method performed by a computer, which includes the ability to control a vehicle.

11. Based on passenger monitoring results, the display of entertainment content is controlled. The method of notifying an event occurs outside the vehicle is determined based on the urgency or importance of communicating information about the event. A program that allows a computer to accomplish something.

Images