Mobile device, information processing device, and information processing method

Abstract

To improve convenience of a passenger of a mobile apparatus.SOLUTION: A mobile apparatus comprises: a metadata giving unit that gives metadata including data associated with a situation within a room to the video data captured while moving; and an image processing unit that edits the video data and generates edited moving image data, on the basis of the metadata. The present technology can be applied to a vehicle for example.SELECTED DRAWING: Figure 37

Description

Technical Field

[0001] The present technology relates to a mobile device, an information processing device, and an information processing method, and particularly relates to a mobile device, an information processing device, and an information processing method that can improve the convenience of passengers in a mobile device.

Background Art

[0002] In recent years, the introduction of AR (Augmented Reality) technology into vehicles has been progressing (see, for example, Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] Thus, it is desired to improve the convenience of passengers such as vehicle drivers by using AR technology or the like.

[0005] The present technology has been made in view of such a situation, and aims to improve the convenience of passengers in a mobile device such as a vehicle.

Means for Solving the Problems

[0006] A mobile device according to a first aspect of the present technology includes a metadata adding unit that adds metadata including data related to the indoor situation to captured video data captured while moving, and an image processing unit that edits the captured video data based on the metadata to generate edited video data.

[0007] The first aspect of the information processing method of this technology includes a mobile device adding metadata, which includes data about the conditions inside a room, to video data captured while the mobile device is moving, and editing the video data based on the metadata to generate edited video data.

[0008] The second aspect of this technology is a mobile device that includes a metadata addition unit that adds metadata, including data about the conditions inside the room, to video data captured while moving.

[0009] The second aspect of this technology involves an information processing method in which a mobile device adds metadata, including data about the conditions inside a room, to video data captured while the device is moving.

[0010] The third aspect of this technology is an information processing device which includes an image processing unit that edits the captured video data and generates edited video data based on metadata, which includes data relating to the conditions inside the room of the mobile device, that is attached to the captured video data captured while the mobile device is moving.

[0011] The third aspect of this technology is an information processing method which includes an information processing device editing the captured video data, which is attached to the captured video data taken while the mobile device is moving, and generating edited video data.

[0012] In the first aspect of this technology, metadata including data about the indoor conditions is added to video data captured while moving, and the video data is edited based on the metadata to generate edited video data.

[0013] In the second aspect of this technology, metadata including data about the indoor environment is added to the video data captured while moving.

[0014] In a third aspect of the present technology, based on metadata including data regarding the indoor situation of the mobile device, which is attached to the captured video data captured while the mobile device is moving, the captured video data is edited to generate edited video data.

Brief Description of the Drawings

[0015] [Figure 1] It is a block diagram showing a configuration example of a vehicle control system. [Figure 2] It is a diagram showing an example of a sensing area. [Figure 3] It is a diagram showing an example of the installation positions of a camera, LiDAR, and radar. [Figure 4] It is a schematic diagram showing the front inside the vehicle. [Figure 5] It is a schematic diagram of the driver's seat and the passenger seat area seen from above. [Figure 6] It is a schematic diagram of the passenger seat seen from the left side. [Figure 7] It is a schematic diagram of the driver's seat and the passenger seat area seen from the left rear. [Figure 8] It is a schematic diagram of the front of the vehicle seen from the driver's seat. [Figure 9] It is a schematic diagram of the passenger seat seen from the left rear. [Figure 10] It is a diagram showing an example of the display content in the center of the center display. [Figure 11] It is a diagram showing an example of the display content in the center of the center display. [Figure 12] It is a diagram showing an example of the display content in the center of the center display. [Figure 13] It is a diagram showing an example of the display content at the left end of the center display. [Figure 14] It is a diagram showing an example of the display content of the head-up display. [Figure 15] It is a diagram showing an example of the display content of the head-up display. [Figure 16] It is a diagram showing an example of the display content of the digital rearview mirror. [Figure 17]It is a diagram showing an example of the display content of the steering wheel display. [Figure 18] It is a diagram showing an example of the display content of the steering wheel display. [Figure 19] It is a diagram showing an example of the display content of the rear entertainment display. [Figure 20] It is a front view of the vehicle and an enlarged view near the logo. [Figure 21] It is a left side view of the vehicle. [Figure 22] It is an enlarged view of the tip of the left side of the vehicle. [Figure 23] It is a rear view of the vehicle. [Figure 24] It is a diagram of the area near the left headlight of the vehicle. [Figure 25] It is a diagram of the area near the left headlight of the vehicle. [Figure 26] It is a diagram of the area near the left headlight of the vehicle. [Figure 27] It is a schematic diagram of the interior of the vehicle as seen from the right. [Figure 28] It is a schematic diagram of the area near the driver's seat and the front passenger seat of the vehicle. [Figure 29] It is a schematic diagram of the area near the dashboard of the vehicle. [Figure 30] It is an enlarged view of the steering wheel of the vehicle. [Figure 31] It is a view of the driver's seat as seen from the left rear diagonally. [Figure 32] It is a diagram showing an example of the installation position of the ToF camera for driver photography. [Figure 33] It is a diagram showing an example of the installation position of the ToF camera for driver photography. [Figure 34] It is a schematic diagram showing an example of an image captured by the ToF camera installed at the installation positions in FIGS. 32 and 33. [Figure 35] It is a schematic diagram showing an example of an image captured by the ToF camera installed near the steering wheel. [Figure 36] It is a diagram showing an example of the installation position of the ToF camera for driver photography. [Figure 37] This is a block diagram showing an example of the configuration of the information processing unit. [Figure 38] This diagram shows an example of the light emission pattern when the power is turned on. [Figure 39] This diagram shows an example of the light emission pattern when the power is turned on. [Figure 40] This figure shows an example of the light emission pattern when the vehicle is in motion and the headlights are off. [Figure 41] This figure shows an example of the light emission pattern when the vehicle is in motion and the headlights are on. [Figure 42] This figure shows an example of the light emission pattern when the brakes are applied. [Figure 43] This diagram shows an example of the light emission pattern when the turn signal is activated. [Figure 44] This diagram shows an example of the light emission pattern when the door is open. [Figure 45] This diagram shows an example of the light emission pattern when the door is closed. [Figure 46] This figure shows an example of the light emission pattern when the car is parked. [Figure 47] This figure shows an example of the light emission pattern when the power is off. [Figure 48] This is a flowchart to explain the image processing. [Figure 49] This video shows an example of the object detection mode setting. [Figure 50] This shows an example of a video when the prediction mode is set. [Figure 51] This video shows an example of the search mode setting. [Figure 52] This video shows an example of the search mode setting. [Figure 53] This is a flowchart to explain the editing process. [Figure 54] This is a schematic diagram showing the interior of the vehicle viewed from the right. [Figure 55] This is a schematic diagram showing the front of the vehicle interior. [Figure 56]This is a schematic diagram showing the area in front of the driver's seat inside a vehicle. [Figure 57] This is a close-up view of the area around the steering wheel. [Figure 58] This figure shows an example of the installation location of a ToF camera in a camera module. [Figure 59] This figure shows an example of the shooting range of a ToF camera. [Figure 60] This figure shows other examples of ToF camera installation locations. [Figure 61] This figure shows other examples of ToF camera installation locations. [Figure 62] This figure shows other examples of ToF camera installation locations. [Figure 63] This figure shows other examples of ToF camera installation locations. [Figure 64] This diagram shows examples of camera module installation locations. [Figure 65] This is a diagram showing an example of the internal configuration of a camera module. [Figure 66] This diagram shows variations in the camera's installation position. [Figure 67] This diagram shows an example of the camera's field of view. [Figure 68] This figure shows an example of the display section for the CMS under normal conditions. [Figure 69] This figure shows a first example of the display on the CMS display unit when a vehicle in an adjacent lane approaches. [Figure 70] This figure shows an example of the display on the CMS (Control Monitor System) when the turn signal is activated. [Figure 71] This figure shows a second example of the display on the CMS display unit when a vehicle in an adjacent lane is approaching. [Figure 72] This figure shows an example of the display on the CMS (Control Monitor System) when the turn signal is in operation. [Figure 73] This figure shows an example of the required field of view. [Figure 74] This diagram illustrates an example of changing the display range of a CMS display unit based on the shift position. [Figure 75]This diagram illustrates how to set the display range for the display section of a CMS. [Figure 76] This diagram illustrates how to set the display range for the display section of a CMS. [Figure 77] This figure shows examples of zoom ratios and display ranges for the display section of a CMS. [Figure 78] This diagram illustrates how to control the display range when looking into the display unit for the CMS. [Figure 79] This diagram illustrates how to control the display range when looking into the display unit for the CMS. [Figure 80] This diagram illustrates how to control the display range when the shift position is set to reverse and the display unit for the CMS is viewed. [Figure 81] This diagram illustrates how to control the display range when the shift position is set to reverse and the display unit for the CMS is viewed. [Figure 82] This diagram illustrates how to control the display range when the shift position is set to reverse and the display unit for the CMS is viewed. [Figure 83] This diagram illustrates how to control the footlights when looking down at your feet. [Figure 84] This diagram illustrates the control method for the CMS display unit when the door opener is touched. [Figure 85] This diagram illustrates the control method for the CMS display unit when the door opener is touched. [Figure 86] This diagram illustrates the control method for the CMS display unit when the door opener is touched. [Figure 87] This figure shows the illumination status of the driver's seat display and steering wheel before the driver becomes aware of them. [Figure 88] This figure shows the illumination status of the driver's seat display and steering wheel when the driver is aware of the vehicle. [Figure 89]This figure shows the illumination status of the driver's seat display and steering wheel when the autonomous driving system is ready. [Figure 90] This diagram shows the illumination status of the driver's seat display and steering wheel when autonomous driving is initiated. [Figure 91] This diagram shows the illumination status of the driver's seat display and steering wheel during lane changes while the vehicle is in autonomous driving mode. [Figure 92] This figure shows the illumination status of the driver's seat display and steering wheel when a hazard is detected. [Figure 93] This is a diagram illustrating an example of guidance display during button operation. [Figure 94] This is a diagram illustrating an example of guidance display during button operation. [Figure 95] This figure shows an example of the display on the driver's seat display unit during lip reading. [Figure 96] This diagram illustrates how to control the warning sound that alerts the driver in front of the vehicle. [Figure 97] This diagram illustrates how to control the sound effects that occur when autonomous driving starts. [Figure 98] This diagram illustrates how to control sound effects during acceleration in autonomous driving. [Figure 99] This diagram illustrates how to control sound effects during deceleration in autonomous driving. [Figure 100] This diagram illustrates how to control sound effects before voice messages are output. [Figure 101] This diagram illustrates how to control the warning sound for vehicles approaching from the left rear. [Figure 102] This diagram illustrates how to control the warning sound for vehicles approaching from the left rear. [Figure 103] This diagram illustrates how to control the warning sound for vehicles approaching from the left rear. [Figure 104] This diagram illustrates how to control the warning sound for vehicles approaching from the left rear. [Figure 105] This is a diagram illustrating how to control audio for passengers in the rear seats. [Modes for carrying out the invention]

[0016] The following describes the configurations for implementing this technology. The explanation will proceed in the following order. 1. Example of a vehicle control system configuration 2. First Embodiment 3. Modified form of the first embodiment 4. Second Embodiment 5. Modified form of the second embodiment 6. Third Embodiment 7. Modified form of the third embodiment 8. Other

[0017] <<1. Example of Vehicle Control System Configuration>> Figure 1 is a block diagram showing an example configuration of a vehicle control system 11, which is an example of a mobile device control system to which this technology is applied.

[0018] The vehicle control system 11 is installed in the vehicle 1 and performs processing related to driving assistance and autonomous driving of the vehicle 1.

[0019] The vehicle control system 11 includes a processor 21, a communication unit 22, a map information storage unit 23, a GNSS (Global Navigation Satellite System) receiver 24, an external recognition sensor 25, an in-vehicle sensor 26, a vehicle sensor 27, a recording unit 28, a driving assistance / automatic driving control unit 29, a DMS (Driver Monitoring System) 30, an HMI (Human Machine Interface) 31, and a vehicle control unit 32.

[0020] The processor 21, communication unit 22, map information storage unit 23, GNSS receiver unit 24, external recognition sensor 25, in-vehicle sensor 26, vehicle sensor 27, recording unit 28, driving assistance / autonomous driving control unit 29, driver monitoring system (DMS) 30, human-machine interface (HMI) 31, and vehicle control unit 32 are interconnected via a communication network 41. The communication network 41 consists of an in-vehicle communication network or bus conforming to any standard such as CAN (Controller Area Network), LIN (Local Interconnect Network), LAN (Local Area Network), FlexRay (registered trademark), or Ethernet (registered trademark). In addition, each part of the vehicle control system 11 may be directly connected without going through the communication network 41, for example, by near-field communication (NFC (Near Field Communication)) or Bluetooth (registered trademark).

[0021] In the following, when each part of the vehicle control system 11 communicates via the communication network 41, the description of the communication network 41 will be omitted. For example, when the processor 21 and the communication unit 22 communicate via the communication network 41, it will simply be described as the processor 21 and the communication unit 22 communicating.

[0022] The processor 21 is composed of various processors, such as a CPU (Central Processing Unit), an MPU (Micro Processing Unit), and an ECU (Electronic Control Unit). The processor 21 controls the entire vehicle control system 11.

[0023] The communication unit 22 communicates with various devices inside and outside the vehicle, other vehicles, servers, base stations, etc., and sends and receives various types of data. For example, when communicating with the outside, the communication unit 22 receives from the outside programs for updating the software that controls the operation of the vehicle control system 11, map information, traffic information, and information about the area around vehicle 1. For example, the communication unit 22 transmits to the outside information about vehicle 1 (for example, data indicating the status of vehicle 1, recognition results by the recognition unit 73, etc.) and information about the area around vehicle 1. For example, the communication unit 22 performs communication corresponding to vehicle emergency notification systems such as e-Call.

[0024] The communication method of the communication unit 22 is not particularly limited. Furthermore, multiple communication methods may be used.

[0025] For communication with the vehicle interior, for example, the communication unit 22 wirelessly communicates with in-vehicle equipment using communication methods such as Wi-Fi, Bluetooth, NFC, and WUSB (Wireless USB). For example, the communication unit 22 also communicates with in-vehicle equipment via wired connections (and cables if necessary) using communication methods such as USB (Universal Serial Bus), HDMI (High-Definition Multimedia Interface, registered trademark), or MHL (Mobile High-definition Link).

[0026] Here, "devices inside the vehicle" refers to devices that are not connected to the communication network 41 inside the vehicle. For example, this could include mobile devices and wearable devices carried by passengers such as the driver, or information devices brought into the vehicle and temporarily installed.

[0027] For example, the communication unit 22 communicates with servers and the like located on an external network (e.g., the Internet, a cloud network, or a carrier-specific network) via a base station or access point using wireless communication methods such as 4G (fourth-generation mobile communication system), 5G (fifth-generation mobile communication system), LTE (Long Term Evolution), and DSRC (Dedicated Short Range Communications).

[0028] For example, the communication unit 22 uses P2P (Peer To Peer) technology to communicate with terminals located near the vehicle (for example, terminals of pedestrians or shops, or MTC (Machine Type Communication) terminals). For example, the communication unit 22 performs V2X communication. V2X communication includes, for example, vehicle-to-vehicle communication with other vehicles, vehicle-to-infrastructure communication with roadside devices, etc., vehicle-to-home communication with homes, and vehicle-to-pedestrian communication with terminals carried by pedestrians, etc.

[0029] For example, the communication unit 22 receives electromagnetic waves transmitted by road traffic information communication systems (VICS (Vehicle Information and Communication System), registered trademark), such as radio beacons, optical beacons, and FM multiplex broadcasting.

[0030] The map information storage unit 23 stores maps acquired from external sources and maps created by the vehicle 1. For example, the map information storage unit 23 stores 3D high-precision maps, global maps with lower precision than high-precision maps but covering a wide area, and so on.

[0031] High-precision maps include, for example, dynamic maps, point cloud maps, and vector maps (also called ADAS (Advanced Driver Assistance System) maps). A dynamic map is, for example, a map consisting of four layers: dynamic information, semi-dynamic information, semi-static information, and static information, and is provided from an external server. A point cloud map is a map composed of point clouds (point cloud data). A vector map is a map that maps information such as the location of lanes and traffic lights to a point cloud map. Point cloud maps and vector maps may be provided from an external server, for example, or they may be created in vehicle 1 as maps for matching with the local map described later, based on sensing results from radar 52, LiDAR 53, etc., and stored in the map information storage unit 23. Furthermore, when high-precision maps are provided from an external server, in order to reduce communication capacity, map data of, for example, several hundred meters square, relating to the planned route that vehicle 1 will travel is acquired from the server.

[0032] The GNSS receiver 24 receives GNSS signals from GNSS satellites and supplies them to the driving assistance / automatic driving control unit 29.

[0033] The external recognition sensor 25 is equipped with various sensors used to recognize the external conditions of the vehicle 1, and supplies sensor data from each sensor to various parts of the vehicle control system 11. The types and number of sensors equipped with the external recognition sensor 25 are arbitrary.

[0034] For example, the external recognition sensor 25 includes a camera 51, a radar 52, a LiDAR (Light Detection and Ranging, Laser Imaging Detection and Ranging) 53, and an ultrasonic sensor 54. The number of cameras 51, radar 52, LiDAR 53, and ultrasonic sensors 54 is arbitrary, and examples of the sensing areas of each sensor will be described later.

[0035] Furthermore, camera 51 may be any type of camera with an arbitrary imaging method, such as a ToF (Time Of Flight) camera, stereo camera, monocular camera, or infrared camera, as needed.

[0036] Furthermore, for example, the external recognition sensor 25 includes environmental sensors for detecting weather, climate, brightness, etc. Environmental sensors include, for example, raindrop sensors, fog sensors, sunlight sensors, snow sensors, illuminance sensors, etc.

[0037] Furthermore, for example, the external recognition sensor 25 includes a microphone used for detecting sounds around the vehicle 1 and the location of sound sources.

[0038] The in-vehicle sensor 26 is equipped with various sensors for detecting information inside the vehicle and supplies sensor data from each sensor to various parts of the vehicle control system 11. The types and number of sensors equipped in the in-vehicle sensor 26 are arbitrary.

[0039] For example, the in-vehicle sensor 26 includes a camera, radar, seat sensor, steering wheel sensor, microphone, biosensor, etc. The camera can be any type of camera, such as a ToF camera, stereo camera, monocular camera, or infrared camera. The biosensor is installed, for example, on the seat or steering wheel, to detect various biometric information of the occupant, such as the driver.

[0040] The vehicle sensor 27 is equipped with various sensors for detecting the state of the vehicle 1 and supplies sensor data from each sensor to various parts of the vehicle control system 11. The types and number of sensors equipped in the vehicle sensor 27 are arbitrary.

[0041] For example, the vehicle sensor 27 includes a speed sensor, an acceleration sensor, an angular velocity sensor (gyro sensor), and an inertial measurement unit (IMU). For example, the vehicle sensor 27 includes a steering angle sensor for detecting the steering angle of the steering wheel, a yaw rate sensor, an accelerator sensor for detecting the amount of operation of the accelerator pedal, and a brake sensor for detecting the amount of operation of the brake pedal. For example, the vehicle sensor 27 includes a rotation sensor for detecting the rotation speed of the engine or motor, an air pressure sensor for detecting the air pressure of the tires, a slip ratio sensor for detecting the slip ratio of the tires, and a wheel speed sensor for detecting the rotation speed of the wheels. For example, the vehicle sensor 27 includes a battery sensor for detecting the remaining charge and temperature of the battery, and an impact sensor for detecting external impacts.

[0042] The recording unit 28 includes, for example, magnetic storage devices such as ROM (Read Only Memory), RAM (Random Access Memory), and HDD (Hard Disk Drive), semiconductor storage devices, optical storage devices, and magneto-optical storage devices. The recording unit 28 records various programs and data used by each part of the vehicle control system 11. For example, the recording unit 28 records a rosbag file containing messages sent and received by ROS (Robot Operating System) on which application programs related to autonomous driving operate. For example, the recording unit 28 includes an EDR (Event Data Recorder) and a DSSAD (Data Storage System for Automated Driving) to record information about vehicle 1 before and after an event such as an accident.

[0043] The driving assistance / automatic driving control unit 29 controls the driving assistance and automatic driving of the vehicle 1. For example, the driving assistance / automatic driving control unit 29 includes an analysis unit 61, an action planning unit 62, and an operation control unit 63.

[0044] The analysis unit 61 performs analysis processing on the vehicle 1 and its surroundings. The analysis unit 61 comprises a self-position estimation unit 71, a sensor fusion unit 72, and a recognition unit 73.

[0045] The self-position estimation unit 71 estimates the vehicle's position based on sensor data from the external recognition sensor 25 and a high-precision map stored in the map information storage unit 23. For example, the self-position estimation unit 71 generates a local map based on sensor data from the external recognition sensor 25 and estimates the vehicle's position by matching the local map with the high-precision map. The position of the vehicle 1 is based on, for example, the center of the rear wheel relative to the axle.

[0046] Local maps are, for example, three-dimensional high-precision maps created using technologies such as SLAM (Simultaneous Localization and Mapping), or occupancy grid maps. Three-dimensional high-precision maps are, for example, the point cloud maps mentioned above. Occupancy grid maps divide the three-dimensional or two-dimensional space around vehicle 1 into grids of a predetermined size and show the occupancy status of objects on a grid-by-grid basis. The occupancy status of objects is indicated, for example, by the presence or absence of an object or the probability of its existence. Local maps are also used, for example, in the detection and recognition processing of the external conditions of vehicle 1 by the recognition unit 73.

[0047] The self-position estimation unit 71 may estimate the vehicle 1's own position based on the GNSS signal and sensor data from the vehicle sensor 27.

[0048] The sensor fusion unit 72 performs sensor fusion processing to obtain new information by combining multiple different types of sensor data (for example, image data supplied from the camera 51 and sensor data supplied from the radar 52). Methods for combining different types of sensor data include integration, fusion, and union.

[0049] The recognition unit 73 performs detection and recognition processing of the external conditions of the vehicle 1.

[0050] For example, the recognition unit 73 performs detection and recognition processing of the external conditions of the vehicle 1 based on information from the external recognition sensor 25, information from the self-position estimation unit 71, information from the sensor fusion unit 72, etc.

[0051] Specifically, for example, the recognition unit 73 performs detection and recognition processing of objects around the vehicle 1. Object detection processing includes, for example, detecting the presence, size, shape, position, and movement of objects. Object recognition processing includes, for example, recognizing attributes such as the type of object or identifying a specific object. However, detection processing and recognition processing are not necessarily clearly separated and may overlap.

[0052] For example, the recognition unit 73 detects objects around the vehicle 1 by performing clustering, which classifies point clouds based on sensor data such as LiDAR or radar into clusters of points. This allows the presence, size, shape, and position of objects around the vehicle 1 to be detected.

[0053] For example, the recognition unit 73 detects the movement of objects around the vehicle 1 by performing tracking that follows the movement of clusters of points classified by clustering. This allows the velocity and direction of travel (movement vector) of objects around the vehicle 1 to be detected.

[0054] For example, the recognition unit 73 recognizes the types of objects around the vehicle 1 by performing object recognition processing such as semantic segmentation on the image data supplied from the camera 51.

[0055] Objects that can be detected or recognized include, for example, vehicles, people, bicycles, obstacles, structures, roads, traffic lights, traffic signs, and road markings.

[0056] For example, the recognition unit 73 performs recognition processing of traffic rules around the vehicle 1 based on the map stored in the map information storage unit 23, the self-position estimation result, and the recognition result of objects around the vehicle 1. Through this processing, for example, the location and status of traffic signals, the content of traffic signs and road markings, the content of traffic regulations, and the lanes that can be driven on are recognized.

[0057] For example, the recognition unit 73 performs recognition processing of the environment surrounding the vehicle 1. The surrounding environment to be recognized may include, for example, weather, temperature, humidity, brightness, and road surface conditions.

[0058] The action planning unit 62 creates an action plan for vehicle 1. For example, the action planning unit 62 creates an action plan by performing route planning and route following processes.

[0059] Global path planning is the process of planning the general route from the start to the goal. This path planning also includes a process called local path planning, which involves generating a track that allows vehicle 1 to move safely and smoothly in its vicinity, taking into account the motion characteristics of vehicle 1 along the planned path.

[0060] Route following is the process of planning actions to ensure that a vehicle travels safely and accurately along a planned route within a planned time. For example, the target speed and target angular velocity of vehicle 1 are calculated.

[0061] The motion control unit 63 controls the operation of the vehicle 1 in order to realize the action plan created by the action planning unit 62.

[0062] For example, the motion control unit 63 controls the steering control unit 81, the brake control unit 82, and the drive control unit 83 to perform acceleration / deceleration control and direction control so that the vehicle 1 moves along the trajectory calculated by the trajectory plan. For example, the motion control unit 63 performs cooperative control aimed at realizing ADAS functions such as collision avoidance or impact mitigation, follow-me driving, vehicle speed maintenance driving, collision warning for the vehicle itself, and lane departure warning for the vehicle itself. For example, the motion control unit 63 performs cooperative control aimed at autonomous driving, such as driving autonomously without driver operation.

[0063] The DMS30 performs driver authentication and driver status recognition based on sensor data from the in-vehicle sensors 26 and input data entered into the HMI31. The driver status to be recognized may include, for example, physical condition, alertness, concentration level, fatigue level, gaze direction, intoxication level, driving operation, and posture.

[0064] Furthermore, the DMS30 may perform authentication processing for passengers other than the driver and recognition processing for the status of said passengers. Also, for example, the DMS30 may perform recognition processing of the conditions inside the vehicle based on sensor data from the in-vehicle sensor 26. Examples of conditions inside the vehicle to be recognized include temperature, humidity, brightness, and odor.

[0065] The HMI31 is used for inputting various data and instructions, and generates input signals based on the input data and instructions, which are then supplied to each part of the vehicle control system 11. For example, the HMI31 includes operating devices such as a touch panel, buttons, microphones, switches, and levers, as well as operating devices that allow input by methods other than manual operation, such as voice or gestures. The HMI31 may also be, for example, a remote control device using infrared or other radio waves, or an externally connected device such as a mobile device or wearable device that is compatible with the operation of the vehicle control system 11.

[0066] Furthermore, the HMI31 generates and outputs visual, auditory, and tactile information to the occupant or outside the vehicle, and performs output control to control the output content, timing, and method. Visual information is information shown by images and light, such as operation screens, vehicle status displays, warning displays, and monitor images showing the surroundings of vehicle 1. Auditory information is information shown by voice, such as guidance, warning sounds, and warning messages. Tactile information is information given to the occupant's sense of touch, such as force, vibration, and movement.

[0067] Examples of devices that output visual information include display devices, projectors, navigation systems, instrument panels, CMS (Camera Monitoring Systems), electronic mirrors, and lamps. Display devices may include not only devices with conventional displays, but also devices that display visual information within the passenger's field of view, such as head-up displays, transparent displays, and wearable devices with AR (Augmented Reality) functionality.

[0068] Devices that output auditory information include, for example, audio speakers, headphones, and earphones.

[0069] Devices that output tactile information include, for example, haptic elements using haptic technology. Haptic elements can be installed in, for example, a steering wheel or a seat.

[0070] The vehicle control unit 32 controls various parts of the vehicle 1. The vehicle control unit 32 includes a steering control unit 81, a brake control unit 82, a drive control unit 83, a body system control unit 84, a light control unit 85, and a horn control unit 86.

[0071] The steering control unit 81 detects and controls the state of the steering system of the vehicle 1. The steering system includes, for example, a steering mechanism with a steering wheel, an electric power steering system, etc. The steering control unit 81 includes, for example, a control unit such as an ECU that controls the steering system, an actuator that drives the steering system, etc.

[0072] The brake control unit 82 detects and controls the state of the brake system of the vehicle 1. The brake system includes, for example, a brake mechanism including a brake pedal, an ABS (Antilock Brake System), etc. The brake control unit 82 includes, for example, a control unit such as an ECU that controls the brake system, an actuator that drives the brake system, etc.

[0073] The drive control unit 83 detects and controls the state of the vehicle 1's drive system. The drive system includes, for example, an accelerator pedal, a drive force generating device for generating driving force such as an internal combustion engine or drive motor, and a drive force transmission mechanism for transmitting driving force to the wheels. The drive control unit 83 includes, for example, a control unit such as an ECU for controlling the drive system, and actuators for driving the drive system.

[0074] The body system control unit 84 detects and controls the state of the body system of the vehicle 1. The body system includes, for example, a keyless entry system, a smart key system, power window devices, power seats, an air conditioning system, airbags, seat belts, a shift lever, etc. The body system control unit 84 includes, for example, a control unit such as an ECU that controls the body system, and actuators that drive the body system.

[0075] The light control unit 85 detects and controls the status of various lights on the vehicle 1. Examples of lights to be controlled include headlights, taillights, fog lights, turn signals, brake lights, projection lights, and bumper displays. The light control unit 85 includes a control unit such as an ECU that controls the lights, and actuators that drive the lights.

[0076] The horn control unit 86 detects and controls the state of the car horn of the vehicle 1. The horn control unit 86 includes, for example, a control unit such as an ECU that controls the car horn, and an actuator that drives the car horn.

[0077] Figure 2 shows an example of the sensing area of ​​the external recognition sensor 25 in Figure 1, which includes the camera 51, radar 52, LiDAR 53, and ultrasonic sensor 54.

[0078] Sensing regions SA1F and SA1B show examples of the sensing regions of the ultrasonic sensor 54. Sensing region SA1F covers the area around the front end of vehicle 1. Sensing region SA1B covers the area around the rear end of vehicle 1.

[0079] The sensing results in sensing areas SA1F and SA1B are used, for example, to assist with parking vehicle 1.

[0080] Sensing areas SA2F to SA2B show examples of sensing areas for short-range or medium-range radar 52. Sensing area SA2F covers a position further in front of vehicle 1 than sensing area SA1F. Sensing area SA2B covers a position further in rear of vehicle 1 than sensing area SA1B. Sensing area SA2L covers the rear periphery of the left side of vehicle 1. Sensing area SA2R covers the rear periphery of the right side of vehicle 1.

[0081] The sensing results in sensing region SA2F are used, for example, to detect vehicles or pedestrians in front of vehicle 1. The sensing results in sensing region SA2B are used, for example, to prevent collisions behind vehicle 1. The sensing results in sensing regions SA2L and SA2R are used, for example, to detect objects in the blind spots to the sides of vehicle 1.

[0082] Sensing areas SA3F to SA3B show examples of sensing areas by camera 51. Sensing area SA3F covers a position further in front of vehicle 1 than sensing area SA2F. Sensing area SA3B covers a position further in rear of vehicle 1 than sensing area SA2B. Sensing area SA3L covers the periphery of the left side of vehicle 1. Sensing area SA3R covers the periphery of the right side of vehicle 1.

[0083] The sensing results in sensing region SA3F are used, for example, for the recognition of traffic lights and traffic signs, and for lane departure prevention support systems. The sensing results in sensing region SA3B are used, for example, for parking assistance and surround view systems. The sensing results in sensing regions SA3L and SA3R are used, for example, for surround view systems.

[0084] Sensing area SA4 shows an example of the sensing area of ​​LiDAR 53. Sensing area SA4 covers a position further in front of vehicle 1 than sensing area SA3F. On the other hand, sensing area SA4 has a narrower range in the lateral direction than sensing area SA3F.

[0085] The sensing results in sensing region SA4 are used, for example, for emergency braking, collision avoidance, and pedestrian detection.

[0086] Sensing area SA5 shows an example of the sensing area of ​​the long-range radar 52. Sensing area SA5 covers a position further in front of vehicle 1 than sensing area SA4. On the other hand, sensing area SA5 has a narrower range in the lateral direction than sensing area SA4.

[0087] The sensing results in sensing region SA5 are used, for example, in ACC (Adaptive Cruise Control).

[0088] Furthermore, the sensing areas of each sensor may take on various configurations other than those shown in Figure 2. Specifically, the ultrasonic sensor 54 may also sense the sides of the vehicle 1, or the LiDAR 53 may be configured to sense the rear of the vehicle 1.

[0089] <<2. First Embodiment>> Next, a first embodiment of the present technology will be described with reference to Figures 3 to 19.

[0090] <Examples of installation locations for the external recognition sensor 25 and the in-vehicle sensor 26> Figure 3 shows examples of the installation locations of the camera 51, radar 52, and LiDAR 53 in the external recognition sensor 25 and the camera in the in-vehicle sensor 26 of vehicle 1A, which is a first embodiment of vehicle 1 to which this technology is applied. In this example, the external recognition sensor 25 includes cameras 51FC to 51BR, radars 52FC to 52BR, and LiDARs 53F to 53B. The in-vehicle sensor 26 includes cameras 151FL to 151BR.

[0091] Camera 51FC is located near the center of the front of vehicle 1A. Camera 51FC photographs the area in front of vehicle 1A.

[0092] Camera 51FL is located near the left end of the front of vehicle 1A. Camera 51FL photographs the area diagonally to the left and in front of vehicle 1A.

[0093] Camera 51FR is located near the right end of the front of vehicle 1A. Camera 51FR photographs the area diagonally to the right and in front of vehicle 1A.

[0094] Camera 51M is located near the front center of the interior of vehicle 1A. Camera 51M photographs the area in front of vehicle 1A through the windshield 104.

[0095] Camera 51SL is located on the left side of vehicle 1A, near the front edge of the door of the driver's seat 101. Camera 51SL photographs the left rear (left side and rear) of vehicle 1A.

[0096] Camera 51SR is located on the right side of vehicle 1A, near the front edge of the passenger seat 102 door, which is positioned next to the driver's seat 101. Camera 51SR photographs the right rear (right side and rear) of vehicle 1A.

[0097] Camera 51BC is located near the center of the rear end of vehicle 1A. Camera 51BC photographs the area behind vehicle 1A.

[0098] Camera 51BL is located near the left end of the rear of vehicle 1A. Camera 51BL captures the area diagonally to the left rear of vehicle 1A.

[0099] Camera 51BR is located near the right end of the rear of vehicle 1A. Camera 51BR photographs the area diagonally to the right rear of vehicle 1A.

[0100] Radar 52FC is located near the center of the front of vehicle 1A. Radar 52FC performs sensing in front of vehicle 1A.

[0101] Radar 52FL is located near the left end of the front of vehicle 1A. Radar 52FL performs sensing in the diagonally front left area of ​​vehicle 1A.

[0102] Radar 52FR is located near the right edge of the front of vehicle 1A. Radar 52FR performs sensing in the area diagonally to the right and in front of vehicle 1A.

[0103] Radar 52SL is located on the front left side of vehicle 1A. Radar 52SL performs sensing on the left side of vehicle 1A.

[0104] Radar 52SR is located on the front right side of vehicle 1A. Radar 52SL performs sensing on the right side of vehicle 1A.

[0105] Radar 52BC is located near the center of the rear end of vehicle 1A. Radar 52BC performs sensing of the area behind vehicle 1A.

[0106] Radar 52BL is located near the left end of the rear of vehicle 1A. Radar 52BL performs sensing of the area diagonally to the left rear of vehicle 1A.

[0107] Radar 52BR is located near the right edge of the rear end of vehicle 1A. Radar 52BR performs sensing of the area diagonally to the right rear of vehicle 1A.

[0108] The LiDAR53F is located near the center of the front of vehicle 1A. The LiDAR53F performs sensing in front of vehicle 1A.

[0109] The LiDAR53L is located on the front left side of vehicle 1A. The LiDAR53L performs sensing on the left side of vehicle 1A.

[0110] The LiDAR53R is located on the front right side of vehicle 1A. The LiDAR53R performs sensing on the right side of vehicle 1A.

[0111] The LiDAR 53B is located near the center of the rear end of vehicle 1A. The LiDAR 53B performs sensing of the area behind vehicle 1A.

[0112] Camera 151FL is installed inside the vehicle, near the front left of the driver's seat 101. Camera 151FL takes pictures of the area around the driver (the person sitting in the driver's seat 101).

[0113] Camera 151FR is installed inside the vehicle, near the front right of the passenger seat 102. Camera 151FR takes pictures of the area around the passenger sitting in the passenger seat 102.

[0114] Camera 151BL is located inside the vehicle, near the front of the left rear seat 103L. Camera 151BL captures images of the area around the passenger sitting in rear seat 103L.

[0115] Camera 151BR is located inside the vehicle, near the front of the right rear seat 103R. Camera 151BR captures images of the area around the passenger sitting in rear seat 103R.

[0116] Hereafter, when there is no need to distinguish between rear seat 103L and rear seat 103R individually, they will simply be referred to as rear seat 103.

[0117] <Example of in-car display configuration> Next, with reference to Figures 4 to 9, an example of the configuration of a display (display unit) installed inside the vehicle 1A, which constitutes part of the HMI 31 and forms part of the display system of the vehicle 1A will be described. Figure 4 is a schematic diagram showing the front of the interior of the vehicle. Figure 5 is a schematic diagram of the area around the driver's seat 101 and passenger seat 102 viewed from above. Figure 6 is a schematic diagram of the passenger seat 102 viewed from the left side. Figure 7 is a schematic diagram of the area around the driver's seat 101 and passenger seat 102 viewed from the left rear. Figure 8 is a schematic diagram of the front of the vehicle 1A viewed from the driver's seat 101. Figure 9 is a schematic diagram of the passenger seat 102 viewed from the left rear.

[0118] The interior of the vehicle is equipped with a center display 201, a console display 202, a head-up display (only display 203 is shown), a digital rearview mirror 204, a steering wheel display 205, and a rear entertainment display 206.

[0119] The center display 201 is positioned in front of the driver's seat 101 and the passenger seat 102, extending horizontally from left to right on the front of the dashboard 105. Depending on the orientation of the display, the center display 201 is broadly divided into the left end 201L, the central part 201C, and the right end 201R. In other words, the center display 201 has a configuration in which the left end 201L, the central part 201C, and the right end 201R, which have different orientations, are continuous in the left-right direction and integrated. The left end 201L, the central part 201C, and the right end 201R can each display independently or as a single integrated display.

[0120] The central section 201C extends horizontally from near the left edge of the driver's seat 101 to near the right edge of the passenger seat 102, in front of the driver's seat 101 and the passenger seat 102, and faces rearward (towards the rear of the vehicle 1A) as viewed from the driver's seat 101 or the passenger seat 102. Furthermore, as shown in Figure 6, the central section 201C is angled upward. This makes the angle of incidence of the line of sight when the driver sitting in the driver's seat 101 and the passenger sitting in the passenger seat 102 look at the central section 201C closer to vertical, improving visibility.

[0121] The left end 201L and the right end 201R are provided approximately symmetrically at both ends of the center display 201. The left end 201L bends inward (towards the vehicle interior) at the left end of the center display 201, is angled inward relative to the central part 201C, and faces diagonally rear to the right (diagonally rear to the right of vehicle 1A) as viewed from the driver's seat 101 or passenger seat 102. The right end 201R bends inward (towards the vehicle interior) at the right end of the center display 201, is angled inward relative to the central part 201C, and faces diagonally rear to the left (diagonally rear to the left of vehicle 1A) as viewed from the driver's seat 101 or passenger seat 102.

[0122] The angle of the left end 201L with respect to the central part 201C is adjusted so that, for example, the angle of reflection of the angle of incidence of a standard driver's line of sight to the left end 201L points in the appropriate direction to the left rear of vehicle 1A. The angle of the right end 201R with respect to the central part 201C is adjusted so that, for example, the angle of reflection of the angle of incidence of a standard driver's line of sight to the right end 201R points in the appropriate direction to the right rear of vehicle 1A.

[0123] The console display 202 is located on the console 107, which is situated between the driver's seat 101 and the passenger seat 102, and is positioned below the central part 201C of the center display 201.

[0124] The console 107 extends in the front-to-back direction between the driver's seat 101 and the passenger seat 102. As shown in Figure 6, the console 107 is broadly divided into a front end 107A, a middle section 107B, and a rear end 107C, from front to back.

[0125] The tip 107A of the console 107 is inclined diagonally upward toward the front of the vehicle 1A. A console display 202 is provided at the tip of the tip 107A. Behind (below) the console display 202 on the tip 107A, there is an operating section 109 equipped with, for example, multiple physical switches, keys, etc. (not shown).

[0126] The console display 202 is configured, for example, as a two-dimensional or three-dimensional touch panel, and can be operated by touching or bringing a finger or the like close to it. The console display 202 faces the rear of the vehicle 1A. Furthermore, the console display 202 is angled diagonally upward at approximately the same angle as the central part 201C of the center display 201. This creates a sense of unity, as if the center display 201 and the console display 202 are connected as a single unit. In addition, the visibility of the console display 202 is improved, similar to the central part 201C of the center display 201.

[0127] The middle section 107B of the console 107 is inclined diagonally downward towards the front of the vehicle 1A, in contrast to the front section 107A. A circular controller 108 is provided near the center of the middle section 107B. The inclination angle of the middle section 107B is set to an angle that makes it easy for the driver to place their hands, thereby improving the operability of the controller 108.

[0128] The rear end 107C of the console 107 is approximately horizontal, for example, to make it easier for the driver to rest their arms.

[0129] The head-up display includes a display 203 located in front of the driver's seat 101, as shown in Figure 4. For example, the display 203 may be part of the windshield 104, or it may be provided separately from the windshield 104. In the latter case, for example, the display 203 is attached to the windshield 104. Then, using AR technology, visual information is projected onto the display 203, so that the visual information is superimposed within the driver's field of view.

[0130] Furthermore, the display 203 is positioned higher than, for example, the display of a conventional in-vehicle head-up display. This makes it possible to superimpose visual information onto vehicles further ahead of vehicle 1A (for example, 100m ahead).

[0131] The digital rearview mirror 204 is used in place of the conventional rearview mirror and is also called a smart rearview mirror. Like the conventional rearview mirror, the digital rearview mirror 204 is located slightly in front of the center of the upper edge of the windshield 104 and positioned above the central part 201C of the center display 201.

[0132] The steering wheel display 205 is located in the center of the steering wheel 106.

[0133] The rear entertainment display 206 is located on the back of the passenger seat 102, more specifically, on the back of the headrest of the passenger seat 102, as shown in Figure 7 or Figure 9.

[0134] The rear entertainment display 206 can also be installed on the backs of both the driver's seat 101 and the passenger seat 102, or only on the back of the driver's seat 101.

[0135] The displays on the center display 201, console display 202, head-up display, digital rear mirror 204, steering wheel display 205, and rear entertainment display 206 are controlled, for example, by the display control unit, which is part of the functions of the HMI 31.

[0136] <Examples of content displayed on each screen inside the vehicle> Next, with reference to Figures 10 to 17, examples of the content displayed on each in-vehicle display will be described.

[0137] <Example of content displayed on the center display 201> First, let's look at an example of what the center display 201 shows.

[0138] The central part 201C of the center display 201 displays, for example, information to assist driving or images of the surroundings of the vehicle 1A.

[0139] For example, the central section 201C displays information related to the driving status of vehicle 1A. For example, as shown in Figure 10, turn signal information 301, speed information 302, and battery information 303 are displayed side by side.

[0140] The turn signal information 301 indicates the operating status of the turn signals of vehicle 1A. The left and right arrows in the turn signal information 301 flash or light up in conjunction with the left and right turn signals, respectively.

[0141] Speed ​​information 302 indicates the speed and shift position of vehicle 1A. The speed of vehicle 1A is indicated by a bar extending horizontally and a number. For example, speed information 302 may also indicate the speed limit.

[0142] Battery information 303 shows the remaining battery level of the vehicle's 1A battery. The battery level is indicated by a bar extending horizontally and by a number.

[0143] For example, the central section 201C displays an image showing the road conditions ahead when there is a road extending to the left or right in front of vehicle 1A at a T-junction or intersection. For example, an image is displayed that is a composite of images from cameras 51FL, 51FC, and 51FR, that is, an image of approximately 180 degrees centered on the front of vehicle 1A, as seen from near the front of vehicle 1A. This allows the driver to easily recognize vehicles approaching from the left or right on the road ahead.

[0144] Furthermore, T-junctions or crossroads in front of vehicle 1A are detected by the recognition unit 73 based, for example, on images from camera 51M and sensor data from LiDAR 53F.

[0145] For example, the central section 201C displays information to assist with parking when vehicle 1A is parked. For example, as shown in Figure 11, images from cameras 51BL, 51BC, and 51BR, i.e., images of the rear of vehicle 1A, are displayed. In addition, the parking position 321 is displayed in a flashing manner within the image of the rear of vehicle 1A. For example, when automatic parking is performed, the parking position 321 indicates the position set for parking, and when manual parking is performed, it indicates a parking position or a recommended parking position.

[0146] In this case, for example, the 360-degree images of the vehicle 1A taken by each camera 51 (hereinafter referred to as wide-angle images) may be rotated and displayed in the central section 201C. The range of the wide-angle image displayed in the central section 201C may be rotated according to the driver's posture, for example. Specifically, for example, the range of the wide-angle image displayed in the central section 201C may be shifted based on the driver's position and head orientation relative to a predetermined reference position, thereby rotating the wide-angle image displayed in the central section 201C.

[0147] This allows you to check the parking location, easily find a suitable parking spot, and safely park vehicle 1A.

[0148] The parking position 321 is detected by the recognition unit 73 based on images from cameras 51BL, 51BC, and 51BR, as well as sensor data from LiDAR 53B.

[0149] For example, when vehicle 1A is backing up, the central section 201C displays images from cameras 51BL, 51BC, and 51BR, that is, images of the area behind vehicle 1A.

[0150] For example, as shown in Figure 12, an image 341 of the rear of vehicle 1A is displayed. Also, for example, if an obstacle is detected behind vehicle 1A, an alert display 342 flashes within image 341. An image 343 showing the location of the obstacle is also displayed. The vehicle in image 343 is vehicle 1A. Then, an alert display 344 showing the location of the obstacle flashes around the vehicle in image 343. In this example, it is shown that there is an obstacle to the right and slightly behind vehicle 1A.

[0151] This allows the driver to safely reverse vehicle 1A.

[0152] Obstacles behind vehicle 1A are detected by the recognition unit 73 based on images from cameras 51BL, 51BC, and 51BR, as well as sensor data from radars 52BL, 52BC, 52BR, and LiDAR 53B.

[0153] For example, if the central unit 201C detects the siren of an emergency vehicle (e.g., an ambulance, fire truck, police car, etc.), it displays information indicating the direction of the siren's source. This allows the driver to accurately recognize the direction of the emergency vehicle and take appropriate action, such as taking cover.

[0154] The direction of the siren sound source is detected by the recognition unit 73, for example, based on the sound detected by the microphone provided by the external recognition sensor 25.

[0155] For example, the central unit 201C displays an alert if a person is detected in the shadows around the vehicle 1A. For instance, information indicating the direction of the person is displayed as an alert. This allows the driver to avoid collisions or contact with people hiding in the shadows.

[0156] Furthermore, people hiding behind objects are detected by the recognition unit 73 based on sensor data from each radar 52 and each LiDAR 53, for example.

[0157] For example, the central unit 201C displays an alert when an object that is difficult for the human eye to see (e.g., an animal in the dark) is detected in a dark environment around the vehicle 1A. For example, if an image of the area in front of the vehicle 1A, captured by camera 51M, is displayed, a frame indicating the location of the detected object is superimposed as an alert. This allows the driver to avoid collisions or contact with the detected object.

[0158] Objects that are difficult to see are detected by the recognition unit 73, for example, based on the difference in brightness in the image from camera 51M.

[0159] For example, the central unit 201C displays information indicating the predicted direction of travel when the Lane Keep Assist (LKA) function is activated while vehicle 1A is backing up. The direction of travel is indicated, for example, by lines or arrows. This allows the driver to know the direction they are backing up in advance, enabling them to respond appropriately in emergencies or dangerous situations.

[0160] For example, the central unit 201C displays images from cameras 151FL and 151FR installed in the rear seat 103. This allows the driver to monitor, for example, a child sitting in the rear seat 103.

[0161] For example, if the central section 201C is displaying an image of the area around the vehicle 1A, the display range of the image may be changed based on at least one of the driver's gaze direction and posture.

[0162] The driver's gaze direction and posture are recognized by the DMS30, for example, based on images from camera 151FL. The driver's posture is also recognized, for example, the position and orientation of the head, as well as the seating position.

[0163] Furthermore, the central section 201C can be divided into three sections depending on the situation: the driver's seat area in front of the driver's seat 101, the central area between the driver's seat 101 and the passenger seat 102, and the passenger seat area in front of the passenger seat 102. The driver's seat area, the central area, and the passenger seat area can each be displayed independently.

[0164] The driver's area primarily displays information for the driver. For example, the information mentioned above that assists with driving is displayed there.

[0165] The central area displays infotainment-related information, such as audio, video, websites, and maps.

[0166] In the passenger seat area, for example, infotainment-related information for the passenger in the front seat is displayed.

[0167] The left end 201L and the right end 201R of the center display 201 are primarily used as digital outer mirrors (electronic side mirrors) that replace conventional side mirrors. In other words, the left end 201L and the right end 201R are used for the CMS. For example, the left end 201L displays an image of the left rear of vehicle 1A taken by camera 51SL. The right end 201R displays an image of the right rear of vehicle 1A taken by camera 51SR.

[0168] Furthermore, the left end 201L, the center 201C, and the right end 201R of the center display 201 are continuous in the horizontal direction. Therefore, when the driver shifts their gaze from the center 201C to the left end 201L or the right end 201R, or from the left end 201L or the right end 201R to the center 201C, they only need to move their gaze horizontally. As a result, the distance and direction of the driver's gaze movement are reduced, improving visibility.

[0169] Additionally, for example, the leftmost unit 201L will display an alert if it detects a vehicle, motorcycle, bicycle, pedestrian, etc., approaching from the left side or rear of vehicle 1A.

[0170] For example, if a vehicle is approaching from the left rear of vehicle 1A, a frame 361 will be displayed to alert the driver to the approaching vehicle, as shown in Figure 13. In addition, an icon 362 indicating the position of the approaching vehicle will flash. The upper right rectangle of the icon 362 represents vehicle 1A, and the lower left half of the rectangle represents the approaching vehicle.

[0171] Similarly, the rightmost unit 201R displays an alert if it detects a vehicle, motorcycle, bicycle, pedestrian, etc., approaching from the right side or rear of vehicle 1A.

[0172] Vehicles approaching from the left and right sides and rear of vehicle 1A are detected by the recognition unit 73 based on images from cameras 51SL, 51SR, 51BL, 51BC, and 51BR, as well as sensor data from radars 52BL, 52BC, 52BR, and LiDAR 53B.

[0173] Furthermore, based on at least one of the driver's gaze direction and posture, the display control unit adjusts, for example, the shooting direction of cameras 51SL and 51SR. As a result, the range of the image displayed at the left end 201L and the right end 201R is appropriately adjusted according to the driver's gaze direction and posture, similar to conventional physical side mirrors. Consequently, the driver can check the situation in the desired range diagonally behind the vehicle 1A without feeling any discomfort.

[0174] Furthermore, the display range of the images at the left end 201L and the right end 201R is changed based on the operation of the turn signal, for example. For instance, when the turn signal is not activated, the field of view is set to the normal angle, and when the turn signal is activated, the field of view is widened. This allows the driver to accurately perceive the surroundings of vehicle 1A and safely change the direction of travel of vehicle 1A.

[0175] Furthermore, since the leftmost section 201L, the central section 201C, and the rightmost section 201R are connected as one unit, it is possible to display a single screen across the entire center display 201. For example, it is possible to display images of the vehicle 1A's surroundings, map information, or infotainment-related information across the entire center display 201.

[0176] <Example of what is displayed on console display 202> Next, we will describe an example of what is displayed on console display 202.

[0177] For example, the console display 202 displays an operation screen for controlling the in-vehicle air conditioning system. The driver or other passengers use the displayed operation screen to operate the in-vehicle air conditioning equipment.

[0178] For example, at least a portion of the operation of the in-vehicle air conditioning equipment can also be performed using the control unit 109.

[0179] For example, the console display 202 displays an operation screen for manipulating the information displayed on the center display 201. For example, a passenger such as the driver can use the displayed operation screen to scroll, zoom in, zoom out, switch, etc., of the information (for example, a map) displayed in the central part 201C of the center display 201.

[0180] As mentioned above, the center display 201 and the console display 202 are integrated seamlessly. Therefore, passengers can operate the information displayed on the center display 201 using the control screen of the console display 202 with a natural feel, improving usability.

[0181] The controller 108 can be pressed, rotated, and tilted in predetermined directions (for example, forward, backward, left, or right), and is used, for example, to operate the in-car audio system. For example, when the controller 108 is pressed, music is played or stopped, and the volume is adjusted by rotating the controller 108.

[0182] <Example of content displayed on a head-up display> The head-up display's display 203, for example, displays information to assist driving.

[0183] For example, display 203 displays an alert depending on the surrounding conditions of vehicle 1A.

[0184] For example, as shown in Figure 14, if there is a sharp curve ahead of vehicle 1A, an alert display 381 is shown. The alert display 381 indicates the direction of the curve with an arrow and the curvature of the curve with the shape (e.g., length, thickness, etc.) and color of the arrow. This allows the driver to recognize the presence of a sharp curve in advance and drive through the curve safely.

[0185] For example, as shown in Figure 15, if a large vehicle such as a truck is present in the lane adjacent to the lane in which vehicle 1A is traveling, the alert display 401 will flash. The alert display 401 is an alert that notifies the driver that a large vehicle such as a truck is approaching in the lane to the right of vehicle 1A. This allows the driver to recognize the approach of a large vehicle in advance and avoid a collision or contact with the large vehicle.

[0186] For example, when the emergency brake of vehicle 1A is activated, the display 203 displays an alert notifying the driver that the emergency brake has been activated. This informs the driver of the cause of the emergency stop, providing them with a sense of security.

[0187] For example, the display 203 displays information indicating the vehicle ahead that is being followed while ACC (Adaptive Cruise Control) is in operation. For example, a frame surrounding the vehicle being followed is displayed. This allows the driver to confirm the operation of ACC and predict the path of vehicle 1A, enabling them to take appropriate action, for example, in emergencies or dangerous situations.

[0188] For example, the display 203 displays the planned route using arrows or other symbols as needed while the lane change assist function is operating. This allows the driver to recognize the lane change in advance and take appropriate action, for example, in emergencies or dangerous situations.

[0189] <Example of display content for Digital Rear Mirror 204> The digital rearview mirror 204 displays, for example, an image of the rear of the vehicle 1A taken by the camera 51BC.

[0190] For example, the digital rearview mirror 204 displays an alert if another vehicle is approaching from behind vehicle 1A. For example, as shown in Figure 16, if the distance between vehicle 1A and vehicle 421 behind it falls below a predetermined threshold, an alert display 422 flashes to draw the driver's attention.

[0191] The distance to the vehicle behind is detected by the recognition unit 73 based on sensor data from, for example, radar 52BC and LiDAR 53B.

[0192] Furthermore, the shooting direction of the camera 51BC is adjusted based on, for example, at least one of the driver's gaze direction and posture. This ensures that the range of the image displayed on the digital rear mirror 204 is appropriately adjusted according to the driver's gaze direction and posture, similar to a conventional physical rearview mirror. As a result, the driver can check the situation in the desired range behind the vehicle 1A without feeling any discomfort.

[0193] <Example of content displayed on the steering wheel display 205> The steering wheel display 205, for example, displays a message to the driver. This allows the driver and vehicle 1A to communicate.

[0194] For example, if the driver is recognized by the DMS 30, the steering wheel display 205 displays a message 441 containing the name of the recognized driver, as shown in Figure 17.

[0195] For example, when the driver inputs a voice command, the steering wheel display 205 displays a message 461 indicating that the voice command has been received, as shown in Figure 18.

[0196] Additionally, for example, the steering wheel display 205 displays messages confirming and indicating completion of payment for highways, toll roads, parking lots, etc.

[0197] <Example of content displayed on the rear entertainment display 206> For example, the rear entertainment display 206 displays infotainment-related information for passengers in the rear seats 103.

[0198] For example, the rear entertainment display 206 displays alerts in emergencies or dangerous situations. For instance, as shown in Figure 19, an alert display 481 flashes to warn passengers in the rear seats 103 during an emergency stop.

[0199] As described above, displaying various types of information on the in-vehicle displays can improve convenience for the driver and other passengers. For example, it can assist in driving vehicle 1A, provide passengers with necessary information, and entertain them.

[0200] <<3. Modified Examples of the First Embodiment>> The following describes a modified version of the first embodiment of the present technology described above.

[0201] This technology can also be applied to other mobility devices that carry people, such as vehicles.

[0202] Furthermore, this technology can also be applied to a mobility device that operates autonomously without the driver's input. In this case, the distinction between the driver's seat 101 and the passenger seat 102 mentioned above is eliminated, and the various displays described above are placed in front of the seats within the mobility device. In addition, the display range of the images shown on the displays is changed based on, for example, the gaze direction and posture of the person sitting in a designated seat.

[0203] <<4. Second Embodiment>> Next, a second embodiment of the present technology will be described with reference to Figures 20 to 53.

[0204] In the figures of the second embodiment, the same reference numerals are used for parts corresponding to those in the first embodiment, and their descriptions are omitted as appropriate.

[0205] <Example of exterior configuration for vehicle 1B> First, with reference to Figures 20 to 26, an example of the exterior configuration of vehicle 1B, which is a second embodiment of vehicle 1 to which this technology is applied, will be described. Below, the description will mainly focus on the configuration of the exterior lighting system of vehicle 1B.

[0206] Figure 20 is a front view of vehicle 1B and a close-up of the area around the logo on the front center of vehicle 1B. Figure 21 is a left side view of vehicle 1B. Figure 22 is a close-up of the front end of the left side of vehicle 1B. Figure 23 is a rear view of vehicle 1B. Figures 24 to 26 are views of the area around the left headlight of vehicle 1B from multiple directions.

[0207] In the following, the left and right sides of vehicle 1B, when facing the direction of travel, will be referred to as the left and right sides of vehicle 1B, respectively. For example, the left and right sides of Figure 20 will be the right and left sides of vehicle 1B, respectively.

[0208] Most of the external lights and sensors of vehicle 1B are positioned along a virtual loop line L1, which is a line that surrounds the body of vehicle 1B in a roughly horizontal direction. Here, being positioned along the loop line L1 includes not only being positioned on the loop line L1, but also being positioned in the vicinity of the loop line L1.

[0209] For example, as shown in Figure 20, accessory lights 601L, 601R, daytime running lights 602L, 602R, headlights 603LU, 603LD, 603RU, and 603RD are located on the front of the body.

[0210] Accessory light 601L extends in the width direction (left-right direction) from the center of the front of the body to near the right edge of headlights 603LU and 603LD. Accessory light 601R extends in the width direction (left-right direction) from the center of the front of the body to near the left edge of headlights 603RU and 603RD.

[0211] Accessory light 601L and accessory light 601R are separated, with a gap between them. Specifically, the right end of accessory light 601L is bent diagonally downward to the right, and the left end of accessory light 601R is bent diagonally upward to the left. The bent portion at the right end of accessory light 601L and the bent portion at the left end of accessory light 601R are positioned approximately parallel to each other with a predetermined distance between them, forming the logo of vehicle 1B. Near the center of the logo, in the gap area A1 between accessory light 601L and accessory light 601R, optical sensors (not shown), such as cameras, radar, or LiDAR, are positioned.

[0212] The daytime running light 602L extends horizontally from the left end of the accessory light 601L to near the left end of the headlights 603LU and 603LD. Furthermore, as shown in Figures 24 to 26, the left end of the daytime running light 602L curves toward the rear of the vehicle 1B and extends in a direction that penetrates the body.

[0213] The daytime running light 602R extends horizontally from the right end of the accessory light 601R to near the right end of the headlights 603RU and 603RD. Although not shown in the diagram, similar to the daytime running light 602L, the right end of the daytime running light 602R curves toward the rear of the vehicle 1B and extends in a direction that penetrates the body.

[0214] The accessory light 601L, the accessory light 601R, the daytime running light 602L, and the daytime running light 602R constitute a front light along the loop line L1 on the front surface of the body. Further, the front light constitutes a front light that is a part of the loop line L1, extends in the vehicle width direction (left - right direction) on the front surface of the body, curves backward at both ends, and extends in a direction penetrating the body.

[0215] The accessory light 601L, the accessory light 601R, the daytime running light 602L, and the daytime running light 602R each include a plurality of LEDs arranged horizontally. Each LED can be individually controlled for on / off, color, brightness, etc.

[0216] Hereinafter, when there is no need to individually distinguish between the accessory light 601L and the accessory light 601R, they are simply referred to as the accessory light 601. Hereinafter, when there is no need to individually distinguish between the daytime running light 602L and the daytime running light 602R, they are simply referred to as the daytime running light 602.

[0217] The headlight 603LU is adjacent to the upper side of the daytime running light 602L, extends horizontally, and curves backward at the left end. The headlight 603LD is adjacent to the lower side of the daytime running light 602L, extends horizontally, and curves backward at the left end. Thus, the headlight 603LU and the headlight 603LD are separated vertically by the daytime running light 602L (front light).

[0218] The headlight 603RU is adjacent to the upper side of the daytime running light 602R, extends horizontally, and curves backward at the right end. The headlight 603RD is adjacent to the lower side of the daytime running light 602R, extends horizontally, and curves backward at the right end. Thus, the headlight 603RU and the headlight 603RD are separated vertically by the daytime running light 602R (front light).

[0219] The headlamps 603LU and 603RU each include a plurality of LEDs arranged in the horizontal and vertical directions and output a low beam. The headlamps 603LD and 603RD are each composed of a plurality of LEDs arranged in the horizontal and vertical directions and output a high beam. Each LED can be individually controlled for on / off, color, brightness, etc.

[0220] Hereinafter, when it is not necessary to individually distinguish between the headlamp 603LU and the headlamp 603LD, they are simply referred to as the headlamp 603L. Hereinafter, when it is not necessary to individually distinguish between the headlamp 603RU and the headlamp 603RD, they are simply referred to as the headlamp 603R. Hereinafter, when it is not necessary to individually distinguish between the headlamp 603L and the headlamp 603R, they are simply referred to as the headlamp 603.

[0221] In this way, by dividing the headlamps 603L and 603R vertically by the front line, the degree of freedom in the design of the headlamps 603L and 603R is improved. For example, the headlamps 603L and 603R can be designed in a design other than a fish-eye style or a drooping-eye style. Also, since the low beam (headlamps 603LU and 603RU) and the high beam (headlamps 603LD and 603RD) are arranged at appropriate positions, the functionality of the headlamp 603 and the safety of the vehicle 1B do not deteriorate.

[0222] Also, for example, as shown in FIG. 21, the turn signal 604L, the auxiliary lamp 605FL, and the auxiliary lamp 605BL are arranged on the left side surface of the body.

[0223] The turn signal 604L extends in the front-rear direction on the extension line of the A pillar 614L and just above the loop line L1.

[0224] The auxiliary light 605FL is positioned behind the door handle 612FL of the left front door 611FL, illuminating the area around the door handle 612FL. Since the door handle 612FL is located just above the loop line L1, the auxiliary light 605FL is also positioned just above the loop line L1.

[0225] Furthermore, for example, a short-range wireless communication device (not shown), such as an NFC, is placed on or near the door knob 612FL.

[0226] The auxiliary light 605BL is positioned behind the door handle 612BL on the left rear door 611BL, illuminating the area around the door handle 612BL. Since the door handle 612BL is located just above the loop line L1, the auxiliary light 605BL is also positioned just above the loop line L1.

[0227] Furthermore, for example, a short-range wireless communication device (not shown), such as an NFC, is placed on or near the door knob 612BL.

[0228] Thus, on the left side of vehicle 1B, the turn signal 604L, auxiliary light 605FL, and auxiliary light 605BL are arranged in the front-to-back direction along the loop line L1.

[0229] The turn signal 604L, auxiliary light 605FL, and auxiliary light 605BL each feature multiple LEDs arranged horizontally. Each LED can be individually controlled for on / off, color, brightness, etc.

[0230] Furthermore, as shown in Figure 21, optical sensors such as cameras, radar, and LiDAR are provided within region A2L near the intersection of the loop line L1 and the extension of the A-pillar 614L. For example, LiDAR 53L is provided below the turn signal 604L and on the loop line L1.

[0231] Furthermore, within region A3L near the intersection of the loop line L1 and the extension of the C pillar 615L, optical sensors (not shown), such as cameras, radar, and LiDAR, are provided.

[0232] In this way, by placing the optical sensor near the loop line L1, even if, for example, the surface color of the optical sensor differs from the body color, the optical sensor is recognized as part of the loop line L1. As a result, the optical sensor blends naturally into the body's appearance without causing any sense of incongruity.

[0233] Although not shown in the diagram, the right side of vehicle 1B also has the same arrangement of turn signal 604R, auxiliary light 605FR, auxiliary light 605BR, door handle 612FR, door handle 612BR, short-range wireless communication device, and optical sensor as the left side.

[0234] Furthermore, as shown in Figure 23, for example, taillights 606CL, 606CR, 606L, 606R, brake lights 607LU, 607LD, 607RU, and 607RD are located on the rear of the body.

[0235] The taillight 606CL extends in the width direction (left-right) from the center of the rear of the body to near the right edge of the brake lights 607LU and 607LD. The taillight 606CR extends horizontally from the center of the rear of the body to near the right edge of the brake lights 607RU and 607RD.

[0236] The taillights 606CL and 606CR are separated from each other, and a gap is provided therebetween. Specifically, the right end portion of the taillight 606CL is bent upward and to the right, and the left end portion of the taillight 606CR is bent downward and to the left. The bent portion at the right end of the taillight 606CL and the bent portion at the left end of the taillight 606CR face each other almost parallelly with a predetermined interval therebetween, forming the logo of the vehicle 1B. In the vicinity of the center of the logo, in the region A4 of the gap between the taillight 606CL and the taillight 606CR, for example, an optical sensor (not shown) such as a camera, a radar, or a LiDAR is disposed.

[0237] The taillight 606L extends horizontally from the left end of the taillight 606CL to the vicinity of the left ends of the brake lights 607LU and 607LD. The left end portion of the taillight 606L is curved forward. The taillight 606R extends horizontally from the right end of the taillight 606CR to the vicinity of the right ends of the brake lights 607RU and 607RD. The right end portion of the taillight 606R is curved forward.

[0238] The taillights 606CL, 606CR, 606L, and 606R form a tailline that extends in the left - right direction on the rear surface of the body and whose both end portions are curved forward. The tailline constitutes a part of the loop line L1.

[0239] The taillights 606CL, 606CR, 606L, and 606R each include a plurality of LEDs arranged horizontally. Each LED can be individually controlled in terms of on / off, color, brightness, etc.

[0240] Hereinafter, when it is not necessary to distinguish the taillight 606CL and the taillight 606CR individually, they are simply referred to as the taillight 606C. Hereinafter, when it is not necessary to distinguish the taillight 606C, the taillight 606L, and the taillight 606R individually, they are simply referred to as the taillight 606.

[0241] Brake light 607LU is adjacent to the upper side of tail light 606L and curves forward at its left end. Brake light 607LD is adjacent to the lower side of tail light 606L and curves forward at its left end. Thus, brake light 607LU and brake light 607LD are separated vertically by tail light 606L.

[0242] Brake light 607RU is adjacent to the upper side of tail light 606R and curves forward at its right end. Brake light 607RD is adjacent to the lower side of tail light 606R and curves forward at its right end. Thus, brake light 607RU and brake light 607RD are separated vertically by tail light 606R (tail line).

[0243] Brake lights 607LU, 607LD, 607RU, and 607RD each feature multiple LEDs arranged horizontally. Each LED can be individually controlled for on / off, color, brightness, etc.

[0244] In the following, when it is not necessary to distinguish between brake light 607LU and brake light 607LD individually, they will simply be referred to as brake light 607L. In the following, when it is not necessary to distinguish between brake light 607RU and brake light 607RD individually, they will simply be referred to as brake light 607R. In the following, when it is not necessary to distinguish between brake light 607L and brake light 607R individually, they will simply be referred to as brake light 607.

[0245] Furthermore, the body color differs along the lower edge of the windshield 104, the lower edge of the window 613FL of door 611FL, the lower edge of the window 613BL of door 611BL, the lower edge of the window 613FR (not shown) of door 611FR, and the lower edge of the window 613BR (not shown) of door 611BR, with the body color differing on either side of a loop line L2, which is a virtual line that surrounds the body of vehicle 1B in a roughly horizontal direction.

[0246] For example, a black line is formed along the loop line L2 by chrome plating. Above the loop line L2, the body is finished in a black color. For example, the body above the loop line L2 is painted black. In addition, the windshield 104, windows 613FL, 613BL, 613FR, 613BR, and the rear window 616 are tinted with a black smoke.

[0247] On the other hand, below the loop line L2, the body is painted in a different color than the upper part. The color of the lower part of the body is not particularly limited.

[0248] Furthermore, cameras 51SL and 51SR are provided along the loop line L2 near the front and lower ends of windows 613FL and 613FR (not shown). Cameras 51SL and 51SR each photograph the left rear or right rear of vehicle 1B.

[0249] In this way, it is possible to improve the design of vehicle 1B while avoiding a decrease in safety or functionality.

[0250] For example, in the exterior of vehicle 1B, two roughly parallel loop lines, L1 and L2, become virtually recognizable. This makes the vehicle appear lower and gives it a sporty impression.

[0251] Furthermore, as mentioned above, the design freedom of the headlights is increased. In addition, because each light is positioned appropriately, the functionality of each light and the safety of vehicle 1B are not compromised.

[0252] Furthermore, the front line (front lights) curves backward at both ends and extends in a direction that penetrates the body, giving the impression that the front line penetrates the body and connects to the LiDAR53L on the left side and the LiDAR53R on the right side.

[0253] Furthermore, by arranging sensors around vehicle 1B along the loop line L1, it gives the impression that the area around vehicle 1B is being watched (monitored), which is effective in preventing theft and vandalism of vehicle 1B.

[0254] <Example of interior configuration for vehicle 1B> Next, an example of the interior configuration of vehicle 1B will be described with reference to Figures 27 to 36.

[0255] First, with reference to Figures 27 to 30, the devices arranged along the loop line L11 will be described.

[0256] Figure 27 is a schematic diagram of the interior of vehicle 1B viewed from the right. Figure 28 is a schematic diagram of the area around the driver's seat 101 and passenger seat 102 of vehicle 1B. Figure 29 is a schematic diagram of the area around the dashboard of vehicle 1B. Figure 30 is a magnified view of the steering wheel 651 of vehicle 1B.

[0257] Inside vehicle 1B, multiple user interface devices are centrally located along a virtual loop line L11 (Figure 27), which surrounds the interior in a roughly horizontal direction, and various interfaces are integrated within this loop line.

[0258] Here, the user interface devices include, for example, output devices that output visual information, auditory information, and tactile information, and operation devices used for various operations. Furthermore, "arranged along the loop line L11" includes not only cases where the devices are placed on the loop line L11, but also cases where they are placed in the vicinity of the loop line L11.

[0259] Loop line L11 is positioned at the same height as loop line L1 on the exterior of vehicle 1B. Furthermore, loop line L11 is slightly tilted upward from front to rear. This is because the rear seats 103L and 103R are higher than the driver's seat 101 and passenger seat 102.

[0260] For example, the display devices that make up the HMI31 in Figure 1 are arranged along the loop line L11.

[0261] For example, as shown in Figures 27 to 29, the center display 201 is positioned on the front of the dashboard in front of the driver's seat 101 and the passenger seat 102, extending in the vehicle width direction (left-right direction) just above the loop line L11.

[0262] As described above, the center display 201 is broadly divided into the left end, the center, and the right end depending on the orientation of the display. The left end, center, and right end of the center display 201 can each display independently or as a single integrated display. The left end and right end of the center display 201 are mainly used as digital outer mirrors (electronic side mirrors) that replace conventional side mirrors. For example, the left end displays an image of the left rear of vehicle 1B taken by camera 51SL (Figure 20). The right end displays an image of the right rear of vehicle 1B taken by camera 51SR (Figure 20).

[0263] Furthermore, as shown in Figures 28 to 30, the steering wheel 651 is positioned on the loop line L11 in front of the driver's seat 101.

[0264] Furthermore, as shown in Figures 29 and 30, illumination 652 is provided around the central part of the steering wheel 651. The illumination 652 comprises multiple LEDs arranged in a circle around the central part of the steering wheel 651. Each LED can be individually controlled for on / off, color, brightness, etc. Therefore, the illumination 652 has variable color, brightness, and light-emitting area (range of illumination).

[0265] An airbag is housed in the center of the steering wheel 651. When the airbag deploys, the portion of the steering wheel 651 shown by the dotted line in the center of Figure 30 splits open. The illumination 652 is positioned to avoid this split portion in the center. This prevents fragments or harmful substances from being scattered when the airbag deploys.

[0266] Furthermore, the speakers that make up the HMI31 in Figure 1 are arranged along the loop line L11.

[0267] Specifically, as shown in Figures 27 and 28, speaker 653FL is embedded near loop line L11 on the inside of door 611FL on the driver's side 101. As shown in Figure 28, speaker 653FR is embedded near loop line L11 on the inside of door 611FR on the passenger side 102. As shown in Figure 27, speaker 653BL is embedded near loop line L11 on the inside of door 611BL on the left rear seat 103L side. Although not shown in the illustration, speaker 653BR is embedded near loop line L11 on the inside of door 611BR on the right rear seat 103R side.

[0268] Although not shown in the diagram, speakers (hereinafter referred to as seat speakers) are individually embedded below the headrests of the driver's seat 101, passenger seat 102, rear seat 103L, and rear seat 103R. Furthermore, the shape of the seats and the position of the seat speakers have been adjusted so that people of various heights (sitting heights) can clearly hear the sound from the seat speakers in each seat.

[0269] Speakers 653FL to 653BR, positioned along the loop line L11, are used, for example, to output sound to the entire interior of the vehicle (all passengers inside the vehicle).

[0270] Furthermore, the 653FL or 653FR speakers enable 360-degree real audio. This 360-degree real audio allows, for example, users to enjoy videos, music, and other content with immersive sound inside the vehicle. Additionally, the location of obstacles and other hazardous objects surrounding the vehicle 1B can be indicated by the direction of sound output.

[0271] On the other hand, the seat speakers at each seat are used, for example, to output private sounds primarily for the individual passengers sitting in each seat. In other words, the sound output from each seat speaker is controlled individually.

[0272] Note that this speaker placement is just one example and can be changed. For example, the number of speakers placed on loop line L11 may be increased. For example, speakers may be placed on the dashboard at the front of vehicle 1B.

[0273] Furthermore, as shown in Figure 27, loop lights 654 are positioned slightly above loop line L11, at approximately the same height as the external loop line L1, and approximately parallel to loop line L11, surrounding the interior of the vehicle. Loop lights 654 are downlights equipped with multiple LEDs embedded in the interior of the vehicle in an approximately horizontal arrangement, and are mainly used as auxiliary lighting or interior lighting. Each LED can be individually controlled in terms of on / off, color, brightness, etc.

[0274] Furthermore, the Loop Light 654 does not necessarily need to surround the entire perimeter of the vehicle interior; it may surround only a portion of the interior without continuous coverage.

[0275] Furthermore, various operating devices are arranged along the loop line L11.

[0276] For example, as described above, the steering wheel 651 is positioned on the loop line L11 in front of the driver's seat 101.

[0277] Furthermore, as shown in Figure 30, a rod-shaped operating body, the stalk lever 655, is provided behind the steering wheel 651, extending from the steering column (not shown) in the vehicle width direction (lateral direction (rightward direction)). The stalk lever 655 is movable in the vertical direction, and by moving it in the vertical direction, the shift position of the vehicle 1B is switched. In other words, the stalk lever 655 constitutes a tilt-type shift lever that is movable in the vertical direction. Note that the stalk lever 655 may be either a straight type that moves straight in the vertical direction, or a column type that moves in a zigzag pattern in the vertical direction.

[0278] The shift positions set by the Stork Lever 655 are arranged from top to bottom as follows: R (Reverse), N (Neutral), D (Drive), and A (Automatic). That is, moving the Stork Lever 655 from top to bottom switches the shift positions in the order of R, N, D, A. Moving the Stork Lever 655 from bottom to top switches the shift positions in the order of A, D, N, R.

[0279] Furthermore, a button 656 is provided at the tip of the stalk lever 655, which can be pressed in the axial direction of the stalk lever 655. When the button 656 is pressed, the shift position of the vehicle 1B switches to P (parking).

[0280] Furthermore, an indicator 657 is provided circumferentially on the side of the stalk lever 655. The indicator 657 is positioned so that it can be seen from the driver's seat 101 through the gaps in the spokes of the steering wheel 651.

[0281] The indicator 657 changes color depending on the set shift position. For example, when the shift position is set to parking, the indicator 657 is red. When the shift position is set to drive, the indicator 657 is white. When the shift position is set to autopilot and autopilot is available, the indicator 657 is green. When the shift position is set to autopilot and autopilot is active, the indicator 657 is blue.

[0282] As shown by the dotted line in Figure 30, the indicator 657 may be positioned outside the outer circumference of the steering wheel 651 when viewed from the driver's seat 101. This makes the indicator 657 visible from outside the steering wheel 651. The indicator 657 may also be positioned at both the solid and dotted line positions.

[0283] In this way, by providing an indicator 657 on the operating mechanism (stork lever 655) used to switch the shift position, the driver can intuitively understand the meaning of the color of the indicator 657 and intuitively and reliably recognize the shift position.

[0284] Alternatively, the shift position setting may be displayed on the center display 201, for example. In this case, the shift positions will be displayed in the order of the setting direction by the stalk lever 655. That is, the shift positions will be displayed from top to bottom in the order of R (reverse), N (neutral), D (drive), and A (autonomous driving).

[0285] Furthermore, for example, a dial that rotates around an axis (circumferentially) may be provided at the axial end or middle of the stalk lever 655, so that the shift position can be switched by rotating the dial.

[0286] Furthermore, as shown in Figure 28, the air vents 658FC, 658FL, and 658FR of the air conditioner (A / C) are located on the loop line L11. Specifically, air vent 658FR is located between the driver's seat 101 and the passenger seat 102, directly below the center display 201. Air vent 658FL is located near the joint of the door 611FL on the driver's seat 101 side, directly below the center display 201. Air vent 658FR is located near the joint of the door 611FR on the passenger seat 102 side, directly below the center display 201.

[0287] Furthermore, as shown in Figure 29, the air outlet 658FC is provided with knobs 659CL and 659CR for changing the airflow direction. As shown in Figure 29, the air outlet 658FL is provided with knob 659FL for changing the airflow direction. Although not shown in the illustration, the air outlet 658FR is provided with knob 659FR for changing the airflow direction, in the same position as knob 659FL on the air outlet 658FL. Thus, knobs 659CL, 659CR, 659FL, and 659FR are provided on the loop line L11.

[0288] Furthermore, as shown in Figure 29, an operating unit 660 equipped with switches and the like for making various settings for the headlights 603 is located on the loop line L11, slightly to the right and behind the steering wheel 651.

[0289] Furthermore, door openers are positioned on the loop line L11. For example, as shown in Figure 28, the door opener 661FL is positioned on the loop line L11 near the center in the front-to-rear direction of the door 611FL on the driver's side 101. Similarly, the door opener 661FR is positioned on the loop line L11 near the center in the front-to-rear direction of the door 611FR on the passenger side 102.

[0290] In this way, by arranging various user interface devices along the loop line L11, distracting noise is removed from the driver's line of sight, providing an environment that makes it easier to concentrate on driving. Furthermore, by concentrating the various devices near the loop line L11, it becomes possible to intuitively recognize and operate the location of the various devices. In addition, the concentration of various devices near the loop line L11 and the placement of the loop light 654 give the impression that the interior of vehicle 1B is being watched over. Moreover, the vertical movement of the driver's and other passengers' eyes is reduced, which helps to suppress motion sickness among passengers.

[0291] Figure 31 is a view of the driver's seat 101 from the left rear.

[0292] The tablet device 662L is mounted on the back of the driver's seat 101, more specifically, on the back of the headrest of the driver's seat 101.

[0293] The tablet terminal 662L, for example, constitutes the rear entertainment display 206 shown in Figure 9, and presents infotainment-related information to passengers in the rear seats 103L and accepts their input regarding the presented information. Furthermore, the tablet terminal 662L can display alerts in emergencies or dangerous situations.

[0294] Although not shown in the diagram, for example, a tablet terminal 662R similar to the tablet terminal 662L is provided on the back of the passenger seat 102.

[0295] Alternatively, for example, ToF cameras may be installed near tablet terminals 662L and 662R. This would allow for the recognition of passengers operating tablet terminals 662L and 662R based on images captured by the ToF cameras.

[0296] Next, with reference to Figures 32 to 36, examples of installation positions for a ToF camera for photographing the direction of the driver's seat 101 (the driver sitting in the driver's seat 101) will be described.

[0297] As shown in Figures 32 and 33, the ToF camera is installed, for example, at installation position P1 around the digital rear mirror 204.

[0298] The digital rearview mirror 204 is used to check the area behind the vehicle 1B in place of a conventional rearview mirror, and is also called a smart rearview mirror or digital rearview mirror. As shown in Figure 28, the digital rearview mirror 204 is mounted at the upper end and near the center of the windshield 104, similar to a conventional rearview mirror, slightly behind the windshield 104, and positioned above the center of the center display 201. Also, as shown in Figures 32 and 33, the digital rearview mirror 204 is mounted near the ceiling via a mounting part 663, diagonally to the right and in front of the driver's seat 101.

[0299] The installation position P1 is located on the ceiling of the vehicle 1B, near the rear end of the left side of the mounting portion 663, and near the upper left of the digital rearview mirror 204.

[0300] Figure 34 schematically shows an example of an image captured by a ToF camera when it is installed at installation position P1.

[0301] By installing the ToF camera at installation position P1, the upper body of the driver 681 sitting in the driver's seat 101 is captured from a viewpoint looking down from the right front and above. This allows the driver's face to be captured from the right front and above, enabling the recognition unit 73 (Figure 1) to recognize the driver's gaze, for example. In addition, the driver's upper body, including the steering wheel 651, can be captured from an overhead view, enabling the recognition unit 73 to recognize the driver's posture, for example. Furthermore, because the probability of obstacles appearing between the driver 681 and the digital rearview mirror 204 is low, the driver's gaze and posture can be reliably recognized.

[0302] On the other hand, Figure 35 shows an example of an image captured by a ToF camera when the ToF camera is placed on or near the steering wheel 651.

[0303] In this case, the area around the driver's face is photographed from a forward-looking perspective. Therefore, the recognition unit 73 can recognize the driver's gaze. However, it is anticipated that obstacles such as the driver's hands are likely to appear between the driver 681 and the steering wheel 651, making it difficult to recognize the gaze. In addition, since the entire upper body of the driver 681 is not photographed, it becomes difficult for the recognition unit 73 to recognize the driver's posture.

[0304] Figure 36 shows a modified example of the installation position of the ToF camera.

[0305] For example, a ToF camera may be installed at the leftmost mounting position P2 of the digital rearview mirror 204.

[0306] For example, a ToF camera may be installed on the ceiling of vehicle 1B at installation position P3, which is behind the digital rear mirror 204 (in front of vehicle 1B from the digital rear mirror 204) and closer to the driver's seat 101 than the digital rear mirror 204 (to the left of the digital rear mirror 204).

[0307] For example, a ToF camera may be installed on the ceiling of vehicle 1B at installation position P4, which is in front of the digital rear mirror 204 (behind vehicle 1B from the digital rear mirror 204) and closer to the driver's seat 101 than the digital rear mirror 204 (to the left of the digital rear mirror 204).

[0308] Regardless of whether the ToF camera is installed at installation position P2 through P4, it will be possible to recognize the driver's gaze and posture based on the images captured by the ToF camera, just as when it is installed at installation position P1.

[0309] Alternatively, a conventional optical mirror may be used instead of the digital rearview mirror 204. Furthermore, a camera of a different type than the ToF camera may be used.

[0310] Furthermore, for example, a ToF camera that photographs the passenger sitting in the passenger seat 102 may be installed in a position symmetrical to the camera that photographs the driver.

[0311] <Example configuration of Information Processing Unit 701> Figure 37 is a block diagram showing an example configuration of the information processing unit 701 realized by the processor 21, HMI 31, etc., shown in Figure 1.

[0312] The information processing unit 701 comprises an image processing unit 711, a metadata assignment unit 712, and an output control unit 713.

[0313] The image processing unit 711 performs various image processing and editing on the video data obtained by the camera 51. The image processing unit 711 also converts sensor data obtained by optical sensors other than the camera, such as the radar 52 and LiDAR 53, into images.

[0314] The metadata assignment unit 712 assigns metadata to the video data obtained by the camera 51.

[0315] The output control unit 713 controls the output of visual, auditory, and tactile information from the HMI 31. For example, the output control unit 713 controls the output of visual information from the center display 201, tablet terminals 662L and 662R, and the illumination 652 of the steering wheel 651. For example, the output control unit 713 controls the output of auditory information (sound) from speakers 653FL to 653BR and the seat speakers.

[0316] <Operation of Vehicle 1B> Next, an example of the operation of vehicle 1B will be described with reference to Figures 38 to 53.

[0317] In the following, when there is no need to distinguish between door 611FL, door 611FR, door 611BL, and door 611BR individually, they will simply be referred to as door 611.

[0318] <Lighting System Emission Pattern> First, with reference to Figures 38 to 47, an example of the light emission pattern of the lighting system of vehicle 1B will be described.

[0319] The light control unit 85 (Figure 1) performs coordinated control of the exterior lights and interior lights of the vehicle.

[0320] Specifically, Figures 38 to 47 show examples of the light emission patterns of accessory light 601L, accessory light 601R, daytime running light 602L, daytime running light 602R, headlight 603L, headlight 603R, turn signal 604L, turn signal 604R, auxiliary light 605FL, auxiliary light 605FR, auxiliary light 605BL, auxiliary light 605BR, taillight 606L, taillight 606R, taillight 606CL, taillight 606CR, brake light 607L, brake light 607R, loop light 654, and main light 751.

[0321] The main light 751 consists of LEDs arranged almost along the outer edge of the ceiling inside vehicle 1B, and is used as the main lighting for the interior. Each LED can be individually controlled for on / off, color, brightness, etc.

[0322] Furthermore, in Figures 38 to 47, headlights 603LU and 603LD, headlights 603RU and 603RD, brake lights 607LU and 607LD, and brake lights 607RU and 607RD are not distinguished from each other but are shown together as a single unit.

[0323] Unless otherwise specified, the accessory lights 601L, 601R, daytime running lights 602L, 602R, headlights 603L, 603R, turn signals 604L, 604R, auxiliary lights 605FL, 605FR, 605BL, and 605BR shall emit white light. Unless otherwise specified, the taillights 606L, 606R, 606CL, 606CR, brake lights 607L, and 607R shall emit red light. Unless otherwise specified, the loop light 654 and main light 751 shall emit orange light.

[0324] Furthermore, in the following diagrams, illuminated lights will be colored black, and unilluminated lights will be colored white.

[0325] <When powered on> Figures 38 and 39 show examples of the illumination patterns when vehicle 1B is powered on. Figure 38 shows the state when all lights that are to be illuminated when the power is turned on are lit. Figure 39 shows the final state of the lights when the power is turned on.

[0326] When vehicle 1B's power is turned off, all lights are off.

[0327] When the power of vehicle 1B is turned on, the accessory lights 601L, 601R, daytime running lights 602L, 602R, turn signals 604L, 604R, auxiliary lights 605FL, 605FR, 605BL, 605BR, taillights 606L, 606R, 606CL, and 606CR illuminate.

[0328] At this time, as indicated by arrows A11 and A12 in the diagram, the lights illuminate so that light flows around the body from the center of the front to the center of the back.

[0329] Specifically, the logos on the right edge of the accessory light 601L and the left edge of the accessory light 601R will light up for a short time first.

[0330] Next, the LEDs of accessory light 601L light up sequentially from right to left. Next, the LEDs of daytime running light 602L light up sequentially from right to left. Next, the LEDs of turn signal 604L light up sequentially from front to rear. Next, the LEDs of auxiliary light 605FL light up sequentially from front to rear. Next, the LEDs of auxiliary light 605BL light up sequentially from front to rear. Next, the LEDs of tail light 606L light up sequentially from left to right. Finally, the LEDs of tail light 606CL light up sequentially from left to right. As a result, each light illuminates in a counterclockwise direction from the center of the front of the body to the center of the rear, as indicated by arrow A11.

[0331] Similarly, the LEDs of accessory light 601R illuminate sequentially from left to right. Next, the LEDs of daytime running light 602R illuminate sequentially from left to right. Next, the LEDs of turn signal 604R illuminate sequentially from front to rear. Next, the LEDs of auxiliary light 605FR illuminate sequentially from front to rear. Next, the LEDs of auxiliary light 605BR illuminate sequentially from front to rear. Next, the LEDs of tail light 606R illuminate sequentially from right to left. Finally, the LEDs of tail light 606CR illuminate sequentially from right to left. As a result, each light illuminates in a clockwise direction from the center of the front of the body to the center of the rear, as indicated by arrow A12.

[0332] The lights for both the left-turning and right-turning directions are illuminated simultaneously.

[0333] Next, the loop light 654 and the main light 751 are turned on. At this time, the loop light 654 and the main light 751 are turned on so that their overall brightness gradually increases.

[0334] When the loop light 654 and the main light 751 have finished turning on, the state will be as shown in Figure 38.

[0335] Next, as shown in Figure 39, some of the lights turn off. At this time, as indicated by arrows A13 and A14 in the figure, the lights turn off sequentially from the front of the left side and the front of the right side of the body to the center of the back.

[0336] Specifically, the LEDs in the turn signal 604L turn off sequentially from the front to the rear. Next, the LEDs in the auxiliary light 605FL turn off sequentially from the front to the rear. Next, the LEDs in the auxiliary light 605BL turn off sequentially from the front to the rear. Next, the LEDs in the taillight 606L turn off sequentially from the left end to the right end. Finally, the LEDs in the taillight 606CL turn off sequentially from the left end to the right end. As a result, as indicated by arrow A13, each light turns off sequentially in a counterclockwise direction from the front of the left side of the body to the center of the rear.

[0337] Similarly, the LEDs of the turn signal 604R are turned off sequentially from the front to the rear. Next, the LEDs of the auxiliary light 605FR are turned off sequentially from the front to the rear. Next, the LEDs of the auxiliary light 605BR are turned off sequentially from the front to the rear. Next, the LEDs of the taillight 606R are turned off sequentially from the right end to the left end. Finally, the LEDs of the taillight 606CR are turned off sequentially from the right end to the left end. As a result, each light is turned off sequentially in a clockwise direction from the front of the right side of the body to the center of the rear, as indicated by arrow A14.

[0338] The left-hand and right-hand rotating lights on this device will be switched off simultaneously.

[0339] <While driving and with headlights 603 turned off> Figure 40 shows an example of the light emission pattern when vehicle 1B is being driven (shift position is set to driving or autonomous driving) and the headlights 603 are turned off.

[0340] In this state, the accessory light 601L, accessory light 601R, daytime running light 602L, and daytime running light 602R will be illuminated.

[0341] The daytime running lights 602L and 602R are required to remain illuminated while vehicle 1B is in operation due to legal regulations and other restrictions. On the other hand, the accessory lights 601L and 601R must be dimmed or turned off while vehicle 1B is in operation due to legal regulations and other restrictions. Therefore, the brightness of the accessory lights 601L and 601R is set lower than normal.

[0342] <While driving and with headlights 603 on> Figure 41 shows an example of a light emission pattern when vehicle 1B is being driven (shift position is set to driving or autonomous driving) and the headlights 603 are illuminated.

[0343] Comparing Figure 41 with Figure 40, the difference is that headlights 603L and 603R, taillights 606L, 606R, 606CL, and 606CR are illuminated. In other words, taillights 606L, 606R, 606CL, and 606CR illuminate in conjunction with the illumination of headlights 603L and 603R.

[0344] <When the brakes are applied> Figure 42 shows an example of the light emission pattern when the brakes are applied.

[0345] Comparing the state in Figure 42 with the state in Figure 40, the difference is that brake lights 607L and 607R are illuminated. In other words, when the brakes are applied, brake lights 607L and 607R illuminate.

[0346] <When the turn signal is activated> Figure 43 shows an example of the light emission pattern when the left turn signal is activated.

[0347] Comparing Figure 43 with Figure 40, the differences are that the daytime running lights 602L and taillights 606L change color, and the turn signals 604L flash.

[0348] Specifically, the LEDs in the daytime running light 602L change from white to orange sequentially from right to left. After all the LEDs in the daytime running light 602L have changed to orange, the LEDs in the turn signal 604L light up in orange sequentially from front to rear.

[0349] In parallel with this, the LEDs in the 606L taillight change sequentially from white to orange, from right to left.

[0350] Next, the LEDs in the daytime running light 602L change from orange to white sequentially, from right to left. After all the LEDs in the daytime running light 602L have changed to white, the LEDs in the turn signal 604L turn off sequentially, from front to rear.

[0351] In parallel with this, the LEDs in the 606L taillight change sequentially from orange to white, from right to left.

[0352] The same process is repeated below.

[0353] <If door 611 is open> Figure 44 shows an example of the light emission pattern when one of the doors 611 of vehicle 1B is open.

[0354] At this time, the accessory light 601L, accessory light 601R, day running light 602L, day running light 602R, tail light 606L, tail light 606R, tail light 606CL, tail light 606CR, loop light 654, and main light 751 are illuminated.

[0355] <When door 611 is closed> Figure 45 shows an example of the light emission pattern when the state changes from one of the doors 611 of vehicle 1B being open to all of the doors 611 being closed.

[0356] Comparing the state in Figure 45 with the state in Figure 46, the difference is that the main light 751 is off. In other words, the main light 751 is turned off when the door 611 is closed.

[0357] <When parking> Figure 46 shows an example of the illumination pattern when the shift position of vehicle 1B is set to parking.

[0358] In this case, the accessory light 601L, accessory light 601R, daytime running light 602L, daytime running light 602R, turn signal 604L, turn signal 604R, tail light 606L, tail light 606R, tail light 606CL, tail light 606CR, and loop light 654 will illuminate.

[0359] <When powered off> Figure 47 shows an example of the illumination pattern when the power is turned off while the shift position of vehicle 1B is set to parking.

[0360] In this case, the accessory light 601L, accessory light 601R, daytime running light 602L, daytime running light 602R, turn signal 604L, turn signal 604R, tail light 606L, tail light 606R, tail light 606CL, tail light 606CR, and loop light 654 will be turned off.

[0361] At this time, as indicated by arrows A15 and A16, the lights turn off in a flowing manner from the side of the body to the center of the front. Also, as indicated by arrows A17 and A18, the lights turn off in a flowing manner from the side of the body to the center of the back.

[0362] Specifically, the LEDs in the turn signal 604L are turned off sequentially from the rear to the front. Next, the LEDs in the daytime running light 602L are turned off sequentially from the left to the right. Then, the LEDs in the accessory light 601L are turned off sequentially from the left to the right.

[0363] In parallel with this, the LEDs of the turn signal 604R turn off sequentially from the rear to the front. Next, the LEDs of the daytime running light 602R turn off sequentially from the right end to the left end. Then, the LEDs of the accessory light 601R turn off sequentially from the right end to the left end.

[0364] Simultaneously, the LEDs in taillight 606L are turned off sequentially from left to right. Next, the LEDs in taillight 606CL are turned off sequentially from left to right.

[0365] Simultaneously, the LEDs in taillight 606R are turned off sequentially from right to left. Next, the LEDs in taillight 606CR are turned off sequentially from right to left.

[0366] Also, the Loop Light 654 gradually turns off.

[0367] <When a person approaches vehicle 1B> Next, although not shown in the diagram, an example of a light emission pattern when the recognition unit 73 detects a person approaching vehicle 1B will be described.

[0368] For example, a light located near the approaching person (near the approaching location) will illuminate or flash. Specifically, for example, when the approach of a person is detected near door 611FL, the turn signal 604L and auxiliary light 605FL near door 611FL will illuminate or flash. Furthermore, for example, after the turn signal 604L and auxiliary light 605FL illuminate, other lights may also illuminate sequentially.

[0369] Furthermore, for example, if a person's approach is detected, the lights may always illuminate or flash in the same pattern, regardless of the person's position. For example, the lights may illuminate in a way that creates a flowing light around the vehicle.

[0370] Furthermore, the manner in which the lights turn on or flash (e.g., color, pattern, etc.) may change depending on the person approaching. For example, if the approaching person is recognized as a pre-registered user (e.g., the driver or their family), a white light may flash. On the other hand, if the recognition of the approaching person fails (if the approaching person is not recognized as a pre-registered user), a red light may flash.

[0371] For example, the body control unit 84 may be configured to automatically unlock the doors when it detects the approach of a registered user.

[0372] Furthermore, the conditions under which the lights are turned on or flashed in response to a person's approach may be limited. For example, the lights may be turned on or flashed in response to a person's approach only when the power of vehicle 1B is off, or when there is no one inside vehicle 1B.

[0373] In this way, when approaching persons is detected, for example, by turning on or flashing the external lights, it can be made clear to those around that vehicle 1B is monitoring its surroundings, thereby preventing theft or vandalism of vehicle 1B. Furthermore, when approaching persons is detected, for example, by turning on or flashing the interior lights of the vehicle, it can be made clear to those around that vehicle 1B is also monitoring its interior.

[0374] <Lighting pattern of illumination 652 for steering wheel 651> Next, we will describe the illumination pattern of the steering wheel 651's illumination 652.

[0375] For example, the light control unit 85 lights up or flashes the illumination 652 in a pattern appropriate to the situation, based on at least one of the following: the situation of the vehicle 1B, the situation around the vehicle 1B, and the situation of the passengers. The light emission pattern of the illumination 652 is defined by at least one of the following: color, brightness, flashing pattern, light movement, and light emission area.

[0376] Specifically, for example, when vehicle 1B is ready for autonomous driving, several short bands of light rotate approximately once around illumination 652. Then, the entire illumination 652 flashes. This allows the driver to reliably recognize that autonomous driving is ready.

[0377] Furthermore, for example, when vehicle 1B changes lanes using autonomous driving, the entire illumination 652 lights up first. After illumination 652 turns off, the LEDs of illumination 652 in the direction that vehicle 1B is moving to change lanes flash. Then, after the lane change is completed, the entire illumination 652 lights up and then turns off. In this way, lane change notifications are provided from before the lane change until the lane change is completed, giving passengers a sense of security.

[0378] Furthermore, for example, if the driver is approaching an obstacle or falling asleep at the wheel, the top of the illumination 652 will flash red to alert them. This allows the driver to quickly detect and avoid danger.

[0379] Furthermore, for example, when voice recognition is in operation, multiple bands of light move from side to side above the illumination 652. This allows the driver to recognize that the voice recognition function is working correctly.

[0380] <Image Processing> Next, the imaging process performed by vehicle 1B will be explained with reference to the flowchart in Figure 48.

[0381] This process starts, for example, when the power of vehicle 1B is turned on, and ends when the power of vehicle 1B is turned off.

[0382] In step S1, camera 51 photographs the surroundings and interior of vehicle 1B. Specifically, camera 51 is composed of multiple cameras, each camera photographing the surroundings or interior (room) of vehicle 1B. Each camera supplies the video data obtained from the photography to the information processing unit 701.

[0383] In step S2, the image processing unit 711 performs image processing as necessary.

[0384] For example, the image processing unit 711 superimposes visual information onto each frame of the video data, as will be described later with reference to Figures 49 to 52.

[0385] Furthermore, for example, the image processing unit 711 performs image processing such as noise cancellation on each frame of the video data based on sensor data from the in-vehicle sensor 26 and the vehicle sensor 27. For example, if it is raining, the image processing unit 711 removes noise such as raindrops in the frame.

[0386] In step S3, the metadata assignment unit 712 assigns metadata to the video data.

[0387] For example, the metadata assignment unit 712 assigns metadata relating to at least one of the following to each video data taken while in motion, such as the shooting location, shooting date and time, the condition of the vehicle 1B, the condition inside the vehicle, and the surrounding conditions.

[0388] Metadata related to the shooting location includes, for example, at least one of the following: the position of vehicle 1B at the time of shooting, the direction of shooting, and the position of the camera used for shooting.

[0389] Metadata related to the date and time of shooting includes, for example, at least one of the following: year, month, and day, time, etc.

[0390] Metadata regarding the status of vehicle 1B includes, for example, at least one of the following: speed, acceleration, direction of travel, destination, and the status of vehicle 1B (e.g., presence or absence of malfunction, presence or absence of accident, charge level, etc.).

[0391] Metadata regarding the conditions inside the vehicle includes, for example, passenger identification information (e.g., name or ID), passenger seat location, passenger status (e.g., activities, napping, etc.), speech recognition results of conversations inside the vehicle, and activity level inside the vehicle. Activity level is set based on, for example, the volume of conversations inside the vehicle and passenger movements.

[0392] Metadata regarding the surrounding environment includes, for example, at least one of the following: weather, temperature, humidity, brightness, location and type of surrounding objects (e.g., other vehicles, pedestrians, obstacles, traffic signs, landmarks, etc.), and whether or not an event (e.g., accident, construction, etc.) has occurred and its type.

[0393] Furthermore, the metadata assignment unit 712 may, for example, assign metadata entered by the user (passenger) to the video data. For example, a movie button may be provided on the steering wheel 651, and the driver may press the movie button when a scene that they want to save as video data is captured. Examples of situations where a scene that they want to save as video data is captured include, for example, when the scenery around the vehicle 1B is beautiful, or when an accident or other trouble occurs. For example, the metadata assignment unit 712 may, using the movie button, assign metadata to video data captured during a period specified by the user indicating that it needs to be saved.

[0394] The unit to which metadata is assigned can be set arbitrarily. For example, metadata can be assigned on a frame-by-frame basis, on a unit consisting of multiple frames, or on a video data basis. Furthermore, the unit to which metadata is assigned may be variable depending on the type of metadata.

[0395] In step S4, vehicle 1B stores video data. Specifically, the image processing unit 711 causes the recording unit 28 to store the video data.

[0396] In addition, at this time, audio data recorded around and inside vehicle 1B may be stored together with video data.

[0397] In step S5, vehicle 1B displays a video. Specifically, the output control unit 713 causes the HMI 31 to display a video based on the video data.

[0398] For example, the videos shown in Figures 49 to 52 are displayed on the center display 201.

[0399] Figure 49 shows an example of a video displayed when the display mode is set to object detection mode. In this example, a vehicle and a person are selected as the detection targets. A frame 801 surrounding the detected vehicle and a frame 802 surrounding the person are displayed in the video, which was filmed in front of vehicle 1B. In addition, the words (CAR and PERSON) indicating the type of object detected are displayed within frames 801 and 802.

[0400] Figure 50 shows an example of a video that is displayed when the display mode is set to prediction mode. In this example, the predicted position of a moving device (in this example, a person) detected in the video 0.5 seconds later is displayed. In this example, the current position of the person is shown by a dotted line, and the predicted position of the person 0.5 seconds later is shown by a solid line and enclosed by a frame 811.

[0401] Figure 51 shows an example of a video displayed when the display mode is set to search mode. Here, an example of searching for and displaying a parking space is shown. Specifically, a frame 821 indicating a parking space is displayed.

[0402] For example, multiple parking locations may be displayed. Alternatively, parking locations may be displayed using a bird's-eye view.

[0403] Figure 52 shows an example of a video displayed when the display mode is set to search mode. This example shows how to search for pre-registered people and display the detected people. Specifically, a frame 831 is displayed in the video surrounding the pre-registered people. The name of the detected person (Sakura) is also displayed in frame 831.

[0404] For example, the center display 201 may display a video of the direction selected by the driver or other passenger (e.g., forward, backward, left, right, etc.). Alternatively, the center display 201 may display a 360-degree video of the area around the vehicle 1B while scrolling.

[0405] Furthermore, for example, sensor data obtained by radar 52 or LiDAR 53 may be converted into images and displayed on the center display 201. Alternatively, visual information based on data obtained by two or more sensors from camera 51, radar 52, and LiDAR 53 may be superimposed on the video and displayed.

[0406] Furthermore, for example, the planned direction of travel may be superimposed and displayed on the video based on the route plan.

[0407] Furthermore, for example, the movement of other vehicles, predicted based on information obtained through vehicle-to-vehicle communication, may be superimposed and displayed on the video.

[0408] After that, the process returns to step S1, and steps S1 through S5 are repeatedly executed.

[0409] <Video editing process> Next, with reference to Figure 53, the video editing process performed by vehicle 1B will be described.

[0410] This process begins, for example, when an instruction to edit a video is input to the HMI31. The instruction to edit a video includes, for example, the period during which the video to be edited was filmed (hereinafter referred to as the editing period), and the conditions for extracting frames to be included in the video.

[0411] In step S51, the image processing unit 711 performs video editing.

[0412] For example, the image processing unit 711 extracts frames from each video data (hereinafter referred to as "recorded video data") taken within the editing period (hereinafter referred to as "recorded video data") based on metadata, etc., to be included in the edited video data (hereinafter referred to as "edited video data").

[0413] For example, the image processing unit 711 extracts frames that have metadata indicating that they need to be saved.

[0414] Furthermore, for example, the image processing unit 711 extracts frames that satisfy the given conditions based on the metadata.

[0415] For example, the image processing unit 711 extracts frames based on the route of vehicle 1B within the editing period and metadata. For example, the image processing unit 711 detects the route of vehicle 1B within the editing period based on location information contained in the map data and the metadata of the video data. Then, for example, if vehicle 1B was traveling on a road along the coast, the image processing unit 711 prioritizes extracting frames that show the direction of the sea. For example, if vehicle 1B was traveling on a high place such as a mountain, the image processing unit 711 prioritizes extracting frames that show the direction looking down on the surroundings. Furthermore, for example, the image processing unit 711 prioritizes extracting frames that show landmarks around vehicle 1B based on the direction of travel and turning direction of vehicle 1B.

[0416] For example, the image processing unit 711 extracts frames from when a specific event occurred, based on metadata. For instance, if vehicle 1B is involved in an accident, or if an accident occurs around vehicle 1B, the image processing unit 711 prioritizes extracting frames from the time period before and after the accident. Alternatively, for example, the image processing unit 711 prioritizes extracting frames taken in the direction from which the accident occurred.

[0417] For example, if the image processing unit 711 detects a surge in activity inside the vehicle based on the activity level contained in the metadata, it prioritizes extracting frames that capture the interior of the vehicle. Furthermore, the image processing unit 711 also extracts audio data corresponding to the extracted frames of the vehicle interior.

[0418] Alternatively, for example, the output control unit 713 may display a list of video data or the actual video on the center display 201 and the tablet terminal 662L, and the image processing unit 711 may edit the video based on user instructions.

[0419] For example, the user may be allowed to select video data or frames that they want to include in the edited video data. In this case, for example, various types of information, visual effects, and visual information such as doodles may be superimposed on the video frames displayed on the center display 201 and the tablet terminal 662L.

[0420] The image processing unit 711 then generates edited video data, for example, by combining the extracted frames. For example, the image processing unit 711 generates edited video data by connecting the extracted frames in chronological order, or by arranging or overlapping multiple frames within the same frame. In addition, the image processing unit 711 superimposes visual information onto the extracted frames as needed, for example, based on metadata.

[0421] In step S52, the vehicle 1B saves video data. For example, the image processing unit 711 stores the edited video data in the recording unit 28. Alternatively, for example, the image processing unit 711 transmits the edited video data via the communication unit 22 to a server or an information processing terminal (e.g., a smartphone, tablet, personal computer, etc.) owned by the passenger for storage.

[0422] After that, the video editing process is complete.

[0423] In this way, videos taken of the surroundings and interior of vehicle 1B can be easily edited. This makes it easy to generate video data such as travel memories, videos with scenic views, and videos recording accident details.

[0424] <<5. Modified Examples of the Second Embodiment>> The following describes a modified version of the second embodiment of the present technology described above.

[0425] For example, the recorded video data with pre-edited metadata attached may be stored or copied outside of vehicle 1B, and the video may be edited by an external device (e.g., a server (cloud), smartphone, tablet, personal computer, etc.). Alternatively, the video may be edited jointly by the external device and vehicle 1B. Furthermore, the metadata may be added by an external device.

[0426] Furthermore, while the above explanation shows an example where the front and tail lines of the loop line L1 are illuminated, it is also possible to choose not to illuminate them, for example, considering design or legal regulations.

[0427] Furthermore, while the above explanation shows examples where the central part of the front line and the central part of the tail line are divided, it is also possible to connect them without dividing them, for example, taking design into consideration.

[0428] Alternatively, optical sensors such as cameras, radar, and LiDAR may be placed inside the headlight 603.

[0429] Furthermore, although the above explanation uses the example of a left-hand drive vehicle 1B, this technology can of course be applied to right-hand drive vehicles as well. When applying this technology to a right-hand drive vehicle, the exterior and interior layouts described above will be appropriately modified to suit a right-hand drive vehicle.

[0430] Furthermore, the type of vehicle to which this technology can be applied is not particularly limited. In addition, this technology can be applied to other mobile devices besides vehicles, such as personal mobility devices, airplanes, ships, construction machinery, and agricultural machinery. Mobile devices to which this technology can be applied also include, for example, drones and robots that perform ambient photography without a human on board.

[0431] <<6. Third Embodiment>> Next, a third embodiment of the present technology will be described with reference to Figures 54 to 105.

[0432] In the figures of the third embodiment, the same reference numerals are used for parts corresponding to those in the first and second embodiments, and their descriptions are omitted as appropriate. Furthermore, any configurations present in the first and second embodiments that are not shown or described in the third embodiment are also present in the third embodiment, unless an alternative configuration is shown or described, or unless it is explained that such configuration will be omitted.

[0433] <Example of interior configuration for vehicle 1C> First, we will explain an example of the interior configuration of vehicle 1C with reference to Figures 54 to 57.

[0434] Figure 54 is a schematic diagram of the interior of vehicle 1C viewed from the right. Figure 55 is a schematic diagram showing the front of the interior of vehicle 1C. Figure 56 is a schematic diagram showing the front of the driver's seat 101 inside the interior of vehicle 1C. Figure 57 is a magnified view of the area around the steering wheel 1001 in vehicle 1C.

[0435] In Figure 54, the parts of vehicle 1C that differ significantly from vehicles 1A and 1B are circled. Specifically, the steering wheel 1001, camera module 1007, and camera module 1008 are the parts that differ significantly.

[0436] As shown in Figure 57, the steering wheel 1001 differs from the steering wheel 651 in Figure 30 in the position of the illumination 1002. The illumination 1002 comprises multiple LEDs arranged in a ring shape at a predetermined distance inward from the outer circumference of the center of the steering wheel 1001. Each LED can be individually controlled for on / off, color, brightness, etc. Therefore, the illumination 1002 has variable color, brightness, and light-emitting area (range of illumination). The logo of vehicle 1C is formed inside the illumination 1002.

[0437] An airbag (not shown) is housed in the center of the steering wheel 1001. The illumination 1002 is positioned to avoid the area where the center of the steering wheel 1001 would split when the airbag deploys. This prevents fragments or harmful substances from the illumination 1002 from scattering when the airbag deploys.

[0438] The left spoke of the steering wheel 1001 is provided with an operating section 1003 equipped with multiple buttons. The right spoke of the steering wheel 1001 is provided with an operating section 1004 equipped with multiple buttons.

[0439] As shown in Figure 57, a rod-shaped operating body, the stalk lever 1005, is provided behind the steering wheel 1001, extending to the left from the steering column (not shown). Additionally, a rod-shaped operating body, the stalk lever 1006, is provided behind the steering wheel 1001, extending to the right from the steering column (not shown). The stalk lever 1006 has a configuration similar to, for example, the stalk lever 655 in Figure 30.

[0440] As shown in FIGS. 54, 55, and 57, the camera module 1007 is provided on the dashboard 105, slightly closer to the driver's seat 101 than the center in the left - right direction. The camera module 1007 has a shape like a part of a truncated cone cut out vertically and has a vertical flat surface on its side. The camera module 1007 is oriented such that the flat surface on the side faces the direction of the driver's seat 101 and, as will be described later, captures an area that at least includes the head of the driver sitting in the driver's seat 101.

[0441] As shown in FIG. 55, the camera module 1008 is provided at the tip of the ceiling of the vehicle 1C and at the center in the left - right direction, near the digital rear - view mirror 204. The camera module 1008 includes two ToF cameras (not shown). As will be described later, each ToF camera captures the driver and the passenger in the passenger seat 102, respectively.

[0442] A circular light 1009 is provided on the lower surface of the camera module 1008. Also, the camera module 1008 incorporates a microphone (not shown) and the like for collecting the voices of passengers.

[0443] Also, in this example, as shown in FIG. 55, the central part 201C of the center display 201 is divided into a display part 201CL in front of the driver's seat 101, a display part 201CC between the driver's seat 101 and the passenger seat 102, and a display part 201CR in front of the passenger seat 102. As described above, it is also possible to connect the display parts 201CL, 201CC, and 201CR to form one display part. A display part 201LL is provided at the left end 201L of the center display 201. A display part 201RR is provided at the right end 201R of the center display 201.

[0444] <Example of Installation Position of ToF Camera> Next, referring to FIGS. 58 to 63, an example of the installation position of the ToF camera will be described.

[0445] Figure 58 shows examples of the installation positions of the ToF camera in camera module 1008. The upper left diagram of Figure 58 is a schematic view of camera module 1008 from the lower left. The lower left diagram of Figure 58 is a schematic view of camera module 1008 from the lower left. The right diagram of Figure 58 is a schematic view of camera module 1008 from the lower front.

[0446] The camera module 1008 has a shape in which a truncated cone is attached to the rear end (rear side of vehicle 1C) of a base that has been cut diagonally from both sides of a rectangular prism. As described above, the camera module 1008 is positioned at the front of the ceiling of vehicle 1C and in the center in the left-right direction.

[0447] A ToF camera (hereinafter referred to as ToF camera P101L) is built into the camera module 1008 at position P101L, near the front left end of the frustoconical portion. The optical axis of the ToF camera P101L is directed towards the midpoint between the eyes of a driver with a standard build, for example, when a driver of standard build is seated in the driver's seat 101, which is positioned in a standard location.

[0448] Figure 59 shows an example of the shooting range of the ToF camera P101L. Regardless of the driver DR1's physique and the position of the driver's seat 101, the ToF camera P101L can capture almost the entire body of the driver DR1, including from the head to near the feet, from the upper right.

[0449] A ToF camera (hereinafter referred to as ToF camera P101R) is built into the camera module 1008 at position P101R, near the front right end of the frustoconical portion. Although not shown in the diagram, ToF camera P101R, like ToF camera P101L, is capable of capturing almost the entire body of the passenger sitting in the passenger seat 102, including from the head to the feet, from an upper left angle, regardless of the passenger's physique or the position of the passenger seat 102.

[0450] A black, glossy cover 1008A is provided on the side of the truncated cone portion of the camera module 1008. Cover 1008A makes the lenses of the ToF cameras P101L and P101R less visible from the outside. In addition, it is possible to install cameras or sensors other than the ToF cameras P101L and P101R on the back of cover 1008A.

[0451] For example, DMS30 (Figure 1) recognizes the driver's posture and movement by performing skeletal recognition, etc., based on images from the ToF camera P101L. For example, DMS30 performs driver recognition processing (e.g., individual recognition) based on images from the ToF camera P101L.

[0452] Similarly, for example, DMS30 recognizes the posture and movement of the passenger in the passenger seat 102 by performing skeletal recognition of the passenger based on images from ToF camera P101R. For example, DMS30 performs passenger recognition processing based on images from ToF camera P101L.

[0453] Next, other examples of ToF camera installation locations will be described with reference to Figures 60 to 63.

[0454] For example, as shown in the upper diagram of Figure 60, a ToF camera (hereinafter referred to as ToF camera P101C) is built into the rear end position P101C of the camera module 1008.

[0455] The diagram below Figure 60 shows an example of the shooting range of the ToF camera P101C. As shown, the ToF camera P101C can capture the area A101L including the headrest of the rear seat 103L, and the area A101R including the headrest of the rear seat 103R. Therefore, the ToF camera P101C can capture the upper body including the head of the occupant in the rear seat 103L, and the upper body including the head of the occupant in the rear seat 103R, at the same time.

[0456] As shown in Figure 60, the aforementioned seat speakers are embedded below the headrest of each seat. Specifically, seat speaker 1021 is embedded below the headrest of the driver's seat 101. Seat speaker 1022 is embedded below the headrest of the passenger seat 102. Seat speaker 1023L is embedded below the headrest of the left rear seat 103L. Seat speaker 1023R is embedded below the headrest of the right rear seat 103R.

[0457] Although not shown in the illustrations, the speakers 653FL to 653BR described above are embedded in each door, as shown in Figures 27 and 28, thereby achieving 360-degree real audio.

[0458] Figures 61 to 63 show examples of installation locations for ToF cameras mounted on the rear seat 103L.

[0459] For example, a ToF camera (hereinafter referred to as ToF camera P111L) is installed on the ceiling of vehicle 1C at position P111L, just above the frame above door 611BL. The ToF camera P111L can, for example, photograph almost the entire body of a passenger in the rear seat 103L from a diagonal upper left direction. In addition, a reading light (not shown) capable of illuminating within range A111 is provided around the ToF camera P111L.

[0460] For example, a ToF camera (hereinafter referred to as ToF camera P112L) is installed at position P112L near the center of the width direction of the upper edge of the tablet terminal 662L. The ToF camera P112L can, for example, photograph the upper body of a passenger in the rear seat 103L from the front.

[0461] Although not shown in the diagram, ToF cameras are installed on the rear seat 103R at positions P111R and P112R, similar to positions P111L and P112L for the rear seat 103L. Hereinafter, the ToF camera installed at position P111R will be referred to as ToF camera P111R, and the ToF camera installed at position P112R will be referred to as ToF camera P112R.

[0462] For example, DMS30 recognizes the posture and movement of the passenger in the rear seat 103L by performing skeletal recognition of the passenger based on images from at least one of ToF cameras P111L and P112L. For example, DMS30 performs passenger recognition processing based on images from at least one of ToF cameras P111L and P112L. For example, DMS30 performs lip reading of the passenger based on images from ToF camera P112L.

[0463] Similarly, for example, DMS30 recognizes the posture and movement of the occupant in the rear seat 103R by performing skeletal recognition of the occupant based on images from at least one of ToF cameras P111R and P112R. For example, DMS30 performs occupant recognition processing based on images from at least one of ToF cameras P111R and P112R. For example, DMS30 performs lip reading of the occupant based on images from ToF camera P112L.

[0464] For example, as shown in Figure 61, the light control unit 85 controls the on / off status and brightness of the reading light based on the actions of the passenger in the rear seat 103L. Also, for example, the light control unit 85 controls the brightness of the illumination range A112 of the loop light 654 near the rear seat 103L based on the actions of the passenger in the rear seat 103L. For example, if the light control unit 85 detects that the passenger is reading documents, it will turn on the reading light or brighten the illumination range A112 of the loop light 654.

[0465] Furthermore, for example, the same lighting control as for the rear seat 103L is also performed for the passenger seat 102 and the rear seat 103R.

[0466] For example, as shown in Figure 62, the DMS 30 can recognize the position and shape of the left ear EL1 (not shown) and right ear ER1 of the passenger DR2 in the rear seat 103L based on the image from the ToF camera P111L. In response, the audio control unit, which is part of the functions of the HMI 31, controls the seat speaker 1023L in the rear seat 103L to control the sound image, sound field, and volume of various sounds directed to the passenger DR2, in accordance with the position and shape of the passenger DR2's ears.

[0467] Furthermore, in the driver's seat 101, passenger seat 102, and rear seat 103R, for example, various audio controls such as sound image, sound field, and volume are performed for the occupants of each seat, similar to the rear seat 103L.

[0468] For example, as shown in the lower right diagram of Figure 63, the body control unit 84 controls the position and orientation of the rear seat 103R and the position and orientation of the tablet terminal 662R based on the physique of the passenger in the rear seat 103R. For example, the rear seat 103R is reclined deeply to accommodate the passenger's physique, and the tablet terminal 662R moves toward the rear seat 103R, as indicated by arrow A113.

[0469] Similarly, the position and orientation of the rear seat 103L and the tablet terminal 662L are controlled based on the physique of the passenger in the rear seat 103L.

[0470] <Details of Camera Module 1007> Next, the details of the camera module 1007 will be described with reference to Figures 64 to 67.

[0471] Figure 64 shows an example of the installation location of the camera module 1007. As described above, the camera module 1007 is installed on the dashboard 105, slightly closer to the driver's seat 101 than the center in the left-right direction.

[0472] Figure 65 shows an example of the internal configuration of the camera module 1007.

[0473] The camera module 1007 comprises a housing 1041, a camera 1042, LEDs 1043-1 and 1043-2, and a cover 1044.

[0474] In the following, when it is not necessary to distinguish between LED1043-1 and LED1043-2 individually, they will simply be referred to as LED1043.

[0475] Camera 1042, as well as LEDs 1043-1 and 1043-2, are arranged within the housing 1041 so as to be aligned horizontally. Camera 1042 is positioned closer to the driver's seat 101 than LEDs 1043-1 and 1043-2. ​​The optical axes of camera 1042, as well as LEDs 1043-1 and 1043-2, are approximately parallel and each points towards the driver's seat 101. Camera 1042 is capable of capturing images of at least the area from the top of the driver's head to the Adam's apple, regardless of the driver's physique or the position of the driver's seat 101.

[0476] Camera 1042 is a camera capable of two types of shooting: RGB shooting (color shooting) and IR shooting (infrared shooting). Camera 1042 automatically switches between RGB shooting and IR shooting based on conditions such as ambient brightness.

[0477] LED1043 is a light source that emits IR light (infrared light). By providing two LED1043s, LED1043-1 and LED1043-2, it is possible to suppress the amount of IR light emitted by each LED1043 while maintaining sufficient light output, thereby suppressing the heat generated by each LED1043.

[0478] The cover 1044 is located on the flat side of the housing 1041, in front of the camera 1042 and each LED 1043. The cover 1044 makes the lens of the camera 1042 and the light-emitting parts of each LED 1043 less visible from the outside.

[0479] For example, in bright environments such as daytime, camera 1042 automatically performs RGB photography to acquire a color image. On the other hand, in dark environments such as nighttime, each LED 1043 automatically lights up, and camera 1042 automatically performs IR photography to acquire a monochrome image. This ensures that the driver's head is reliably captured regardless of the ambient light.

[0480] For example, DMS30 performs lip-reading of the driver based on the image from camera 1042 and recognizes the content of the driver's speech. For example, DMS30 performs pupil detection, gaze detection, arousal determination, and recognition processing of the driver based on the image from camera 1042.

[0481] Next, with reference to Figure 66, a modified example of the installation position of camera 1042 will be described.

[0482] The left side of Figure 66 is a schematic diagram of the area around the steering wheel 1001 of vehicle 1C, viewed from the upper right. The right side of Figure 66 is a schematic diagram of the area in front of the driver's seat 101.

[0483] For example, the camera 1042 could be installed at position P121 on the dashboard 105 in front of the driver's seat 101. In this case, the driver's face can be photographed from the front. On the other hand, when the steering wheel 1001 rotates, the spokes of the steering wheel 1001 obstruct the view of the camera 1042.

[0484] For example, camera 1042 could be installed at position P122 near the left edge of the dashboard 105. In this case, symmetrically with camera module 1007, the driver's face would be photographed from the left front.

[0485] For example, it is conceivable to install the camera 1042 at position P123 on the steering column of the steering wheel 1001. In this case, the driver's face can be photographed at close range and from the front. On the other hand, similar to position P121, when the steering wheel 1001 rotates, the spokes of the steering wheel 1001 obstruct the view of the camera 1042. Also, the driver's view of the display unit 201CL of the center display 201 is obstructed.

[0486] For example, the camera 1042 could be built into a position P124 on the right side of the steering wheel 1001, near the upper edge of the central part 201C of the center display 201. In this case, the driver's face would be photographed from the front right. On the other hand, there are limitations on the storage space for the camera 1042, so it needs to be miniaturized. Also, the camera 1042 and the LED 1043 may be positioned far apart.

[0487] For example, it is conceivable to embed the camera 1042 at position P125 near the upper left corner of the left edge 201L of the center display 201. In this case, the driver's face would be photographed from the left front. On the other hand, due to space constraints for housing the camera 1042, miniaturization of the camera 1042 is necessary. Also, the camera 1042 and the LED 1043 may be positioned separately.

[0488] For example, one could place a smartphone in the center of the steering wheel 1001 and use the smartphone to photograph the driver's face. In this case, the driver's face can be photographed from the front. However, there is a risk that the smartphone may be damaged when the airbag deploys. Also, if the steering wheel 1001 rotates, the driver's face will be photographed at an angle.

[0489] For example, it is also conceivable to incorporate the camera 1042 inside the A pillar (not shown) on the driver's seat side of the vehicle 1C. In this case, the driver's face is photographed from the upper left diagonal direction. On the other hand, since there are restrictions on the storage space of the camera 1042, it is necessary to miniaturize the camera 1042. Also, the camera 1042 and the LED 1043 may be arranged separately.

[0490] For example, if the camera 1042 can be sufficiently wide-angled, the camera module 1008 may be arranged at the center in the left-right direction of the dashboard 105 so that the optical axis of the camera 1042 faces the rear of the vehicle 1C.

[0491] FIG. 67 shows an example of the imaging range of the camera 1042 in this case. In this example, the entire headrest of the driver's seat 101, the entire headrest of the passenger seat 102, a part of the headrest of the rear seat 103L, and a part of the headrest of the rear seat 103R are included in the imaging range. Therefore, with one camera 1042, it becomes possible to image the entire head of the driver, the entire head of the passenger in the passenger seat 102, a part of the head of the passenger in the rear seat 103L, and a part of the head of the passenger in the rear seat 103R.

[0492] <Operating example of CMS> Next, referring to FIGS. 68 to 85, an operating example of the CMS of the vehicle 1C will be described. Hereinafter, the display example of the display unit 201LL at the left end 201L of the center display 201 will be mainly described.

[0493] For example, the display control unit, which is a part of the function of the HMI 31, causes the display unit 201LL to display an image based on the image of the camera 51SL. More specifically, the display control unit sets a display range within the image of the camera 51SL and causes the display unit 201LL to display the image within the set display range.

[0494] Also, the recognition unit 73 performs object recognition based on the image of the camera 51SL or the like. The display control unit controls the display of the display unit 201LL based on the recognition result of the object and the state of the vehicle 1C or the like.

[0495] Figure 68 shows an example of the display on the display unit 201LL under normal conditions. In this example, a portion of the body of vehicle 1C, vehicle 1101, which is a following vehicle traveling in the lane to the left of vehicle 1C, and the road surface 1102 are displayed.

[0496] For example, when vehicle 1101 approaches vehicle 1C and the distance between vehicle 1101 and vehicle 1C falls within a predetermined range, a vertically elongated, band-shaped visual effect, bar 1103, is superimposed on the left edge of the display unit 201LL, as shown by the diagonal lines in Figure 69. Bar 1103 is displayed, for example, when vehicle 1101 is within the range visible to the display unit 201LL and the distance between it and vehicle 1C is within a predetermined range. The color of bar 1103 is a conspicuous color, such as yellow.

[0497] The driver will be alerted by the display of bar 1103, and will be able to reliably prevent accidents.

[0498] The display characteristics of bar 1103, such as its color, brightness, shape, flashing pattern, and movement, are not limited to this example and can be changed as desired. For example, the display characteristics of bar 1103 may change according to the level of danger.

[0499] For example, if the left turn signal of vehicle 1C is turned on, there is a possibility that vehicle 1C will turn left (change direction to the left) or change lanes to the left lane, increasing the risk of collision or contact with vehicle 1101. In response to this, for example, as shown in Figure 70, the width of bar 1103 increases and it starts flashing. This further attracts the driver's attention.

[0500] Furthermore, for example, if the left turn signal of vehicle 1C is turned on, the flashing speed of bar 1103 changes according to the degree of danger. For example, the faster the absolute speed of vehicle 1101 or its relative speed to vehicle 1C, the faster the flashing speed of bar 1103 becomes. For example, the closer vehicle 1101 gets, in other words, the shorter the distance between vehicle 1101 and vehicle 1C, the faster the flashing speed of bar 1103 becomes. In addition, for example, a warning sound may be sounded in synchronization with the flashing of bar 1103.

[0501] For example, if the absolute speed of vehicle 1101 or its relative speed to vehicle 1C is above a predetermined threshold, the timing at which bar 1103 is displayed will be earlier. That is, bar 1103 will be displayed from the point when vehicle 1101 is located further away from vehicle 1C.

[0502] Furthermore, as shown in Figure 71, for example, when vehicle 1101 is recognized from the left rear, a triangular visual effect, mark 1104, indicating the presence of vehicle 1101, may be superimposed and displayed near vehicle 1101. Mark 1104 moves in accordance with the movement of vehicle 1101.

[0503] This allows the driver to reliably recognize the position of the vehicle 1101, even when the visibility of the image displayed on the display unit 201LL is poor due to fog, haze, or other factors.

[0504] The location of vehicle 1101 may be indicated by a different display method than the mark 1104 in Figure 71. For example, a frame surrounding vehicle 1101 may be displayed.

[0505] Although a detailed explanation will be omitted, the display section 201RR at the right end 201R of the center display 201 also displays similar information based on the status of the vehicle traveling in the lane to the right of vehicle 1C and the operation of the right turn signal.

[0506] Furthermore, while the above explanation has shown an example of issuing a warning to other vehicles, the same process is applied when issuing warnings to moving objects other than vehicles (for example, motorcycles, etc.).

[0507] Furthermore, for example, the display control unit controls the range of the image displayed on the display unit 201LL in accordance with the operation of the turn signal.

[0508] Figure 72 shows an example of changing the display range of the image displayed on the display unit 201LL in accordance with the operation of the left turn signal.

[0509] Specifically, Figures 72A and 72B show examples of images captured by camera 51SL. The images show vehicles 1121-1 through 1121-4, etc., traveling in the lane to the left of vehicle 1C.

[0510] For example, if the left turn signal of vehicle 1C is turned off, the image in area A131a of Figure 72 is extracted, and the image in area A132a is displayed on the display unit 201LL. Area A132a includes vehicles 1121-2 through 1121-4, but does not include vehicle 1121-1.

[0511] On the other hand, when the left turn signal of vehicle 1C is turned on, the image in area A131b of Figure 72B is extracted, and the image in area A132b is displayed on the display unit 201LL. Areas A131b and A132b are wider than areas A131a and A132a. That is, the display range of the display unit 201LL is zoomed out (expanded) and widened. As a result, the rear end of vehicle 1121-1 is now included in area A132b, in addition to vehicles 1121-2 through 1121-4.

[0512] This allows the driver to see a wider area of ​​the lane to the left of vehicle 1C when the left turn signal is activated. As a result, the driver can change lanes or turn left more safely.

[0513] Furthermore, for example, if the display range of the display unit 201LL is zoomed out, an icon or frame indicating that it has been zoomed out may be displayed on the display unit 201LL.

[0514] Here, UN ECE-R46 (United Nations Economic Commission for Europe Regulation 46) defines the required field of view (hereinafter referred to as the mandatory field of view) that must be visible in a CMS. The mandatory field of view varies depending on the vehicle's passenger capacity, weight, etc., but Figure 73 shows an example.

[0515] In this example, the required field of view consists of regions A133L, A133R, and A133C.

[0516] Area A133L is the region enclosed by points P132L through P136L to the left of vehicle 1C. Specifically, reference point P131L is the leftmost point of vehicle 1C, corresponding to the position of the driver's eyes in the longitudinal direction of vehicle 1C. Point P132L is 4m behind reference point P131L. Point P133L is 60m behind reference point P131L. Point P134L is 1m to the left of point P132L. Point P135L is 20m behind and 4m to the left of reference point P131L. Point P136L is 60m behind and 4m to the left of reference point P131L.

[0517] Area A133R is the region enclosed by points P132R through P136R to the right of vehicle 1C. Specifically, reference point P131R is the rightmost point of vehicle 1C, corresponding to the position of the driver's eyes in the longitudinal direction of vehicle 1C. Point P132R is 4m behind reference point P131R. Point P133R is 60m behind reference point P131R. Point P134R is 1m to the right of point P132R. Point P135R is 20m behind reference point P131R and 4m to the right. Point P136R is 60m behind reference point P131R and 4m to the right.

[0518] Area A133C is located more than 60m behind the driver's eye position of vehicle 1C and is a 20m wide area centered on the left-right center of vehicle 1C.

[0519] Therefore, for example, as shown in Figure 72, when the display range of the display unit 201LL is changed in accordance with the operation of the left turn signal, the display range is changed so that area A133L is always included.

[0520] Although a detailed explanation will be omitted, the display range of the center display 201RR is controlled in a similar manner in accordance with the operation of the right-hand turn signal. In addition, the display range of the display unit 201RR is changed so that area A133R is always included.

[0521] Furthermore, for example, the display control unit controls the display range of the display unit 201LL based on the shift position of the vehicle 1C.

[0522] Figure 74 shows an example of changing the display range of the display unit 201LL based on the shift position of vehicle 1C.

[0523] Specifically, Figures 74A and 74B show examples of images captured by camera 51SL. The images show vehicles 1141-1 and 1141-2, etc., parked side by side to the left rear of vehicle 1C.

[0524] For example, before the shift position of vehicle 1C is set to reverse, an image is extracted from area A141a in Figure 74, and an image from area A142a is displayed on the display unit 201LL.

[0525] On the other hand, after the shift position of vehicle 1C is set to reverse, the image in area A141b of Figure 74B is extracted, and the image in area A142b is displayed on the display unit 201LL. Areas A141b and A142b are wider than areas A141a and A142a. That is, the display range of the display unit 201LL is zoomed out and expanded. At this time, for example, the display range of the display unit 201LL may be automatically moved downward so that a wider area of ​​the ground around vehicle 1C is displayed.

[0526] Although a detailed explanation will be omitted, the display range of the center display 201RR also changes based on the shift position of the vehicle 1C, similar to the display unit 201LL.

[0527] This allows the driver to see a wider area to the left rear of vehicle 1C. As a result, the driver can park vehicle 1C more safely.

[0528] Furthermore, for example, the driver can change the display range of display unit 201LL and display unit 201RR by operating the operation screen displayed on the console display 202 in Figure 75.

[0529] This operation screen displays buttons 1151, 1152, touchpad 1153, 1154, and 1155.

[0530] When button 1151 is pressed, the display range of the left display unit 201LL becomes the target of the setting. When button 1152 is pressed, the display range of the right display unit 201RR becomes the target of the setting.

[0531] When the touchpad 1153 is operated, the display ranges of the display unit 201LL and the display unit 201CC move. In other words, the display ranges of the display unit 201LL and the display unit 201CC move in the direction in which the touchpad 1153 is pressed.

[0532] When button 1154 is pressed, the currently set display range is memorized.

[0533] When button 1155 is pressed, the change in the display range is canceled, and the display returns to its previous state.

[0534] Furthermore, for example, the zoom ratio (Zoom Factor) of the display range of display units 201LL and 201RR can be set using other buttons on the console display 202 (not shown).

[0535] Here, the display range can be set within the configurable range A151 shown in Figure 76. In Figure 76, the parts corresponding to those in Figure 72 are denoted by the same reference numerals, and their explanations are omitted as appropriate.

[0536] For example, the size of the extraction range A152 extracted from the captured image for display is set based on the set zoom ratio. That is, the extraction range A152 is zoomed in (reduced) and zoomed out (expanded) based on the zoom ratio. Then, the image within a predetermined display range A153 within the extraction range A152 is displayed on the display unit 201LL.

[0537] In this case, the display range A153 can be moved within the configurable range A151. The configurable range A151 is set to the range in which area A133L is included within the display range A153.

[0538] In contrast, if, for example, area A133L extends beyond the display range A153, as shown in Figure 76B, the driver will be warned. For example, a warning message will be displayed on the display unit 201LL, and a warning sound will be emitted.

[0539] Alternatively, for example, the movement range of the display range A153 may be restricted so that the display range A153 does not go outside the settable range A151, regardless of the driver's operation.

[0540] Figure 77 shows an example of the display range of the display unit 201LL when the zoom ratio is changed from 0.8 to 1.3 in increments of 0.1.

[0541] The smaller the zoom ratio, the larger the display area, allowing for a wider area to be viewed. On the other hand, the size of objects such as following vehicles displayed on the 201LL display unit becomes smaller.

[0542] Conversely, as the zoom ratio increases, the display range becomes smaller, and the size of objects such as following vehicles displayed on the display unit 201LL increases. On the other hand, the number of pixels in the extracted image decreases, and the image clarity deteriorates.

[0543] For example, the adjustable zoom ratio range for the driver's side display unit 201LL is set to 1.0 to 1.2. The adjustable zoom ratio range for the passenger side display unit 201RR is set to 1.0 to 1.1.

[0544] Here, the adjustable zoom range for the driver's side display unit 201LL differs from that of the passenger side display unit 201RR. This is because the distance from the driver to the display unit 201LL is different from the distance to the display unit 201CC, and the direction of the driver's line of sight to the display unit 201LL is different from the direction of the driver's line of sight to the display unit 201CC. In other words, the way the driver sees the display unit 201LL is different from how the driver sees the display unit 201CC.

[0545] Furthermore, as mentioned above, the zoom ratio of display units 201LL and 201CC is reduced when the turn signal is activated or the shift position is set to reverse. With this in mind, the lower limit of the adjustable zoom ratio range is restricted to 1.0. For example, if the above-mentioned changes to the zoom ratio do not occur, the lower limit of the adjustable zoom ratio range can be lowered to 0.8 or 0.9.

[0546] In this way, the driver can move the display range A153 to their preferred position within the range that includes area A133L, or zoom in or out of the display range A153, by operating the console display 202. Furthermore, the driver can easily adjust the display range A153 without taking their eyes off the screen by, for example, moving their finger or clicking on the console display 202 while maintaining their driving posture.

[0547] Furthermore, in the case of physical side mirrors, for example, the driver can change the field of view through the side mirror by leaning their face closer to it and peering into it. For example, when parking, the driver may peer into the side mirror to check for obstacles on the ground around the vehicle.

[0548] Similarly, with the display unit 201LL, the driver can change the display range by bringing their face closer to the display unit 201LL and peering at it.

[0549] For example, the DMS30 detects the driver's head position and orientation by performing skeletal recognition of the driver based on the image from the ToF camera P101L. Based on the driver's head position and orientation, if the DMS30 recognizes that the driver is looking into the display unit 201LL, it detects the angle at which the driver is looking into the display unit 201LL.

[0550] In response to this, when vehicle 1C is stopped, that is, when vehicle 1C's speed is 0, the display control unit controls the display range of the display unit 201LL based on the angle at which the driver looks at the display unit 201LL.

[0551] For example, Figure 78 shows the display range of the display unit 201LL before the driver looks at it. In this example, a part of the vehicle 1C's body is displayed in the lower right corner of the display unit 201LL. Also, the upper part of object 1161 located to the left rear of vehicle 1C is displayed.

[0552] In this case, for example, if the driver looks at the display unit 201LL from an oblique upward direction, the display range of the display unit 201LL changes as shown in Figure 79. Specifically, the display range of the display unit 201LL moves downward. As a result, the lower part of the object 1161 is also displayed on the display unit 201LL, making it possible for the driver to see the entire object 1161.

[0553] Furthermore, as shown by the diagonal lines in the figure, a line 1162, which is a band-shaped visual effect indicating a predetermined area of ​​the ground around vehicle 1C, is superimposed and flashes. This allows the driver to quickly recognize objects that vehicle 1C may collide with or come into contact with, and to move vehicle 1C while avoiding the recognized objects.

[0554] Furthermore, for example, Figure 80 shows the display range of the display unit 201LL before the driver looks at it. In this example, a part of the body of vehicle 1C is displayed at the right edge of the display unit 201LL. A part of vehicle 1181 to the left rear is displayed in the upper left of the display unit 201LL. A parking line 1182 is displayed between vehicle 1C and vehicle 1181. The upper edge of object 1183 is displayed to the left of vehicle 1C and at the lower edge of the display unit 201LL.

[0555] For example, if the driver sets the shift position to reverse in order to park vehicle 1C, the display range of the display unit 201LL zooms out, as described above with reference to Figure 74. As a result, the display range of the display unit 201LL expands, as shown in Figure 81, and the area in which vehicle 1C, vehicle 1181, and object 1183 are displayed widens.

[0556] Furthermore, for example, if the driver looks at the display unit 201LL from an oblique upward direction, the display range of the display unit 201LL moves downward, as shown in Figure 82. As a result, the entire object 1183 is displayed on the display unit 201LL. Consequently, the driver can park the vehicle 1C while reliably avoiding the object 1183.

[0557] Although a detailed explanation will be omitted, if the driver looks at the display unit 201RR of the center display 201, the display range will move in the same way as the display unit 201LL.

[0558] Furthermore, for example, if the driver leans closer to the digital rearview mirror 204 and looks into it, the display range of the digital rearview mirror 204 will also move.

[0559] For example, based on the position and orientation of the driver's head, the DMS30 detects the angle at which the driver is looking into the digital rearview mirror 204 if it recognizes that the driver is looking into the digital rearview mirror 204.

[0560] In response to this, when the vehicle 1C is stopped, that is, when the vehicle 1C's speed is 0, the display control unit controls the display range of the digital rear mirror 204 based on the angle at which the driver looks into the digital rear mirror 204.

[0561] Furthermore, for example, as shown in Figure 83, if the driver DR1 is detected to be looking down at their feet, the light control unit 85 may turn on a footlight (not shown) to illuminate the area around the driver's feet. This allows the driver to easily find, for example, keys or other items that they have dropped at their feet.

[0562] Furthermore, for example, the DMS30 detects the driver's left hand movement by performing skeletal recognition of the driver based on the image from the ToF camera P101L. The recognition unit 73 detects objects around the door 611FL of the driver's seat 101 outside the vehicle 1C based on sensing data such as the image from the camera 51SL. The display unit 201LL then displays a warning when an object is detected within a predetermined range around the door 611FL of the driver's seat 101, and the driver attempts to open the door 611FL, for example, when the driver's left hand touches the door opener 661FL of the door 611FL.

[0563] For example, Figure 84 shows an example of the display on the display unit 201LL before the driver's left hand touches the door opener 661FL. A portion of the vehicle 1C's body is displayed at the right edge of the display unit 201LL. To the left of vehicle 1C, a pedestrian 1201 approaching vehicle 1C from behind is displayed.

[0564] In this case, for example, as shown in Figure 85, when it is detected that the driver DR1's left hand has touched the door opener 661FL, the display unit 201LL displays a warning, as shown by the diagonal lines in Figure 86. Specifically, a vertically elongated strip-shaped visual effect, a bar 1202, is superimposed and flashes at the left end of the display unit 201LL. The color of the bar 1202 is a conspicuous color, such as yellow.

[0565] In addition, a warning sound may be emitted at this time.

[0566] This will alert driver DR1 to pedestrian 1201 and prevent door 611FL from hitting pedestrian 1201 when it is opened.

[0567] In this case, for example, the body control unit 84 may increase the torque of the door 611FL to make it more difficult to open the door 611FL.

[0568] Furthermore, for example, if the passenger in the passenger seat 102 touches the door opener 661FR of the door 611FR, the display unit 201RR on the passenger seat 102 side may display a similar warning.

[0569] <Example of operation of display unit 201CL and illumination 1002> Next, with reference to Figures 87 to 95, examples of the operation of the display unit 201CL of the center display 201 and the illumination 1002 of the steering wheel 1001 will be described.

[0570] For example, the display control unit displays content on the display unit 201CL based on at least one of the following: the status of the vehicle 1C, the status of the area around the vehicle 1C, and the status of the passengers.

[0571] For example, the light control unit 85 lights up or flashes the illumination 1002 in a pattern appropriate to the situation, based on at least one of the following: the situation of the vehicle 1C, the situation around the vehicle 1C, and the situation of the passengers. The light emission pattern of the illumination 1002 is defined by at least one of the following: color, brightness, flashing pattern, light movement, and light emission area.

[0572] For example, when a driver gets into vehicle 1C and sits in the driver's seat 101, the DMS 30 performs driver recognition processing based on at least one of the images from the ToF camera P101L and the image from camera 1042 of camera module 1007. The display unit 201CL and illumination 1002 then notify that the driver has been recognized.

[0573] For example, Figure 87 shows the state of the display unit 201CL and illumination 1002 before the driver recognizes them.

[0574] The display unit 201CL shows the status of vehicle 1C. Specifically, it displays the shift position, charge level, speed, and remaining driving range.

[0575] Illumination 1002 is turned off.

[0576] Figure 88 shows the state of the display unit 201CL and illumination 1002 when the driver is recognized.

[0577] The display unit 201CL displays the image, name, and message of the recognized driver. After that, the display unit 201CL returns to the state shown in Figure 87.

[0578] Although detailed illustrations are omitted, the illumination 1002 lights up in a predetermined pattern. For example, multiple short strips of white light move across the illumination 1002 in a predetermined flowing pattern.

[0579] This ensures that the driver is notified that vehicle 1C has recognized them.

[0580] In this case, for example, the body control unit 84 may set the position and angle of the driver's seat 101 and the position and angle of the steering wheel 1001 based on the recognized driver's physique or the driver's preferences set in advance. This sets a driving position suitable for the recognized driver.

[0581] Furthermore, for example, the display control unit may adjust the display position of the display unit 201CL of the center display 201 based on the recognized driver's eye position, as well as the position and angle of the steering wheel 1001. This sets the display range of the display unit 201CL to a position that is easily visible without being obstructed by the steering wheel 1001, according to the recognized driver.

[0582] Furthermore, for example, when vehicle 1C is ready for autonomous driving, the display unit 201CL and illumination 1002 notify that preparation for autonomous driving is complete.

[0583] Figure 89 shows the state of the display unit 201CL and illumination 1002 when the vehicle is ready for autonomous driving.

[0584] In addition to the information shown in Figure 87, the display unit 201CL also displays icon 1311, indicating that preparation for autonomous driving is complete. To the left of icon 1311, the speed limit of the road on which vehicle 1 is traveling is displayed.

[0585] Although detailed illustrations are omitted, the illumination 1002 lights up in a predetermined pattern. For example, multiple short strips of white light move across the illumination 1002 in a predetermined flowing pattern.

[0586] This allows the driver to be sure that the vehicle is ready for autonomous driving.

[0587] Furthermore, for example, when the automatic driving of vehicle 1C is started, the display unit 201CL and illumination 1002 notify the start of automatic driving.

[0588] Figure 90 shows the state of the display unit 201CL and illumination 1002 when autonomous driving starts.

[0589] The display unit 201CL displays, for example, an image of the area in front of vehicle 1C, captured by camera 51FC. The display unit 201CL also displays, for example, the shift position, charge level, speed, remaining range, speed limit of the road being driven on, fuel efficiency, etc. To the right of the aforementioned icon 1311, the display unit 201CL displays icon 1312, which indicates that autonomous driving is in progress. In addition, for example, the color of icon 1311 changes when autonomous driving begins.

[0590] Although detailed illustrations are omitted, the illumination 1002 lights up in a predetermined pattern. For example, multiple short white bands of light move across the illumination 1002 in a predetermined pattern, gradually changing to blue. Finally, the entire illumination 1002 lights up blue, and remains blue during automatic operation.

[0591] This allows the driver to be sure that autonomous driving has started. In addition, when illumination 1002 lights up, passengers other than the driver can also be sure that autonomous driving has started.

[0592] Figure 91 shows the state of the display unit 201CL and illumination 1002 when vehicle 1C changes lanes to the right lane during autonomous driving.

[0593] When vehicle 1C changes lanes, the display unit 201CL displays guidance information 1313, including an animation to notify the vehicle of the lane change and its direction, superimposed on the image in front of vehicle 1C.

[0594] Furthermore, the blue light of the entire illumination 1002 turns off, and the predetermined flashing area A201 on the right side of illumination 1002 flashes blue. Also, when the steering wheel 1001 rotates clockwise to change lanes, the flashing area A201 rotates in the opposite direction (counterclockwise) to match the rotation angle of the steering wheel 1001. After changing lanes, when the steering wheel 1001 rotates counterclockwise to return to its original position, the flashing area A201 rotates in the opposite direction (clockwise) to match the rotation angle of the steering wheel 1001. As a result, the flashing area A201 does not rotate with the steering wheel 1001 and remains in a fixed position when viewed from the outside. After the rotation angle of the steering wheel 1001 returns to 0 degrees, the entire illumination 1002 lights up blue, just as before the lane change.

[0595] This allows the driver to recognize in advance that vehicle 1C will change lanes.

[0596] Similarly, when vehicle 1C changes lanes to the left lane, guidance information 1313 is superimposed on the display unit 201CL, and a predetermined flashing range on the left side of the illumination 1002 flashes in blue.

[0597] Similarly, the same process is performed when, for example, vehicle 1C changes direction (e.g., turns left or right). For example, guidance information, including an animation to notify the vehicle of the change of direction, is superimposed on the display unit 201CL, and a predetermined flashing range of the illumination 1002 corresponding to the direction of the change of direction flashes in blue.

[0598] Figure 92 shows the state of the display unit 201CL and illumination 1002 when a dangerous condition is detected around vehicle 1C.

[0599] The display unit 201CL displays, for example, a message 1321 and an icon 1322 indicating the detected status and how to deal with it.

[0600] Furthermore, the blue light of the entire illumination 1002 turns off, and a predetermined flashing area A211 on the upper side of the illumination 1002 begins to flash. For example, the color and flashing speed of the light in flashing area A211 change based on the detected level of danger. For example, if the danger level is high, flashing area A211 flashes red, and if the danger level is low, flashing area A211 flashes yellow. Also, the flashing speed increases as the danger level increases, and decreases as the danger level decreases.

[0601] Additionally, for example, a warning sound is emitted in conjunction with the flashing of the flashing range A211.

[0602] Furthermore, for example, the DMS30 detects the positions of the driver's left and right hands by performing skeletal recognition of the driver based on images captured by the ToF camera P101L. The DMS30 then detects the position where the driver's left hand fingers touch the control section 1003 of the steering wheel 1001, and the position where the driver's right hand fingers touch the control section 1004 of the steering wheel 1001.

[0603] For example, if the driver's left hand fingers touch the operation unit 1003, or if the driver's right hand fingers touch the operation unit 1004, the display control unit displays guidance on the display unit 201CL indicating the function and operation method of the button that was touched.

[0604] For example, as shown in Figure 93, when the driver DR1's left thumb touches the leftmost button on the control unit 1003, guidance information 1341 is displayed near the center and bottom edge of the display unit 201CL in the left-right direction, as shown in Figure 94. The guidance information 1341 indicates the function and operation method of the button that the driver DR1's finger touched.

[0605] Note that in Figure 94, the hands of driver DR1 have been omitted from the illustration for clarity.

[0606] Alternatively, similar guidance information may be displayed on the head-up display's display 203.

[0607] This allows driver DR1 to understand the function and operation method of the button they are trying to operate without having to move their gaze significantly towards the control unit 1003 or 1004. For example, driver DR1 can understand the function and operation method of the button they are trying to operate even in dark conditions such as at night. For example, even if driver DR1 is not familiar with driving vehicle 1C, they can easily understand the function and operation method of the button they are trying to operate.

[0608] Figure 95 shows the state of the display unit 201CL and illumination 1002 when the DMS30 is performing at least one of speech recognition and lip reading.

[0609] In this case, the display unit 201CL displays an animation 1361 indicating that at least one of speech recognition and lip reading is being performed.

[0610] Although detailed illustrations are omitted here, illumination 1002 represents the movement of light in conjunction with the movement of animation 1361.

[0611] For example, the same animation as animation 1361 may also be displayed on the display unit 201CC of the center display 201.

[0612] This ensures that the driver or other passengers are reliably notified that at least one of voice recognition and / or lip-reading is being performed. As a result, it is possible to prevent, for example, the passengers from suspecting that data such as conversations are being collected without their consent.

[0613] <Examples of voice control> Next, an example of voice control for vehicle 1C will be described with reference to Figures 96 to 105.

[0614] As described above, vehicle 1C is capable of real-time 360-degree audio. Furthermore, the driver's seat 101, passenger seat 102, rear seat 103L, and rear seat 103L are each equipped with seat speakers 1021, 1022, 1023L, and 1023R, respectively. Therefore, the audio control unit, which is part of the HMI 31's functions, can individually and freely control the sound image, sound field, volume, etc., for each occupant in each seat.

[0615] The DMS30 can recognize the shape and position of the driver's ears, for example, based on images from the ToF camera P101L. Therefore, the audio control unit can appropriately set the sound image, sound field, volume, etc., for the driver by controlling the seat speaker 1021, etc., in the driver's seat 101 based on the position and shape of the driver's ears.

[0616] Similarly, the DMS30 can recognize the shape and position of both ears of the passenger in the passenger seat 102 based on the image from the ToF camera P101R, for example. Therefore, the sound control unit can appropriately set the sound image, sound field, volume, etc. for the passenger by controlling the seat speaker 1022, etc., in the passenger seat 102 based on the position and shape of the passenger's ears.

[0617] Similarly, the DMS30 can recognize the shape and position of both ears of the passenger in the rear seat 103L based on images from, for example, the ToF camera P111L or the ToF camera P112L. Therefore, the sound control unit can appropriately set the sound image, sound field, volume, etc. for the passenger by controlling the seat speaker 1023L, etc., in the rear seat 103L based on the position and shape of the passenger's ears.

[0618] Similarly, the DMS30 can recognize the shape and position of both ears of the passenger in the rear seat 103R based on images from, for example, the ToF camera P111R or the ToF camera P112R. Therefore, the sound control unit can appropriately set the sound image, sound field, volume, etc. for the passenger by controlling the seat speaker 1023R, etc., in the rear seat 103R based on the position and shape of the passenger's ears.

[0619] Thus, vehicle 1C can output sound individually to each passenger in each seat, and can also individually control the sound image, sound field, volume, etc. Using this characteristic, the following functions can be realized, for example.

[0620] Figure 96 shows an example of a method for controlling the warning sound when there is a hazard such as an obstacle in front of vehicle 1.

[0621] Sound fields A301L and A301R schematically represent the sound fields of the warning sound directed to the driver in the driver's seat 101. Specifically, the triangles representing sound fields A301L and A301R indicate the position of the virtual sound source and the extent of the sound field. Specifically, the vertices of the triangle indicated by black circles indicate the position of the virtual sound source. The two sides extending from the vertex indicating the virtual sound source indicate the direction in which the sound field expands.

[0622] Furthermore, sound fields A301L and A301R are represented by cones in three-dimensional space. Therefore, sound fields A301L and A301R spread out conically in the direction of the triangle's edges from the virtual sound source.

[0623] For example, first, sound field A301L is set to point to the right near the driver's left ear. Sound field A301R is set to point to the left near the driver's right ear. As a result, the driver will hear warning sounds in both their left and right ears.

[0624] Then, as indicated by the arrow, sound field A301L rotates clockwise around the driver's face until it is directly in front of the driver. Similarly, sound field A301R rotates counterclockwise around the driver's face until it is directly in front of the driver, as indicated by the arrow.

[0625] Therefore, the driver perceives the warning sounds as rotating around their face, eventually merging into one and localizing directly in front of them. Subsequently, a warning message such as "Look straight ahead" is output from directly in front of the driver.

[0626] Furthermore, if the direction in which the driver is being alerted (for example, the direction in which an obstacle exists) is not directly in front, the warning sound will be localized in the direction in which the driver is being alerted. Therefore, the driver will perceive the warning sound as moving and localizing in the direction in which the driver is being alerted.

[0627] This allows the driver to be alerted to potential hazards such as obstacles, and to reliably recognize the direction in which the hazard exists.

[0628] Figure 97 shows an example of how to control the sound effects output when autonomous driving starts.

[0629] First, the sound field A302 for sound effects is set to be at the driver's ear level and facing the driver from directly in front of them. Then, the sound field A302 splits into left and right, with one rotating counterclockwise around the driver's face to directly behind the driver, and the other rotating clockwise around the driver's face to directly behind the driver.

[0630] Therefore, the driver perceives the sound effects as being emitted from the front, then splitting to the left and right, rotating around their face, and finally merging into one, localized behind them. This effectively simulates the start of autonomous driving.

[0631] Figure 98 shows an example of a method for controlling the sound effect output when vehicle 1C accelerates during autonomous driving.

[0632] First, the sound field A303 is set to face to the right, near the driver's left ear. Then, the sound field A303 rotates clockwise around the driver's face until it faces to the left, near the driver's right ear.

[0633] Therefore, the driver perceives the sound effect as rotating clockwise in front of their face, from the left ear to the right ear.

[0634] Figure 99 shows an example of a method for controlling the sound effect output when vehicle 1C decelerates during autonomous driving.

[0635] First, the sound field A304 for the sound effect is set to face left, near the driver's right ear. Then, the sound field A304 rotates counterclockwise around the driver's face until it faces right, near the driver's left ear.

[0636] Therefore, the driver perceives the sound effect as rotating counterclockwise in front of their face, from the right ear to the left ear.

[0637] As a result, the driver will be able to intuitively recognize the acceleration and deceleration of vehicle 1C during autonomous driving by the direction in which the sound effect moves.

[0638] Furthermore, sound effects for settings other than the speed of vehicle 1C may also be controlled as shown in Figures 98 and 99. For example, when the setting value increases, a sound effect may be output as shown in Figure 98, and when the setting value decreases, a sound effect may be output as shown in Figure 99.

[0639] Figure 100 shows an example of a method for controlling sound effects that are output before voice messages such as navigation messages are output.

[0640] First, the sound field A305 for the sound effect is set to face the driver from the left rear at the height of the driver's ears. Then, the sound field A305 rotates clockwise around the driver's head until it faces the driver from the right front.

[0641] Therefore, the driver perceives the sound effect as rotating counterclockwise around the back of their head, from the left rear to the right front. This rotation of the sound effect behind the driver's head makes it easier for the driver to notice than if it rotated in front of their head. Consequently, the number of times the driver misses the audio message that follows the sound effect is reduced.

[0642] Figures 101 to 104 show examples of control methods for a warning sound that alerts the driver to a vehicle 1401 approaching from the left rear.

[0643] For example, when the left turn signal of vehicle 1401 is turned on, a warning sound is emitted when the distance between vehicle 1401 and vehicle 1C falls within a predetermined threshold. Figure 101 shows the sound field A306 at the start of the warning sound emission. The orientation of the sound field A306 is set to be approximately equal to the direction in which the driver's head in vehicle 1C is viewed from vehicle 1401. The virtual sound source for the warning sound is set at a predetermined distance away from the driver's head in the direction of vehicle 1401.

[0644] Subsequently, as shown in Figures 102 to 104, the sound field A306 moves as vehicle 1401 approaches vehicle 1C. Specifically, the sound field A306 moves so that it always faces approximately the same direction as the direction in which vehicle 1401 would view the driver's head in vehicle 1C. In addition, the sound field A306 moves so that the distance between the virtual sound source of the warning sound and the driver's head is approximately proportional to the distance between vehicle 1401 and vehicle 1C.

[0645] Therefore, the driver can recognize the approach of vehicle 1401 by the warning sound. In addition, the driver can intuitively perceive the speed, distance, and direction of vehicle 1401 by the movement of the warning sound.

[0646] Furthermore, the seat speakers at each seat can be used to individually output audio so that it can only be heard by the passenger in that seat. For example, as shown in Figure 105, the seat speaker 1023L can be used to output audio so that it can only be heard by the passenger sitting in the rear seat 103L.

[0647] This feature allows, for example, passengers in each seat to make voice or video calls to the outside world without other passengers hearing. It also allows, for example, passengers within vehicle 1C to engage in voice chat with each other.

[0648] <Examples of lip reading> As described above, DMS30 performs lip-reading of the driver based on the image from camera 1042 of camera module 1007. As described above, DMS30 performs lip-reading of the passenger in the rear seat 103L based on the image from ToF camera P112L. DMS30 performs lip-reading of the passenger in the rear seat 103R based on the image from ToF camera P112R. For example, DMS30 can also perform lip-reading of the passenger in the front passenger seat 102 using the image from ToF camera P101R.

[0649] For example, DMS30 can improve the accuracy of speech recognition by using lip-reading as an aid to speech recognition of passengers at each seat. Alternatively, for example, DMS30 can recognize the content of speech uttered by passengers at each seat using only lip-reading.

[0650] This allows the DMS30 to accurately recognize the speech of passengers in each seat, for example, when it is noisy inside vehicle 1C and it is difficult to collect sound with the microphone, or when passengers cannot speak loudly because a child is sleeping.

[0651] Furthermore, for example, HMI31 can convert the recognized speech content of a passenger in one seat into audio and output it through the seat speakers in other seats, thereby conveying the content of the speech to passengers in other seats. For example, HMI31 can convert the recognized speech content of a passenger in one seat into text and display it on the display in front of other seats, thereby conveying the content of the speech to passengers in other seats. This makes it possible for passengers in each seat to communicate even when it is noisy inside the vehicle 1C and voices are difficult to hear or when it is not possible to speak loudly.

[0652] Furthermore, for example, the driver can operate vehicle 1C without speaking by using lip-reading. Specifically, for example, the driver can operate vehicle 1C by pressing a predetermined button on the control section 1003 or control section 1004 of the steering wheel 1001 and making the same mouth movements as when inputting a predetermined voice command. In this case, the driver may or may not speak.

[0653] More specifically, for example, the driver can adjust the air conditioning settings in vehicle 1C by mouthing voice commands such as "cool my hands," "cool my feet," or "ventilate the room." For example, the driver can control music playback by mouthing voice commands such as "play some wake-up music," "play my usual music," or "play a song we can all sing along to." For example, the driver can give instructions to a navigation application running in vehicle 1C, such as searching for a destination, by mouthing voice commands such as "tell me about restaurants around here," "find a cafe around here," or "tell me about nearby charging stations."

[0654] This allows the driver to operate vehicle 1C using voice commands, even when it is noisy inside the vehicle 1C or when they cannot speak loudly.

[0655] <Examples of gesture input> For example, the DMS30 can recognize the gestures of each passenger in each seat of vehicle 1C based on images from ToF cameras installed for each seat. This allows each passenger to operate vehicle 1C using gestures.

[0656] For example, a passenger can increase the volume of the audio of the content being played in vehicle 1C by pointing into the air and rotating their index finger clockwise. Conversely, a passenger can decrease the volume of the audio of the content being played in vehicle 1C by pointing into the air and rotating their index finger counterclockwise. For example, a passenger can mute the volume of the audio of the content being played in vehicle 1C by holding up their index finger and bringing it close to their lips.

[0657] For example, a passenger can operate the touch panel on the display in front of their seat by pointing their index finger at the display and moving their finger, without actually touching the display. For example, a driver can launch a navigation application by pointing to the display unit 201CC of the center display 201. Also, for example, a driver can display information about the pointed location or set the pointed location as a destination by pointing to the map within the navigation application.

[0658] <<7. Modified Examples of the Third Embodiment>> The following describes a modified example of the third embodiment of the present technology described above.

[0659] The above explanation uses the example of a left-hand drive vehicle (Vehicle 1C), but this technology can, of course, also be applied to right-hand drive vehicles. When applying this technology to a right-hand drive vehicle, the interior layout described above will be appropriately modified to suit a right-hand drive vehicle.

[0660] Furthermore, this technology can also be applied to a mobility device that operates autonomously without the driver's input. In this case, the distinction between the driver's seat 101 and the passenger seat 102 mentioned above is eliminated, and the various displays described above are placed in front of the seats within the mobility device. In addition, the display range of the images shown on the displays is changed based on, for example, the gaze direction and posture of the person sitting in a designated seat.

[0661] Furthermore, this technology is not limited to any particular type of vehicle to which it can be applied. In addition, this technology can be applied to other mobile devices besides vehicles, such as personal mobility devices, airplanes, ships, construction machinery, and agricultural machinery.

[0662] <<8. Others>> <Example of computer configuration> The series of processes described above can be executed by hardware or by software. When the series of processes are executed by software, the programs that make up that software are installed on the computer (for example, processor 21, etc.).

[0663] The programs executed by the computer may be programs that are processed chronologically in the order described herein, or they may be programs that are processed in parallel or at necessary times, such as when a call is made.

[0664] Furthermore, in this specification, a system means a collection of multiple components (devices, modules (parts), etc.), regardless of whether all components are located in the same enclosure or not. Therefore, multiple devices housed in separate enclosures and connected via a network, and a single device in which multiple modules are housed in one enclosure, are both considered systems.

[0665] Furthermore, the embodiments of this technology are not limited to those described above, and various modifications are possible without departing from the spirit of this technology.

[0666] For example, this technology can be configured as cloud computing, where a single function is shared and processed collaboratively by multiple devices via a network.

[0667] Furthermore, each step described in the flowchart above can be performed by a single device, or it can be divided and performed by multiple devices.

[0668] Furthermore, if a single step includes multiple processes, those processes can be executed by a single device or shared among multiple devices.

[0669] <Examples of configuration combinations> This technology can also be configured as follows:

[0670] (1) A first display extending horizontally in front of the first seat of the mobile device Prepare, The left edge of the first display is facing diagonally to the right and rear as viewed from the first seat. The central part of the first display faces backward when viewed from the first seat. The right edge of the first display is facing diagonally to the left and rear as viewed from the first seat. Display system. (2) The leftmost part of the first display shows an image of the left rear of the mobile device. The rightmost part of the first display shows an image of the right rear of the mobile device. The display system described in (1) above. (3) The left and right edges of the first display are used as digital outer mirrors. The display system described in (2) above. (4) A display control unit that controls the display of images at the left and right edges of the first display. The display system further comprising the features described in (2) or (3) above. (5) The display control unit controls the display range of the left and right edges of the first display based on at least one of the following: the gaze direction of the person sitting in the first seat, the person's posture, the operation of the turn signals, and the shift position. The display system described in (4) above. (6) When the moving device is stopped, the display control unit moves the display area of ​​the first display, whichever of the left and right edges the person is looking at, based on the angle at which the person is looking. The display system described in (5) above. (7) The display control unit superimposes a visual effect indicating a predetermined area around the mobile device onto the image on the left or right edge of the first display that the person is looking at. The display system described in (6) above. (8) The display control unit zooms out the display range of the left and right ends of the first display in the direction in which the turn signal is turned on. The display system described in any of (5) to (7) above. (9) The display control unit zooms out the display range when the shift position is set to reverse. The display system described in any of (5) to (8) above. (10) The display control unit overlays a visual effect based on the recognition result of an object located diagonally to the left rear of the mobile device onto the image at the left end of the first display, and overlays a visual effect based on the recognition result of an object located diagonally to the right rear of the mobile device onto the image at the right end of the first display. The display system described in any of (5) to (9) above. (11) The display control unit changes the display mode of the visual effect based on the degree of danger to the object. The display system described in (10) above. (12) The display control unit changes the display mode of the visual effect at the left end of the first display when the left turn signal is turned on, and changes the display mode of the visual effect at the right end of the first display when the right turn signal is turned on. The display system described in (11) above. (13) A recognition unit that recognizes an object to the left rear of the moving device based on an image of the left rear of the moving device, and recognizes an object to the right rear of the moving device based on an image of the right rear of the moving device. The display system further comprises any of the above (10) to (12). (14) The central part of the first display shows an image of the surroundings of the mobile device. The display system described in any of (1) to (13) above. (15) A display control unit controls the display range of an image displayed in the center of the first display based on at least one of the gaze direction and posture of a person sitting in the first seat. The display system described in (14) further comprises the above. (16) The central part of the first display shows information to assist in the operation of the mobile device. The display system described in any of (1) to (15) above. (17) The central portion of the first display extends horizontally in front of the first seat and the second seat located next to the first seat, and is divisible into a first display area in front of the first seat, a second display area in front of the second seat, and a third display area between the first and second display areas. The display system described in any of (1) to (16) above. (18) The first display area displays information to assist in the operation of the mobile device. The second and third display areas display infotainment-related information. The display system described in (17) above. (18A) The central part of the first display is facing diagonally upwards. The display system described in any of (1) to (17) above. (19) The left and right edges of the first display are bent inward from the center of the first display towards the moving device. The display system described in any of (1) to (18A) above. (20) A second display is provided below the first display and facing backward from the perspective of the first seat. The display system further comprises any of the above (1) to (19). (twenty one) The second display consists of a two-dimensional or three-dimensional touch panel and displays an operation screen for the information displayed on the first display. The display system described in (20) above. (21A) The second display shows the operation screen for the air conditioning of the mobile device. The display system described in (20) or (21) above. (21B) The second display is oriented diagonally upwards. The display system described in any of (20) to (21A) above. (21C) The central part of the first display is angled upward at approximately the same angle as the second display. The display system described in (21B) above. (twenty two) The second display is located on the console between the first seat and the second seat, which is positioned next to the first seat. The display system described in any of (20) to (21C) above. (twenty three) A display control unit that controls the display of images at the left and right edges of the first display. Furthermore, The leftmost part of the first display shows an image of the left rear of the mobile device. The rightmost part of the first display shows an image of the right rear of the moving device. The second display is comprised of a two-dimensional or three-dimensional touch panel and displays the operation screen for setting the display range of the left and right edges of the first display. The display control unit changes the display range based on the operation on the operation screen. The display system described in any of (20) to (22) above. (twenty four) The display control unit changes the display range of the left edge of the first display within a range that includes a predetermined first area diagonally to the left rear of the mobile device, and changes the display range of the right edge of the first display within a range that includes a predetermined second area diagonally to the right rear of the mobile device, based on the operation on the operation screen. The display system described in (23) above. (twenty five) The display control unit causes the range in which the display range can be zoomed out and zoomed in to differ between the left and right ends of the first display. The display system described in (23) or (24) above. (26) A third display is provided in front of the first seat and displays visual information superimposed within the field of view of the person sitting in the first seat. The display system further comprises any of the above (1) to (25). (27) The third display shows information to assist in the operation of the mobile device. The display system described in (26) above. (28) A fourth display is provided above the first display and facing backward from the perspective of the first seat. The display system further comprises any of the above (1) to (27). (29) The fourth display is used as a digital rearview mirror. The display system described in (28) above. (30) A display control unit controls the display range of an image displayed on the fourth display based on at least one of the gaze direction and posture of a person sitting in the first seat. The display system described in (29) further comprises the above. (31) The fifth display located in the center of the steering wheel of the aforementioned mobile device The display system further comprises any of the above (1) to (30). (31A) The fifth display shows a message to the person sitting in the first seat. The display system described in (31) above. (32) The sixth display is provided on the back of the first seat and at least one of the second seats arranged next to the first seat. The display system further comprises any of the above (1) to (31A). (33) The left edge of the first display faces diagonally rear to the right of the moving device. The central part of the first display faces the rear of the mobile device, The rightmost edge of the first display is facing diagonally to the left rear of the mobile device. The display system described in any of (1) to (32) above. (34) The aforementioned first seat is the driver's seat. The display system described in any of (1) to (33) above. (35) A display extending horizontally in front of the seat of the mobility device Prepare, The left edge of the aforementioned display faces diagonally to the right and rear as viewed from the seat. The central part of the aforementioned display faces backward when viewed from the seat. The right edge of the aforementioned display is facing diagonally to the left and rear as viewed from the seat. Display device. (36) A display extending horizontally in front of the seat of the mobility device has its left end pointing diagonally to the right rear as viewed from the seat, its central part pointing to the rear as viewed from the seat, and its right end pointing diagonally to the left rear as viewed from the seat. The left end of the display shows an image of the diagonally left rear of the mobility device, and the right end of the display shows an image of the diagonally right rear of the mobility device. Display method. (37) A display extending horizontally in front of the seat, with its left end facing diagonally rear to the right as viewed from the seat, its central part facing rear as viewed from the seat, and its right end facing diagonally rear to the left as viewed from the seat. A mobile device. (38) A metadata addition unit that adds metadata to video data shot while moving, An image processing unit that edits the captured video data based on the metadata and generates edited video data. A mobile device equipped with the following features. (39) The image processing unit generates the edited video data by extracting and combining frames from multiple video data captured by multiple cameras based on the metadata. The moving device described in (38) above. (40) The image processing unit extracts frames from a plurality of captured video data based on the path of the moving device and the metadata. The mobile device described in (39) above. (41) The aforementioned video data is video data of the surroundings of the mobile device or the interior of the room. The moving device described in (39) or (40) above. (42) The image processing unit extracts frames from the video data of the room based on the activity level of the room. The mobile device described in (41) above. (43) The image processing unit superimposes visual information onto the frames of the captured video data based on the metadata. The moving device according to any one of (38) to (42) above. (44) The image processing unit edits the captured video data based on user instructions. The moving device described in any of (38) to (43) above. (45) The metadata assignment unit assigns metadata to the recorded video data relating to one or more of the following: the shooting location, the shooting date and time, the status of the mobile device, the status of the room inside the mobile device, and the status of the surroundings of the mobile device. The moving device according to any one of (38) to (44) above. (46) The metadata assignment unit assigns the metadata entered by the user to the recorded video data. The moving device described in any of (38) to (45) above. (47) A rearview mirror or a camera installed around the rearview mirror to photograph the direction of the driver's seat. A mobile device. (48) The aforementioned rearview mirror is installed near the ceiling diagonally in front of the driver's seat. The moving device described in (47) above. (49) The camera is installed on the ceiling around the location where the rearview mirror is mounted. The moving device described in (48) above. (50) The aforementioned camera is installed closer to the driver's seat than the rearview mirror. The moving device described in (48) or (49) above. (51) Based on the image captured by the aforementioned camera, a first recognition unit performs at least one of the following: driver gaze recognition, posture recognition, motion recognition, and personal recognition. The mobile device further comprises any of the above (47) to (50). (52) Based on at least one of the driver's gaze, posture, and actions recognized by the first recognition unit, a display control unit controls the display of the digital outer mirror of the mobile device. The mobile device further comprising the mobile device described in (51) above. (53) A second recognition unit recognizes objects around the driver's seat door based on sensing data from the area around the driver's seat door on the outside of the mobile device. Furthermore, When the first recognition unit recognizes the driver's action of opening the door, and the second recognition unit recognizes an object around the door, the display control unit superimposes a visual effect based on the object recognition result onto the image on the digital outer mirror on the driver's side. The mobile device described in (52) above. (54) When the first recognition unit detects that the driver is looking into the digital outer mirror, the display control unit moves the display range of the digital outer mirror based on the angle at which the driver is looking. The moving device described in (52) or (53) above. (55) When the first recognition unit recognizes that the driver has touched the control body of the mobile device, the display control unit causes information indicating guidance regarding the control body to be displayed on the display in front of the driver's seat. The moving device according to any one of (52) to (54) above. (56) The aforementioned camera is a ToF camera. The mobile device according to any one of (47) to (55) above. (57) A rod-shaped operating body used to change the shift position, the operating body having an indicator on its side that changes color depending on the set shift position. A mobile device. (58) The order in which the shift position can be set by the aforementioned operating device is reverse, neutral, drive, and then automatic driving. The mobile device described in (57) above. (59) The aforementioned operating body consists of a lever that switches the shift position by being moved in the vertical direction. The moving device described in (58) above. (60) The operating body includes a dial that switches the shift position by rotating it around an axis. The moving device described in (58) above. (61) The tip of the operating body is provided with a button that can be pressed in the axial direction of the operating body and is used to set the shift position to parking. The mobile device described in any of (58) to (60) above. (62) The indicator is provided circumferentially on the side of the operating body. The mobile device described in any of (57) to (61) above. (63) The operating body is provided so as to extend laterally from the steering column to the moving device, The indicator is positioned at least one of the following locations: one visible from the driver's seat through the gaps in the spokes of the steering wheel, and another outside the outer circumference of the steering wheel as viewed from the driver's seat. The mobile device according to any one of (57) to (62) above. (64) A steering wheel having a ring-shaped illumination in the center, in which at least one of the following is variable: color, brightness, and light-emitting area; A light control unit that controls the illumination and A mobile device equipped with the following features. (65) The light control unit turns on or flashes the illumination based on at least one of the following: the status of the mobile device, the status of the surroundings of the mobile device, and the status of the passenger. The moving device described in (64) above. (66) The light control unit turns on or flashes the illumination in a pattern according to the situation. The mobile device described in (65) above. (67) The pattern is defined by at least one of the following: color, brightness, flashing pattern, movement of light, and the light-emitting region. The mobile device described in (66) above. (68) The light control unit illuminates or flashes the portion of the illumination corresponding to the direction of the lane change or direction change when the mobile device changes lanes or direction. The moving device described in (66) or (67) above. (69) The light control unit rotates the part that turns on or flashes the illumination in the opposite direction to the steering wheel, in accordance with the rotation angle of the steering wheel. The moving device described in (68) above. (70) The light control unit turns on or flashes the illumination in a predetermined pattern when at least one of voice recognition and lip reading is being performed. The mobile device according to any one of the above (66) to (69). (71) The aforementioned central section houses the airbag, The illumination is positioned to avoid the area in the center that splits when the airbag deploys. The mobile device described in any of (64) to (70) above. (72) Multiple user interface devices are arranged along a loop line that encircles the interior of the room almost horizontally. Mobile device. (73) The device for the user interface includes at least one of an output device and an operating device. The mobile device described in (72) above. (74) The output device includes at least one of a display extending horizontally in front of the moving device and illumination provided around the center of the steering wheel. The mobile device described in (73) above. (75) The output device includes a speaker provided in the door of the mobile device. The mobile device according to any one of (73) to (74) above. (76) The operating device includes at least one of the following: a door opener for the door of the mobile device, a device for adjusting the airflow direction of the air conditioner, and a device for adjusting the headlights. The mobile device according to any one of (73) to (75) above. (77) Loop lights are provided, approximately parallel to the aforementioned loop line, to surround at least a portion of the perimeter of the interior of the room. The mobile device further comprises any of the above (72) to (76). (78) A light control unit that performs coordinated control of the external lights of the mobile device and the loop lights. The mobile device further comprising the mobile device described in (77) above. (79) An air outlet for the air conditioner is provided along the aforementioned loop line. The mobile device according to any one of (72) to (78) above. (80) Multiple seat speakers, which are speakers individually installed at each seat, An output control unit that individually controls the sound output from each of the aforementioned seat speakers. A mobile device equipped with the following features. (81) Each of the aforementioned seat speakers is positioned below the headrest of each of the aforementioned seats. The moving device described in (80) above. (82) Multiple line speakers are speakers arranged along a loop line that encloses the room almost horizontally. Furthermore, The output control unit further controls the sound output from the line speaker. The moving device described in (80) or (81) above. (83) The output control unit controls each line speaker to output sound directed towards the entire cabin, and each seat speaker to output sound directed towards each passenger sitting in each seat. The moving device described in (82) above. (84) The output control unit moves the sound field of the warning sound to the driver in a direction that attracts the driver's attention. The moving device described in (80) to (83) above. (85) The output control unit rotates the sound field of a sound effect indicating an increase in a predetermined set value of the moving device in a first direction around the head of the person sitting in the seat, and rotates the sound field of a sound effect indicating a decrease in the set value in a second direction opposite to the first direction around the person's head. The moving device according to any one of (80) to (84) above. (86) The output control unit rotates the sound field of the sound effect output before the voice message so that it is behind the head of the person sitting in the seat. The moving device according to any one of (80) to (85) above. (87) The output control unit moves the sound field of the warning sound for an object behind the moving device in accordance with the movement of the object. The moving device according to any one of (80) to (86) above. (88) A camera module mounted on the dashboard captures a diagonal front view of the driver's seat. A mobile device. (89) The camera module switches between color and infrared photography. The moving device described in (88) above. (90) The camera module is equipped with a light source that outputs infrared light, and outputs the infrared light when performing infrared imaging. The moving device described in (89) above. (91) The camera module switches between color and infrared photography based on the ambient brightness. The moving device described in (89) or (90) above. (92) The mobile device according to any one of (89) to (91), further comprising a recognition unit that performs lip-reading of the driver based on the image of the camera module.

[0671] Furthermore, the effects described herein are merely illustrative and not limiting; other effects may also occur. [Explanation of symbols]

[0672] 1 Vehicle, 11 Vehicle control system, 21 Processor, 25 External recognition sensor, 26 In-vehicle sensor, 30 DMS, 31 HMI, 51 Camera, 52 Radar, 53 LiDAR, 73 Recognition unit, 84 Body system control unit, 85 Light control unit, 101 Driver's seat, 102 Passenger seat, 103, 103L, 103R Rear seats, 104 Windshield, 106 Steering wheel, 107 Console, 108 Controller, 109 Operation unit, 151FL to 151BR Camera, 201 Center display, 201C Center section, 201CL to 201CR Display section, 201L Left end, 201LL Display section, 201R Right end, 201RR Display section, 202 Console display, 203 Display, 204 Digital rearview mirror, 205 Steering wheel display, 206 Rear entertainment display, 601 Accessory lights, 602 Daytime running lights, 603 Headlights, 604 Turn signals, 605 Auxiliary lights, 606 Taillights, 607 Brake lights, 651 Steering wheel, 652 Illumination, 653FL to BR Speakers, 654 Loop lights, 655 Stork levers, 656 Buttons, 657 Indicators, 658FC to 658FR Air vents, 659CL to 659FR Knobs, 660 Control section, 661FL, 661FR Door openers, 662L, 662R Tablet terminals, 663 Mounting section, 701 Information processing section, 711 Image processing unit, 712 Metadata assignment unit, 713 Output control unit, 751 Main light, 1001 Steering wheel, 1002 Illumination, 1003,1004 Operation unit, 1007,1008 Camera module, 1021 to 1023R Seat speaker, 1042 Camera, 1043 LED

Claims

1. In a mobile device, A metadata assignment unit assigns metadata to video data shot while moving, including data relating to one or more of the following: the conditions inside the room, the shooting location, the shooting date and time, the condition of the mobile device, and the conditions around the mobile device. An image processing unit generates edited video data by extracting and combining frames from multiple video data captured by multiple cameras based on the path of the mobile device and the metadata. A mobile device equipped with the following features.

2. The metadata includes the activity level of the room. The mobile device according to claim 1.

3. The image processing unit extracts frames from the recorded video data of the room based on the activity level. The mobile device according to claim 2.

4. If the image processing unit detects a rise in the room based on the activity level, it preferentially extracts frames from the video data of the room that were captured. The mobile device according to claim 3.

5. The metadata assignment unit sets the activity level based on at least one of the following: the volume of conversation in the room and the movement of the passengers. A mobile device according to any one of claims 2 to 4.

6. The aforementioned video data is video data of the surroundings of the mobile device or the interior of the room. A mobile device according to any one of claims 1 to 5.

7. The image processing unit superimposes visual information onto the frames of the captured video data based on the metadata. A mobile device according to any one of claims 1 to 6.

8. The image processing unit edits the captured video data based on user instructions. A mobile device according to any one of claims 1 to 7.

9. The metadata assignment unit assigns the metadata entered by the user to the recorded video data. A mobile device according to any one of claims 1 to 8.

10. The image processing unit preferentially extracts frames of the captured video data that have been captured in a specific direction along the path. A mobile device according to any one of claims 1 to 9.

11. The image processing unit preferentially extracts frames of the captured video data that show landmarks around the mobile device. A mobile device according to any one of claims 1 to 10.

12. The image processing unit preferentially extracts frames of the captured video data when a specific event occurs. A mobile device according to any one of claims 1 to 11.

13. The image processing unit preferentially extracts frames of the captured video data from the time period before and after the event, or frames of the captured video data that were captured in the direction in which the event occurred. The mobile device according to claim 12.

14. The image processing unit generates the edited video data by connecting the extracted frames in chronological order, or by arranging or overlapping multiple extracted frames within the same frame. A mobile device according to any one of claims 1 to 13.

15. The mobile device, The system involves adding metadata to video data shot while moving, including data relating to one or more of the following: the conditions inside the room, the shooting location, the date and time of shooting, the condition of the mobile device, and the conditions around the mobile device. Based on the route of the mobile device and the metadata, frames are extracted from multiple video data captured by multiple cameras and combined to generate edited video data. Information processing methods including

16. A metadata set attached to video footage captured while a mobile device is moving includes data relating to one or more of the following: the interior of the mobile device, the shooting location, the shooting date and time, the status of the mobile device, and the surrounding conditions of the mobile device; and an image processing unit that generates edited video data by extracting and combining frames from multiple video footage captured by multiple cameras based on the route of the mobile device. Information processing device.

17. The metadata includes the activity level of the room. The information processing apparatus according to claim 16.

18. The image processing unit extracts frames from the recorded video data of the room based on the activity level. The information processing apparatus according to claim 17.

19. If the image processing unit detects a rise in the room based on the activity level, it preferentially extracts frames from the video data of the room that were captured. The information processing apparatus according to claim 18.

20. The aforementioned video data is video data of the surroundings of the mobile device or the interior of the room. An information processing apparatus according to any one of claims 16 to 19.

21. The image processing unit superimposes visual information onto the frames of the captured video data based on the metadata. An information processing apparatus according to any one of claims 16 to 20.

22. The image processing unit edits the captured video data based on user instructions. An information processing apparatus according to any one of claims 16 to 21.

23. The image processing unit preferentially extracts frames of the captured video data that have been captured in a specific direction along the path. An information processing apparatus according to any one of claims 16 to 22.

24. The image processing unit preferentially extracts frames of the captured video data that show landmarks around the mobile device. An information processing apparatus according to any one of claims 16 to 23.

25. The image processing unit preferentially extracts frames of the captured video data when a specific event occurs. An information processing apparatus according to any one of claims 16 to 24.

26. The image processing unit preferentially extracts frames of the captured video data from the time period before and after the event, or frames of the captured video data that were captured in the direction in which the event occurred. The information processing apparatus according to claim 25.

27. ​​The image processing unit generates the edited video data by connecting the extracted frames in chronological order, or by arranging or overlapping multiple extracted frames within the same frame. An information processing apparatus according to any one of claims 16 to 26.

28. Information processing device, The system generates edited video data by extracting and combining frames from multiple video data captured by multiple cameras based on metadata attached to video data captured while a mobile device is moving, including data relating to one or more of the following: the interior of the mobile device, the shooting location, the shooting date and time, the status of the mobile device, and the surrounding conditions of the mobile device, and based on the route of the mobile device. Information processing methods, including those mentioned.

Images