Mobile apparatus, information processing apparatus and information processing method

Abstract

To improve convenience of a passenger of a mobile apparatus.SOLUTION: A mobile apparatus comprises: a metadata giving unit that gives metadata including data associated with a situation within a room to the video data captured while moving; and an image processing unit that edits the video data and generates edited moving image data, on the basis of the metadata. The present technology can be applied to a vehicle for example.SELECTED DRAWING: Figure 37

Description

[Technical field]

[0001] The present technology relates to a mobile device, an information processing device, and an information processing method, and more particularly to a mobile device, an information processing device, and an information processing method that can improve convenience for a passenger of the mobile device. [Background technology]

[0002] In recent years, AR (Augmented Reality) technology has been increasingly introduced into vehicles (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] International Publication No. 2019 / 044536 Summary of the Invention [Problem to be solved by the invention]

[0004] In this way, it is desirable to use AR technology and the like to improve convenience for vehicle drivers and other passengers.

[0005] The present technology has been made in consideration of such circumstances, and is intended to improve convenience for passengers of transportation devices such as vehicles. [Means for solving the problem]

[0006] A mobile device according to a first aspect of the present technology includes a metadata addition unit that adds metadata including data related to indoor conditions to shot video data shot while moving, and an image processing unit that edits the shot video data based on the metadata to generate edited video data.

[0007] An information processing method according to a first aspect of the present technology includes a mobile device adding metadata including data relating to indoor conditions to captured video data captured while moving, and editing the captured video data based on the metadata to generate edited video data.

[0008] A mobile device according to a second aspect of the present technology includes a metadata adding unit that adds metadata including data related to a situation inside a room to video data captured while moving.

[0009] An information processing method according to a second aspect of the present technology includes adding metadata including data relating to a situation inside a room to video data captured by a mobile device while the mobile device is moving.

[0010] An information processing device according to a third aspect of the present technology includes an image processing unit that edits captured video data captured while moving on a mobile device based on metadata including data regarding the indoor situation of the mobile device, and generates edited video data.

[0011] An information processing method according to a third aspect of the present technology includes an information processing device editing captured video data captured while moving on a mobile device based on metadata including data relating to an indoor situation of the mobile device, the metadata being assigned to the captured video data, to generate edited video data.

[0012] In a first aspect of the present technology, metadata including data regarding indoor conditions is added to shot video data shot while moving, and the shot video data is edited based on the metadata to generate edited video data.

[0013] In a second aspect of the present technology, metadata including data related to a situation inside a room is added to video data captured while moving.

[0014] In a third aspect of the present technology, the captured video data is edited based on metadata, including data regarding the indoor situation of the mobile device, that is added to the captured video data captured while moving on the mobile device, and edited video data is generated. [Brief description of the drawings]

[0015] [Figure 1] FIG. 1 is a block diagram showing an example of the configuration of a vehicle control system. [Diagram 2] FIG. 2 is a diagram showing an example of a sensing region. [Diagram 3] FIG. 1 is a diagram showing examples of installation positions of a camera, a LiDAR, and a radar. [Figure 4] FIG. [Diagram 5] FIG. 2 is a schematic diagram showing the driver's seat and passenger seat areas viewed from above. [Figure 6] FIG. 2 is a schematic diagram of the passenger seat as seen from the left side. [Figure 7] FIG. 2 is a schematic diagram of the driver's seat and passenger seat area viewed from the left rear. [Figure 8] FIG. 2 is a schematic diagram showing the front of the vehicle as viewed from the driver's seat. [Figure 9] FIG. 2 is a schematic diagram of the passenger seat as seen from the left rear. [Figure 10] FIG. 4 is a diagram showing an example of the display content in the center part of the center display. [Figure 11] FIG. 4 is a diagram showing an example of the display content in the center part of the center display. [Figure 12] FIG. 4 is a diagram showing an example of the display content in the center part of the center display. [Figure 13] FIG. 11 is a diagram showing an example of the display content at the left end of the center display. [Figure 14] FIG. 4 is a diagram illustrating an example of display content of a head-up display. [Figure 15] FIG. 4 is a diagram illustrating an example of display content of a head-up display. [Figure 16] FIG. 11 is a diagram showing an example of the display content of the digital rear-view mirror. [Figure 17]FIG. 4 is a diagram showing an example of the display content of a steering wheel display. [Figure 18] FIG. 4 is a diagram showing an example of the display content of a steering wheel display. [Figure 19] FIG. 4 is a diagram showing an example of the display content of a rear entertainment display. [Figure 20] FIG. 2 is a front view of the vehicle and an enlarged view of the area around the logo. [Figure 21] FIG. [Figure 22] FIG. 2 is an enlarged view of the tip portion of the left side of the vehicle. [Figure 23] FIG. [Figure 24] FIG. 2 is a view of the left headlight area of ​​the vehicle. [Diagram 25] FIG. 2 is a view of the left headlight area of ​​the vehicle. [Figure 26] FIG. 2 is a view of the left headlight area of ​​the vehicle. [Figure 27] FIG. 2 is a schematic view of the interior of the vehicle as viewed from the right side. [Figure 28] FIG. 2 is a schematic diagram of the vicinity of the driver's seat and passenger seat of the vehicle. [Figure 29] FIG. 1 is a schematic diagram of the vicinity of a dashboard of a vehicle. [Diagram 30] FIG. 2 is an enlarged view of the steering wheel of the vehicle. [Diagram 31] This is a view of the driver's seat seen from diagonally rear left. [Diagram 32] FIG. 1 is a diagram showing an example of an installation position of a ToF camera for photographing a driver. [Diagram 33] FIG. 1 is a diagram showing an example of an installation position of a ToF camera for photographing a driver. [Diagram 34] FIG. 34 is a schematic diagram showing an example of an image captured by a ToF camera installed at the installation position shown in FIGS. 32 and 33. [Diagram 35] FIG. 1 is a schematic diagram showing an example of an image captured by a ToF camera installed near a steering wheel. [Diagram 36] FIG. 1 is a diagram showing an example of an installation position of a ToF camera for photographing a driver. [Figure 37] 4 is a block diagram showing a configuration example of an information processing unit. FIG. [Figure 38] FIG. 11 is a diagram showing an example of a light emission pattern of a light when the power is turned on. [Figure 39] FIG. 11 is a diagram showing an example of a light emission pattern of a light when the power is turned on. [Diagram 40] 11A and 11B are diagrams illustrating examples of light emission patterns of lights when driving and the headlights are turned off. [Diagram 41] FIG. 13 is a diagram showing an example of a light emission pattern when the headlights are turned on while driving. [Diagram 42] FIG. 11 is a diagram showing an example of a light emission pattern of a light when the brakes are applied. [Diagram 43] 11A and 11B are diagrams illustrating examples of light emission patterns of lights when a turn signal is activated. [Diagram 44] FIG. 13 is a diagram showing an example of a light emission pattern when the door is open. [Diagram 45] FIG. 13 is a diagram showing an example of a light emission pattern when a door is closed. [Figure 46] FIG. 11 is a diagram showing an example of a light emission pattern of a light when parking. [Figure 47] FIG. 13 is a diagram showing an example of a light emission pattern of a light when the power is off. [Figure 48] 11 is a flowchart illustrating a photographing process. [Figure 49] 13 shows an example of a video when the object detection mode is set. [Figure 50] 13 shows an example of a moving image when a prediction mode is set. [Figure 51] 13 shows an example of a video when the search mode is set. [Figure 52] 13 shows an example of a video when the search mode is set. [Diagram 53] 13 is a flowchart illustrating an editing process. [Figure 54] This is a schematic diagram of the interior of the vehicle looking down from the right side. [Figure 55] FIG. [Figure 56]FIG. 2 is a schematic diagram showing the interior of a vehicle in front of the driver's seat. [Figure 57] FIG. [Figure 58] 1A to 1C are diagrams illustrating examples of installation positions of ToF cameras in a camera module. [Figure 59] FIG. 1 is a diagram illustrating an example of a shooting range of a ToF camera. [Figure 60] FIG. 11 is a diagram showing another example of the installation position of the ToF camera. [Figure 61] FIG. 11 is a diagram showing another example of the installation position of the ToF camera. [Figure 62] FIG. 11 is a diagram showing another example of the installation position of the ToF camera. [Figure 63] FIG. 11 is a diagram showing another example of the installation position of the ToF camera. [Figure 64] 1A and 1B are diagrams illustrating examples of installation positions of camera modules. [Figure 65] FIG. 2 is a diagram illustrating an example of the internal configuration of a camera module. [Figure 66] 13A and 13B are diagrams illustrating modified examples of the installation positions of the cameras. [Figure 67] FIG. 2 is a diagram showing an example of a shooting range of a camera. [Figure 68] FIG. 13 is a diagram showing an example of a display on the display unit for CMS in a normal state. [Figure 69] FIG. 11 is a diagram showing a first example of a display on a display unit for a CMS when a vehicle in an adjacent lane is approaching. [Figure 70] FIG. 13 is a diagram showing an example of a display on the display unit for CMS when a direction indicator is on. [Figure 71] FIG. 13 is a diagram showing a second example of the display on the display unit for the CMS when a vehicle in an adjacent lane is approaching. [Figure 72] 13A and 13B are diagrams illustrating an example of a display on a display unit for a CMS when a direction indicator is in operation. [Figure 73] FIG. 13 is a diagram showing an example of a required visibility range. [Figure 74] 11 is a diagram for explaining an example of changing the display range of a CMS display unit based on a shift position. FIG. [Figure 75]FIG. 13 is a diagram for explaining a method for setting the display range of a display unit for a CMS. [Figure 76] FIG. 13 is a diagram for explaining a method for setting the display range of a display unit for a CMS. [Figure 77] 11A and 11B are diagrams illustrating an example of a zoom ratio and a display range of a display unit for a CMS. [Figure 78] FIG. 13 is a diagram for explaining a method for controlling the display range when looking into a display unit for CMS. [Figure 79] FIG. 13 is a diagram for explaining a method for controlling the display range when looking into a display unit for CMS. [Figure 80] 13 is a diagram for explaining a method for controlling the display range when the shift position is set to reverse and the driver looks into the CMS display unit. FIG. [Figure 81] 13 is a diagram for explaining a method for controlling the display range when the shift position is set to reverse and the driver looks into the CMS display unit. FIG. [Figure 82] 13 is a diagram for explaining a method for controlling the display range when the shift position is set to reverse and the driver looks into the CMS display unit. FIG. [Figure 83] 13 is a diagram for explaining a method of controlling footlights when looking down at the feet. FIG. [Figure 84] 13 is a diagram for explaining a method of controlling a display unit for a CMS when a door opener is touched. FIG. [Figure 85] 13 is a diagram for explaining a method of controlling a display unit for a CMS when a door opener is touched. FIG. [Figure 86] 13 is a diagram for explaining a method of controlling a display unit for a CMS when a door opener is touched. FIG. [Figure 87] 13A and 13B are diagrams showing the illumination states of the display unit for the driver's seat and the steering wheel before the driver is recognized. [Figure 88] 13A and 13B are diagrams showing the illumination states of the display unit for the driver's seat and the steering wheel when the driver recognizes the vehicle. [Figure 89]A figure showing the state of the driver's seat display and steering wheel illumination when preparations for autonomous driving are complete. [Figure 90] A figure showing the state of the illumination of the display unit for the driver's seat and the steering wheel when autonomous driving starts. [Figure 91] A figure showing the state of illumination of the driver's seat display and steering wheel when changing lanes during autonomous driving. [Figure 92] 11A and 11B are diagrams showing the illumination states of the display unit for the driver's seat and the steering wheel when a danger is detected. [Figure 93] 11A and 11B are diagrams for explaining examples of guidance displays when operating a button. [Figure 94] 11A and 11B are diagrams for explaining examples of guidance displays when operating a button. [Figure 95] 13A and 13B are diagrams illustrating an example of a display on a display unit for a driver's seat during lip reading. [Figure 96] 11A and 11B are diagrams for explaining a method of controlling a warning sound that alerts people in front of the vehicle. [Figure 97] FIG. 13 is a diagram for explaining a method for controlling sound effects at the start of autonomous driving. [Figure 98] 11A and 11B are diagrams for explaining a method for controlling sound effects during acceleration during autonomous driving. [Figure 99] 11A and 11B are diagrams for explaining a method for controlling sound effects during deceleration during autonomous driving. [Figure 100] 11A and 11B are diagrams for explaining a method of controlling a sound effect before a voice message is output. [Figure 101] 11A and 11B are diagrams for explaining a method of controlling a warning sound for a vehicle approaching from the left rear. [Figure 102] 11A and 11B are diagrams for explaining a method of controlling a warning sound for a vehicle approaching from the left rear. [Figure 103] 11A and 11B are diagrams for explaining a method of controlling a warning sound for a vehicle approaching from the left rear. [Figure 104] 11A and 11B are diagrams for explaining a method of controlling a warning sound for a vehicle approaching from the left rear. [Figure 105] FIG. 13 is a diagram for explaining a method of controlling audio for passengers in the rear seats. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0016] Hereinafter, an embodiment of the present technology will be described in the following order. 1. Example of vehicle control system configuration 2. First embodiment 3. Modification of the First Embodiment 4. Second embodiment 5. Modification of the Second Embodiment 6. Third embodiment 7. Modification of the Third Embodiment 8.Other

[0017] <<1. Example of vehicle control system configuration>> FIG. 1 is a block diagram showing an example of the configuration of a vehicle control system 11, which is an example of a mobility device control system to which the present technology is applied.

[0018] The vehicle control system 11 is provided in the vehicle 1 and performs processing related to driving assistance and automatic driving of the vehicle 1.

[0019] The vehicle control system 11 includes a processor 21, a communication unit 22, a map information storage unit 23, a GNSS (Global Navigation Satellite System) receiving unit 24, an external recognition sensor 25, an in-vehicle sensor 26, a vehicle sensor 27, a recording unit 28, a driving assistance / automatic driving control unit 29, a DMS (Driver Monitoring System) 30, an HMI (Human Machine Interface) 31, and a vehicle control unit 32.

[0020] The processor 21, the communication unit 22, the map information storage unit 23, the GNSS receiving unit 24, the external recognition sensor 25, the in-vehicle sensor 26, the vehicle sensor 27, the recording unit 28, the driving support / automatic driving control unit 29, the driver monitoring system (DMS) 30, the human machine interface (HMI) 31, and the vehicle control unit 32 are connected to each other via a communication network 41. The communication network 41 is configured by an in-vehicle communication network or bus conforming to any standard such as CAN (Controller Area Network), LIN (Local Interconnect Network), LAN (Local Area Network), FlexRay (registered trademark), Ethernet (registered trademark), etc. Note that each unit of the vehicle control system 11 may be directly connected to each other by, for example, near field communication (NFC (Near Field Communication)) or Bluetooth (registered trademark) without going through the communication network 41.

[0021] In the following description, when each unit of the vehicle control system 11 communicates with each other via the communication network 41, the description of the communication network 41 will be omitted. For example, when the processor 21 and the communication unit 22 communicate with each other via the communication network 41, it will be simply described that the processor 21 and the communication unit 22 communicate with each other.

[0022] The processor 21 is configured by various processors such as a CPU (Central Processing Unit), an MPU (Micro Processing Unit), an ECU (Electronic Control Unit), etc. The processor 21 controls the vehicle control system 11 as a whole.

[0023] The communication unit 22 communicates with various devices inside and outside the vehicle, other vehicles, servers, base stations, etc., and transmits and receives various data. As communication with the outside of the vehicle, for example, the communication unit 22 receives from the outside a program for updating software that controls the operation of the vehicle control system 11, map information, traffic information, information about the surroundings of the vehicle 1, etc. For example, the communication unit 22 transmits information about the vehicle 1 (for example, data indicating the state of the vehicle 1, recognition results by the recognition unit 73, etc.), information about the surroundings of the vehicle 1, etc., to the outside. For example, the communication unit 22 performs communication corresponding to a vehicle emergency notification system such as e-Call.

[0024] There is no particular limitation on the communication method of the communication unit 22. A plurality of communication methods may be used.

[0025] For example, the communication unit 22 performs wireless communication with devices in the vehicle using a communication method such as wireless LAN, Bluetooth, NFC, WUSB (Wireless USB), etc. For example, the communication unit 22 performs wired communication with devices in the vehicle using a communication method such as USB (Universal Serial Bus), HDMI (High-Definition Multimedia Interface, registered trademark), or MHL (Mobile High-definition Link) via a connection terminal (and a cable, if necessary) not shown.

[0026] Here, the in-vehicle device is, for example, a device that is not connected to the communication network 41 in the vehicle. For example, a mobile device or a wearable device carried by a passenger such as a driver, an information device brought into the vehicle and temporarily installed, etc. are assumed.

[0027] For example, the communication unit 22 communicates with servers and the like existing on an external network (e.g., the Internet, a cloud network, or an operator-specific network) via a base station or an access point using a wireless communication method such as 4G (fourth generation mobile communication system), 5G (fifth generation mobile communication system), LTE (Long Term Evolution), or DSRC (Dedicated Short Range Communications).

[0028] For example, the communication unit 22 communicates with a terminal (for example, a pedestrian or a store terminal, or an MTC (Machine Type Communication) terminal) present in the vicinity of the vehicle by using a P2P (Peer To Peer) technique. For example, the communication unit 22 performs V2X communication. The V2X communication includes, for example, vehicle-to-vehicle communication with other vehicles, vehicle-to-infrastructure communication with roadside devices, vehicle-to-home communication, and vehicle-to-pedestrian communication with terminals held by pedestrians.

[0029] For example, the communication unit 22 receives electromagnetic waves transmitted by a road traffic information and communication system (VICS (Vehicle Information and Communication System), registered trademark), such as a radio beacon, an optical beacon, or an FM multiplex broadcast.

[0030] The map information storage unit 23 stores maps acquired from an external source and maps created by the vehicle 1. For example, the map information storage unit 23 stores a three-dimensional high-precision map, a global map that has lower precision than a high-precision map and covers a wide area, and the like.

[0031] The high-precision map is, for example, a dynamic map, a point cloud map, a vector map (also called an ADAS (Advanced Driver Assistance System) map), etc. The dynamic map is, for example, a map consisting of four layers of dynamic information, quasi-dynamic information, quasi-static information, and static information, and is provided from an external server or the like. The point cloud map is a map configured of a point cloud (point cloud data). The vector map is a map in which information such as the positions of lanes and traffic lights is associated with the point cloud map. The point cloud map and the vector map may be provided from, for example, an external server or the like, or may be created by the vehicle 1 based on sensing results by the radar 52, the LiDAR 53, etc. as a map for matching with a local map to be described later, and may be stored in the map information storage unit 23. In addition, when a high-precision map is provided from an external server or the like, map data of, for example, several hundred meters square, related to the planned route along which the vehicle 1 will travel is acquired from the server or the like in order to reduce communication capacity.

[0032] The GNSS receiver unit 24 receives GNSS signals from GNSS satellites and supplies them to the driving assistance / automatic driving control unit 29.

[0033] The external recognition sensor 25 includes various sensors used to recognize the situation outside the vehicle 1, and supplies sensor data from each sensor to each unit of the vehicle control system 11. The type and number of sensors included in the external recognition sensor 25 are arbitrary.

[0034] For example, the external recognition sensor 25 includes a camera 51, a radar 52, a LiDAR (Light Detection and Ranging, Laser Imaging Detection and Ranging) 53, and an ultrasonic sensor 54. The number of cameras 51, radars 52, LiDARs 53, and ultrasonic sensors 54 is arbitrary, and an example of a sensing area of ​​each sensor will be described later.

[0035] As the camera 51, a camera of any imaging method, such as a ToF (Time Of Flight) camera, a stereo camera, a monocular camera, or an infrared camera, may be used as necessary.

[0036] Furthermore, for example, the external recognition sensor 25 includes an environmental sensor for detecting the weather, climate, brightness, etc. The environmental sensor includes, for example, a raindrop sensor, a fog sensor, a sunlight sensor, a snow sensor, an illuminance sensor, etc.

[0037] Furthermore, for example, the external recognition sensor 25 includes a microphone used to detect sounds around the vehicle 1 and the positions of sound sources.

[0038] The in-vehicle sensor 26 includes various sensors for detecting information inside the vehicle, and supplies sensor data from each sensor to each unit of the vehicle control system 11. The in-vehicle sensor 26 may include any type and any number of sensors.

[0039] For example, the in-vehicle sensor 26 includes a camera, a radar, a seating sensor, a steering wheel sensor, a microphone, a biosensor, etc. The camera may be a camera of any imaging method, such as a ToF camera, a stereo camera, a monocular camera, an infrared camera, etc. The biosensor is provided, for example, on a seat or a steering wheel, and detects various types of bioinformation of a passenger such as a driver.

[0040] The vehicle sensor 27 includes various sensors for detecting the state of the vehicle 1, and supplies sensor data from each sensor to each unit of the vehicle control system 11. The type and number of sensors included in the vehicle sensor 27 are arbitrary.

[0041] For example, the vehicle sensor 27 includes a speed sensor, an acceleration sensor, an angular velocity sensor (gyro sensor), and an inertial measurement unit (IMU). For example, the vehicle sensor 27 includes a steering angle sensor that detects the steering angle of the steering wheel, a yaw rate sensor, an accelerator sensor that detects the amount of operation of an accelerator pedal, and a brake sensor that detects the amount of operation of a brake pedal. For example, the vehicle sensor 27 includes a rotation sensor that detects the number of rotations of the engine or the motor, an air pressure sensor that detects the air pressure of the tires, a slip ratio sensor that detects the slip ratio of the tires, and a wheel speed sensor that detects the rotation speed of the wheels. For example, the vehicle sensor 27 includes a battery sensor that detects the remaining charge and temperature of the battery, and an impact sensor that detects an impact from the outside.

[0042] The recording unit 28 includes, for example, a magnetic storage device such as a ROM (Read Only Memory), a RAM (Random Access Memory), a HDD (Hard Disc Drive), a semiconductor storage device, an optical storage device, and a magneto-optical storage device. The recording unit 28 records various programs and data used by each part of the vehicle control system 11. For example, the recording unit 28 records a rosbag file including messages transmitted and received by a ROS (Robot Operating System) on which an application program related to autonomous driving runs. For example, the recording unit 28 includes an EDR (Event Data Recorder) and a DSSAD (Data Storage System for Automated Driving), and records information on the vehicle 1 before and after an event such as an accident.

[0043] The driving support / automatic driving control unit 29 controls driving support and automatic driving of the vehicle 1. For example, the driving support / automatic driving control unit 29 includes an analysis unit 61, an action planning unit 62, and an operation control unit 63.

[0044] The analysis unit 61 performs an analysis process of the vehicle 1 and the surrounding situation. The analysis unit 61 includes a self-position estimation unit 71, a sensor fusion unit 72, and a recognition unit 73.

[0045] The self-position estimation unit 71 estimates the self-position of the vehicle 1 based on the sensor data from the external recognition sensor 25 and the high-precision map stored in the map information storage unit 23. For example, the self-position estimation unit 71 generates a local map based on the sensor data from the external recognition sensor 25, and estimates the self-position of the vehicle 1 by matching the local map with the high-precision map. The position of the vehicle 1 is based on, for example, the center of the rear wheel pair axle.

[0046] The local map is, for example, a three-dimensional high-precision map or an occupancy grid map created using a technique such as SLAM (Simultaneous Localization and Mapping). The three-dimensional high-precision map is, for example, the above-mentioned point cloud map. The occupancy grid map is a map in which the three-dimensional or two-dimensional space around the vehicle 1 is divided into grids of a predetermined size, and the occupancy state of an object is indicated on a grid-by-grid basis. The occupancy state of an object is indicated, for example, by the presence or absence of an object and the probability of its existence. The local map is also used, for example, in detection processing and recognition processing of the external situation of the vehicle 1 by the recognition unit 73.

[0047] The self-position estimation unit 71 may estimate the self-position of the vehicle 1 based on the GNSS signal and sensor data from the vehicle sensor 27.

[0048] The sensor fusion unit 72 performs a sensor fusion process to obtain new information by combining a plurality of different types of sensor data (for example, image data supplied from the camera 51 and sensor data supplied from the radar 52). Methods for combining different types of sensor data include integration, fusion, and association.

[0049] The recognition unit 73 performs detection processing and recognition processing of the situation outside the vehicle 1.

[0050] For example, the recognition unit 73 performs detection processing and recognition processing of the situation outside the vehicle 1 based on information from the external recognition sensor 25, information from the self-position estimation unit 71, information from the sensor fusion unit 72, and the like.

[0051] Specifically, for example, the recognition unit 73 performs detection processing and recognition processing of objects around the vehicle 1. The object detection processing is, for example, processing to detect the presence or absence, size, shape, position, movement, etc. of an object. The object recognition processing is, for example, processing to recognize attributes such as the type of object, or to identify a specific object. However, the detection processing and the recognition processing are not necessarily clearly separated, and may overlap.

[0052] For example, the recognition unit 73 performs clustering to classify a point cloud based on sensor data such as LiDAR or radar into clusters of points, thereby detecting objects around the vehicle 1. This allows the presence or absence, size, shape, and position of objects around the vehicle 1 to be detected.

[0053] For example, the recognition unit 73 performs tracking to follow the movement of the clusters of the point cloud classified by clustering, thereby detecting the movement of the objects around the vehicle 1. As a result, the speed and traveling direction (movement vector) of the objects around the vehicle 1 are detected.

[0054] For example, the recognition unit 73 performs object recognition processing such as semantic segmentation on the image data supplied from the camera 51 to recognize the type of object around the vehicle 1.

[0055] Note that objects to be detected or recognized may include, for example, vehicles, people, bicycles, obstacles, structures, roads, traffic lights, traffic signs, road markings, and the like.

[0056] For example, the recognition unit 73 performs a recognition process of traffic rules around the vehicle 1 based on the map stored in the map information storage unit 23, the estimation result of the vehicle's own position, and the recognition result of objects around the vehicle 1. Through this process, for example, the positions and states of traffic signals, the contents of traffic signs and road markings, the contents of traffic regulations, and lanes on which travel is possible are recognized.

[0057] For example, the recognition unit 73 performs a process of recognizing the environment around the vehicle 1. Examples of the surrounding environment to be recognized include the weather, temperature, humidity, brightness, and road surface condition.

[0058] The behavior planning unit 62 creates a behavior plan for the vehicle 1. For example, the behavior planning unit 62 creates the behavior plan by performing route planning and route following processing.

[0059] Global path planning is a process for planning a rough route from the start to the goal. This route planning also includes a process for local path planning, which is called trajectory planning and allows the vehicle 1 to proceed safely and smoothly in the vicinity of the vehicle 1, taking into account the motion characteristics of the vehicle 1 on the route planned by the route planning.

[0060] Path following is a process of planning an operation for traveling safely and accurately along a route planned by a route plan within a planned time. For example, a target speed and a target angular velocity of the vehicle 1 are calculated.

[0061] The operation control unit 63 controls the operation of the vehicle 1 in order to realize the action plan created by the action planning unit 62.

[0062] For example, the operation control unit 63 controls the steering control unit 81, the brake control unit 82, and the drive control unit 83 to perform acceleration / deceleration control and direction control so that the vehicle 1 proceeds along the trajectory calculated by the trajectory plan. For example, the operation control unit 63 performs cooperative control for the purpose of realizing ADAS functions such as collision avoidance or impact mitigation, following driving, maintaining vehicle speed, collision warning for the vehicle itself, and lane departure warning for the vehicle itself. For example, the operation control unit 63 performs cooperative control for the purpose of automatic driving that autonomously drives without the driver's operation.

[0063] The DMS 30 performs authentication processing of the driver and recognition processing of the driver's state, etc., based on the sensor data from the in-vehicle sensor 26 and the input data input to the HMI 31. Examples of the driver's state to be recognized include physical condition, alertness level, concentration level, fatigue level, line of sight direction, level of intoxication, driving operation, posture, etc.

[0064] The DMS 30 may perform authentication processing of passengers other than the driver and recognition processing of the conditions of the passengers. For example, the DMS 30 may perform recognition processing of the conditions inside the vehicle based on sensor data from the in-vehicle sensor 26. Conditions inside the vehicle to be recognized include, for example, temperature, humidity, brightness, odor, and the like.

[0065] The HMI 31 is used to input various data, instructions, etc., generates input signals based on the input data, instructions, etc., and supplies them to each part of the vehicle control system 11. For example, the HMI 31 includes operation devices such as a touch panel, buttons, microphones, switches, and levers, as well as operation devices that allow input by means of voice, gestures, etc. other than manual operation. The HMI 31 may be, for example, a remote control device that uses infrared rays or other radio waves, or an externally connected device such as a mobile device or wearable device that supports the operation of the vehicle control system 11.

[0066] The HMI 31 also performs output control to generate and output visual information, auditory information, and tactile information for the occupant or the outside of the vehicle, as well as to control the output content, output timing, output method, etc. Visual information is information displayed by images or light, such as an operation screen, a status display of the vehicle 1, a warning display, a monitor image showing the situation around the vehicle 1, etc. Auditory information is information displayed by sound, such as guidance, warning sound, warning message, etc. Tactile information is information provided to the occupant's sense of touch by force, vibration, movement, etc.

[0067] Possible devices for outputting visual information include, for example, a display device, a projector, a navigation device, an instrument panel, a CMS (Camera Monitoring System), an electronic mirror, a lamp, etc. The display device may be a device having a normal display, or may be a device that displays visual information within the field of view of a passenger, such as, for example, a head-up display, a see-through display, or a wearable device with an AR (Augmented Reality) function.

[0068] Devices that output auditory information include, for example, audio speakers, headphones, earphones, and the like.

[0069] As a device for outputting tactile information, for example, a haptic element using haptic technology is assumed, and the haptic element is provided, for example, on a steering wheel, a seat, or the like.

[0070] The vehicle control unit 32 controls each part of the vehicle 1. The vehicle control unit 32 includes a steering control unit 81, a brake control unit 82, a drive control unit 83, a body system control unit 84, a light control unit 85, and a horn control unit 86.

[0071] The steering control unit 81 detects and controls the state of the steering system of the vehicle 1. The steering system includes, for example, a steering mechanism including a steering wheel, an electric power steering, etc. The steering control unit 81 includes, for example, a control unit such as an ECU that controls the steering system, and an actuator that drives the steering system.

[0072] The brake control unit 82 detects and controls the state of the brake system of the vehicle 1. The brake system includes, for example, a brake mechanism including a brake pedal, an ABS (Antilock Brake System), etc. The brake control unit 82 includes, for example, a control unit such as an ECU that controls the brake system, and an actuator that drives the brake system.

[0073] The drive control unit 83 detects and controls the state of the drive system of the vehicle 1. The drive system includes, for example, an accelerator pedal, a drive force generating device for generating drive force such as an internal combustion engine or a drive motor, a drive force transmission mechanism for transmitting the drive force to the wheels, etc. The drive control unit 83 includes, for example, a control unit such as an ECU that controls the drive system, and an actuator that drives the drive system.

[0074] The body system control unit 84 detects and controls the states of the body system systems of the vehicle 1. The body system systems include, for example, a keyless entry system, a smart key system, a power window device, a power seat, an air conditioning system, an airbag, a seat belt, a shift lever, etc. The body system control unit 84 includes, for example, a control unit such as an ECU that controls the body system systems, and an actuator that drives the body system systems.

[0075] The light control unit 85 detects and controls the states of various lights of the vehicle 1. Examples of lights to be controlled include headlights, backlights, fog lights, turn signals, brake lights, projections, and bumper displays. The light control unit 85 includes a control unit such as an ECU that controls the lights, and an actuator that drives the lights.

[0076] The horn control unit 86 detects and controls the state of the car horn of the vehicle 1. The horn control unit 86 includes, for example, a control unit such as an ECU that controls the car horn, and an actuator that drives the car horn.

[0077] FIG. 2 is a diagram showing an example of a sensing area by the camera 51, the radar 52, the LiDAR 53, and the ultrasonic sensor 54 of the external recognition sensor 25 in FIG.

[0078] Sensing area SA1F and sensing area SA1B are examples of sensing areas of the ultrasonic sensor 54. Sensing area SA1F covers the periphery of the front end of the vehicle 1. Sensing area SA1B covers the periphery of the rear end of the vehicle 1.

[0079] The sensing results in the sensing area SA1F and the sensing area SA1B are used for, for example, parking assistance for the vehicle 1.

[0080] Sensing area SA2F to sensing area SA2B show examples of sensing areas of a short-range or medium-range radar 52. Sensing area SA2F covers a position farther in front of the vehicle 1 than sensing area SA1F. Sensing area SA2B covers a position farther in the rear of the vehicle 1 than sensing area SA1B. Sensing area SA2L covers the rear periphery of the left side of the vehicle 1. Sensing area SA2R covers the rear periphery of the right side of the vehicle 1.

[0081] The sensing results in the sensing area SA2F are used, for example, to detect vehicles, pedestrians, and the like that exist in front of the vehicle 1. The sensing results in the sensing area SA2B are used, for example, for a collision prevention function behind the vehicle 1. The sensing results in the sensing areas SA2L and SA2R are used, for example, to detect objects in blind spots on the sides of the vehicle 1.

[0082] Sensing area SA3F to sensing area SA3B show examples of sensing areas by the camera 51. Sensing area SA3F covers a position farther in front of the vehicle 1 than sensing area SA2F. Sensing area SA3B covers a position farther in the rear of the vehicle 1 than sensing area SA2B. Sensing area SA3L covers the periphery of the left side of the vehicle 1. Sensing area SA3R covers the periphery of the right side of the vehicle 1.

[0083] The sensing results in the sensing area SA3F are used, for example, for recognizing traffic signals and traffic signs, lane departure prevention support systems, etc. The sensing results in the sensing area SA3B are used, for example, for parking support and surround view systems, etc. The sensing results in the sensing areas SA3L and SA3R are used, for example, for surround view systems, etc.

[0084] Sensing area SA4 is an example of a sensing area of ​​the LiDAR 53. Sensing area SA4 covers a position farther than sensing area SA3F in front of the vehicle 1. On the other hand, sensing area SA4 has a narrower range in the left-right direction than sensing area SA3F.

[0085] The sensing results in the sensing area SA4 are used for, for example, emergency braking, collision avoidance, pedestrian detection, and the like.

[0086] Sensing area SA5 is an example of a sensing area of ​​the long-distance radar 52. Sensing area SA5 covers a position farther than sensing area SA4 in front of the vehicle 1. On the other hand, sensing area SA5 has a narrower range in the left-right direction than sensing area SA4.

[0087] The sensing result in the sensing area SA5 is used for, for example, ACC (Adaptive Cruise Control) and the like.

[0088] The sensing area of ​​each sensor may have various configurations other than that shown in Fig. 2. Specifically, the ultrasonic sensor 54 may also sense the sides of the vehicle 1, and the LiDAR 53 may sense the rear of the vehicle 1.

[0089] <<2. First embodiment>> Next, a first embodiment of the present technology will be described with reference to FIGS.

[0090] <Examples of installation positions of the external recognition sensor 25 and the in-vehicle sensor 26> 3 shows an example of the installation positions of the camera 51, the radar 52, and the LiDAR 53 of the external recognition sensor 25 of the vehicle 1A according to the first embodiment of the present technology, and the camera of the in-vehicle sensor 26. In this example, the external recognition sensor 25 includes the cameras 51FC to 51BR, the radars 52FC to 52BR, and the LiDARs 53F to 53B. The in-vehicle sensor 26 includes the cameras 151FL to 151BR.

[0091] The camera 51FC is provided near the center of the front end of the vehicle 1A. The camera 51FC captures an image of the area ahead of the vehicle 1A.

[0092] The camera 51FL is provided near the left end of the front end of the vehicle 1A. The camera 51FL captures an image of the area diagonally forward to the left of the vehicle 1A.

[0093] The camera 51FR is provided near the right end of the front end of the vehicle 1A and captures an image diagonally to the right front of the vehicle 1A.

[0094] The camera 51M is provided near the center of the front of the interior of the vehicle 1A. The camera 51M captures an image of the area in front of the vehicle 1A through the windshield 104.

[0095] Camera 51SL is provided on the left side of vehicle 1A near the front end of the door of driver's seat 101. Camera 51SL photographs the area diagonally rear left (left side and rear) of vehicle 1A.

[0096] Camera 51SR is provided on the right side of vehicle 1A near the front end of the door of passenger seat 102 arranged next to driver's seat 101. Camera 51SR photographs the right rear (right side and rear) of vehicle 1A.

[0097] The camera 51BC is provided near the center of the rear end of the vehicle 1A. The camera 51BC photographs the rear of the vehicle 1A.

[0098] The camera 51BL is provided near the left end of the rear end of the vehicle 1A. The camera 51BL captures an image of the area diagonally rear to the left of the vehicle 1A.

[0099] The camera 51BR is provided near the right end of the rear end of the vehicle 1A and captures an image of the area diagonally rear to the right of the vehicle 1A.

[0100] The radar 52FC is provided near the center of the front end of the vehicle 1A. The radar 52FC senses the area ahead of the vehicle 1A.

[0101] The radar 52FL is provided near the left end of the front end of the vehicle 1A. The radar 52FL performs sensing in the diagonally forward left direction of the vehicle 1A.

[0102] The radar 52FR is provided near the right end of the front end of the vehicle 1A. The radar 52FR performs sensing in the diagonally forward right direction of the vehicle 1A.

[0103] The radar 52SL is provided in front of the left side of the vehicle 1A. The radar 52SL performs sensing on the left side of the vehicle 1A.

[0104] The radar 52SR is provided in front of the right side of the vehicle 1A. The radar 52SL senses the right side of the vehicle 1A.

[0105] The radar 52BC is provided near the center of the rear end of the vehicle 1A. The radar 52BC senses the area behind the vehicle 1A.

[0106] The radar 52BL is provided near the left end of the rear end of the vehicle 1A. The radar 52BL performs sensing in the left rear diagonal direction of the vehicle 1A.

[0107] The radar 52BR is provided near the right end of the rear end of the vehicle 1A. The radar 52BR performs sensing in the diagonally rear right area of ​​the vehicle 1A.

[0108] The LiDAR 53F is provided near the center of the front end of the vehicle 1A. The LiDAR 53F performs sensing of the area ahead of the vehicle 1A.

[0109] The LiDAR 53L is provided in front of the left side of the vehicle 1A. The LiDAR 53L performs sensing on the left side of the vehicle 1A.

[0110] The LiDAR 53R is provided in front of the right side of the vehicle 1A. The LiDAR 53R performs sensing on the right side of the vehicle 1A.

[0111] The LiDAR 53B is provided near the center of the rear end of the vehicle 1A. The LiDAR 53B performs sensing behind the vehicle 1A.

[0112] Camera 151FL is provided inside the vehicle near the left front of driver's seat 101. Camera 151FL captures images of the surroundings of the driver (a person sitting in driver's seat 101).

[0113] The camera 151FR is provided inside the vehicle near the right front of the passenger seat 102. The camera 151FR captures an image of the surroundings of a passenger sitting in the passenger seat 102.

[0114] The camera 151BL is provided in the vehicle near the front of the left rear seat 103L. The camera 151BL captures an image of the surroundings of a passenger sitting in the rear seat 103L.

[0115] The camera 151BR is provided in the vehicle near the front of the right rear seat 103R, and captures an image of the surroundings of a passenger sitting in the rear seat 103R.

[0116] Hereinafter, when there is no need to distinguish between the rear seats 103L and 103R, they will simply be referred to as the rear seats 103.

[0117] <Example of in-car display configuration> Next, with reference to Figs. 4 to 9, a configuration example of a display (display unit) that is provided inside the vehicle 1A, constitutes part of the HMI 31, and constitutes the display system of the vehicle 1A will be described. Fig. 4 is a schematic diagram showing the front of the vehicle interior. Fig. 5 is a schematic diagram of the driver's seat 101 and the passenger seat 102 periphery viewed from above. Fig. 6 is a schematic diagram of the passenger seat 102 viewed from the left side. Fig. 7 is a schematic diagram of the driver's seat 101 and the passenger seat 102 periphery viewed from the left rear. Fig. 8 is a schematic diagram of the front of the vehicle 1A viewed from the driver's seat 101. Fig. 9 is a schematic diagram of the passenger seat 102 viewed from the left rear.

[0118] A center display 201, a console display 202, a head-up display (only display 203 is shown), a digital rear-view mirror 204, a steering wheel display 205, and a rear entertainment display 206 are provided inside the vehicle.

[0119] The center display 201 is provided in front of the driver's seat 101 and the passenger seat 102, extending left and right on the front surface of the dashboard 105. The center display 201 is largely divided into a left end portion 201L, a central portion 201C, and a right end portion 201R depending on the orientation of the display. That is, the center display 201 has a configuration in which the left end portion 201L, the central portion 201C, and the right end portion 201R, which are oriented in different directions, are continuous in the left-right direction and integrated. The left end portion 201L, the central portion 201C, and the right end portion 201R can each display independently, or can display integrally.

[0120] Central portion 201C extends laterally from near the left end of driver's seat 101 to near the right end of passenger seat 102 in front of driver's seat 101 and passenger seat 102, and faces rearward (rear of vehicle 1A) when viewed from driver's seat 101 or passenger seat 102. Central portion 201C faces diagonally upward as shown in Fig. 6. This makes the angle of incidence of the line of sight of the driver sitting in driver's seat 101 and the passenger sitting in passenger seat 102 when looking at central portion 201C closer to vertical, improving visibility.

[0121] The left end 201L and the right end 201R are provided substantially symmetrically on both the left and right ends of the center display 201. The left end 201L is bent inward (toward the inside of the vehicle) at the left end of the center display 201, is angled toward the interior of the vehicle with respect to the central portion 201C, and faces diagonally rear right (diagonally rear right of the vehicle 1A) as viewed from the driver's seat 101 or the passenger seat 102. The right end 201R is bent inward (toward the inside of the vehicle) at the right end of the center display 201, is angled toward the interior of the vehicle with respect to the central portion 201C, and faces diagonally rear left (diagonally rear left of the vehicle 1A) as viewed from the driver's seat 101 or the passenger seat 102.

[0122] The angle of the left end 201L with respect to the center 201C is adjusted, for example, so that the reflection angle with respect to the incident angle of the line of sight of a standard driver to the left end 201L faces an appropriate direction diagonally behind the vehicle 1A. The angle of the right end 201R with respect to the center 201C is adjusted, for example, so that the reflection angle with respect to the incident angle of the line of sight of a standard driver to the right end 201R faces an appropriate direction diagonally behind the vehicle 1A.

[0123] The console display 202 is provided in the console 107 provided between the driver's seat 101 and the passenger seat 102 , and is arranged below the central portion 201 C of the center display 201 .

[0124] The console 107 extends in the front-rear direction between the driver's seat 101 and the passenger seat 102. As shown in Fig. 6, the console 107 is roughly divided, from the front, into a front end portion 107A, a middle portion 107B, and a rear end portion 107C.

[0125] A tip portion 107A of the console 107 is inclined obliquely upward toward the front of the vehicle 1A. A console display 202 is provided at the tip of the tip portion 107A. In addition, at the tip portion 107A, behind (below) the console display 202, an operation unit 109 including, for example, a plurality of physical switches, keys, and the like (not shown) is provided.

[0126] The console display 202 is, for example, configured with a two-dimensional or three-dimensional touch panel, and can be operated by touching or approaching with a finger or the like. The console display 202 faces the rear of the vehicle 1A. The console display 202 faces diagonally upward at approximately the same angle as the central portion 201C of the center display 201. This creates a sense of unity between the center display 201 and the console display 202, as if they were continuously connected. Also, like the central portion 201C of the center display 201, the visibility of the console display 202 is improved.

[0127] Contrary to the tip portion 107A, the middle portion 107B of the console 107 is inclined obliquely downward toward the front of the vehicle 1A. A circular controller 108 is provided near the center of the middle portion 107B. The inclination angle of the middle portion 107B is set to an angle at which the driver can easily place his / her hand, thereby improving the operability of the controller 108.

[0128] A rear end portion 107C of the console 107 is substantially horizontal so that the driver can easily place his / her arms thereon, for example.

[0129] The head-up display includes a display 203 provided in front of the driver's seat 101, as shown in Fig. 4. For example, the display 203 may be configured as a part of the windshield 104, or may be provided separately from the windshield 104. In the latter case, for example, the display 203 is attached to the windshield 104. Then, visual information is projected onto the display 203 using AR technology, and the visual information is superimposed and displayed within the field of vision of the driver.

[0130] The position of the display 203 is set to, for example, a position higher than that of a conventional in-vehicle head-up display. This makes it possible to superimpose visual information on vehicles and the like that are farther ahead of the vehicle 1A (for example, 100 m ahead).

[0131] The digital rearview mirror 204 is used in place of a conventional rearview mirror and is also called a smart rearview mirror. The digital rearview mirror 204 is provided at the top end of the windshield 104, slightly in front of the center, and above the center portion 201C of the center display 201, just like a conventional rearview mirror.

[0132] The steering wheel display 205 is provided in the center of the steering wheel 106.

[0133] As shown in FIG. 7 or FIG. 9, the rear entertainment display 206 is provided on the back of the passenger seat 102, more specifically, on the back of the headrest of the passenger seat 102.

[0134] The rear entertainment display 206 may be provided, for example, on the back of both the driver's seat 101 and the passenger seat 102, or may be provided only on the back of the driver's seat 101.

[0135] The displays of the center display 201, console display 202, head-up display, digital rear mirror 204, steering wheel display 205, and rear entertainment display 206 are controlled, for example, by a display control unit which is part of the functions of the HMI 31.

[0136] <Examples of what is displayed on each display inside the vehicle> Next, examples of the display contents of each display inside the vehicle will be described with reference to Figs.

[0137] <Example of display contents on center display 201> First, an example of the display content of the center display 201 will be described.

[0138] A central portion 201C of the center display 201 displays, for example, driving support information and images of the surroundings of the vehicle 1A.

[0139] For example, the central portion 201C displays information related to the running state of the vehicle 1A, etc. For example, as shown in Fig. 10, turn signal information 301, speed information 302, and battery information 303 are displayed side by side.

[0140] The direction indicator information 301 indicates the operating state of the direction indicators of the vehicle 1A. The left and right arrows in the direction indicator information 301 flash or light up in conjunction with the left and right direction indicators, respectively.

[0141] The speed information 302 indicates the speed and gear shift position of the vehicle 1A. The speed of the vehicle 1A is indicated by a bar extending in the left-right direction and a number. For example, the speed information 302 may further indicate the speed limit.

[0142] The battery information 303 indicates the remaining charge of the battery of the vehicle 1A. The remaining charge of the battery is indicated by a bar extending in the left-right direction and a number.

[0143] For example, when there is a road extending in the left and right direction in front of the vehicle 1A at a T-junction or a crossroads, the center section 201C displays an image showing the condition of the road ahead. For example, an image obtained by combining images from the cameras 51FL, 51FC, and 51FR, that is, an image in a direction of about 180 degrees centered on the front of the vehicle 1A as seen from near the tip of the vehicle 1A, is displayed. This allows the driver to easily recognize vehicles approaching from the left or right on the road ahead.

[0144] A T-junction, a crossroad, or the like ahead of the vehicle 1A is detected by the recognition unit 73 based on, for example, an image from the camera 51M and sensor data from the LiDAR 53F.

[0145] For example, the center portion 201C displays information to assist parking when parking the vehicle 1A. For example, as shown in Fig. 11, images from the cameras 51BL, 51BC, and 51BR, i.e., an image of the rear of the vehicle 1A, are displayed. Also, a parking position 321 is displayed in a flashing manner within the image of the rear of the vehicle 1A. The parking position 321 indicates, for example, a position set for parking when automatic parking is performed, and indicates a position where parking is possible or a recommended parking position when manual parking is performed.

[0146] At this time, for example, a 360-degree image (hereinafter referred to as a wide-angle image) around the vehicle 1A captured by each camera 51 may be rotated and displayed in the central portion 201C. The range of the wide-angle image displayed in the central portion 201C may be rotated according to the driver's posture, for example. Specifically, for example, the range of the wide-angle image displayed in the central portion 201C may be shifted based on the driver's position and head direction relative to a predetermined reference position, thereby rotating the wide-angle image displayed in the central portion 201C.

[0147] This allows the user to confirm the parking position, easily find a suitable parking position, and safely park the vehicle 1A.

[0148] The parking position 321 is detected by the recognition unit 73 based on, for example, images from the cameras 51BL, 51BC, and 51BR, and sensor data from the LiDAR 53B.

[0149] For example, when the vehicle 1A is backing up, the central portion 201C displays images from the cameras 51BL, 51BC, and 51BR, that is, an image of the rear of the vehicle 1A.

[0150] For example, as shown in FIG. 12, an image 341 of the rear of the vehicle 1A is displayed. Furthermore, for example, if an obstacle is detected behind the vehicle 1A, an alert display 342 flashes in the image 341. Furthermore, an image 343 indicating the position of the obstacle is displayed. The vehicle in the image 343 indicates the vehicle 1A. Then, around the vehicle in the image 343, an alert display 344 indicating the position of the obstacle flashes. In this example, it is shown that an obstacle is present diagonally to the right rear of the vehicle 1A.

[0151] This allows the driver to safely back up the vehicle 1A.

[0152] Note that an obstacle behind the vehicle 1A is detected by the recognition unit 73 based on, for example, images from the cameras 51BL, 51BC, and 51BR, and sensor data from the radars 52BL, 52BC, 52BR, and LiDAR 53B.

[0153] For example, when the siren of an emergency vehicle (e.g., an ambulance, a fire engine, a police car, etc.) is detected, the central unit 201C displays information indicating the direction of the siren's sound source. This allows the driver to accurately recognize the direction of the emergency vehicle and take appropriate measures such as waiting in the direction of the emergency vehicle.

[0154] The direction of the sound source of the siren is detected by the recognition unit 73 based on the sound detected by a microphone provided in the external recognition sensor 25, for example.

[0155] For example, when a person is detected in the shade around the vehicle 1A, the center unit 201C displays an alert. For example, information indicating the direction in which the person is located is displayed as the alert display. This allows the driver to avoid colliding with or coming into contact with the person in the shade.

[0156] It should be noted that a person in the shadow is detected by the recognition unit 73 based on the sensor data from each radar 52 and each LiDAR 53, for example.

[0157] For example, when the surroundings of the vehicle 1A are dark and an object that is difficult for the human eye to see (such as an animal in the dark) is detected, the center unit 201C displays an alert. For example, when an image of the front of the vehicle 1A captured by the camera 51M is displayed, a frame or the like indicating the position of the detected object is superimposed as an alert display. This allows the driver to avoid colliding with or coming into contact with the detected object.

[0158] Note that an object that is difficult to see is detected by the recognition unit 73 based on, for example, a difference in brightness in the image from the camera 51M.

[0159] For example, when the LKA (Lane Keep Assist) function is operating while the vehicle 1A is backing up, the central section 201C displays information indicating the predicted traveling direction. The traveling direction is indicated by, for example, a line or an arrow. This allows the driver to know the direction to back up in advance, and enables the driver to take appropriate action in the event of an emergency or danger, for example.

[0160] For example, center portion 201C displays images from cameras 151FL and 151FR installed in rear seat 103. This allows the driver to monitor the state of a child sitting in rear seat 103, for example.

[0161] For example, when the central portion 201C displays an image of the surroundings of the vehicle 1A, the display range of the image may be changed based on at least one of the driver's line of sight and posture.

[0162] The line of sight direction and posture of the driver are recognized by the DMS 30, for example, based on an image from the camera 151FL. In addition, the position and direction of the head, as well as the sitting position, etc., are recognized as the posture of the driver.

[0163] Furthermore, depending on the situation, central portion 201C can be divided into three areas: a driver's seat area in front of driver's seat 101, a central area between driver's seat 101 and passenger seat 102, and a passenger seat area in front of passenger seat 102. The driver's seat area, central area, and passenger seat area can each be displayed independently.

[0164] In the driver's seat area, information mainly for the driver is displayed, such as the above-mentioned driving support information.

[0165] The central area displays infotainment (in-vehicle infotainment) related information, such as audio, video, websites, maps, etc.

[0166] The passenger seat area displays, for example, infotainment-related information for the passenger in the passenger seat.

[0167] The left end 201L and the right end 201R of the center display 201 are mainly used as a digital outer mirror (electronic side mirror) that replaces a conventional side mirror. That is, the left end 201L and the right end 201R are used for the CMS. For example, the left end 201L displays an image of the left diagonally rear of the vehicle 1A captured by the camera 51SL. The right end 201R displays an image of the right diagonally rear of the vehicle 1A captured by the camera 51SR.

[0168] The left end 201L, the center 201C, and the right end 201R of the center display 201 are continuous in the horizontal direction. Therefore, when the driver moves his / her gaze from the center 201C to the left end 201L or the right end 201R, or from the left end 201L or the right end 201R to the center 201C, he / she only needs to move his / her gaze horizontally. As a result, the distance and direction of movement of the driver's gaze are reduced, improving visibility.

[0169] Also, for example, the left end portion 201L displays an alert when a vehicle, motorcycle, bicycle, pedestrian, or the like approaching from the left side or rear of the vehicle 1A is detected.

[0170] For example, when a vehicle is approaching from the left rear of vehicle 1A, a frame 361 is displayed to alert the driver to the approaching vehicle, as shown in Fig. 13. Also, an icon 362 indicating the position of the approaching vehicle is displayed in a blinking manner. The rectangle in the upper right corner of icon 362 indicates vehicle 1A, and the half rectangle in the lower left corner indicates the approaching vehicle.

[0171] Similarly, the right end portion 201R displays an alert when a vehicle, motorbike, bicycle, pedestrian, or the like approaching from the right side or rear of the vehicle 1A is detected.

[0172] In addition, vehicles approaching from the left and right sides and rear of vehicle 1A are detected by the recognition unit 73 based on, for example, images from camera 51SL, camera 51SR, camera 51BL, camera 51BC, and camera 51BR, as well as sensor data from radar 52BL, radar 52BC, radar 52BR, and LiDAR 53B.

[0173] Based on at least one of the driver's line of sight and posture, for example, the display control unit adjusts the shooting directions of the cameras 51SL and 51SR. This allows the range of images displayed on the left end 201L and the right end 201R to be appropriately adjusted according to the driver's line of sight and posture, similar to a conventional physical side mirror. As a result, the driver can check the situation in the desired range diagonally behind the vehicle 1A without feeling any discomfort.

[0174] Also, for example, the display range of the image of the left end 201L and the right end 201R is changed based on the operation of the turn indicator. For example, when the turn indicator is not operating, the normal angle of view is set, and when the turn indicator is operating, the angle of view is widened. This allows the driver to accurately recognize the situation around the vehicle 1A and safely change the traveling direction of the vehicle 1A.

[0175] In addition, since the left end portion 201L, the center portion 201C, and the right end portion 201R are connected together, it is also possible to display one screen on the entire center display 201. For example, it is also possible to display an image of the surroundings of the vehicle 1A, map information, infotainment-related information, or the like on the entire center display 201.

[0176] <Example of the display contents of the console display 202> Next, an example of the display content of the console display 202 will be described.

[0177] For example, the console display 202 displays an operation screen for operating an air conditioning system in the vehicle. A passenger such as a driver operates the air conditioning system in the vehicle using the displayed operation screen.

[0178] For example, at least a part of the operation of the air conditioning equipment in the vehicle can also be performed using the operation unit 109.

[0179] For example, the console display 202 displays an operation screen for operating information displayed on the center display 201. For example, a passenger such as a driver uses the displayed operation screen to scroll, enlarge, reduce, switch, etc., information (e.g., a map, etc.) displayed in the central portion 201C of the center display 201.

[0180] As described above, there is a sense of unity between the center display 201 and the console display 202. Therefore, the passenger can operate the information displayed on the center display 201 using the operation screen of the console display 202 in a natural manner, improving operability.

[0181] The controller 108 can be pressed, rotated, or tilted in a predetermined direction (e.g., forward, backward, left, right) and is used, for example, to operate an audio system in a car. For example, pressing the controller 108 plays or stops music, and rotating the controller 108 adjusts the volume.

[0182] <Example of head-up display content> The display 203 of the head-up display displays, for example, information to assist driving.

[0183] For example, the display 203 displays an alert in accordance with the situation around the vehicle 1A.

[0184] 14, when there is a sharp curve ahead of vehicle 1A, alert display 381 is displayed. Alert display 381 indicates the direction of the curve with an arrow, and indicates the curvature of the curve with the shape of the arrow (e.g., length, thickness, etc.), color, etc. This allows the driver to recognize the presence of a sharp curve in advance, and to drive around the sharp curve safely.

[0185] For example, as shown in Fig. 15, when a large vehicle such as a truck is present in a lane adjacent to the lane in which vehicle 1A is traveling, alert display 401 is displayed flashing. Alert display 401 is an alert display that notifies the driver that a large vehicle such as a truck is approaching in the lane to the right of vehicle 1A. This allows the driver to recognize the approach of the large vehicle in advance and to avoid collision or contact with the large vehicle.

[0186] For example, when the emergency brake of the vehicle 1A is applied, the display 203 displays an alert notifying the driver that the emergency brake has been applied. This notifies the driver of the cause of the emergency stop, and gives the driver a sense of security.

[0187] For example, the display 203 displays information indicating a vehicle ahead to be followed during the operation of ACC (Adaptive Cruise Control). For example, a frame or the like surrounding the vehicle to be followed is displayed. This allows the driver to check the operation of ACC and predict the route of the vehicle 1A, and to appropriately respond to, for example, an emergency or danger.

[0188] For example, while the lane change assist function is operating, the display 203 displays the planned route as necessary using arrows, etc. This allows the driver to recognize lane changes in advance and to take appropriate action in the event of an emergency or danger, for example.

[0189] <Example of display content of digital rearview mirror 204> The digital rear-view mirror 204 displays, for example, an image of the rear of the vehicle 1A captured by the camera 51BC.

[0190] For example, the digital rear-view mirror 204 displays an alert when another vehicle is approaching behind the vehicle 1A. For example, as shown in Fig. 16, when the distance between the vehicle 1A and a vehicle 421 behind the vehicle 1A falls below a predetermined threshold, an alert display 422 flashes to alert the driver.

[0191] The inter-vehicle distance to the rear vehicle is detected by the recognition unit 73 based on, for example, sensor data from the radar 52BC and the LiDAR 53B.

[0192] Also, for example, the shooting direction of the camera 51BC is adjusted based on at least one of the line of sight and the posture of the driver. This allows the range of the image displayed on the digital rear mirror 204 to be appropriately adjusted according to the line of sight and the posture of the driver, similar to a conventional physical rearview mirror. As a result, the driver can check the situation of the desired range behind the vehicle 1A without feeling uncomfortable.

[0193] <Example of display contents of steering wheel display 205> The steering wheel display 205 displays, for example, a message for the driver, thereby enabling communication between the driver and the vehicle 1A.

[0194] For example, when a driver is recognized by the DMS 30, the steering wheel display 205 displays a message 441 including the name of the recognized driver, as shown in FIG.

[0195] For example, when the driver inputs a voice command, the steering wheel display 205 displays a message 461 indicating that the voice command has been accepted, as shown in FIG.

[0196] Also, for example, the steering wheel display 205 displays messages indicating confirmation and completion of payment of fees for expressways, toll roads, parking lots, etc.

[0197] <Example of display contents of rear entertainment display 206> For example, the rear entertainment display 206 displays infotainment-related information for passengers in the rear seats 103 .

[0198] For example, the rear entertainment display 206 displays an alert in the event of an emergency, danger, etc. For example, as shown in Fig. 19, an alert display 481 for calling the attention of passengers in the rear seat 103 is displayed in a flashing manner during an emergency stop.

[0199] As described above, by displaying various kinds of information on each display inside the vehicle, it is possible to improve the convenience of the passengers, such as the driver. For example, it is possible to assist the driving of the vehicle 1A, provide necessary information to the passengers, and entertain the passengers.

[0200] <<3. Modification of the First Embodiment>> A modification of the first embodiment of the present technology described above will be described below.

[0201] The present technology can also be applied to, for example, transportation devices other than vehicles that people ride on.

[0202] The present technology can also be applied to a moving device that performs automatic driving without the need for a passenger to drive. In this case, the above-mentioned distinction between the driver's seat 101 and the passenger seat 102 disappears, and the above-mentioned various displays are arranged in front of the seats in the moving device. Also, for example, the display range of an image displayed on the display is changed based on the line of sight, posture, etc. of a person sitting in a specific seat.

[0203] <<4. Second embodiment>> Next, a second embodiment of the present technology will be described with reference to FIGS.

[0204] In each drawing of the second embodiment, parts corresponding to those in the first embodiment are given the same reference numerals, and the description thereof will be omitted as appropriate.

[0205] <Exterior configuration example of vehicle 1B> First, a configuration example of the exterior of a vehicle 1B according to a second embodiment of the vehicle 1 to which the present technology is applied will be described with reference to Fig. 20 to Fig. 26. The following description will mainly focus on the configuration of an exterior lighting system of the vehicle 1B.

[0206] Fig. 20 is a front view of vehicle 1B and an enlarged view of the vicinity of the logo of vehicle 1B at the center of the front. Fig. 21 is a left side view of vehicle 1B. Fig. 22 is an enlarged view of the tip of the left side of vehicle 1B. Fig. 23 is a rear view of vehicle 1B. Figs. 24 to 26 are views of the vicinity of the left headlight of vehicle 1B viewed from multiple directions.

[0207] In the following description, the left and right sides of the vehicle 1B when viewed in the traveling direction are respectively the left and right sides of the vehicle 1B. For example, the left and right sides of FIG. 20 are respectively the right and left sides of the vehicle 1B.

[0208] Many of the lights and sensors on the exterior of the vehicle 1B are arranged along a loop line L1, which is an imaginary line that surrounds the periphery of the body of the vehicle 1B in a substantially horizontal direction. Here, "arranged along the loop line L1" includes not only the case where they are arranged on the loop line L1, but also the case where they are arranged in the vicinity of the loop line L1.

[0209] For example, as shown in FIG. 20, accessory light 601L, accessory light 601R, day running light 602L, day running light 602R, headlight 603LU, headlight 603LD, headlight 603RU, and headlight 603RD are disposed on the front of the body.

[0210] The accessory light 601L extends in the vehicle width direction (left-right direction) from the center of the front face of the body to near the right ends of the headlights 603LU and 603LD. The accessory light 601R extends in the vehicle width direction (left-right direction) from the center of the front face of the body to near the left ends of the headlights 603RU and 603RD.

[0211] The accessory light 601L and the accessory light 601R are separated from each other, and a gap is provided between them. Specifically, the right end of the accessory light 601L is bent diagonally downward to the right, and the left end of the accessory light 601R is bent diagonally upward to the left. The bent portion at the right end of the accessory light 601L and the bent portion at the left end of the accessory light 601R face each other substantially in parallel with a predetermined interval therebetween, forming a logo of the vehicle 1B. An optical sensor (not shown), such as a camera, radar, or LiDAR, is disposed in an area A1 in the gap between the accessory light 601L and the accessory light 601R near the center of the logo.

[0212] The day running light 602L extends horizontally from the left end of the accessory light 601L to near the left ends of the headlights 603LU and 603LD. As shown in Fig. 24 to Fig. 26, the left end of the day running light 602L curves toward the rear of the vehicle 1B and extends in a direction penetrating the body.

[0213] The day running light 602R extends horizontally from the right end of the accessory light 601R to near the right ends of the headlights 603RU and 603RD. Although not shown, the right end of the day running light 602R curves toward the rear of the vehicle 1B and extends in a direction penetrating the body, similar to the day running light 602L.

[0214] The accessory light 601L, the accessory light 601R, the day running light 602L, and the day running light 602R form a front light along a loop line L1 on the front side of the body. The front light forms a front line that is a part of the loop line L1, extends in the vehicle width direction (left-right direction) on the front side of the body, curves backward at both ends, and extends in a direction penetrating the body.

[0215] The accessory light 601L, the accessory light 601R, the day running light 602L, and the day running light 602R each include a plurality of LEDs arranged in a horizontal direction. Each LED can be individually controlled to be turned on / off, have a color, a brightness, and the like.

[0216] In the following, when there is no need to distinguish between the accessory light 601L and the accessory light 601R, they will simply be referred to as the accessory light 601. In the following, when there is no need to distinguish between the day running light 602L and the day running light 602R, they will simply be referred to as the day running light 602.

[0217] The headlight 603LU is adjacent to the upper side of the day running light 602L, extends horizontally, and curves backward at the left end. The headlight 603LD is adjacent to the lower side of the day running light 602L, extends horizontally, and curves backward at the left end. In this way, the headlight 603LU and the headlight 603LD are separated above and below by the day running light 602L (front line).

[0218] The headlight 603RU is adjacent to the upper side of the day running light 602R, extends horizontally, and curves backward at the right end. The headlight 603RD is adjacent to the lower side of the day running light 602R, extends horizontally, and curves backward at the right end. In this way, the headlight 603RU and headlight 603RD are separated above and below by the day running light 602R (front line).

[0219] The headlights 603LU and 603RU each have multiple LEDs arranged horizontally and vertically, and output low beams. The headlights 603LD and 603RD each have multiple LEDs arranged horizontally and vertically, and output high beams. Each LED can be individually controlled to turn on / off, and to change color, brightness, etc.

[0220] In the following, when there is no need to distinguish between the headlights 603LU and 603LD, they will simply be referred to as headlights 603L. In the following, when there is no need to distinguish between the headlights 603RU and 603RD, they will simply be referred to as headlights 603R. In the following, when there is no need to distinguish between the headlights 603L and 603R, they will simply be referred to as headlights 603.

[0221] In this way, by dividing the headlights 603L and 603R into upper and lower parts by the front line, the degree of freedom in designing the headlights 603L and 603R is improved. For example, the headlights 603L and 603R can be designed in a style other than slanted eyes or droopy eyes. In addition, since the low beams (headlights 603LU and 603RU) and high beams (headlights 603LD and 603RD) are arranged in appropriate positions, the functionality of the headlights 603 and the safety of the vehicle 1B are not reduced.

[0222] Also, for example, as shown in FIG. 21, a turn signal 604L, an auxiliary light 605FL, and an auxiliary light 605BL are disposed on the left side surface of the body.

[0223] The turn signal 604L extends in the front-rear direction on an extension of the A-pillar 614L, immediately above the loop line L1.

[0224] The auxiliary light 605FL is disposed behind a door knob 612FL of the left front door 611FL and illuminates the area around the door knob 612FL. Since the door knob 612FL is disposed immediately above the loop line L1, the auxiliary light 605FL is also disposed immediately above the loop line L1.

[0225] Furthermore, for example, a short-range wireless communication device (not shown) such as NFC is disposed on the doorknob 612FL or in the vicinity of the doorknob 612FL.

[0226] The auxiliary light 605BL is disposed behind a door knob 612BL of the rear left door 611BL, and illuminates the area around the door knob 612BL. Since the door knob 612BL is disposed immediately above the loop line L1, the auxiliary light 605BL is also disposed immediately above the loop line L1.

[0227] Furthermore, for example, a short-range wireless communication device (not shown) such as NFC is disposed on the doorknob 612BL or in the vicinity of the doorknob 612BL.

[0228] In this manner, on the left side of the vehicle 1B, the turn signal 604L, the auxiliary light 605FL, and the auxiliary light 605BL are aligned in the front-rear direction along the loop line L1.

[0229] The turn signal 604L, the auxiliary light 605FL, and the auxiliary light 605BL each include a plurality of LEDs arranged in a horizontal direction. Each LED can be individually controlled to be turned on / off, have a color, or have a brightness, etc.

[0230] 21, an optical sensor such as a camera, a radar, or a LiDAR is provided in an area A2L near the intersection of the loop line L1 and an extension of the A-pillar 614L. For example, a LiDAR 53L is provided below the turn signal 604L and on the loop line L1.

[0231] Furthermore, an optical sensor (not shown), such as a camera, a radar, or a LiDAR, is provided in an area A3L near the intersection of the loop line L1 and an extension of the C-pillar 615L.

[0232] In this way, by locating the optical sensor near the loop line L1, even if the color of the surface of the optical sensor is different from the color of the body, the optical sensor is recognized as being part of the loop line L1, so that the optical sensor blends naturally into the exterior appearance of the body without creating an unnatural feeling.

[0233] Although not shown, a turn signal 604R, an auxiliary light 605FR, an auxiliary light 605BR, a door handle 612FR, a door handle 612BR, a near field communication device, and an optical sensor are arranged on the right side of the vehicle 1B in the same positions as on the left side.

[0234] Further, for example, as shown in FIG. 23, tail light 606CL, tail light 606CR, tail light 606L, tail light 606R, brake light 607LU, brake light 607LD, brake light 607RU, and brake light 607RD are arranged on the rear surface of the body.

[0235] The tail light 606CL extends in the vehicle width direction (left-right direction) from the center of the rear face of the body to near the right ends of the brake lights 607LU and 607LD. The tail light 606CR extends horizontally from the center of the rear face of the body to near the right ends of the brake lights 607RU and 607RD.

[0236] The tail light 606CL and the tail light 606CR are separated from each other with a gap therebetween. Specifically, the right end of the tail light 606CL is bent obliquely upward to the right, and the left end of the tail light 606CR is bent obliquely downward to the left. The bent portion at the right end of the tail light 606CL and the bent portion at the left end of the tail light 606CR face each other substantially in parallel with a predetermined gap therebetween, forming the logo of the vehicle 1B. An optical sensor (not shown), such as a camera, radar, or LiDAR, is disposed in an area A4 in the gap between the tail light 606CL and the tail light 606CR near the center of the logo.

[0237] The tail light 606L extends horizontally from the left end of the tail light 606CL to near the left ends of the brake light 607LU and the brake light 607LD. The left end of the tail light 606L curves forward. The tail light 606R extends horizontally from the right end of the tail light 606CR to near the right ends of the brake light 607RU and the brake light 607RD. The right end of the tail light 606R curves forward.

[0238] The tail light 606CL, the tail light 606CR, the tail light 606L, and the tail light 606R form a tail line that extends in the left-right direction on the rear surface of the body and has both ends curved forward. The tail line forms a part of the loop line L1.

[0239] Each of the tail lights 606CL, 606CR, 606L, and 606R includes a plurality of LEDs arranged in a horizontal direction. Each LED can be individually controlled to be turned on / off, have a color, or have a brightness, etc.

[0240] In the following, when there is no need to distinguish between the tail light 606CL and the tail light 606CR, they will simply be referred to as the tail light 606C. In the following, when there is no need to distinguish between the tail light 606C, the tail light 606L, and the tail light 606R, they will simply be referred to as the tail light 606.

[0241] The brake light 607LU is adjacent to the upper side of the tail light 606L and curves forward at the left end. The brake light 607LD is adjacent to the lower side of the tail light 606L and curves forward at the left end. In this way, the brake light 607LU and the brake light 607LD are separated from each other by the tail light 606L.

[0242] The brake light 607RU is adjacent to the upper side of the tail light 606R and curves forward at the right end. The brake light 607RD is adjacent to the lower side of the tail light 606R and curves forward at the right end. In this way, the brake light 607RU and the brake light 607RD are separated above and below by the tail light 606R (tail line).

[0243] The brake light 607LU, the brake light 607LD, the brake light 607RU, and the brake light 607RD each include a plurality of LEDs arranged in a horizontal direction. Each LED can be individually controlled to be turned on / off, have a color, a brightness, and the like.

[0244] In the following, when there is no need to distinguish between the brake lights 607LU and 607LD, they will simply be referred to as the brake lights 607L. In the following, when there is no need to distinguish between the brake lights 607RU and 607RD, they will simply be referred to as the brake lights 607R. In the following, when there is no need to distinguish between the brake lights 607L and 607R, they will simply be referred to as the brake lights 607.

[0245] In addition, the body color differs along the lower end of windshield 104, the lower end of window 613FL of door 611FL, the lower end of window 613BL of door 611BL, the lower end of window 613FR (not shown) of door 611FR, and the lower end of window 613BR (not shown) of door 611BR, along the loop line L2, which is an imaginary line that surrounds the body of vehicle 1B in an approximately horizontal direction.

[0246] For example, a black line is formed by chrome plating along the loop line L2. The area above the loop line L2 is colored black. For example, the area above the loop line L2 is painted black. The windshield 104, the window 613FL, the window 613BL, the window 613FR, the window 613BR, and the rear window 616 are tinted black.

[0247] On the other hand, the body below the loop line L2 is painted in a color different from that above the loop line L2. The color of the body below the loop line L2 is not particularly limited.

[0248] Further, cameras 51SL and 51SR are provided near the front and bottom ends of the windows 613FL and 613FR (not shown) along the loop line L2. The cameras 51SL and 51SR capture images of the left rear and right rear of the vehicle 1B, respectively.

[0249] In this manner, the design of the vehicle 1B can be improved while avoiding any deterioration in safety and functionality.

[0250] For example, in the exterior of the vehicle 1B, two substantially parallel loop lines L1 and L2 are virtually recognized, which makes the vehicle body appear lower and gives a sporty impression.

[0251] As described above, the degree of freedom in designing the headlights is improved. Furthermore, since each light is disposed at an appropriate position, the functionality of each light and the safety of the vehicle 1B are not reduced.

[0252] In addition, since the front line (front light) curves backward at both ends and extends in a direction penetrating the body, the impression is given that the front line penetrates the body and is connected to the LiDAR53L on the left side and the LiDAR53R on the right side.

[0253] Furthermore, by arranging sensors around vehicle 1B along loop line L1, the impression is given that the area around vehicle 1B is being watched (surveilled), which is effective in preventing theft and vandalism of vehicle 1B.

[0254] <Example of vehicle 1B interior configuration> Next, an example of the interior configuration of the vehicle 1B will be described with reference to Figs.

[0255] First, the devices arranged along the loop line L11 will be described with reference to FIGS.

[0256] Fig. 27 is a schematic diagram of the interior of vehicle 1B seen from the right. Fig. 28 is a schematic diagram of the vicinity of driver's seat 101 and passenger seat 102 of vehicle 1B. Fig. 29 is a schematic diagram of the vicinity of the dashboard of vehicle 1B. Fig. 30 is an enlarged view of steering wheel 651 of vehicle 1B.

[0257] Inside the vehicle 1B, multiple user interface devices are concentrated along a loop line L11 (Figure 27), which is an imaginary line that surrounds the perimeter of the interior in an approximately horizontal direction, and various interfaces are aggregated.

[0258] Here, the device for the user interface includes, for example, an output device that outputs visual information, auditory information, and tactile information, and an operation device used for various operations. In addition, being arranged along the loop line L11 includes not only being arranged on the loop line L11, but also being arranged in the vicinity of the loop line L11.

[0259] The loop line L11 is disposed at the same height as the loop line L1 outside the vehicle 1B. The loop line L11 is also tilted slightly upward from the front to the rear. This is because the positions of the rear seats 103L and 103R are higher than the positions of the driver's seat 101 and the passenger seat 102.

[0260] For example, the display devices constituting the HMI 31 in FIG. 1 are arranged along the loop line L11.

[0261] For example, as shown in Figures 27 to 29, a center display 201 is disposed on the front of the dashboard in front of the driver's seat 101 and the passenger seat 102, extending in the vehicle width direction (left and right direction) immediately above the loop line L11.

[0262] As described above, the center display 201 is largely divided into a left end, a center portion, and a right end portion depending on the orientation of the display. The left end, center portion, and right end portion of the center display 201 can display information independently of each other, or can display information together. The left end and right end portions of the center display 201 are mainly used as digital outer mirrors (electronic side mirrors) that replace conventional side mirrors. For example, the left end portion displays an image of the left rear of the vehicle 1B captured by the camera 51SL (FIG. 20). The right end portion displays an image of the right rear of the vehicle 1B captured by the camera 51SR (FIG. 20).

[0263] As shown in FIGS. 28 to 30, in front of the driver's seat 101, a steering wheel 651 is disposed on the loop line L11.

[0264] 29 and 30, illumination 652 is provided along the periphery of the central part of steering wheel 651. Illumination 652 includes a plurality of LEDs arranged circumferentially along the periphery of the central part of steering wheel 651. Each LED can be individually controlled to be turned on / off, color, brightness, etc. Therefore, the color, brightness, and light-emitting area (light-emitting range) of illumination 652 are variable.

[0265] An airbag is stored in the center of steering wheel 651. When the airbag deploys, the central portion of steering wheel 651 shown by the dotted line in Fig. 30 will crack. Illumination 652 is positioned to avoid this central crackable portion. This prevents fragments of illumination 652 and harmful substances from scattering when the airbag deploys.

[0266] Moreover, the speakers constituting the HMI 31 in FIG. 1 are arranged along the loop line L11.

[0267] Specifically, as shown in Fig. 27 and Fig. 28, a speaker 653FL is embedded near the loop line L11 on the inside of the door 611FL on the driver's seat 101 side. As shown in Fig. 28, a speaker 653FR is embedded near the loop line L11 on the inside of the door 611FR on the passenger's seat 102 side. As shown in Fig. 27, a speaker 653BL is embedded near the loop line L11 on the inside of the door 611BL on the left rear seat 103L side. Although not shown, a speaker 653BR is embedded near the loop line L11 on the inside of the door 611BR on the right rear seat 103R side.

[0268] Although not shown, speakers (hereinafter referred to as seat speakers) are embedded individually below the headrests of the driver's seat 101, passenger seat 102, rear seat 103L, and rear seat 103R. Furthermore, the shape of the seats and the positions of the seat speakers are adjusted so that people of various heights (sitting heights) can clearly hear the sound from the seat speakers of each seat.

[0269] Speakers 653FL to 653BR arranged along loop line L11 are used, for example, to output sounds directed to the entire interior of the vehicle (all passengers in the vehicle).

[0270] Moreover, the speakers 653FL to 653FR realize 360-degree real audio. By realizing 360-degree real audio, for example, it is possible to enjoy videos, music, and the like with realistic sound inside the vehicle. In addition, the position of a dangerous object such as an obstacle existing around the vehicle 1B can be notified by the output direction of the sound.

[0271] On the other hand, the seat speakers of each seat are used, for example, mainly to output private sounds for the individual passengers sitting in each seat. In other words, the sounds output from each seat speaker are individually controlled.

[0272] This arrangement of speakers is merely an example and may be changed. For example, the number of speakers arranged on the loop line L11 may be increased. For example, speakers may be arranged on the dashboard in the front of the vehicle 1B.

[0273] 27, loop lights 654 are arranged slightly above the loop line L11 and at approximately the same height as the external loop line L1, approximately parallel to the loop line L11, so as to surround the interior of the vehicle. The loop lights 654 are downlights equipped with multiple LEDs embedded in the vehicle so as to be aligned approximately horizontally, and are mainly used as auxiliary lighting or interior decoration. Each LED can be individually controlled to be turned on / off, and its color, brightness, etc.

[0274] It should be noted that the loop light 654 does not necessarily have to surround the entire interior of the vehicle, but may be non-continuous and surround only a portion of the interior of the vehicle.

[0275] Furthermore, various operating devices are arranged along the loop line L11.

[0276] For example, as described above, in front of the driver's seat 101, the steering wheel 651 is disposed on the loop line L11.

[0277] As shown in FIG. 30, a stalk lever 655, which is a rod-shaped operating body, is provided at the rear of the steering wheel 651 so as to extend from a steering column (not shown) in the vehicle width direction (lateral direction (rightward)). The stalk lever 655 can be moved in the vertical direction, and by moving it in the vertical direction, the shift position of the vehicle 1B is changed. In other words, the stalk lever 655 constitutes a tilt-type shift lever that is movable in the vertical direction. Note that the stalk lever 655 may be, for example, either a straight type that moves straight up and down, or a column type that moves in a zigzag manner up and down.

[0278] The order in which the shift positions are set by the stalk lever 655 is, from top to bottom, R (reverse), N (neutral), D (drive), and A (automatic driving). That is, when the stalk lever 655 is moved downward from top to bottom, the shift positions are changed in the order of R, N, D, and A. When the stalk lever 655 is moved upward from bottom to top, the shift positions are changed in the order of A, D, N, and R.

[0279] Further, a button 656 that can be pressed in the axial direction of the stalk lever 655 is provided at the tip of the stalk lever 655. When the button 656 is pressed, the shift position of the vehicle 1B is switched to P (parking).

[0280] Furthermore, an indicator 657 is provided circumferentially on the side surface of the stalk lever 655. The indicator 657 is disposed at a position visible through gaps between the spokes of the steering wheel 651 when viewed from the driver's seat 101.

[0281] The color of the indicator 657 changes depending on the shift position that is set. For example, when the shift position is set to parking, the indicator 657 is red. When the shift position is set to drive, the indicator 657 is white. When the shift position is set to automatic driving and automatic driving is possible, the indicator 657 is green. When the shift position is set to automatic driving and automatic driving is active, the indicator 657 is blue.

[0282] 30, the indicator 657 may be disposed outside the outer periphery of the steering wheel 651 as viewed from the driver's seat 101. This makes the indicator 657 visible from outside the steering wheel 651. Also, the indicator 657 may be disposed in both the solid line and dotted line positions.

[0283] In this way, by providing indicator 657 on the operating body (stalk lever 655) used to change the shift position, the driver can intuitively understand the meaning represented by the color of indicator 657 and intuitively and reliably recognize the shift position.

[0284] Also, for example, the setting state of the shift position may be displayed on the center display 201 or the like. In this case, for example, the shift positions are displayed in the order of the setting direction by the stalk lever 655. That is, the shift positions are displayed in the order of R (reverse), N (neutral), D (drive), and A (automatic driving) from the top.

[0285] Furthermore, for example, a dial that rotates around an axis (circumferential direction) may be provided at the axial tip or middle of the stalk lever 655, and the shift position may be changed by rotating the dial.

[0286] 28, an air outlet 658FC, an air outlet 658FL, and an air outlet 658FR of the air conditioner (A / C) are arranged on the loop line L11. Specifically, the air outlet 658FR is arranged immediately below the center display 201 between the driver's seat 101 and the passenger seat 102. The air outlet 658FL is arranged immediately below the center display 201 near the joint of the door 611FL on the driver's seat 101 side. The air outlet 658FR is arranged immediately below the center display 201 near the joint of the door 611FR on the passenger seat 102 side.

[0287] Furthermore, as shown in Fig. 29, the air outlet 658FC is provided with a knob 659CL and a knob 659CR for changing the air direction. As shown in Fig. 29, the air outlet 658FL is provided with a knob 659FL for changing the air direction. Although not shown, the air outlet 658FR is provided with a knob 659FR for changing the air direction at the same position as the knob 659FL of the air outlet 658FL. In this way, the knobs 659CL, 659CR, 659FL, and 659FR are provided on the loop line L11.

[0288] As shown in FIG. 29, an operation unit 660 including switches and the like for making various settings for the headlights 603 is disposed on the loop line L11 at the rear right side of the steering wheel 651.

[0289] Further, a door opener is disposed on the loop line L11. For example, as shown in Fig. 28, a door opener 661FL is disposed on the loop line L11 near the center in the front-rear direction of the door 611FL on the driver's seat 101 side. Similarly, a door opener 661FR is disposed on the loop line L11 near the center in the front-rear direction of the door 611FR on the passenger's seat 102 side.

[0290] In this way, by arranging various user interface devices and the like along the loop line L11, noise that interferes with driving is removed from the driver's line of sight, providing an environment in which it is easy to concentrate on driving. In addition, by concentrating various devices near the loop line L11, it becomes possible to intuitively recognize the positions of various devices and operate them. Furthermore, by concentrating various devices near the loop line L11 and arranging the loop light 654, an impression is given that the inside of the vehicle 1B is being watched. In addition, the vertical movement of the driver's or other passenger's eyes is reduced, suppressing car sickness in the passengers.

[0291] FIG. 31 is a view of the driver's seat 101 seen from the left rear.

[0292] The tablet terminal 662L is provided on the back of the driver's seat 101, more specifically, on the back of the headrest of the driver's seat 101.

[0293] 9, presents infotainment-related information to passengers in the rear seats 103L, and accepts operations on the presented information. In addition, for example, the tablet terminal 662L displays an alert in the event of an emergency or danger.

[0294] Although not shown in the drawings, for example, a tablet terminal 662R similar to tablet terminal 662L is provided on the back of passenger seat 102.

[0295] Also, for example, ToF cameras may be provided near the tablet terminals 662L and 662R, respectively. This makes it possible to recognize, for example, passengers operating the tablet terminals 662L and 662R based on images captured by the ToF cameras.

[0296] Next, with reference to Figs. 32 to 36, examples of installation positions of a ToF camera for capturing an image in the direction of the driver's seat 101 (the driver sitting in the driver's seat 101) will be described.

[0297] As shown in FIGS. 32 and 33, the ToF camera is installed at an installation position P1 around the digital rear mirror 204, for example.

[0298] The digital rearview mirror 204 is used to check the rear of the vehicle 1B in place of a conventional rearview mirror, and is also called a smart rearview mirror or a digital rearview mirror. As shown in Fig. 28 and other drawings, the digital rearview mirror 204 is provided at the upper end and near the center of the windshield 104, slightly toward the rear of the vehicle 1B from the windshield 104, like a conventional rearview mirror, and is disposed above the center of the center display 201. Also, as shown in Figs. 32 and 33, the digital rearview mirror 204 is installed near the ceiling via an attachment part 663, diagonally in front of the right of the driver's seat 101.

[0299] The installation position P1 is located near the rear end of the left side surface of the attachment portion 663 on the ceiling of the vehicle 1B, and is located near the upper left of the digital rear-view mirror 204.

[0300] FIG. 34 shows a schematic example of an image captured by the ToF camera when the ToF camera is installed at installation position P1.

[0301] By installing the ToF camera at the installation position P1, the upper body of the driver 681 sitting in the driver's seat 101 is photographed diagonally from above and to the right. This allows the face of the driver 681 to be photographed diagonally from above and to the right, and for example, the recognition unit 73 (FIG. 1) can recognize the line of sight of the driver 681. In addition, the upper body of the driver 681, including the steering wheel 651, can be photographed from a bird's-eye view, and for example, the recognition unit 73 can recognize the posture of the driver 681. Furthermore, since there is a low probability that an obstacle will appear between the driver 681 and the digital rear mirror 204, the line of sight and posture of the driver 681 can be stably recognized.

[0302] On the other hand, FIG. 35 shows an example of an image captured by the ToF camera when the ToF camera is placed on or near the steering wheel 651.

[0303] In this case, the vicinity of the face of the driver 681 is photographed from the front as if looking up. Therefore, the recognition unit 73 can recognize the line of sight of the driver 681. However, it is expected that an obstacle such as the driver 681's hand is likely to appear between the driver 681 and the steering wheel 651, and a situation where the line of sight cannot be recognized is likely to occur. In addition, since the entire upper body of the driver 681 is not photographed, it becomes difficult for the recognition unit 73 to recognize the posture of the driver 681.

[0304] FIG. 36 shows a modified example of the installation position of the ToF camera.

[0305] For example, a ToF camera may be installed at installation position P2 on the left edge of the digital rear mirror 204.

[0306] For example, a ToF camera may be installed at installation position P3 on the ceiling of vehicle 1B, behind the digital rear-view mirror 204 (forward of vehicle 1B from the digital rear-view mirror 204) and closer to the driver's seat 101 than the digital rear-view mirror 204 (to the left of the digital rear-view mirror 204).

[0307] For example, a ToF camera may be installed at installation position P4 on the ceiling of vehicle 1B, in front of the digital rear-view mirror 204 (rear of vehicle 1B from the digital rear-view mirror 204) and closer to the driver's seat 101 than the digital rear-view mirror 204 (to the left of the digital rear-view mirror 204).

[0308] Regardless of whether the ToF camera is installed at installation position P2 to installation position P4, it is possible to recognize the driver's line of sight and posture based on the images captured by the ToF camera, just as when the ToF camera is installed at installation position P1.

[0309] It should be noted that a conventional optical mirror may be used instead of the digital rear mirror 204. Also, a camera of a type different from the ToF camera may be used.

[0310] Furthermore, for example, a ToF camera that captures an image of a passenger sitting in the passenger seat 102 may be installed in a position that is symmetrical to the camera that captures the image of the driver.

[0311] <Example of configuration of information processing unit 701> FIG. 37 is a block diagram showing an example of the configuration of an information processing unit 701 realized by the processor 21, the HMI 31, etc. in FIG.

[0312] The information processing unit 701 includes an image processing unit 711 , a metadata adding unit 712 , and an output control unit 713 .

[0313] The image processing unit 711 performs various image processing and editing on the video data obtained by the camera 51. The image processing unit 711 also converts sensor data obtained by optical sensors other than the camera, such as the radar 52 and the LiDAR 53, into an image.

[0314] The metadata adding unit 712 adds metadata to the video data obtained by the camera 51 .

[0315] The output control unit 713 controls the output of visual information, auditory information, and tactile information in the HMI 31. For example, the output control unit 713 controls the output of visual information from the center display 201, the tablet terminal 662L, the tablet terminal 662R, and the illumination 652 of the steering wheel 651. For example, the output control unit 713 controls the output of auditory information (sound) from the speakers 653FL to 653BR and the seat speaker.

[0316] <Vehicle 1B Operation> Next, an example of the operation of the vehicle 1B will be described with reference to FIGS.

[0317] In the following description, the doors 611FL, 611FR, 611BL, and 611BR will be simply referred to as door 611 unless there is a need to distinguish between them.

[0318] <Lighting system emission pattern> First, examples of light emission patterns of the lighting system of the vehicle 1B will be described with reference to Figs.

[0319] The light control unit 85 (FIG. 1) performs coordinated control of the lights outside the vehicle and the lights inside the vehicle.

[0320] Specifically, Figures 38 to 47 show examples of light emission patterns of the accessory light 601L, accessory light 601R, day running light 602L, day running light 602R, headlight 603L, headlight 603R, turn signal 604L, turn signal 604R, auxiliary light 605FL, auxiliary light 605FR, auxiliary light 605BL, auxiliary light 605BR, tail light 606L, tail light 606R, tail light 606CL, tail light 606CR, brake light 607L, brake light 607R, loop light 654, and main light 751.

[0321] The main light 751 includes LEDs arranged along the periphery of the ceiling of the vehicle 1B, and is used as the main lighting in the vehicle. Each LED can be individually controlled to be turned on / off, and to have a different color and brightness.

[0322] Furthermore, in Figs. 38 to 47, headlights 603LU and 603LD, headlights 603RU and 603RD, brake lights 607LU and 607LD, and brake lights 607RU and 607RD are not distinguished from one another and are illustrated collectively.

[0323] In the following, unless otherwise specified, the accessory light 601L, the accessory light 601R, the day running light 602L, the day running light 602R, the headlight 603L, the headlight 603R, the turn signal 604L, the turn signal 604R, the auxiliary light 605FL, the auxiliary light 605FR, the auxiliary light 605BL, and the auxiliary light 605BR are assumed to be illuminated in white. In the following, unless otherwise specified, the tail light 606L, the tail light 606R, the tail light 606CL, the tail light 606CR, the brake light 607L, and the brake light 607R are assumed to be illuminated in red. In the following, unless otherwise specified, the loop light 654 and the main light 751 are assumed to be illuminated in orange.

[0324] In the following description, basically, in each diagram, lights that are lit will be painted black, and lights that are not lit will be painted white.

[0325] <When power is turned on> Fig. 38 and Fig. 39 show examples of light emission patterns when the vehicle 1B is powered on. Fig. 38 shows the state when all lights that are to be turned on when the vehicle 1B is powered on are turned on. Fig. 39 shows the final state of the lights when the vehicle 1B is powered on.

[0326] When vehicle 1B is powered off, all lights are off.

[0327] When the power of vehicle 1B is turned on, accessory light 601L, accessory light 601R, day running light 602L, day running light 602R, turn signal 604L, turn signal 604R, auxiliary light 605FL, auxiliary light 605FR, auxiliary light 605BL, auxiliary light 605BR, tail light 606L, tail light 606R, tail light 606CL, and tail light 606CR are illuminated.

[0328] At this time, as indicated by arrows A11 and A12 in the figure, the light is turned on so that the light flows around the body from the center of the front of the body to the center of the back of the body.

[0329] Specifically, the logo portion at the right end of the accessory light 601L and the logo portion at the left end of the accessory light 601R are first lit for a while.

[0330] Next, the LEDs of the accessory light 601L are turned on in order from the right end to the left end. Next, the LEDs of the day running light 602L are turned on in order from the right end to the left end. Next, the LEDs of the turn signal 604L are turned on in order from the front end to the rear end. Next, the LEDs of the auxiliary light 605FL are turned on in order from the front end to the rear end. Next, the LEDs of the auxiliary light 605BL are turned on in order from the front end to the rear end. Next, the LEDs of the tail light 606L are turned on in order from the left end to the right end. Finally, the LEDs of the tail light 606CL are turned on in order from the left end to the right end. As a result, as shown by the arrow A11, each light is turned on so that the light flows counterclockwise from the center of the front of the body to the center of the rear.

[0331] Similarly, the LEDs of the accessory light 601R are turned on in order from the left end to the right end. Next, the LEDs of the day running light 602R are turned on in order from the left end to the right end. Next, the LEDs of the turn signal 604R are turned on in order from the front end to the rear end. Next, the LEDs of the auxiliary light 605FR are turned on in order from the front end to the rear end. Next, the LEDs of the auxiliary light 605BR are turned on in order from the front end to the rear end. Next, the LEDs of the tail light 606R are turned on in order from the right end to the left end. Finally, the LEDs of the tail light 606CR are turned on in order from the right end to the left end. As a result, as shown by the arrow A12, each light is turned on so that the light flows clockwise from the center of the front of the body to the center of the rear.

[0332] The left and right lights will be turned on simultaneously.

[0333] Next, the loop light 654 and the main light 751 are turned on. At this time, the loop light 654 and the main light 751 are turned on so that the entirety of each of them gradually becomes brighter.

[0334] When the loop light 654 and the main light 751 finish lighting, the state shown in FIG. 38 is reached.

[0335] Next, some of the lights are turned off as shown in Fig. 39. At this time, as shown by arrows A13 and A14 in the figure, the lights are turned off in order from the front of the left side and the front of the right side of the body to the center of the back.

[0336] Specifically, the LEDs of the turn signal 604L are turned off in order from the front end to the rear end. Next, the LEDs of the auxiliary light 605FL are turned off in order from the front end to the rear end. Next, the LEDs of the auxiliary light 605BL are turned off in order from the front end to the rear end. Next, the LEDs of the tail light 606L are turned off in order from the left end to the right end. Finally, the LEDs of the tail light 606CL are turned off in order from the left end to the right end. As a result, as shown by the arrow A13, each light is turned off in order counterclockwise from the front of the left side of the body to the center of the back.

[0337] Similarly, the LEDs of the turn signal 604R are turned off in order from the front end to the rear end. Next, the LEDs of the auxiliary light 605FR are turned off in order from the front end to the rear end. Next, the LEDs of the auxiliary light 605BR are turned off in order from the front end to the rear end. Next, the LEDs of the tail light 606R are turned off in order from the right end to the left end. Finally, the LEDs of the tail light 606CR are turned off in order from the right end to the left end. As a result, as shown by the arrow A14, each light is turned off in order clockwise from the front of the right side of the body to the center of the back.

[0338] The lights rotate left and right and turn off simultaneously.

[0339] <When driving and headlights 603 are off> FIG. 40 shows an example of a light emission pattern when the vehicle 1B is being driven (the shift position is set to driving or automatic driving) and the headlights 603 are turned off.

[0340] In this state, the accessory light 601L, the accessory light 601R, the day running light 602L, and the day running light 602R are turned on.

[0341] The day running lights 602L and 602R are always turned on while the vehicle 1B is being driven due to restrictions such as laws and regulations. On the other hand, the accessory lights 601L and 601R must be dimmed or turned off while the vehicle 1B is being driven due to restrictions such as laws and regulations. Therefore, the brightness of the accessory lights 601L and 601R is set lower than normal.

[0342] <When driving and headlights 603 are on> FIG. 41 shows an example of a light emission pattern when the vehicle 1B is being driven (the shift position is set to driving or automatic driving) and the headlights 603 are turned on.

[0343] Comparing Fig. 41 with Fig. 40, it is different in that the headlight 603L, the headlight 603R, the taillight 606L, the taillight 606R, the taillight 606CL, and the taillight 606CR are turned on. That is, in accordance with the lighting of the headlight 603L and the headlight 603R, the taillight 606L, the taillight 606R, the taillight 606CL, and the taillight 606CR are turned on.

[0344] <When braking> FIG. 42 shows an example of the lighting pattern when the brakes are applied.

[0345] Comparing the state of Figure 42 with the state of Figure 40, it is different in that the brake light 607L and the brake light 607R are turned on. That is, the brake light 607L and the brake light 607R are turned on by the brakes being applied.

[0346] <When turn signal is operating> FIG. 43 shows an example of the illumination pattern when the left turn signal is activated.

[0347] Comparing FIG. 43 with FIG. 40, the differences are that the daytime running lights 602L and the tail lights 606L change color, and the turn signals 604L flash.

[0348] Specifically, the LEDs of the daytime running light 602L change from white to orange in order from the right end to the left end. After all the LEDs of the daytime running light 602L have changed to orange, the LEDs of the turn signal 604L light up orange in order from the front end to the rear end.

[0349] At the same time, the LEDs in the taillight 606L change from white to orange in order from the right end to the left end.

[0350] Next, the LEDs of the daytime running light 602L change from orange to white in order from the right end to the left end. After all the LEDs of the daytime running light 602L have changed to white, the LEDs of the turn signal 604L are turned off in order from the front end to the rear end.

[0351] At the same time, the LEDs on the taillight 606L change from orange to white in order from the right end to the left end.

[0352] Thereafter, the same operation is repeated.

[0353] <If door 611 is open> FIG. 44 shows an example of a light emission pattern when any of the doors 611 of the vehicle 1B is open.

[0354] At this time, the accessory light 601L, the accessory light 601R, the day running light 602L, the day running light 602R, the tail light 606L, the tail light 606R, the tail light 606CL, the tail light 606CR, the loop light 654, and the main light 751 are turned on.

[0355] <When Door 611 closes> FIG. 45 shows an example of a light emission pattern when the state changes from one of the doors 611 of the vehicle 1B being open to all of the doors 611 being closed.

[0356] Comparing the state of Figure 45 with the state of Figure 46, it is different in that the main light 751 is turned off. That is, when the door 611 is closed, the main light 751 is turned off.

[0357] <When parking> FIG. 46 shows an example of a light emission pattern when the shift position of the vehicle 1B is set to parking.

[0358] In this case, the accessory light 601L, the accessory light 601R, the day running light 602L, the day running light 602R, the turn signal 604L, the turn signal 604R, the tail light 606L, the tail light 606R, the tail light 606CL, the tail light 606CR, and the loop light 654 are turned on.

[0359] <When power is off> FIG. 47 shows an example of a light emission pattern when the power is turned off from a state in which the shift position of the vehicle 1B is set to parking.

[0360] In this case, the accessory light 601L, the accessory light 601R, the day running light 602L, the day running light 602R, the turn signal 604L, the turn signal 604R, the tail light 606L, the tail light 606R, the tail light 606CL, the tail light 606CR, and the loop light 654 are turned off.

[0361] At this time, the lights are turned off in a flowing manner from the side of the body to the center of the front as shown by arrows A15 and A16, and from the side of the body to the center of the back as shown by arrows A17 and A18.

[0362] Specifically, the LEDs of the turn signal 604L are turned off in order from the rear end to the front end. Next, the LEDs of the day running light 602L are turned off in order from the left end to the right end. Next, the LEDs of the accessory light 601L are turned off in order from the left end to the right end.

[0363] In parallel with this, the LEDs of the turn signal 604R are turned off in order from the rear end to the front end. Next, the LEDs of the day running light 602R are turned off in order from the right end to the left end. Next, the LEDs of the accessory light 601R are turned off in order from the right end to the left end.

[0364] In parallel with this, the LEDs of the tail light 606L are turned off in order from the left end to the right end. Next, the LEDs of the tail light 606CL are turned off in order from the left end to the right end.

[0365] In parallel with this, the LEDs of the tail light 606R are turned off in order from the right end to the left end. Next, the LEDs of the tail light 606CR are turned off in order from the right end to the left end.

[0366] Additionally, the loop light 654 gradually goes out.

[0367] <When a person approaches vehicle 1B> Next, although illustration is omitted, an example of a light emission pattern when the recognition unit 73 detects the approach of a person to the vehicle 1B will be described.

[0368] For example, a light located near the approaching person (near the place where the person approached) lights up or flashes. Specifically, for example, when the approach of a person near the door 611FL is detected, the turn signal 604L and the auxiliary light 605FL near the door 611FL lights up or flashes. Furthermore, for example, after the turn signal 604L and the auxiliary light 605FL are turned on, other lights may also be turned on in sequence.

[0369] Also, for example, when the approach of a person is detected, the lights may be turned on or flashed in the same pattern regardless of the location of the approach. For example, the lights may be turned on so that the light flows around the vehicle.

[0370] Furthermore, for example, the manner in which the light is turned on or blinks (for example, color, pattern, etc.) may be changed depending on the approaching person. For example, if the approaching person is recognized as a pre-registered user (for example, the driver or his / her family member), a white light may be blinked. On the other hand, for example, if the approaching person fails to be recognized (if the approaching person is not recognized as a pre-registered user), a red light may be blinked.

[0371] For example, the body system control unit 84 may automatically unlock the doors when it detects the approach of a registered user.

[0372] In addition, the conditions for turning on or blinking the lights in response to the approach of a person may be limited. For example, the lights may be turned on or blinked in response to the approach of a person only when the power of the vehicle 1B is turned off or when there is no person inside the vehicle 1B.

[0373] In this way, when the approach of a person is detected, for example, by turning on or blinking the exterior lights, it is made known to those around that the vehicle 1B is monitoring the surroundings, and it is possible to prevent theft or vandalism of the vehicle 1B. Also, when the approach of a person is detected, for example, by turning on or blinking the interior lights, it is made known to those around that the vehicle 1B is also monitoring the interior of the vehicle.

[0374] <Lighting pattern of steering wheel 651 illumination 652> Next, the light emission pattern of the illumination 652 of the steering wheel 651 will be described.

[0375] For example, the light control unit 85 turns on or blinks the illumination 652 in a pattern according to the situation based on at least one of the situation of the vehicle 1B, the situation around the vehicle 1B, and the situation of the passenger. The light emission pattern of the illumination 652 is defined by at least one of the color, brightness, blinking pattern, light movement, and light-emitting area, for example.

[0376] Specifically, for example, when the vehicle 1B is ready for autonomous driving, multiple short light bands rotate approximately once around the illumination 652. Then, the entire illumination 652 flashes. This allows the driver to reliably recognize that the vehicle is ready for autonomous driving.

[0377] Also, for example, when vehicle 1B changes lanes by autonomous driving, first, the entire illumination 652 is turned on. Then, after illumination 652 is turned off, the LED of illumination 652 in the direction in which vehicle 1B moves to change lanes flashes. Then, after the lane change is completed, the entire illumination 652 is turned on and then turned off. In this way, a notification of the lane change is given from before the lane change until the lane change is completed, which gives a sense of security to passengers.

[0378] Furthermore, for example, when the driver needs to be alerted to an approaching obstacle, falling asleep, or the like, the upper part of the illumination 652 flashes in red, allowing the driver to quickly sense and avoid danger.

[0379] Also, for example, when voice recognition is being performed, multiple light bands move left and right in the upper part of the illumination 652. This allows the driver to recognize that the voice recognition function is operating normally.

[0380] <Shooting process> Next, the photographing process executed by the vehicle 1B will be described with reference to the flowchart of FIG.

[0381] This process is started, for example, when the power of the vehicle 1B is turned on, and is ended when the power of the vehicle 1B is turned off.

[0382] In step S1, the camera 51 captures the surroundings and interior of the vehicle 1B. Specifically, the camera 51 is composed of a plurality of cameras, each of which captures the surroundings or interior (room) of the vehicle 1B. Each camera supplies video data obtained by capturing the video to the information processing unit 701.

[0383] In step S2, the image processing unit 711 performs image processing as necessary.

[0384] For example, the image processing unit 711 superimposes visual information onto each frame of the video data, as will be described later with reference to FIGS.

[0385] Furthermore, for example, the image processing unit 711 performs image processing such as noise cancellation on each frame of the video data based on the sensor data from the in-vehicle sensor 26 and the vehicle sensor 27. For example, when it is raining, the image processing unit 711 removes noise such as raindrops from within the frame.

[0386] In step S3, the metadata adding unit 712 adds metadata to the video data.

[0387] For example, the metadata assignment unit 712 assigns metadata regarding at least one of the shooting location, shooting date and time, the status of the vehicle 1B, the status inside the vehicle, and the surrounding status to each video data shot while traveling, etc.

[0388] The metadata regarding the shooting location includes, for example, at least one of the position of the vehicle 1B at the time of shooting, the shooting direction, the position of the camera used for shooting, and the like.

[0389] The metadata relating to the shooting date and time includes, for example, at least one of the date and time when the image was shot.

[0390] The metadata regarding the status of the vehicle 1B includes, for example, at least one of the speed, acceleration, traveling direction, destination, and state of the vehicle 1B (for example, the presence or absence of an obstacle, the presence or absence of an accident, the amount of charge, etc.).

[0391] The metadata regarding the situation inside the vehicle includes, for example, identification information of the passengers (e.g., name, ID, etc.), the seat positions of the passengers, the situation of the passengers (e.g., behavior, dozing, etc.), voice recognition results of conversations inside the vehicle, activity levels inside the vehicle, etc. The activity levels are set based on, for example, the volume of conversations inside the vehicle and the movements of the passengers.

[0392] Metadata regarding the surrounding conditions may include at least one of the following: weather, temperature, humidity, brightness, the position and type of surrounding objects (e.g., other vehicles, pedestrians, obstacles, traffic signs, landmarks, etc.), the presence or absence and type of events (e.g., accidents, construction, etc.), etc.

[0393] Also, for example, the metadata adding unit 712 may add metadata input by a user (passenger) to video data. For example, a movie button may be provided on the steering wheel 651, and the driver may press the movie button when a scene or the like for which video data is to be saved has been shot. Examples of scenes for which video data is to be saved include when the scenery around the vehicle 1B is good, or when a problem such as an accident has occurred. For example, the metadata adding unit 712 adds metadata indicating that saving is necessary to video data shot during a period specified by the user using the movie button.

[0394] The unit for assigning metadata can be set arbitrarily. For example, metadata is assigned on a frame-by-frame basis, on a unit basis consisting of multiple frames, or on a video data basis. Also, the unit for assigning metadata may be variable depending on the type of metadata.

[0395] In step S4, the vehicle 1B accumulates the video data. Specifically, the image processing unit 711 causes the recording unit 28 to store the video data.

[0396] At this time, sound data recorded around and inside the vehicle 1B may be stored together with the video data.

[0397] In step S5, the vehicle 1B displays the video. Specifically, the output control unit 713 causes the HMI 31 to display the video based on the video data.

[0398] For example, the moving images shown in FIGS.

[0399] Fig. 49 shows an example of a moving image displayed when the display mode is set to the object detection mode. In this example, a vehicle and a person are selected as detection targets. A frame 801 surrounding a vehicle and a frame 802 surrounding a person detected in a moving image captured in front of the vehicle 1B are displayed. Characters indicating the type of object detected (CAR and PERSON) are also displayed within the frames 801 and 802.

[0400] 50 shows an example of a moving image displayed when the display mode is set to the prediction mode. In this example, the predicted position of a moving device (in this example, a person) detected in the moving image 0.5 seconds later is displayed. In this example, the current position of the person is shown by a dotted line, and the predicted position of the person 0.5 seconds later is shown by a solid line and is surrounded by a frame 811.

[0401] 51 shows an example of a moving image displayed when the display mode is set to the search mode. Here, an example is shown in which possible parking locations are searched for and displayed. Specifically, a frame 821 showing possible parking locations is displayed.

[0402] For example, a plurality of parking positions may be displayed. Also, for example, parking positions may be displayed using a bird's-eye view.

[0403] Fig. 52 shows an example of a moving image displayed when the display mode is set to the search mode, for example. Here, an example is shown in which a person who has been registered in advance is searched for and the detected person is displayed. Specifically, a frame 831 that surrounds the person who has been registered in advance is displayed in the moving image. Also, the name of the detected person (Sakura) is displayed in the frame 831.

[0404] For example, a video of a direction (for example, forward, backward, left, right, etc.) selected by a passenger such as a driver may be displayed on the center display 201. Also, for example, a 360-degree video of the surroundings of the vehicle 1B may be displayed on the center display 201 while scrolling.

[0405] Also, for example, sensor data obtained by the radar 52 and the LiDAR 53 may be converted into an image and displayed on the center display 201. Also, visual information based on data obtained by two or more types of sensors among the camera 51, the radar 52, and the LiDAR 53 may be superimposed on a moving image and displayed.

[0406] Furthermore, for example, the planned direction of travel, etc. may be displayed superimposed on the video based on the route plan.

[0407] In addition, for example, the movement of other vehicles predicted based on information obtained through vehicle-to-vehicle communication may be superimposed on the video and displayed.

[0408] Thereafter, the process returns to step S1, and the processes from step S1 to step S5 are repeatedly executed.

[0409] <Video editing processing> Next, the video editing process executed by the vehicle 1B will be described with reference to FIG.

[0410] This process is started, for example, when an instruction to edit a moving image is input to the HMI 31. The instruction to edit a moving image includes, for example, the period during which the moving image to be edited was shot (hereinafter, referred to as the period to be edited), conditions for extracting frames to be included in the moving image, and the like.

[0411] In step S51, the image processing unit 711 edits the moving image.

[0412] For example, based on metadata, etc., the image processing unit 711 extracts frames to be included in the edited video data (hereinafter referred to as edited video data) from each video data (hereinafter referred to as shot video data) shot within the editing period.

[0413] For example, the image processing unit 711 extracts frames to which metadata indicating that storage is required has been added.

[0414] Also, for example, the image processing unit 711 extracts frames that satisfy given conditions based on metadata.

[0415] For example, the image processing unit 711 extracts frames based on the route of the vehicle 1B during the editing period and metadata. For example, the image processing unit 711 detects the route of the vehicle 1B during the editing period based on the map data and the location information included in the metadata of the video data. Then, for example, when the vehicle 1B is traveling on a road along the sea, the image processing unit 711 preferentially extracts frames photographed in the direction of the sea. For example, when the vehicle 1B is traveling on a high ground such as a mountain, the image processing unit 711 preferentially extracts frames photographed in a direction overlooking the surroundings. Furthermore, for example, the image processing unit 711 preferentially extracts frames that include landmarks and the like around the vehicle 1B based on the traveling direction and turning direction of the vehicle 1B.

[0416] For example, the image processing unit 711 extracts a frame when a specific event occurs based on the metadata. For example, when the vehicle 1B is involved in an accident or when an accident occurs around the vehicle 1B, the image processing unit 711 preferentially extracts frames from the time period before and after the accident. Also, for example, the image processing unit 711 preferentially extracts frames captured in the direction in which the accident occurred.

[0417] For example, when the image processing unit 711 detects excitement inside the vehicle based on the activity level included in the metadata, the image processing unit 711 preferentially extracts frames taken inside the vehicle. Also, for example, the image processing unit 711 extracts audio data corresponding to the extracted frames of the vehicle.

[0418] Also, for example, the output control unit 713 may cause the center display 201 and the tablet terminal 662L to display a list of video data or actual video, and the image processing unit 711 may edit the video based on instructions from the user.

[0419] For example, the user may select video data and frames that he or she wishes to include in the edited video data. In this case, for example, various types of information, visual effects, graffiti, and other visual information may be superimposed on the video frames displayed on the center display 201 and the tablet terminal 662L.

[0420] Then, the image processing unit 711 generates edited video data, for example, by combining the extracted frames. For example, the image processing unit 711 generates edited video data by connecting the extracted frames in chronological order, arranging or overlapping multiple frames within the same frame, etc. Furthermore, the image processing unit 711 superimposes visual information on the extracted frames as necessary, for example, based on metadata.

[0421] In step S52, the vehicle 1B saves the video data. For example, the image processing unit 711 stores the edited video data in the recording unit 28. In addition, for example, the image processing unit 711 transmits the edited video data to a server or an information processing terminal (for example, a smartphone, a tablet terminal, a personal computer, etc.) held by the passenger via the communication unit 22, and stores the edited video data.

[0422] After that, the video editing process ends.

[0423] In this way, the video of the surroundings and interior of the vehicle 1B can be easily edited. This makes it possible to easily generate video data that records memories of a trip, video data that extracts scenic scenes, video data that records the situation of an accident, and the like.

[0424] <<5. Modification of the Second Embodiment>> A modification of the above-described second embodiment of the present technology will be described below.

[0425] For example, the captured video data with the metadata added before editing may be stored or copied outside vehicle 1B, and the video may be edited by a device external to vehicle 1B (e.g., a server (cloud), a smartphone, a tablet terminal, a personal computer, etc.). Also, for example, the video may be edited by a device external to vehicle 1B in cooperation with vehicle 1B. Furthermore, for example, metadata may be added by a device external to vehicle 1B.

[0426] In addition, in the above description, an example has been shown in which the front line and the tail line of the loop line L1 are illuminated, but they may not be illuminated, taking into consideration, for example, design, legal regulations, and the like.

[0427] Furthermore, in the above description, an example has been shown in which the central portion of the front line and the central portion of the tail line are separated, but they may be connected without being separated, taking into consideration, for example, the design.

[0428] Also, for example, an optical sensor such as a camera, a radar, or a LiDAR may be disposed inside the headlight 603.

[0429] Furthermore, in the above description, an example in which the vehicle 1B is a left-hand drive vehicle has been shown, but the present technology can of course also be applied to a right-hand drive vehicle. When the present technology is applied to a right-hand drive vehicle, the above-mentioned vehicle exterior and interior layouts are appropriately changed to suit the right-hand drive vehicle.

[0430] In addition, the type of vehicle to which the present technology can be applied is not particularly limited. In addition, the present technology can be applied to moving devices such as personal mobility, airplanes, ships, construction machines, agricultural machines, etc., in addition to vehicles. In addition, the moving devices to which the present technology can be applied include, for example, moving devices such as drones and robots that capture images of the surroundings without a person on board.

[0431] <<6. Third embodiment>> Next, a third embodiment of the present technology will be described with reference to FIGS.

[0432] In each drawing of the third embodiment, the same reference numerals are used for the parts corresponding to the first and second embodiments, and the description thereof will be omitted as appropriate. Furthermore, the configurations included in the first and second embodiments and not shown or described in the third embodiment are basically included in the third embodiment, except when an alternative configuration is shown or described, or when a description is given to the effect that the configuration is deleted.

[0433] <Example of vehicle 1C interior configuration> First, with reference to Figs. 54 to 57, an example of the interior configuration of the vehicle 1C will be described.

[0434] Fig. 54 is a schematic diagram of the interior of the vehicle 1C looking down from the right. Fig. 55 is a schematic diagram showing the front of the interior of the vehicle 1C. Fig. 56 is a schematic diagram showing the front of the driver's seat 101 in the interior of the vehicle 1C. Fig. 57 is an enlarged view of the vicinity of the steering wheel 1001 of the vehicle 1C.

[0435] 54, the parts of the vehicle 1C that are significantly different from the vehicles 1A and 1B are circled. Specifically, the steering wheel 1001, the camera module 1007, and the camera module 1008 are the parts that are significantly different.

[0436] As shown in FIG. 57, the steering wheel 1001 is different from the steering wheel 651 in FIG. 30 in the position of the illumination 1002. The illumination 1002 includes a plurality of LEDs arranged in a ring shape at a predetermined distance inside from the outer periphery of the central part of the steering wheel 1001. Each LED can be individually controlled to be turned on / off, color, brightness, etc. Therefore, the color, brightness, and light-emitting area (range that emits light) of the illumination 1002 are variable. The logo of the vehicle 1C is formed on the inside of the illumination 1002.

[0437] An airbag (not shown) is housed in the center of the steering wheel 1001. The illumination 1002 is arranged to avoid the area where the center of the steering wheel 1001 may crack when the airbag is deployed. This prevents fragments of the illumination 1002 and harmful substances from scattering when the airbag is deployed.

[0438] An operation unit 1003 having a plurality of buttons is provided on the left spoke of the steering wheel 1001. An operation unit 1004 having a plurality of buttons is provided on the right spoke of the steering wheel 1001.

[0439] 57, a stalk lever 1005, which is a rod-shaped operating body, is provided so as to extend leftward from a steering column (not shown) behind the steering wheel 1001. Also, a stalk lever 1006, which is a rod-shaped operating body, is provided so as to extend rightward from the steering column (not shown) behind the steering wheel 1001. The stalk lever 1006 is configured similarly to the stalk lever 655 in FIG. 30, for example.

[0440] As shown in FIGS. 54, 55, and 57, the camera module 1007 is provided on the dashboard 105, slightly closer to the driver's seat 101 than the center in the left-right direction. The camera module 1007 has a shape like a part of a truncated cone cut out vertically, and has a vertical flat surface on its side. The camera module 1007 has the flat surface on its side facing the direction of the driver's seat 101, and as will be described later, photographs a range including at least the head of the driver sitting in the driver's seat 101.

[0441] As shown in FIG. 55, the camera module 1008 is provided at the tip of the ceiling of the vehicle 1C and at the center in the left-right direction, near the digital rearview mirror 204. The camera module 1008 is equipped with two ToF cameras (not shown). As will be described later, each ToF camera photographs the driver and the passenger in the passenger seat 102, respectively.

[0442] A circular light 1009 is provided on the lower surface of the camera module 1008. The camera module 1008 also incorporates a microphone (not shown) and the like for collecting the voices of passengers and the like.

[0443] Also, in this example, as shown in FIG. 55, the central part 201C of the center display 201 is divided into a display part 201CL in front of the driver's seat 101, a display part 201CC between the driver's seat 101 and the passenger seat 102, and a display part 201CR in front of the passenger seat 102. As described above, it is also possible to connect the display part 201CL, the display part 201CC, and the display part 201CR to form one display part. A display part 201LL is provided at the left end 201L of the center display 201. A display part 201RR is provided at the right end 201R of the center display 201.

[0444] <Example of Installation Position of ToF Camera> Next, with reference to FIGS. 58 to 63, an example of the installation position of the ToF camera will be described.

[0445] Fig. 58 shows an example of the installation position of the ToF camera in the camera module 1008. The diagram at the top left of Fig. 58 is a schematic diagram of the camera module 1008 viewed from a diagonally downward left direction. The diagram at the bottom left of Fig. 58 is a schematic diagram of the camera module 1008 viewed from a diagonally downward left direction. The diagram at the right of Fig. 58 is a schematic diagram of the camera module 1008 viewed from a diagonally downward front direction.

[0446] The camera module 1008 has a shape in which a half of a truncated cone is connected to the rear end (at the rear of the vehicle 1C) of a base obtained by cutting the left and right sides of a rectangular parallelepiped diagonally. As described above, the camera module 1008 is disposed at the tip of the ceiling of the vehicle 1C and in the center in the left-right direction.

[0447] A ToF camera (hereinafter referred to as ToF camera P101L) is built in at position P101L near the front end on the left side of the truncated cone portion of the camera module 1008. The optical axis of the ToF camera P101L is directed toward the middle of the driver's eyes when a driver of a standard build sits in the driver's seat 101 arranged in a standard position, for example.

[0448] 59 shows an example of the imaging range of the ToF camera P101L. The ToF camera P101L can capture almost the entire body of the driver DR1, including from the head to the feet, from an upper right diagonal direction, regardless of the physique of the driver DR1 and the position of the driver's seat 101.

[0449] A ToF camera (hereinafter referred to as ToF camera P101R) is built into position P101R near the front end on the right side of the truncated cone portion of camera module 1008. Although not shown in the drawings, the ToF camera P101R, like the ToF camera P101L, is capable of capturing an image of almost the entire body of a passenger, including from the head to the feet, from an upper left diagonal direction, regardless of the physique of the passenger sitting in passenger seat 102 and the position of passenger seat 102.

[0450] A cover 1008A with a black gloss finish is provided on the side of the truncated cone portion of the camera module 1008. The cover 1008A makes it difficult to see the lenses of the ToF camera P101L and the ToF camera P101R from the outside. In addition, cameras and sensors other than the ToF camera P101L and the ToF camera P101R can be provided on the back side of the cover 1008A.

[0451] For example, the DMS 30 (FIG. 1) recognizes the driver's posture and movement by performing skeleton recognition of the driver based on the image of the ToF camera P101L. For example, the DMS 30 performs recognition processing of the driver (for example, personal recognition) based on the image of the ToF camera P101L.

[0452] Similarly, for example, the DMS 30 recognizes the posture and movement of the passenger in the passenger seat 102 by performing bone structure recognition or the like on the basis of an image from the ToF camera P101R. For example, the DMS 30 performs recognition processing of the passenger on the basis of an image from the ToF camera P101L.

[0453] Next, other examples of the installation position of the ToF camera will be described with reference to FIGS.

[0454] For example, as shown in the upper diagram of FIG. 60, a ToF camera (hereinafter referred to as ToF camera P101C) is built into the rear end portion of the camera module 1008 at position P101C.

[0455] The lower diagram of Fig. 60 shows an example of the imaging range of the ToF camera P101C. In this way, the ToF camera P101C can image an area A101L including the headrest of the rear seat 103L and an area A101R including the headrest of the rear seat 103R. Therefore, the ToF camera P101C can simultaneously image the upper body including the head of the passenger in the rear seat 103L and the upper body including the head of the passenger in the rear seat 103R.

[0456] As shown in Fig. 60, the above-mentioned seat speakers are embedded below the headrest of each seat. Specifically, seat speaker 1021 is embedded below the headrest of driver's seat 101. Seat speaker 1022 is embedded below the headrest of passenger seat 102. Seat speaker 1023L is embedded below the headrest of left rear seat 103L. Seat speaker 1023R is embedded below the headrest of right rear seat 103R.

[0457] Although not shown, the speakers 653FL to 653BR described above with reference to FIGS. 27 and 28 are embedded in each door, achieving 360-degree real audio.

[0458] 61 to 63 show examples of the installation positions of the ToF camera installed in the rear seat 103L.

[0459] For example, a ToF camera (hereinafter referred to as ToF camera P111L) is provided at a position P111L immediately above the upper frame of the door 611BL on the ceiling of the vehicle 1C. The ToF camera P111L can capture, for example, almost the entire body of an occupant in the rear seat 103L from an upper left diagonal direction. In addition, a reading light (not shown) capable of irradiating illumination light within a range A111 is provided around the ToF camera P111L.

[0460] For example, a ToF camera (hereinafter referred to as ToF camera P112L) is provided at a position P112L near the center in the width direction of the upper end of the tablet terminal 662L. The ToF camera P112L can capture an image of a portion including the upper body of an occupant in the rear seat 103L from the front, for example.

[0461] Although not shown in the drawings, ToF cameras are provided for rear seat 103R at positions P111R and P112R similar to positions P111L and P112L for rear seat 103L. Hereinafter, the ToF camera provided at position P111R will be referred to as ToF camera P111R, and the ToF camera provided at position P112R will be referred to as ToF camera P112R.

[0462] For example, the DMS30 recognizes the posture and movement of the passenger in the rear seat 103L by performing skeleton recognition or the like of the passenger based on at least one image of the ToF camera P111L and the ToF camera P112L. For example, the DMS30 performs recognition processing of the passenger based on at least one image of the ToF camera P111L and the ToF camera P112L. For example, the DMS30 performs lip reading of the passenger based on the image of the ToF camera P112L.

[0463] Similarly, for example, the DMS30 recognizes the posture and movement of the passenger in the rear seat 103R by performing skeleton recognition or the like of the passenger based on at least one image of the ToF camera P111R and the ToF camera P112R. For example, the DMS30 performs recognition processing of the passenger based on at least one image of the ToF camera P111R and the ToF camera P112R. For example, the DMS30 performs lip reading of the passenger based on the image of the ToF camera P112L.

[0464] 61, the light control unit 85 controls the on / off and brightness of the reading light based on the behavior of the passenger in the rear seat 103L. Also, for example, the light control unit 85 controls the brightness of the illumination range A112 of the loop light 654 near the rear seat 103L based on the behavior of the passenger in the rear seat 103L. For example, when the light control unit 85 recognizes the passenger's action of reading a document or the like, it turns on the reading light or brightens the illumination range A112 of the loop light 654.

[0465] For example, the same lighting control as that for the rear seat 103L is executed in the passenger seat 102 and the rear seat 103R.

[0466] 62, the DMS30 can recognize the position and shape of the left ear EL1 (not shown) and right ear ER1 of the passenger DR2 in the rear seat 103L based on an image of the ToF camera P111L. In response to this, the audio control unit, which is a part of the function of the HMI31, controls the seat speaker 1023L of the rear seat 103L to control the sound image, sound field, volume, etc. of various sounds for the passenger DR2 according to the position and shape of the ears of the passenger DR2.

[0467] For example, in the driver's seat 101, the passenger seat 102, and the rear seat 103R, the sound images, sound fields, volumes, and the like of various sounds for passengers in each seat are controlled in the same manner as in the rear seat 103L.

[0468] For example, as shown in the lower right diagram of Fig. 63, the body system control unit 84 controls the position and attitude of the rear seat 103R and the position and attitude of the tablet terminal 662R based on the physique of the occupant in the rear seat 103R, etc. For example, the rear seat 103R is deeply reclined according to the physique of the occupant, and the tablet terminal 662R moves in a direction approaching the rear seat 103R, as shown by the arrow A113.

[0469] Similarly, the position and attitude of the rear seat 103L and the tablet terminal 662L are controlled based on the build of the passenger in the rear seat 103L.

[0470] <Details of camera module 1007> Next, the camera module 1007 will be described in detail with reference to FIGS.

[0471] 64 shows an example of the installation position of the camera module 1007. As described above, the camera module 1007 is provided on the dashboard 105 slightly closer to the driver's seat 101 than the center in the left-right direction.

[0472] FIG. 65 shows an example of the internal configuration of the camera module 1007.

[0473] The camera module 1007 includes a housing 1041 , a camera 1042 , an LED 1043 - 1 and an LED 1043 - 2 , and a cover 1044 .

[0474] In the following description, when there is no need to distinguish between the LED 1043-1 and the LED 1043-2, they will simply be referred to as the LED 1043.

[0475] The camera 1042, and the LEDs 1043-1 and 1043-2 are arranged in the casing 1041 so as to be aligned in the horizontal direction. The camera 1042 is arranged in a position closer to the driver's seat 101 than the LEDs 1043-1 and 1043-2. ​​The optical axes of the camera 1042, and the LEDs 1043-1 and 1043-2 are approximately parallel and each point toward the driver's seat 101. The camera 1042 is capable of capturing an image of at least an area including the top of the driver's head to the Adam's apple, regardless of the driver's physique and the position of the driver's seat 101.

[0476] The camera 1042 is capable of two types of photography: RGB photography (color photography) and IR photography (infrared photography). The camera 1042 automatically switches between RGB photography and IR photography based on conditions such as the surrounding brightness.

[0477] The LED 1043 is a light source that outputs IR light (infrared light). By providing two LEDs 1043, LED 1043-1 and LED 1043-2, the amount of light emitted by each LED 1043 can be reduced while ensuring the amount of IR light, and the amount of heat generated by each LED 1043 can be reduced.

[0478] Cover 1044 is a flat portion on the side of housing 1041, and is disposed in front of camera 1042 and each LED 1043. Cover 1044 makes the lens of camera 1042 and the light-emitting portions of each LED 1043 difficult to see from the outside.

[0479] For example, in a bright environment such as daytime, the camera 1042 automatically performs RGB shooting to obtain a color image. On the other hand, in a dark environment such as nighttime, each LED 1043 is automatically turned on, and the camera 1042 automatically performs IR shooting to obtain a monochrome image. This makes it possible to reliably capture the driver's head regardless of the brightness of the surroundings.

[0480] For example, the DMS 30 performs lip reading of the driver and recognizes the contents of the driver's speech based on the image of the camera 1042. For example, the DMS 30 performs black eye detection, line of sight detection, wakefulness determination, recognition processing, etc. of the driver based on the image of the camera 1042.

[0481] Next, a modified example of the installation position of the camera 1042 will be described with reference to FIG.

[0482] 66 is a schematic diagram of the vicinity of the steering wheel 1001 of the vehicle 1C viewed diagonally from above right. The diagram on the right side of FIG.

[0483] For example, it is conceivable to install camera 1042 at position P121 on dashboard 105 in front of driver's seat 101. In this case, it is possible to capture an image of the driver's face from the front. However, when steering wheel 1001 is rotated, the spokes of steering wheel 1001 block the field of view of camera 1042.

[0484] For example, it is conceivable to install the camera 1042 at a position P122 near the left end of the dashboard 105. In this case, symmetrically to the camera module 1007, the face of the driver is photographed from diagonally forward and to the left.

[0485] For example, it is conceivable to install the camera 1042 at position P123 on the steering column of the steering wheel 1001. In this case, it is possible to capture an image of the driver's face from a close distance and from the front. However, as with position P121, when the steering wheel 1001 is rotated, the spokes of the steering wheel 1001 block the field of view of the camera 1042. Also, the driver's view of the display unit 201CL of the center display 201 is blocked.

[0486] For example, it is conceivable to incorporate the camera 1042 at a position P124 on the right side of the steering wheel 1001 and near the upper end of the central portion 201C of the center display 201. In this case, the face of the driver is photographed from the diagonally forward right. Meanwhile, since there is a restriction on the storage space for the camera 1042, it is necessary to miniaturize the camera 1042. Also, there are cases where the camera 1042 and the LED 1043 are disposed apart from each other.

[0487] For example, it is conceivable to incorporate the camera 1042 in a position P125 near the upper left corner of the left end portion 201L of the center display 201. In this case, the face of the driver is photographed from the diagonally forward left. On the other hand, since there is a restriction on the storage space for the camera 1042, it is necessary to miniaturize the camera 1042. Also, the camera 1042 and the LED 1043 may be disposed apart from each other.

[0488] For example, it is conceivable to place a smartphone in the center of the steering wheel 1001 and use the smartphone to capture an image of the driver's face. In this case, the driver's face can be captured from the front. However, there is a risk that the smartphone may be damaged when the airbag is deployed. Also, when the steering wheel 1001 is rotated, the driver's face is captured at an angle.

[0489] For example, it is also conceivable to incorporate the camera 1042 inside the A-pillar (not shown) on the driver's seat side of the vehicle 1C. In this case, the driver's face is photographed from the upper left diagonal direction. On the other hand, since there are restrictions on the storage space for the camera 1042, it is necessary to miniaturize the camera 1042. Also, the camera 1042 and the LED 1043 may be arranged separately.

[0490] For example, if the camera 1042 can be sufficiently wide-angled, the camera module 1008 may be arranged at the center in the left-right direction of the dashboard 105 so that the optical axis of the camera 1042 faces the rear of the vehicle 1C.

[0491] FIG. 67 shows an example of the imaging range of the camera 1042 in this case. In this example, the entire headrest of the driver's seat 101, the entire headrest of the passenger seat 102, a part of the headrest of the rear seat 103L, and a part of the headrest of the rear seat 103R are included in the imaging range. Therefore, with one camera 1042, it becomes possible to image the entire head of the driver, the entire head of the passenger in the passenger seat 102, a part of the head of the passenger in the rear seat 103L, and a part of the head of the passenger in the rear seat 103R.

[0492] <Operation example of CMS> Next, with reference to FIGS. 68 to 85, an operation example of the CMS of the vehicle 1C will be described. Hereinafter, mainly the display example of the display unit 201LL at the left end 201L of the center display 201 will be described.

[0493] For example, the display control unit, which is a part of the function of the HMI 31, causes the display unit 201LL to display an image based on the image of the camera 51SL. More specifically, the display control unit sets a display range within the image of the camera 51SL and causes the display unit 201LL to display the image within the set display range.

[0494] Also, the recognition unit 73 performs object recognition based on the image of the camera 51SL or the like. The display control unit controls the display of the display unit 201LL based on the recognition result of the object and the state of the vehicle 1C or the like.

[0495] 68 shows an example of a display on the display unit 201LL in a normal state. In this example, a part of the body of the vehicle 1C, a vehicle 1101 that is a following vehicle traveling in the lane to the left of the vehicle 1C and diagonally rearward to the left, and a road surface 1102 are displayed.

[0496] For example, when the vehicle 1101 approaches the vehicle 1C and the distance between the vehicle 1101 and the vehicle 1C falls within a predetermined range, a vertically long band-like visual effect bar 1103 is displayed superimposed on the left end of the display unit 201LL, as shown by diagonal lines in Fig. 69. The bar 1103 is displayed, for example, when the vehicle 1101 is within the range that can be seen on the display unit 201LL and the distance between the vehicle 1C falls within a predetermined range. The color of the bar 1103 is a conspicuous color, for example, yellow.

[0497] The display of the bar 1103 alerts the driver, and an accident can be reliably prevented.

[0498] Note that the display mode of the bar 1103, such as color, brightness, shape, blinking pattern, movement, etc., is not limited to this example and can be changed arbitrarily. For example, the display mode of the bar 1103 changes depending on the degree of danger.

[0499] For example, when the left turn signal of the vehicle 1C is turned on, the vehicle 1C may turn left (change direction to the left) or change lanes to the left lane, increasing the risk of collision or contact with the vehicle 1101. In response to this, for example, as shown in Fig. 70, the bar 1103 becomes wider and flashes, thereby further attracting the driver's attention.

[0500] Furthermore, for example, when the left turn signal of vehicle 1C is turned on, the blinking speed of bar 1103 changes according to the degree of danger. For example, the faster the absolute speed of vehicle 1101 or the relative speed with respect to vehicle 1C, the faster the blinking speed of bar 1103. For example, the closer vehicle 1101 is, in other words, the shorter the distance between vehicle 1101 and vehicle 1C, the faster the blinking speed of bar 1103. Note that, for example, a warning sound may be sounded in synchronization with the blinking of bar 1103.

[0501] For example, when the absolute speed of the vehicle 1101 or the relative speed with respect to the vehicle 1C is equal to or greater than a predetermined threshold, the timing at which the bar 1103 is displayed is advanced. That is, the bar 1103 is displayed from the point in time when the vehicle 1101 is located farther away from the vehicle 1C.

[0502] 71, when a vehicle 1101 is recognized diagonally rearward to the left, a mark 1104, which is a triangular visual effect indicating the presence of the vehicle 1101, may be displayed superimposed near the vehicle 1101. The mark 1104 moves in accordance with the movement of the vehicle 1101.

[0503] This allows the driver to reliably recognize the position of the vehicle 1101 even when the visibility of the image displayed on the display unit 201LL is poor due to fog, haze, or the like, for example.

[0504] The position of the vehicle 1101 may be notified in a display manner different from that of the mark 1104 in Fig. 71. For example, a frame surrounding the periphery of the vehicle 1101 may be displayed.

[0505] Although detailed explanation is omitted, the display unit 201RR at the right end 201R of the center display 201 also displays a similar message based on the situation of the vehicle traveling in the lane to the right of the vehicle 1C and the operation of the right turn signal.

[0506] In addition, in the above description, an example of issuing a warning to other vehicles has been shown, but similar processing is also performed when issuing a warning to a moving body other than a vehicle (for example, a motorcycle, etc.).

[0507] Also, for example, the display control unit controls the range of the image displayed on the display unit 201LL in accordance with the operation of the direction indicator.

[0508] FIG. 72 shows an example in which the display range of an image displayed on the display unit 201LL is changed in accordance with the operation of the left direction indicator.

[0509] 72A and 72B show examples of images captured by the camera 51SL. In the images, vehicles 1121-1 to 1121-4 and the like traveling in the lane to the left of the vehicle 1C are captured.

[0510] For example, when the left turn signal of the vehicle 1C is turned off, the image in the area A131a in A of Fig. 72 is extracted, and the image in the area A132a is displayed on the display unit 201LL. The area A132a includes the vehicles 1121-2 to 1121-4, but does not include the vehicle 1121-1.

[0511] On the other hand, when the left direction indicator of the vehicle 1C is turned on, the image in the area A131b in FIG. 72B is extracted, and the image in the area A132b is displayed on the display unit 201LL. The areas A131b and A132b are larger than the areas A131a and A132a. That is, the display range of the display unit 201LL is zoomed out (expanded) and widened. Then, the rear end of the vehicle 1121-1 is included in the area A132b in addition to the vehicles 1121-2 to 1121-4.

[0512] This allows the driver to check the lane conditions on the left side of the vehicle 1C in a wider range when the driver turns on the left turn signal, which allows the driver to change lanes or turn left more safely.

[0513] For example, when the display range of the display unit 201LL is zoomed out, an icon, frame, or the like indicating that the display range has been zoomed out may be displayed on the display unit 201LL.

[0514] Here, UN ECE-R46 (United Nations Economic Commission for Europe Regulation 46) specifies the visibility range that must be visible in the CMS (hereinafter referred to as the required visibility range). The required visibility range differs depending on the vehicle capacity, weight, etc., and Figure 73 shows an example.

[0515] In this example, the required viewing range consists of area A133L, area A133R, and area A133C.

[0516] The area A133L is an area surrounded by points P132L to P136L on the left side of the vehicle 1C. Specifically, the leftmost point of the vehicle 1C corresponding to the same position as the driver's eye position in the longitudinal direction of the vehicle 1C is set as the reference point P131L. The point P132L is a point 4 m behind the reference point P131L. The point P133L is a point 60 m behind the reference point P131L. The point P134L is a point 1 m away from the point P132L in the left direction. The point P135L is a point 20 m behind and 4 m away from the reference point P131L in the left direction. The point P136L is a point 60 m behind and 4 m away from the reference point P131L in the left direction.

[0517] The area A133R is an area surrounded by points P132R to P136R on the right side of the vehicle 1C. Specifically, the rightmost point of the vehicle 1C corresponding to the same position as the driver's eye position in the longitudinal direction of the vehicle 1C is set as the reference point P131R. The point P132R is a point 4 m behind the reference point P131R. The point P133R is a point 60 m behind the reference point P131R. The point P134R is a point 1 m to the right from the point P132R. The point P135R is a point 20 m behind and 4 m to the right from the reference point P131R. The point P136R is a point 60 m behind and 4 m to the right from the reference point P131R.

[0518] Area A133C is 60 m or more behind the eye position of the driver of vehicle 1C, and is an area 20 m wide centered on the lateral center of vehicle 1C.

[0519] Therefore, for example, as shown in FIG. 72, when the display range of the display unit 201LL is changed in response to the operation of the left direction indicator, the display range is changed so as to always include the area A133L.

[0520] Although detailed description is omitted, the display range of the display unit 201RR of the center display 201 is similarly controlled in response to the operation of the right direction indicator. Also, the display range of the display unit 201RR is changed so that the area A133R is always included.

[0521] Also, for example, the display control unit controls the display range of the display unit 201LL based on the shift position of the vehicle 1C.

[0522] FIG. 74 shows an example in which the display range of the display unit 201LL is changed based on the shift position of the vehicle 1C.

[0523] 74A and 74B show examples of images captured by the camera 51SL. The images show vehicles 1141-1 and 1141-2, etc., parked side by side on the left rear side of the vehicle 1C.

[0524] For example, before the shift position of the vehicle 1C is set to reverse, the image within the area A141a in A of FIG. 74 is extracted, and the image within the area A142a is displayed on the display unit 201LL.

[0525] On the other hand, after the shift position of the vehicle 1C is set to reverse, the image in the area A141b in B of FIG. 74 is extracted, and the image in the area A142b is displayed on the display unit 201LL. The areas A141b and A142b are larger than the areas A141a and A142a. That is, the display range of the display unit 201LL is zoomed out and expanded. At this time, for example, the display range of the display unit 201LL may be automatically moved downward so that the ground around the vehicle 1C is displayed more widely.

[0526] Although detailed description will be omitted, the display range of the display unit 201RR of the center display 201 is also changed based on the shift position of the vehicle 1C in the same manner as the display unit 201LL.

[0527] This allows the driver to check the situation diagonally to the left rear of the vehicle 1C in a wider range, thereby enabling the driver to park the vehicle 1C more safely.

[0528] Also, for example, the driver can change the display range of the display unit 201LL and the display unit 201RR by operating the operation screen displayed on the console display 202 of FIG.

[0529] On this operation screen, a button 1151, a button 1152, a touch pad 1153, a button 1154, and a button 1155 are displayed.

[0530] When the button 1151 is pressed, the display range of the left display unit 201LL becomes the setting target, and when the button 1152 is pressed, the display range of the right display unit 201RR becomes the setting target.

[0531] The display range of display unit 201LL and display unit 201CC moves when touch pad 1153 is operated. That is, the display range of display unit 201LL and display unit 201CC moves in the direction in which touch pad 1153 is pressed.

[0532] When the button 1154 is pressed, the currently set display range is stored.

[0533] When the button 1155 is pressed, the change in the display range is cancelled and the display range returns to the state before the change.

[0534] In addition, for example, by using another button (not shown) of the console display 202, the zoom factor of the display range of the display unit 201LL and the display unit 201RR is set.

[0535] Here, the display range can be set within a settable range A151 shown in Fig. 76. In Fig. 76, the same reference numerals are used for the parts corresponding to those in Fig. 72, and the description thereof will be omitted as appropriate.

[0536] For example, the size of the extraction range A152 extracted for display from the captured image is set based on the set zoom ratio. That is, the extraction range A152 is zoomed in (reduced) and zoomed out (expanded) based on the zoom ratio. Then, an image within a predetermined display range A153 within the extraction range A152 is displayed on the display unit 201LL.

[0537] At this time, the display range A153 can be moved within the settable range A151. The settable range A151 is set to a range in which the area A133L is included within the display range A153.

[0538] 76B, for example, when the area A133L protrudes from the display range A153, a warning is issued to the driver. For example, a warning message is displayed on the display unit 201LL, and a warning sound is output.

[0539] Alternatively, for example, the movement range of the display range A153 may be limited so that the display range A153 does not go outside the settable range A151 regardless of the driver's operation.

[0540] FIG. 77 shows an example of the display range of the display unit 201LL when the zoom ratio is changed from 0.8 to 1.3 in increments of 0.1.

[0541] The smaller the zoom ratio, the larger the display range becomes, making it possible to check a wider range, while the size of objects such as following vehicles displayed on the display unit 201LL becomes smaller.

[0542] On the other hand, as the zoom ratio increases, the display range becomes smaller and the size of objects such as following vehicles displayed on the display unit 201LL becomes larger. On the other hand, the number of pixels of the extracted image decreases and the clarity of the image decreases.

[0543] Therefore, for example, the settable range of the zoom ratio of the display unit 201LL on the driver's seat side is set to a range from 1.0 to 1.2, and the settable range of the zoom ratio of the display unit 201RR on the passenger's seat side is set to a range from 1.0 to 1.1.

[0544] Here, the settable range of the zoom ratio of the display unit 201LL on the driver's seat side is different from the settable range of the zoom ratio of the display unit 201RR on the passenger seat side. This is because the distance from the driver to the display unit 201LL is different from the distance to the display unit 201CC, and the direction of the driver's line of sight to the display unit 201LL is different from the direction of the driver's line of sight to the display unit 201CC. In other words, this is because the way the display unit 201LL looks to the driver is different from the way the display unit 201CC looks to the driver.

[0545] As described above, when the direction indicator is operated or when the shift position is set to reverse, the zoom ratio of the display unit 201LL and the display unit 201CC is reduced. In anticipation of this, the lower limit of the settable range of the zoom ratio is limited to 1.0. For example, when the zoom ratio is not changed as described above, the lower limit of the settable range of the zoom ratio can be reduced to 0.8 or 0.9.

[0546] In this way, the driver can move the display range A153 to a position of his / her choice within the range including the area A133L, or zoom in or out of the display range A153, by operating the console display 202. In addition, the driver can easily adjust the display range A153 without taking his / her eyes off the display range A153, for example, by moving or clicking a finger on the console display 202 while remaining in a driving position.

[0547] In addition, for example, in the case of a physical side mirror, the driver can change the range of what is visible through the side mirror by moving his / her face closer to the side mirror and looking into the side mirror. For example, the driver may look into the side mirror when parking the vehicle to check whether or not there is an obstacle on the ground around the vehicle.

[0548] In response to this, the display range of the display unit 201LL can also be changed by the driver moving his / her face closer to the display unit 201LL and peering into it.

[0549] For example, the DMS 30 detects the position and orientation of the driver's head by performing skeletal recognition of the driver based on the image of the ToF camera P101L. When the DMS 30 recognizes the driver's action of peering into the display unit 201LL based on the position and orientation of the driver's head, it detects the angle at which the driver peers into the display unit 201LL.

[0550] On the other hand, when the vehicle 1C is stopped, that is, when the speed of the vehicle 1C is 0, the display control unit controls the display range of the display unit 201LL based on the angle at which the driver looks into the display unit 201LL.

[0551] For example, Fig. 78 shows the display range of the display unit 201LL before the driver looks inside. In this example, a part of the body of the vehicle 1C is displayed in the lower right corner of the display unit 201LL. Also, the upper part of the object 1161 located to the left rear of the vehicle 1C is displayed.

[0552] In this case, for example, when the driver peers at the display unit 201LL from diagonally above, the display range of the display unit 201LL changes as shown in Fig. 79. Specifically, the display range of the display unit 201LL moves downward. As a result, the lower part of the object 1161 is also displayed on the display unit 201LL, and the driver can visually recognize the entire object 1161.

[0553] In addition, as shown by the diagonal lines in the figure, a line 1162, which is a band-like visual effect indicating an area of ​​the ground within a predetermined range around the vehicle 1C, is displayed superimposed and blinks. This allows the driver to quickly recognize objects that the vehicle 1C may collide with or come into contact with, and to move the vehicle 1C while avoiding the recognized objects.

[0554] Also, for example, Fig. 80 shows the display range of the display unit 201LL before the driver looks inside. In this example, a part of the body of the vehicle 1C is displayed at the right end of the display unit 201LL. A part of a vehicle 1181 at the left rear is displayed at the upper left of the display unit 201LL. A parking line 1182 is displayed between the vehicles 1C and 1181. The upper end of an object 1183 is displayed to the left of the vehicle 1C, at the lower end of the display unit 201LL.

[0555] Here, for example, when the driver sets the shift position to reverse in order to park the vehicle 1C, the display range of the display unit 201LL zooms out as described above with reference to Fig. 74. This causes the display range of the display unit 201LL to expand, and the range in which the vehicle 1C, the vehicle 1181, and the object 1183 are displayed to expand, as shown in Fig. 81.

[0556] Furthermore, for example, when the driver peers at the display unit 201LL from diagonally above, the display range of the display unit 201LL moves downward as shown in Fig. 82. This causes the entire object 1183 to be displayed on the display unit 201LL. As a result, the driver can park the vehicle 1C while reliably avoiding the object 1183.

[0557] Although a detailed description is omitted, when the driver looks into the display unit 201RR of the center display 201, the display range also moves in the same manner as the display unit 201LL.

[0558] Furthermore, for example, when the driver approaches the digital rear-view mirror 204 and looks into it, the display range of the digital rear-view mirror 204 also moves.

[0559] For example, when the DMS 30 recognizes an action of the driver peering into the digital rear-view mirror 204 based on the position and direction of the driver's head, it detects the angle at which the driver peers into the digital rear-view mirror 204.

[0560] On the other hand, when the vehicle 1C is stopped, that is, when the speed of the vehicle 1C is 0, the display control unit controls the display range of the digital rear-view mirror 204 based on the angle at which the driver looks into the digital rear-view mirror 204.

[0561] Furthermore, for example, as shown in Fig. 83, when the driver DR1 is recognized to be looking down at his / her feet, the light control unit 85 may turn on foot lights (not shown) that illuminate the driver's feet. This allows the driver to easily find the keys or the like that he / she has dropped, for example, at his / her feet.

[0562] Also, for example, the DMS30 detects the movement of the driver's left hand by performing skeleton recognition of the driver based on the image of the ToF camera P101L. The recognition unit 73 detects an object around the door 611FL of the driver's seat 101 outside the vehicle 1C based on sensing data such as the image of the camera 51SL. Then, when an object is detected within a predetermined range around the door 611FL of the driver's seat 101, the display unit 201LL displays a warning when the driver performs an action to open the door 611FL, for example, when the driver's left hand touches the door opener 661FL of the door 611FL.

[0563] For example, Fig. 84 shows an example of the display on the display unit 201LL before the driver's left hand touches the door opener 661FL. A part of the body of the vehicle 1C is displayed on the right end of the display unit 201LL. A pedestrian 1201 approaching the vehicle 1C from behind is displayed on the left side of the vehicle 1C.

[0564] In this case, for example, as shown in Fig. 85, when it is recognized that the left hand of the driver DR1 touches the door opener 661FL, the display unit 201LL displays a warning as indicated by diagonal lines in Fig. 86. Specifically, a bar 1202, which is a vertically long band-like visual effect, is displayed overlappingly at the left end of the display unit 201LL and blinks. The color of the bar 1202 is a conspicuous color, for example, yellow.

[0565] At this time, a warning sound may be output.

[0566] This alerts the driver DR1 to the pedestrian 1201, and prevents the door 611FL from hitting the pedestrian 1201 when the door 611FL is opened.

[0567] At this time, for example, the body system control section 84 may increase the torque of the door 611FL to make it difficult to open the door 611FL.

[0568] Also, for example, when the hand of a passenger in the passenger seat 102 touches the door opener 661FR of the door 611FR, the display unit 201RR on the passenger seat 102 side may display a similar warning.

[0569] <Example of Operation of Display Unit 201CL and Illumination 1002> Next, an example of the operation of the display unit 201CL of the center display 201 and the illumination 1002 of the steering wheel 1001 will be described with reference to FIGS.

[0570] For example, the display control unit causes the display unit 201CL to display content according to the situation based on at least one of the situation of the vehicle 1C, the situation around the vehicle 1C, and the situation of the passenger.

[0571] For example, the light control unit 85 turns on or blinks the illumination 1002 in a pattern according to the situation based on at least one of the situation of the vehicle 1C, the situation around the vehicle 1C, and the situation of the passenger. The light emission pattern of the illumination 1002 is defined by at least one of the color, brightness, blinking pattern, light movement, and light-emitting area, for example.

[0572] For example, when a driver gets into the vehicle 1C and sits in the driver's seat 101, the DMS 30 performs a driver recognition process based on at least one of an image from the ToF camera P101L and an image from the camera 1042 of the camera module 1007. Then, the display unit 201CL and the illumination 1002 notify that the driver has been recognized.

[0573] For example, FIG. 87 shows the state of the display unit 201CL and the illumination 1002 before the driver recognizes the state.

[0574] The display unit 201CL displays the state of the vehicle 1C, specifically, the shift position, the charge amount, the speed, the cruising distance, and the like.

[0575] Illumination 1002 is turned off.

[0576] FIG. 88 shows the state of the display unit 201CL and the illumination 1002 when the driver is recognized.

[0577] The display unit 201CL displays the image and name of the recognized driver, and a message to the driver. After that, the display unit 201CL returns to the state shown in FIG.

[0578] Although detailed illustration is omitted, the illumination 1002 lights up in a predetermined pattern. For example, multiple short white light bands move in a flowing manner over the illumination 1002 in a predetermined pattern.

[0579] This ensures that the driver is notified that the vehicle 1C has recognized the driver.

[0580] At this time, for example, the body system control unit 84 may set the position and angle of the driver's seat 101 and the position and angle of the steering wheel 1001 based on the recognized driver's physique or the driver's preferences set in advance, etc. In this way, a driving position suitable for the recognized driver is set.

[0581] Also, for example, the display control unit may adjust the display position of the display unit 201CL of the center display 201 based on the recognized driver's eye position and the position and angle of the steering wheel 1001. In this way, the display range of the display unit 201CL is set to a position that is easy to see without being obstructed by the steering wheel 1001, according to the recognized driver.

[0582] Furthermore, for example, when the vehicle 1C is ready for automatic driving, the display unit 201CL and the illumination 1002 notify that preparation for automatic driving is complete.

[0583] FIG. 89 shows the state of the display unit 201CL and the illumination 1002 when preparations for automatic driving are complete.

[0584] 87, an icon 1311 indicating that preparation for autonomous driving is complete is displayed on the display unit 201CL. Note that, to the left of the icon 1311, the speed limit of the road on which the vehicle 1 is traveling is displayed.

[0585] Although detailed illustration is omitted, the illumination 1002 lights up in a predetermined pattern. For example, multiple short white light bands move in a flowing manner over the illumination 1002 in a predetermined pattern.

[0586] This allows the driver to be assured that the vehicle is ready for autonomous driving.

[0587] Furthermore, for example, when the autonomous driving of the vehicle 1C is started, the display unit 201CL and the illumination 1002 notify the start of the autonomous driving.

[0588] FIG. 90 shows the state of the display unit 201CL and the illumination 1002 when the autonomous driving starts.

[0589] The display unit 201CL displays, for example, an image of the front of the vehicle 1C captured by the camera 51FC. The display unit 201CL displays, for example, the shift position, the charge amount, the speed, the cruising distance, the speed limit of the road on which the vehicle is traveling, fuel efficiency, and the like. The display unit 201CL displays, to the right of the above-mentioned icon 1311, an icon 1312 indicating that the vehicle is in autonomous driving mode. Also, for example, the color of the icon 1311 changes as autonomous driving begins.

[0590] Although detailed illustration is omitted, the illumination 1002 lights up in a predetermined pattern. For example, multiple short white light bands move in a flowing manner in a predetermined pattern on the illumination 1002 and gradually change to blue. Eventually, the entire illumination 1002 lights up in blue, and the blue light remains on during autonomous driving.

[0591] This allows the driver to be sure that automatic driving has started. In addition, by turning on illumination 1002, passengers other than the driver can also be sure that automatic driving has started.

[0592] FIG. 91 shows the state of the display unit 201CL and the illumination 1002 when the vehicle 1C changes lanes to the right lane during autonomous driving.

[0593] When the vehicle 1C changes lanes, the display unit 201CL displays guidance information 1313 including animation for notifying the driver of the implementation and direction of the lane change, superimposed on an image of the area ahead of the vehicle 1C.

[0594] Also, the blue light of the entire illumination 1002 goes out, and a predetermined blinking range A201 on the right side of the illumination 1002 blinks blue. Also, when the steering wheel 1001 is rotated clockwise to change lanes, the blinking range A201 rotates in the opposite direction (counterclockwise) in accordance with the rotation angle of the steering wheel 1001. Also, when the steering wheel 1001 is rotated counterclockwise to return to its original state after changing lanes, the blinking range A201 rotates in the opposite direction (clockwise) in accordance with the rotation angle of the steering wheel 1001. As a result, the blinking range A201 does not rotate together with the steering wheel 1001, but is maintained in a fixed position as viewed from the outside. Then, after the rotation angle of the steering wheel 1001 returns to 0 degrees, the entire illumination 1002 lights up blue, as before the lane change.

[0595] This allows the driver to know in advance that the vehicle 1C will change lanes.

[0596] Similarly, when the vehicle 1C changes lanes to the left lane, the guidance information 1313 is displayed superimposed on the display unit 201CL, and a predetermined blinking range on the left side of the illumination 1002 blinks in blue.

[0597] In addition, for example, the same process is performed when the vehicle 1C changes direction (for example, turns left or right). For example, guidance information including an animation for notifying the driver of the change of direction and the direction is displayed superimposed on the display unit 201CL, and a predetermined blinking range of the illumination 1002 corresponding to the direction of the change of direction blinks in blue.

[0598] FIG. 92 shows the state of the display unit 201CL and the illumination 1002 when a dangerous situation is detected around the vehicle 1C.

[0599] On the display section 201CL, for example, a message 1321 and an icon 1322 indicating a detected state and a countermeasure method are displayed.

[0600] Furthermore, the blue light of the entire illumination 1002 goes out, and a predetermined blinking range A211 on the upper side of the illumination 1002 blinks. For example, the color of light and the blinking speed of the blinking range A211 change based on the danger level of the detected condition. For example, when the danger level is high, the blinking range A211 blinks red, and when the danger level is low, the blinking range A211 blinks yellow. Furthermore, the higher the danger level, the faster the blinking speed becomes, and the lower the danger level, the slower the blinking speed becomes.

[0601] Also, for example, a warning sound is output in accordance with the blinking of the blinking range A211.

[0602] Also, for example, the DMS 30 detects the positions of the driver's left and right hands by performing skeletal recognition of the driver based on an image captured by the ToF camera P101L. Then, the DMS 30 detects the position where the fingers of the driver's left hand touch the operation unit 1003 of the steering wheel 1001 and the position where the fingers of the driver's right hand touch the operation unit 1004 of the steering wheel 1001.

[0603] For example, when the driver's left hand fingers touch the operation unit 1003, or when the driver's right hand fingers touch the operation unit 1004, the display control unit displays guidance on the display unit 201CL indicating the function of the button touched by the finger, the operation method, etc.

[0604] For example, as shown in Fig. 93, when the thumb of the left hand of the driver DR1 touches the button at the left end of the operation unit 1003, guidance information 1341 is displayed near the center and bottom end in the left-right direction of the display unit 201CL as shown in Fig. 94. The guidance information 1341 indicates the function and operation method of the button touched by the finger of the driver DR1.

[0605] In FIG. 94, the hands of the driver DR1 and the like are omitted for ease of understanding.

[0606] Also, similar guidance information may be displayed on the display 203 of the head-up display.

[0607] This allows the driver DR1 to understand the function and operation method of the button he or she is trying to operate, without significantly moving his or her line of sight in the direction of the operation unit 1003 or the operation unit 1004. For example, the driver DR1 can understand the function and operation method of the button he or she is trying to operate even in a dark state such as at night. For example, even if the driver DR1 is not accustomed to driving the vehicle 1C, he or she can easily understand the function and operation method of the button he or she is trying to operate.

[0608] FIG. 95 shows the state of the display unit 201CL and the illumination 1002 when the DMS 30 is performing at least one of voice recognition and lip reading.

[0609] In this case, an animation 1361 is displayed on the display unit 201CL, which indicates that at least one of voice recognition and lip reading is being performed.

[0610] Although detailed illustration is omitted, the illumination 1002 expresses the movement of light linked to the movement of the animation 1361.

[0611] For example, an animation similar to the animation 1361 may be displayed on the display unit 201CC of the center display 201.

[0612] This ensures that passengers, such as the driver, are notified that at least one of voice recognition and lip reading is being performed, which prevents passengers from suspecting that data, such as conversations, is being collected without their permission.

[0613] <Example of voice control> Next, an example of voice control of the vehicle 1C will be described with reference to Figs.

[0614] As described above, the vehicle 1C can realize 360-degree real audio. The driver's seat 101, the passenger seat 102, the rear seat 103L, and the rear seat 103R are each provided with a seat speaker 1021, a seat speaker 1022, a seat speaker 1023L, and a seat speaker 1023R. Therefore, the audio control unit, which is a part of the function of the HMI 31, can freely control the sound image, sound field, volume, etc. for the passengers in each seat individually.

[0615] The DMS 30 can recognize the shape and position of the driver's ears, for example, based on an image of the ToF camera P101L. Therefore, the audio control unit can appropriately set the sound image, sound field, volume, etc. for the driver by controlling the seat speaker 1021 of the driver's seat 101, etc., based on the position and shape of the driver's ears.

[0616] Similarly, the DMS 30 can recognize the shape and position of both ears of the passenger in the passenger seat 102, for example, based on an image of the ToF camera P101R. Therefore, the audio control unit can appropriately set the sound image, sound field, volume, etc. for the passenger by controlling the seat speaker 1022 of the passenger seat 102, etc., based on the position and shape of the passenger's both ears.

[0617] Similarly, the DMS 30 can recognize the shape and position of both ears of the passenger in the rear seat 103L based on, for example, an image of the ToF camera P111L or the ToF camera P112L. Therefore, the audio control unit can appropriately set the sound image, sound field, volume, etc. for the passenger by controlling the seat speaker 1023L of the rear seat 103L based on the position and shape of both ears of the passenger.

[0618] Similarly, the DMS 30 can recognize the shape and position of both ears of the passenger in the rear seat 103R based on, for example, an image of the ToF camera P111R or the ToF camera P112R. Therefore, the audio control unit can appropriately set the sound image, sound field, volume, etc. for the passenger by controlling the seat speaker 1023R of the rear seat 103R based on the position and shape of both ears of the passenger.

[0619] In this way, the vehicle 1C can output sound to each passenger individually and can individually control the sound image, sound field, volume, etc. By using this characteristic, for example, the following functions can be realized.

[0620] FIG. 96 shows an example of a method for controlling a warning sound when a danger such as an obstacle exists ahead of the vehicle 1. In FIG.

[0621] Sound fields A301L and A301R are schematic representations of sound fields of warning sounds for the driver in the driver's seat 101. Specifically, the triangles representing sound fields A301L and A301R indicate the position of a virtual sound source and the spread of the sound field. Specifically, the apex of the triangle indicated by a black circle indicates the position of the virtual sound source. Two sides extending from the apex indicating the virtual sound source indicate the direction in which the sound field spreads.

[0622] The sound fields A301L and A301R are represented by cones in a three-dimensional space. Therefore, the sound fields A301L and A301R spread in a cone shape in the direction in which the sides of the triangle point from the virtual sound source.

[0623] For example, first, sound field A301L is set to face the right side near the driver's left ear, and sound field A301R is set to face the left side near the driver's right ear, so that the driver hears the warning sound at both the left and right ears.

[0624] Sound field A301L then rotates clockwise around the driver's face until it is directly in front of the driver, as shown by the arrow, and sound field A301R similarly rotates counterclockwise around the driver's face until it is directly in front of the driver, as shown by the arrow.

[0625] Therefore, the driver feels that the left and right warning sounds rotate around the face and are finally integrated into one, localized in front of the driver, after which a warning message such as "Look ahead" is output from directly in front of the driver.

[0626] If the direction in which the driver's attention is called (for example, the direction in which an obstacle exists) is other than the front direction, the warning sound is localized in the direction in which the driver's attention is called. Therefore, the driver feels that the warning sound is moving and localized in the direction in which the driver's attention is called.

[0627] This alerts the driver to dangers such as obstacles, and enables the driver to reliably recognize the direction in which the danger exists.

[0628] FIG. 97 shows an example of a method for controlling sound effects that are output when autonomous driving starts.

[0629] First, sound effect field A302 is set to face the driver from in front of him / her at the height of his / her ears. Then, sound field A302 is divided into left and right, one of which rotates counterclockwise around the driver's face to directly behind the driver, and the other rotates clockwise around the driver's face to directly behind the driver.

[0630] Therefore, the driver feels that the sound effect is output from the front, then splits into two parts, rotates around the face, and finally merges into one and is localized behind the driver, effectively creating the illusion of autonomous driving starting.

[0631] FIG. 98 shows an example of a method for controlling sound effects that are output when the vehicle 1C accelerates during autonomous driving.

[0632] First, the sound effect sound field A303 is set to face the right at the driver's left ear, then the sound field A303 rotates clockwise around the driver's face to a position facing the left at the driver's right ear.

[0633] Therefore, the driver feels as if the sound effect is rotating clockwise in front of his face from his left ear to his right ear.

[0634] FIG. 99 shows an example of a method for controlling sound effects that are output when the vehicle 1C decelerates during autonomous driving.

[0635] First, sound effect field A304 is set to face leftward near the driver's right ear, then sound field A304 rotates counterclockwise around the driver's face to a position facing rightward near the driver's left ear.

[0636] Therefore, the driver feels as if the sound effect is rotating counterclockwise in front of his face from his right ear to his left ear.

[0637] As a result, the driver can intuitively recognize the acceleration and deceleration of the vehicle 1C during automatic driving based on the direction in which the sound effect moves.

[0638] For example, sound effects for set values ​​other than the speed of the vehicle 1C may also be controlled as shown in Figures 98 and 99. For example, when the set value increases, sound effects may be output as shown in Figure 98, and when the set value decreases, sound effects may be output as shown in Figure 99.

[0639] FIG. 100 shows an example of a method for controlling a sound effect that is output before output of a voice message such as navigation.

[0640] First, the sound effect field A305 is set to face the driver from the left rear of the driver at the height of the driver's ears. Then, the sound field A305 rotates clockwise around the driver's head to a position facing the driver from the right front of the driver.

[0641] Therefore, the driver feels as if the sound effect is rotating counterclockwise around the back of the driver's head from the left rear diagonal to the right front diagonal around the head. By rotating behind the driver's head in this way, the driver is more likely to notice the sound effect than if the sound effect were rotating in front of the driver's head. This reduces the number of cases where the driver misses the voice message that follows the sound effect.

[0642] 101 to 104 show an example of a method for controlling a warning sound that alerts the driver to a vehicle 1401 approaching from the rear left.

[0643] For example, when the left turn signal of the vehicle 1401 is on, and the distance between the vehicle 1401 and the vehicle 1C falls within a predetermined threshold, output of a warning sound is started. Fig. 101 shows a sound field A306 at the start of output of a warning sound. The direction of the sound field A306 is set to a direction substantially equal to the direction in which the head of the driver of the vehicle 1C is seen from the vehicle 1401. In addition, a virtual sound source of the warning sound is set at a position a predetermined distance away from the driver's head in the direction of the vehicle 1401.

[0644] 102 to 104, as the vehicle 1401 approaches the vehicle 1C, the sound field A306 moves. Specifically, the sound field A306 moves so as to always face in a direction substantially equal to the direction in which the head of the driver of the vehicle 1C is viewed from the vehicle 1401. Also, the sound field A306 moves so that the distance between the virtual sound source of the warning sound and the head of the driver is substantially proportional to the distance between the vehicle 1401 and the vehicle 1C.

[0645] Therefore, the driver can recognize the approach of the vehicle 1401 by the warning sound. In addition, the driver can intuitively recognize the speed, distance, and direction of the vehicle 1401 by the movement of the warning sound.

[0646] Also, by using the seat speaker of each seat, sound can be output individually so that only the passenger in each seat can hear it. For example, as shown in Fig. 105, sound can be output using the seat speaker 1023L so that only the passenger sitting in the rear seat 103L can hear it.

[0647] By using this function, for example, passengers in each seat can make voice or video calls to the outside without being heard by other passengers. Also, for example, passengers in the vehicle 1C can have voice chats with each other.

[0648] <Examples of lip reading> As described above, the DMS30 performs lip reading of the driver based on the image of the camera 1042 of the camera module 1007. As described above, the DMS30 performs lip reading of the passenger in the back seat 103L based on the image of the ToF camera P112L. The DMS30 performs lip reading of the passenger in the back seat 103R based on the image of the ToF camera P112R. For example, the DMS30 can also perform lip reading of the passenger in the passenger seat 102 using the image of the ToF camera P101R.

[0649] For example, the DMS 30 may use lip reading to aid in recognizing the speech of each seat occupant, thereby improving the accuracy of speech recognition, or, for example, the DMS 30 may use only lip reading to recognize the speech of each seat occupant.

[0650] This enables the DMS30 to recognize with high accuracy the speech content of passengers in each seat, for example, when it is noisy inside the vehicle 1C and it is difficult to collect voice using a microphone, or when passengers cannot speak loudly because a child is sleeping, etc.

[0651] Also, for example, the HMI 31 can convert the recognition result of the speech content of a passenger in a certain seat into voice and output it from the seat speaker of the other seat, thereby conveying the speech content to the passenger in the other seat. For example, the HMI 31 can convert the recognition result of the speech content of a passenger in a certain seat into text and display it on a display in front of the other seat, thereby conveying the speech content to the passenger in the other seat. This enables communication between passengers in each seat even when it is noisy inside the vehicle 1C and voices are difficult to hear or when it is difficult to speak loudly.

[0652] Also, for example, the driver can operate the vehicle 1C without speaking by using lip reading. Specifically, for example, the driver can operate the vehicle 1C by pressing a predetermined button on the operation unit 1003 or the operation unit 1004 of the steering wheel 1001 and moving his / her mouth in the same manner as when inputting a predetermined voice command. At this time, the driver may or may not speak aloud.

[0653] More specifically, for example, the driver can set the air conditioning in the vehicle 1C by making mouth movements to voice commands such as "Keep my hands cool," "Keep my feet cool," and "Get some fresh air." For example, the driver can control music playback by making mouth movements to voice commands such as "Play a song that wakes me up," "Play my usual song," and "Play a song that everyone can sing along to." For example, the driver can give instructions such as searching for a destination to a navigation application running in the vehicle 1C by making mouth movements to voice commands such as "Tell me about restaurants around here," "Find a cafe around here," and "Tell me about nearby charging spots."

[0654] This allows the driver to operate the vehicle 1C using voice commands even when it is noisy inside the vehicle 1C or when the driver cannot speak loudly.

[0655] <Example of gesture input> For example, the DMS 30 can recognize the gestures of the passengers in each seat based on the images of the ToF cameras provided for each seat of the vehicle 1C. This allows each passenger to operate the vehicle 1C using gestures.

[0656] For example, a passenger can increase the volume of the sound of the content being played in the vehicle 1C by pointing at the air and rotating his / her index finger clockwise. On the other hand, a passenger can decrease the volume of the sound of the content being played in the vehicle 1C by pointing at the air and rotating his / her index finger counterclockwise. For example, a passenger can mute the volume of the sound of the content being played in the vehicle 1C by holding his / her index finger up and bringing it close to his / her lips.

[0657] For example, a passenger can operate a touch panel without touching the touch panel provided on the display by pointing at the display in front of the seat with his / her index finger and moving the index finger. For example, a driver can start a navigation application by pointing at the display unit 201CC of the center display 201. Also, for example, a driver can display information on the pointed position or set the pointed position as a destination by pointing at a map in the navigation application.

[0658] <<7. Modification of the Third Embodiment>> A modification of the above-described third embodiment of the present technology will be described below.

[0659] In the above description, an example in which the vehicle 1C is a left-hand drive vehicle has been shown, but the present technology can of course also be applied to a right-hand drive vehicle. When the present technology is applied to a right-hand drive vehicle, the above-described interior layout is appropriately changed to suit the right-hand drive vehicle.

[0660] The present technology can also be applied to a moving device that performs automatic driving without the need for a passenger to drive. In this case, the above-mentioned distinction between the driver's seat 101 and the passenger seat 102 disappears, and the above-mentioned various displays are arranged in front of the seats in the moving device. In addition, the display range of an image displayed on the display is changed based on the line of sight, posture, etc. of a person sitting in a specific seat.

[0661] Furthermore, the type of vehicle to which the present technology can be applied is not particularly limited. In addition, the present technology can be applied to moving devices other than vehicles, such as personal mobility devices, airplanes, ships, construction machines, and agricultural machines.

[0662] <<8.Other>> <Computer configuration example> The above-described series of processes can be executed by hardware or software. When the series of processes is executed by software, a program constituting the software is installed in a computer (e.g., the processor 21, etc.).

[0663] In addition, the program executed by the computer may be a program in which processing is performed chronologically in the order described in this specification, or it may be a program in which processing is performed in parallel or at the required timing, such as when called.

[0664] In this specification, a system refers to a collection of multiple components (devices, modules (parts), etc.), regardless of whether all the components are in the same case. Therefore, multiple devices housed in separate cases and connected via a network, and a single device in which multiple modules are housed in a single case, are both systems.

[0665] Furthermore, the embodiments of the present technology are not limited to the above-described embodiments, and various modifications are possible without departing from the gist of the present technology.

[0666] For example, the present technology can be configured as cloud computing, in which a single function is shared and processed collaboratively by multiple devices via a network.

[0667] Furthermore, each step described in the above flow chart can be executed by one device, or can be shared and executed by a plurality of devices.

[0668] Furthermore, when a single step includes multiple processes, the multiple processes included in the single step can be executed by a single device, or can be shared and executed by multiple devices.

[0669] <Examples of configuration combinations> The present technology can also be configured as follows.

[0670] (1) A first display extending in the left-right direction in front of a first seat of the mobile device is provided. Preparation, a left end portion of the first display facing diagonally rearward to the right as viewed from the first seat; a center portion of the first display faces rearward as seen from the first seat; The right end of the first display faces diagonally rearward to the left as seen from the first seat. Display system. (2) a left end portion of the first display displays an image of a left rear portion of the mobile device; The right end of the first display displays an image of the right rear of the mobile device. The display system according to (1) above. (3) The left and right ends of the first display are used as digital exterior mirrors. The display system according to (2) above. (4) a display control unit that controls display of images at the left and right ends of the first display; The display system according to (2) or (3) further comprises: (5) The display control unit controls a display range of the left end and the right end of the first display based on at least one of a line of sight direction of a person sitting in the first seat, a posture of the person, an operation of a turn signal, and a shift position. The display system according to (4) above. (6) The display control unit moves the display range of one of the left end and the right end of the first display into which the person looks, based on an angle into which the person looks, when the moving device is stopped. The display system according to (5) above. (7) The display control unit displays a visual effect indicating a predetermined range around the mobile device by superimposing it on an image of one of a left end portion and a right end portion of the first display which the person looks into. The display system according to (6) above. (8) The display control unit zooms out the display range in a direction in which the turn indicator is turned on, out of the left end and the right end of the first display. A display system according to any one of (5) to (7). (9) The display control unit zooms out the display range when the shift position is set to reverse. A display system according to any one of (5) to (8). (10) The display control unit causes a visual effect based on a recognition result of an object diagonally rear to the left of the mobile device to be superimposed on an image at a left end of the first display, and causes a visual effect based on a recognition result of an object diagonally rear to the right of the mobile device to be superimposed on an image at a right end of the first display. A display system according to any one of (5) to (9). (11) The display control unit changes a display mode of the visual effect based on a degree of danger to the object. The display system according to (10) above. (12) The display control unit changes a display mode of the visual effect at the left end of the first display when a left direction indicator is turned on, and changes a display mode of the visual effect at the right end of the first display when a right direction indicator is turned on. The display system according to (11) above. (13) a recognition unit that recognizes an object diagonally to the left rear of the mobile device based on an image of the left rear of the mobile device, and recognizes an object diagonally to the right rear of the mobile device based on an image of the right rear of the mobile device; The display system according to any one of (10) to (12) further comprises: (14) A central portion of the first display displays an image of the surroundings of the mobile device. A display system according to any one of (1) to (13). (15) a display control unit that controls a display range of an image displayed in a central portion of the first display based on at least one of a line of sight direction and a posture of a person sitting in the first seat; The display system according to (14) further comprises: (16) A central portion of the first display displays information to assist in driving the mobile device. A display system according to any one of (1) to (15). (17) A central portion of the first display extends in the left-right direction in front of the first seat and a second seat arranged beside the first seat, and can be divided into a first display area in front of the first seat, a second display area in front of the second seat, and a third display area between the first display area and the second display area. A display system according to any one of (1) to (16). (18) the first display area displays information to assist in driving the mobile device; The second display area and the third display area display infotainment-related information. The display system according to (17) above. (18A) The center of the first display faces diagonally upward. A display system according to any one of (1) to (17). (19) The left and right ends of the first display are bent from the center of the first display toward the inside of the mobile device. A display system according to any one of (1) to (18A). (20) A second display is provided below the first display and faces rearward as viewed from the first seat. The display system according to any one of (1) to (19) further comprises: (twenty one) The second display is configured with a two-dimensional or three-dimensional touch panel, and displays an operation screen for the information displayed on the first display. The display system according to (20) above. (21A) The second display displays an operation screen of an air conditioner of the mobile device. The display system according to (20) or (21) above. (21B) The second display faces diagonally upward. A display system according to any one of (20) to (21A). (21C) The center of the first display faces obliquely upward at approximately the same angle as the second display. The display system described in (21B). (twenty two) The second display is provided in a console between the first seat and a second seat arranged next to the first seat. A display system according to any one of (20) to (21C). (twenty three) a display control unit that controls display of images at the left and right ends of the first display; In addition, a left end portion of the first display displays an image of a left rear portion of the mobile device; a right end portion of the first display displays an image of a right rear portion of the mobile device; the second display is configured with a two-dimensional or three-dimensional touch panel and displays the operation screen for setting a display range of a left end portion and a right end portion of the first display; The display control unit changes the display range based on an operation on the operation screen. A display system according to any one of (20) to (22). (twenty four) The display control unit changes the display range of the left end portion of the first display within a range including a predetermined first area diagonally to the left rear of the mobile device, and changes the display range of the right end portion of the first display within a range including a predetermined second area diagonally to the right rear of the mobile device, based on an operation on the operation screen. The display system according to (23). (twenty five) The display control unit causes the range in which the display range can be zoomed out and zoomed in to differ between the left end and the right end of the first display. The display system according to (23) or (24). (26) a third display provided in front of the first seat and displaying visual information superimposed on the visual field of the person sitting in the first seat; The display system according to any one of (1) to (25) further comprises: (27) The third display displays information to assist in driving the mobile device. The display system according to (26) above. (28) a fourth display provided above the first display and facing rearward as viewed from the first seat; The display system according to any one of (1) to (27) further comprises: (29) The fourth display is used as a digital rear-view mirror. The display system according to (28). (30) a display control unit that controls a display range of an image to be displayed on the fourth display based on at least one of a line of sight direction and a posture of a person sitting in the first seat; The display system according to (29) further comprises: (31) A fifth display provided at the center of the steering wheel of the mobile device. The display system according to any one of (1) to (30) further comprises: (31A) The fifth display displays a message for a person sitting in the first seat. The display system according to (31). (32) a sixth display provided on the back of at least one of the first seat and a second seat arranged next to the first seat; The display system according to any one of (1) to (31A) further comprises: (33) a left end portion of the first display facing diagonally rearward to the right of the mobile device; a central portion of the first display facing toward the rear of the mobile device; The right end of the first display faces diagonally left rear of the mobile device. A display system according to any one of (1) to (32). (34) The first seat is a driver's seat. A display system according to any one of (1) to (33). (35) A display extending in the left-right direction in front of the seat of the mobile device is provided. Preparation, a left end of the display facing diagonally rearward to the right as viewed from the seat; a center portion of the display faces rearward as viewed from the seat; The right end of the display faces diagonally rearward to the left as seen from the seat. Display device. (36) The display extends in the left-right direction in front of the seat of the mobile device, with the left end facing diagonally rear to the right as viewed from the seat, the center facing backward as viewed from the seat, and the right end facing diagonally rear to the left as viewed from the seat. The left end of the display displays an image of the left diagonal rear of the mobile device, and the right end of the display displays an image of the right diagonal rear of the mobile device. Display method. (37) A display that extends in the left-right direction in front of a seat, with the left end facing diagonally rear to the right as viewed from the seat, the center facing rear as viewed from the seat, and the right end facing diagonally rear to the left as viewed from the seat. Equipped with mobility equipment. (38) a metadata adding unit that adds metadata to video data captured while moving; an image processing unit that edits the captured video data based on the metadata and generates edited video data; A mobile device comprising: (39) The image processing unit extracts frames from the plurality of pieces of captured video data captured by a plurality of cameras based on the metadata, and combines the frames to generate the edited video data. The moving device according to (38). (40) The image processing unit extracts frames from the plurality of captured video data based on the route of the mobile device and the metadata. The moving device according to (39). (41) The captured video data is video data captured around the mobile device or inside a room. The moving device according to (39) or (40). (42) The image processing unit extracts frames of the captured video data captured in the room based on the activity level in the room. The moving device according to (41). (43) The image processing unit superimposes visual information on a frame of the captured video data based on the metadata. The moving device according to any one of (38) to (42). (44) The image processing unit edits the captured video data based on a user's instruction. The moving device according to any one of (38) to (43). (45) The metadata adding unit adds the metadata related to one or more of a shooting location, a shooting date and time, a status of the mobile device, a status inside the mobile device, and a status around the mobile device to the shot video data. The moving device according to any one of (38) to (44). (46) The metadata adding unit adds the metadata input by a user to the captured video data. The moving device according to any one of (38) to (45). (47) A camera is installed in the rearview mirror or around the rearview mirror to capture the view toward the driver's seat. Equipped with mobility equipment. (48) The rearview mirror is installed near the ceiling diagonally in front of the driver's seat. The moving device according to (47). (49) The camera is installed on the ceiling around a position where the rearview mirror is attached. The moving device according to (48). (50) The camera is installed closer to the driver's seat than the rearview mirror. The moving device according to (48) or (49). (51) A first recognition unit that performs at least one of a driver's line of sight recognition, a driver's posture recognition, a driver's action recognition, and a driver's personal recognition based on an image captured by the camera. The moving device according to any one of (47) to (50) further comprises: (52) a display control unit that controls display of a digital outer mirror of the mobile device based on at least one of the line of sight, posture, and movement of the driver recognized by the first recognition unit; The moving device according to (51) above further comprises: (53) a second recognition unit that recognizes an object around the driver's door based on sensing data around the driver's door on the outside of the mobile device; In addition, When the first recognition unit recognizes an action of the driver to open the door and the second recognition unit recognizes an object around the door, the display control unit displays a visual effect based on a recognition result of the object by superimposing it on the image of the digital outer mirror on the driver's seat side. The mobile device according to (52). (54) The display control unit moves a display range of the digital outer mirror based on an angle at which the driver looks into the digital outer mirror when the first recognition unit recognizes that the driver looks into the digital outer mirror. The moving device according to (52) or (53). (55) The display control unit, when the first recognition unit recognizes that the driver has contacted an operating object of the mobility device, displays information showing guidance regarding the operating object on a display in front of the driver's seat. The mobile device according to any one of (52) to (54). (56) The camera is a ToF camera. A mobile device according to any one of (47) to (55). (57) A rod-shaped operating object used to change the shift position, with an indicator on the side that changes color depending on the shift position that is set. Equipped with mobility equipment. (58) The setting order of the shift position by the operating body is reverse, neutral, drive, and automatic driving. The moving device according to (57). (59) The operating body is a lever that changes the shift position by being moved vertically. The moving device according to (58). (60) The operating body includes a dial that switches the shift position by rotating around an axis. The moving device according to (58). (61) A button is provided at the tip of the operating body, which can be pressed in the axial direction of the operating body and is used to set the shift position to parking. A mobile device according to any one of (58) to (60). (62) The indicator is provided circumferentially on the side surface of the operation body. A mobile device according to any one of (57) to (61). (63) The operation body is provided so as to extend in a lateral direction of the moving device from a steering column, The indicator is disposed at least one of a position visible through a gap between spokes of the steering wheel as seen from the driver's seat and a position outside the outer periphery of the steering wheel as seen from the driver's seat. A mobile device according to any one of (57) to (62). (64) A steering wheel having a ring-shaped illumination in the center, the color, brightness, and / or light-emitting area of ​​which can be changed; A light control unit that controls the illumination; A mobile device comprising: (65) The light control unit turns on or blinks the illumination based on at least one of a state of the mobile device, a state around the mobile device, and a state of a passenger. The mobile device according to (64). (66) The light control unit turns on or blinks the illumination in a pattern according to a situation. The moving device according to (65). (67) The pattern is defined by at least one of a color, a brightness, a blinking pattern, a light movement, and the light-emitting area. The mobile device according to claim 66. (68) The light control unit turns on or blinks a portion of the illumination corresponding to a direction of lane change or direction change when the moving device changes lanes or turns. The moving device according to (66) or (67). (69) The light control unit rotates the portion that turns on or blinks the illumination in a direction opposite to that of the steering wheel in accordance with a rotation angle of the steering wheel. The moving device according to (68). (70) The light control unit turns on or blinks the illumination in a predetermined pattern when at least one of voice recognition and lip reading is being performed. A mobile device according to any one of (66) to (69). (71) The central portion houses an airbag, The illumination is arranged to avoid a portion of the central portion that may crack when the airbag is deployed. A mobile device according to any one of (64) to (70). (72) A plurality of user interface devices are arranged along a loop line that surrounds the perimeter of the room in a substantially horizontal manner. Mobile device. (73) The device for the user interface includes at least one of an output device and an operation device. The mobile device according to (72). (74) The output device includes at least one of a display extending in the left-right direction in front of the mobile device and illumination provided along the periphery of the center of a steering wheel. The moving device according to (73). (75) The output device includes a speaker mounted on a door of the mobile device. A mobile device according to any one of (73) to (74). (76) The operation device includes at least one of a door opener for the door of the mobile device, a device for adjusting the direction of air conditioning, and a device used for setting headlights. The mobile device according to any one of (73) to (75). (77) A loop light is provided that is substantially parallel to the loop line and surrounds at least a portion of the periphery of the room. The moving device according to any one of (72) to (76) further comprises: (78) a light control unit that performs cooperative control of the external light of the mobile device and the loop light; The moving device according to (77) further comprises: (79) An air conditioner outlet is provided along the loop line. A mobile device according to any one of (72) to (78). (80) A plurality of seat speakers which are speakers individually provided in each seat; an output control unit that individually controls the output of sound from each of the seat speakers; A mobile device comprising: (81) Each of the seat speakers is disposed below a headrest of each of the seats. The moving device according to claim 80. (82) A plurality of line speakers are arranged along a loop line that surrounds the perimeter of the room in a substantially horizontal manner. In addition, The output control unit further controls the output of sound from the line speaker. The moving device according to (80) or (81). (83) The output control unit controls the line speakers to output sounds directed to the entire cabin, and the seat speakers to output sounds directed to each passenger sitting in each seat. The moving device according to (82). (84) The output control unit moves a sound field of a warning sound for a driver in a direction to alert the driver. A moving device as described in any one of (80) to (83). (85) The output control unit rotates a sound field of a sound effect indicating an increase in a predetermined setting value of the mobile device in a first direction around the head of a person sitting in the seat, and rotates a sound field of a sound effect indicating a decrease in the setting value in a second direction opposite to the first direction around the head of the person. A mobile device according to any one of (80) to (84). (86) The output control unit rotates a sound field of a sound effect to be output before a voice message behind the head of a person sitting in the seat. A moving device according to any one of (80) to (85). (87) The output control unit moves a sound field of a warning sound for an object behind the moving device in accordance with the movement of the object. A mobile device according to any one of (80) to (86). (88) A camera module is installed on the dashboard and captures the driver's seat from an oblique angle. Equipped with mobility equipment. (89) The camera module switches between color and infrared photography. The moving device according to (88). (90) The camera module includes a light source that outputs infrared light, and outputs the infrared light when infrared photography is performed. The moving device according to (89). (91) The camera module switches between color and infrared photography based on the surrounding brightness. The moving device according to (89) or (90). (92) The moving device according to any one of (89) to (91), further comprising a recognition unit that performs lip reading of the driver based on an image from the camera module.

[0671] It should be noted that the effects described in this specification are merely examples and are not limiting, and other effects may also be obtained. [Explanation of symbols]

[0672] 1 vehicle, 11 vehicle control system, 21 processor, 25 external recognition sensor, 26 in-vehicle sensor, 30 DMS, 31 HMI, 51 camera, 52 radar, 53 LiDAR, 73 recognition unit, 84 body system control unit, 85 light control unit, 101 driver's seat, 102 passenger seat, 103, 103L, 103R rear seat, 104 windshield, 106 steering wheel, 107 console, 108 controller, 109 operation unit, 151FL to 151BR camera, 201 center display, 201C center unit, 201CL to 201CR display unit, 201L left end, 201LL display unit, 201R right end, 201RR display unit, 202 console display, 203 display, 204 digital rear mirror, 205 steering wheel display, 206 rear entertainment display, 601 accessory light, 602 daytime running light, 603 headlight, 604 turn signal, 605 auxiliary light, 606 tail light, 607 brake light, 651 steering wheel, 652 illumination, 653FL to BR speaker, 654 loop light, 655 stalk lever, 656 button, 657 indicator, 658FC to 658FR air outlet, 659CL to 659FR knob, 660 operation unit, 661FL, 661FR door opener, 662L, 662R tablet terminal, 663 mounting unit, 701 information processing unit, 711 Image processing unit, 712, metadata assignment unit, 713, output control unit, 751, main light, 1001, steering wheel, 1002, illumination, 1003, 1004, operation unit, 1007, 1008, camera module, 1021 to 1023R, seat speaker, 1042, camera, 1043, LED

Claims

1. a metadata adding unit that adds metadata including data related to a situation inside a room to video data captured while moving; an image processing unit that edits the captured video data based on the metadata and generates edited video data; A mobile device comprising:

2. The metadata includes the activity level in the room. The mobile device of claim 1 .

3. The image processing unit extracts frames of the captured video data captured inside the room based on the activity level. The mobile device of claim 2.

4. When the image processing unit detects an increase in activity in the room based on the activity level, the image processing unit preferentially extracts frames of the captured video data that capture the room. The mobile device of claim 3.

5. The metadata assigning unit sets the activity level based on at least one of a volume of conversation in the vehicle and a movement of a passenger. The mobile device according to any one of claims 2 to 4.

6. The image processing unit extracts frames from the plurality of pieces of captured video data captured by a plurality of cameras based on the metadata, and combines the frames to generate the edited video data. A mobile device according to any one of claims 1 to 5.

7. The image processing unit extracts frames from the plurality of captured video data based on the route of the mobile device and the metadata. The mobile device of claim 6.

8. The captured video data is video data captured around the mobile device or inside a room. A mobile device according to claim 6 or 7.

9. The image processing unit superimposes visual information on a frame of the captured video data based on the metadata. A mobile device according to any one of claims 1 to 8.

10. The image processing unit edits the captured video data based on a user's instruction. A mobile device according to any one of claims 1 to 9.

11. The metadata further includes data regarding one or more of a location where the image was taken, a date and time the image was taken, a state of the mobile device, and a state of the surroundings of the mobile device. A mobile device according to any one of claims 1 to 10.

12. The metadata adding unit adds the metadata input by a user to the captured video data. A mobile device according to any one of claims 1 to 11.

13. The mobile device Adding metadata including data on indoor conditions to video data captured while moving; editing the captured video data based on the metadata to generate edited video data; An information processing method comprising:

14. A metadata adding unit adds metadata including data on the indoor situation to the video data taken while moving. Equipped with mobility equipment.

15. The metadata includes the activity level in the room.

15. The mobile device of claim 14.

16. The metadata assigning unit sets the activity level based on at least one of a volume of conversation in the vehicle and a movement of a passenger.

16. The mobile device of claim 15.

17. The metadata further includes data regarding one or more of a location where the image was taken, a date and time the image was taken, a state of the mobile device, and a state of the surroundings of the mobile device. A mobile device according to any one of claims 14 to 16.

18. The metadata adding unit adds the metadata input by a user to the captured video data. A mobile device according to any one of claims 14 to 17.

19. The mobile device Adding metadata including data on indoor conditions to video data captured while moving An information processing method comprising:

20. an image processing unit that edits the captured video data based on metadata, which is added to the captured video data captured while moving by the mobile device and includes data on a situation inside the mobile device, to generate edited video data; An information processing device.

21. The metadata includes the activity level in the room. The information processing device according to claim 20.

22. The image processing unit extracts frames of the captured video data captured inside the room based on the activity level. The information processing device according to claim 21.

23. When the image processing unit detects an increase in activity in the room based on the activity level, the image processing unit preferentially extracts frames of the captured video data that capture the interior of the room. The information processing device according to claim 22.

24. The image processing unit extracts frames from the plurality of pieces of captured video data captured by a plurality of cameras based on the metadata, and combines the frames to generate the edited video data. The information processing device according to any one of claims 20 to 23.

25. The image processing unit extracts frames from the plurality of captured video data based on the route of the mobile device and the metadata. The information processing device according to claim 24.

26. The captured video data is video data captured around the mobile device or inside a room.

26. An information processing device according to claim 24 or 25.

27. The image processing unit superimposes visual information on a frame of the captured video data based on the metadata. The information processing device according to any one of claims 20 to 26.

28. The image processing unit edits the captured video data based on a user's instruction. The information processing device according to any one of claims 20 to 27.

29. The metadata further includes data regarding one or more of a location where the image was taken, a date and time the image was taken, a state of the mobile device, and a state of the surroundings of the mobile device. The information processing device according to any one of claims 20 to 28.

30. An information processing device, Based on metadata that includes data on the indoor situation of the mobile device and that is added to the captured video data captured while moving with the mobile device, editing the captured video data to generate edited video data. An information processing method including:

Images