Display system, display device, display method, and mobile device

Abstract

The present technology relates to a display system, a display device, a display method, and a mobile device that can improve convenience for an occupant of a mobile device. The display system includes a first display that extends in a left-right direction in front of a first seat of the mobile device, in which a left end portion of the first display faces a right rear obliquely when viewed from the first seat, a central portion of the first display faces a rear when viewed from the first seat, and a right end portion of the first display faces a left rear obliquely when viewed from the first seat. The present technology can be applied to, for example, an in-vehicle display system.

Description

Technical Field

[0001] This technology relates to display systems, display devices, display methods, and mobile devices, and more specifically to display systems, display devices, display methods, and mobile devices that can improve the convenience of occupants of mobile devices. Background Technology

[0002] In recent years, the introduction of augmented reality (AR) technology into vehicles has been making progress (see, for example, Patent Document 1).

[0003] Reference List

[0004] Patent documents

[0005] Patent Document 1: WO 2019 / 044536 Summary of the Invention

[0006] The problem to be solved by the present invention

[0007] As mentioned above, it is hoped that the convenience of occupants (such as the driver of a vehicle) can be improved by using technologies such as AR.

[0008] This technology was developed in view of this situation and makes it possible to improve the convenience of occupants of mobile devices such as vehicles.

[0009] Problem Solution

[0010] The first aspect of the present technology includes a display system comprising a first display extending in a left-right direction in front of a first seat of a mobile device, wherein the left end portion of the first display faces diagonally to the right rear when viewed from the first seat, the central portion of the first display faces rearward when viewed from the first seat, and the right end portion of the first display faces diagonally to the left rearward when viewed from the first seat.

[0011] The second aspect of the technology includes a display extending in a left-right direction in front of the seat of a mobile device, wherein the left end portion of the display faces diagonally to the right rear when viewed from the seat, the central portion of the display faces rearward when viewed from the seat, and the right end portion of the display faces diagonally to the left rearward when viewed from the seat.

[0012] The third aspect of the display method of this technology includes displaying an image of the mobile device to the left rear through the left-right direction in front of the seat of the mobile device and having a left end portion facing the right rear when viewed from the seat, a central portion facing the rear when viewed from the seat, and a right end portion facing the left rear when viewed from the seat, in a display where an image of the mobile device to the left rear is displayed through the left end portion of the display and an image of the mobile device to the right rear is displayed through the right end portion of the display.

[0013] The fourth aspect of the present technology includes a display that extends in a left-right direction in front of a seat and has a left end portion facing right rearward when viewed from the seat, a central portion facing rearward when viewed from the seat, and a right end portion facing left rearward when viewed from the seat.

[0014] The mobile device of the fifth aspect of the present technology includes: a metadata adding unit that adds metadata to captured mobile image data, the captured mobile image data being captured while in motion; and an image processing unit that edits the captured mobile image data based on the metadata and generates edited mobile image data.

[0015] The mobile device of the sixth aspect of this technology includes a camera mounted on or around a rearview mirror for inspecting the rear of a vehicle and imaging the direction of the driver's seat.

[0016] The seventh aspect of the present invention includes a moving device comprising an operating body, which is a lever-shaped operating body for changing a shift position, and having an indicator disposed on the side, the color of which changes according to the set shift position.

[0017] The eighth aspect of the present technology includes a steering wheel having an annular lighting device, at least one of the color, brightness and luminous area of ​​the lighting device being variable and the lighting device being disposed in a central portion, and a lamp control unit for controlling the lighting device.

[0018] The mobile device of the ninth aspect of this technology includes a device for multiple user interfaces arranged along a generally horizontal ring around an interior periphery.

[0019] The mobile device of the tenth aspect of this technology includes a plurality of seat speakers, which are individual speakers disposed in the seat, and an output control unit for individually controlling the output of sound from each seat speaker.

[0020] The eleventh aspect of this technology includes a mobile device that is mounted on the dashboard and captures images from the driver's seat direction from a diagonal front.

[0021] In a first aspect of the present technology, the left end portion of a first display extending in the left-right direction in front of the first seat of the mobile device faces the right rearward when viewed from the first seat, the central portion of the first display faces the rearward when viewed from the first seat, and the right end portion of the first display faces the left rearward when viewed from the first seat.

[0022] In a second or fourth aspect of this technology, the left end portion of a display extending in the left-right direction in front of the seat of the mobile device faces the right rearward when viewed from the seat, the central portion of the display faces the rearward when viewed from the seat, and the right end portion faces the left rearward when viewed from the seat.

[0023] In a third aspect of this technology, a display extending in the left-right direction in front of the seat of the mobile device and having a left end portion facing right rearward when viewed from the seat, a central portion facing rearward when viewed from the seat, and a right end portion facing left rearward when viewed from the seat, displays an image of the mobile device from the left rearward through the left end portion of the display and displays an image of the mobile device from the right rearward through the right end portion of the display.

[0024] In a fifth aspect of this technology, metadata is added to captured moving image data, which is captured while in motion, the captured moving image data is edited based on the metadata, and edited moving image data is generated.

[0025] In a sixth aspect of this technology, the image in the driver's seat direction is captured by a rearview mirror used for checking the rear or by a camera mounted around the rearview mirror.

[0026] In a seventh aspect of this technology, the color of an indicator provided on the side of a lever-shaped operating body for changing the shift position changes according to the set shift position.

[0027] In an eighth aspect of the present technology, a lighting device arranged in a ring in the central portion of a steering wheel is controlled, wherein at least one of the color, brightness, and luminous area of ​​the lighting device is variable.

[0028] In a ninth aspect of this technology, devices for multiple user interfaces are arranged along a generally horizontal ring around the perimeter of the interior.

[0029] In a tenth aspect of this technology, sound is individually controlled from the output of a plurality of seat speakers, which are individually installed as loudspeakers in the seat.

[0030] In the eleventh aspect of this technology, an image in the direction of the driver's seat is captured from the oblique front by a camera module mounted on the dashboard. Attached Figure Description

[0031] Figure 1 This is a block diagram illustrating an example configuration of a vehicle control system.

[0032] Figure 2 This is a view illustrating an example of the sensing area.

[0033] Figure 3This is a view illustrating an example of the installation locations of a camera, LiDAR, and radar.

[0034] Figure 4 This is a schematic diagram of the front of the vehicle.

[0035] Figure 5 This is a diagram showing the perimeter of the driver's seat and passenger seat when viewed from above.

[0036] Figure 6 This is a diagram of the passenger seats when viewed from the left.

[0037] Figure 7 This is a diagram showing the area around the driver's seat and passenger seat when viewed from the left rear.

[0038] Figure 8 This is a diagram of the front of the vehicle as viewed from the driver's seat.

[0039] Figure 9 This is a diagram of the passenger seats when viewed from the left rear.

[0040] Figure 10 This is a view illustrating an example of the content displayed in the central section of a central display screen.

[0041] Figure 11 This is a view illustrating an example of the content displayed in the central section of a central display screen.

[0042] Figure 12 This is a view illustrating an example of the content displayed in the central section of a central display screen.

[0043] Figure 13 This is an example view of the content displayed on the left side of the central display screen.

[0044] Figure 14 This is a view illustrating an example of the content displayed on a heads-up display.

[0045] Figure 15 This is a view illustrating an example of the content displayed on a heads-up display.

[0046] Figure 16 This is an example of the display content of a digital rearview mirror.

[0047] Figure 17 This is a view illustrating an example of the content displayed on a steering wheel display.

[0048] Figure 18 This is an example of the content displayed on a steering wheel display.

[0049] Figure 19 This is a view showing an example of the content displayed on the entertainment monitor in the background.

[0050] Figure 20 It's a front view of the vehicle and a magnified view near the logo.

[0051] Figure 21 This is a left-side view of the vehicle.

[0052] Figure 22 This is an enlarged view of the far left side of the vehicle.

[0053] Figure 23 This is a rear view of the vehicle.

[0054] Figure 24 This is a view near the left headlight of the vehicle.

[0055] Figure 25 This is a view near the left headlight of the vehicle.

[0056] Figure 26 This is a view near the left headlight of the vehicle.

[0057] Figure 27 This is a diagram showing the interior of the vehicle when viewed from the right.

[0058] Figure 28 This is a diagram showing the area around the driver's seat and passenger seats in the vehicle.

[0059] Figure 29 This is a diagram of the area near the vehicle's dashboard.

[0060] Figure 30 This is a magnified view of the vehicle's steering wheel.

[0061] Figure 31 This is the view of the driver's seat from the left rear.

[0062] Figure 32 This is a view illustrating an example of the mounting location of a ToF camera used to capture images of the driver.

[0063] Figure 33 This is a view illustrating an example of the mounting location of a ToF camera used to capture images of the driver.

[0064] Figure 34 Installed by Figure 32 and Figure 33 A schematic diagram illustrating an example of an image captured by a ToF camera at its installation location.

[0065] Figure 35 This is a schematic diagram illustrating an example of an image captured by a ToF camera mounted near the steering wheel.

[0066] Figure 36This is a view illustrating an example of the mounting location of a ToF camera used to capture images of the driver.

[0067] Figure 37 This is a block diagram illustrating an example configuration of an information processing unit.

[0068] Figure 38 This is an example diagram illustrating the light emission pattern when the power is on.

[0069] Figure 39 This is an example diagram illustrating the light emission pattern when the power is on.

[0070] Figure 40 This is an example diagram illustrating the illumination patterns during operation and when the headlights are off.

[0071] Figure 41 This is an example diagram illustrating the lighting patterns during operation and when the headlights are on.

[0072] Figure 42 This is an illustration of an example of the light emission pattern when braking is applied.

[0073] Figure 43 This is an example diagram illustrating the lighting pattern during turn signal operation.

[0074] Figure 44 This is an example of the lighting pattern when the door is open.

[0075] Figure 45 This is an example of the lighting pattern when the door is closed.

[0076] Figure 46 This is an example diagram illustrating the lighting pattern during parking.

[0077] Figure 47 This is an example diagram illustrating the light emission pattern when the power is off.

[0078] Figure 48 It is a flowchart used to describe the image capture process.

[0079] Figure 49 The illustration shows an example of a moving image when the object detection mode is set.

[0080] Figure 50 The illustration shows an example of a moving image when setting the prediction mode.

[0081] Figure 51 The illustration shows an example of a moving image when setting search mode.

[0082] Figure 52 The illustration shows an example of a moving image when setting search mode.

[0083] Figure 53 It is a flowchart used to describe the editing process.

[0084] Figure 54 This is a diagram showing the interior of the vehicle from the right side, looking down.

[0085] Figure 55 This is a schematic diagram of the front of the vehicle.

[0086] Figure 56 This is a schematic diagram of the area in front of the driver's seat in the vehicle shown.

[0087] Figure 57 This is a magnified view of the area near the steering wheel.

[0088] Figure 58 This is a view illustrating an example of the mounting location of a ToF camera in a camera module.

[0089] Figure 59 This is an example view illustrating the image capture range of a ToF camera.

[0090] Figure 60 This is another example view illustrating the mounting location of a ToF camera.

[0091] Figure 61 This is another example view illustrating the mounting location of a ToF camera.

[0092] Figure 62 This is another example view illustrating the mounting location of a ToF camera.

[0093] Figure 63 This is another example view illustrating the mounting location of a ToF camera.

[0094] Figure 64 This is a view illustrating an example of the camera module's mounting location.

[0095] Figure 65 This is a view showing an example of the configuration inside the illustrated camera module.

[0096] Figure 66 This is a view illustrating an example of modifying the camera's settings position.

[0097] Figure 67 This is an example view illustrating the image capture range of a camera.

[0098] Figure 68 This is a view illustrating a display example of a display unit used in a CMS under normal conditions.

[0099] Figure 69 This is a view illustrating a first display example of the display unit used in the CMS when vehicles in adjacent lanes approach each other.

[0100] Figure 70 This is a view illustrating an example of the display unit used in the CMS when the direction indicator is turned on.

[0101] Figure 71 This is a second display example illustrating the display unit used by the CMS when vehicles in adjacent lanes approach each other.

[0102] Figure 72 This is a view illustrating an example of the display unit used in the CMS when the direction indicator is in operation.

[0103] Figure 73 This is an example view illustrating the range of the desired field of view.

[0104] Figure 74 This is a view used to describe an example of how the display range of the display unit used in CMS changes based on the shift position.

[0105] Figure 75 This is a view used to describe the methods for setting the display range of display units in a CMS.

[0106] Figure 76 This is a view used to describe the methods for setting the display range of display units in a CMS.

[0107] Figure 77 This is a view illustrating an example of the scaling ratio and display range of the display units used in a CMS.

[0108] Figure 78 It is a view used to describe the method of controlling the display range when looking into a display unit used in the CMS.

[0109] Figure 79 It is a view used to describe the method of controlling the display range when viewing a display unit used in a CMS.

[0110] Figure 80 It is a view used to describe the method of controlling the display range when the shift position is set to reverse and the view is directed at the display unit used by the CMS.

[0111] Figure 81 This is a view used to describe a method of controlling the display range when the shift position is set to reverse and the view is directed to a display unit applied to the CMS.

[0112] Figure 82 It is a view used to describe the method of controlling the display range when the shift position is set to reverse and the view is directed at the display unit used by the CMS.

[0113] Figure 83This is a view used to describe how to control footlights when viewed from above.

[0114] Figure 84 This is a view used to describe the method of controlling the display unit for the CMS when the door opener is touched.

[0115] Figure 85 This is a view describing a method for controlling the display unit for the CMS when the door opener is touched.

[0116] Figure 86 This is a view used to describe the method of controlling the display unit for the CMS when the door opener is touched.

[0117] Figure 87 This is a view illustrating the status of the display unit for the driver's seat and the lighting device for the steering wheel before driver identification.

[0118] Figure 88 This is a view illustrating the status of the display unit for the driver's seat and the lighting device for the steering wheel when identifying the driver.

[0119] Figure 89 This is a view illustrating the status of the display unit for the driver's seat and the lighting device for the steering wheel when preparation for autonomous driving is complete.

[0120] Figure 90 This is a view illustrating the status of the display unit for the driver's seat and the lighting device for the steering wheel when autonomous driving begins.

[0121] Figure 91 This is a view illustrating the status of the display unit for the driver's seat and the lighting device for the steering wheel when changing lanes during autonomous driving.

[0122] Figure 92 This is a view illustrating the status of the display unit for the driver's seat and the lighting device for the steering wheel when a hazard is detected.

[0123] Figure 93 This is a sample view used to describe the guide display when a button is activated.

[0124] Figure 94 This is a sample view used to describe the guide display when a button is activated.

[0125] Figure 95 This is a view illustrating an example of a display unit used for a driver's seat during lip reading.

[0126] Figure 96 This is a view used to describe a method for controlling a warning sound that alerts a vehicle to the road ahead.

[0127] Figure 97 This is a view used to describe the method of controlling sound effects when autonomous driving begins.

[0128] Figure 98 This is a view used to describe the method of controlling the sound effects during acceleration during autonomous driving.

[0129] Figure 99 This is a view used to describe the sound effects during deceleration in autonomous driving.

[0130] Figure 100 This is a view used to describe methods for controlling sound effects before outputting voice messages.

[0131] Figure 101 This is a view used to describe a method for controlling the warning sound of a vehicle approaching from the left rear.

[0132] Figure 102 This is a view used to describe a method for controlling the warning sound of a vehicle approaching from the left rear.

[0133] Figure 103 This is a view used to describe a method for controlling the warning sound of a vehicle approaching from the left rear.

[0134] Figure 104 This is a view used to describe a method for controlling the warning sound of a vehicle approaching from the left rear.

[0135] Figure 105 This is a view used to describe the method of controlling the voice of rear-seat occupants. Detailed Implementation

[0136] The modes used to implement this technology will be described below. They will be described in the following order.

[0137] 1. Configuration example of vehicle control system

[0138] 2. First Embodiment

[0139] 3. Modification example of the first embodiment

[0140] 4. Second Embodiment

[0141] 5. Modification example of the second embodiment

[0142] 6. Third embodiment

[0143] 7. Modification example of the third embodiment

[0144] 8. Other

[0145] <<1. Configuration Examples of Vehicle Control Systems>>

[0146] Figure 1 This is a block diagram illustrating a configuration example of a vehicle control system 11, which is an example of a mobile device control system to which this technology is applied.

[0147] The vehicle control system 11 is set up in the vehicle 1 and performs processing related to driving support and autonomous driving of the vehicle 1.

[0148] The vehicle control system 11 includes a processor 21, a communication unit 22, a map information storage unit 23, a Global Navigation Satellite System (GNSS) receiver 24, an external identification sensor 25, an in-vehicle sensor 26, a vehicle sensor 27, a recording unit 28, a driver support-autonomous driving control unit 29, a driver monitoring system (DMS) 30, a human-machine interface (HMI) 31, and a vehicle control unit 32.

[0149] Processor 21, communication unit 22, map information storage unit 23, GNSS receiver unit 24, external identification sensor 25, in-vehicle sensor 26, vehicle sensor 27, recording unit 28, driver support-autonomous driving control unit 29, driver monitoring system (DMS) 30, human-machine interface (HMI) 31, and vehicle control unit 32 are interconnected via communication network 41. Communication network 41 is formed, for example, by an in-vehicle communication network or bus conforming to any standard (such as Controller Area Network (CAN), Local Area Network (LIN), Local Area Network (LAN), FlexRay, Ethernet, etc.). Note that each part of the vehicle control system 11 can be directly connected via, for example, short-range wireless communication (Near Field Communication (NFC)) or Bluetooth, without going through communication network 41.

[0150] Note that in the following text, when each unit of the vehicle control system 11 performs communication via the communication network 41, the description of the communication network 41 is omitted. For example, when the processor 21 and the communication unit 22 communicate with each other via the communication network 41, the simple communication between the processor 21 and the communication unit 22 will be described.

[0151] The processor 21 is composed of various processors, such as a central processing unit (CPU), a microprocessor unit (MPU), and an electronic control unit (ECU). The processor 21 controls the entire vehicle control system 11.

[0152] The communication unit 22 communicates with various devices inside and outside the vehicle, other vehicles, servers, base stations, etc., and transmits and receives various data. For communication with the outside of the vehicle, for example, the communication unit 22 receives programs for updating the software used to control the operation of the vehicle control system 11, map information, traffic information, and information about the surroundings of the vehicle 1. For example, the communication unit 22 transmits information about the vehicle 1 (e.g., data indicating the status of the vehicle 1, the identification results of the identification unit 73, etc.) and information about the surrounding environment of the vehicle 1 to the outside. For example, the communication unit 22 performs communication with vehicle emergency call systems such as eCall.

[0153] It should be noted that there are no particular restrictions on the communication method of communication unit 22. In addition, a variety of communication methods can be used.

[0154] For communication with devices inside the vehicle, for example, communication unit 22 can wirelessly communicate with devices in the vehicle via communication methods such as wireless LAN, Bluetooth, NFC, or wireless USB (WUSB). For example, communication unit 22 can perform wired communication with devices in the vehicle via connection terminals (and cables if necessary) not shown, through communication methods such as Universal Serial Bus (USB), High Definition Multimedia Interface (HDMI, registered trademark), or Mobile High Definition Link (MHL).

[0155] Here, the devices in the vehicle are, for example, devices not connected to the vehicle's communication network 41. For example, consider mobile devices and wearable devices owned by occupants such as the driver, information devices brought into the vehicle and temporarily installed, etc.

[0156] For example, communication unit 22 communicates with servers located on external networks (e.g., the Internet, cloud networks, or operator-specific networks) via wireless communication systems such as fourth-generation mobile communication systems (4G), fifth-generation mobile communication systems (5G), Long Term Evolution (LTE), or Dedicated Short Range Communications (DSRC).

[0157] For example, communication unit 22 uses peer-to-peer (P2P) technology to communicate with terminals located near the vehicle (e.g., terminals belonging to pedestrians or shops, or machine-type communication (MTC) terminals). For example, communication unit 22 performs V2X communication. V2X communication includes, for example, vehicle-to-vehicle communication with other vehicles, vehicle-to-infrastructure communication with roadside equipment, communication with homes (vehicle-to-home), and vehicle-to-pedestrian communication with terminals owned by pedestrians.

[0158] For example, the communication unit 22 uses radio beacons, optical beacons, FM multiplexed broadcasts, etc., to receive electromagnetic waves transmitted by the vehicle information and communication system (VICS, registered trademark).

[0159] Map information storage unit 23 stores maps acquired from external sources and maps created by vehicle 1. For example, map information storage unit 23 stores 3D high-precision maps, global maps that are less accurate than high-precision maps but have a wider coverage area, etc.

[0160] High-precision maps include, for example, dynamic maps, point cloud maps, and vector maps (also known as Advanced Driver Assistance Systems (ADAS) maps). Dynamic maps, for example, are maps comprising four layers: dynamic information, quasi-dynamic information, quasi-static information, and static information, and are provided from external servers, etc. Point cloud maps are maps formed from point clouds (point cloud data). Vector maps are maps in which information such as lane and signal positions are associated with point cloud maps. Point cloud maps and vector maps can be provided from, for example, external servers, or can be created in vehicle 1 and stored in map information storage unit 23 as maps for matching with local maps based on sensing results from radar 52, LiDAR 53, etc., as described later. Furthermore, in the case of providing high-precision maps from external servers, for example, to reduce communication capacity, map data of several hundred square meters related to the planned route that vehicle 1 will travel on from now on is obtained from the server, etc.

[0161] The GNSS receiver unit 24 receives GNSS signals from GNSS satellites and supplies the signals to the driving support-autonomous driving control unit 29.

[0162] The external identification sensor 25 includes various sensors for identifying external conditions of the vehicle 1, and supplies sensor data from each sensor to each part of the vehicle control system 11. The type and number of sensors included in the external identification sensor 25 are arbitrary.

[0163] For example, the external identification sensor 25 includes a camera 51, a radar 52, a light detection and ranging, a laser imaging detection and ranging (LiDAR) 53, and an ultrasonic sensor 54. The number of cameras 51, radar 52, LiDAR 53, and ultrasonic sensors 54 is arbitrary, and examples of the sensing area of ​​each sensor will be described later.

[0164] Note that, for example, a camera using any imaging method such as a time-of-flight (ToF) camera, a stereo camera, a monocular camera, and an infrared camera may be used as camera 51 as needed.

[0165] Furthermore, for example, the external identification sensor 25 includes environmental sensors for detecting weather, climate, brightness, etc. Environmental sensors include, for example, rain sensors, fog sensors, sunlight sensors, snow sensors, illuminance sensors, etc.

[0166] Furthermore, for example, the external identification sensor 25 includes a microphone for detecting the location of sounds, sound sources, etc. around the vehicle 1.

[0167] The in-vehicle sensors 26 include various sensors for detecting information within the vehicle, and supply sensor data from each sensor to each part of the vehicle control system 11. The type and number of sensors included in the in-vehicle sensors 26 are arbitrary.

[0168] For example, in-vehicle sensors 26 include cameras, radar, seating sensors, steering wheel sensors, microphones, biosensors, etc. As cameras, for example, any imaging method can be used, such as ToF cameras, stereo cameras, monocular cameras, and infrared cameras. Biosensors are placed, for example, on seats, steering wheels, etc., and detect various biological information of occupants such as the driver.

[0169] Vehicle sensor 27 includes various sensors for detecting the state of vehicle 1, and supplies sensor data from each sensor to each part of vehicle control system 11. The type and number of sensors included in vehicle sensor 27 are arbitrary.

[0170] For example, vehicle sensor 27 includes a speed sensor, an acceleration sensor, an angular velocity sensor (gyroscope sensor), and an inertial measurement unit (IMU). For example, vehicle sensor 27 includes a steering angle sensor for detecting the steering angle of the steering wheel, a yaw rate sensor, an accelerator sensor for detecting the amount of accelerator pedal operation, and a brake sensor for detecting the amount of brake pedal operation. For example, vehicle sensor 27 includes a rotation sensor for detecting the engine or motor speed, a tire pressure sensor for detecting tire pressure, a slip rate sensor for detecting tire slip ratio, and a wheel speed sensor for detecting wheel rotation speed. For example, vehicle sensor 27 includes a battery sensor for detecting the remaining battery level and temperature, and an impact sensor for detecting external impacts.

[0171] Recording unit 28 includes, for example, read-only memory (ROM), random access memory (RAM), magnetic storage devices such as hard disk drives (HDDs), semiconductor storage devices, optical storage devices, magneto-optical storage devices, etc. Recording unit 28 records various programs, data, etc., used by each unit of the vehicle control system 11. For example, recording unit 28 records rosbag files, which include messages sent and received by the Robot Operating System (ROS) in which applications related to autonomous driving operate. For example, recording unit 28 includes a data storage system (DSSAD) and an event data recorder (EDR) for autonomous driving, and records information about vehicle 1 before and after events (such as accidents).

[0172] The driving support-autonomous driving control unit 29 controls the driving support and autonomous driving of vehicle 1. For example, the driving support-autonomous driving control unit 29 includes an analysis unit 61, an action planning unit 62, and an operation control unit 63.

[0173] The analysis unit 61 analyzes the vehicle 1 and its surroundings. The analysis unit 61 includes a self-position estimation unit 71, a sensor fusion unit 72, and a recognition unit 73.

[0174] The self-position estimation unit 71 estimates the vehicle 1's own position based on sensor data from the external identification sensor 25 and a high-precision map stored in the map information storage unit 23. For example, the self-position estimation unit 71 estimates the vehicle 1's own position by generating a local map based on sensor data from the external identification sensor 25 and matching the local map with the high-precision map. The position of the vehicle 1 is based, for example, on the center of the rear wheel axle.

[0175] The local map is, for example, a high-precision 3D map created using techniques such as Simultaneous Localization and Mapping (SLAM) and occupancy grid mapping. The high-precision 3D map is, for example, a point cloud map as described above. The occupancy grid map divides the 3D or 2D space surrounding vehicle 1 into grids of predetermined sizes and illustrates the occupancy status of objects within each grid unit. The occupancy status of an object is indicated, for example, by its presence, absence, or probability. The local map is also used, for example, by the recognition unit 73 for the detection and recognition of the external environment of vehicle 1.

[0176] Note that the self-position estimation unit 71 can estimate the self-position of vehicle 1 based on GNSS signals and sensor data from vehicle sensor 27.

[0177] The sensor fusion unit 72 performs sensor fusion processing to obtain new information by combining multiple different types of sensor data (e.g., image data supplied from camera 51 and sensor data supplied from radar 52). Methods for combining different types of sensor data include integration, fusion, correlation, etc.

[0178] The identification unit 73 performs detection and identification processing of the external conditions of vehicle 1.

[0179] For example, the identification unit 73 performs detection and identification processing on the external situation of the vehicle 1 based on information from the external identification sensor 25, information from its own position estimation unit 71, and information from the sensor fusion unit 72.

[0180] Specifically, for example, the recognition unit 73 performs object detection processing, recognition processing, etc., around the vehicle 1. Object detection processing includes, for example, detecting the presence or absence of objects, their size, shape, position, movement, etc. Object recognition processing includes, for example, recognizing attributes such as object type or recognizing a specific object. However, detection processing and recognition processing are not always clearly separated and can overlap.

[0181] For example, the identification unit 73 detects objects around the vehicle 1 by performing clustering of point clouds based on sensor data such as LiDAR and radar into each block of the point cloud. Thus, it detects the presence or absence, size, shape, and location of objects around the vehicle 1.

[0182] For example, the recognition unit 73 detects the movement of objects around vehicle 1 by performing tracking to follow the movement of blocks of a point cloud classified by clustering. Therefore, the speed and direction of travel (movement vector) of objects around vehicle 1 are detected.

[0183] For example, the recognition unit 73 identifies the types of objects around the vehicle 1 by performing object recognition processing, such as semantic segmentation, on the image data supplied from the camera 51.

[0184] Note that, for example, vehicles, people, bicycles, obstacles, structures, roads, traffic lights, traffic signs, road signs, etc., are assumed to be objects to be detected or identified.

[0185] For example, the recognition unit 73 performs traffic rule recognition processing around the vehicle 1 based on the map stored in the map information storage unit 23, the estimated result of its own position, and the recognition result of objects around the vehicle 1. Through this processing, for example, the location and status of signals, the content of traffic signs and road signs, the content of traffic rules, and the lanes in which vehicles can travel are identified.

[0186] For example, the recognition unit 73 performs recognition processing of the surrounding environment of the vehicle 1. The surrounding environment, which is the target of detection, includes, for example, weather, temperature, humidity, brightness, and road conditions.

[0187] The motion planning unit 62 creates a motion plan for vehicle 1. For example, the motion planning unit 62 creates a motion plan by executing route planning and route tracking processing.

[0188] Note that route planning (global path planning) is the process of planning a rough route from the starting point to the destination. This route planning is called track planning, and it also includes the processing of track generation (local path planning) that takes into account the motion characteristics of vehicle 1 on the route planned in the route planning, allowing vehicle 1 to travel safely and smoothly near vehicle 1.

[0189] Route following is the planned operation process for safely and accurately traveling on a planned route within a planned timeframe. For example, calculating the target speed and target angular velocity of vehicle 1.

[0190] The operation control unit 63 controls the operation of the vehicle 1 to implement the action plan created by the action planning unit 62.

[0191] For example, the operation control unit 63 controls the steering control unit 81, the braking control unit 82, and the driving control unit 83 to perform acceleration and deceleration control and directional control, so that the vehicle 1 travels on a track calculated by the track plan. For example, the operation control unit 63 performs coordinated control to achieve ADAS functions such as collision avoidance or impact mitigation, following other vehicles, maintaining vehicle speed, collision warning, and lane departure warning. For example, the operation control unit 63 performs coordinated control for the purpose of autonomous driving, so as to drive autonomously without driver intervention.

[0192] The DMS 30 performs driver authentication and driver state recognition processes based on sensor data from the in-vehicle sensor 26 and input data to the HMI 31. For example, physical condition, alertness, concentration, fatigue, gaze direction, level of intoxication, driving operation, and posture are assumed to be the driver's state to be identified.

[0193] Note that DMS 30 can perform authentication processing for occupants other than the driver and identification processing for occupant status. Furthermore, for example, DMS 30 can perform vehicle interior condition identification processing based on sensor data from in-vehicle sensors 26. For example, temperature, humidity, brightness, and odor are assumed to be the vehicle interior conditions to be identified.

[0194] HMI 31 is used to input various data, commands, etc., generate input signals based on the input data and commands, and supply the input signals to each part of the vehicle control system 11. For example, HMI 31 includes operating devices such as touch panels, buttons, microphones, switches, and joysticks, as well as operating devices that allow input via methods other than manual operation, such as voice and gestures. Note that HMI 31 may be, for example, a remote control device using infrared or other radio waves, or an external connection device corresponding to the operation of the vehicle control system 11, such as a mobile device or wearable device.

[0195] Furthermore, the HMI 31 performs output control, which controls the generation and output of visual, auditory, and tactile information from the occupants or the outside of the vehicle, including output content, timing, and method. Visual information includes, for example, images or lights on the operating screen, vehicle 1 status displays, warning displays, and monitor images illustrating the surroundings of vehicle 1. Auditory information includes, for example, information given via voice prompts, such as guidance, warning sounds, and warning messages. Tactile information includes information that provides tactile sensations to the occupants, such as force, vibration, or movement.

[0196] For example, display devices, projectors, navigation devices, instrument panels, camera monitoring systems (CMS), electronic reflectors, lights, etc., are assumed to be devices that output visual information. Display devices can be devices that display visual information in the occupant's field of view, such as head-up displays, transmissive displays, and wearable devices with augmented reality (AR) capabilities, for example, as a supplement to devices with normal displays.

[0197] For example, audio speakers, headphones, and earphones are assumed to be devices that output auditory information.

[0198] For example, tactile elements using haptic technology are assumed to be devices that output tactile information. Tactile elements are installed in, for example, steering wheels and seats.

[0199] The vehicle control unit 32 controls each part of the vehicle 1. The vehicle control unit 32 includes a steering control unit 81, a brake control unit 82, a driving control unit 83, a body system control unit 84, a light control unit 85, and a horn control unit 86.

[0200] The steering control unit 81 performs functions such as detecting and controlling the state of the steering system of vehicle 1. The steering system includes, for example, a steering mechanism including a steering wheel, electric power steering, etc. The steering control unit 81 includes, for example, a control unit such as an ECU that controls the steering system, and an actuator that drives the steering system.

[0201] The brake control unit 82 performs functions such as detecting and controlling the status of the braking system of vehicle 1. The braking system includes, for example, a braking mechanism including a brake pedal, an anti-lock braking system (ABS), etc. The brake control unit 82 includes, for example, a control unit such as an ECU that controls the braking system, and actuators that drive the braking system.

[0202] The driving control unit 83 performs tasks such as detecting and controlling the state of the driving system of vehicle 1. The driving system includes, for example, an accelerator pedal, a drive force generator (such as an internal combustion engine, drive motor, etc.) for generating driving force, and a drive force transmission mechanism for transmitting driving force to the wheels. The driving control unit 83 includes, for example, control units such as an ECU for controlling the driving system and actuators for driving the driving system.

[0203] The body system control unit 84 performs functions such as detecting and controlling the status of the vehicle 1's body systems. The body systems include, for example, keyless entry systems, smart key systems, power windows, power seats, air conditioning, airbags, seat belts, and gearshift levers. The body system control unit 84 includes control units such as ECUs that control the body systems and actuators that drive the body systems.

[0204] The lighting control unit 85 performs functions such as detecting and controlling the status of various lights in the vehicle 1. For example, headlights, backlights, fog lights, turn signals, brake lights, projectors, and bumper displays are assumed to be lights to be controlled. The lighting control unit 85 includes control units such as an ECU that controls the lights and actuators that drive the lights.

[0205] The horn control unit 86 performs functions such as detecting and controlling the status of the vehicle horn of vehicle 1. The horn control unit 86 includes control units such as an ECU that controls the vehicle horn and an actuator that drives the vehicle horn.

[0206] Figure 2 It is a diagram. Figure 1 An example view of the sensing area of ​​the external identification sensor 25, including camera 51, radar 52, LiDAR 53, and ultrasonic sensor 54.

[0207] Sensing regions SA1F and SA1B illustrate examples of the sensing regions of the ultrasonic sensor 54. Sensing region SA1F covers the periphery of the front end of vehicle 1. Sensing region SA1B covers the periphery of the rear end of vehicle 1.

[0208] The sensing results in sensing areas SA1F and SA1B are used, for example, for parking support of vehicle 1.

[0209] Sensing areas SA2F to SA2B illustrate examples of the sensing areas of radar 52 for short or medium range applications. Sensing area SA2F covers a position in front of vehicle 1 that is farther than sensing area SA1F. Sensing area SA2B covers a position behind vehicle 1 that is farther than sensing area SA1B. Sensing area SA2L covers the rear perimeter of the left side of vehicle 1. Sensing area SA2R covers the rear perimeter of the right side of vehicle 1.

[0210] The sensing results in sensing area SA2F are used, for example, to detect vehicles, pedestrians, etc., in front of vehicle 1. The sensing results in sensing area SA2B are used, for example, for rear-end collision avoidance functions of vehicle 1. The sensing results in sensing areas SA2L and SA2R are used, for example, to detect objects in the blind spot on the side of vehicle 1.

[0211] Sensing areas SA3F to SA3B illustrate examples of the sensing areas of camera 51. Sensing area SA3F covers a position further in front of vehicle 1 than sensing area SA2F. Sensing area SA3B covers a position further behind vehicle 1 than sensing area SA2B. Sensing area SA3L covers the perimeter of the left side of vehicle 1. Sensing area SA3R covers the perimeter of the right side of vehicle 1.

[0212] Sensing results in sensing area SA3F are used, for example, to identify traffic lights and signs, and for lane departure prevention support systems. Sensing results in sensing area SA3B are used, for example, for parking assistance, and for surround view systems. Sensing results in sensing areas SA3L and SA3R are used, for example, for surround view systems.

[0213] The illustration of sensing area SA4 shows an example of the sensing area of ​​LiDAR 53. Sensing area SA4 covers a greater distance in front of vehicle 1 than sensing area SA3F. On the other hand, sensing area SA4 is narrower in the left-right direction than sensing area SA3F.

[0214] The sensing results in sensing area SA4 are used for applications such as emergency braking, collision avoidance, and pedestrian detection.

[0215] The illustration of sensing area SA5 shows an example of the sensing area of ​​radar 52 used for long-range applications. Sensing area SA5 covers a greater distance in front of vehicle 1 than sensing area SA4. On the other hand, sensing area SA5 is narrower in the left-right direction than sensing area SA4.

[0216] The sensing results in sensing area SA5 are used, for example, for adaptive cruise control (ACC).

[0217] Note that the corresponding sensing area of ​​the sensor can have, except for Figure 2 Various configurations other than those shown are also possible. Specifically, the ultrasonic sensor 54 can also sense the sides of the vehicle 1, or the LiDAR 53 can sense the rear of the vehicle 1.

[0218] <<2. First Embodiment>>

[0219] Next, we will refer to Figures 3 to 19 The first embodiment of this technology is described.

[0220] <Example of the mounting locations for external identification sensor 25 and in-vehicle sensor 26>

[0221] Figure 3 The illustration shows an example of the mounting positions of cameras 51, radar 52, and LiDAR 53 included in the external identification sensor 25 of vehicle 1A, and cameras disposed in the in-vehicle sensor 26. Vehicle 1A is a first embodiment of vehicle 1 to which this technology is applied. In this example, the external identification sensor 25 includes cameras 51FC to 51BR, radars 52FC to 52BR, and LiDARs 53F to 53B. The in-vehicle sensor 26 includes cameras 151FL to 151BR.

[0222] Camera 51FC is positioned near the center of the far end of vehicle 1A. Camera 51FC captures images of the front of vehicle 1A.

[0223] Camera 51FL is positioned near the far left end of vehicle 1A. Camera 51FL captures images of the left-south-facing side of vehicle 1A.

[0224] Camera 51FR is positioned near the far right end of vehicle 1A. Camera 51FR captures images of the right-south-facing side of vehicle 1A.

[0225] Camera 51M is positioned near the center of the front of the interior of vehicle 1A. Camera 51M captures an image of the front of vehicle 1A through windshield 104.

[0226] Camera 51SL is positioned near the front of the door of driver's seat 101 on the left side of vehicle 1A. Camera 51SL captures images of the left rear (left and rear) side of vehicle 1A.

[0227] Camera 51SR is positioned on the right side of vehicle 1A near the front of the door of passenger seat 102, which is arranged adjacent to driver's seat 101. Camera 51SR captures images of the right rear (right side and rear) of vehicle 1A.

[0228] Camera 51BC is positioned near the center of the rear end of vehicle 1A. Camera 51BC captures images of the rear of vehicle 1A.

[0229] Camera 51BL is positioned near the left end of the rear end of vehicle 1A. Camera 51BL captures images of the left-side rear of vehicle 1A.

[0230] Camera 51BR is positioned near the right end of the rear end of vehicle 1A. Camera 51BR captures images of the right-side rear of vehicle 1A.

[0231] Radar 52FC is positioned near the center of the far end of vehicle 1A. Radar 52FC senses the front of vehicle 1A.

[0232] Radar 52FL is positioned near the far left end of vehicle 1A. Radar 52FL senses the left-south front of vehicle 1A.

[0233] Radar 52FR is positioned near the far right end of vehicle 1A. Radar 52FR senses the right-front angle of vehicle 1A.

[0234] Radar 52SL is positioned on the front left side of vehicle 1A. Radar 52SL senses the left side of vehicle 1A.

[0235] Radar 52SR is positioned on the front right side of vehicle 1A. Radar 52SL senses the right side of vehicle 1A.

[0236] Radar 52BC is positioned near the center of the rear end of vehicle 1A. Radar 52BC senses the rear of vehicle 1A.

[0237] Radar 52BL is positioned near the left rear end of vehicle 1A. Radar 52BL senses the left diagonal rear of vehicle 1A.

[0238] Radar 52BR is positioned near the right end of the rear end of vehicle 1A. Radar 52BR senses the right diagonally rear of vehicle 1A.

[0239] The LiDAR 53F is positioned near the center of the far end of vehicle 1A. The LiDAR 53F senses the front of vehicle 1A.

[0240] The LiDAR 53L is positioned on the front left side of vehicle 1A. The LiDAR 53L senses the left side of vehicle 1A.

[0241] The LiDAR 53R is positioned on the front right side of vehicle 1A. The LiDAR 53R senses the right side of vehicle 1A.

[0242] The LiDAR 53B is positioned near the center of the rear end of vehicle 1A. The LiDAR 53B senses the rear of vehicle 1A.

[0243] Camera 151FL is positioned near the left front of driver's seat 101 in the vehicle. Camera 151FL captures images of the area around the driver (the person sitting in driver's seat 101).

[0244] Camera 151FR is positioned near the right front of passenger seat 102 in the vehicle. Camera 151FR captures images of the surroundings of the occupant sitting in passenger seat 102.

[0245] Camera 151BL is positioned near the front of the rear seat 103L on the left side of the vehicle. Camera 151BL captures images of the surroundings of the occupant sitting in the rear seat 103L.

[0246] Camera 151BR is positioned near the front of the rear seat 103R on the right side of the vehicle. Camera 151BR captures images of the surroundings of the occupant sitting in the rear seat 103R.

[0247] In the following text, without needing to distinguish between the rear seat 103L and the rear seat 103R, they will be referred to as the rear seat 103.

[0248] <Example of display configuration in a vehicle>

[0249] Next, we will refer to Figures 4 to 9 This describes a configuration example of a display (display unit) that is installed inside vehicle 1A, forms part of HMI 31, and forms part of the display system of vehicle 1A. Figure 4 This is a schematic diagram of the front of the vehicle. Figure 5 This is a schematic diagram of the area surrounding the driver's seat 101 and passenger seat 102 when viewed from above. Figure 6 This is a schematic diagram of passenger seat 102 viewed from the left. Figure 7 This is a schematic diagram of the area surrounding the driver's seat 101 and passenger seat 102 when viewed from the left rear. Figure 8 This is a schematic diagram of the front of vehicle 1A as viewed from the driver's seat 101. Figure 9 This is a schematic diagram of passenger seat 102 as viewed from the left rear.

[0250] Inside the vehicle, there are a central display 201, a console display 202, a head-up display (only display 203 is shown), a digital rearview mirror 204, a steering wheel display 205, and a rear entertainment display 206.

[0251] The central display 201 is positioned in front of the driver's seat 101 and the passenger seat 102, extending laterally in front of the instrument panel 105. The central display 201 is roughly divided into a left-hand portion 201L, a central portion 201C, and a right-hand portion 201R, depending on the orientation of the display. Specifically, the central display 201 is configured such that the left-hand portion 201L, the central portion 201C, and the right-hand portion 201R, which have different orientations, are continuous and integrated in the left-right direction. The left-hand portion 201L, the central portion 201C, and the right-hand portion 201R can be displayed independently or as a single unit.

[0252] The central portion 201C extends laterally from near the left end of the driver's seat 101 to near the right end of the passenger seat 102 in front of the driver's seat 101 and the passenger seat 102, and faces rearward (rear of vehicle 1A) when viewed from either the driver's seat 101 or the passenger seat 102. Furthermore, as... Figure 6As shown, the central portion 201C faces diagonally upward. Therefore, when the driver sitting in the driver's seat 101 and the occupant sitting in the passenger seat 102 look at the central portion 201C, the angle of incidence of their line of sight is close to vertical, thus improving visibility.

[0253] The left end portion 201L and the right end portion 201R are arranged substantially symmetrically at the left and right ends of the central display 201. The left end portion 201L curves inward (inside the vehicle) at the left end of the central display 201, forming an angle inward relative to the central portion 201C, and faces diagonally to the right rear (diagonally to the right rear of vehicle 1A) when viewed from the driver's seat 101 or passenger seat 102. The right end portion 201R curves inward (inside the vehicle) at the right end of the central display 201, forming an angle inward relative to the central portion 201C, and faces diagonally to the left rear (diagonally to the left rear of vehicle 1A) when viewed from the driver's seat 101 or passenger seat 102.

[0254] The angle of the left end portion 201L relative to the central portion 201C is adjusted such that, for example, the reflection angle of the driver's standard line of sight relative to the angle of incidence on the left end portion 201L faces an appropriate direction to the left rear of the vehicle 1A. The angle of the right end portion 201R relative to the central portion 201C is adjusted such that, for example, the reflection angle of the driver's standard line of sight relative to the angle of incidence on the right end portion 201R faces an appropriate direction to the right rear of the vehicle 1A.

[0255] The console display 202 is disposed on the console 107 located between the driver's seat 101 and the passenger seat 102, and is arranged below the central portion 201C of the central display 201.

[0256] The console 107 extends in the fore-and-aft direction between the driver's seat 101 and the passenger seat 102. (As...) Figure 6 As shown, the console 107 is roughly divided into a remote portion 107A, a middle portion 107B, and a rear portion 107C from the front in sequence.

[0257] The distal portion 107A of the console 107 is tilted upwards towards the front of the vehicle 1A. The console display 202 is located at the distal end of the distal portion 107A. Furthermore, in the distal portion 107A, behind (below) the console display 202, for example, an operating unit 109 including multiple physical switches, keys, etc. (not shown) is provided.

[0258] The console display 202 includes, for example, a two-dimensional or three-dimensional touch panel and can be operated by touch or by bringing a finger close to it. The console display 202 faces the rear of the vehicle 1A. Furthermore, the console display 202 faces diagonally upwards at an angle substantially similar to that of the central portion 201C of the central display 201. This creates a sense of unity where the central display 201 and the console display 202 are continuously connected. Moreover, like the central portion 201C of the central display 201, the visibility of the console display 202 is improved.

[0259] The middle portion 107B of the control console 107 is tilted downwards towards the front of the vehicle 1A, opposite to the far portion 107A. A circular controller 108 is positioned near the center of the middle portion 107B. The tilt angle of the middle portion 107B is set so that the driver can easily place his or her hands, which improves the operability of the controller 108.

[0260] For example, the rear portion 107C of the console 107 is roughly horizontal, allowing the driver to easily place his or her arm.

[0261] like Figure 4 As shown, the head-up display includes a display 203 disposed in front of the driver's seat 101. For example, the display 203 may be formed as a part of the windshield 104, or it may be disposed separately from the windshield 104. In the latter case, for example, the display 203 is attached to the windshield 104. Then, by using AR technology to project visual information onto the display 203, the visual information is superimposed and displayed in the driver's field of vision.

[0262] Note that, for example, the position of display 203 is set higher than that of a conventional in-vehicle head-up display. This makes it possible to overlay visual information of vehicles, etc., at a greater distance in front of the vehicle 1A (e.g., 100m in front).

[0263] The digital rearview mirror 204 is used to replace the conventional rearview mirror and is also referred to as the smart rearview mirror. Like the conventional rearview mirror, the digital rearview mirror 204 is positioned near the upper end of the windshield 104 and slightly forward of its center, and is arranged above the central portion 201C of the central display 201.

[0264] The steering wheel display 205 is located in the center of the steering wheel 106.

[0265] The rear entertainment display 206 is disposed on the back of the passenger seat 102, and more specifically, on the back of the headrest of the passenger seat 102, such as... Figure 7 or Figure 9 As shown in the image.

[0266] Note that, for example, the rear entertainment display 206 may be installed on the back of both the driver's seat 101 and the passenger seat 102, or it may be installed only on the back of the driver's seat 101.

[0267] Note that the displays of the central display 201, console display 202, head-up display, digital rearview mirror 204, steering wheel display 205, and rear entertainment display 206 are controlled, for example, by a display control unit that is part of the functionality of the HMI 31.

[0268] <Examples of what each display in the vehicle shows>

[0269] Next, we will refer to Figures 10 to 17 Examples describing the content displayed on each display in the vehicle.

[0270] <Example of content displayed on central display 201>

[0271] First, an example of the content displayed on the central display 201 will be described.

[0272] The central portion 201C of the central display 201 shows information such as driving support and images of the vehicle 1A around it.

[0273] For example, the central section 201C displays information related to the driving status of vehicle 1A. For example, such as... Figure 10 As shown, the direction indicator information 301, speed information 302, and battery information 303 are displayed side by side.

[0274] The direction indicator information 301 indicates the operating status of the direction indicator of vehicle 1A. The left and right arrows in the direction indicator information 301 flash or light in conjunction with the left-right direction direction indicator, respectively.

[0275] Speed ​​information 302 indicates the speed and shift position of vehicle 1A. The speed of vehicle 1A is indicated by bars and numbers extending in the left-right direction. Note that, for example, speed information 302 may also indicate a speed limit.

[0276] Battery information 303 indicates the remaining battery charge of vehicle 1A. The remaining battery charge is indicated by bars and numbers extending from left to right.

[0277] For example, the central section 201C displays an image illustrating the road conditions in front of vehicle 1A at locations such as T-junctions or intersections where a road extending in a left-right direction exists in front of vehicle 1A. For instance, it displays a composite image from cameras 51FL, 51FC, and 51FR; that is, an image taken from approximately a 180-degree angle centered on the front of vehicle 1A when viewed from the far end of vehicle 1A. Therefore, the driver can easily identify vehicles approaching from the left or right on the road ahead.

[0278] Note that the recognition unit 73 detects T-junctions, intersections, etc. in front of the vehicle 1A based on, for example, images from camera 51M and sensor data from LiDAR 53F.

[0279] For example, when vehicle 1A is parked, the central section 201C displays information to support parking. For example, as... Figure 11 As shown, images from cameras 51BL, 51BC, and 51BR are displayed, specifically images of the area behind vehicle 1A. Furthermore, a parking position 321 is displayed by flashing in the image of the area behind vehicle 1A. The parking position 321 indicates, for example, the position set for parking when automatic parking is performed, and indicates a parking location or a recommended parking location when manual parking is performed.

[0280] At this time, for example, the 360-degree image of the vehicle 1A captured by each camera 51 (hereinafter referred to as the wide-angle image) can be rotated and displayed on the central portion 201C. Note that, for example, the range of the wide-angle image displayed on the central portion 201C can be rotated according to the driver's posture. Specifically, for example, the range of the wide-angle image displayed in the central portion 201C can be moved based on the driver's position relative to a predetermined reference position, the direction of the head, etc., thereby rotating the wide-angle image displayed in the image of the central portion 201C.

[0281] This makes it possible to identify parking locations, easily find suitable parking spots, and park the vehicle safely 1A.

[0282] Note that the parking location 321 is detected by the recognition unit 73 based on images from, for example, cameras 51BL, 51BC and 51BR, as well as sensor data from LiDAR 53B.

[0283] For example, the central section 201C displays images from cameras 51BL, 51BC and 51BR, that is, images of the area behind vehicle 1A when vehicle 1A is reversing.

[0284] For example, such as Figure 12As shown, image 341 displays the area behind vehicle 1A. Furthermore, for example, if an obstacle is detected behind vehicle 1A, alarm display 342 flashes in image 341. Additionally, image 343 shows the location of the obstacle. The vehicle in image 343 is vehicle 1A. Then, alarm display 344, indicating the location of the obstacle, flashes around the vehicle in image 343. In this example, an obstacle is shown to the right rear of vehicle 1A.

[0285] Therefore, the driver can safely reverse vehicle 1A.

[0286] Note that the identification unit 73 detects obstacles behind the vehicle 1A based on, for example, images from cameras 51BL, 51BC and 51BR and sensor data from radars 52BL, 52BC, 52BR and LiDAR 53B.

[0287] For example, when the siren of an emergency vehicle (e.g., ambulance, fire truck, police car, etc.) is detected, the central section 201C displays information indicating the direction of the siren's sound source. Therefore, the driver can accurately identify the direction of the emergency vehicle and take appropriate measures such as retreating.

[0288] Note that, for example, the identification unit 73 detects the direction of the alarm's sound source based on the sound detected by the microphone included in the external identification sensor 25.

[0289] For example, if a person is detected in the shadows surrounding vehicle 1A, the central section 201C displays an alarm. Information indicating the direction of the person is displayed as part of the alarm. Therefore, the driver can avoid collisions or contact with people in the shadows.

[0290] Note that, for example, the identification unit 73 detects people in the shadows based on sensor data from each radar 52 and each LiDAR 53.

[0291] For example, when an object that is difficult for the human eye to see (e.g., an animal existing in the dark) is detected in the dark surroundings of vehicle 1A, the central section 201C displays an alarm. For example, when displaying an image of the area in front of vehicle 1A captured by camera 51M, a box indicating the location of the detected object is overlaid as an alarm display. Therefore, the driver can avoid collision or contact with the detected object.

[0292] Note that the recognition unit 73 detects objects that are difficult to see, for example, based on brightness differences in images from the camera 51M.

[0293] For example, when the Lane Keeping Assist (LKA) function is operating while the vehicle is reversing (1A), the central section 201C displays information indicating the predicted driving direction. The driving direction is indicated, for example, by lines or arrows. Therefore, the driver can know the direction of reversing in advance and can, for example, respond appropriately to emergencies or hazards.

[0294] For example, the central section 201C displays images from cameras 151FL and 151FR installed in the rear seat 103. Therefore, the driver can monitor, for example, the status of a child sitting in the rear seat 103.

[0295] Note that, for example, in the case where the central portion 201C displays an image of the area surrounding the vehicle 1A, the display range of the image can be changed based on at least one of the driver's line of sight and posture.

[0296] For example, DMS 30 identifies the driver's gaze direction and posture based on images from camera 151FL. Furthermore, factors such as head position and orientation, and seating position are identified as driver posture.

[0297] Furthermore, depending on the situation, the central section 201C may be divided into three parts: the driver's seat area in front of the driver's seat 101, the central area between the driver's seat 101 and the passenger seat 102, and the passenger seat area in front of the passenger seat 102. The driver's seat area, the central area, and the passenger seat area can be displayed independently.

[0298] Information primarily intended for the driver is displayed in the driver's seat area. For example, the aforementioned information used to support driving is displayed.

[0299] For example, in the central area, information related to infotainment (in-vehicle infotainment) is displayed, such as audio, video, websites, and maps.

[0300] For example, in the passenger seating area, infotainment-related information for occupants in their seats is displayed.

[0301] The left-hand portion 201L and the right-hand portion 201R of the central display 201 are primarily used as digital exterior rearview mirrors (electronic side mirrors) to replace conventional side mirrors. Specifically, the left-hand portion 201L and the right-hand portion 201R are used in the CMS. For example, the left-hand portion 201L displays an image of the vehicle 1A from the left rear, captured by camera 51SL. The right-hand portion 201R displays an image of the vehicle 1A from the right rear, captured by camera 51SR.

[0302] Note that the left portion 201L, the central portion 201C, and the right portion 201R of the central display 201 are continuous in the horizontal direction. Therefore, when the driver moves their gaze from the central portion 201C to the left portion 201L or the right portion 201R, or vice versa, the driver only needs to move their gaze horizontally. This reduces the distance and direction of the driver's gaze movement and improves visibility.

[0303] In addition, for example, if a vehicle, motorcycle, bicycle, pedestrian, or other object is detected approaching from the left or rear of vehicle 1A, the left end portion 201L displays an alarm.

[0304] For example, when a vehicle approaches from the left rear of vehicle 1A, such as Figure 13 As shown, a box 361 displays a warning of an approaching vehicle. Additionally, an icon 362 indicates the location of the approaching vehicle by flashing. The upper right corner of icon 362 shows vehicle 1A, and the lower left corner shows the approaching vehicle.

[0305] Similarly, if a vehicle, motorcycle, bicycle, pedestrian, or other object is detected approaching from the right or rear of vehicle 1A, the right-hand section 201R displays an alarm.

[0306] Note that the identification unit 73 detects vehicles approaching from the left or right and rear of the vehicle 1A, for example, based on images from cameras 51SL, 51SR, 51BL, 51BC and 51BR, and sensor data from radars 52BL, 52BC and 52BR and LiDAR 53B.

[0307] Note that the imaging capture direction of cameras 51SL and 51SR is adjusted, for example, by the display control unit based on at least one of the driver's gaze direction and posture. Therefore, as in conventional physical side mirrors, the range of the images displayed on the left end portion 201L and the right end portion 201R is appropriately adjusted according to the driver's gaze direction and posture. Thus, the driver can check the situation within a desired range diagonally behind the vehicle 1A without any discomfort.

[0308] Furthermore, for example, the display range of the images in the left portion 201L and the right portion 201R changes based on the operation of the direction indicator. For example, a normal viewing angle is set when the direction indicator is not in operation, while the viewing angle is widened when the direction indicator is in operation. Therefore, the driver can accurately identify the situation around the vehicle 1A and safely change the driving direction of the vehicle 1A.

[0309] Note that since the left portion 201L, the central portion 201C, and the right portion 201R are connected as one, it is possible to display a single image on the entire central display 201. For example, it is possible to display images of the area around the vehicle 1A, map information, infotainment-related information, etc., on the entire central display 201.

[0310] <Example of content displayed on console monitor 202>

[0311] Next, an example of the content displayed on the console display 202 will be described.

[0312] For example, the console display 202 shows an operating screen for operating the air conditioning equipment in the vehicle. Occupants such as the driver operate the air conditioning in the vehicle by using the displayed operating screen.

[0313] Note that, for example, at least a portion of the operation of the air conditioning in the vehicle can be performed using the operation unit 109.

[0314] For example, the console display 202 displays an operation screen for manipulating the information displayed on the central display 201. For example, an occupant such as a driver can use the displayed operation screen to scroll, zoom in, zoom out, switch, etc., on the information (e.g., a map, etc.) displayed on the central portion 201C of the central display 201.

[0315] As described above, the central display 201 and the console display 202 have a unified feel. Therefore, occupants can operate the information displayed on the central display 201 with a natural feeling by using the operating screen of the console display 202, thus improving operability.

[0316] Note that the controller 108 can be pushed, rotated, and tilted in a predetermined direction (e.g., forward, backward, left, and right), and is used, for example, to operate an audio system in a vehicle. For example, when the controller 108 is pressed, music is played or stopped, and the volume is adjusted by rotating the controller 108.

[0317] <Example of content displayed on a heads-up display>

[0318] The head-up display 203 shows information such as driving support.

[0319] For example, display 203 shows an alarm based on the situation around vehicle 1A.

[0320] For example, such as Figure 14As shown, when there is a sharp turn in front of vehicle 1A, warning display 381 is displayed. Warning display 381 indicates the direction of the turn with an arrow, and indicates the curvature of the turn with the shape of the arrow (e.g., length, thickness, etc.) and color. Therefore, the driver can recognize the presence of a sharp turn in advance and can safely drive through the sharp turn.

[0321] For example, such as Figure 15 As shown, in the event of a large vehicle, such as a truck, in the lane adjacent to the driving lane of vehicle 1A, a flashing warning display 401 is shown. Warning display 401 is a warning display used to notify the driver that a large vehicle, such as a truck, is approaching in the lane to the right of vehicle 1A. Therefore, the driver can recognize the approach of the large vehicle in advance and can avoid collision or contact with the large vehicle.

[0322] For example, when emergency braking is applied to vehicle 1A, display 203 shows an alert notifying the driver that emergency braking has been applied. Thus, the driver is informed of the reason for the emergency stop, which can provide the driver with a sense of security.

[0323] For example, when ACC (Adaptive Cruise Control) is in operation, display 203 shows information indicating the vehicle to be followed. For example, it may display a box surrounding the vehicle to be followed. Therefore, the driver can confirm the operation of ACC and predict the route of the vehicle 1A, and can respond appropriately to emergencies, dangers, etc.

[0324] For example, when the lane change support function is in operation, the display 203 may show the route to be traveled via arrows or the like if necessary. Therefore, the driver can anticipate lane changes and take appropriate action, for example, in emergency or dangerous situations.

[0325] <Example of the content displayed on digital rearview mirror 204>

[0326] The digital rearview mirror 204 displays, for example, an image of the area behind the vehicle 1A captured by the camera 51BC.

[0327] For example, the digital rearview mirror 204 displays an alert when another vehicle approaches behind vehicle 1A. For example, as... Figure 16 As shown, when the inter-vehicle distance between vehicle 1A and vehicle 421 behind vehicle 1A is less than a predetermined threshold, an alarm display 422 flashes to alert the driver.

[0328] Note that, for example, the identification unit 73 detects the inter-vehicle distance to the vehicle behind based on sensor data from radar 52BC and LiDAR 53B.

[0329] Furthermore, for example, the image capture direction of camera 51BC is adjusted based on at least one of the driver's gaze direction and posture. Therefore, as in a conventional physical rearview mirror, the range of the image displayed on digital rearview mirror 204 is appropriately adjusted according to the driver's gaze direction or posture. Thus, the driver can confirm the situation within a desired range behind vehicle 1A without any discomfort.

[0330] <Example of the content displayed on the steering wheel display 205>

[0331] The steering wheel display 205 can, for example, display messages to the driver. Therefore, the driver and vehicle 1A can communicate with each other.

[0332] For example, if the DMS 30 identifies the driver, the steering wheel display 205 displays a message 441 containing the name of the identified driver, such as... Figure 17 As shown in the image.

[0333] For example, when the driver inputs a voice command, the steering wheel display 205 displays a message 461 indicating that the voice command has been accepted, such as... Figure 18 As shown in the image.

[0334] In addition, for example, the steering wheel display 205 displays messages indicating confirmation and completion of toll payments for highways, toll roads, parking lots, etc.

[0335] <Example of the content displayed on the rear entertainment display 206>

[0336] For example, the rear entertainment display 206 displays infotainment-related information to the occupants in the rear seat 103.

[0337] For example, the rear entertainment display 206 displays alarms in emergency or dangerous situations. For example, such as Figure 19 As shown, an alarm display 481 is shown that flashes during an emergency stop to alert the occupants in the rear seat 103.

[0338] As described above, by displaying various information on each display in the vehicle, it is possible to improve the convenience of occupants such as the driver. For example, it is possible to support driving the vehicle, provide occupants with necessary information, and entertain them.

[0339] <<3. Modification Examples of the First Embodiment>>

[0340] In the following text, a modified example of the first embodiment of the present technology will be described.

[0341] This technology can be applied to mobile devices, for example, other than vehicles carrying people.

[0342] Furthermore, this technology can also be applied to, for example, autonomous driving mobile devices in which the occupants do not drive. In this case, there is no distinction between the driver's seat 101 and the passenger seat 102 as described above, and the various displays described above are arranged in front of the seats in the mobile device. Furthermore, for example, the display range of the images displayed on the displays can be changed based on the line of sight and posture of the person sitting in the designated seat.

[0343] <<4. Second Embodiment>>

[0344] Next, we will refer to Figures 20 to 53 A second embodiment of this technology is described.

[0345] Note that in each figure of the second embodiment, the parts corresponding to the parts of the first embodiment are given the same reference numerals, and their descriptions will be omitted as appropriate.

[0346] <External configuration example of vehicle 1B>

[0347] First, refer to Figures 20 to 26 An example of the external configuration of vehicle 1B, which is a second embodiment of vehicle 1 to which this technology is applied, will be described. Hereinafter, the configuration of the external lighting system of vehicle 1B will be described in detail.

[0348] Figure 20 This is a front view of vehicle 1B and a magnified view near the vehicle 1B logo in the front center. Figure 21 This is the left-side view of vehicle 1B. Figure 22 This is an enlarged view of the far left side of vehicle 1B. Figure 23 This is the rear view of vehicle 1B. Figures 24 to 26 This is a view of the area near the left headlight of vehicle 1B when viewed from multiple directions.

[0349] Note that, in the following text, the left and right sides facing the direction of travel of vehicle 1B will be referred to as the left and right sides of vehicle 1B, respectively. For example, Figure 20 The left and right sides are the right and left sides of vehicle 1B, respectively.

[0350] Most of the lights and sensors on the exterior of vehicle 1B are arranged along a ring line L1, which is a virtual line surrounding the perimeter of the vehicle body in a substantially horizontal direction. Here, "arranged along the ring line L1" includes not only those arranged on the ring line L1, but also those arranged near the ring line L1.

[0351] For example, such as Figure 20As shown, accessory lights 601L, accessory lights 601R, daytime running lights 602L, daytime running lights 602R, headlights 603LU, headlights 603LD, headlights 603RU and headlights 603RD are arranged on the front surface of the vehicle body.

[0352] The accessory light 601L extends from the center of the front surface of the vehicle body, in the vehicle width direction (left-right direction), near the right end of the headlights 603LU and 603LD. The accessory light 601R extends from the center of the front surface of the vehicle body, in the vehicle width direction (left-right direction), near the left end of the headlights 603RU and 603RD.

[0353] Accessory lights 601L and 601R are separate, with a gap between them. Specifically, the right end portion of accessory light 601L curves diagonally downward to the right, and the left end portion of accessory light 601R curves diagonally upward to the left. The curved portions at the right end of accessory light 601L and the curved portions at the left end of accessory light 601R face each other substantially parallel to each other, separated by a predetermined interval, to form the emblem of vehicle 1B. For example, an optical sensor (not shown) such as a camera, radar, or LiDAR is arranged in area A1 of the gap between accessory lights 601L and 601R, near the center of the emblem.

[0354] Daytime running lights 602L extend horizontally from the left end of accessory lights 601L to near the left end of headlights 603LU and 603LD. Furthermore, as... Figures 24 to 26 As shown, the left end of the daytime running light 602L bends toward the rear of the vehicle 1B and extends in a direction through the vehicle body.

[0355] Daytime running light 602R extends horizontally from the right end of accessory light 601R near the right end of headlight 603RU and headlight 603RD. Furthermore, although not shown, similar to daytime running light 602L, the right end portion of daytime running light 602R curves towards the rear of vehicle 1B and extends in a direction through the vehicle body.

[0356] Accessory lights 601L, accessory lights 601R, daytime running lights 602L and 602R form headlights along the ring line L1 on the front surface of the vehicle body. In addition, the headlights constitute a front line as part of the ring line L1, extending on the front surface of the vehicle body in the vehicle width direction (left-right direction), curving backward at both ends, and extending in the direction through the vehicle body.

[0357] Accessory lights 601L, 601R, 602L, and 602R each have multiple LEDs arranged horizontally. The on / off state, color, and brightness of each LED can be controlled individually.

[0358] Note that in the following text, unless it is necessary to distinguish between accessory lights 601L and 601R, they will be referred to as accessory lights 601. Similarly, unless it is necessary to distinguish between daytime running lights 602L and 602R, they will be referred to as daytime running lights 602.

[0359] The headlight 603LU is adjacent to the upper side of the daytime running light 602L, extends horizontally, and curves backward at the left end. The headlight 603LD is adjacent to the lower side of the daytime running light 602L, extends horizontally, and curves backward at the left end. In this way, the headlights 603LU and 603LD are separated vertically by the daytime running light 602L (front line).

[0360] The headlight 603RU is adjacent to the upper side of the daytime running light 602R, extends horizontally, and curves backward at the right end. The headlight 603RD is adjacent to the lower side of the daytime running light 602R, extends horizontally, and curves backward at the right end. In this way, the headlights 603RU and 603RD are separated vertically by the daytime running light 602R (front line).

[0361] Headlights 603LU and 603RU each include multiple LEDs arranged horizontally and vertically, and output low beam. Headlights 603LD and 603RD each include multiple LEDs arranged horizontally and vertically, and output high beam. The on / off state, color, brightness, etc. of each LED can be controlled individually.

[0362] Note that in the following text, unless it is necessary to distinguish between headlight 603LU and headlight 603LD, they will be referred to as headlight 603L. Unless it is necessary to distinguish between headlight 603RU and headlight 603RD, they will be referred to as headlight 603R. Unless it is necessary to distinguish between headlight 603L and headlight 603R, they will be referred to as headlight 603.

[0363] By separating the headlights 603L and 603R vertically from the front end in this manner, the design freedom of the headlights 603L and 603R is increased. For example, the headlights 603L and 603R can be designed differently from either the upturned or downturned eye styles. Furthermore, since the low beams (headlights 603LU and 603RU) and high beams (headlights 603LD and 603RD) are positioned appropriately, the functionality of the headlights 603 and the safety of the vehicle 1B are not compromised.

[0364] In addition, for example, such as Figure 21 As shown, the turn signal light 604L, auxiliary light 605FL, and auxiliary light 605BL are arranged on the left side of the vehicle body.

[0365] The turn signal light 604L extends in the front-rear direction directly above the loop line L1 on the extension line of the A-pillar 614L.

[0366] The auxiliary light 605FL is located behind the door handle 612FL of the left front door 611FL and illuminates the vicinity of the door handle 612FL. Since the door handle 612FL is located directly above the loop line L1, the auxiliary light 605FL is also located directly above the loop line L1.

[0367] In addition, for example, a short-range wireless communication device (not shown) such as NFC is arranged near the door handle 612FL.

[0368] The auxiliary light 605BL is located behind the door handle 612BL of the left rear door 611BL and illuminates the vicinity of the door handle 612BL. Since the door handle 612BL is located directly above the loop line L1, the auxiliary light 605BL is also located directly above the loop line L1.

[0369] In addition, for example, a short-range wireless communication device (not shown) such as NFC is arranged in the vicinity of the door handle 612BL.

[0370] In this manner, on the left side of vehicle 1B, the turn signal light 604L, auxiliary light 605FL, and auxiliary light 605BL are arranged along the loop line L1 in the front-rear direction.

[0371] The turn signal light 604L, auxiliary light 605FL, and auxiliary light 605BL each have multiple LEDs arranged horizontally. The on / off state, color, and brightness of each LED can be controlled individually.

[0372] In addition, such as Figure 21 As shown, for example, an optical sensor such as a camera, radar, or LiDAR is placed in the area A2L near the intersection of the extended line of the loop line L1 and the A-pillar 614L. For example, a LiDAR 53L is placed below the turn signal light 604L and above the loop line L1.

[0373] Furthermore, for example, an optical sensor such as a camera, radar, or LiDAR (not shown) may be placed in the region A3L near the intersection of the extension of the loop L1 and the C-pillar 615L.

[0374] By arranging optical sensors near the ring road L1 in this way, for example, even if the surface color of the optical sensors differs from the body color, the optical sensors can still be identified as part of the ring road L1. Therefore, the optical sensors blend naturally into the vehicle's appearance without creating any visual discomfort.

[0375] Note that, although not shown, on the right side of vehicle 1B, the turn signal 604R, auxiliary light 605FR, auxiliary light 605BR, door handle 612FR, door handle 612BR, short-range wireless communication device, and optical sensor are also located in similar positions to those on the left side.

[0376] Moreover, for example, such as Figure 23 As shown, taillights 606CL, 606CR, 606L, 606R, brake lights 607LU, 607LD, 607RU, and 607RD are arranged on the rear surface of the vehicle body.

[0377] Taillight 606CL extends from the center of the rear surface of the vehicle body towards the right end of brake lights 607LU and 607LD in the vehicle width direction (left-right direction). Taillight 606CR extends horizontally from the center (towards the right end) of brake lights 607RU and 607RD in the vehicle width direction of the rear surface of the vehicle body.

[0378] Taillights 606CL and 606CR are separate, with a gap between them. Specifically, the right end of taillight 606CL curves diagonally upward to the right, while the left end of taillight 606CR curves diagonally downward to the left. The curved portions at the right and left ends of taillights 606CL and 606CR are substantially parallel to each other, with a predetermined interval to form the emblem of vehicle 1B. For example, an optical sensor (not shown) such as a camera, radar, or LiDAR is located in area A4 of the gap near the center of the emblem between taillights 606CL and 606CR.

[0379] Taillight 606L extends horizontally from the left end of taillight 606CL towards the vicinity of the left ends of brake lights 607LU and 607LD. The left end portion of taillight 606L curves forward. Taillight 606R extends horizontally from the right end of taillight 606CR towards the vicinity of the right ends of brake lights 607RU and 607RD. The right end portion of taillight 606R curves forward.

[0380] Taillights 606CL, 606CR, 606L, and 606R form a tail line on the rear surface of the vehicle body that extends in a left-right direction and curves forward at both ends. The tail line forms part of loop L1.

[0381] Taillights 606CL, 606CR, 606L, and 606R each have multiple LEDs arranged horizontally. The on / off state, color, and brightness of each LED can be controlled individually.

[0382] Note that in the following text, unless it is necessary to distinguish between taillight 606CL and taillight 606CR, they will be referred to as taillight 606C. Similarly, unless it is necessary to distinguish between taillight 606C, taillight 606L, and taillight 606R, they will be referred to as taillight 606.

[0383] Brake light 607LU is adjacent to the upper side of taillight 606L and bends forward at the left end. Brake light 607LD is adjacent to the lower side of taillight 606L and bends forward at the left end. In this way, brake light 607LU and brake light 607LD are separated vertically by taillight 606L.

[0384] Brake light 607RU is adjacent to the upper side of taillight 606R and bends forward at the right end. Brake light 607RD is adjacent to the lower side of taillight 606R and bends forward at the right end. In this way, brake light 607RU and brake light 607RD are separated vertically by taillight 606R (tail line).

[0385] Brake lights 607LU, 607LD, 607RU, and 607RD each have multiple LEDs arranged horizontally. The on / off state, color, and brightness of each LED can be controlled individually.

[0386] Note that in the following text, unless it is necessary to distinguish between brake lamp 607LU and brake lamp 607LD, they will be referred to as brake lamp 607L. Continuing below, unless it is necessary to distinguish between brake lamp 607RU and brake lamp 607RD, they will be referred to as brake lamp 607R. Continuing below, unless it is necessary to distinguish between brake lamp 607L and brake lamp 607R, they will be referred to as brake lamp 607.

[0387] Furthermore, along the lower end of the windshield 104, the lower end of the window 613FL of door 611FL, the lower end of the window 613BL of door 611BL, the lower end of the window 613FR (not shown) of door 611FR, and the lower end of the window 613BR (not shown) of door 611BR, the body color is different from the ring line L2, which is a virtual line around the body of vehicle 1B in a roughly horizontal direction as a boundary.

[0388] For example, a black line is formed along ring line L2 by chrome plating. Then, the area above ring line L2 is grouped in black. For example, the body is painted black above ring line L2. In addition, the windshield 104, windows 613FL, 613BL, 613FR, 613BR and rear window 616 are smoked black.

[0389] On the other hand, below the L2 ring road, the vehicle body is painted a different color than the upper part. Note that there are no particular restrictions on the color of the lower part of the vehicle body.

[0390] In addition, cameras 51SL and 51SR are positioned along loop L2 near the front and lower ends of windows 613FL and 613FR (not shown). Cameras 51SL and 51SR capture images of the vehicle 1B from the left rear or right rear, respectively.

[0391] As mentioned above, the design can be improved while avoiding a deterioration in the safety and functionality of vehicle 1B.

[0392] For example, in the exterior of vehicle 1B, two roughly parallel ring lines L1 and L2 are virtually identified. This makes the vehicle appear low and gives a sporty impression.

[0393] Furthermore, as mentioned above, this increases the freedom in designing the headlights. Moreover, since each light is placed in its proper position, the functionality of each light and the safety of the vehicle are not compromised.

[0394] In addition, the front line (headlights) curves backward at both ends and extends in the direction that runs through the body, giving the impression that the front line runs through the body and connects with the LiDAR 53L on the left side and the LiDAR 53R on the right side.

[0395] Furthermore, by placing sensors around vehicle 1B along loop L1, an impression can be given of the area around vehicle 1B, and theft and vandalism of vehicle 1B can be prevented.

[0396] <Example of interior configuration of vehicle 1B>

[0397] Next, we will refer to Figures 27 to 36 Describes an example of the interior configuration of vehicle 1B.

[0398] First, refer to Figures 27 to 30 Describe the equipment arranged along loop L11.

[0399] Figure 27 This is a schematic diagram of the interior of vehicle 1B when viewed from the right side. Figure 28 This is a schematic diagram of the area near the driver's seat 101 and passenger seat 102 of vehicle 1B. Figure 29 This is a diagram of the area near the dashboard of vehicle 1B. Figure 30 This is an enlarged view of the steering wheel 651 of vehicle 1B.

[0400] Inside vehicle 1B, equipment for multiple user interfaces is centrally arranged, and various interfaces are arranged along loop L11. Figure 27 The cluster, loop L11 is a virtual line that surrounds the interior in the basic horizontal direction.

[0401] Here, the devices for the user interface include, for example, output devices that output visual, auditory, and tactile information, as well as operating devices for various operations. Furthermore, the arrangement along loop L11 includes not only arrangements on loop L11, but also arrangements near loop L11.

[0402] The ring line L11 is positioned at the same height as the ring line L1 on the outer side of vehicle 1B. Furthermore, the ring line L11 slopes slightly upwards from front to back. This is because the rear seats 103L and 103R are positioned higher than the driver's seat 101 and passenger seat 102.

[0403] For example, constituting Figure 1 The display devices of HMI 31 are arranged along loop L11.

[0404] For example, such as Figures 27 to 29 As shown, the central display 201 is arranged to extend directly above the loop line L11 on the front surface of the dashboard in front of the driver's seat 101 and the passenger seat 102 in the vehicle width direction (left-right direction).

[0405] As described above, the central display 201 is roughly divided into a left section, a central section, and a right section according to the orientation of the display. The left, central, and right sections of the central display 201 can be displayed individually or as a single unit. The left and right sections of the central display 201 primarily function as digital exterior rearview mirrors (electronic side mirrors) that replace conventional side mirrors. For example, the left section displays the image from camera 51SL (… Figure 20 Image captured by camera 51SR from the left rear. The right portion shows the image captured by camera 51SR. Figure 20 (Image captured from the right rear of vehicle 1B)

[0406] In addition, such as Figures 28 to 30 As shown, the steering wheel 651 is positioned in front of the driver's seat 101 on the loop L11.

[0407] Moreover, such as Figure 29 and Figure 30 As shown, an illumination device 652 is arranged around the periphery of the central portion of the steering wheel 651. The illumination device 652 includes multiple LEDs arranged circumferentially around the periphery of the central portion of the steering wheel 651. The on / off state, color, brightness, etc., of each LED can be controlled individually. Therefore, the illumination device 652 has variable color, brightness, and luminous area (luminous range).

[0408] Note that the airbag is housed in the central section of the steering wheel 651. Then, when the airbag is deployed, Figure 30The central portion of the steering wheel 651 is split, as indicated by the dotted line. The lighting device 652 is arranged to avoid the split portion in the central portion. This prevents debris and harmful substances from the lighting device 652 from dispersing when the airbag is activated.

[0409] In addition, constituting Figure 1 The speakers of the HMI 31 are arranged along loop L11.

[0410] Specifically, such as Figure 27 and Figure 28 As shown, speaker 653FL is embedded near the loop L11 inside the door 611FL on the driver's seat 101 side. Figure 28 As shown, speaker 653FR is embedded near loop L11 within the door 611FR on the side of passenger seat 102. Figure 27 As shown, speaker 653BL is embedded near loop L11 within door 611BL on the left rear seat 103L side. Although not shown, speaker 653BR is embedded near loop L11 within door 611BR on the right rear seat 103R side.

[0411] Furthermore, although not shown, speakers (hereinafter referred to as seat speakers) are embedded under the headrests of the driver's seat 101, passenger seat 102, rear seat 103L, and rear seat 103R, respectively. Moreover, the shape of the seats and the position of the seat speakers have been adjusted so that people of various heights (sitting heights) can clearly hear the sound from the seat speakers of each seat.

[0412] The speakers 653FL to 653BR, arranged along loop L11, are used, for example, to output sound directed throughout the interior of the vehicle (to all occupants of the vehicle).

[0413] Furthermore, 360-degree realistic audio is achieved through speakers 653FL to 653FR. For example, by achieving 360-degree realistic audio, it is possible to enjoy animated images, music, etc., in the vehicle with realistic sound. In addition, the location of dangerous objects such as obstacles around the vehicle 1B can be notified by the direction of sound output.

[0414] On the other hand, the seat speakers for each seat are used, for example, primarily to output private sound for the individual occupants sitting in each seat. That is, the sound output from each seat speaker is controlled individually.

[0415] Note that the speaker arrangement is an example and can be changed. For example, the number of speakers arranged on loop L11 can be increased. For example, the speakers can be placed on the dashboard at the front of vehicle 1B.

[0416] In addition, such as Figure 27As shown, the ring light 654 surrounds the vehicle interior at a height substantially the same as the outer ring light L11, slightly above the ring line L11, and is substantially parallel to the ring line L11. The ring light 654 is a downlight equipped with multiple LEDs embedded in the vehicle in a horizontal row, primarily used for auxiliary lighting and interior trim. The on / off state, color, brightness, etc., of each LED can be individually controlled.

[0417] Note that the ring light 654 does not have to surround the entire perimeter of the vehicle interior, and can discontinuously surround only a portion of the perimeter of the vehicle interior.

[0418] Furthermore, various operating equipment is arranged along the L11 ring road.

[0419] For example, as described above, the steering wheel 651 is arranged on the ring line L11 in front of the driver's seat 101.

[0420] In addition, such as Figure 30 As shown, a lever 655, which is a rod-shaped operating body, is disposed behind the steering wheel 651 and extends from the steering column (not shown) in the vehicle width direction (lateral (right direction)). The lever 655 can be moved in the vertical direction, and the shift position of the vehicle 1B can be switched by moving the lever 655 in the vertical direction. That is, the lever 655 constitutes a tilting gear lever that can be moved in the vertical direction. Note that the lever 655 can be, for example, a linear type that moves linearly in the vertical direction or a column type that moves in a Z-shaped manner in the vertical direction.

[0421] The gear shifting position setting sequence via the lever 655, from top to bottom, is reverse (R), neutral (N), drive (D), and autonomous driving (A). That is, when the lever 655 moves from top to bottom, the gear shifting position changes in the order of R, N, D, and A. When the lever 655 is moved from bottom to top, the gear shifting position changes in the order of A, D, N, and R.

[0422] Furthermore, a button 656 is provided at the tip of the lever 655, which can be pressed in the axial direction of the lever 655. When the button 656 is pressed, the shift position of the vehicle 1B is switched to park (P).

[0423] Furthermore, an indicator 657 is provided in a circumferential shape on the side of the control lever 655. The indicator 657 is positioned where it is visible from the gap between the spokes of the steering wheel 651 when viewed from the driver's seat 101.

[0424] The indicator 657 changes color according to the set shift position. For example, when the shift position is set to park, the indicator 657 turns red. When the shift position is set to drive, the indicator 657 turns white. When the shift position is set to autonomous driving and autonomous driving is possible, the indicator 657 turns green. When the shift position is set to autonomous driving and autonomous driving is in operation, the indicator 657 turns blue.

[0425] Note that, as Figure 30 As shown by the dotted line, the indicator 657 can be positioned on the outer periphery of the steering wheel 651 when viewed from the driver's seat 101. This makes the indicator 657 visible from outside the steering wheel 651. Furthermore, the indicator 657 can be positioned at both the solid line and the dotted line locations.

[0426] In this way, by setting an indicator 657 on the operating body (lever 655) used for shifting gears, the driver can intuitively understand the meaning of the color of the indicator 657 and can intuitively and definitively identify the shift position.

[0427] Furthermore, for example, the shift position setting status can be displayed on a central display 201 or similar device. In this case, for example, the shift positions are displayed in the order of the setting direction via the lever 655. That is, the shift positions are displayed from top to bottom in the order of R (reverse), N (neutral), D (drive), and A (autonomous driving).

[0428] Furthermore, for example, a dial that rotates about the axis (circumferential direction) can be provided at the tip or middle of the control lever 655 in the axial direction, and the shift position can be switched by rotating the dial.

[0429] In addition, such as Figure 28 As shown, air conditioning (A / C) vents 658FC, 658FL, and 658FR are arranged on ring line L11. Specifically, vent 658FR is located between the driver's seat 101 and the passenger seat 102, directly below the central display 201. Vent 658FL is located near the junction of the door 611FL on the driver's seat 101 side, directly below the central display 201. Vent 658FR is located near the junction of the door 611FR on the passenger seat 102 side, directly below the central display 201.

[0430] Moreover, such as Figure 29 As shown, the vent 658FC is equipped with knobs 659CL and 659CR for changing the airflow direction. Figure 29As shown, vent 658FL is provided with a knob 659FL for changing the airflow direction. Although not shown, vent 658FR is provided with a knob 659FR for changing the airflow direction at a position similar to that of knob 659FL of vent 658FL. Therefore, knobs 659CL, 659CR, 659FL and 659FR are provided on loop L11.

[0431] In addition, such as Figure 29 As shown, an operating unit 660, including switches for performing various settings of the headlight 603, is arranged on the ring line L11 on the right side of the back of the steering wheel 651.

[0432] Furthermore, a door opener is placed on loop L11. For example, as... Figure 28 As shown, the door opener 661FL is arranged on a loop L11 near the center in the front-rear direction of the door 611FL on the driver's seat 101 side. Similarly, the door opener 661FR is arranged on a loop L11 near the center in the front-rear direction of the door 611FR on the passenger seat 102 side.

[0433] By arranging various user interface devices along the ring road L11 in this manner, distracting noises are removed from the driver's line of sight, providing an environment conducive to driving concentration. Furthermore, by clustering various devices near the ring road L11, it becomes possible to visually identify and operate these devices. Moreover, by clustering various devices near the ring road L11 and arranging the ring lights 654, it is possible to provide an impression of the interior of the vehicle 1B. Additionally, it reduces the vertical movement of the eyes of occupants such as the driver, and suppresses motion sickness.

[0434] Figure 31 This is a view of the driver's seat 101 from the left rear.

[0435] The tablet terminal 662L is located on the back of the driver's seat 101, and more specifically, on the back of the headrest of the driver's seat 101.

[0436] For example, the 662L tablet terminal is composed of Figure 9 The rear entertainment display 206 displays infotainment-related information to occupants in the rear seat 103L and allows them to interact with the presented information. Additionally, for example, the tablet terminal 662L displays alarms in emergency situations or dangerous situations.

[0437] Note that, although not shown, a tablet terminal 662R similar to the tablet terminal 662L is provided on the back of, for example, passenger seat 102.

[0438] Furthermore, for example, a ToF camera can be placed near each of the tablet terminals 662L and 662R. Therefore, for example, it is possible to identify the occupants operating the tablet terminals 662L and 662R based on images captured by the ToF cameras.

[0439] Next, we will refer to Figures 32 to 36 An example of the mounting location of a ToF camera used to capture an image of the orientation of the driver's seat 101 (the driver sitting in the driver's seat 101).

[0440] like Figure 32 and Figure 33 As shown, the ToF camera is mounted at a mounting location P1, for example, around the digital rearview mirror 204.

[0441] The digital rearview mirror 204 replaces the conventional rearview mirror for inspecting the rear of the vehicle 1B, and is also known as a smart rearview mirror or digital rearview mirror. For example... Figure 28 As shown, the digital rearview mirror 204 is positioned near the upper end and center of the windshield 104 and, like a conventional interior rearview mirror, is positioned slightly towards the rear of the vehicle 1B relative to the windshield 104, and is positioned above the central portion of the central display 201. Furthermore, as... Figure 32 and Figure 33 As shown, the digital rearview mirror 204 is mounted near the roof via mounting portions 663 in front of and to the right of the driver's seat 101.

[0442] The mounting position P1 is located near the rear end of the left side of the mounting part 663, and near the upper left of the digital rearview mirror 204 on the roof of vehicle 1B.

[0443] Figure 34 An example of an image captured by a ToF camera is illustrated schematically with the camera mounted at mounting position P1.

[0444] By mounting a ToF camera at mounting position P1, an image of the upper body of the driver 681 sitting in the driver's seat 101 is captured from the oblique right side and as if looking down from above. Therefore, an image of the driver 681's face can be captured from the oblique upper right, and, for example, the recognition unit 73 ( Figure 1 It can recognize the driver's line of sight. Furthermore, it can provide a bird's-eye view. Figure 1 The system captures an image of the upper body of the driver 681, including the steering wheel 651, and the recognition unit 73 can, for example, recognize the driver 681's posture. Moreover, since the probability of an obstacle appearing between the driver 681 and the digital rearview mirror 204 is low, the driver 681's line of sight and posture can be reliably recognized.

[0445] on the other hand, Figure 35 The illustration shows an example of an image captured by a ToF camera when the ToF camera is positioned on or near the steering wheel 651.

[0446] In this scenario, an image of the area near the driver 681's face is captured so that the view is from the front upwards. Therefore, the recognition unit 73 can identify the driver 681's gaze. However, obstacles such as the driver 681's hands can easily appear between the driver 681 and the steering wheel 651, and it is assumed that the gaze may not be recognized. Furthermore, since an image of the driver 681's entire upper body is not captured, the recognition unit 73 has difficulty recognizing the driver 681's posture.

[0447] Figure 36 The illustration shows an example of modifying the installation location of a ToF camera.

[0448] For example, a ToF camera can be installed at mounting position P2 on the left side of the digital rearview mirror 204.

[0449] For example, on the roof of vehicle 1B, a ToF camera can be mounted behind the digital rearview mirror 204 (from the front side of vehicle 1B of the digital rearview mirror 204) and at a mounting position P3 that is closer to the driver's seat 101 than the digital rearview mirror 204 (to the left of the digital rearview mirror 204).

[0450] For example, on the roof of vehicle 1B, a ToF camera can be mounted in front of digital rearview mirror 204 (from the rear side of vehicle 1B of digital rearview mirror 204) and at a mounting position P4 that is closer to driver's seat 101 than digital rearview mirror 204 (to the left of digital rearview mirror 204).

[0451] Regardless of whether the ToF camera is installed in any of the installation positions P2 to P4, it is possible to identify the driver's gaze and posture based on the images captured by the ToF camera, as is the case when it is installed in installation position P1.

[0452] Note that a conventional optical mirror can be used instead of the digital rearview mirror 204. Furthermore, a camera of a different type than the ToF camera can be used.

[0453] Moreover, for example, a ToF camera that captures images of occupants sitting in passenger seat 102 can be mounted in a position symmetrical to that of a camera that captures images of the driver.

[0454] <Configuration Example of Information Processing Unit 701>

[0455] Figure 37 The diagram is drawn by Figure 1A block diagram illustrating a configuration example of the information processing unit 701 implemented by the processor 21, HMI 31, etc.

[0456] The information processing unit 701 includes an image processing unit 711, a metadata addition unit 712, and an output control unit 713.

[0457] The image processing unit 711 performs various image processing and editing operations on the moving image data acquired by the camera 51. Furthermore, the image processing unit 711 images sensor data acquired by optical sensors other than the camera (such as radar 52 and LiDAR 53).

[0458] Metadata adding unit 712 adds metadata to the moving image data obtained by camera 51.

[0459] The output control unit 713 controls the output of visual, auditory, and tactile information in the HMI 31. For example, the output control unit 713 controls the output of visual information from the central display 201, the tablet terminal 662L, the tablet terminal 662R, and the lighting device 652 of the steering wheel 651. For example, the output control unit 713 controls the output of auditory information (sound) from the speakers 653FL to 653BR and the seat speakers.

[0460] <Operation of Vehicle 1B>

[0461] Next, we will refer to Figures 38 to 53 An example describing the operation of vehicle 1B.

[0462] Note that in the following text, unless it is necessary to distinguish between door 611FL, door 611FR, door 611BL, and door 611BR, they will be referred to as door 611.

[0463] <Light emission patterns of lighting systems>

[0464] First, refer to Figures 38 to 47 An example describing the luminous pattern of the lighting system of vehicle 1B.

[0465] Lamp control unit 85 ( Figure 1 It performs coordinated control of the lights on the exterior and interior of the vehicle.

[0466] Specifically, Figures 38 to 47The illustration shows examples of the illumination modes of accessory lights 601L, 601R, daytime running lights 602L, 602R, headlights 603L and 603R, turn signals 604L and 604R, auxiliary lights 605FL, 605FR, 605BL and 605BR, taillights 606L, 606R, 606CL and 606CR, brake lights 607L and 607R, ring lights 654, and main lights 751.

[0467] Note that the main light 751 is equipped with LEDs arranged along the outer perimeter of the ceiling inside vehicle 1B and serves as the main lighting device in the vehicle. The on / off state, color, brightness, etc. of each LED can be controlled individually.

[0468] In addition, Figures 38 to 47 In this document, the headlights 603LU and 603LD, 603RU and 603RD, brake lights 607LU and 607LD, and brake lights 607RU and 607RD are indistinguishable from each other and are described together.

[0469] Note that, unless otherwise stated below, accessory lights 601L, 601R, daytime running lights 602L, 602R, headlights 603L, 603R, turn signals 604L, 604R, auxiliary lights 605FL, 605FR, 605BL, and 605BR emit white light. In the following text, unless otherwise stated, taillights 606L, 606R, 606CL, 606CR, brake lights 607L, and 607R emit red light. In the following text, unless otherwise stated, ring lights 654 and main lights 751 emit orange light.

[0470] Furthermore, in the following text, it is assumed that, in each figure, lit lights are painted black and unlit lights are painted white.

[0471] <When powered on>

[0472] Figure 38 and Figure 39 The illustration shows an example of the light-emitting pattern when the power is turned on in vehicle 1B. Figure 38 The diagram illustrates the state when all lights are on when the power is turned on. Figure 39 The diagram illustrates the final state of the lamp when it is powered on.

[0473] When vehicle 1B loses power, all lights turn off.

[0474] Then, when the power of vehicle 1B is turned on, accessory lights 601L, accessory lights 601R, daytime running lights 602L, daytime running lights 602R, turn signal lights 604L, turn signal lights 604R, auxiliary lights 605FL, auxiliary lights 605FR, auxiliary lights 605BL, auxiliary lights 605BR, taillights 606L, taillights 606R, taillights 606CL and taillights 606CR are turned on.

[0475] At this point, as indicated by arrows A11 and A12 in the diagram, the lights are turned on, causing light to flow around the vehicle from the center of the front surface to the center of the rear surface.

[0476] Specifically, the logo sections on the right end of accessory light 601L and the left end of accessory light 601R are turned on for a period of time first.

[0477] Next, the LEDs turn on sequentially from right to left of the accessory light 601L. Next, the LEDs turn on sequentially from right to left of the daytime running light 602L. Next, the LEDs turn on sequentially from front to rear of the turn signal light 604L. Next, the LEDs turn on sequentially from front to rear of the auxiliary light 605FL. Next, the LEDs turn on sequentially from front to rear of the auxiliary light 605BL. Next, the LEDs turn on sequentially from left to right of the taillight 606L. Finally, the LEDs turn on sequentially from left to right of the taillight 606CL. Therefore, as indicated by arrow A11, each light turns on, causing light to flow counter-clockwise from the center of the front surface of the vehicle body to the center of the rear surface.

[0478] Similarly, the LEDs are turned on sequentially from left to right of the accessory light 601R. Next, the LEDs are turned on sequentially from left to right of the daytime running lights 602R. Next, the LEDs are turned on sequentially from front to rear of the turn signal light 604R. Next, the LEDs are turned on sequentially from front to rear of the auxiliary light 605FR. Next, the LEDs are turned on sequentially from front to rear of the auxiliary light 605BR. Next, the LEDs are turned on sequentially from right to left of the taillight 606R. Finally, the LEDs are turned on sequentially from right to left of the taillight 606CR. Thus, as indicated by arrow A12, each light is turned on, causing light to flow clockwise from the center of the front surface of the body to the center of the rear surface.

[0479] The counter-clockwise and clockwise light activation processes are performed simultaneously.

[0480] Next, the ring light 654 and the main light 751 are turned on. At this time, the ring light 654 and the main light 751 are on, so the overall brightness gradually increases.

[0481] Then, when the ring light 654 and the main light 751 are fully illuminated, the desired effect is achieved. Figure 38 The state shown in the image.

[0482] Next, some lights will be turned off, such as... Figure 39 As shown in the diagram. At this time, as indicated by arrows A13 and A14 in the diagram, the lights turn off sequentially from the front of the left side of the vehicle body and the front of the right side to the center of the rear surface.

[0483] Specifically, the LEDs turn off sequentially from the front to the rear of the turn signal light 604L. Next, the LEDs turn off sequentially from the front to the rear of the auxiliary light 605FL. Next, the LEDs turn off sequentially from the front to the rear of the auxiliary light 605BL. Next, the LEDs turn off sequentially from the left to the right of the taillight 606L. Finally, the LEDs turn off sequentially from the left to the right of the taillight 606CL. Therefore, as shown by arrow A13, each light turns off sequentially counterclockwise from the front of the left side of the vehicle body to the center of the rear surface.

[0484] Similarly, the LEDs turn off sequentially from the front to the rear of the turn signal light 604R. Next, the LEDs turn off sequentially from the front to the rear of the auxiliary light 605FR. Next, the LEDs turn off sequentially from the front to the rear of the auxiliary light 605BR. Next, the LEDs turn off sequentially from the right end to the left end of the taillight 606R. Finally, the LEDs turn off sequentially from the right end to the left end of the taillight 606CR. Therefore, as shown by arrow A14, each light turns off sequentially clockwise from the front of the right side of the vehicle body to the center of the rear surface.

[0485] The counter-clockwise and clockwise light turns off simultaneously.

[0486] <While driving and when headlights 603 are off>

[0487] Figure 40 The illustration shows an example of the illumination mode when vehicle 1B is being driven (shift position is set to drive or autonomous driving) and headlights 603 are off.

[0488] In this state, accessory lights 601L, 601R, daytime running lights 602L, and daytime running lights 602R are turned on.

[0489] It should be noted that, due to restrictions such as laws and regulations, daytime running lights 602L and 602R are always on when vehicle 1B is in operation. On the other hand, due to restrictions such as laws and regulations, accessory lights 601L and 601R need to be dimmed or turned off when vehicle 1B is being driven. Therefore, the brightness of accessory lights 601L and 601R is set lower than usual.

[0490] <While driving and when headlights 603 are on>

[0491] Figure 41The illustration shows an example of the illumination pattern when vehicle 1B is being driven (the gear shift position is set to drive or autonomous driving) and the headlights 603 are on.

[0492] Compare Figure 41 and Figure 40 The difference lies in the activation of the headlights 603L, 603R, 606L, 606R, 606CL, and 606CR. Specifically, the taillights 606L, 606R, 606CL, and 606CR activate in response to the activation of the headlights 603L and 603R.

[0493] When braking is applied

[0494] Figure 42 The illustration shows an example of the light pattern when braking is applied.

[0495] Compare Figure 42 state and Figure 40 The difference lies in the state of brake lights 607L and 607R. That is, when braking is applied, brake lights 607L and 607R are turned on.

[0496] When the turn signal is activated

[0497] Figure 43 The illustration shows an example of the illumination pattern when a left turn signal is being operated.

[0498] Compare Figure 43 and Figure 40 The difference lies in the color change of the daytime running lights 602L and taillights 606L, and the flashing of the turn signal lights 604L.

[0499] Specifically, the LEDs change from white to orange sequentially from the right end to the left end of the daytime running light 602L. After all the LEDs in the daytime running light 602L have turned orange, the LEDs from the front end to the rear end of the turn signal light 604L turn orange sequentially.

[0500] In parallel, the LEDs change from white to orange sequentially from the right end to the left end of the 606L taillight.

[0501] Next, the LEDs change from orange to white sequentially from the right end to the left end of the daytime running light 602L. After all the LEDs in the daytime running light 602L have turned white, the LEDs turn off sequentially from the front end to the rear end of the turn signal light 604L.

[0502] In parallel, the LEDs change from orange to white sequentially from the right end to the left end of the 606L taillight.

[0503] Similar operations will be repeated below.

[0504] <When door 611 is open>

[0505] Figure 44 An example of the illumination pattern is illustrated when any of the doors 611 of vehicle 1B is open.

[0506] At this time, accessory lights 601L, accessory lights 601R, daytime running lights 602L, daytime running lights 602R, taillights 606L, taillights 606R, taillights 606CL, taillights 606CR, ring lights 654, and main lights 751 are turned on.

[0507] When door 611 is closed

[0508] Figure 45 The illustration shows an example of the illumination pattern when the state of any door 611 of vehicle 1B is changed to the state of all doors 611 being closed.

[0509] Compare Figure 45 state and Figure 46 The difference lies in the state of the main light 751 being off. That is, when the door 611 is closed, the main light 751 is off.

[0510] When parking

[0511] Figure 46 The illustration shows an example of the illumination pattern when the shift position of vehicle 1B is set to park.

[0512] In this situation, accessory lights 601L, accessory lights 601R, daytime running lights 602L, daytime running lights 602R, turn signals 604L, turn signals 604R, taillights 606L, taillights 606R, taillights 606CL, taillights 606CR, and ring lights 654 are turned on.

[0513] When the power is off

[0514] Figure 47 The illustration shows an example of the illumination pattern when the power is turned off from the shift position of vehicle 1B to a parked state.

[0515] In this situation, accessory lights 601L, accessory lights 601R, daytime running lights 602L, daytime running lights 602R, turn signals 604L, turn signals 604R, taillights 606L, taillights 606R, taillights 606CL, taillights 606CR, and ring lights 654 are turned off.

[0516] At this point, as indicated by arrows A15 and A16, the lights are turned off so that they flow from the sides of the vehicle body towards the center of the front surface. Additionally, as indicated by arrows A17 and A18, the lights are turned off so that they flow from the sides of the vehicle body towards the center of the rear surface.

[0517] Specifically, the LEDs turn off sequentially from the rear to the front of the turn signal light 604L. Next, the LEDs turn off sequentially from the left to the right of the daytime running lights 602L. Next, the LEDs turn off sequentially from the left to the right of the accessory lights 601L.

[0518] Simultaneously, the LEDs turn off sequentially from the rear to the front of the turn signal 604R. Next, the LEDs turn off sequentially from the right to the left of the daytime running lights 602R. Finally, the LEDs turn off sequentially from the right to the left of the accessory lights 601R.

[0519] Simultaneously, the LEDs are turned off sequentially from left to right of taillight 606L. Next, the LEDs are turned off sequentially from left to right of taillight 606CL.

[0520] Simultaneously, the LEDs are turned off sequentially from right to left of taillight 606R. Next, the LEDs are turned off sequentially from right to left of taillight 606CR.

[0521] In addition, the ring light 654 gradually turns off.

[0522] When a person approaches the area surrounding vehicle 1B

[0523] Next, although not illustrated, an example of the illumination pattern when the recognition unit 73 detects a person approaching the vehicle 1B will be described, for example.

[0524] For example, lights near a person approaching (near the location of the approach) can be turned on or flashed. Specifically, for example, when a person is detected approaching the vicinity of door 611FL, the turn signal light 604L and the auxiliary light 605FL near door 611FL can be turned on or flashed. Moreover, for example, after the turn signal light 604L and the auxiliary light 605FL are turned on, other lights can be turned on sequentially.

[0525] Furthermore, for example, upon detecting the approach of a person, the lights can always be turned on or flashing in a similar pattern, regardless of the location of the approach. For instance, the lights can be turned on to cause light to flow around the vehicle.

[0526] Furthermore, for example, the mode of turning on or flashing the light can be changed based on the person approaching (e.g., color, pattern, etc.). For example, if the approaching person is identified as a pre-registered user (e.g., a driver, his or her family member, etc.), the white light can flash. On the other hand, for example, if the approaching person is not identified (if the approaching person is not identified as a pre-registered user), the red light can flash.

[0527] Note that, for example, the door locks can automatically unlock if the body system control unit 84 detects the approach of a registered user.

[0528] Furthermore, for example, the conditions under which the lights turn on or flash when a person approaches can be limited. For example, the lights may turn on or flash only when the power to vehicle 1B is turned off or when no one is in vehicle 1B.

[0529] In this way, when a person is detected approaching, for example by turning on or flashing the exterior lights, the vehicle 1B can be alerted to the surroundings that it is monitoring the area, and theft or vandalism of the vehicle 1B can be prevented. Furthermore, when a person is detected approaching, for example by turning on or flashing the lights inside the vehicle, it is possible to alert the surroundings that the vehicle 1B is also monitoring the interior of the vehicle.

[0530] <Lighting mode of the lighting device 652 of steering wheel 651>

[0531] Next, the illumination mode of the lighting device 652 of the steering wheel 651 will be described.

[0532] For example, the lamp control unit 85 illuminates or flashes the lighting device 652 in a certain pattern based on at least one of the following: the situation of vehicle 1B, the situation around vehicle 1B, and the situation of the occupants. The light emission pattern of the lighting device 652 is defined, for example, by at least one of color, brightness, flashing pattern, light movement, and light emission area.

[0533] Specifically, for example, when vehicle 1B is ready for autonomous driving, multiple short light strips rotate approximately one revolution around lighting device 652. Then, the entire lighting device 652 flashes. Thus, the driver can confidently recognize that it is ready for autonomous driving.

[0534] Furthermore, for example, when vehicle 1B changes lanes autonomously, the overall lighting device 652 first turns on. Then, after the lighting device 652 has turned off, the LEDs of the lighting device 652 flash in the direction in which vehicle 1B moves to change lanes. Then, after the lane change is complete, the overall lighting device 652 turns on again and then off. In this way, lane change notification is provided from before the lane change to the completion of the lane change, which can give occupants a sense of security.

[0535] Furthermore, for example, in situations that attract the driver's attention, such as when approaching an obstacle or falling asleep, the upper part of the lighting device 652 flashes red. This allows the driver to quickly detect and avoid danger.

[0536] Furthermore, for example, when voice recognition is being performed, multiple light strips move left and right on the upper part of the lighting device 652. Therefore, the driver can recognize that the voice recognition function is operating normally.

[0537] Image capture and processing

[0538] Next, we will refer to Figure 48 The flowchart describes the image capture processing performed by vehicle 1B.

[0539] This process, for example, begins when the power to vehicle 1B is turned on and ends when the power to vehicle 1B is turned off.

[0540] In step S1, camera 51 captures images of the surroundings and interior of vehicle 1B. Specifically, camera 51 includes multiple cameras, and each camera captures images of the surroundings or interior of vehicle 1B. Each camera supplies the moving image data obtained through image capture to information processing unit 701.

[0541] In step S2, the image processing unit 711 performs image processing when necessary.

[0542] For example, the image processing unit 711 overlays visual information onto each frame of the moving image data, as will be referred to later. Figures 49 to 52 Described.

[0543] Furthermore, for example, the image processing unit 711 performs image processing such as noise reduction on each frame of the moving image data based on sensor data from the in-vehicle sensor 26 and the vehicle sensor 27. For example, in the case of rain, the image processing unit 711 removes noise such as raindrops from the frame.

[0544] In step S3, the metadata adding unit 712 adds metadata to the moving image data.

[0545] For example, the metadata adding unit 712 adds metadata about at least one of the following: image capture location, image capture date and time, vehicle 1B condition, vehicle interior condition, and surrounding condition to each piece of moving image data captured during movement, etc.

[0546] Metadata regarding the image capture location includes at least one of the following: the location of vehicle 1B at the time of image capture, the image capture orientation, and the location of the camera used for image capture.

[0547] Metadata about the date and time of image capture includes at least one of the following: the date and time when the image was captured.

[0548] Metadata about vehicle 1B includes at least one of the following: vehicle 1B's speed, acceleration, direction of travel, destination, or status (e.g., whether there is a malfunction, whether there is an accident, charge level, etc.).

[0549] Metadata regarding the in-vehicle situation includes, for example, occupant identification information (e.g., name, ID, etc.), passenger seat position, occupant behavior (e.g., actions, drowsiness, etc.), voice recognition results of conversations within the vehicle, and the activity level within the vehicle. For instance, activity levels can be set based on the volume of conversations within the vehicle and occupant movement.

[0550] Metadata about the surrounding environment includes, for example, weather, temperature, humidity, brightness, the type of surrounding objects (e.g., another vehicle, pedestrian, obstacle, traffic sign, landmark, etc.), the presence and type of events (e.g., accident, construction, etc.).

[0551] Furthermore, for example, the metadata adding unit 712 can add metadata input by the user (occupant) to the moving image data. For example, the steering wheel 651 may be equipped with a movie button, and the driver can press the movie button when a scenario where moving image data is expected to be saved is being captured. For example, scenarios such as beautiful scenery around vehicle 1B or a trouble such as an accident are assumed to be scenarios where moving image data is expected to be saved. For example, the metadata adding unit 712 adds metadata indicating that moving image data captured during the period specified by the user using the movie button must be saved.

[0552] Note that the unit for providing metadata can be set arbitrarily. For example, metadata can be added in units of frames, units including multiple frames, or units of moving image data. Furthermore, depending on the type of metadata, the unit for providing metadata can be variable.

[0553] In step S4, vehicle 1B stores moving image data. Specifically, image processing unit 711 stores the moving image data in recording unit 28.

[0554] Note that at this time, the audio data recorded around and inside vehicle 1B can be stored together with the moving image data.

[0555] In step S5, vehicle 1B displays a moving image. Specifically, output control unit 713 causes HMI 31 to display a moving image based on the moving image data.

[0556] For example, Figures 49 to 52 The moving image shown is displayed on the central display 201.

[0557] For example, Figure 49The illustration shows an example of a moving image displayed when the display mode is set to object detection mode. In this example, a vehicle and a person are selected as detection targets. Boxes 801 surrounding the vehicle and 802 surrounding the person are then displayed in the captured moving image in front of vehicle 1B. Furthermore, characters indicating the type of object detected in boxes 801 and 802 (car and person) are displayed.

[0558] For example, Figure 50 The illustration shows an example of a moving image displayed when the display mode is set to prediction mode. In this example, the predicted position of a moving device (in this example, a person) detected in the moving image is shown 0.5 seconds later. In this example, the person's current position is indicated by a dotted line, and the predicted position of the person 0.5 seconds later is indicated by a solid line and enclosed in a box 811.

[0559] For example, Figure 51 The illustration shows an example of a moving image displayed when the display mode is set to search mode. Here, an example of searching for and displaying available parking locations is shown. Specifically, boxes 821 indicating available parking locations are displayed.

[0560] Note that, for example, multiple available parking locations can be displayed. Additionally, for example, available parking locations can be displayed in a bird's-eye view.

[0561] For example, Figure 52 The illustration shows an example of a moving image displayed when the display mode is set to search mode. Here, an example of searching for pre-registered people and displaying the detected people is shown. Specifically, in the moving image, boxes 831 surrounding the pre-registered people are displayed. Furthermore, the name (Sakura) of the detected person is displayed in box 831.

[0562] Note that, for example, a moving image in a direction selected by an occupant such as a driver (e.g., forward, backward, left, right, etc.) can be displayed on the central display 201. Furthermore, for example, a 360-degree moving image of the vehicle 1B can be displayed and scrolled on the central display 201.

[0563] Furthermore, sensor data obtained by, for example, radar 52 or LiDAR 53 can be imaged and displayed on the central display 201. Additionally, visual information based on data obtained by two or more of the sensors, including camera 51, radar 52, and LiDAR 53, can be overlaid and displayed on the moving image.

[0564] Moreover, for example, the planned direction of travel can be overlaid and displayed on the moving image based on the route plan.

[0565] Furthermore, for example, the movement of another vehicle, predicted by information obtained from vehicle-to-vehicle communication, can be overlaid and displayed on the moving image.

[0566] After that, the process returns to step S1, and the processes of steps S1 to S5 are repeated.

[0567] <Mobile Image Editing and Processing>

[0568] Next, we will refer to Figure 53 Describes the motion image editing process performed by vehicle 1B.

[0569] For example, this process begins when an instruction to edit a moving image is input into HMI 31. The instruction to edit a moving image includes, for example, capturing the time period of the moving image to be edited (hereinafter referred to as the editing target time period), conditions for extracting frames to be included in the moving image, etc.

[0570] In step S51, the image processing unit 711 edits the moving image.

[0571] For example, the image processing unit 711 can extract frames to be included in the edited moving image data (hereinafter referred to as the edited moving image data) from the moving image data captured within each editing target time period (hereinafter referred to as the captured moving image data) based on metadata, etc.

[0572] For example, the image processing unit 711 extracts frames to which metadata indicating that it is necessary to save is added.

[0573] In addition, for example, the image processing unit 711 extracts frames that meet given conditions based on metadata.

[0574] For example, the image processing unit 711 extracts frames based on the vehicle 1B's route and metadata within the target editing period. For example, the image processing unit 711 detects the vehicle 1B's route within the target editing period based on location information and metadata from moving image data included in the map data. Then, for example, if the vehicle 1B is traveling along a road by the sea, the image processing unit 711 prioritizes extracting frames that capture images in the direction of the sea. For example, if the vehicle 1B is traveling on a mountain (such as a mountain peak), the image processing unit 711 prioritizes extracting frames that capture images in the direction of looking down at the surroundings. Furthermore, for example, the image processing unit 711 prioritizes extracting boxes showing landmarks and the like around the vehicle 1B based on the vehicle 1B's driving direction, turning direction, etc.

[0575] For example, the image processing unit 711 extracts frames based on metadata when a specific event occurs. For example, in the case of an accident involving vehicle 1B or an accident occurring around vehicle 1B, the image processing unit 711 prioritizes extracting frames from the time zone before and after the accident. Furthermore, for example, the image processing unit 711 prioritizes extracting frames in the direction in which the accident occurred.

[0576] For example, in the case of detecting excitement in a vehicle based on activity levels included in metadata, the image processing unit 711 prioritizes extracting frames of images captured inside the vehicle. Furthermore, for example, the image processing unit 711 also extracts audio data corresponding to the frames extracted from the vehicle.

[0577] Furthermore, for example, the output control unit 713 can display a list of moving image data and actual moving images on a central display 201, a tablet terminal 662L, etc., and the image processing unit 711 can edit the moving images based on user instructions.

[0578] For example, a user can select moving image data or frames he or she wants to include in the edited moving image data. In this case, for example, various information, visual effects, and visual information such as graffiti can be overlaid on frames of moving images displayed on the central display 201 and the tablet terminal 662L.

[0579] Then, the image processing unit 711 generates edited motion image data by, for example, combining the extracted frames. For example, the image processing unit 711 generates edited motion image data by connecting the extracted frames in a time sequence, arranging or overlaying multiple frames in the same frame, etc. Furthermore, the image processing unit 711 may, when necessary, overlay visual information onto the extracted frames, for example, based on metadata.

[0580] In step S52, vehicle 1B saves the moving image data. For example, image processing unit 711 stores the edited moving image data in recording unit 28. Alternatively, for example, image processing unit 711 transmits the edited moving image data to a server or an information processing terminal owned by the occupant (e.g., smartphone, tablet, personal computer, etc.) via communication unit 22 and stores the edited moving image data.

[0581] After this, the image editing process is complete.

[0582] In this way, it is possible to easily edit captured moving images of the surroundings and interior of vehicle 1B. Therefore, it is possible to easily generate, for example, moving image data recording memories of a trip, moving image data obtained by extracting beautiful scenery, moving image data recording accident situations, etc.

[0583] <<5. Modification Examples of the Second Embodiment>>

[0584] In the following text, a modified example of the second embodiment of the present technology will be described.

[0585] For example, captured moving images, with metadata added before editing, can be stored or copied outside of vehicle 1B, and devices outside vehicle 1B (e.g., servers (cloud), smartphones, tablets, or personal computers) can edit the moving images. Furthermore, for example, devices outside vehicle 1B and vehicle 1B can jointly edit the moving images. Moreover, for example, metadata can be added by devices outside vehicle 1B.

[0586] Furthermore, as described above, it has been stated that the front and rear lines of loop L1 are illuminated; however, for example, considering design, laws, and regulations, the front and rear lines may not be illuminated.

[0587] Furthermore, the above description has already illustrated an example of separating the central part of the front line and the central part of the tail line. However, for example, considering design considerations, the central part of the front line and the central part of the tail line can be connected without separation.

[0588] In addition, for example, optical sensors such as cameras, radar or LiDAR can be arranged in the headlight 603.

[0589] Furthermore, while the above description has illustrated vehicle 1B as an example of a left-hand drive vehicle, this technology can certainly be applied to right-hand drive vehicles. When applying this technology to a right-hand drive vehicle, the exterior and interior layout of the vehicle can be appropriately modified according to the vehicle's orientation.

[0590] Furthermore, there are no particular restrictions on the types of vehicles to which this technology can be applied. In addition to vehicles, this technology can also be applied to mobile devices such as personal mobility devices, aircraft, ships, construction machinery, and agricultural machinery. Moreover, mobile devices to which this technology can be applied include, for example, mobile devices that capture images of their surroundings without a human presence, such as drones or robots.

[0591] <<6. Third Embodiment>>

[0592] Next, we will refer to Figures 54 to 105 A third embodiment of this technology is described.

[0593] Note that in each figure of the third embodiment, the same reference numerals are given to the parts corresponding to those in the first and second embodiments, and descriptions thereof will be appropriately omitted. Furthermore, except where alternative configurations are illustrated or described, and where such components are described as having been removed, components included in the first and second embodiments but not shown or described in the third embodiment are generally assumed to be included in the third embodiment.

[0594] <Example of interior configuration of vehicle 1C>

[0595] First, refer to Figures 54 to 57 A configuration example describing the interior of vehicle 1C.

[0596] Figure 54 This is a schematic diagram of the interior of vehicle 1C viewed from right to left. Figure 55 This is a schematic diagram of the front interior of vehicle 1C. Figure 56 This is a schematic diagram of the area in front of the driver's seat 101 inside the vehicle 1C. Figure 57 This is an enlarged view of the area near the steering wheel 1001 in vehicle 1C.

[0597] exist Figure 54 In the diagram, the parts of vehicle 1C that are significantly different from those of vehicles 1A and 1B are circled. Specifically, the steering wheel 1001, camera module 1007, and camera module 1008 are the significantly different parts.

[0598] like Figure 57 As shown, with Figure 30 Compared to steering wheel 651, the lighting device 1002 of steering wheel 1001 is positioned differently. The lighting device 1002 includes multiple LEDs arranged in a ring on the inner side of the central portion of the steering wheel 1001 at a predetermined distance from its outer periphery. The on / off state, color, and brightness of each LED can be individually controlled. Therefore, the lighting device 1002 has variable color, brightness, and luminous area (luminous range). The logo of vehicle 1C is formed inside the lighting device 1002.

[0599] Note that the airbag (not shown) is housed in the central portion of the steering wheel 1001. The lighting device 1002 is then arranged to avoid the portion of the central portion of the steering wheel 1001 that would be exposed when the airbag is activated. This prevents debris and harmful substances from the lighting device 1002 from dispersing when the airbag is activated.

[0600] The left spoke of the steering wheel 1001 is equipped with an operation unit 1003 having multiple buttons. The right spoke of the steering wheel 1001 is equipped with an operation unit 1004 having multiple buttons.

[0601] like Figure 57As shown, the joystick 1005 as a rod-shaped operating body is provided behind the steering wheel 1001 and extends leftward from a steering column (not shown). Further, the joystick 1006 as a rod-shaped operating body is provided behind the steering wheel 1001 and extends rightward from a steering column (not shown). For example, the joystick 1006 has a configuration similar to the configuration of the joystick 655 shown in Figure 30 As shown.

[0602] As Figure 54 , Figure 55 and Figure 57 As shown, the camera module 1007 is provided on the instrument panel 105 and is closer to the driver's seat 101 than the center in the left-right direction. The camera module 1007 has a shape as if a part of a frustum of a cone is vertically cut off and has a vertically flat portion on the side surface. The flat portion on the side surface of the camera module 1007 points toward the driver's seat 101, and as described later, the camera module 1007 captures an image of at least a range including the head of the driver sitting on the driver's seat 101.

[0603] As Figure 55 As shown, the camera module 1008 is provided at the front end of the ceiling of the vehicle 1C and at the center in the left-right direction near the digital rearview mirror 204. The camera module 1008 has two ToF cameras (not shown). The ToF cameras respectively capture images of the occupants on the driver and passenger seats 102, as will be described later.

[0604] The circular lamp 1009 is provided on the lower surface of the camera module 1008. Further, the camera module 1008 incorporates a built-in microphone (not shown) for collecting sounds of the occupants and the like.

[0605] Further, in this example, as Figure 55 As shown, the central portion 201C of the central display 201 is divided into a display unit 201CL in front of the driver's seat 101, a display unit 201CC between the driver's seat 101 and the passenger seat 102, and a display unit 201CR in front of the passenger seat 102. Note that, as described above, it is also possible to connect the display unit 201CL, the display unit 201CC, and the display unit 201CR to form one display unit. The display unit 201LL is provided on the left end portion 201L of the central display 201. The display unit 201RR is provided on the right end portion 201R of the central display 201.

[0606] <Examples of Mounting Positions of ToF Cameras>

[0607] Next, examples of the mounting positions of the ToF cameras will be described with reference to Figures 58 to 63 As shown.

[0608] Figure 58 The illustration shows an example of the mounting location of the ToF camera in camera module 1008. Figure 58 The top left view is a schematic diagram of the camera module 1008 when viewed from a diagonal left-downward direction. Figure 58 The bottom left view is a schematic diagram of the camera module 1008 when viewed from the bottom left. Figure 58 The right view is a schematic diagram of the camera module 1008 when viewed from a diagonal downward angle from the front.

[0609] The camera module 1008 has a shape in which a platform with a semi-conical truncated cone is connected to the rear end (rear side of vehicle 1C) of a platform obtained by obliquely cutting off the left and right sides of a cuboid. As described above, the camera module 1008 is arranged at the front end of the roof of vehicle 1C and at the center in the left-right direction.

[0610] The ToF camera (hereinafter referred to as ToF camera P101L) is located near the front end of the left side of the frustum portion of the camera module 1008. For example, when a driver of standard build is sitting in the driver's seat 101 arranged in a standard position, the optical axis of the ToF camera P101L is pointed towards the center of the driver's eyes.

[0611] Figure 59 The illustration shows an example of the image capture range of the ToF camera P101L. Regardless of the size of the driver DR1 and the position of the driver's seat 101, the ToF camera P101L can capture images of almost the entire body of the driver DR1 from head to toe in an oblique upper right direction.

[0612] The ToF camera (hereinafter referred to as ToF camera P101R) is located in the position P101R near the front part of the right side of the truncated cone portion of the camera module 1008. Although not shown, like the ToF camera P101L, the ToF camera P101R can capture images of almost the entire body of the occupant from head to toe in an oblique upper-left direction, regardless of the occupant's size and the position of the passenger seat 102.

[0613] A glossy black cover 1008A is provided on the side of the truncated cone portion of the camera module 1008. The cover 1008A makes it difficult to see the lenses of the ToF cameras P101L and P101R from the outside. Furthermore, it is possible to house cameras and sensors other than the ToF cameras P101L and P101R on the back of the cover 1008A.

[0614] For example, DMS 30 ( Figure 1The driver's posture and movement are identified by recognizing the driver's skeleton, etc., based on images from the ToF camera P101L. For example, the DMS 30 performs driver recognition processing (e.g., personal recognition) based on images from the ToF camera P101L.

[0615] Similarly, for example, DMS 30 identifies the occupant's posture and movement by recognizing the skeletal structure of the occupant on the passenger seat 102 based on images from the ToF camera P101R. For example, DMS 30 performs occupant recognition processing based on images from the ToF camera P101L.

[0616] Next, we will refer to Figures 60 to 63 Another example describing the installation location of a ToF camera.

[0617] For example, such as Figure 60 As shown in the top view, the ToF camera (hereinafter referred to as ToF camera P101C) is integrated in the rear portion of the camera module 1008 at location P101C.

[0618] Figure 60 The bottom view illustrates an example of the image capture range of the ToF camera P101C. As described above, the ToF camera P101C can capture images of regions including area A101L containing the headrest of rear seat 103L and area A101R containing the headrest of rear seat 103R. Therefore, the ToF camera P101C can simultaneously capture images of the upper body including the head of the occupant in rear seat 103L and the upper body including the head of the occupant in rear seat 103R.

[0619] Note that, as Figure 60 As shown, the aforementioned seat speakers are embedded under the headrest of each seat. Specifically, seat speaker 1021 is embedded under the headrest of the driver's seat 101. Seat speaker 1022 is embedded under the headrest of the passenger seat 102. Seat speaker 1023L is embedded under the headrest of the left rear seat 103L. Seat speaker 1023R is embedded under the headrest of the right rear seat 103R.

[0620] Furthermore, although not illustrated, the above references... Figure 27 and Figure 28 The described speakers 653FL to 653BR are embedded in the corresponding doors to achieve 360-degree true audio.

[0621] Figures 61 to 63 The illustration shows an example of the mounting position of a ToF camera relative to the rear seat 103L.

[0622] For example, a ToF camera (hereinafter referred to as ToF camera P111L) is positioned at P111L on the roof of vehicle 1C, directly above the frame above the upper part of door 611BL. ToF camera P111L can capture an image of almost the entire body of the occupant in the rear seat 103L, for example, from an oblique, upper left direction. Furthermore, a reading light (not shown) capable of emitting illumination within a range A111 is provided around ToF camera P111L.

[0623] For example, a ToF camera (hereinafter referred to as ToF camera P112L) is provided at a position P112L near the center of the upper end of the tablet terminal 662L in the width direction. The ToF camera P112L can, for example, capture images from the front including the upper body of the occupant in the rear seat 103L.

[0624] Note that, although not illustrated, ToF cameras are positioned at positions P111R and P112R relative to the rear seat 103R, positions similar to those at positions P111L and P112L relative to the rear seat 103L. In the following text, the ToF camera positioned at P111R will be referred to as ToF camera P111R, and the ToF camera positioned at P112R will be referred to as ToF camera P112R.

[0625] For example, DMS 30 identifies the occupant's posture and movement by performing skeletal recognition, etc., of the occupant in the rear seat 103L based on images from at least one of the ToF cameras P111L and P112L. For example, DMS 30 performs occupant recognition processing based on images from at least one of the ToF cameras P111L and P112L. For example, DMS 30 performs lip reading of the occupant based on images from the ToF camera P112L.

[0626] Similarly, for example, DMS 30 identifies the occupant's posture and movement by recognizing the skeleton, etc., of the occupant in the rear seat 103R based on images from at least one of the ToF cameras P111R and P112R. For example, DMS 30 performs occupant recognition processing based on images from at least one of the ToF cameras P111R and P112R. For example, DMS 30 performs lip reading of the occupant based on images from the ToF camera P112L.

[0627] For example, such as Figure 61As shown, the lamp control unit 85 controls the reading light's on / off state and brightness based on the actions of the occupant in the rear seat 103L. Furthermore, for example, the lamp control unit 85 controls the brightness of the illumination range A112 of the ring light 654 near the rear seat 103L based on the actions of the occupant in the rear seat 103L. For example, the lamp control unit 85 turns on the reading light or brightens the illumination range A112 of the ring light 654 when it detects the occupant reading a document.

[0628] Note that, for example, lighting controls similar to those for the rear seat 103L are implemented in passenger seat 102 and rear seat 103R.

[0629] For example, such as Figure 62 As shown, the DMS 30 can recognize the position and shape of the left ear EL1 (not shown) and right ear ER1 of the occupant DR2 in the rear seat 103L based on the image of the ToF camera P111L. On the other hand, the audio control unit, as part of the function of the HMI 31, controls the seat speaker 1023L of the rear seat 103L to control the sound image, sound field, volume, etc. of various types of sound for the occupant DR2 corresponding to the position and shape of the occupant DR2's ears.

[0630] Note that, for example, in the driver's seat 101, passenger seat 102 and rear seat 103R, the sound image, sound field, volume and other aspects of various types of sounds for the occupants in each seat are controlled similarly to those in the rear seat 103L.

[0631] For example, such as Figure 63 As shown in the lower right view, the vehicle body system control unit 84 controls the position and posture of the rear seat 103R and the position of the tablet terminal 662R based on the physique of the occupant on the rear seat 103R. For example, the rear seat 103R is deeply reclined according to the occupant's physique, and the tablet terminal 662R moves in a direction close to the rear seat 103R as indicated by arrow A113.

[0632] Note that, similarly, the position and posture of the rear seat 103L and the tablet terminal 662L are controlled based on the physique of the occupant in the rear seat 103L.

[0633] <Details of Camera Module 1007>

[0634] Next, we will refer to Figures 64 to 67 Describe the details of camera module 1007.

[0635] Figure 64 An example of the mounting location of the camera module 1007 is illustrated. As described above, the camera module 1007 is positioned on the dashboard 105, closer to the driver's seat 101 in the left-right direction than the center.

[0636] Figure 65 The illustration shows a configuration example inside the camera module 1007.

[0637] The camera module 1007 includes a housing 1041, a camera 1042, LEDs 1043-1 and 1043-2, and a cover 1044.

[0638] Note that in the following text, unless there is a need to distinguish between LED 1043-1 and LED1043-2, they will be referred to as LED 1043.

[0639] Camera 1042, along with LEDs 1043-1 and 1043-2, are arranged horizontally within housing 1041. Camera 1042 is positioned closer to the driver's seat 101 than LEDs 1043-1 and 1043-2. ​​The optical axis of camera 1042 is substantially parallel to the optical axes of LEDs 1043-1 and 1043-2 and each faces the driver's seat 101. Camera 1042 can capture images that include at least the area from the top of the driver's head to their Adam's apple, regardless of the driver's physique or the position of the driver's seat 101.

[0640] Camera 1042 is capable of performing both RGB image capture (color image capture) and IR image capture (infrared image capture). Camera 1042 automatically switches between RGB image capture and IR image capture, for example, based on conditions such as ambient brightness.

[0641] LED 1043 is the light source that outputs IR light (infrared light). By setting two LEDs 1043, LED 1043-1 and LED 1043-2, it is possible to suppress the amount of light emitted from each LED 1043 and suppress the heat generated by each LED 1043, while ensuring the amount of IR light.

[0642] Cover 1044 is a flat portion of the side of housing 1041 and is positioned in front of camera 1042 and each LED 1043. Cover 1044 makes it difficult to see the lens of camera 1042 and the light-emitting portion of each LED 1043 from the outside.

[0643] For example, in bright environments such as daytime, camera 1042 automatically performs RGB image capture and acquires a color image. On the other hand, in dark environments such as nighttime, each LED 1043 automatically turns on, and camera 1042 automatically performs IR image capture and acquires a monochrome image. Therefore, an image of the driver's head can be reliably captured regardless of the ambient brightness.

[0644] For example, DMS 30 performs driver lip reading and recognizes the content of the driver's speech based on images from camera 1042. DMS 30 also performs driver iris detection, gaze detection, arousal determination, and recognition processing based on images from camera 1042, for example.

[0645] Next, we will refer to Figure 66 Example of modifying the mounting location of camera 1042.

[0646] Figure 66 The left view is a schematic diagram taken from a right-angled top view of the area near the steering wheel 1001 of vehicle 1C. Figure 66 The right view is a schematic diagram of the area near the front of the driver's seat 101.

[0647] For example, it is conceivable to mount the camera 1042 at position P121 on the dashboard 105 in front of the driver's seat 101. In this case, an image of the driver's face can be captured from the front. On the other hand, when the steering wheel 1001 is rotated, the spokes of the steering wheel 1001 obstruct the field of view of the camera 1042.

[0648] For example, it is conceivable to mount camera 1042 at a position P122 near the left end of instrument panel 105. In this case, the driver's facial image is captured from the left diagonal front, compared to camera module 1007.

[0649] For example, it is conceivable to mount the camera 1042 at position P123 on the steering column of the steering wheel 1001. In this case, an image of the driver's face can be captured from the front at a short distance. On the other hand, as in position P121, when the steering wheel 1001 is rotated, the spokes of the steering wheel 1001 obstruct the field of view of the camera 1042. Furthermore, the driver's field of view is obstructed relative to the display unit 201CL of the central display 201.

[0650] For example, it is conceivable that camera 1042 is combined at position P124 on the right side of steering wheel 1001 and near the upper end of the central portion 201C of central display 201. In this case, the driver's facial image is captured from the right oblique front. On the other hand, since the storage space of camera 1042 is limited, it is necessary to reduce the size of camera 1042. Furthermore, camera 1042 and LED 1043 can be arranged separately from each other.

[0651] For example, it is conceivable to incorporate the camera 1042 at the position P125 in the upper left corner of the left end portion 201L of the center display 201. In this case, the facial image of the driver is captured from the left front obliquely. On the other hand, since the storage space of the camera 1042 is limited, it is necessary to reduce the size of the camera 1042. In addition, the camera 1042 and the LED 1043 can be arranged separately from each other.

[0652] For example, it is conceivable to place a smart phone in the central portion of the steering wheel 1001 and capture an image of the driver's face with the smart phone. In this case, an image of the driver's face can be captured from the front. On the other hand, when the airbag is activated, the smart phone may be damaged. In addition, when the steering wheel 1001 rotates, the image of the driver's face becomes inclined when captured.

[0653] For example, it is conceivable to incorporate the camera 1042 in the A pillar (not shown) on the driver's seat side of the vehicle 1C. In this case, the facial image of the driver is captured from the left upper obliquely. On the other hand, since the storage space of the camera 1042 is limited, it is necessary to reduce the size of the camera 1042. In addition, the camera 1042 and the LED 1043 can be arranged separately from each other.

[0654] For example, if the camera 1042 can be sufficiently widened, the camera module 1008 can be placed in the center of the instrument panel 105 in the left - right direction such that the optical axis of the camera 1042 faces the rear of the vehicle 1C.

[0655] Figure 67 An example of the image capture range of the camera 1042 in this case is illustrated. In this example, the entire headrest of the driver's seat 101, the entire headrest of the passenger seat 102, a part of the headrest of the rear seat 103L, and a part of the headrest of the rear seat 103R are included in the image capture range. Therefore, images of the entire head of the driver, the entire head of the occupant on the passenger seat 102, a part of the head of the occupant on the rear seat 103L, and a part of the head of the occupant on the rear seat 103R can be captured with one camera 1042.

[0656] <CMS Operation Example>

[0657] Next, reference will be made to Figures 68 to 85 Describe an operation example of the CMS of the vehicle 1C. Hereinafter, a display example of the display unit 201LL in the left end portion 201L of the center display 201 will be mainly described.

[0658] For example, as part of the functionality of HMI 31, the display control unit causes the display unit 201LL to display an image based on the image from camera 51SL. More specifically, the display control unit sets a display area within the image from camera 51SL and causes the display unit 201LL to display the image within the set display area.

[0659] Furthermore, the recognition unit 73 performs object recognition processing based on images from the camera 51SL, etc. The display control unit controls the display unit 201LL to display based on the object recognition results, the vehicle 1C status, etc.

[0660] Figure 68 The illustration shows a display example of the display unit 201LL in its normal state. In this example, a portion of the vehicle body of vehicle 1C, a following vehicle 1101 traveling in the left adjacent lane diagonally behind vehicle 1C, and road surface 1102 are displayed.

[0661] Then, for example, when vehicle 1101 approaches vehicle 1C and the distance between vehicle 1101 and vehicle 1C is within a predetermined range, such as... Figure 69 As shown diagonally, a vertically elongated visual effect, bar 1103 is superimposed and displayed at the left end of display unit 201LL. For example, bar 1103 is displayed when vehicle 1101 is within the range shown by display unit 201LL and the distance from vehicle 1C is within a predetermined range. The color of bar 1103 is a prominent color, such as yellow.

[0662] The display of bar 1103 can alert the driver and reliably prevent accidents.

[0663] Note that the display mode of bar 1103 (such as color, brightness, shape, blinking mode, and movement) is not limited to this example and can be changed arbitrarily. For example, the display mode of bar 1103 may be changed according to the level of risk.

[0664] For example, if the left-hand direction indicator of vehicle 1C is activated, vehicle 1C may turn left (change direction to the left) or change lanes to the left lane, increasing the risk of collision or contact with vehicle 1101. Therefore, for example, as... Figure 70 As shown, bar 1103 widens and flashes. Therefore, it can provide more alertness to the driver.

[0665] Furthermore, for example, when the direction indicator for the left direction of vehicle 1C is activated, the flashing speed of bar 1103 changes according to the level of danger. For example, the faster the absolute speed of vehicle 1101 or its relative speed with respect to vehicle 1C, the faster bar 1103 flashes. For example, the closer vehicle 1101 is, in other words, the shorter the distance between vehicle 1101 and vehicle 1C, the faster bar 1103 flashes. Note that, for example, a warning sound may be activated synchronously with the flashing of bar 1103.

[0666] For example, if the absolute speed of vehicle 1101 or its relative speed with respect to vehicle 1C is equal to or greater than a predetermined threshold, the timing of display bar 1103 will be earlier. That is, bar 1103 will be displayed from a point in time when vehicle 1101 is farther away from vehicle 1C.

[0667] In addition, for example, such as Figure 71 As shown, at the point in time when vehicle 1101 is identified to the left and slightly behind, a marker 1104, serving as a triangular visual effect indicating the presence of vehicle 1101, can be superimposed and displayed near vehicle 1101. The marker 1104 moves as vehicle 1101 moves.

[0668] Therefore, for example, even in cases where the visibility of the image displayed on the display unit 201LL is poor due to fog, haze, or other conditions, the driver can reliably identify the position of the vehicle 1101.

[0669] Note that the location notification for vehicle 1101 can be obtained through [communication / communication]. Figure 71 The display mode for marker 1104 is given in different display modes. For example, the frame surrounding vehicle 1101 can be displayed.

[0670] Note that, although detailed descriptions are omitted, the display unit 201RR of the right-hand portion 201R of the central display 201 performs a similar display based on the situation of vehicles traveling in the right adjacent lane of vehicle 1C and the operation of the right-hand direction indicator.

[0671] Furthermore, while the above description illustrates an example of drawing attention to another vehicle, similar processing is performed when drawing attention to a moving object other than a vehicle (e.g., a motorcycle, etc.).

[0672] Furthermore, for example, the display control unit controls the range of the image displayed on the display unit 201LL based on the operation of the direction indicator.

[0673] Figure 72 The illustration shows an example of changing the display range of an image displayed on display unit 201LL based on the operation of the direction indicator in the left direction.

[0674] Specifically, Figure 72 Figures A and B in the diagram illustrate examples of images captured by camera 51SL. The image shows vehicles 1121-1 through 1121-4, etc., traveling in the left lane of vehicle 1C.

[0675] For example, when the direction indicator for the left direction of vehicle 1C is off, extract Figure 72 The image in region A131a is displayed on display unit 201LL, and the image in region A132a is displayed on display unit 201LL. Region A132a includes vehicles 1121-2 to 1121-4, but does not include vehicle 1121-1.

[0676] On the other hand, with the left-hand direction indicator of vehicle 1C activated, extraction... Figure 72 The image in region A131b of B is displayed, and the image in region A132b is displayed on display unit 201LL. Regions A131b and A132b are wider than regions A131a and A132a. That is, the display range of display unit 201LL is zoomed out and expanded. Then, in addition to vehicles 1121-2 to 1121-4, the rear end portion of vehicle 1121-1 is also included in region A132b.

[0677] Therefore, when the left-hand turn indicator is activated, the driver can perceive the situation in the left lane of vehicle 1C over a wider range. This allows the driver to change lanes or turn left more safely.

[0678] Note that, for example, when the display range of the display unit 201LL is extended, icons, boxes, etc. indicating that it has been extended can be displayed on the display unit 201LL.

[0679] Here, Article 46 of the United Nations Economic Commission for Europe (UN ECE-R46) defines the range of the field of view that is always visible in the CMS (hereinafter referred to as the required field of view). The required field of view varies depending on the vehicle's seating capacity, weight, etc., and Figure 73 An example is shown in the illustration.

[0680] In this example, the required field of view includes regions A133L, A133R, and A133C.

[0681] Area A133L is the region enclosed by points P132L to P136L on the left side of vehicle 1C. Specifically, the leftmost point of vehicle 1C corresponding to the position of the driver's eye in the front-rear direction is set as reference point P131L. Point P132L is 4m behind reference point P131L. Point P133L is 60m behind reference point P131L. Point P134L is 1m to the left of point P132L. Point P135L is 20m behind reference point P131L and 4m to the left of reference point P131L. Point P136L is 60m behind reference point P131L and 4m to the left of reference point P131L.

[0682] Area A133R is the region enclosed by points P132R to P136R on the right side of vehicle 1C. Specifically, the rightmost point of vehicle 1C, corresponding to the driver's eye position in the front-rear direction, is designated as reference point P131R. Point P132R is 4m behind reference point P131R. Point P133R is 60m behind reference point P131R. Point P134R is 1m to the right of point P132R. Point P135R is 20m behind and 4m to the right of reference point P131R. Point P136R is 60m behind and 4m to the right of reference point P131R.

[0683] Area A133C is an area 60m or more away from the driver's eye position in the left-right direction of vehicle 1C, and with a width of 20m.

[0684] Therefore, for example, such as Figure 72 As shown, when the display range of display unit 201LL changes according to the operation of the direction indicator in the left direction, the display range is changed so that region A133L is always included.

[0685] Note that, although detailed descriptions are omitted, the display unit 201RR of the central display 201 also controls a similar display range based on the operation of the direction indicator in the right direction. Furthermore, the display range of the display unit 201RR is adjusted so that area A133R is always included.

[0686] In addition, for example, the display control unit controls the display range of the display unit 201LL based on the shift position of the vehicle 1C.

[0687] Figure 74 The illustration shows an example of the display range of the display unit 201LL, which changes the shift position based on vehicle 1C.

[0688] Specifically, Figure 74 Figures A and B in the diagram illustrate examples of images captured by camera 51SL. The image shows vehicles 1141-1 and 1141-2, etc., parked side by side to the left rear of vehicle 1C.

[0689] For example, before the shift position of vehicle 1C is set to reverse, extract... Figure 74 The image in region A141a of A is displayed on display unit 201LL, and the image in region A142a is displayed on display unit 201LL.

[0690] On the other hand, after the shift position of vehicle 1C is set to reverse, extraction Figure 74 The image in region A141b of the vehicle 1C is displayed on display unit 201LL, and the image in region A142b is displayed on display unit 201LL. Regions A141b and A142b are wider than regions A141a and A142a. That is, the display range of display unit 201LL is extended and expanded. At this time, for example, the display range of display unit 201LL can automatically move downward to display the ground around vehicle 1C more widely.

[0691] Note that, although detailed descriptions are omitted, the display range in the display unit 201RR of the central display 201 also changes similarly to the display range in the display unit 201LL, based on the gear shift position of the vehicle 1C.

[0692] This allows the driver to inspect the area diagonally to the left rear of vehicle 1C from a wider range. Therefore, the driver can park vehicle 1C more safely.

[0693] In addition, for example, the driver can operate Figure 75 The operation screen displayed on the console monitor 202 is used to change the display range of the display unit 201LL and the display unit 201RR.

[0694] This operation screen displays buttons 1151, 1152, touchpad 1153, 1154, and 1155.

[0695] When button 1151 is pressed, the display area of ​​the left display unit 201LL becomes the setting target. When button 1152 is pressed, the display area of ​​the right display unit 201RR becomes the setting target.

[0696] When the touchpad 1153 is operated, the display areas of the display units 201LL and 201CC move. That is, the display areas of the display units 201LL and 201CC move in the direction in which the touchpad 1153 is pressed.

[0697] When button 1154 is pressed, the currently set display range is stored.

[0698] When button 1155 is pressed, the change in the display range is canceled and the display returns to its previous state.

[0699] In addition, for example, the scaling ratio (scaling factor) of the display range of display unit 201LL and display unit 201RR is set by another button (not shown) on console display 202.

[0700] Here, the display range can be set Figure 76 The settable range A151 is shown. Note that, within... Figure 76 In, with Figure 72 The corresponding parts are indicated by the same reference numerals, and descriptions of them will be omitted as appropriate.

[0701] For example, the size of the extraction range A152 extracted from the captured image is set for display based on a set scaling ratio. That is, the extraction range A152 is zoomed in (decreased) and zoomed out (expanded) based on the scaling ratio. Then, the image in the predetermined display range A153 within the extraction range A152 is displayed on the display unit 201LL.

[0702] At this time, the display range A153 can be moved within the settable range A151. The settable range A151 is set such that region A133L is included within the display range A153.

[0703] On the other hand, for example, such as Figure 76 As shown in B, if the area A133L extends beyond the display range A153, a warning is issued to the driver. For example, a warning message is displayed on the display unit 201LL and a warning sound is output.

[0704] Alternatively, for example, the range of motion of the display range A153 can be limited so that the display range A153 does not exceed the settable range A151 regardless of the driver's operation.

[0705] Figure 77 The illustration shows an example of the display range of display unit 201LL when the scaling ratio changes from 0.8 to 1.3 in 0.1 increments.

[0706] As the scaling ratio decreases, the display area becomes larger, making it possible to confirm a wider range. On the other hand, the size of objects displayed on the display unit 201LL, such as following vehicles, becomes smaller.

[0707] At the same time, as the scaling ratio becomes larger, the display area becomes smaller and the size of objects displayed on the display unit 201LL, such as following vehicles, becomes larger. On the other hand, the number of pixels in the extracted image decreases, and the image sharpness decreases.

[0708] Therefore, for example, the adjustable range of the scaling ratio of the display unit 201LL on the driver's seat side is set to 1.0 to 1.2. The adjustable range of the scaling ratio of the display unit 201RR on the passenger seat side is set to 1.0 to 1.1.

[0709] Here, the adjustable range of the scaling ratio of the display unit 201LL on the driver's seat side differs from the adjustable range of the scaling ratio of the display unit 201RR on the passenger seat side. This is because the distance from the driver to the display unit 201LL is different from the distance to the display unit 201CC, and the direction of the driver's line of sight relative to the display unit 201LL is different from the direction of the driver's line of sight relative to the display unit 201CC. That is, for the driver, the appearance of the display unit 201LL is different from the appearance of the display unit 201CC.

[0710] Furthermore, as described above, when the direction indicator is in operation or when the shift position is set to reverse, the scaling ratio of display units 201LL and 201CC is reduced. Based on this assumption, the lower limit of the settable range of the scaling ratio is limited to 1.0. For example, without changing the aforementioned scaling ratio, the lower limit of the settable range of the scaling ratio can be reduced to 0.8 or 0.9.

[0711] In this way, by operating the console display 202, the driver can move the display range A153 to his or her preferred position within the area A133L, and can zoom in or out on the display range A153. Furthermore, for example, the driver can easily adjust the display range A153 without distracting his or her gaze by moving his or her finger or tapping the console display 202 while maintaining a driving posture.

[0712] Furthermore, for example, in the case of physical side mirrors, a driver can change the range of visibility through the side mirror by bringing his or her face close to the mirror and looking at it. For example, a driver can check for obstacles on the ground around the vehicle by looking at the side mirror when parking.

[0713] On the other hand, the display range of the display unit 201LL can be similarly changed by the driver taking actions such as moving his or her face closer to the display unit 201LL and looking at the display unit 201LL.

[0714] For example, DMS 30 detects the position and orientation of the driver's head by recognizing the driver's skeleton, etc., based on images from the ToF camera P101L. When the driver's gaze at the display unit 201LL is detected based on the position and orientation of the driver's head, DMS 30 detects the angle at which the driver gazes at the display unit 201LL.

[0715] On the other hand, when the vehicle 1C is stopped, that is, when the speed of the vehicle 1C is zero, the display control unit controls the display range of the display unit 201LL based on the angle at which the driver looks at the display unit 201LL.

[0716] For example, Figure 78 The illustration shows the display area of ​​display unit 201LL before the driver looks at it. In this example, a portion of the vehicle body 1C is displayed in the lower right corner of display unit 201LL. Additionally, the upper part of object 1161 to the left rear of vehicle 1C is also shown.

[0717] In this situation, for example, when the driver takes an action such as looking at the display unit 201LL from an upward angle, the display range of the display unit 201LL is as follows: Figure 79 The change is shown in the figure. Specifically, the display area of ​​the display unit 201LL is moved downward. Therefore, the lower part of the object 1161 is also displayed on the display unit 201LL, and the driver can visually recognize the entire object 1161.

[0718] Furthermore, as shown diagonally in the view, bar visual effect lines 1162 indicating the ground area within a predetermined range around vehicle 1C are superimposed, displayed, and flashed. Therefore, the driver can quickly identify objects that vehicle 1C may collide with or come into contact with, and move vehicle 1C while avoiding the identified objects.

[0719] In addition, for example, Figure 80 The illustration shows the display area of ​​display unit 201LL before the driver's gaze. In this example, a portion of the body of vehicle 1C is displayed at the right end of display unit 201LL. A portion of vehicle 1181 to the left rear is displayed at the upper left of display unit 201LL. A parking line 1182 is displayed between vehicle 1C and vehicle 1181. The upper end of object 1183 is displayed to the left of vehicle 1C and at the lower end of display unit 201LL.

[0720] Here, for example, when the driver sets the gear shift to reverse in order to park vehicle 1C, the display range of display unit 201LL is extended, as referenced above. Figure 74 As stated above. Therefore, as Figure 81As shown, the display range of display unit 201LL is expanded, and the display range of vehicle 1C, vehicle 1181 and object 1183 is expanded.

[0721] Furthermore, for example, when the driver performs an action such as looking at the display unit 201LL from an upward angle, the display range of the display unit 201LL shifts downward, such as... Figure 82 As shown in the diagram. Therefore, the entire object 1183 is displayed on the display unit 201LL. Thus, the driver can park the vehicle 1C while reliably avoiding the object 1183.

[0722] Note that, although detailed descriptions are omitted, the display range moves similarly to the display range of the display unit 201LL when the driver is looking at the display unit 201RR of the central display 201.

[0723] Furthermore, for example, when the driver brings his or her face close to and looks at the digital rearview mirror 204, the display range of the digital rearview mirror 204 also shifts.

[0724] For example, DMS 30 detects the angle at which the driver is looking at the digital rearview mirror 204 based on the position and direction of the driver's head.

[0725] On the other hand, when the vehicle 1C is stopped, that is, when the speed of the vehicle 1C is zero, the display control unit controls the display range of the digital rearview mirror 204 based on the angle at which the driver is looking at the digital rearview mirror 204.

[0726] Moreover, for example, such as Figure 83 As shown, when the driver DR1 is detected looking down at his or her feet, the light control unit 85 can turn on the footlights (not shown) that illuminate the driver's feet. Therefore, for example, if the driver drops the key or other items at his or her feet, the driver can easily find the dropped key or other items.

[0727] Furthermore, for example, DMS 30 detects the movement of the driver's left hand by recognizing the driver's skeleton, etc., based on images from the ToF camera P101L. The recognition unit 73 detects objects outside the vehicle 1C around the door 611FL of the driver's seat 101 based on sensing data such as images from the camera 51SL. Then, if an object is detected within a predetermined range around the door 611FL of the driver's seat 101, when the driver takes an action to open the door 611FL, for example, when the driver's left hand touches the door opener 661FL of the door 611FL, the display unit 201LL displays a warning.

[0728] For example, Figure 84 The illustration shows a display example of display unit 201LL before the driver's left hand touches the door opener 661FL. A portion of the vehicle body 1C is displayed at the right end of display unit 201LL. On the left side of vehicle 1C, a pedestrian 1201 is shown approaching vehicle 1C from behind.

[0729] In this case, for example, such as Figure 85 As shown, when the driver DR1's left hand is detected touching the door opener 661FL, the display unit 201LL displays the following: Figure 86 The warning is shown diagonally in the image. Specifically, at the left end of display unit 201LL, a vertically elongated visual effect bar 1202 is superimposed, displayed, and flashes. The color of bar 1202 is a prominent color, such as yellow.

[0730] Note that a warning sound can be output at this time.

[0731] This draws the driver DR1’s attention to pedestrian 1201 and prevents door 611FL from hitting pedestrian 1201 when it is opened.

[0732] Note that at this time, for example, the body system control unit 84 can increase the torque of the door 611FL to make the door 611FL difficult to open.

[0733] Furthermore, for example, if an occupant in passenger seat 102 touches the door opener 661FR of door 611FR with their hand, the display unit 201RR on the side of passenger seat 102 may display a similar warning.

[0734] <Operational Example of Display Unit 201CL and Lighting Device 1002>

[0735] Next, we will refer to Figures 87 to 95 This describes an operational example of the display unit 201CL of the central display 201 and the lighting device 1002 of the steering wheel 1001.

[0736] For example, the display control unit causes the display unit 201CL to display content based on at least one of the following: the situation of the vehicle 1C, the situation around the vehicle 1C, and the situation of the occupants.

[0737] For example, the lamp control unit 85 activates or flashes the lighting device 1002 in a certain mode based on at least one of the conditions of the vehicle 1C, the conditions around the vehicle 1C, and the conditions of the occupants. The light emission mode of the lighting device 1002 is defined, for example, by at least one of color, brightness, flashing mode, light movement, and light emission area.

[0738] For example, when the driver enters the vehicle 1C and sits in the driver's seat 101, the DMS 30 performs driver recognition processing based on at least one of the images from the ToF camera P101L and the camera 1042 of the camera module 1007. Then, the display unit 201CL and the lighting device 1002 provide a notification that the driver has been recognized.

[0739] For example, Figure 87 The illustration shows the state of the display unit 201CL and the lighting device 1002 before driver identification.

[0740] The vehicle's status is displayed on the display unit 201CL. Specifically, it displays the gear shift position, charge level, speed, and cruising range.

[0741] Lighting device 1002 is turned off.

[0742] Figure 88 The illustration shows the status of the display unit 201CL and the lighting device 1002 when the driver is identified.

[0743] Display unit 201CL displays the identified driver's image and name, as well as the message to the driver. Afterward, display unit 201CL returns to... Figure 87 The state shown in the image.

[0744] Although detailed illustrations are omitted, the lighting device 1002 is turned on in a predetermined mode. For example, multiple short white light strips move on the lighting device 1002 in a predetermined pattern.

[0745] Therefore, the driver was indeed notified that vehicle 1C had identified the driver.

[0746] At this time, for example, the vehicle system control unit 84 can set the position and angle of the driver's seat 101, the position and angle of the steering wheel 1001, etc., based on the identified driver's physique or pre-set driver preferences. Therefore, a driving position suitable for the identified driver is set.

[0747] Furthermore, for example, the display control unit can adjust the display position of the display unit 201CL of the central display 201 based on the identified position of the driver's eyes, the position and angle of the steering wheel 1001, etc. Therefore, according to the identified driver, the display range of the display unit 201CL is set in a position that is easy to view and will not be obstructed by the steering wheel 1001.

[0748] Furthermore, for example, when the vehicle 1C is ready for autonomous driving, the display unit 201CL and the lighting device 1002 provide a notification that the preparation for autonomous driving is complete.

[0749] Figure 89The illustration shows the status of the display unit 201CL and the lighting device 1002 when preparation for autonomous driving is complete.

[0750] Apart from Figure 87 In addition to the content shown, the display unit 201CL also displays an icon 1311 indicating that autonomous driving preparation is complete. Note that to the left of icon 1311, the speed limit of the road on which vehicle 1 is traveling is displayed.

[0751] Although detailed descriptions are omitted, the lighting device 1002 is turned on in a predetermined mode. For example, multiple short white light strips move on the lighting device 1002 in a predetermined pattern.

[0752] Therefore, the driver can confidently recognize that preparations for autonomous driving have been completed.

[0753] In addition, for example, when autonomous driving of vehicle 1C begins, display unit 201CL and lighting device 1002 provide a notification that autonomous driving has begun.

[0754] Figure 90 The illustration shows the status of the display unit 201CL and the lighting device 1002 when autonomous driving begins.

[0755] Display unit 201CL displays, for example, an image of the front of vehicle 1C captured by camera 51FC. On display unit 201CL, for example, shift position, charge level, speed, cruising range, speed limit of the road on which the vehicle travels, fuel consumption, etc., are displayed. On display unit 201CL, to the right of the aforementioned icon 1311, an icon 1312 indicating that autonomous driving is in progress is displayed. Furthermore, for example, the color of icon 1311 changes as autonomous driving begins.

[0756] Although detailed descriptions are omitted, the lighting device 1002 is activated in a predetermined mode. For example, multiple short white light strips move in a predetermined pattern above the lighting device 1002 while gradually turning blue. Finally, the entire lighting device 1002 is activated in blue and remains blue during autonomous driving.

[0757] Therefore, the driver can confidently recognize that autonomous driving has begun. Furthermore, by activating the lighting device 1002, occupants other than the driver can confidently recognize that autonomous driving has begun.

[0758] Figure 91 The illustration shows the status of the display unit 201CL and the lighting device 1002 when the vehicle 1C changes lanes to the right lane during autonomous driving.

[0759] When vehicle 1C changes lanes, display unit 201CL overlays and displays guidance information 1313, including animations to notify of the execution and direction of the lane change, on the image in front of vehicle 1C.

[0760] Additionally, the blue light of the entire lighting device 1002 is turned off, and the predetermined flashing range A201 on the right side of the lighting device 1002 flashes blue. Furthermore, when the steering wheel 1001 is rotated clockwise to change lanes, the flashing range A201 rotates in the opposite direction (counter-clockwise) according to the rotation angle of the steering wheel 1001. Conversely, when the steering wheel 1001 is rotated counter-clockwise to return to its original position after changing lanes, the flashing range A201 rotates in the opposite direction (clockwise) according to the rotation angle of the steering wheel 1001. Therefore, the flashing range A201 remains in a constant position when viewed from the outside without rotating with the steering wheel 1001. Then, after the rotation angle of the steering wheel 1001 returns to zero degrees, the entire lighting device 1002 turns on in blue, similar to before changing lanes.

[0761] Therefore, drivers can anticipate when vehicle 1C will change lanes.

[0762] Note that when vehicle 1C changes lanes to the left lane, similarly, guidance information 1313 is superimposed on display unit 201CL and displayed, and the predetermined flashing range on the left side of lighting device 1002 flashes in blue.

[0763] Furthermore, for example, a similar process is performed when the vehicle 1C changes direction (e.g., turns left or right). For example, guidance information including animations for notifying the execution of the direction change and the direction is overlaid and displayed on the display unit 201CL, and a predetermined flashing range corresponding to the direction change direction of the lighting device 1002 flashes blue.

[0764] Figure 92 The illustration shows the status of the display unit 201CL and the lighting device 1002 when a dangerous situation is detected around the vehicle 1C.

[0765] On the display unit 201CL, for example, a message 1321 indicating the detected status, countermeasures, etc., and an icon 1322 are displayed.

[0766] Furthermore, the blue light of the entire lighting device 1002 is turned off, and the predetermined flashing range A211 on the lighting device 1002 flashes. For example, the color and flashing speed of the light in the flashing range A211 change based on the detected level of danger. For example, in a high-risk situation, the flashing range A211 flashes red, while in a low-risk situation, the flashing range A211 flashes yellow. Moreover, the higher the risk, the faster the flashing speed, and the lower the risk, the slower the flashing speed.

[0767] In addition, for example, a warning sound is output based on the flashing range A211.

[0768] Furthermore, for example, DMS 30 detects the positions of the driver's left and right hands by performing driver skeleton recognition, etc., based on images captured by the ToF camera P101L. Then, DMS 30 detects the positions of the driver's left-hand fingers touching the operation unit 1003 of the steering wheel 1001 and the positions of the driver's right-hand fingers touching the operation unit 1004 of the steering wheel 1001.

[0769] For example, in the display control unit, when the driver's left-hand finger touches the operation unit 1003, or when the driver's right-hand finger touches the operation unit 1004, guidance indicating the function, operation method, etc. of the button touched by the finger is displayed on the display unit 201CL.

[0770] For example, such as Figure 93 As shown, when the driver DR1's left thumb touches the leftmost button of the operation unit 1003, as Figure 94 As shown, guidance information 1341 is displayed near the center in the left-right direction and at the lower end of the display unit 201CL. Guidance information 1341 indicates the function and operation method of the button touched by the driver DR1's finger.

[0771] It should be noted that, Figure 94 In order to make the view easier to understand, the driver DR1's hands, etc., are not shown.

[0772] In addition, similar guidance information can be displayed on the head-up display's display 203.

[0773] Therefore, the driver DR1 can grasp the function and operation method of the button without moving their gaze in the direction of the operation unit 1003 or operation unit 1004. For example, the driver DR1 can grasp the function and operation method of the button even in dark conditions such as at night. For example, even if the driver DR1 is not used to driving vehicle 1C, they can easily grasp the function and operation method of the button.

[0774] Figure 95The illustration shows the state of the display unit 201CL and the lighting device 1002 when the DMS 30 performs at least one of speech recognition and lip reading.

[0775] In this case, the display unit 201CL displays an animation 1361 indicating that at least one of speech recognition and lip reading is being performed.

[0776] Although detailed descriptions are omitted, the lighting device 1002 describes the movement of light associated with the movement of the animation 1361.

[0777] Note that, for example, an animation similar to animation 1361 can be displayed on the display unit 201CC of the central display 201.

[0778] This ensures that occupants, such as the driver, are notified that at least one of speech recognition and lip reading is being performed. Therefore, for example, it is possible to prevent occupants from suspecting the unauthorized collection of data such as conversations.

[0779] <Examples related to audio control>

[0780] Next, we will refer to Figures 96 to 105 An example describing the audio control of vehicle 1C.

[0781] As described above, vehicle 1C can achieve 360-degree realistic audio. Furthermore, the driver's seat 101, passenger seat 102, rear seat 103L, and rear seat 103L each include seat speakers 1021, 1022, 1023L, and 1023R. Therefore, the audio control unit, as part of the functionality of HMI 31, can freely control the sound image, sound field, volume, etc., for each occupant in each seat.

[0782] The DMS 30 can identify the shape and position of the driver's two ears based on images from, for example, a ToF camera P101L. Therefore, the audio control unit can appropriately set the sound image, sound field, volume, etc. for the driver by controlling the seat speakers 1021 of the driver's seat 101 based on the position and shape of the driver's two ears.

[0783] Similarly, DMS 30 can identify the shape and position of the occupant's two ears on the passenger seat 102 based on images from, for example, a ToF camera P101R. Therefore, the audio control unit can appropriately set the sound image, sound field, volume, etc. for the occupant by controlling the seat speakers 1022, etc., of the passenger seat 102 based on the position and shape of the occupant's two ears.

[0784] Similarly, DMS 30 can identify the shape and position of the two ears of the occupant in the rear seat 103L based on images from, for example, a ToF camera P111L or a ToF camera P112L. Therefore, the audio control unit can appropriately set the sound image, sound field, volume, etc. for the occupant by controlling the seat speakers 1023L, etc. in the rear seat 103L, based on the position and shape of the occupant's two ears.

[0785] Similarly, DMS 30 can identify the shape and position of the two ears of the occupant in the rear seat 103R based on images from, for example, a ToF camera P111R or a ToF camera P112R. Therefore, the audio control unit can control the seat speakers 1023R, etc. in the rear seat 103R, to appropriately set the sound image, sound field, volume, etc. for the occupant based on the position and shape of the occupant's two ears.

[0786] In this way, vehicle 1C can output sound to each occupant individually, and can control the sound image, sound field, volume, etc., independently. Utilizing this feature, for example, the following functions can be achieved.

[0787] Figure 96 The illustration shows an example of a method for controlling a warning sound when there is a hazard, such as an obstacle, in front of vehicle 1.

[0788] Sound fields A301L and A301R schematically illustrate the sound fields for a warning sound to the driver in driver's seat 101. Specifically, the triangles representing sound fields A301L and A301R indicate the location of the virtual sound source and the dispersion of the sound field. Specifically, the vertex indicated by the black circle at the center of the triangle indicates the location of the virtual sound source. The two sides extending from the vertex indicating the virtual sound source indicate the direction of the sound field dispersion.

[0789] Note that sound fields A301L and A301R are represented by cones in three-dimensional space. Therefore, sound fields A301L and A301R are each distributed in a cone shape from the virtual sound source toward the side of the triangle.

[0790] For example, firstly, sound field A301L is set to face the driver's left ear from the right. Sound field A301R is set to face the driver's right ear from the left. Therefore, the driver hears the warning sound in both their left and right ears.

[0791] Then, sound field A301L rotates clockwise around the driver's face to the front of the driver, as indicated by the arrow. Similarly, sound field A301R rotates counterclockwise around the driver's face to the front of the driver, as indicated by the arrow.

[0792] Therefore, after hearing the warning sound in each of the left and right ears, the driver feels the two warning sounds swirling around their face, eventually merging into one and focusing on the front. Afterward, a warning message such as "Look ahead" is output from in front of the driver.

[0793] Note that when the direction that draws the driver's attention (e.g., the direction of an obstacle) is not forward, the warning sound is localized in the direction that draws the driver's attention. Therefore, the driver perceives the warning sound as moving and being localized in the direction that draws their attention.

[0794] This alerts the driver to hazards such as obstacles and allows them to confidently identify the direction in which the hazard may be located.

[0795] Figure 97 The illustration shows an example of how to control the sound effects output when autonomous driving begins.

[0796] First, the sound field A302 for the sound effects is set to face the driver from in front, at ear level. Then, the sound field A302 is divided into left and right sections, one rotating counter-clockwise around the driver's face to directly behind the driver, and the other rotating clockwise around the driver's face to directly behind the driver.

[0797] Therefore, the driver perceives the sound effect as being output from the front, splitting into left and right segments, rotating around the face, and finally merging into one sound positioned behind. This effectively initiates autonomous driving.

[0798] Figure 98 The illustration shows an example of a method for controlling the output of sound effects when vehicle 1C accelerates during autonomous driving.

[0799] First, the sound field A303 is set to face the driver's left ear to the right. Then, the sound field A303 rotates clockwise around the driver's face until the driver's right ear faces to the left.

[0800] Therefore, the driver perceives the sound effect rotating clockwise from the left ear to the right ear in front of the face.

[0801] Figure 99 The illustration shows an example of a method for controlling the output of sound effects when vehicle 1C decelerates during autonomous driving.

[0802] First, the sound field A304 for the sound effects is set to face the driver's right ear from the left. Then, the sound field A304 rotates counter-clockwise around the driver's face until the driver's left ear faces to the right.

[0803] Therefore, the driver perceives the sound effect rotating counterclockwise from the right ear to the left ear in front of the face.

[0804] As mentioned above, the driver can intuitively identify the acceleration and deceleration of the vehicle 1C during autonomous driving by the direction of the sound effect movement.

[0805] Note that, for example, it can also be like Figure 98 and Figure 99 The controls shown are for sound effects based on set values ​​other than the speed of vehicle 1C. For example, when the set value is increased, the sound effect can be output as follows: Figure 98 The sound effect shown can be output as follows when the set value is decreased: Figure 99 The sound effects shown.

[0806] Figure 100 The illustration shows an example of a method for controlling the sound effects output before voice messages such as navigation are output.

[0807] First, the sound field A305 is set at ear level, facing the driver from the driver's left rear. Then, the sound field A305 rotates clockwise around the driver's head to a position facing the driver from the driver's right front.

[0808] Therefore, the driver perceives the sound effect as rotating counter-clockwise behind the head, from the left rear to the right front. In this way, the sound effect rotates behind the driver's head, making it easier for the driver to notice it than if the sound effect rotated in front of the head. This reduces the likelihood of the driver missing voice messages after an audio effect.

[0809] Figures 101 to 104 The illustration shows an example of a method for controlling a warning sound that draws attention to a vehicle 1401 approaching from the left rear.

[0810] For example, if the direction indicator for the left direction of vehicle 1401 is turned on and the distance between vehicle 1401 and vehicle 1C is within a predetermined threshold, a warning sound will be output. Figure 101 The diagram shows the sound field A306 when the warning sound is first output. The direction of the sound field A306 is set to be substantially the same as the direction of the driver's head in vehicle 1C as seen from vehicle 1401. In addition, the virtual sound source of the warning sound is set at a predetermined distance from the driver's head in the direction of vehicle 1401.

[0811] After that, as Figures 102 to 104As shown, as vehicle 1401 approaches vehicle 1C, sound field A306 moves. Specifically, sound field A306 moves so that it is always oriented in a direction substantially the same as the direction of the driver's head when viewing vehicle 1C from vehicle 1401. Furthermore, the movement of sound field A306 makes the distance between the virtual source of the warning sound and the driver's head approximately proportional to the distance between vehicle 1401 and vehicle 1C.

[0812] Therefore, the driver can recognize the approach of vehicle 1401 through the warning sound. In addition, the driver can visually identify the speed, distance, and direction of vehicle 1401 by observing the movement of the warning sound.

[0813] Furthermore, each seat's speaker can be used to output sound individually, so that only the occupant in each seat can hear the sound. For example, as Figure 105 As shown, the seat speaker 1023L can be used to output sound so that only the occupant sitting in the rear seat 103L can hear the sound.

[0814] For example, using this feature, each occupant in one seat can make voice or video calls to the outside world without being overheard by other occupants. Additionally, for example, voice chat can be performed between occupants in vehicle 1C.

[0815] <Example of lip reading>

[0816] As described above, DMS 30 performs lip reading of the driver based on images from camera 1042 of camera module 1007. As described above, DMS 30 performs lip reading of the occupant in rear seat 103L based on images from ToF camera P112L. DMS 30 performs lip reading of the occupant in rear seat 103R based on images from ToF camera P112R. Note that, for example, DMS 30 can also perform lip reading of the occupant in passenger seat 102 using images from ToF camera P101R.

[0817] For example, DMS 30 can improve speech recognition accuracy by using lip reading to assist with speech recognition for each occupant in each seat. Alternatively, for example, DMS 30 can recognize the content of each occupant's speech by using only lip reading.

[0818] Therefore, in situations such as when the interior of vehicle 1C is noisy and it is difficult to collect speech with a microphone, or when a child is sleeping and cannot speak loudly, DMS 30 can recognize the speech of each occupant in each seat with high accuracy.

[0819] Furthermore, for example, HMI 31 can transmit speech content from one occupant's speech to another occupant by converting the speech content of one occupant into speech and outputting that speech from a speaker in another seat. Alternatively, HMI 31 can transmit speech content from one occupant's speech content to another occupant by converting the speech content of one occupant into text and displaying that text on a display in front of another seat. This enables communication between occupants in different seats, even in noisy or poorly audible environments within the vehicle 1C.

[0820] Furthermore, for example, the driver can operate the vehicle 1C quietly by using lip reading. Specifically, for example, the driver can operate the vehicle 1C by pressing a predetermined button on the operation unit 1003 or operation unit 1004 of the steering wheel 1001 and making mouth movements similar to those used to input a predetermined voice command. Note that the driver may or may not speak at this time.

[0821] More specifically, for example, a driver can adjust the air conditioning in the vehicle's central control system (NC) by making mouth movements to give voice commands such as "cool hands," "cool feet," or "refresh the air." Similarly, a driver can play music by making mouth movements to give voice commands such as "play an eye-opening song," "play a normal song," or "play a song everyone can sing along to." Furthermore, a driver can give instructions such as searching for a destination to a navigation app operating in the vehicle's NC by making mouth movements to give voice commands such as "tell me about nearby restaurants," "find a coffee shop nearby," or "tell me about nearby charging points."

[0822] Therefore, even in noisy environments inside the vehicle 1C or when it is not possible to speak loudly, the driver can use voice commands to operate the vehicle 1C.

[0823] <Examples of gesture input>

[0824] For example, DMS 30 can recognize the gestures of the occupant in each seat based on images from ToF cameras installed for each seat in vehicle 1C. Therefore, each occupant can operate vehicle 1C by using gestures.

[0825] For example, an occupant can increase the audio volume of content played in vehicle 1C by pointing to the air and rotating their index finger clockwise. Conversely, an occupant can decrease the audio volume of content played in vehicle 1C by pointing to the air and rotating their index finger counterclockwise. For example, an occupant can mute the volume of content played in vehicle 1C by raising their index finger and bringing it close to their lips.

[0826] For example, an occupant can operate the touchpad by pointing their index finger at the display in front of the seat and moving their index finger, without actually touching the touchpad on the display. For example, a driver can launch a navigation application by pointing at the display unit 201CC of the central display 201. Furthermore, for example, a driver can display information about the pointed location or set the pointed location as the destination by pointing at the map in the navigation application.

[0827] <<7. Modification Examples of the Third Embodiment>>

[0828] In the following text, a modified example of the third embodiment of the present technology will be described.

[0829] The above description has illustrated vehicle 1C as an example of a left-hand drive vehicle, but this technology can certainly be applied to right-hand drive vehicles. When applying this technology to a right-hand drive vehicle, the interior layout of the vehicle can be appropriately modified according to the vehicle's driving style.

[0830] Furthermore, this technology can also be applied to, for example, autonomous driving mobile devices in which the occupants do not drive. In this case, there is no distinction between the driver's seat 101 and the passenger seat 102 as described above, and the various displays described above are arranged in front of the seats in the mobile device. Furthermore, for example, the display range of the images displayed on the displays can be changed based on the line of sight and posture of the person sitting in the designated seat.

[0831] Furthermore, there are no particular restrictions on the types of vehicles to which this technology can be applied. In addition to vehicles, this technology can also be applied to mobile devices such as personal mobility devices, aircraft, ships, construction machinery, and agricultural machinery.

[0832] <<8. Others>>

[0833] <Computer Configuration Example>

[0834] The above series of processes can be performed by hardware or by software. In the case where the series of processes are performed by software, the program constituting the software is installed in the computer (e.g., processor 21, etc.).

[0835] Note that the program executed by the computer may be a program for processing in a time sequence in the order described in this specification, or a program for parallel processing or processing at necessary timed intervals (such as when a call is made).

[0836] Furthermore, in this specification, a system refers to a collection of multiple components (devices, modules (parts), etc.), and it is irrelevant whether all components are housed in the same housing. Therefore, multiple devices housed in different housings and connected via a network, and a single device in which multiple modules are housed in one housing, are both systems.

[0837] Furthermore, the embodiments of this technology are not limited to the above embodiments, and various modifications can be made without departing from the spirit of this technology.

[0838] For example, this technology can be configured using cloud computing, where one function is shared and processed jointly by multiple devices via a network.

[0839] Furthermore, each step described in the flowchart above can be performed by a single device, or by multiple devices in a shared manner.

[0840] Furthermore, when a step includes multiple processes, these processes can be executed by multiple devices in a shared manner, in addition to being executed by one device.

[0841] <Example of configuration combinations>

[0842] This technology can also be configured as follows. (1)

[0844] A display system, comprising:

[0845] A first display extends in a left-right direction in front of the first seat of the mobile device.

[0846] The left end of the first display faces diagonally to the right and rear when viewed from the first seat.

[0847] The central portion of the first display faces rearward when viewed from the first seat, and

[0848] The right end of the first display faces diagonally to the left and rear when viewed from the first seat. (2)

[0850] According to the display system described in (1) above, wherein

[0851] The left-hand portion of the first display shows an image of the mobile device diagonally rear to the left, and

[0852] The right side of the first display shows an image of the mobile device diagonally behind and to the right. (3)

[0854] According to the display system described in (2) above, wherein

[0855] The left and right portions of the first display are used as digital exterior rearview mirrors. (4)

[0857] The display system according to (2) or (3) above also includes

[0858] The display control unit controls the display of images on the left and right portions of the first display. (5)

[0860] According to the display system described in (4) above, wherein

[0861] The display control unit controls the display range of the left and right portions of the first display based on at least one of the following: the line of sight of the person sitting in the first seat, the person's posture, the operation of the direction indicator, and the gear shift position. (6)

[0863] According to the display system described in (5) above, wherein

[0864] When the moving device stops, the display control unit moves the display range of the left and right portions of the first display to the side that the person is looking at, based on the angle of the person's gaze. (7)

[0866] According to the display system described in (6) above, wherein

[0867] The display control unit overlays and displays visual effects indicating a predetermined range around the mobile device onto the image of the person looking at the left and right portions of the first display. (8)

[0869] According to any one of (5) to (7) above, the display system wherein

[0870] The display control unit extends the display range of the left and right portions of the first display in the direction in which the direction indicator is turned on. (9)

[0872] The display system according to any one of (5) to (8) above, wherein

[0873] When the shift position is set to reverse, the display control unit zooms out the display range. (10)

[0875] According to any one of (5) to (9) above, the display system wherein

[0876] The display control unit overlays and displays the visual effect of the recognition result of the object diagonally behind the left side of the mobile device on the image of the left side of the first display, and overlays and displays the visual effect of the recognition result of the object diagonally behind the right side of the mobile device on the image of the right side of the first display. (11)

[0878] According to the display system described in (10) above, wherein

[0879] The display control unit changes the display mode of the visual effect based on the degree of danger of the object. (12)

[0881] According to the display system described in (11) above, wherein

[0882] The display control unit changes the display mode of the visual effect at the left end of the first display when the left-direction direction indicator is turned on, and changes the display mode of the visual effect at the right end of the first display when the right-direction direction indicator is turned on. (13)

[0884] The display system according to any one of (10) to (12) above further includes

[0885] The recognition unit identifies objects to the left and behind the mobile device based on an image to the left and behind the mobile device, and identifies objects to the right and behind the mobile device based on an image to the right and behind the mobile device. (14)

[0887] The display system according to any one of (1) to (13) above, wherein

[0888] The central portion of the first display shows an image of the area surrounding the mobile device. (15)

[0890] The display system described in (14) above also includes

[0891] The display control unit controls the display range of the image displayed in the central portion of the first display based on at least one of the line of sight and posture of the person sitting in the first seat. (16)

[0893] The display system according to any one of (1) to (15) above, wherein

[0894] The central portion of the first display shows information supporting the operation of the mobile device. (17)

[0896] The display system according to any one of (1) to (16) above, wherein

[0897] The central portion of the first display extends in a left-right direction in front of the first seat and the second seat arranged next to the first seat, and can be divided into a first display area in front of the first seat, a second display area in front of the second seat, and a third display area between the first display area and the second display area. (18)

[0899] According to the display system described in (17) above, wherein

[0900] The first display area displays information supporting operation of mobile devices, and

[0901] The second and third display areas display infotainment-related information.

[0902] (18A)

[0903] The display system according to any one of (1) to (17) above, wherein

[0904] The central portion of the first display faces diagonally upwards. (19)

[0906] The display system according to any one of (1) to (18A) above, wherein

[0907] The left and right portions of the first display bend inward from the center of the first display toward the interior of the mobile device. (20)

[0909] The display system according to any one of (1) to (19) above further includes

[0910] A second display is positioned below the first display and faces rearward when viewed from the first seat. (twenty one)

[0912] According to the display system described above (20), wherein

[0913] The second display is formed by a two-dimensional or three-dimensional touch panel and displays an operation screen for the information displayed on the first display.

[0914] (21A)

[0915] According to the display system described in (20) or (21) above, wherein

[0916] The second display shows the operation screen of the air conditioner on the mobile device.

[0917] (21B)

[0918] The display system according to any one of (20) to (21A) above, wherein

[0919] The second display faces diagonally upwards.

[0920] (21C)

[0921] According to the display system described above (21B), wherein

[0922] The central portion of the first display faces diagonally upward at an angle substantially similar to that of the second display. (twenty two)

[0924] The display system according to any one of (20) to (21C) above, wherein

[0925] The second display is located in the console between the first seat and the second seat arranged next to the first seat. (twenty three)

[0927] The display system according to any one of (20) to (22) above further includes

[0928] The display control unit controls the display of images on the left and right portions of the first display.

[0929] The left side of the first display shows an image from the left rear of the mobile device.

[0930] The right side of the first display shows an image of the mobile device diagonally behind and to the right.

[0931] The second display is formed by a two-dimensional or three-dimensional touch panel and displays an operation screen configured to set the display range of the left and right portions of the first display.

[0932] The display control unit changes the display area based on the operation of the screen. (twenty four)

[0934] According to the display system described above (23), wherein

[0935] Based on the operation of the operation screen, the display control unit changes the display range of the left end portion of the first display within a predetermined first area including the left rear of the mobile device, and changes the display range of the right end portion of the first display within a predetermined second area including the right rear of the mobile device. (25)

[0937] According to the display system described in (23) or (24) above, wherein

[0938] The display control unit enables the zooming in and out range of the display to differ between the left and right portions of the first display. (26)

[0940] The display system according to any one of (1) to (25) above further includes

[0941] A third display is positioned in front of the first seat and displays visual information superimposed on the field of vision of the person sitting in the first seat. (27)

[0943] According to the display system described above (26), wherein

[0944] The third display shows information that supports operation of mobile devices. (28)

[0946] The display system according to any one of (1) to (27) above further includes

[0947] A fourth display is positioned above the first display and faces rearward when viewed from the first seat. (29)

[0949] According to the display system described above (28), wherein

[0950] The fourth display is used as a digital rearview mirror. (30)

[0952] According to the display system described above (29), it also includes

[0953] The display control unit controls the display range of the image displayed on the fourth display based on at least one of the line of sight and posture of the person sitting in the first seat. (31)

[0955] The display system according to any one of (1) to (30) above further includes

[0956] A fifth display is located at the center of the steering wheel of the mobile device.

[0957] (31A)

[0958] According to the display system described in (31) above, wherein

[0959] The fifth display shows a message to the person sitting in the first seat. (32)

[0961] The display system according to any one of (1) to (31A) above further includes

[0962] A sixth display is disposed on the back of at least one of the first seat and the second seat arranged next to the first seat. (33)

[0964] The display system according to any one of (1) to (32) above, wherein

[0965] The left end of the first display faces the right rear of the mobile device.

[0966] The central portion of the first display faces the rear of the mobile device, and

[0967] The right end of the first display faces the left rear of the mobile device. (34)

[0969] The display system according to any one of (1) to (33) above, wherein

[0970] The first seat is the driver's seat. (35)

[0972] A display device, including

[0973] The display extends in a left-right direction in front of the seat of the mobile device.

[0974] The left side of the monitor faces slightly to the right and rear when viewed from the seat.

[0975] The central portion of the monitor faces backward when viewed from the seat, and

[0976] The right side of the monitor faces slightly to the left and rear when viewed from the seat. (36)

[0978] A display method, including

[0979] In a display that extends in the left-right direction in front of the seat of the mobile device and has a left end portion facing the right rear when viewed from the seat, a central portion facing the rear when viewed from the seat, and a right end portion facing the left rear when viewed from the seat, the left end portion of the display displays an image of the mobile device facing the left rear, and the right end portion of the display displays an image of the mobile device facing the right rear. (37)

[0981] A mobile device, comprising

[0982] The display extends in the left-right direction in front of the seat and has a left end portion facing right rearward when viewed from the seat, a central portion facing rearward when viewed from the seat, and a right end portion facing left rearward when viewed from the seat. (38)

[0984] A mobile device, comprising:

[0985] Metadata adding unit, which adds metadata to captured moving image data, which is captured while in motion; and

[0986] An image processing unit that edits captured moving image data based on metadata and generates edited moving image data. (39)

[0988] According to the mobile device described in (38) above, wherein

[0989] The image processing unit extracts frames from multiple captured motion image data captured by multiple cameras based on metadata, and combines the frames to generate edited motion image data. (40)

[0991] According to the mobile device described above (39), wherein

[0992] The image processing unit extracts frames from the multiple captured mobile image data based on the mobile device's path and metadata. (41)

[0994] According to the mobile device described in (39) or (40) above, wherein

[0995] Captured motion image data is obtained by capturing images of the surroundings or interior of a mobile device. (42)

[0997] According to the mobile device described in (41) above, wherein

[0998] The image processing unit extracts frames of captured moving image data obtained by capturing internal images based on internal activity. (43)

[1000] The mobile device according to any one of (38) to (42) above, wherein

[1001] The image processing unit overlays visual information onto frames of captured moving image data based on metadata. (44)

[1003] The mobile device according to any one of (38) to (43) above, wherein

[1004] The image processing unit edits the captured moving image data based on user instructions. (45)

[1006] The mobile device according to any one of (38) to (44) above, wherein

[1007] The metadata addition unit adds one or more metadata items to the mobile image data, including image capture location, image capture date and time, mobile device conditions, mobile device interior conditions, and mobile device surrounding conditions. (46)

[1009] The mobile device according to any one of (38) to (45) above, wherein

[1010] The metadata addition unit adds user-inputted metadata to the captured mobile image data. (47)

[1012] A mobile device, comprising

[1013] A camera is mounted on or around a rearview mirror used for inspecting the rear of the vehicle and to image the direction of the driver's seat. (48)

[1015] According to the mobile device described in (47) above, wherein

[1016] The rearview mirror is installed near the ceiling diagonally in front of the driver's seat. (49)

[1018] According to the mobile device described in (48) above, wherein

[1019] The camera is mounted on the roof around the location where the rearview mirror is attached to the interior of the vehicle. (50)

[1021] According to the mobile device described in (48) or (49) above, wherein

[1022] The camera is mounted closer to the driver's seat than the rearview mirror. (51)

[1024] The mobile device according to any one of (47) to (50) above further includes

[1025] A first identification unit performs at least one of driver gaze recognition, gesture recognition, action recognition, and personal recognition based on images captured by a camera. (52)

[1027] The mobile device according to (51) above also includes

[1028] The display control unit controls the display of the digital exterior rearview mirror of the mobile device based on at least one of the driver's gaze, posture, and movement identified by the first recognition unit. (53)

[1030] The mobile device according to (52) above also includes

[1031] The second identification unit identifies objects around the driver's seat door based on sensing data around the door outside the mobile device.

[1032] In the case where the first recognition unit recognizes the driver's action of opening the door and the second recognition unit recognizes objects around the door, the display control unit overlays and displays the visual effect based on the object recognition result on the image of the digital exterior rearview mirror on the driver's seat side. (54)

[1034] According to the mobile device described in (52) or (53) above, wherein

[1035] When the first recognition unit detects that the driver is looking at the digital exterior rearview mirror, the display control unit moves the display range of the digital exterior rearview mirror based on the angle at which the driver is looking at it. (55)

[1037] The mobile device according to any one of (52) to (54) above, wherein

[1038] When the first identification unit detects that the driver is in contact with the operating body of the mobile device, the display control unit displays information indicating guidance for the operating body on the display in front of the driver's seat. (56)

[1040] The mobile device according to any one of (47) to (55) above, wherein

[1041] The camera is a Time-of-Flight (ToF) camera. (57)

[1043] A mobile device, comprising

[1044] The control body is a lever-shaped control body for changing the shift position, and has an indicator provided on the side, which has a color that changes according to the set shift position. (58)

[1046] According to the mobile device described above (57), wherein

[1047] The shift positions of the control unit are set in the order of reverse, neutral, drive, and autonomous driving. (59)

[1049] According to the mobile device described above (58), wherein

[1050] The control unit includes a joystick that switches gear positions by moving it in the up-down direction. (60)

[1052] The mobile device according to (58) above also includes

[1053] The operating unit includes a dial, through which the shift position is switched by rotating about an axis. (61)

[1055] The mobile device according to any one of (58) to (60) above further includes

[1056] A button located at the top of the control body can be pressed in the axial direction of the control body and is used to set the shift position to stop. (62)

[1058] The mobile device according to any one of (57) to (61) above, wherein

[1059] The indicator is arranged in a circumferential shape on the side of the operating body. (63)

[1061] The mobile device according to any one of (57) to (62) above, wherein

[1062] The operating body is configured to extend from the steering column in the transverse direction of the moving device, and

[1063] The indicator is positioned at least one of the following locations: one visible from the gap between the spokes of the steering wheel when viewed from the driver's seat, and the other located outside the outer periphery of the steering wheel when viewed from the driver's seat. (64)

[1065] A mobile device, comprising:

[1066] A steering wheel having a ring-shaped lighting device, the color, brightness, and at least one of the luminous area of ​​the lighting device being variable, and the lighting device being disposed in the central portion; and

[1067] The lamp control unit controls the lighting device. (65)

[1069] According to the mobile device described above (64), wherein

[1070] The lighting control unit turns on or flashes the lighting device based on at least one of the conditions of the mobile device, the conditions around the mobile device, and the conditions of the occupants. (66)

[1072] According to the mobile device described above (65), wherein

[1073] The lighting control unit turns the lighting on or flashes according to the situation. (67)

[1075] According to the mobile device described in (66) above, wherein

[1076] The pattern is defined by at least one of color, brightness, blinking pattern, light movement, and luminous area. (68)

[1078] According to the mobile device described in (66) or (67) above, wherein

[1079] When a moving device changes lanes or direction, the light control unit activates or flashes a portion of the lighting device corresponding to the direction of the lane change or direction change. (69)

[1081] According to the mobile device described above (68), wherein

[1082] The light control unit rotates the part of the lighting device that is turned on or flashing in the opposite direction to the rotation of the steering wheel, based on the angle of rotation of the steering wheel. (70)

[1084] The mobile device according to any one of (66) to (69) above, wherein

[1085] When performing at least one of voice recognition and lip reading, the lamp control unit turns on or flashes the lighting device in a predetermined mode. (71)

[1087] The mobile device according to any one of (64) to (70) above, wherein

[1088] The central section houses the airbag, and

[1089] The lighting is positioned to avoid the central section that splits open when the airbag is activated. (72)

[1091] A mobile device, comprising

[1092] A device for multiple user interfaces, arranged along a roughly horizontal ring around the interior perimeter. (73)

[1094] According to the mobile device described in (72) above, wherein

[1095] Devices used for user interfaces include at least one of output devices and operating devices. (74)

[1097] According to the mobile device described in (73) above, wherein

[1098] The output device includes at least one of a display extending in a left-right direction in the front portion of the mobile device and a lighting device disposed around the periphery of the central portion of the steering wheel. (75)

[1100] The mobile device according to any one of (73) to (74) above, wherein

[1101] The output devices include speakers mounted on the door of the mobile device. (76)

[1103] The mobile device according to any one of (73) to (75) above, wherein

[1104] The operating equipment includes at least one of the following: a door opener for a moving device, a device for adjusting the airflow direction of an air conditioner, and a device for setting headlights. (77)

[1106] The mobile device according to any one of (72) to (76) above further includes

[1107] A ring light that surrounds at least a portion of the inner perimeter, with the ring line generally parallel to it. (78)

[1109] The mobile device according to (77) above also includes

[1110] The lamp control unit performs coordinated control of the external lamps and ring lights of the mobile device. (79)

[1112] The mobile device according to any one of (72) to (78) above, wherein

[1113] Air conditioning vents are installed along the loop. (80)

[1115] A mobile device, comprising:

[1116] Multiple seat speakers, which are individually installed speakers within the seat; and

[1117] Output control unit, which individually controls the sound output from each seat speaker. (81)

[1119] According to the mobile device described above (80), wherein

[1120] Each seat speaker is located under the headrest of each seat. (82)

[1122] The mobile device according to (80) or (81) above also includes

[1123] Multiple linear loudspeakers, arranged along a loop that roughly horizontally surrounds the inner perimeter.

[1124] The output control unit also controls the sound output from the linear speaker. (83)

[1126] According to the mobile device described in (82) above, wherein

[1127] The output control unit performs control to output sound from each linear speaker toward the entire interior, and from each seat speaker to output sound for each occupant sitting in each seat. (84)

[1129] According to the mobile device described in (80) to (83) above, wherein

[1130] The output control unit moves the sound field of the warning sound for the driver in the direction that will attract the driver's attention. (85)

[1132] The mobile device according to any one of (80) to (84) above, wherein

[1133] The output control unit causes the sound field of the sound effect indicating an increase in a predetermined setting value for the moving device to rotate around the head of the person sitting in the seat along a first direction, and causes the sound field of the sound effect indicating a decrease in a predetermined setting value to rotate around the head of the person along a second direction opposite to the first direction. (86)

[1135] The mobile device according to any one of (80) to (85) above, wherein

[1136] The output control unit causes the sound field of the audio effect output before the voice message to rotate behind the head of the person sitting in the seat. (87)

[1138] The mobile device according to any one of (80) to (86) above, wherein

[1139] The output control unit adjusts the sound field of the warning sound for objects behind the moving device according to the movement of the object. (88)

[1141] A mobile device, comprising

[1142] A camera module that is mounted on the dashboard and captures images from the driver's seat direction from a diagonal angle. (89)

[1144] According to the mobile device described in (88) above, wherein

[1145] The camera module switches between color image capture and infrared image capture. (90)

[1147] According to the mobile device described above (89), wherein

[1148] The camera module includes a light source that outputs infrared light, and outputs infrared light when performing infrared image capture. (91)

[1150] According to the mobile device described in (89) or (90) above, wherein

[1151] The camera module switches between color image capture and infrared image capture based on ambient brightness. (92)

[1153] The mobile device according to any one of (89) to (91) above further includes

[1154] The recognition unit performs lip reading of the driver based on images from the camera module.

[1155] Note that the effects described in this manual are merely examples and not limitations, and other effects can be set.

[1156] List of reference numerals

[1157] 1 vehicle

[1158] 11 Vehicle Control System

[1159] 21 processors

[1160] 25 External identification sensors

[1161] 26 In-vehicle sensors

[1162] 30 DMS

[1163] 31 HMI

[1164] 51 cameras

[1165] 52 radar

[1166] 53 LiDAR

[1167] 73 Identification Units

[1168] 84 Body System Control Unit

[1169] 85 lamp control unit

[1170] 101 Driver's Seat

[1171] 102 Passenger Seats

[1172] Rear seats of 103, 103L, and 103R

[1173] 104 Windshield

[1174] 106 Steering Wheel

[1175] 107 Console

[1176] 108 controller

[1177] 109 Operation Unit

[1178] 151FL to 151BR cameras

[1179] 201 Central Display

[1180] 201C Central Section

[1181] 201CL to 201CR display units

[1182] 201L Left end

[1183] 201LL Display Unit

[1184] 201R Right end

[1185] 201RR Display Unit

[1186] 202 Console Display

[1187] 203 Monitor

[1188] 204 Digital Rearview Mirror

[1189] 205 Steering Wheel Display

[1190] 206 Rear Entertainment Display

[1191] 601 Accessory Light

[1192] 602 daytime running lights

[1193] 603 Headlights

[1194] 604 Turn signal light

[1195] 605 Auxiliary Light

[1196] 606 taillights

[1197] 607 Brake Light

[1198] 651 Steering Wheel

[1199] 652 Lighting fixtures

[1200] 653FL to BR speaker

[1201] 654 Ring Light

[1202] 655 joystick

[1203] 656 button

[1204] 657 indicator

[1205] 658FC to 658FR ventilation openings

[1206] 659CL to 659FR knobs

[1207] 660 Operation Unit

[1208] 661FL, 661FR door openers

[1209] 662L and 662R tablet terminals

[1210] 663 Installation Section

[1211] 701 Information Processing Unit

[1212] 711 Image Processing Unit

[1213] 712 Metadata Addition Unit

[1214] 713 Output Control Unit

[1215] 751 Main Light

[1216] 1001 Steering Wheel

[1217] 1002 Lighting fixtures

[1218] Operation units 1003 and 1004

[1219] Camera modules 1007 and 1008

[1220] Seat speakers (1021 to 1023R)

[1221] 1042 camera

[1222] 1043 LED

Claims

1. A display system, comprising A first display extends in a left-to-right direction in front of the first seat of the mobile device. in, The left end of the first display faces diagonally to the right and rear when viewed from the first seat. The central portion of the first display faces rearward when viewed from the first seat, and The right end of the first display faces diagonally to the left and rear when viewed from the first seat. The display control unit controls the display of images at the left and right ends of the first display. The display control unit overlays and displays the visual effect based on the recognition result of an object diagonally behind and to the left of the mobile device onto the image on the left side of the first display, and overlays and displays the visual effect based on the recognition result of an object diagonally behind and to the right of the mobile device onto the image on the right side of the first display. Wherein, when the distance between the object to the left or right diagonally behind the display control unit and the mobile device is within a predetermined range, a bar is superimposed and displayed on the left end of the image on the left side of the first display or the right end of the image on the right side of the first display. And when the absolute speed of the object or the relative speed of the object with respect to the mobile device is equal to or greater than a predetermined threshold, the bar is displayed from the time point when the object is farther away from the mobile device.

2. The display system according to claim 1, wherein The left-hand portion of the first display shows an image of the mobile device diagonally rear to the left, and The right side of the first display shows an image of the mobile device diagonally behind and to the right.

3. The display system according to claim 2, wherein The left and right portions of the first display are used as digital exterior rearview mirrors.

4. The display system according to claim 1, wherein The display control unit controls the display range of the left and right portions of the first display based on at least one of the following: the line of sight of the person sitting in the first seat, the person's posture, the operation of the direction indicator, and the gear shift position.

5. The display system according to claim 4, wherein When the moving device stops, the display control unit moves the display range of the left and right portions of the first display to the side that the person is looking at, based on the angle of the person's gaze.

6. The display system according to claim 5, wherein The display control unit overlays and displays visual effects indicating a predetermined range around the mobile device onto the image of the person looking at the left and right portions of the first display.

7. The display system according to claim 4, wherein The display control unit extends the display range of the left and right portions of the first display in the direction in which the direction indicator is turned on.

8. The display system according to claim 4, wherein When the shift position is set to reverse, the display control unit zooms out the display range.

9. The display system according to claim 1, wherein The display control unit changes the display mode of the visual effect based on the degree of danger of the object.

10. The display system according to claim 9, wherein The display control unit changes the display mode of the visual effect at the left end of the first display when the left-direction direction indicator is turned on, and changes the display mode of the visual effect at the right end of the first display when the right-direction direction indicator is turned on.

11. The display system according to claim 1, further comprising: The recognition unit identifies objects to the left and behind the mobile device based on an image to the left and behind the mobile device, and identifies objects to the right and behind the mobile device based on an image to the right and behind the mobile device.

12. The display system according to claim 1, wherein The central portion of the first display shows an image of the area surrounding the mobile device.

13. The display system according to claim 12, further comprising: The display control unit controls the display range of the image displayed in the central portion of the first display based on at least one of the line of sight and posture of the person sitting in the first seat.

14. The display system according to claim 1, wherein The central portion of the first display shows information supporting the operation of the mobile device.

15. The display system according to claim 1, wherein The central portion of the first display extends in a left-right direction in front of the first seat and the second seat arranged next to the first seat, and can be divided into a first display area in front of the first seat, a second display area in front of the second seat, and a third display area between the first display area and the second display area.

16. The display system according to claim 15, wherein The first display area displays information supporting operation of mobile devices, and The second and third display areas display infotainment-related information.

17. The display system according to claim 1, wherein The left and right portions of the first display bend inward from the center of the first display toward the interior of the mobile device.

18. The display system according to claim 1, further comprising: A second display is positioned below the first display and faces rearward when viewed from the first seat.

19. The display system according to claim 18, wherein The second display is formed by a two-dimensional or three-dimensional touch panel and displays an operation screen for the information displayed on the first display.

20. The display system according to claim 18, wherein The second display is located in the console between the first seat and the second seat arranged next to the first seat.

21. The display system according to claim 18, wherein, The left side of the first display shows an image of the mobile device diagonally behind and to the left. The right side of the first display shows an image of the mobile device diagonally behind and to the right. The second display is formed by a two-dimensional or three-dimensional touch panel and displays an operation screen configured to set the display range of the left and right portions of the first display. The display control unit changes the display area based on the operation of the screen.

22. The display system according to claim 21, wherein Based on the operation of the operation screen, the display control unit changes the display range of the left end portion of the first display within a predetermined first area including the left rear of the mobile device, and changes the display range of the right end portion of the first display within a predetermined second area including the right rear of the mobile device.

23. The display system according to claim 21, wherein The display control unit enables the zooming in and out range of the display to differ between the left and right portions of the first display.

24. The display system according to claim 1, further comprising: A third display is positioned in front of the first seat and displays visual information superimposed on the field of vision of the person sitting in the first seat.

25. The display system according to claim 24, wherein The third display shows information that supports operation of mobile devices.

26. The display system according to claim 1, further comprising: A fourth display is positioned above the first display and faces rearward when viewed from the first seat.

27. The display system according to claim 26, wherein The fourth display is used as a digital rearview mirror.

28. The display system according to claim 27, further comprising: The display control unit controls the display range of the image displayed on the fourth display based on at least one of the line of sight and posture of the person sitting in the first seat.

29. The display system according to claim 1, further comprising: A fifth display is located at the center of the steering wheel of the mobile device.

30. The display system according to claim 1, further comprising: A sixth display is disposed on the back of at least one of the first seat and a second seat arranged next to the first seat.

31. The display system according to claim 1, wherein The left end of the first display faces the right rear of the mobile device. The central portion of the first display faces the rear of the mobile device, and The right end of the first display faces the left rear of the mobile device.

32. The display system according to claim 1, wherein The first seat is the driver's seat.

33. A display device, comprising The display extends in a left-right direction in front of the seat of the mobile device. in, The left side of the monitor faces slightly to the right and rear when viewed from the seat. The central portion of the monitor faces backward when viewed from the seat, and The right side of the monitor faces slightly to the left and rear when viewed from the seat. The display control unit controls the display of images on the left and right sides of the display. The display control unit overlays and displays the visual effect based on the recognition result of an object diagonally behind and to the left of the mobile device onto the image on the left side of the display, and overlays and displays the visual effect based on the recognition result of an object diagonally behind and to the right of the mobile device onto the image on the right side of the display. Wherein, when the distance between the object to the left or right diagonally behind the display control unit and the moving device is within a predetermined range, a bar is superimposed and displayed on the left end of the image on the left side of the display or the right end of the image on the right side of the display. And when the absolute speed of the object or the relative speed of the object with respect to the moving device is equal to or greater than a predetermined threshold, the bar is displayed from the time point when the object is farther away from the moving device.

34. A display method, comprising: In a display extending in the left-right direction in front of the seat of the mobile device, and having a left end portion facing right rearward when viewed from the seat, a central portion facing rearward when viewed from the seat, and a right end portion facing left rearward when viewed from the seat, the left end portion of the display displays an image of the mobile device from left rearward, and the right end portion of the display displays an image of the mobile device from right rearward. Controls the display of the image on the left and right sides of the monitor. The visual effect based on the recognition result of an object diagonally behind and to the left of the mobile device is overlaid and displayed on the image at the left end of the display, and the visual effect based on the recognition result of an object diagonally behind and to the right of the mobile device is overlaid and displayed on the image at the right end of the display. in, When the distance between an object to the left or right rear and the mobile device is within a predetermined range, a bar is superimposed and displayed on the left end of the image on the left side of the display or the right end of the image on the right side of the display. Furthermore, when the absolute velocity of the object or the relative velocity of the object with respect to the mobile device is equal to or greater than a predetermined threshold, the bar is displayed from the time point when the object is farther away from the mobile device.

35. A mobile device, comprising The display extends in a left-right direction in front of the seat and has a left end portion facing diagonally to the right rear when viewed from the seat, a central portion facing rearward when viewed from the seat, and a right end portion facing diagonally to the left rear when viewed from the seat. The display control unit controls the display of images on the left and right sides of the display. The display control unit overlays and displays the visual effect based on the recognition result of an object diagonally behind and to the left of the mobile device onto the image on the left side of the display, and overlays and displays the visual effect based on the recognition result of an object diagonally behind and to the right of the mobile device onto the image on the right side of the display. in, When the distance between the object to the left or right and the moving device is within a predetermined range, the display control unit superimposes and displays a bar on the left end of the image on the left side of the display or the right end of the image on the right side of the display. Furthermore, when the absolute velocity of the object or the relative velocity of the object with respect to the moving device is equal to or greater than a predetermined threshold, the bar is displayed from the point in time when the object is farther away from the moving device.

Images