Camera module, information processing system, information processing method, and information processing device

Abstract

The present technology relates to a camera module, an information processing system, an information processing method, and an information processing device that make it possible to curb the complexity of a system which includes a camera in a moving device such as a vehicle. The camera module comprises a first camera, a second camera, and a housing that accommodates the first camera and the second camera. In a state in which the camera module is installed in a moving device, the optical axis of the first camera is oriented diagonally rearward of the moving device and the optical axis of the second camera is oriented in a lateral direction or diagonally forward of the moving device. The present technology can be applied, for example, to a self-driving vehicle.

Description

[Technical Field]

[0001] The present technology relates to a camera module, an information processing system, an information processing method, and an information processing device, and more particularly to a camera module, an information processing system, an information processing method, and an information processing device that enable effective use of a camera in a mobile device. [Background technology]

[0002] In recent years, development has progressed on CMS (Camera Monitoring System), which displays images of the rear of the vehicle on an in-car display instead of conventional door mirrors.

[0003] For example, conventionally, door mirrors for use in CMS have been proposed that are equipped with a main camera that faces the rear of the vehicle and captures images to be displayed on an in-vehicle display, and a sub-camera that detects fogging of the lens, etc. (see, for example, Patent Documents 1 and 2).

[0004] In addition, in recent years, the number of cameras installed in vehicles has been increasing in order to support CMS and autonomous driving. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Application Publication No. 2018-197021 [Patent Document 2] Japanese Patent Application Publication No. 2019-51769 Summary of the Invention [Problem to be solved by the invention]

[0006] However, as the number of cameras increases, the system architecture of the vehicle including each camera becomes more complex, and the design of the camera installation positions, wiring, etc. becomes more difficult.

[0007] The present technology has been made in view of such circumstances, and is intended to make it possible to suppress the complication of a system including a camera in a mobile device such as a vehicle. [Means for solving the problem]

[0008] A camera module according to a first aspect of the present technology includes a first camera, a second camera, and a housing for storing the first camera and the second camera, and when installed on a mobile device, the optical axis of the first camera faces diagonally rearward from the mobile device, and the optical axis of the second camera faces laterally or diagonally forward from the mobile device.

[0009] An information processing system according to a second aspect of the present technology includes a first camera whose optical axis faces diagonally rearward of a mobile device, a second camera housed in the same housing as the first camera and whose optical axis faces closer to the front of the mobile device than the optical axis of the first camera, a first display used in a CMS that displays an image based on a first image captured by the first camera, a recognition unit that recognizes objects outside the mobile device based on a second image captured by the second camera, and a display control unit that controls the display on the first display of the first image and visual effects based on the results of object recognition by the recognition unit.

[0010] An information processing method according to a third aspect of the present technology controls the display of an image on a display for a CMS based on a first image taken by a first camera whose optical axis faces diagonally rearward of a mobile device, recognizes objects outside the mobile device based on a second image taken by a second camera housed in the same housing as the first camera and whose optical axis faces closer to the front of the mobile device than the optical axis of the first camera, and controls the display of visual effects based on the results of the object recognition.

[0011] An information processing device according to a fourth aspect of the present technology includes a display control unit that controls the display of a display used in a CMS, and the display control unit controls the display of visual effects based on the results of object recognition outside the mobile device, based on an image based on a first image taken by a first camera whose optical axis faces diagonally rearward of the mobile device, and a second image taken by a second camera housed in the same housing as the first camera and whose optical axis faces in a direction closer to the front of the mobile device than the optical axis of the first camera.

[0012] In a first aspect of the present technology, when installed on a mobile device, the optical axis of the first camera is directed diagonally rearward from the mobile device, and the optical axis of the second camera is directed laterally or diagonally forward from the mobile device.

[0013] In a second or third aspect of the present technology, the display of an image is controlled based on a first image taken by a first camera whose optical axis faces diagonally rearward of the mobile device, object recognition outside the mobile device is performed based on a second image taken by a second camera stored in the same housing as the first camera and whose optical axis faces in a direction closer to the front of the mobile device than the optical axis of the first camera, and the display of a visual effect based on the results of the object recognition is controlled.

[0014] In a fourth aspect of the present technology, the display of visual effects is controlled based on the results of object recognition outside the mobile device, based on an image based on a first image taken by a first camera whose optical axis faces diagonally rearward of the mobile device, and a second image taken by a second camera stored in the same housing as the first camera and whose optical axis faces in a direction closer to the front of the mobile device than the optical axis of the first camera. [Brief explanation of the drawings]

[0015] [Figure 1] FIG. 1 is a block diagram illustrating an example of the configuration of a vehicle control system. [Figure 2] FIG. 1 is a diagram illustrating an example of installation positions of a camera, a LiDAR, a radar, and an ultrasonic sensor. [Figure 3] FIG. 2 is a diagram illustrating an example of a photographing range of a fisheye camera. [Figure 4] FIG. [Figure 5] The left side of the vehicle. [Figure 6] FIG. 2 is a diagram illustrating an example of the configuration of a camera module. [Figure 7] FIG. 2 is a diagram illustrating an example of the configuration of a camera module. [Figure 8] FIG. 2 is a diagram illustrating an example of the configuration of a camera module. [Figure 9] FIG. 2 is an exploded view showing a configuration example of a camera module. [Figure 10] FIG. 2 is a diagram illustrating an example of the arrangement of cameras in a camera module. [Figure 11] FIG. 2 is a diagram illustrating an example of the arrangement of cameras in a camera module. [Figure 12] FIG. 2 is a schematic diagram of the interior of the vehicle viewed from the right. [Figure 13] FIG. 2 is a schematic diagram showing the front interior of the vehicle. [Figure 14] FIG. 2 is a schematic diagram showing the area in front of the driver's seat of the vehicle. [Figure 15] FIG. 2 is an enlarged view of the vicinity of the steering wheel of the vehicle. [Figure 16] 1A and 1B are diagrams illustrating examples of installation positions of ToF cameras in a camera module. [Figure 17] FIG. 1 is a diagram illustrating an example of a photographing range of a ToF camera. [Figure 18] FIG. 2 is a schematic view of the interior of the vehicle as seen from the front. [Figure 19] FIG. 2 is a schematic diagram of the driver's seat as seen from the left rear. [Figure 20] FIG. 1 is a block diagram illustrating an example of the configuration of an information processing system. [Figure 21] FIG. 1 is a block diagram illustrating a specific configuration example of an information processing system. [Figure 22] FIG. 10 is a diagram showing an example of a display on the display unit for CMS in a normal state. [Figure 23] 10A and 10B are diagrams illustrating an example of a display on a display unit for a CMS when a vehicle in an adjacent lane is approaching. [Figure 24] FIG. 10 is a diagram showing an example of a display on the display unit for CMS when a direction indicator is on. [Figure 25] 10A and 10B are diagrams illustrating an example of a display on a display unit for a CMS when a vehicle in an adjacent lane is approaching. [Figure 26] FIG. 10 is a diagram showing an example of a display on the display unit for CMS when a direction indicator is on. [Figure 27] FIG. 10 is a diagram illustrating an example of a required field of view range. [Figure 28] 10 is a diagram illustrating an example of changing the display range of a display unit for a CMS based on a shift position. FIG. [Figure 29] FIG. 10 is a diagram for explaining a method for setting the display range of the display unit for CMS. [Figure 30] FIG. 10 is a diagram for explaining the display range when looking into the CMS display unit. [Figure 31] FIG. 10 is a diagram for explaining the display range when looking into the CMS display unit. [Figure 32] FIG. 10 is a diagram for explaining the display range when the shift position is set to reverse and the driver looks into the CMS display unit. [Figure 33] FIG. 10 is a diagram for explaining the display range when the shift position is set to reverse and the driver looks into the CMS display unit. [Figure 34] FIG. 10 is a diagram for explaining the display range when the shift position is set to reverse and the driver looks into the CMS display unit. [Figure 35] 10 is a diagram for explaining a method of controlling a display unit for a CMS when a door opener is touched. FIG. [Figure 36] 10 is a diagram for explaining a method of controlling a display unit for a CMS when a door opener is touched. FIG. [Figure 37] 10 is a diagram for explaining a method of controlling a display unit for a CMS when a door opener is touched. FIG. [Figure 38] 10A and 10B are diagrams showing the illumination states of the display unit for the driver's seat and the steering wheel before the driver is recognized. [Figure 39]10A and 10B are diagrams illustrating the illumination states of the display unit for the driver's seat and the steering wheel when the driver is recognized. [Figure 40] FIG. 10 is a diagram showing the illumination state of the display unit for the driver's seat and the steering wheel when preparations for autonomous driving are complete. [Figure 41] FIG. 10 is a diagram showing the illumination state of the display unit for the driver's seat and the steering wheel when autonomous driving starts. [Figure 42] 10A and 10B are diagrams showing the illumination state of the display unit for the driver's seat and the steering wheel when changing lanes during autonomous driving. [Figure 43] 10A and 10B are diagrams illustrating the illumination states of the display unit for the driver's seat and the steering wheel when a danger is detected. [Figure 44] 10A and 10B are diagrams for explaining a method for controlling a warning sound that alerts a driver to a vehicle approaching from the rear left. [Figure 45] 10A and 10B are diagrams for explaining a method for controlling a warning sound that alerts a driver to a vehicle approaching from the rear left. [Figure 46] 10A and 10B are diagrams for explaining a method for controlling a warning sound that alerts a driver to a vehicle approaching from the rear left. [Figure 47] 10A and 10B are diagrams for explaining a method for controlling a warning sound that alerts a driver to a vehicle approaching from the rear left. [Figure 48] FIG. 21 is a block diagram showing a first modified example of the information processing system of FIG. 20. [Figure 49] FIG. 21 is a block diagram showing a second modified example of the information processing system of FIG. 20. [Figure 50] FIG. 21 is a block diagram showing a third modified example of the information processing system of FIG. 20. [Figure 51] FIG. 1 is a block diagram showing an example of a method for realizing ADAS+HMI+Body DC. [Figure 52] FIG. 22 is a block diagram showing a modified example of the information processing system of FIG. 21. [Figure 53] FIG. 2 is a block diagram showing an example of the configuration of a video splitter. [Figure 54]FIG. 1 is a block diagram illustrating an example of the configuration of a fisheye camera. DETAILED DESCRIPTION OF THE INVENTION

[0016] Hereinafter, embodiments of the present technology will be described in the following order. 1. Example of vehicle control system configuration 2. Embodiment 3. Variations 4.Other

[0017] <<1. Example of vehicle control system configuration>> FIG. 1 is a block diagram showing an example of the configuration of a vehicle control system 11, which is an example of a mobility device control system to which the present technology is applied.

[0018] The vehicle control system 11 is provided in the vehicle 1 and performs processing related to driving assistance and automatic driving of the vehicle 1.

[0019] The vehicle control system 11 includes a processor 21, a communication unit 22, a map information storage unit 23, a GNSS (Global Navigation Satellite System) receiving unit 24, an external recognition sensor 25, an in-vehicle sensor 26, a vehicle sensor 27, a recording unit 28, a driving assistance / autonomous driving control unit 29, a DMS (Driver Monitoring System) 30, an HMI (Human Machine Interface) 31, and a vehicle control unit 32.

[0020] The processor 21, communication unit 22, map information storage unit 23, GNSS receiving unit 24, external recognition sensor 25, in-vehicle sensor 26, vehicle sensor 27, recording unit 28, cruise assist / autonomous driving control unit 29, driver monitoring system (DMS) 30, human-machine interface (HMI) 31, and vehicle control unit 32 are interconnected via a communication network 41. The communication network 41 is configured by an in-vehicle communication network or bus conforming to any standard such as CAN (Controller Area Network), LIN (Local Interconnect Network), LAN (Local Area Network), FlexRay (registered trademark), or Ethernet (registered trademark). Note that the components of the vehicle control system 11 may be directly connected to each other via, for example, near field communication (NFC) or Bluetooth (registered trademark) without using the communication network 41.

[0021] In the following description, when each unit of the vehicle control system 11 communicates via the communication network 41, the description of the communication network 41 will be omitted. For example, when the processor 21 and the communication unit 22 communicate via the communication network 41, it will simply be described that the processor 21 and the communication unit 22 communicate with each other.

[0022] The processor 21 is configured by various processors such as a CPU (Central Processing Unit), an MPU (Micro Processing Unit), an ECU (Electronic Control Unit), etc. The processor 21 controls the vehicle control system 11 as a whole.

[0023] The communication unit 22 communicates with various devices inside and outside the vehicle, other vehicles, servers, base stations, etc., and transmits and receives various types of data. As communication with the outside of the vehicle, for example, the communication unit 22 receives from the outside a program for updating software that controls the operation of the vehicle control system 11, map information, traffic information, information about the surroundings of the vehicle 1, etc. For example, the communication unit 22 transmits information about the vehicle 1 (for example, data indicating the state of the vehicle 1, the recognition result by the recognition unit 73, etc.), information about the surroundings of the vehicle 1, etc., to the outside. For example, the communication unit 22 performs communication compatible with a vehicle emergency notification system such as e-call.

[0024] There is no particular limitation on the communication method of the communication unit 22. A plurality of communication methods may be used.

[0025] For example, the communication unit 22 performs wireless communication with devices inside the vehicle using a communication method such as wireless LAN, Bluetooth, NFC, or WUSB (Wireless USB). For example, the communication unit 22 performs wired communication with devices inside the vehicle using a communication method such as USB (Universal Serial Bus), HDMI (High-Definition Multimedia Interface, registered trademark), or MHL (Mobile High-Definition Link) via a connection terminal (and a cable, if necessary) not shown.

[0026] Here, the in-vehicle device refers to, for example, a device that is in the vehicle and is not connected to the communication network 41. Examples of such devices include mobile devices and wearable devices carried by passengers such as the driver, and information devices that are brought into the vehicle and temporarily installed.

[0027] For example, the communication unit 22 communicates with a server or the like existing on an external network (e.g., the Internet, a cloud network, or an operator-specific network) via a base station or an access point using a wireless communication method such as 4G (fourth generation mobile communication system), 5G (fifth generation mobile communication system), LTE (Long Term Evolution), or DSRC (Dedicated Short Range Communications).

[0028] For example, the communication unit 22 communicates with a terminal present in the vicinity of the vehicle (for example, a terminal of a pedestrian or a store, or an MTC (Machine Type Communication) terminal) using P2P (Peer To Peer) technology. For example, the communication unit 22 performs V2X communication. V2X communication includes, for example, vehicle-to-vehicle communication with another vehicle, vehicle-to-infrastructure communication with a roadside unit or the like, vehicle-to-home communication, and vehicle-to-pedestrian communication with a terminal carried by a pedestrian or the like.

[0029] For example, the communication unit 22 receives electromagnetic waves transmitted by a road traffic information and communication system (VICS (Vehicle Information and Communication System), registered trademark) such as a radio beacon, an optical beacon, or FM multiplex broadcasting.

[0030] The map information storage unit 23 stores maps acquired from an external source and maps created by the vehicle 1. For example, the map information storage unit 23 stores a three-dimensional high-precision map, a global map that is less accurate than a high-precision map and covers a wide area, and the like.

[0031] Examples of high-precision maps include dynamic maps, point cloud maps, and vector maps (also referred to as ADAS (Advanced Driver Assistance System) maps). Dynamic maps are maps consisting of four layers of information, for example, dynamic information, quasi-dynamic information, quasi-static information, and static information, and are provided from an external server or the like. Point cloud maps are maps configured from point clouds (point cloud data). Vector maps are maps in which information such as the positions of lanes and traffic lights is associated with the point cloud map. The point cloud map and the vector map may be provided from an external server or the like, or may be created by the vehicle 1 based on sensing results from the radar 52, the LiDAR 53, and the like as a map for matching with a local map (described later), and stored in the map information storage unit 23. Furthermore, when a high-precision map is provided from an external server or the like, map data of, for example, an area of ​​several hundred square meters related to the planned route along which the vehicle 1 will travel is acquired from the server or the like in order to reduce communication capacity.

[0032] The GNSS receiver 24 receives GNSS signals from GNSS satellites and supplies them to the driving assistance / automated driving control unit 29 .

[0033] The external recognition sensor 25 includes various sensors used to recognize the situation outside the vehicle 1, and supplies sensor data from each sensor to each part of the vehicle control system 11. The type and number of sensors included in the external recognition sensor 25 are arbitrary.

[0034] For example, the external recognition sensor 25 includes a camera 51, a radar 52, a LiDAR (Light Detection and Ranging, Laser Imaging Detection and Ranging) 53, and an ultrasonic sensor 54. The number of cameras 51, radars 52, LiDARs 53, and ultrasonic sensors 54 is arbitrary, and an example of the sensing area of ​​each sensor will be described later.

[0035] As the camera 51, a camera of any imaging method such as a ToF (Time Of Flight) camera, a stereo camera, a monocular camera, or an infrared camera may be used as needed.

[0036] Furthermore, for example, the external recognition sensor 25 includes an environmental sensor for detecting the weather, climate, brightness, etc. The environmental sensor includes, for example, a raindrop sensor, a fog sensor, a sunlight sensor, a snow sensor, an illuminance sensor, etc.

[0037] Furthermore, for example, the external recognition sensor 25 includes a microphone used to detect sounds around the vehicle 1 and the positions of sound sources.

[0038] The interior sensor 26 includes various sensors for detecting information inside the vehicle, and supplies sensor data from each sensor to each unit of the vehicle control system 11. The interior sensor 26 may include any type and any number of sensors.

[0039] For example, the in-vehicle sensors 26 include a camera, radar, a seating sensor, a steering wheel sensor, a microphone, a biometric sensor, etc. The camera may be a camera of any imaging method, such as a ToF camera, a stereo camera, a monocular camera, or an infrared camera. The biometric sensor is provided, for example, on a seat or steering wheel, and detects various types of biometric information of a passenger such as a driver.

[0040] The vehicle sensor 27 includes various sensors for detecting the state of the vehicle 1, and supplies sensor data from each sensor to each unit of the vehicle control system 11. The type and number of sensors included in the vehicle sensor 27 are arbitrary.

[0041] For example, the vehicle sensor 27 includes a speed sensor, an acceleration sensor, an angular velocity sensor (gyro sensor), and an inertial measurement unit (IMU). For example, the vehicle sensor 27 includes a steering angle sensor that detects the steering angle of the steering wheel, a yaw rate sensor, an accelerator sensor that detects the amount of accelerator pedal operation, and a brake sensor that detects the amount of brake pedal operation. For example, the vehicle sensor 27 includes a rotation sensor that detects the number of rotations of the engine or motor, an air pressure sensor that detects tire air pressure, a slip ratio sensor that detects tire slip ratio, and a wheel speed sensor that detects the rotation speed of the wheels. For example, the vehicle sensor 27 includes a battery sensor that detects the remaining battery level and temperature, and an impact sensor that detects external impacts.

[0042] The recording unit 28 includes, for example, a magnetic storage device such as a read-only memory (ROM), a random access memory (RAM), or a hard disk drive (HDD), a semiconductor storage device, an optical storage device, or a magneto-optical storage device. The recording unit 28 records various programs and data used by each component of the vehicle control system 11. For example, the recording unit 28 records a rosbag file including messages transmitted and received by a robot operating system (ROS) on which an application program related to autonomous driving runs. For example, the recording unit 28 includes an event data recorder (EDR) or a data storage system for automated driving (DSSAD), and records information about the vehicle 1 before and after an event such as an accident.

[0043] The driving assistance / automatic driving control unit 29 controls driving assistance and automatic driving of the vehicle 1. For example, the driving assistance / automatic driving control unit 29 includes an analysis unit 61, an action planning unit 62, and an operation control unit 63.

[0044] The analysis unit 61 performs an analysis process of the vehicle 1 and the surrounding situation. The analysis unit 61 includes a self-position estimation unit 71, a sensor fusion unit 72, and a recognition unit 73.

[0045] The self-position estimation unit 71 estimates the self-position of the vehicle 1 based on the sensor data from the external recognition sensor 25 and the high-precision map stored in the map information storage unit 23. For example, the self-position estimation unit 71 generates a local map based on the sensor data from the external recognition sensor 25 and estimates the self-position of the vehicle 1 by matching the local map with the high-precision map. The position of the vehicle 1 is based on, for example, the center of the rear wheel pair axle.

[0046] The local map is, for example, a three-dimensional high-precision map or an occupancy grid map created using a technology such as SLAM (Simultaneous Localization and Mapping). The three-dimensional high-precision map is, for example, the point cloud map described above. The occupancy grid map is a map in which the three-dimensional or two-dimensional space around the vehicle 1 is divided into grids of a predetermined size and the occupancy state of an object is indicated on a grid-by-grid basis. The occupancy state of an object is indicated, for example, by the presence or absence of an object and its probability of existence. The local map is also used, for example, in the detection process and recognition process of the situation outside the vehicle 1 by the recognition unit 73.

[0047] The self-position estimation unit 71 may estimate the self-position of the vehicle 1 based on the GNSS signal and sensor data from the vehicle sensor 27.

[0048] The sensor fusion unit 72 performs sensor fusion processing to obtain new information by combining multiple different types of sensor data (for example, image data supplied from the camera 51 and sensor data supplied from the radar 52). Methods for combining different types of sensor data include integration, fusion, and association.

[0049] The recognition unit 73 performs detection processing and recognition processing of the situation outside the vehicle 1.

[0050] For example, the recognition unit 73 performs detection processing and recognition processing of the situation outside the vehicle 1 based on information from the external recognition sensor 25, information from the self-position estimation unit 71, information from the sensor fusion unit 72, etc.

[0051] Specifically, for example, the recognition unit 73 performs detection processing and recognition processing of objects around the vehicle 1. The object detection processing is, for example, processing to detect the presence or absence, size, shape, position, movement, etc. of an object. The object recognition processing is, for example, processing to recognize attributes such as the type of object, or to identify a specific object. However, the detection processing and the recognition processing are not necessarily clearly separated, and may overlap.

[0052] For example, the recognition unit 73 performs clustering to classify a point cloud based on sensor data such as LiDAR or radar into clusters of points, thereby detecting objects around the vehicle 1. This allows the presence, size, shape, and position of objects around the vehicle 1 to be detected.

[0053] For example, the recognition unit 73 performs tracking to follow the movement of clusters of point clouds classified by clustering, thereby detecting the movement of objects around the vehicle 1. As a result, the speed and traveling direction (movement vector) of the objects around the vehicle 1 are detected.

[0054] For example, the recognition unit 73 performs object recognition processing such as semantic segmentation on the image data supplied from the camera 51 to recognize the type of object around the vehicle 1.

[0055] Note that objects to be detected or recognized may include, for example, vehicles, people, bicycles, obstacles, structures, roads, traffic lights, traffic signs, road markings, and the like.

[0056] For example, the recognition unit 73 performs a recognition process of traffic rules around the vehicle 1 based on the map stored in the map information storage unit 23, the estimation result of the vehicle's own position, and the recognition result of objects around the vehicle 1. Through this process, for example, the positions and states of traffic signals, the contents of traffic signs and road markings, the contents of traffic regulations, and lanes on which travel is permitted are recognized.

[0057] For example, the recognition unit 73 performs a process of recognizing the environment around the vehicle 1. The environment to be recognized may include, for example, the weather, temperature, humidity, brightness, and road surface conditions.

[0058] The behavior planning unit 62 creates a behavior plan for the vehicle 1. For example, the behavior planning unit 62 creates the behavior plan by performing route planning and route following processing.

[0059] Global path planning is a process for planning a rough route from the start to the goal. This route planning also includes a process called trajectory planning, which takes into account the motion characteristics of the vehicle 1 on the route planned by the route planning and generates a trajectory (local path planning) that allows the vehicle 1 to proceed safely and smoothly in the vicinity of the vehicle 1.

[0060] Path following is a process of planning an operation for safely and accurately traveling along a route planned by a route planner within a planned time. For example, a target speed and a target angular velocity of the vehicle 1 are calculated.

[0061] The operation control unit 63 controls the operation of the vehicle 1 in order to realize the action plan created by the action planning unit 62.

[0062] For example, the operation control unit 63 controls the steering control unit 81, the brake control unit 82, and the drive control unit 83 to perform acceleration / deceleration control and direction control so that the vehicle 1 travels along the trajectory calculated by the trajectory plan. For example, the operation control unit 63 performs cooperative control aimed at realizing ADAS functions such as collision avoidance or impact mitigation, following driving, vehicle speed maintenance driving, collision warning for the vehicle itself, and lane departure warning for the vehicle itself. For example, the operation control unit 63 performs cooperative control aimed at automatic driving, which allows the vehicle to travel autonomously without relying on driver operation.

[0063] The DMS 30 performs processes such as authenticating the driver and recognizing the driver's state based on the sensor data from the in-vehicle sensors 26 and the input data input to the HMI 31. Examples of the driver's state to be recognized include physical condition, alertness, concentration, fatigue, gaze direction, level of intoxication, driving operation, and posture.

[0064] The DMS 30 may be configured to perform authentication processing for passengers other than the driver and recognition processing for the conditions of the passengers. Furthermore, for example, the DMS 30 may be configured to perform recognition processing for the conditions inside the vehicle based on sensor data from the in-vehicle sensor 26. Possible conditions inside the vehicle to be recognized include, for example, temperature, humidity, brightness, and odor.

[0065] The HMI 31 is used to input various data, instructions, etc., generates input signals based on the input data, instructions, etc., and supplies them to each component of the vehicle control system 11. For example, the HMI 31 includes operation devices such as a touch panel, buttons, a microphone, switches, and levers, as well as operation devices that allow input by voice, gestures, etc., other than manual operation. The HMI 31 may be, for example, a remote control device that uses infrared or other radio waves, or an externally connected device such as a mobile device or wearable device that supports operation of the vehicle control system 11.

[0066] The HMI 31 also performs output control to generate and output visual information, auditory information, and tactile information for the occupant or the outside of the vehicle, as well as to control the output content, output timing, output method, etc. Visual information is information displayed by images or light, such as an operation screen, a status display of the vehicle 1, a warning display, and a monitor image showing the situation around the vehicle 1. Auditory information is information displayed by sound, such as guidance, warning sounds, and warning messages. Tactile information is information imparted to the occupant's sense of touch by force, vibration, movement, etc.

[0067] Possible devices for outputting visual information include, for example, a display device, a projector, a navigation device, an instrument panel, a CMS (Camera Monitoring System), an electronic mirror, a lamp, etc. The display device may be a device having a normal display, or may be a device that displays visual information within the field of view of a passenger, such as a head-up display, a see-through display, or a wearable device with an AR (Augmented Reality) function.

[0068] Possible devices for outputting auditory information include, for example, audio speakers, headphones, and earphones.

[0069] A device that outputs tactile information may be, for example, a haptic element that uses haptic technology, which may be provided on a steering wheel, a seat, or the like.

[0070] The vehicle control unit 32 controls each part of the vehicle 1. The vehicle control unit 32 includes a steering control unit 81, a brake control unit 82, a drive control unit 83, a body system control unit 84, a light control unit 85, and a horn control unit 86.

[0071] The steering control unit 81 detects and controls the state of the steering system of the vehicle 1. The steering system includes, for example, a steering mechanism including a steering wheel, an electric power steering, etc. The steering control unit 81 includes, for example, a control unit such as an ECU that controls the steering system, an actuator that drives the steering system, etc.

[0072] The brake control unit 82 detects and controls the state of the brake system of the vehicle 1. The brake system includes, for example, a brake mechanism including a brake pedal, an ABS (Antilock Brake System), etc. The brake control unit 82 includes, for example, a control unit such as an ECU that controls the brake system, and an actuator that drives the brake system.

[0073] The drive control unit 83 detects and controls the state of the drive system of the vehicle 1. The drive system includes, for example, an accelerator pedal, a drive force generating device for generating drive force such as an internal combustion engine or a drive motor, and a drive force transmission mechanism for transmitting the drive force to the wheels. The drive control unit 83 includes, for example, a control unit such as an ECU for controlling the drive system, and an actuator for driving the drive system.

[0074] The body system control unit 84 detects and controls the states of the body system systems of the vehicle 1. The body system systems include, for example, a keyless entry system, a smart key system, a power window device, a power seat, an air conditioning system, an airbag, a seat belt, a shift lever, etc. The body system control unit 84 includes, for example, a control unit such as an ECU that controls the body system systems, and an actuator that drives the body system systems.

[0075] The light control unit 85 detects and controls the states of various lights of the vehicle 1. Examples of lights to be controlled include headlights, backlights, fog lights, turn signals, brake lights, projections, and bumper displays. The light control unit 85 includes a control unit such as an ECU that controls the lights, an actuator that drives the lights, and the like.

[0076] The horn control unit 86 detects and controls the state of the car horn of the vehicle 1. The horn control unit 86 includes, for example, a control unit such as an ECU that controls the car horn, and an actuator that drives the car horn.

[0077] <<2. Embodiments>> Next, an embodiment of the present technology will be described with reference to FIGS.

[0078] <Configuration example of the external recognition sensor 25 and the in-vehicle sensor 26> 2 shows an example of the installation positions of the camera 51, radar 52, LiDAR 53, and ultrasonic sensor 54 provided in the external recognition sensor 25 of the vehicle 1, and the cameras provided in the interior sensor 26. In this example, the external recognition sensor 25 includes cameras 51FC1 to 51BC3, radars 52FC to 52BR, LiDARs 53F to 53B, and ultrasonic sensors 54FL1 to 54BR. The interior sensor 26 includes cameras 101L to 102.

[0079] The camera 51FC1 is provided near the center of the front end of the vehicle 1. The camera 51FC1 captures images ahead of the vehicle 1. For example, a fisheye camera is used as the camera 51FC1. The camera 51FC1 is used for, for example, ADAS and surround view. The surround view is a function that displays, for example, an image of the surroundings of the vehicle 1 or an overhead image of the surroundings of the vehicle 1 seen from above.

[0080] The camera 51FC2 is provided near the center of the front of the interior of the vehicle 1. The camera 51FC2 captures an image of the area in front of the vehicle 1 through the windshield. The camera 51FC2 is used in, for example, an ADAS. Images from the camera 51FC2 are recorded in, for example, a drive recorder.

[0081] The cameras 51FL and 51FR are arranged on the left and right at a predetermined distance apart in the front of the interior of the vehicle 1, forming a stereo camera. The cameras 51FL and 51FR capture images of the area in front of the vehicle 1 through the windshield. The cameras 51FL and 51FR are used, for example, in ADAS. The cameras 51FL and 51FR have a higher resolution than the camera 51FC2, for example.

[0082] As will be described later, the cameras 51SL1 to 51SL4 constitute one camera module 122L (FIG. 9, etc.), and are provided on the left side of the vehicle 1 near the front end of the driver's seat door.

[0083] The camera 51SL1 captures an image diagonally forward to the left of the vehicle 1. The camera 51SL1 is used in, for example, an ADAS.

[0084] The camera 51SL2 captures images to the left of the vehicle 1. The camera 51SL2 is, for example, a fisheye camera with a wider angle than the cameras 51SL1, 51SL3, and 51SL4. The camera 51SL2 is used for, for example, ADAS and surround view.

[0085] The camera 51SL3 captures an image diagonally rearward to the left of the vehicle 1. For example, a camera with a higher resolution than the camera 51SL1 is used as the camera 51SL3. The camera 51SL3 is used, for example, in an ADAS.

[0086] Camera 51SL4 photographs the area diagonally rear to the left of vehicle 1. The optical axis of camera 51SL4 faces in a direction closer to the rear of vehicle 1 than the optical axis of camera 51SL3. In other words, the optical axis of camera 51SL3 faces in a direction closer to the left (lateral direction) of vehicle 1 than the optical axis of camera 51SL4. Camera 51SL4 is used, for example, in a CMS.

[0087] As will be described later, the cameras 51SR1 to 51SR4 constitute one camera module 122R (FIG. 4, etc.), and are provided on the right side of the vehicle 1 near the front end of the passenger door.

[0088] The camera 51SR1 captures an image diagonally forward to the right of the vehicle 1. The camera 51SR1 is used in, for example, an ADAS.

[0089] The camera 51SR2 captures images to the right of the vehicle 1. For example, a fisheye camera with a wider angle than the cameras 51SR1, 51SR3, and 51SR4 is used as the camera 51SR2. The camera 51SR2 is used for, for example, ADAS and surround view.

[0090] The camera 51SR3 captures an image diagonally rearward to the right of the vehicle 1. For example, a camera with a higher resolution than the camera 51SR1 is used as the camera 51SR3. The camera 51SR3 is used, for example, in an ADAS.

[0091] Camera 51SR4 photographs the area diagonally rear to the right of vehicle 1. The optical axis of camera 51SR4 faces in a direction closer to the rear of vehicle 1 than the optical axis of camera 51SR3. In other words, the optical axis of camera 51SR3 faces in a direction closer to the right (lateral direction) of vehicle 1 than the optical axis of camera 51SR4. Camera 51SR4 is used, for example, in a CMS.

[0092] The camera 51BC1 is provided near the center of the rear end of the vehicle 1. The camera 51BC1 captures an image behind the vehicle 1. For example, a fisheye camera is used as the camera 51BC1. The camera 51BC1 is used for, for example, ADAS and surround view.

[0093] The camera 51BC2 and the camera 51BC3 are provided near the rear center of the interior of the vehicle 1. The camera 51BC2 and the camera 51BC3 capture images of the rear of the vehicle 1 through the rear window. The camera 51BC2 is used, for example, in a CMS. The camera 51BC3 is used, for example, in an ADAS. Images from the camera 51BC3 are recorded, for example, in a drive recorder.

[0094] The radar 52FC is provided near the center of the front end of the vehicle 1. The radar 52FC senses the area ahead of the vehicle 1.

[0095] The radar 52FL is provided near the left end of the front of the vehicle 1. The radar 52FL senses the area diagonally forward to the left of the vehicle 1.

[0096] The radar 52FR is provided near the right end of the front of the vehicle 1. The radar 52FR performs sensing in the diagonally forward right direction of the vehicle 1.

[0097] The radar 52BC is provided near the center of the rear end of the vehicle 1. The radar 52BC senses the area behind the vehicle 1.

[0098] The radar 52BL is provided near the left end of the rear end of the vehicle 1. The radar 52BL senses the area diagonally rearward and to the left of the vehicle 1.

[0099] The radar 52BR is provided near the right end of the rear end of the vehicle 1. The radar 52BR performs sensing of the area diagonally rear right of the vehicle 1.

[0100] The LiDAR 53F is provided near the center of the front end of the vehicle 1. The LiDAR 53F senses the area ahead of the vehicle 1.

[0101] The LiDAR 53L is provided in front of the left side of the vehicle 1. The LiDAR 53L performs sensing of the left direction of the vehicle 1.

[0102] The LiDAR 53R is provided in front of the right side of the vehicle 1. The LiDAR 53R performs sensing in the right direction of the vehicle 1.

[0103] The LiDAR 53B is provided near the center of the rear end of the vehicle 1. The LiDAR 53B performs sensing behind the vehicle 1.

[0104] The ultrasonic sensor 54FL1 is provided slightly to the left of the center of the front end of the vehicle 1. The ultrasonic sensor 54FL1 performs sensing in the front of the vehicle 1 slightly to the left.

[0105] The ultrasonic sensor 54FL2 is provided near the left end of the front of the vehicle 1. The ultrasonic sensor 54FL1 senses the area diagonally forward and to the left of the vehicle 1.

[0106] The ultrasonic sensor 54FR1 is provided slightly to the right of the center of the front end of the vehicle 1. The ultrasonic sensor 54FR1 performs sensing in the front of the vehicle 1 slightly to the right.

[0107] The ultrasonic sensor 54FR2 is provided near the right end of the front of the vehicle 1. The ultrasonic sensor 54FR1 senses the vehicle 1 in a diagonal direction to the right in front of the vehicle 1.

[0108] The ultrasonic sensor 54SL1 is provided on the front left side of the body of the vehicle 1. The ultrasonic sensor 54SL1 senses the area in front of the vehicle 1 in the left direction.

[0109] The ultrasonic sensor 54SL2 is provided at the rear of the left side of the body of the vehicle 1. The ultrasonic sensor 54SL2 is located at the rear of the vehicle 1 and performs sensing in the left direction.

[0110] The ultrasonic sensor 54SR1 is provided in front of the right side of the body of the vehicle 1. The ultrasonic sensor 54SR1 senses the area in front of the vehicle 1 in the right direction.

[0111] The ultrasonic sensor 54SR2 is provided at the rear of the right side of the body of the vehicle 1. The ultrasonic sensor 54SR2 is located at the rear of the vehicle 1 and performs sensing in the right direction.

[0112] The ultrasonic sensor 54BL is provided near the left end of the rear end of the vehicle 1. The ultrasonic sensor 54BL senses the area diagonally rearward and leftward of the vehicle 1.

[0113] The ultrasonic sensor 54BR is provided near the right end of the rear end of the vehicle 1. The ultrasonic sensor 54BR senses the vehicle 1 in a diagonal direction obliquely rearward to the right.

[0114] The camera 101L is installed in the vehicle diagonally above the right of the driver's seat. The camera 101L takes images of the area around the driver's seat. For example, a ToF camera is used as the camera 101L.

[0115] The camera 101R is installed in the vehicle diagonally above and to the left of the passenger seat. The camera 101R takes images of the area around the passenger seat. For example, a ToF camera is used as the camera 101R.

[0116] The camera 102 is installed on the dashboard inside the vehicle, slightly to the left of the center, and captures images of the area around the driver's seat.

[0117] <Example of fisheye camera coverage> FIG. 3 shows an example of the imaging range of a camera 51FC1, a camera 51SL2, a camera 51SR2, and a camera 51BC1 that use fisheye cameras.

[0118] Camera 51FC1 is capable of capturing images in a shooting range A1F exceeding 180 degrees in front of vehicle 1. Camera 51SL2 is capable of capturing images in a shooting range A1L exceeding 180 degrees to the left of vehicle 1. Camera 51SR2 is capable of capturing images in a shooting range A1R exceeding 180 degrees to the right of vehicle 1. Camera 51BC1 is capable of capturing images in a shooting range A1B exceeding 180 degrees behind vehicle 1. Therefore, as indicated by the diagonal lines in the figure, the four fisheye cameras, cameras 51FC1 to 51BC1, are capable of capturing images of 360 degrees around vehicle 1.

[0119] As described above, the cameras 51FC1 to 51BC1 are used for ADAS and surround view. For example, by combining images from the cameras 51FC1 to 51BC1, a bird's-eye view image of the 360-degree area around the vehicle 1 viewed from above is generated and displayed.

[0120] <Details of the camera module 122L and camera module 122R> Next, the camera modules 122L and 122R will be described in detail with reference to Figures 4 to 11. As described above, the camera module 122L includes the cameras 51SL1 to 51SL4. The camera module 122R includes the cameras 51SR1 to 51SR4.

[0121] First, examples of the installation positions of the camera module 122L and the camera module 122R will be described with reference to FIGS.

[0122] 4 is a front view of the vehicle 1. FIG.

[0123] Camera module 122L is provided in the driver's door 121FL at a position similar to that of a conventional door mirror. Specifically, camera module 122L is provided near the front end of the body of driver's door 121FL, near the boundary between the body and the window.

[0124] Camera module 122R is provided in a passenger door 121FR (not shown) at the same position as camera module 122L.

[0125] Next, a configuration example of the camera module 122L will be described with reference to FIGS.

[0126] FIG. 6A is a schematic diagram of the periphery of the camera module 122L of the vehicle 1 as viewed from the front, slightly diagonally leftward. FIG. 6B is a schematic diagram of a portion of the camera module 122L as viewed from substantially the same direction as FIG. 6A, with the camera module 122L partially disassembled. FIG. 7A is a schematic diagram of the periphery of the camera module 122L of the vehicle 1 as viewed from the front, diagonally leftward. FIG. 7B is a schematic diagram of a portion of the camera module 122L as viewed from substantially the same direction as FIG. 7A, with the camera module 122L partially disassembled. FIG. 8A is a schematic diagram of the periphery of the camera module 122L of the vehicle 1 as viewed from the rear. FIG. 8B is a schematic diagram of the camera module 122L as viewed from substantially the same direction as FIG. 8A. FIG. 9 is an exploded view of the camera module 122L. FIG. 10 is a diagram of the installation positions of the cameras 51SL1 to 51SL4 in the camera module 122L as viewed from the front. FIG. 11 is a diagram of the installation positions of the cameras 51SL1 to 51SL4 in the camera module 122L as viewed from above.

[0127] 9, cameras 51SL1 to 51SL4 are built into housing 131L and fixed to the bottom plate of housing 131L. Cameras 51SL1 to 51SL4 are each directed in a different direction outside housing 131L so that they are not within each other's angles of view. Space is provided behind cameras 51SL1 to 51SL4 so that cables can be routed.

[0128] Camera 51SL1 is disposed so that its optical axis faces horizontally and diagonally slightly to the left from the front of vehicle 1, and its lens looks out from the front of housing 131L. That is, the optical axis of camera 51SL1 faces horizontally to vehicle 1 and faces in a direction rotated slightly leftward from the front direction in the yaw direction of vehicle 1.

[0129] The angle of view of the camera 51SL1 is, for example, 120 degrees, so that the camera 51SL1 can capture a wide range of images from the front of the vehicle 1 to the diagonally forward left.

[0130] Camera 51SL2 is arranged so that its optical axis faces diagonally downward (for example, 32 degrees downward from the horizontal) and to the left (lateral direction) of vehicle 1, and its lens can be seen from the left side surface of housing 131L.

[0131] As described above with reference to FIG. 3, the camera 51SL2 is configured as a fisheye camera, and is capable of capturing images in a capturing range A1L of more than 180 degrees to the left of the vehicle 1.

[0132] Here, the optical axis of a camera normally used for surround view is directed vertically downward. In contrast, the optical axis of camera 51SL2 is directed diagonally downward. Camera 51SL2 is also configured as a fisheye camera. Therefore, camera 51SL2 can capture a wide range of images, from the ground around the left side of vehicle 1 to the tops of vehicles and motorbikes in the adjacent lane to the left.

[0133] Camera 51SL3 is disposed so that its optical axis is horizontal and faces diagonally rear left of vehicle 1, and its lens is viewed from the left side surface of housing 131L. That is, the optical axis of camera 51SL3 faces horizontally relative to vehicle 1, and faces in a direction rotated leftward by a predetermined angle from the left direction in the yaw direction of vehicle 1.

[0134] The angle of view of the camera 51SL3 is, for example, 120 degrees, so that the camera 51SL3 can capture a wide range of images from the left direction of the vehicle 1 to the left rear diagonal direction.

[0135] Camera 51SL4 is positioned so that its optical axis faces slightly diagonally downward and diagonally rearward to the left of vehicle 1, and the lens looks out from near the boundary between the left side surface and rear surface of housing 131L. That is, the optical axis of camera 51SL4 faces slightly downward from the horizontal direction of vehicle 1, and faces in a direction rotated leftward by a predetermined angle from the left direction in the yaw direction of vehicle 1. In addition, the optical axis of camera 51SL4 faces closer to the rear of vehicle 1 than the optical axis of camera 51SL3.

[0136] Therefore, camera 51SL4 can capture images in the left rear direction, which is closer to the rear of vehicle 1 than camera 51SL3. In addition, by tilting the optical axis of camera 51SL4 slightly downward from the horizontal, it becomes possible to capture a wide image of the road surface diagonally rearward and to the left of vehicle 1.

[0137] For example, the width of the housing 131L is 160 mm, the depth is 202 mm, and the height is 69 mm. The housing 131L is made of metal, and can release heat from the cameras 51SL1 to 51SL4 to the outside.

[0138] In this way, the four cameras 51SL1 to 51SL4, which have different shooting directions and purposes, are compactly arranged in the camera module 122L. Therefore, compared to when the cameras 51SL1 to 51SL4 are arranged individually, the space in the vehicle 1 can be used more effectively and the design is improved. Furthermore, the degree of freedom in arranging the cameras 51SL1 to 51SL4 (camera module 122L) is improved, making it easier to design the layout and wiring of sensors, including other cameras. This prevents the system of the vehicle 1, including each camera, from becoming too complicated and makes it simpler.

[0139] Although not shown, in the camera module 122R, the cameras 51SR1 to 51SR4 are arranged symmetrically with the cameras 51SL1 to 51SL4 in the camera module 122L.

[0140] <Interior configuration example of vehicle 1> Next, an example of the interior configuration of the vehicle 1 will be described with reference to FIGS.

[0141] FIG. 12 is a schematic diagram of the interior of vehicle 1 viewed from the right. FIG. 13 is a schematic diagram showing the front of the interior of vehicle 1. FIG. 14 is a schematic diagram showing the front of driver's seat 201 in the interior of vehicle 1. FIG. 15 is an enlarged view of the area around steering wheel 206 of vehicle 1. FIG. 16 is a diagram showing an example of the installation positions of camera 101L and camera 101R in camera module 214. The upper left diagram of FIG. 16 is a schematic diagram of camera module 214 viewed from the diagonally lower left. The lower left diagram of FIG. 16 is a schematic diagram of camera module 214 viewed from the diagonally lower left. The right diagram of FIG. 16 is a schematic diagram of camera module 214 viewed from the diagonally lower front. FIG. 17 shows an example of the imaging range of camera 101L. FIG. 18 is a schematic diagram of the interior of vehicle 1 viewed from the front. FIG. 19 is a schematic diagram of driver's seat 201 viewed from the diagonally rear left.

[0142] <Example of installation location of passenger cameras> First, an example of the installation position of the camera for passengers in the vehicle 1 will be described.

[0143] 13 and 15, camera 102 is provided on dashboard 205, slightly closer to driver's seat 201 than the center in the left-right direction. Camera 102 has a shape like a part of a truncated cone cut out vertically, and has a vertical flat portion on its side. The flat portion on the side of camera 102 is directed toward driver's seat 201, and captures an image of an area that includes at least the head of the driver sitting in driver's seat 201.

[0144] The camera 102 is capable of two types of photography: RGB photography (color photography) and IR photography (infrared photography). The camera 102 automatically switches between RGB photography and IR photography based on conditions such as the ambient brightness. The camera 102 also has a built-in LED that emits IR light (infrared light).

[0145] For example, in a bright environment such as daytime, the camera 102 automatically performs RGB photography to capture a color image. On the other hand, in a dark environment such as nighttime, the camera 102 automatically turns on the LED and automatically performs IR photography to capture a monochrome image. This ensures that the driver's head can be captured reliably regardless of the ambient brightness.

[0146] The DMS 30 (FIG. 1) performs lip reading of the driver and recognizes the content of the driver's speech based on, for example, the image from the camera 102. The DMS 30 performs, for example, the detection of the driver's pupils, the detection of the driver's line of sight, the determination of wakefulness, recognition processing, and the like based on the image from the camera 102.

[0147] 13 and 16, camera module 214 incorporating camera 101L and camera 101R is provided at the tip of the ceiling of vehicle 1, in the center in the left-right direction, near digital rearview mirror 234. Camera module 214 has a shape in which a base formed by cutting the left and right sides of a rectangular parallelepiped diagonally is connected to the rear end (rear side of vehicle 1) of the base.

[0148] Camera 101L is built in at position P1L near the front end on the left side of the truncated cone portion of camera module 214. The optical axis of camera 101L is directed toward the middle of the driver's eyes when, for example, a driver of average build sits in driver's seat 201 placed in a standard position.

[0149] 17 shows an example of the imaging range of the camera 101L. As shown in this example, the camera 101L can capture almost the entire body of the driver DR1, from the head to the feet, from an upper right diagonal direction, regardless of the physique of the driver DR1 and the position of the driver's seat 201.

[0150] Camera 101R is built in at position P1R near the front end on the right side of the truncated cone portion of camera module 214. Although not shown in the drawings, camera 101R, like camera 101L, is capable of capturing an image of almost the entire body of a passenger sitting in passenger seat 202, from the diagonally upper left direction, including from the head to the feet, regardless of the physique of the passenger sitting in passenger seat 202 or the position of passenger seat 202.

[0151] A glossy black cover 214A is provided on the side of the truncated cone portion of camera module 214. Cover 214A makes it difficult to see the lenses of camera 101L and camera 101R from the outside. It is also possible to provide cameras and sensors other than camera 101L and camera 101R on the back side of cover 214A.

[0152] For example, the DMS 30 recognizes the posture and movement of the driver by performing skeleton recognition of the driver based on the image of the camera 101L. For example, the DMS 30 performs recognition processing of the driver (for example, personal recognition) based on the image of the camera 101L.

[0153] Similarly, for example, the DMS 30 recognizes the posture and movement of the occupant by performing skeleton recognition or the like of the occupant in the passenger seat 202 based on the image of the camera 101R. For example, the DMS 30 performs recognition processing of the occupant based on the image of the camera 101L.

[0154] A circular light 215 is provided on the underside of the camera module 214. The camera module 214 also has a built-in microphone (not shown) for collecting the voice of the passenger and the like.

[0155] <Example of configuration around the steering wheel 206> Next, an example of the configuration around the steering wheel 206 will be described.

[0156] As shown in FIG. 15, illumination 207 is provided in the center of steering wheel 206. Illumination 207 includes a plurality of LEDs arranged in a ring shape at a predetermined distance inward from the outer periphery of the center of steering wheel 206. Each LED can be individually controlled for on / off, color, brightness, etc. Therefore, the color, brightness, and light-emitting area (light-emitting range) of illumination 207 are variable. The logo of vehicle 1 is formed inside illumination 207.

[0157] An airbag (not shown) is housed in the center of the steering wheel 206. The illumination 207 is positioned to avoid the area where the center of the steering wheel 206 may break when the airbag is deployed. This prevents fragments of the illumination 207 and harmful substances from scattering when the airbag is deployed.

[0158] An operation unit 208 having a plurality of buttons is provided on the left spoke of the steering wheel 206. An operation unit 209 having a plurality of buttons is provided on the right spoke of the steering wheel 206.

[0159] 15, a stalk lever 210, which is a rod-shaped operating body, is provided so as to extend leftward from a steering column (not shown) at the rear of the steering wheel 206. In addition, a stalk lever 211, which is a rod-shaped operating body, is provided so as to extend rightward from a steering column (not shown) at the rear of the steering wheel 206.

[0160] <Display configuration example> Next, an example of the configuration of the display inside the vehicle 1 will be described.

[0161] The interior of the vehicle 1 is provided with a center display 231, a console display 232, a head-up display (only a display 233 is shown), a digital rearview mirror 234, and tablet terminals 235L and 235R.

[0162] 13, center display 231 is provided in front of driver's seat 201 and passenger seat 202, extending laterally on the front surface of dashboard 205. Center display 231 is roughly divided into left end 231L, central portion 231C, and right end 231R depending on the orientation of the display. That is, center display 231 has a configuration in which left end 231L, central portion 231C, and right end 231R, which are oriented in different directions, are continuous and integrated in the left-right direction. Left end 231L, central portion 231C, and right end 231R can each display independently, or can display integrally.

[0163] The central portion 231C extends left and right in front of the driver's seat 201 and passenger seat 202 from near the left end of the driver's seat 201 to near the right end of the passenger seat 202, and faces rearward (rearward of the vehicle 1) when viewed from the driver's seat 201 or passenger seat 202.

[0164] For example, the central portion 231C displays information to assist driving, an image of the surroundings of the vehicle 1, etc. For example, the central portion 231C displays information related to the driving state of the vehicle 1, etc. For example, the central portion 231C displays the operating state of the turn signals of the vehicle 1. For example, the central portion 231C displays the speed and shift position of the vehicle 1. For example, the central portion 231C displays the remaining battery charge of the vehicle 1. For example, the central portion 231C displays an image showing the condition of the road ahead. For example, the central portion 231C displays information to assist parking when parking the vehicle 1. For example, the central unit 231C can display a portion of a 360-degree image of the surroundings of the vehicle 1 captured by the cameras 51FC1, 51SL2, 51SR2, and 51BC1, and can rotate the displayed range. For example, the central unit 231C displays a bird's-eye view image of the surroundings of the vehicle 1 viewed from above, obtained by combining the images of the cameras 51FC1, 51SL2, 51SR2, and 51BC1.

[0165] For example, when the vehicle 1 is backing up, the center portion 231C displays an image of the rear of the vehicle 1. For example, when the LKA (Lane Keep Assist) function is operating when the vehicle 1 is backing up, the center portion 231C displays information indicating the predicted traveling direction.

[0166] In this example, central portion 231C of center display 231 is divided into display portion 231CL in front of driver's seat 201, display portion 231CC between driver's seat 201 and passenger seat 202, and display portion 231CR in front of passenger seat 202. For example, display portions 231CL to 231CR can be linked to form a single display portion. For example, display portion 231CL and display portion 231CC can be linked to form a single display portion. For example, display portion 231CC and display portion 231CR can be linked to form a single display portion. The above-described display example can be displayed on any of the display portions of central portion 231C.

[0167] For example, the display unit 231CL mainly displays information for the driver. For example, information to assist driving is displayed. For example, the display unit 231CC displays infotainment-related information (in-vehicle infotainment) such as audio, video, websites, and maps. For example, the display unit 231CR displays infotainment-related information for the passenger in the front passenger seat.

[0168] The left end 231L and the right end 231R of the center display 231 are provided approximately symmetrically on the left and right ends of the center display 231. The left end 231L bends inward (toward the interior of the vehicle) at the left end of the center display 231, is angled toward the interior of the vehicle with respect to the central portion 231C, and faces diagonally rear to the right (diagonally rear to the right of the vehicle 1) when viewed from the driver's seat 201 or the passenger seat 202. The right end 231R bends inward (toward the interior of the vehicle) at the right end of the center display 231, is angled toward the interior of the vehicle with respect to the central portion 231C, and faces diagonally rear to the left (diagonally rear to the left of the vehicle 1) when viewed from the driver's seat 201 or the passenger seat 202.

[0169] The angle of left end 231L with respect to center 231C is adjusted, for example, so that the reflection angle with respect to the incident angle of the line of sight of a standard driver to left end 231L faces an appropriate direction diagonally rear to the left of vehicle 1. The angle of right end 231R with respect to center 231C is adjusted, for example, so that the reflection angle with respect to the incident angle of the line of sight of a standard driver to right end 231R faces an appropriate direction diagonally rear to the right of vehicle 1.

[0170] A display unit 231LL is provided at the left end 231L, and a display unit 231RR is provided at the right end 231R.

[0171] The left end 231L and right end 231R of the center display 231 are mainly used as digital outer mirrors (electronic side mirrors) that replace conventional side mirrors. That is, the left end 231L and right end 231R are used for the CMS. For example, the display section 231LL of the left end 231L displays an image of the left rear view of the vehicle 1 captured by the camera 51SL4. The display section 231RR of the right end 231R displays an image of the right rear view of the vehicle 1 captured by the camera 51SR4.

[0172] The left end 231L, center portion 231C, and right end 231R of the center display 231 are continuous in the horizontal direction. Therefore, when the driver moves his / her gaze from the center portion 231C to the left end 231L or the right end 231R, or from the left end 231L or the right end 231R to the center portion 231C, he / she only needs to move his / her gaze horizontally. As a result, the distance and direction of movement of the driver's gaze are reduced, improving visibility.

[0173] Furthermore, for example, the left end 231L displays an alert when a vehicle, motorcycle, bicycle, pedestrian, etc. approaching from the left side or rear of the vehicle 1 is detected. Similarly, the right end 231R displays an alert when a vehicle, motorcycle, bicycle, pedestrian, etc. approaching from the right side or rear of the vehicle 1 is detected.

[0174] Vehicles approaching from the left, right, left, and rear sides of vehicle 1 are detected by recognition unit 73 based on, for example, images (sensor data) from camera 51SL2, camera 51SL3, camera 51SR2, camera 51SR3, camera 51BC1, and camera 51BC3, as well as sensor data from radar 52BL, radar 52BC, radar 52BR, and LiDAR 53B.

[0175] Since the left end 231L, the center 231C, and the right end 231R are connected together, it is possible to display one screen on the entire center display 231. For example, it is possible to display an image of the surroundings of the vehicle 1, map information, infotainment-related information, or the like on the entire center display 231.

[0176] As shown in FIGS. 12 and 13, the console display 232 is provided on the console 212 provided between the driver's seat 201 and the passenger seat 202, and is arranged below the center portion 231C of the center display 231.

[0177] Specifically, the console 212 extends in the front-to-rear direction between the driver's seat 201 and the passenger's seat 202. The tip of the console 212 is inclined obliquely upward toward the front of the vehicle 1, and a console display 232 is provided thereon.

[0178] The console display 232 is configured, for example, with a two-dimensional or three-dimensional touch panel, and can be operated by touching or approaching with a finger or the like. The console display 232 faces toward the rear of the vehicle 1. The console display 232 faces diagonally upward at approximately the same angle as the central portion 231C of the center display 231. This creates a sense of unity between the center display 231 and the console display 232, as if they were connected together. Similar to the central portion 231C of the center display 231, the visibility of the console display 232 is improved.

[0179] For example, the console display 232 displays an operation screen for operating an air conditioning system in the vehicle. A driver or other passenger operates the air conditioning system in the vehicle using the displayed operation screen.

[0180] For example, the console display 232 displays an operation screen for operating information displayed on the center display 231. For example, a passenger such as a driver uses the displayed operation screen to scroll, enlarge, reduce, switch, etc., information (for example, a map) displayed in the central portion 231C of the center display 231.

[0181] As described above, there is a sense of unity between the center display 231 and the console display 232. Therefore, the passenger can naturally use the operation screen of the console display 232 to operate the information displayed on the center display 231, improving operability.

[0182] For example, the console display 232 displays an operation screen for setting the display range of the display units 231LL and 231RR of the center display 231. For example, the driver uses the displayed operation screen to zoom in, zoom out, move, etc. the display range of the display units 231LL and 231RR.

[0183] The middle part of the console 212 is inclined diagonally downward toward the front of the vehicle 1, opposite to the tip part. A circular controller 213 is provided near the center of the middle part. The inclination angle of the middle part of the console 212 is set to an angle that allows the driver to easily place their hands on it, thereby improving the operability of the controller 213.

[0184] Controller 213 can be pressed, rotated, or tilted in a predetermined direction (e.g., forward, backward, left, or right), and is used, for example, to operate an audio system in a car. For example, pressing controller 213 plays or stops music, and rotating controller 213 adjusts the volume.

[0185] The rear end of the console 212 is generally horizontal so that the driver can easily place their arms thereon, for example.

[0186] As shown in Fig. 13, the head-up display includes a display 233 provided in front of the driver's seat 201. For example, the display 233 may be configured as part of the windshield 204, or may be provided separately from the windshield 204. In the latter case, for example, the display 233 is attached to the windshield 204. Then, visual information is projected onto the display 233 using AR technology, and the visual information is superimposed and displayed within the field of view of the driver.

[0187] The display 233 of the head-up display displays, for example, information to assist driving. For example, the display 233 displays an alert in accordance with the situation around the vehicle 1. For example, the display 233 displays information indicating a vehicle ahead that is to be followed while the ACC (Adaptive Cruise Control) is operating. For example, the display 233 displays the planned route to be traveled with an arrow or the like as needed while the lane change assist function is operating.

[0188] Digital rearview mirror 234 is used in place of a conventional rearview mirror and is also called a smart rearview mirror. As shown in Fig. 13, digital rearview mirror 234 is provided at the top edge of windshield 204, slightly in front of the center, and above center portion 231C of center display 231, similar to a conventional rearview mirror.

[0189] The digital rearview mirror 234 displays, for example, an image of the rear of the vehicle 1 captured by the camera 51BC2. For example, when another vehicle is approaching behind the vehicle 1, the digital rearview mirror 234 displays an alert.

[0190] Furthermore, for example, the display range of digital rearview mirror 234 is adjusted based on at least one of the driver's line of sight and posture. As a result, the range of the image displayed on digital rearview mirror 234 is appropriately adjusted according to the driver's line of sight and posture, just like a conventional physical rearview mirror.

[0191] As shown in FIG. 19, the tablet terminal 235L is provided on the back of the driver's seat 201, more specifically, on the back of the headrest of the driver's seat 201.

[0192] The tablet terminal 235L, for example, presents infotainment-related information to a passenger in the rear seat 203L and accepts operations on the presented information. In addition, for example, the tablet terminal 235L displays an alert in the event of an emergency or danger.

[0193] Although not shown in the drawings, for example, a tablet terminal 235R similar to the tablet terminal 235L is provided on the back of the passenger seat 202.

[0194] The displays of the center display 231, the console display 232, the head-up display, the digital rearview mirror 234, the tablet terminal 235L, and the tablet terminal 235R are controlled by a display control unit that is part of the functions of the HMI 31, for example.

[0195] As described above, by displaying various types of information on each display inside the vehicle, convenience for the driver and other passengers can be improved. For example, the information can assist the driver in driving the vehicle 1, provide necessary information to the passengers, and entertain the passengers.

[0196] 19, for example, a ToF camera may be provided at a position P2L near the top and center of the tablet terminal 235L. This makes it possible to recognize, for example, the passenger operating the tablet terminal 235L based on an image captured by the ToF camera.

[0197] The tablet terminal 235R can also be provided with a ToF camera.

[0198] <Speaker installation position> Next, examples of the installation positions of speakers inside the vehicle 1 will be described.

[0199] As shown in Fig. 12, speaker 236FL is embedded near the rear of door opener 216FL in door 121FL on the driver's seat 201 side. Although not shown, speaker 236FR is embedded near the rear of door opener 216FR in door 121FR on the passenger's seat 202 side. Speaker 236BL is embedded near the center of door 121BL on the rear seat 203L side. Although not shown, speaker 236BR is embedded near the center of door 121BR on the rear seat 203R side.

[0200] As shown in Fig. 18, a seat speaker 237 is embedded below the headrest of the driver's seat 201. A seat speaker 238 is embedded below the headrest of the passenger seat 202. A seat speaker 239L is embedded in the left rear seat 203L. A seat speaker 239R is embedded in the right rear seat 203R.

[0201] The shape of the seats and the position of the seat speakers have been adjusted so that people of various heights (sitting heights) can clearly hear the sound from the seat speakers in each seat.

[0202] Speakers 236FL to 236BR embedded in doors 121FL to 121BR are used, for example, to output sound to the entire vehicle interior (all passengers inside the vehicle). Speakers 236FL to 236FR also realize 360-degree real audio. By realizing 360-degree real audio, for example, it becomes possible to enjoy videos, music, etc. with realistic sound inside the vehicle. Furthermore, the location of dangerous objects such as obstacles present around vehicle 1 can be notified by the direction of sound output.

[0203] On the other hand, the seat speakers of each seat are used, for example, to output private sounds for the individual passengers sitting in each seat, that is, the sounds output from each seat speaker are individually controlled.

[0204] For example, the speakers may be arranged on the dashboard 205 at the front of the vehicle 1.

[0205] <Vehicle 1 system architecture> Next, the system architecture of the vehicle 1 will be described with reference to FIGS.

[0206] FIG. 20 shows an example of the configuration of an information processing system 301, which is an example of the system architecture of the vehicle 1.

[0207] The information processing system 301 is a system that employs a domain-type E / E (electrical / electronic) architecture.

[0208] In the information processing system 301, three domain controllers (DCs), namely, an ePWT (e-Power) domain controller (DC) 321, an ADAS domain controller (DC) 331, and an HMI+body domain controller (DC) 341, are connected to a secure gateway 311 via a network. The information processing system 301 is divided into three domains D1 to D3, one for each DC.

[0209] The ePWT DC321 controls the transfer and processing of data within the domain D1, which includes the components and control parts of the powertrain of the vehicle 1. The ePWT DC321 is connected to, for example, a front EDU (Electric Drive Unit) 322, a rear EDU 323, a charger ECU 324, an EPS (Electric Power Steering) ECU 325, a brake ECU 326, and a damper ECU 327 via a network.

[0210] The front EDU 322 is a unit that drives the front wheels of the vehicle 1.

[0211] The rear EDU 323 is a unit that drives the rear wheels of the vehicle 1.

[0212] The charger ECU 324 controls a charger for the battery for driving the front EDU 322, the rear EDU 323, and the like.

[0213] The EPS ECU 325 controls the electric power steering of the vehicle 1.

[0214] The brake ECU 326 controls the brakes of the vehicle 1 .

[0215] The damper ECU 327 controls the dampers of the vehicle 1 .

[0216] The ADAS DC 331 controls the transfer and processing of data within the domain D2, which includes components that realize the ADAS of the vehicle 1. The ADAS DC 331 is connected to, for example, a radar 52, a LiDAR 53, a camera 332, a USS (ultrasonic sensor) ECU 333, a GNSS ECU 334, a data logger 335, and the like via a network.

[0217] The cameras 332 include, for example, all of the cameras 51 of the vehicle 1 except for the camera 51SL4, the camera 51SR4, and the camera 51BC2 for the CMS.

[0218] The USS ECU 333 controls the ultrasonic sensor 54 .

[0219] The GNSS ECU 334 controls the GNSS receiver 24 and the like.

[0220] The data logger 335 records data related to the ADAS processing.

[0221] The HMI+body DC341 controls the transfer and processing of data within a domain D3 that includes components that realize the HMI of the vehicle 1, as well as body system components and control components of the vehicle 1. To the HMI+body DC341, for example, via a network, a door controller 342, a wiper controller 343, a seat controller 344, display units 231CL to 231CR of the central part 231C of the center display 231, a tablet terminal 235L, a tablet terminal 235R, an interface 345, and the like are connected. In addition, for example, a CMS 349 is connected to the HMI+body DC341.

[0222] The door controller 342 controls the doors 121FL to 121BR of the vehicle 1.

[0223] The wiper controller 343 controls the wipers of the vehicle 1.

[0224] The seat controller 344 controls, for example, the positions and attitudes of the driver's seat 201, the passenger seat 202, the rear seat 203L, and the rear seat 203R.

[0225] The interface 345 includes, for example, components and control parts that perform various interfaces other than the display units 231CL to 231CR, the tablet terminal 235L, and the tablet terminal 235R. For example, the interface 345 includes speakers 236FL to 236BR, and seat speakers 237 to 239R.

[0226] The CMS 346 includes, for example, the components and control parts of the CMS. The CMS 349 includes, for example, a camera 51SL4, a camera 51SR4, a camera 51BC2, and a display unit 231LL at the left end 231L and a display unit 231RR at the right end 231R of the center display 231.

[0227] Furthermore, the secure gateway 311 is connected to a TCM (Telematics Computing Unit) 312 that communicates with the outside world.

[0228] FIG. 21 shows a specific example of the configuration of the information processing system 301, mainly related to the CMS 349, the camera module 122L, and the camera module 122R.

[0229] The secure gateway 311 is connected to the ADAS DC 331, the HMI+Body DC 341, and the ToF ECU 361. The secure gateway 311 is connected to the ADAS DC 331, the HMI+Body DC 341, and the ToF ECU 361 via, for example, Ethernet.

[0230] The ADAS DC 331 is connected to the cameras 51SL1 to 51SL3 provided in the camera module 122L, the cameras 51SR1 to 51SR3 provided in the camera module 122R, the camera 51BC1, the radar 52BC, and the LiDAR 53B.

[0231] The ToF ECU 361 is connected to a camera 101L and a camera 101R that are configured as ToF cameras.

[0232] The HMI+body DC 341 is connected to the CMS ECU 362, the display units 231CL to 231CR of the center display 231, and the sound amplifier 363. The HMI+body DC 341 and the CMS ECU 362 are connected via, for example, a USB cable.

[0233] The camera 51SL4 included in the camera module 122L, the camera 51SR4 included in the camera module 122R, and the display units 231LL and 231RR of the center display 231 are connected to the CMS ECU 362. That is, of the camera module 122L, the cameras 51SL1 to 51SL3 are connected to the ADAS DC 331 and belong to domain D2, while the camera 51SL4 is connected to the HMI+body DC 341 via the CMS ECU 362 and belongs to domain D3. Of the camera module 122R, the cameras 51SR1 to 51SR3 are connected to the ADAS DC 331 and belong to domain D2, while the camera 51SR4 is connected to the HMI+body DC 341 via the CMS ECU 362 and belongs to domain D3.

[0234] A speaker 364 is connected to the sound amplifier 363 .

[0235] The camera 51SL1 captures an image of the area diagonally forward to the left of the vehicle 1, as described above.

[0236] As described above, the camera 51SL2 is configured as a fish-eye camera, and captures the left direction of the vehicle 1.

[0237] The camera 51SL3 captures an image of the area diagonally rear left of the vehicle 1, as described above.

[0238] The camera 51SR1 captures an image of the area diagonally forward to the right of the vehicle 1, as described above.

[0239] As described above, the camera 51SR2 is configured as a fisheye camera, and captures images to the right of the vehicle 1.

[0240] The camera 51SR3 captures an image of the area diagonally rear right of the vehicle 1, as described above.

[0241] As described above, the camera 51BC1 is configured as a fish-eye camera, and captures an image of the area behind the vehicle 1.

[0242] The radar 52BC performs sensing behind the vehicle 1 as described above.

[0243] The LiDAR 53B performs sensing behind the vehicle 1 as described above.

[0244] The ADAS DC 331 realizes, for example, a part of the recognition unit 73 of the vehicle 1. For example, the ADAS DC 331 recognizes objects outside the vehicle 1 based on sensor data from the cameras 51SL1 to 51SL3, the cameras 51SR1 to 51SR3, the camera 51BC1, the radar 52BC, and the LiDAR 53B. The ADAS DC 331 supplies data indicating the recognition results to other domains, etc., via the secure gateway 311.

[0245] The ToF ECU 361 realizes, for example, a part of the recognition unit 73 of the vehicle 1. For example, the ToF ECU 361 recognizes the posture, movement, etc. of the driver based on an image from the camera 101L. The ToF ECU 361 recognizes the posture, movement, etc. of the passenger in the passenger seat 202 based on an image from the camera 101R. The ToF ECU 361 supplies data indicating the recognition result to other domains, etc. via the secure gateway 311.

[0246] The camera 51SL4 captures an image of the area diagonally rear left of the vehicle 1, as described above.

[0247] The camera 51SR4 captures an image of the right rear of the vehicle 1 as described above.

[0248] The CMS ECU 362 controls the CMS. For example, the CMS ECU 362 is part of the function of the HMI 31 and implements a display control unit that controls the displays of the display units 231LL and 231RR. For example, the CMS ECU 362 controls the display by the display unit 231LL of the image of the camera 51SL4. For example, the CMS ECU 362 controls the display by the display unit 231RR of the image of the camera 51SR4. For example, the CMS ECU 362 controls the displays by the display units 231LL and 231RR of the visual effects based on the object recognition results by the ADAS DC 331.

[0249] For example, the HMI + body DC 341 implements a display control unit that controls the displays of the display units 231CL to 231CR of the center display 231.

[0250] The speaker 364 includes, for example, the speakers 236FL to 236BR, and the seat speakers 237 to 239R.

[0251] The sound amplifier 363 amplifies various voices output from the speaker 364.

[0252] The HMI + body DC 341 is, for example, part of the function of the HMI 31 and implements an audio control unit that controls the output of various voices from the speaker 364.

[0253] Thus, in the vehicle 1, by adopting the domain type E / E architecture, the complication of the information processing system 301 can be suppressed and made simpler. As a result, the number of components and wiring inside the vehicle can be reduced.

[0254] <Operation Example of CMS> Next, referring to FIGS. 22 to 37, an operation example of the CMS of the vehicle 1 will be described. Specifically, a display example of the display unit 231LL at the left end portion 231L of the center display 231 will be described.

[0255] The display control unit, which is a part of the functions of the HMI 31, causes the display unit 231LL to display an image based on the image captured by the camera 51SL4. More specifically, the display control unit sets a display range within the image captured by the camera 51SL4, and causes the display unit 231LL to display the image within the set display range.

[0256] The recognition unit 73 performs object recognition based on sensor data from the cameras 51SL1 to 51SL3, the cameras 51SR1 to 51SR3, the camera 51BC1, the radar 52BC, and the LiDAR 53B, and recognizes objects outside the vehicle 1. The display control unit controls the display on the display unit 231LL based on the object recognition results, the state of the vehicle 1, etc.

[0257] 22 shows an example of the display on the display unit 231LL in normal operation. In this example, a part of the body of the vehicle 1, a vehicle 401 that is a following vehicle traveling in the adjacent lane to the left of the vehicle 1, and a road surface 402 are displayed.

[0258] For example, when vehicle 401 approaches vehicle 1 and the distance between vehicle 401 and vehicle 1 falls within a predetermined range, a vertically elongated band-shaped visual effect, bar 403, is displayed superimposed on the left edge of display unit 231LL, as indicated by diagonal lines in Fig. 23. Bar 403 is displayed, for example, when vehicle 401 is within the range displayed on display unit 231LL and the distance between vehicle 401 and vehicle 1 is within a predetermined range. The color of bar 403 is a conspicuous color, for example, yellow.

[0259] The display of the bar 403 alerts the driver, and an accident can be reliably prevented.

[0260] The display mode of the bar 403, such as the color, brightness, shape, blinking pattern, and movement, is not limited to this example and can be changed as desired. For example, the display mode of the bar 403 changes depending on the level of danger.

[0261] For example, when the left turn signal of vehicle 1 is turned on, there is a possibility that vehicle 1 will turn left (change direction to the left) or change lanes to the left lane, increasing the risk of collision or contact with vehicle 401. In response to this, for example, as shown in Fig. 24, bar 403 becomes wider and flashes, thereby further attracting the driver's attention.

[0262] Furthermore, for example, when the left turn signal of vehicle 1 is turned on, the blinking speed of bar 403 changes depending on the degree of danger. For example, the faster the absolute speed of vehicle 401 or the relative speed with respect to vehicle 1, the faster the blinking speed of bar 403. For example, the closer vehicle 401 is, in other words, the shorter the distance between vehicle 401 and vehicle 1, the faster the blinking speed of bar 403. Note that, for example, a warning sound may be sounded in synchronization with the blinking of bar 403.

[0263] For example, when the absolute speed of vehicle 401 or the relative speed with respect to vehicle 1 is equal to or greater than a predetermined threshold, the timing at which bar 403 is displayed is advanced. In other words, bar 403 is displayed from the point in time when vehicle 401 is positioned farther away from vehicle 1.

[0264] 25, when a vehicle 401 is recognized diagonally rearward to the left, a mark 404, which is a triangular visual effect indicating the presence of the vehicle 401, may be superimposed and displayed near the vehicle 401. The mark 404 moves in accordance with the movement of the vehicle 401.

[0265] This allows the driver to reliably recognize the position of the vehicle 401 even when visibility of the image displayed on the display unit 231LL is poor due to fog or haze, for example.

[0266] The location of the vehicle 401 may be notified using a display mode different from the mark 404 in Fig. 25. For example, a frame surrounding the periphery of the vehicle 401 may be displayed.

[0267] Although detailed explanation will be omitted, the same display is also made on the display section 231RR at the right end 231R of the center display 231 based on the state of the vehicle traveling in the lane to the right of vehicle 1 and the operation of the right turn signal.

[0268] Furthermore, in the above explanation, an example of issuing a warning to other vehicles has been shown, but similar processing is also performed when issuing a warning to moving bodies other than vehicles (for example, motorcycles, etc.).

[0269] Furthermore, for example, the display control unit controls the range of the image displayed on the display unit 231LL in accordance with the operation of the direction indicator.

[0270] FIG. 26 shows an example in which the range of the image displayed on the display unit 231LL is changed in accordance with the operation of the left direction indicator.

[0271] 26A and 26B show examples of images captured by the camera 51SL4. Vehicles 421-1 to 421-4 traveling in the lane to the left of the vehicle 1 are shown in the image.

[0272] For example, when the left turn signal of the vehicle 1 is off, the image in the area A21a in A of Fig. 26 is extracted, and the image in the area A22a is displayed on the display unit 231LL. The area A22a includes the vehicles 421-2 to 421-4, but does not include the vehicle 421-1.

[0273] On the other hand, when the left turn signal of the vehicle 1 is turned on, the image in the area A21b in FIG. 26B is extracted, and the image in the area A22b is displayed on the display unit 231LL. The areas A21b and A22b are larger than the areas A21a and A22a. That is, the display range of the display unit 231LL is zoomed out (expanded) and widened. The rear end of the vehicle 421-1 is included in the area A22b in addition to the vehicles 421-2 to 421-4.

[0274] This allows the driver to check the lane conditions on the left side of the vehicle 1 over a wider area when turning on the left turn signal, which allows the driver to change lanes or turn left more safely.

[0275] For example, when the display range of the display unit 231LL is zoomed out, an icon, frame, or the like indicating that the display range has been zoomed out may be displayed on the display unit 231LL.

[0276] Here, UN ECE-R46 (United Nations Economic Commission for Europe Regulation 46) specifies the range of visibility that must be visible in CMS (hereinafter referred to as the required visibility range). The required visibility range differs depending on the vehicle's capacity, weight, etc., and Figure 27 shows an example.

[0277] In this example, the required field of view is made up of area A23L, area A23R, and area A23C.

[0278] Area A23L is an area surrounded by points P22L to P26L on the left side of vehicle 1. Specifically, the point on the left edge of vehicle 1 that corresponds to the same position as the driver's eyes in the longitudinal direction of vehicle 1 is set as reference point P21L. Point P22L is a point 4 m behind reference point P21L. Point P23L is a point 60 m behind reference point P21L. Point P24L is a point 1 m to the left of point P22L. Point P25L is a point 20 m behind and 4 m to the left of reference point P21L. Point P26L is a point 60 m behind and 4 m to the left of reference point P21L.

[0279] Area A23R is an area surrounded by points P22R to P26R on the right side of vehicle 1. Specifically, the point on the right edge of vehicle 1 that corresponds to the same position as the driver's eyes in the longitudinal direction of vehicle 1 is set as reference point P21R. Point P22R is a point 4 m behind reference point P21R. Point P23R is a point 60 m behind reference point P21R. Point P24R is a point 1 m to the right of point P22R. Point P25R is a point 20 m behind and 4 m to the right of reference point P21R. Point P25R is a point 60 m behind and 4 m to the right of reference point P21R.

[0280] The area A23C is located 60 m or more behind the eye position of the driver of the vehicle 1, and is an area 20 m wide with the center of the vehicle 1 in the lateral direction as its center.

[0281] Therefore, for example, as shown in FIG. 26, when the display range of the display unit 231LL is changed in response to the operation of the left direction indicator, the display range is changed so that the area A23L is always included.

[0282] Although detailed description will be omitted, the display range of the display unit 231RR of the center display 231 is also controlled in a similar manner in response to the operation of the right turn signal. The display range of the display unit 231RR is also changed so that the area A23R is always included.

[0283] Furthermore, for example, the display control unit controls the display range of the display unit 231LL based on the shift position of the vehicle 1.

[0284] FIG. 28 shows an example in which the display range of the display unit 231LL is changed based on the shift position of the vehicle 1.

[0285] 28A and 28B show examples of images captured by camera 51SL4. In the images, vehicles 441-1 and 441-2 are parked side by side to the left and rear of vehicle 1.

[0286] For example, before the shift position of the vehicle 1 is set to reverse, the image in the area A31a in A of FIG. 28 is extracted, and the image in the area A32a is displayed on the display unit 231LL.

[0287] On the other hand, after the shift position of the vehicle 1 is set to reverse, the image in area A31b in FIG. 28B is extracted, and the image in area A32b is displayed on the display unit 231LL. Areas A31b and A32b are larger than areas A31a and A32a. That is, the display range of the display unit 231LL is zoomed out and expanded. At this time, for example, the display range of the display unit 231LL may be automatically moved downward so that a wider view of the ground around the vehicle 1 is displayed.

[0288] Although detailed description will be omitted, the display range of the display unit 231RR of the center display 231 is also changed based on the shift position of the vehicle 1, similar to the display unit 231LL.

[0289] This allows the driver to check a wider range of the situation diagonally rearward and to the left of the vehicle 1. As a result, the driver can park the vehicle 1C more safely.

[0290] In addition, for example, the driver can set the display ranges of the display units 231LL and 231RR by operating the operation screen displayed on the console display 232 shown in Fig. 13 etc. For example, the driver can move the display range of the display unit 231LL and the display range of the display unit 231RR individually, or set the zoom factor for each display range individually.

[0291] Here, the display range can be set within a settable range A41 shown in Fig. 29. In Fig. 29, parts corresponding to those in Fig. 26 are given the same reference numerals, and descriptions thereof will be omitted as appropriate.

[0292] For example, the size of the extraction range A42 extracted for display from the captured image is set based on the set zoom ratio. That is, the extraction range A42 is zoomed in (reduced) and zoomed out (expanded) based on the zoom ratio. Then, an image within a predetermined display range A43 within the extraction range A42 is displayed on the display unit 231LL.

[0293] At this time, the display range A43 can be moved within the settable range A41. The settable range A41 is set to a range in which the area A23L is included within the display range A43.

[0294] In contrast to this, for example, when the area A23L protrudes from the display range A43 as shown in B of Fig. 29, a warning is issued to the driver. For example, a warning message is displayed on the display unit 231LL and a warning sound is output.

[0295] Alternatively, for example, the movement range of the display range A43 may be limited so that the display range A43 does not go outside the settable range A41 regardless of the driver's operation.

[0296] In this way, the driver can move the display range A43 to a position of their choice within the range that includes the area A23L, or zoom in or out of the display range A43, by operating the console display 232. Furthermore, the driver can adjust the display range A43 without taking their eyes off the screen, for example, by moving or clicking their finger on the console display 232 while remaining in a driving position.

[0297] In addition, for example, in the case of a physical side mirror, the driver can change the range of what is visible through the side mirror by moving their face closer to the side mirror and looking into it. For example, the driver may look into the side mirror when parking or the like to check for the presence or absence of obstacles on the ground around the vehicle.

[0298] Similarly, the display range of the display unit 231LL can be changed by the driver bringing his / her face closer to the display unit 231LL and peering into it.

[0299] For example, the DMS 30 detects the position and orientation of the driver's head by performing skeletal recognition of the driver based on the image of the camera 101L. When the DMS 30 recognizes the driver's action of looking into the display unit 231LL based on the position and orientation of the driver's head, it detects the angle at which the driver looks into the display unit 231LL.

[0300] On the other hand, when the vehicle 1 is stopped, that is, when the speed of the vehicle 1 is 0, the display control unit controls the display range of the display unit 231LL based on the angle at which the driver looks into the display unit 231LL.

[0301] 30 shows the display range of the display unit 231LL before the driver looks inside. In this example, a part of the body of the vehicle 1 is displayed in the lower right corner of the display unit 231LL. Also, a part of the upper part of an object 461 located to the left rear of the vehicle 1 is displayed.

[0302] In this case, for example, when the driver looks at the display unit 231LL from diagonally above, the display range of the display unit 231LL changes as shown in Fig. 31. Specifically, the display range of the display unit 231LL moves downward. As a result, the lower part of the object 461 is also displayed on the display unit 231LL, and the driver can see the entire object 461.

[0303] Additionally, as indicated by the diagonal lines in the figure, a line 462, which is a band-like visual effect indicating an area of ​​the ground within a predetermined range around the vehicle 1, is displayed superimposed and blinking. This allows the driver to quickly recognize objects that the vehicle 1 may collide with or come into contact with, and to move the vehicle 1 while avoiding the recognized objects.

[0304] 32 shows the display range of the display unit 231LL before the driver looks in. In this example, a part of the body of the vehicle 1 is displayed at the right end of the display unit 231LL. A part of the vehicle 481 on the left rear is displayed at the upper left of the display unit 231LL. A parking line 482 is displayed between the vehicles 1 and 481. The upper end of an object 483 is displayed to the left of the vehicle 1, at the lower end of the display unit 231LL.

[0305] Here, for example, when the driver sets the shift position to reverse to park the vehicle 1, the display range of the display unit 231LL zooms out as described above with reference to Fig. 28. This causes the display range of the display unit 231LL to expand, and the range in which the vehicle 1, the vehicle 481, and the object 483 are displayed to expand, as shown in Fig. 33.

[0306] Furthermore, for example, when the driver looks at the display unit 231LL from diagonally above, the display range of the display unit 231LL moves downward as shown in Fig. 34. This causes the entire object 483 to be displayed on the display unit 231LL. As a result, the driver can park the vehicle 1 while reliably avoiding the object 483.

[0307] Although detailed description will be omitted, when the driver looks into the display unit 231RR of the center display 231, the display range also moves in the same way as the display unit 231LL.

[0308] When the vehicle 1 is moving, that is, when the speed of the vehicle 1 is not 0, the display range does not change even if the driver looks into the display units 231LL and 231RR.

[0309] Furthermore, for example, when the driver approaches the digital rearview mirror 234 and looks into it, the display range of the digital rearview mirror 234 also moves.

[0310] For example, when the DMS 30 recognizes that the driver is looking into the digital rearview mirror 234 based on the position and orientation of the driver's head, it detects the angle at which the driver is looking into the digital rearview mirror 234.

[0311] On the other hand, when the vehicle 1 is stopped, that is, when the speed of the vehicle 1 is 0, the display control unit controls the display range of the digital rearview mirror 234 based on the angle at which the driver looks into the digital rearview mirror 234.

[0312] Furthermore, for example, the DMS 30 detects the movement of the driver's left hand by performing skeletal recognition of the driver based on the image from the camera 101L. The recognition unit 73 detects objects around the door 121FL of the driver's seat 201 outside the vehicle 1 based on images from the cameras 51SL2 and 51SL3. If an object is detected within a predetermined range around the door 121FL of the driver's seat 201, the display unit 231LL displays a warning when the driver makes an action to open the door 121FL, for example, when the driver's left hand touches the door opener 216FL of the door 121FL.

[0313] 35 shows an example of the display on the display unit 231LL before the driver's left hand touches the door opener 216FL. A part of the body of the vehicle 1 is displayed on the right edge of the display unit 231LL. A pedestrian 501 approaching the vehicle 1 from behind is displayed on the left side of the vehicle 1.

[0314] In this case, for example, as shown in Fig. 36, when the left hand of the driver DR1 touches the door opener 216FL, the display unit 231LL displays a warning, as indicated by the diagonal lines in Fig. 37. Specifically, a vertically long, band-like visual effect, a bar 502, is displayed superimposed on the left edge of the display unit 231LL and flashes. The color of the bar 502 is a conspicuous color, for example, yellow.

[0315] At this time, a warning sound may be output.

[0316] This alerts the driver DR1 to the pedestrian 501, and prevents the door 121FL from hitting the pedestrian 501 when the door 121FL is opened.

[0317] At this time, the body system control section 84 may increase the torque of the door 121FL to make it difficult to open the door 121FL.

[0318] Also, for example, when the hand of a passenger in the passenger seat 202 touches the door opener 216FR of the door 121FR on the passenger seat 202 side, the display unit 231RR on the passenger seat 202 side may display a similar warning.

[0319] <Example of operation of the display unit 231CL and the illumination 207> Next, with reference to FIGS. 38 to 43, an example of the operation of the display section 231CL of the center display 231 and the illumination 207 of the steering wheel 206 will be described.

[0320] For example, the display control unit causes the display unit 231CL to display content according to the situation based on at least one of the situation of the vehicle 1, the situation around the vehicle 1, and the situation of the passenger.

[0321] For example, the light control unit 85 turns on or flashes the illumination 207 in a pattern according to the situation, based on at least one of the situation of the vehicle 1, the situation around the vehicle 1, and the situation of the passengers. The light emission pattern of the illumination 207 is defined by at least one of the color, brightness, flashing pattern, light movement, and light-emitting area, for example.

[0322] For example, when a driver gets into vehicle 1 and sits in driver's seat 201, DMS 30 performs a driver recognition process based on at least one of the image from camera 101L and the image from camera 102. Then, display unit 231CL and illumination 207 notify that the driver has been recognized.

[0323] For example, FIG. 38 shows the state of the display unit 231CL and the illumination 207 before the driver recognizes the state.

[0324] The display unit 231CL displays the state of the vehicle 1. Specifically, the shift position, the charge amount, the speed, the cruising range, and the like are displayed.

[0325] Illumination 207 is turned off.

[0326] FIG. 39 shows the state of the display unit 231CL and the illumination 207 when the driver is recognized.

[0327] The display unit 231CL displays the image and name of the recognized driver and a message to the driver. After that, the display unit 231CL returns to the state shown in FIG.

[0328] Although not shown in detail, the illumination 207 lights up in a predetermined pattern. For example, multiple short white light strips move in a flowing manner over the illumination 207 in a predetermined pattern.

[0329] This ensures that the driver is notified that the vehicle 1 has recognized the driver.

[0330] At this time, for example, the body system control unit 84 may set the position and angle of the driver's seat 201 and the position and angle of the steering wheel 206 based on the recognized driver's physique or the driver's preferences set in advance, etc. In this way, a driving position suitable for the recognized driver is set.

[0331] Furthermore, for example, the display control unit may adjust the display position of the display unit 231CL of the center display 231 based on the recognized eye position of the driver and the position and angle of the steering wheel 206. In this way, the display range of the display unit 231CL is set to a position that is easy to see without being obstructed by the steering wheel 206, according to the recognized driver.

[0332] Furthermore, for example, when the vehicle 1 is ready for autonomous driving, the display unit 231CL and the illumination 207 notify that the preparation for autonomous driving is complete.

[0333] For example, FIG. 40 shows the state of the display unit 231CL and the illumination 207 when preparations for automatic driving are complete.

[0334] 39, an icon 601 indicating that preparation for autonomous driving has been completed is displayed on the display unit 231CL. The speed limit of the road on which the vehicle 1 is traveling is displayed to the left of the icon 601.

[0335] Although not shown in detail, the illumination 207 lights up in a predetermined pattern. For example, multiple short white light strips move in a flowing manner over the illumination 207 in a predetermined pattern.

[0336] This allows the driver to be sure that the vehicle is ready for automated driving.

[0337] Furthermore, for example, when the autonomous driving of the vehicle 1 is started, the display unit 231CL and the illumination 207 notify the start of the autonomous driving.

[0338] FIG. 41 shows the state of the display unit 231CL and the illumination 207 when the automatic driving starts.

[0339] The display unit 231CL displays, for example, an image of the area ahead of the vehicle 1 captured by the camera 51FC1. The display unit 231CL displays, for example, the gear shift position, the charge level, the speed, the cruising range, the speed limit of the road on which the vehicle is traveling, fuel efficiency, etc. The display unit 231CL displays an icon 602 indicating that the vehicle is in autonomous driving to the right of the above-mentioned icon 601. Also, for example, the color of the icon 601 changes as autonomous driving begins.

[0340] Although detailed illustration is omitted, the illumination 207 lights up in a predetermined pattern. For example, multiple short white light bands gradually change to blue while flowing in a predetermined pattern across the illumination 207. Eventually, the entire illumination 207 lights up in blue, and remains in this blue state during autonomous driving.

[0341] This allows the driver to be sure that automatic driving has started. Also, by lighting up illumination 207, passengers other than the driver can be sure that automatic driving has started.

[0342] FIG. 42 shows the state of the display unit 231CL and the illumination 207 when the vehicle 1 changes lanes to the right lane during automatic driving.

[0343] When the vehicle 1C changes lanes, the display unit 231CL displays guidance information 603 including animation for notifying the driver of the execution and direction of the lane change, superimposed on an image of the area ahead of the vehicle 1C.

[0344] Furthermore, the blue light of the entire illumination goes out, and a predetermined blinking range A101 on the right side of the illumination 207 blinks blue. Furthermore, when the steering wheel 206 is rotated clockwise to change lanes, the blinking range A101 rotates in the opposite direction (counterclockwise) in accordance with the rotation angle of the steering wheel 206. Furthermore, when the steering wheel 206 is rotated counterclockwise to return to its original state after changing lanes, the blinking range A101 rotates in the opposite direction (clockwise) in accordance with the rotation angle of the steering wheel 206. As a result, the blinking range A101 does not rotate together with the steering wheel 206, but is maintained in a fixed position when viewed from the outside. Then, after the rotation angle of the steering wheel 206 returns to 0 degrees, the entire illumination 207 lights up in blue, just as it was before the lane change.

[0345] This allows the driver to know in advance that the vehicle 1 will change lanes.

[0346] Similarly, when the vehicle 1 changes lanes to the left lane, the guidance information 603 is displayed superimposed on the display unit 231CL, and a predetermined blinking range on the left side of the illumination 207 blinks in blue.

[0347] Further, for example, the same process is performed when the vehicle 1 changes direction (for example, turns left or right). For example, guidance information including animation for notifying the driver of the change of direction and the direction is superimposed and displayed on the display unit 231CL, and a predetermined blinking range of the illumination 207 corresponding to the direction of the change of direction blinks in blue.

[0348] FIG. 43 shows the state of the display unit 231CL and the illumination 207 when a dangerous situation is detected around the vehicle 1.

[0349] The display unit 231CL displays, for example, a message 611 and an icon 612 indicating the detected state and how to deal with it.

[0350] Furthermore, the blue light of the entire illumination 207 goes out, and a predetermined blinking range A111 above the illumination 207 blinks. For example, the color of light and blinking speed of the blinking range A111 change based on the detected level of danger. For example, if the level of danger is high, the blinking range A111 blinks red, and if the level of danger is low, the blinking range A111 blinks yellow. Furthermore, the higher the level of danger, the faster the blinking speed, and the lower the level of danger, the slower the blinking speed.

[0351] Also, for example, a warning sound is output in accordance with the blinking of the blinking range A111.

[0352] <Example of voice control> Next, an example of voice control of the vehicle 1 will be described with reference to FIGS.

[0353] As described above, vehicle 1 can realize 360-degree real audio. Driver's seat 201, passenger seat 202, rear seat 203L, and rear seat 203R are each equipped with seat speaker 237, seat speaker 238, seat speaker 239L, and seat speaker 239R. Therefore, HMI 31 can freely and individually control the sound image, sound field, volume, and the like for the passengers in each seat.

[0354] DMS 30 can recognize the shape and position of the driver's ears based on the image from camera 101 L. Therefore, the audio control unit can appropriately set the sound image, sound field, volume, etc. for the driver by controlling seat speaker 237 of driver's seat 201 based on the position and shape of the driver's ears.

[0355] Similarly, based on the image from camera 101R, DMS 30 can recognize the shape and position of both ears of the passenger in passenger seat 202. Therefore, the audio control unit can appropriately set the sound image, sound field, volume, etc. for the passenger by controlling seat speaker 238, etc. of passenger seat 202 based on the position and shape of both ears of the passenger.

[0356] Similarly, the DMS 30 can recognize the shape and position of both ears of the passenger in the rear seat 203L based on an image captured by the ToF camera provided on the tablet terminal 235L. Therefore, the audio control unit can appropriately set the sound image, sound field, volume, etc. for the passenger by controlling the seat speaker 239L of the rear seat 203L based on the position and shape of both ears of the passenger.

[0357] Similarly, DMS30 can recognize the shape and position of both ears of the passenger in rear seat 203R based on images from the ToF camera provided on tablet terminal 235R. Therefore, the audio control unit can appropriately set the sound image, sound field, volume, etc. for the passenger by controlling seat speaker 239R of rear seat 203R based on the position and shape of both ears of the passenger.

[0358] In this way, the vehicle 1 can output sound individually to each passenger in each seat, and can also individually control the sound image, sound field, volume, and the like.

[0359] This allows, for example, passengers such as the driver to be notified of the position, movement, etc. of objects around the vehicle 1 by a warning sound.

[0360] 44 to 47 show examples of methods for controlling a warning sound that calls attention to a vehicle 701 approaching from the rear left.

[0361] For example, when the distance between vehicle 701 and vehicle 1 falls within a predetermined threshold, the output of the warning sound begins. Fig. 44 shows the sound field A151 at the start of the output of the warning sound.

[0362] The apex of the conical sound field A151 indicates the position of the virtual sound source. The direction in which the cone spreads from the apex of the sound field A151 indicates the direction in which the sound field spreads.

[0363] The direction of the sound field A151 is set to be substantially the same as the direction in which the head of the driver of the vehicle 1 is viewed from the vehicle 701. In addition, the virtual sound source of the warning sound is set at a position a predetermined distance away from the driver's head in the direction of the vehicle 701.

[0364] 45 to 47, as the vehicle 701 approaches the vehicle 1, the sound field A151 moves. Specifically, the sound field A151 moves so as to always face in a direction substantially equal to the direction in which the head of the driver of the vehicle 1 is viewed from the vehicle 701. The sound field A151 also moves so that the distance between the virtual sound source of the warning sound and the head of the driver is substantially proportional to the distance between the vehicle 701 and the vehicle 1.

[0365] Therefore, the warning sound allows the driver to recognize the approach of the vehicle 701. Furthermore, the movement of the warning sound allows the driver to intuitively recognize the speed, distance, and direction of the vehicle 701.

[0366] <<3. Modifications>> Hereinafter, modifications of the above-described embodiment of the present technology will be described.

[0367] <System Architecture Variations> The configuration of the information processing system 301 in FIGS. 20 and 21 can be changed as appropriate.

[0368] Here, modified examples of the system architecture of the vehicle 1 will be described with reference to Fig. 48 to Fig. 50. Specifically, modified examples of the configuration of the domain controller (domain) of the information processing system 301 will be described. Note that Fig. 48 to Fig. 50 omit illustration of the configuration under the domain controller.

[0369] Fig. 48 shows an example of the configuration of an information processing system 801, which is a first modified example of the information processing system 301. In the figure, parts corresponding to those in Fig. 20 are given the same reference numerals, and their explanation will be omitted as appropriate.

[0370] Compared to the information processing system 301, the information processing system 801 is provided with an HMI DC 811 and a body DC 812 instead of the HMI+body DC 341. That is, the HMI+body DC 341 is divided into the HMI DC 811 and the body DC 812. Therefore, the domain D3 in Fig. 20 is divided into an HMI-related domain and a body-related domain.

[0371] Fig. 49 shows a configuration example of an information processing system 821 which is a second modified example of the information processing system 301. In the figure, parts corresponding to those in the information processing system 801 of Fig. 48 are given the same reference numerals, and descriptions thereof will be omitted as appropriate.

[0372] Compared to the information processing system 801, the information processing system 821 is provided with an ADAS+HMI DC 831 instead of the ADAS DC 331 and HMI DC 811. That is, the ADAS DC 331 and the HMI DC 811 are integrated. Therefore, the ADAS-related domain D2 ( FIG. 20 ) and the HMI-related domain are integrated.

[0373] Fig. 50 shows a configuration example of an information processing system 841 which is a third modified example of the information processing system 301. In the figure, parts corresponding to those in the information processing system 801 of Fig. 48 are given the same reference numerals, and descriptions thereof will be omitted as appropriate.

[0374] Compared to the information processing system 801, the information processing system 841 is provided with an ADAS+HMI+Body DC851 instead of the ADAS DC331, HMI DC811, and Body DC812. That is, the ADAS DC331, HMI DC811, and Body DC812 are integrated. Therefore, the domain D2 and domain D3 in FIG. 20 are integrated.

[0375] 51, the ADAS+HMI+Body DC 851 can be realized by one piece of hardware 871 using virtualization technology. Specifically, functions corresponding to the ADAS DC 313, the HMI DC 811, and the Body DC 812 may be realized by guest VMs (Virtual Machines) 882-1 to 882-3 operating on a hypervisor 881 in the hardware 871, respectively.

[0376] Fig. 52 shows an example of the configuration of an information processing system 901, which is a modified example of the information processing system 301 in Fig. 21. Note that the same reference numerals are used to designate parts corresponding to those in the information processing system 301 in Fig. 21, and descriptions thereof will be omitted where appropriate.

[0377] The information processing system 901 differs from the information processing system 301 in FIG. 21 in that a video splitter 911 is provided.

[0378] The camera 51SL2, the camera 51SR2, and the camera 51BC1 are connected to a video splitter 911.

[0379] The video splitter 911 splits each of the image signals from the camera 51SL2, the camera 51SR2, and the camera 51BC1 into two lines of image signals and supplies them to the ADAS DC 331 and the HMI+body DC 341. Therefore, the camera 51SL2, the camera 51SR2, and the camera 51BC1 are connected to the ADAS DC 331 and the HMI+body DC 341 via the video splitter 911. That is, the camera 51SL2, the camera 51SR2, and the camera 51BC1 belong to both the domain D2 and the domain D3 shown in FIG.

[0380] Although not shown in the figure, the camera 51FC1 is also connected to the ADAS DC331 and the HMI+body DC341 via the video splitter 911.

[0381] For example, the HMI+body DC341 generates a surround view image based on images from the cameras 51FC1, 51SL2, 51SR2, and 51BC1. The HMI+body DC341 displays the surround view image on one or more of the display units 231CL to 231CR of the center display 231.

[0382] Fig. 53 shows an example of the configuration of the video splitter 911. Although Fig. 53 shows only the circuit related to the camera 51SL2, similar circuits are provided for the other cameras 51FC1, 51SR2, and 51BC1.

[0383] The video splitter 911 includes an input terminal 921, a deserializer 922, serializers 923-1 and 923-2, and output terminals 924-1 and 924-2. The deserializer 922 is connected to the input terminal 921 and the serializers 923-1 and 923-2. The serializer 923-1 is connected to the output terminal 924-1. The serializer 923-2 is connected to the output terminal 924-2.

[0384] An image signal from camera 51SL2 is input to deserializer 922 via input terminal 921. Deserializer 922 branches the image signal and supplies it to serializers 923-1 and 923-2. Serializer 923-1 converts the image signal into a serial signal and supplies it to ADAS DC331 via output terminal 924-1. Serializer 923-2 converts the image signal into a serial signal and supplies it to HMI+body DC341 via output terminal 924-2.

[0385] For example, instead of using the video splitter 911, the fish-eye camera 941 shown in FIG. 54 may be used as the camera 51FC1, the camera 51SL2, the camera 51SR2, and the camera 51BC1.

[0386] The fisheye camera 941 includes a lens 951, a serializer 952, and output terminals 953-1 and 953-2. Note that the circuitry for capturing and processing images is not shown.

[0387] The fisheye camera 941 branches an image signal obtained by capturing an image using a serializer 952. The branched image signal is supplied to the ADAS DC 331 via an output terminal 953-1, and is supplied to the HMI+body DC 341 via an output terminal 953-2.

[0388] As described above, it is possible to flexibly change the system architecture of vehicle 1. By appropriately designing the system architecture, it is possible to suppress the complexity of the system including the cameras, for example, and make it simpler.

[0389] <Modifications of the External Recognition Sensor 25 and the In-Vehicle Sensor 26> The configuration example of the external recognition sensor 25 and the in-vehicle sensor 26 in FIG. 2 can be modified as appropriate.

[0390] For example, the number of cameras in the camera module 122L can be changed.

[0391] Specifically, for example, cameras 51SL1 and 51SL2 can be integrated into a single wide-angle, high-resolution camera. For example, cameras 51SL2 and 51SL3 can be integrated into a single wide-angle, high-resolution camera. For example, cameras 51SL3 and 51SL4 can be integrated into a single wide-angle, high-resolution camera. For example, cameras 51SL1 to 51SL3 can be integrated into a single wide-angle, high-resolution camera. For example, cameras 51SL1 to 51SL4 can be integrated into a single high-resolution fisheye camera.

[0392] The same applies to the camera module 122R.

[0393] For example, it is possible to combine the camera 51BC1 and the radar 52BC.

[0394] For example, it is possible to combine the radar 52BC and the LiDAR 53B.

[0395] For example, it is possible to omit one of the cameras 51SL2 and 51SR2, which are configured as fisheye cameras.

[0396] <Other variations> For example, one of the images of the digital outer mirror displayed on the display unit 231LL and the display unit 231RR of the center display 231 may be displayed on the central portion 231C.

[0397] In the above description, an example has been shown in which the vehicle 1 is a left-hand drive vehicle, but the present technology can of course also be applied to a right-hand drive vehicle. When the present technology is applied to a right-hand drive vehicle, the above-described exterior and interior layouts of the vehicle are appropriately changed to suit the right-hand drive vehicle.

[0398] The present technology can also be applied to, for example, a transportation device that performs automatic driving without the need for a passenger to drive. In this case, the above-mentioned distinction between the driver's seat 201 and the passenger's seat 202 disappears, and the above-mentioned various displays are arranged in front of the seats in the transportation device. Also, for example, the display range of an image displayed on the display is changed based on the line of sight, posture, etc. of a person sitting in a specific seat.

[0399] Furthermore, the type of vehicle to which the present technology can be applied is not particularly limited. The present technology can also be applied to mobile devices other than vehicles, such as personal mobility, transport robots, airplanes, ships, construction machinery, and agricultural machinery. Examples of mobile devices to which the present technology can be applied include mobile devices that capture images of their surroundings without a person on board, such as drones and robots.

[0400] <<4. Other>> <Example of computer configuration> The above-described series of processes can be executed by hardware or software. When the series of processes are executed by software, a program constituting the software is installed in a computer (for example, the processor 21, etc.).

[0401] The program executed by the computer may be a program that processes in chronological order according to the order described in this specification, or may be a program that processes in parallel or at the required timing, such as when called.

[0402] In this specification, a system refers to a collection of multiple components (devices, modules (components), etc.), regardless of whether all the components are contained in the same housing. Therefore, multiple devices housed in separate housings and connected via a network, and a single device housed in a single housing with multiple modules, are both systems.

[0403] Furthermore, the embodiments of the present technology are not limited to the above-described embodiments, and various modifications are possible within the scope of the gist of the present technology.

[0404] For example, this technology can be configured as cloud computing, in which a single function is shared and processed collaboratively by multiple devices via a network.

[0405] Furthermore, each step described in the above flowchart can be executed by one device, or can be shared and executed by multiple devices.

[0406] Furthermore, when one step includes multiple processes, the multiple processes included in that one step can be executed by one device or can be shared and executed by multiple devices.

[0407] <Configuration combination example> The present technology can also be configured as follows.

[0408] (1) The first camera; A second camera; a housing that houses the first camera and the second camera; Equipped with When the camera is installed on a mobile device, the optical axis of the first camera faces diagonally rearward of the mobile device, and the optical axis of the second camera faces laterally or diagonally forward of the mobile device. Camera module. (2) further comprising at least one of a third camera housed in the housing and a fourth camera housed in the housing; When installed on the mobile device, the optical axis of the third camera faces diagonally rearward of the mobile device and faces in a direction closer to the lateral direction of the mobile device than the optical axis of the first camera, and the optical axis of the fourth camera faces diagonally forward of the mobile device and faces in a direction closer to the front direction of the mobile device than the optical axis of the second camera. The camera module according to (1) above. (3) the first camera is used in a CMS (Camera Monitoring System), the second camera is used for surround view; the third camera is used for an ADAS (Advanced Driver Assistance System), The fourth camera is used for ADAS. The camera module according to (2) above. (4) The second camera has a wider angle than the first camera, the third camera, and the fourth camera. The camera module according to (3) above. (5) The second camera is a fisheye camera. The camera module according to (4) above. (6) When the second camera is installed on the moving device, the optical axis of the second camera faces diagonally downward. The camera module according to (4) or (5). (7) The second camera is also used for ADAS. The camera module according to any one of (3) to (6). (8) The third camera has a higher resolution than the fourth camera. The camera module according to any one of (3) to (7). (9) The second camera has a wider angle than the first camera. The camera module according to (1) above. (10) When the second camera is installed on the moving device, the optical axis of the second camera faces diagonally downward. The camera module according to (9) above. (11) the first camera is used in at least one of a CMS and an ADAS; The second camera is used for at least one of a surround view and an ADAS. The camera module according to (1) above. (12) The housing is made of metal The camera module according to any one of (1) to (11). (13) The camera module is installed near the front edge of the front door of the mobile device. The camera module according to any one of (1) to (11). (14) a first camera whose optical axis faces diagonally rearward of the mobile device; a second camera housed in the same housing as the first camera and having an optical axis facing in a direction closer to the front of the mobile device than the optical axis of the first camera; a first display used in the CMS and configured to display an image based on the first image captured by the first camera; a recognition unit that recognizes an object outside the mobile device based on a second image captured by the second camera; a display control unit that controls display, on the first display, of the first image and a visual effect based on a result of object recognition by the recognition unit; and An information processing system comprising: (15) a first domain controller that controls a first domain including the first camera and the display control unit; a second domain controller that controls a second domain including the second camera and the recognition unit; The information processing system according to (14) above, further comprising: (16) The first domain further includes the second camera. The information processing system according to (15) above, (17) A second display extending in the left-right direction at the front of the room of the mobile device is provided. Further preparation, The display control unit further controls display of an image based on the second image on the second display. The information processing system according to (14) or (15) above. (18) The display control unit controls the second display to display a surround view using the second image. The information processing system according to (17) above. (19) The first display and the second display are integrated. The information processing system according to (17) or (18). (20) The optical axis of the second camera faces laterally or diagonally forward of the mobile device. The information processing system according to any one of (14) to (19). (twenty one) further comprising at least one of a third camera housed in the housing and a fourth camera housed in the housing; The optical axis of the third camera faces diagonally rearward of the mobile device and faces in a direction closer to the lateral direction of the mobile device than the optical axis of the first camera, and the optical axis of the fourth camera faces diagonally forward of the mobile device and faces in a direction closer to the front direction of the mobile device than the optical axis of the second camera. The information processing system according to (20) above. (twenty two) The recognition unit further recognizes an object outside the mobile device based on at least one of a third image taken by the third camera and a fourth image taken by the fourth camera. The information processing system according to (21) above. (twenty three) a first domain controller that controls a first domain including the first camera and the display control unit; a second domain controller that controls a second domain including the second camera, the recognition unit, and at least one of the third camera and the fourth camera; The information processing system according to (21) or (22) further comprises: (twenty four) The optical axis of the second camera is directed diagonally rearward of the mobile device and is directed in a direction closer to the lateral direction of the mobile device than the optical axis of the first camera. The information processing system according to any one of (14) to (19). (twenty five) controlling display on a display for the CMS of an image based on a first image captured by a first camera whose optical axis faces diagonally rearward of the mobile device; performing object recognition outside the mobile device based on a second image captured by a second camera housed in the same housing as the first camera and having an optical axis facing in a direction closer to a front direction of the mobile device than the optical axis of the first camera; Controlling the display of a visual effect based on the result of the object recognition by the display. Information processing methods. (26) A display control unit that controls the display of the display used in the CMS Preparation, The display control unit controls the display of a visual effect based on a result of object recognition outside the mobile device, based on an image based on a first image taken by a first camera whose optical axis faces diagonally rearward of the mobile device, and a second image taken by a second camera housed in the same housing as the first camera and whose optical axis faces in a direction closer to the front of the mobile device than the optical axis of the first camera. Information processing device.

[0409] The effects described in this specification are merely examples and are not limiting, and other effects may also be present. [Explanation of symbols]

[0410] 1 vehicle, 11 vehicle control system, 21 processor, 25 external recognition sensor, 26 in-vehicle sensor, 30 DMS, 31 HMI, 51, 51FC1 to 51BC3 camera, 52, 52FC to 52BR radar, 53, 53F to 53B LiDAR, 73 recognition unit, 84 body system control unit, 85 light control unit, 101L, 101R, 102 camera, 122L, 122R camera module, 131L, 131R housing, 231 center display, 231C center unit, 231CL to 231CR display unit, 231L left end, 231LL display unit, 231R right end, 231RR display unit, 301 information processing system, 311 secure gateway, 321 ePWT DC, 331 ADAS DC, 341 HMI+Body DC, 349 CMS, 361 ToF ECU, 362 CMS ECU, 801 Information Processing System, 811 HMI DC, 812 Body DC, 821 Information Processing System, 831 ADAS+HMI DC, 841 Information Processing System, 851 ADAS+HMI+Body DC

Claims

1. A first camera used in a CMS (Camera Monitoring System), a second camera used for surround view; a third camera used for an Advanced Driver Assistance System (ADAS); a fourth camera used for ADAS; a housing that houses the first camera, the second camera, the third camera, and the fourth camera; Equipped with When installed on a mobile device, the optical axis of the first camera faces diagonally rearward of the mobile device, the optical axis of the second camera faces laterally or diagonally forward of the mobile device, the optical axis of the third camera faces diagonally rearward of the mobile device and faces in a direction closer to the lateral direction of the mobile device than the optical axis of the first camera, and the optical axis of the fourth camera faces diagonally forward of the mobile device and faces in a direction closer to the front direction of the mobile device than the optical axis of the second camera. Camera module.

2. The second camera has a wider angle than the first camera, the third camera, and the fourth camera. The camera module of claim 1 .

3. The second camera is a fisheye camera. The camera module according to claim 2 .

4. When the second camera is installed on the moving device, the optical axis of the second camera faces diagonally downward. The camera module according to claim 2 .

5. The second camera is also used for ADAS. The camera module of claim 1 .

6. The third camera has a higher resolution than the fourth camera. The camera module of claim 1 .

7. The second camera has a wider angle than the first camera. The camera module of claim 1 .

8. When the second camera is installed on the moving device, the optical axis of the second camera faces diagonally downward. The camera module according to claim 7 .

9. The housing is made of metal The camera module of claim 1 .

10. The camera module is installed near the front edge of the front door of the mobile device. The camera module of claim 1 .

11. a first camera whose optical axis faces diagonally rearward of the mobile device; a second camera housed in the same housing as the first camera and having an optical axis facing in a direction closer to the front of the mobile device than the optical axis of the first camera; a third camera housed in the housing, the optical axis of which faces diagonally rearward of the mobile device and faces in a direction closer to the lateral direction of the mobile device than the optical axis of the first camera; a fourth camera housed in the housing, the optical axis of which faces diagonally forward of the mobile device and faces in a direction closer to the front of the mobile device than the optical axis of the second camera; a first display used in the CMS and configured to display an image based on the first image captured by the first camera; a recognition unit that recognizes an object outside the mobile device based on a second image taken by the second camera, a third image taken by the third camera, and a fourth image taken by the fourth camera; a display control unit that controls display, on the first display, of the first image and a visual effect based on a result of object recognition by the recognition unit; and An information processing system comprising:

12. a first domain controller that controls a first domain including the first camera and the display control unit; a second domain controller that controls a second domain including the second camera and the recognition unit; The information processing system according to claim 11 , further comprising:

13. The first domain further includes the second camera. The information processing system according to claim 12 , comprising:

14. a second display extending in the left-right direction at the front of the room of the mobile device; Further preparation, The display control unit further controls display of an image based on the second image on the second display. The information processing system according to claim 11.

15. The display control unit controls the second display to display a surround view using the second image. The information processing system according to claim 14.

16. The first display and the second display are integrated. The information processing system according to claim 14.

17. The optical axis of the second camera faces laterally or diagonally forward of the mobile device. The information processing system according to claim 11.

18. a first domain controller that controls a first domain including the first camera and the display control unit; a second domain controller that controls a second domain including the second camera, the recognition unit, the third camera, and the fourth camera; The information processing system according to claim 11 , further comprising:

19. The optical axis of the second camera is directed obliquely rearward of the mobile device and is directed in a direction closer to the lateral direction of the mobile device than the optical axis of the first camera. The information processing system according to claim 11.

20. Controlling display of an image based on a first image captured by a first camera whose optical axis faces diagonally rearward of the mobile device on a display for the CMS; performing object recognition outside the mobile device based on a second image taken by a second camera stored in the same housing as the first camera and having an optical axis facing in a direction closer to the front of the mobile device than the optical axis of the first camera; a third image taken by a third camera stored in the same housing and having an optical axis facing diagonally rearward of the mobile device and facing in a direction closer to the side of the mobile device than the optical axis of the first camera; and a fourth image taken by a fourth camera stored in the same housing and having an optical axis facing diagonally forward of the mobile device and facing in a direction closer to the front of the mobile device than the optical axis of the second camera; Controlling the display of a visual effect based on the result of the object recognition by the display. Information processing methods.

Images