Vehicle display control device, vehicle display control system, vehicle display control method and vehicle display control program

Abstract

To provide a vehicle display control device, a vehicle display control system, a vehicle display control method and a vehicle display control program capable of appropriately displaying a camera image and another display to a passenger in a scene where the camera image is shown.SOLUTION: A control section 11 of a display system ECU 5x enables a camera image to be displayed by coordinating display areas of a plurality of displays 2a to 2c that make up a PtoP display 2. The control unit 11 controls the display of the camera image by setting whether to display the camera image on at least two of the display areas 2a to 2c in coordination, or on a single central information display 2b, depending on the autonomous driving level and the situation inside or outside the vehicle.SELECTED DRAWING: Figure 3A

Description

[Technical Field]

[0001] The present disclosure relates to a vehicle display control device, a vehicle display control system, a vehicle display control method, and a vehicle display control program. [Background technology]

[0002] A conventional driving assistance device is disclosed in Patent Document 1. According to the driving assistance device disclosed in Patent Document 1, when a road surface recognition unit determines that the road surface of an intersection has been recognized, an image switching unit switches the image displayed on a display device that displays driving guidance for a vehicle to an image of the intersection captured by a camera. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2020-112927 Summary of the Invention [Problem to be solved by the invention]

[0004] As described in the Background Technology section, in a scene where the display image on the display device is switched to camera footage depending on the situation around the vehicle, if there is content other than camera footage that can be displayed on the display device, it is necessary to change the display mode of the camera footage to suit the situation, and there is room for further consideration in this regard.

[0005] The present disclosure has been made in consideration of the above circumstances, and its purpose is to provide a vehicle display control device, a vehicle display control system, a vehicle display control method, and a vehicle display control program that are capable of appropriately displaying camera images and other displays to occupants in scenes where camera images are displayed. [Means for solving the problem]

[0006] According to the invention of claim 1, the vehicle display control device is capable of displaying camera images by coordinating the display areas of multiple display devices. The display control unit controls the display of the camera images by setting whether the display areas of the camera images are displayed on at least two or more of the multiple display devices in a coordinated manner or on a single display device, depending on the autonomous driving level and the conditions inside or outside the vehicle. When controlling the display of the camera images on the display devices, the display control unit can change whether the camera images are displayed on multiple displays in a coordinated manner or on a single display, depending on the conditions. This allows the camera images and other displays to be displayed appropriately to occupants in various scenes where camera images are displayed. [Brief explanation of the drawings]

[0007] [Figure 1] External configuration diagram of the cockpit system in the first embodiment [Figure 2] FIG. 1 shows a configuration and control image of a PtoP display in the first embodiment. [Figure 3A] FIG. 1 is a block diagram illustrating a vehicle display control system according to a first embodiment. [Figure 3B] An explanatory diagram of the hardware and software structure in the first embodiment. [Figure 4] Flowchart 1 showing the outline of the processing contents in the first embodiment [Figure 5] Schematic diagram 1 showing a notification mode in the first embodiment [Figure 6] Flowchart 2 outlining the processing content in the first embodiment [Figure 7] Flowchart 3 showing an outline of the processing contents in the first embodiment [Figure 8] Flowchart 4 outlining the processing content in the first embodiment [Figure 9] Schematic diagram 2 showing the notification mode in the first embodiment [Figure 10] Flowchart 5 outlining the processing content in the first embodiment [Figure 11]Flowchart 6 showing the outline of the processing contents in the first embodiment [Figure 12] Flowchart No. 7 showing the outline of the processing contents in the first embodiment [Figure 13] Schematic diagram 3 showing the notification mode in the first embodiment [Figure 14] Flowchart No. 8 showing the outline of the processing contents in the first embodiment [Figure 15] Flowchart No. 9 showing the outline of the processing contents in the first embodiment [Figure 16] Flowchart No. 10 outlining the processing content in the first embodiment [Figure 17] Flowchart No. 11 showing an outline of the processing contents in the first embodiment [Figure 18] Schematic diagram 4 showing the notification mode in the first embodiment [Figure 19] Flowchart No. 12 outlining the processing content in the first embodiment [Figure 20] FIG. 10 is an explanatory diagram of a display target in a display area of ​​a display device according to a first embodiment; [Figure 21] External configuration diagram of a cockpit system in a second embodiment [Figure 22] FIG. 10 is an explanatory diagram of a display target of a display device according to a second embodiment; [Figure 23] External configuration diagram of a cockpit system in a third embodiment [Figure 24] FIG. 10 is an explanatory diagram of a display target of a display device according to a third embodiment; [Figure 25] External configuration diagram of a cockpit system according to a fourth embodiment [Figure 26] FIG. 10 is an explanatory diagram of a display target of a display device according to a fourth embodiment; [Figure 27] An explanatory diagram of the hardware and software structure in the fifth embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0008] Hereinafter, several embodiments of a vehicle display control device, a vehicle display control system, a vehicle display control method, and a vehicle display control program will be described. Parts that perform the same functions in each embodiment will be assigned the same reference numerals, and descriptions thereof may be omitted.

[0009] (First embodiment) The first embodiment will be described with reference to Figures 1 to 20. As shown in Figures 1 and 2, a vehicle display control system 1 is configured using a cockpit system 4 equipped with a pillar-to-pillar display 2. Hereinafter, the pillar-to-pillar display 2 will be referred to as a PtoP display 2.

[0010] As shown in Figures 1 and 2, the PtoP display 2 as a display device is configured with multiple displays 2a-2c arranged side by side to form a landscape orientation. Each of the displays 2a-2c of the PtoP display 2 is a large display configured as a liquid crystal display, an organic EL display, or the like, and is mounted on the dashboard between the left pillar PL and the right pillar PR of the vehicle. The display area of ​​the PtoP display 2 includes at least the display area of ​​the driver's seat front (hereinafter referred to as "D seat front") display 2a mounted in the area in front of the driver's seat, the display area of ​​the passenger seat front (hereinafter referred to as "P seat front") display 2c mounted in the area in front of the other passenger seats, and the display area of ​​a center information display (CID) 2b mounted in the intermediate area.

[0011] In the example shown in FIGS. 1 and 2, the display 2a for displaying vehicle information refers to a display device installed in front of the D seat, and refers to a display device for mainly displaying vehicle information SP such as meter information. On the other hand, a P seat front display 2c is installed in front of the passenger seat (hereinafter referred to as "in front of the P seat") as a display device. The P seat front display 2c is a display device that mainly displays information necessary for the P seat occupant, such as entertainment information, and information for the P seat occupant to enjoy entertainment in the vehicle. The center information display 2b refers to a display device installed between the D seat and the P seat, and refers to a display device that mainly displays a menu screen for displaying a selection screen for an application to be started, an execution screen for the application, and information necessary for the driver and the P seat occupant. However, although modified examples will be described in the embodiments described later, the number, installation conditions, and configuration of the display devices are merely examples, and are not limited to the display configuration shown in this embodiment.

[0012] The PtoP display 2 is configured to be able to display various image content and text content in full graphic display on each of the displays 2a to 2c, such as meter images, images captured by the surrounding camera 23 (see Figure 3A), entertainment information such as still images and video entertainment images, and navigation information using navigation images including map images of the area around the current location.

[0013] As shown in FIGS. 2 and 3A, a large number of ECUs 5 are configured in the vehicle and connected to an in-vehicle network 25. The ECUs 5 are classified into a display system ECU 5x, an in-vehicle control system ECU 5w, a periphery monitoring system ECU 5y, a driving control system ECU 5z, and a DCM 5v (or DCU) that communicates with the outside of the vehicle. DCM stands for Data Communication Module, and DCU stands for Data Communication Unit. The DCM 5v is a module for wireless communication with an external center 50 provided outside the vehicle. Here, the ECUs 5 are classified as described above, but this classification method is an example. Some ECUs 5 may be classified across multiple categories.

[0014] The interior control system ECU 5w may be a well-known ECU 5 for door lock control, an ECU 5 for reclining control, an ECU 5 for air conditioning control, an ECU 5 for horn and lamp control, etc., and controls interior control actuators to perform various controls other than those related to driving control. For example, when the interior control system ECU 5w receives an instruction to open / close the door locks of the D seat, P seat, right rear seat, or left rear seat, it individually controls the door locks. Furthermore, when the interior control system ECU 5w receives a step-by-step instruction to recline the D seat, P seat, right rear seat, or left rear seat, it controls the reclining state of the specified seat. Furthermore, when the interior control system ECU 5w receives an instruction to operate the horn, it turns the horn on / off, and when it receives an instruction to operate various lamps, it turns the headlights or brake lights on / off and turns the turn indicators and hazard lights on / off.

[0015] The driving control system ECU 5z may be a well-known vehicle control ECU, engine control ECU, motor control ECU, brake control ECU, steering control ECU, etc., but may also be configured as an integrated ECU that integrates all or some of these. The driving control system ECU 5z may also include an autonomous driving ECU. The autonomous driving ECU is an Autonomous Driving Electric Control Unit. The driving control system ECU 5z performs driving assistance control using well-known driving actuators and various sensors, such as an electromagnetic throttle, brake actuator, EPS motor, accelerator position sensor, brake pedal force sensor, steering torque sensor, and vehicle speed sensor.

[0016] When the cruise control system ECU 5z inputs an automatic control signal to the autonomous driving ECU, it activates a driving actuator (not shown) to perform the corresponding predetermined level of driving assistance or autonomous driving. For example, Level I driving assistance can perform automatic braking to avoid collisions with obstacles, following a preceding vehicle, or lane-staying control to prevent the vehicle from drifting out of the lanes on either side. Level II autonomous driving can perform a combination of Level I driving assistance or an autonomous driving mode under specific conditions, such as automatically overtaking slower vehicles on a highway or automatically merging onto a highway. Note that Level II autonomous driving requires the driver to monitor the autonomous driving. At Level III or higher autonomous driving, the system performs all driving tasks while monitoring the vehicle.

[0017] Each ECU 5 is mainly composed of an SoC 30 or 31 (see below) incorporating a microcomputer equipped with various storage units 6 such as a processor, cache memory, RAM, and ROM, an I / O 15, and a bus connecting these. The storage unit 6 represents a non-transitory tangible storage medium that non-temporarily stores computer-readable programs and data. The non-transitory tangible storage medium is realized by a semiconductor memory or the like. Each ECU 5 is communicably connected to other ECUs 5 provided in the vehicle via a communication control unit 7 and an in-vehicle network 25.

[0018] In this embodiment, as shown in Fig. 2, a plurality of ECUs 5 constituting a display-system ECU 5x constitute an HCU as an information processing device 10. HCU stands for Human Machine Interface Control Unit. The display-system ECU 5x shares the processing power of its internal physical resources and controls display on each of the displays 2a to 2c of the PtoP display 2; for example, each ECU 5 constituting the display-system ECU 5x individually processes display on each of the displays 2a to 2c of the PtoP display 2. Although the in-vehicle network 25 is shown connected between the ECUs 5 constituting the display-system ECU 5x, they may also be connected by a dedicated line.

[0019] 3A, the information processing device 10 includes a control unit 11, a calculation unit 12, a storage unit 6, a display processing unit 13, a sound processing unit 14, an I / O 15 that manages input or output from various devices, a communication control unit 7 that manages communication with other ECUs 5, and a wireless communication unit 27. Here, a form will be described in which output signals from major components such as a position detector 19, an operation input unit 21, an occupant monitor 22, a peripheral camera 23, and a distance detection sensor 24 shown in FIG. 2 are input to the information processing device 10 via the I / O 15, but signals may also be input from other ECUs 5 that configure an in-vehicle control system ECU 5w, a peripheral monitoring system ECU 5y, and a driving control system ECU 5z via an in-vehicle network 25.

[0020] Based on the control of the control unit 11, the calculation device 12 calculates the area to be displayed on the display screen of the PtoP display 2 for the content of images, sentences, characters, or symbols (hereinafter referred to as images, etc.) stored in the memory unit 6, calculates in which area of ​​the display screen of the PtoP display 2 the content of images, etc. will be displayed, and on which area it will be superimposed, and outputs the content of images, etc. to the control unit 11.

[0021] The control unit 11 has a function of controlling the display of content such as images on the display screen of the PtoP display 2 through the display processing unit 13. The display processing unit 13 processes the display of content such as images on the display screen of the PtoP display 2 based on the control of the control unit 11. Content such as images can be displayed superimposed on each display layer on each display screen of the PtoP display 2.

[0022] The control unit 11 also has a function as a sound notification control unit that controls the notification of sound from the speaker 18 via the sound processing unit 14. The sound processing unit 14 receives a call voice input from the microphone 17 and outputs a call voice from the speaker 18 based on the control of the control unit 11. When the sound processing unit 14 receives text or character content from the control unit 11, it converts the content into voice and reads it out through the speaker 18. The information processing device 10 also performs voice recognition on the voice input through the microphone 17.

[0023] The position detector 19 detects the position with high accuracy using a well-known GNSS receiver such as a GPS, and inertial sensors such as an acceleration sensor and a gyro sensor (not shown). The position detector 19 outputs a position detection signal to the control unit 11 via the I / O 15. The position identification unit 11a of the control unit 11 functions as an ADAS locator that sequentially locates the current position of the vehicle with high accuracy based on map information input from a map data input device and the position detection signal of the position detector 19. ADAS is an abbreviation for Advanced Driver Assistance Systems.

[0024] The map data input device is configured with a non-volatile memory or the like, and stores map data including various data such as link data and node data. The link data includes various data such as link identification data that identifies the links that make up roads on the map, link length data that indicates the length of the link, link orientation data that indicates the orientation of the link, link travel time data that indicates the time required to travel along the link, node coordinate data that indicates the coordinates of the nodes that make up the start and end points of the link, and link attribute data that indicates the attributes of the road. The link attribute data includes road symbols using icons that include prefecture names, city names, and information on national and prefectural roads, and character symbol information that indicates town names.

[0025] The node data includes various types of data such as node identification data that identifies nodes present on the map, node coordinate data that indicates the coordinates of the nodes, node name data that indicates the name of the node, node type data that indicates the type of node such as an intersection, connecting link data that identifies the links connecting to the nodes, etc. The map data also has associated therewith character symbol information that indicates the names of facilities such as parks using identification information such as character codes.

[0026] The display processing unit 13 can generate image content of a map of any location based on these map data, and can also generate the image content so as to include some or all of the character and symbol information described above.

[0027] The vehicle position calculated by the navigation process is expressed in a coordinate system consisting of latitude and longitude, where, for example, the X-axis represents longitude and the Y-axis represents latitude. Note that the vehicle position can be measured based on information on a travel distance obtained based on a sensing result from a vehicle speed sensor mounted on the vehicle, and various other configurations can be adopted as long as the vehicle position can be identified.

[0028] The operation input unit 21 is composed of a touch panel configured on a predetermined display out of the multiple displays 2a to 2c that make up the PtoP display 2, such as the center information display 2b or the P seat front display 2c, or mechanical switches provided on the periphery of the PtoP display 2. When an operation input is made to the operation input unit 21, it is accepted via the I / O 15 and the operation input information is output to the control unit 11. The operation input unit 21 includes, for example, a power switch, a horn switch, a turn switch, a hazard lamp switch, an automatic control switch, and the like.

[0029] A vehicle occupant can issue an operation command to the control unit 11 by operating the operation input unit 21. The control unit 11 executes control based on the operation signal from the operation input unit 21. The operation input unit 21 also enables selection of content to be displayed on the PtoP display 2, operation of the air conditioning, operation of the audio, and input of navigation functions. For example, when a navigation app is running and the vehicle occupant inputs a destination into the operation input unit 21, the control unit 11 can execute navigation processing to provide guidance from the current position of the vehicle to the destination while displaying a map screen using the display processing unit 13.

[0030] When a vehicle occupant operates the operation input unit 21 while the vehicle is stopped, the occupant can instruct to move the display area of ​​the map or change the scale. When the display processing unit 13 receives an instruction through the control unit 11, it can move the display area within the image content and change the scale of the map by enlarging or reducing the display area. This allows the map to be displayed in any area, such as around the current position of the vehicle or around the destination, and the scale of the map display can also be changed.

[0031] The occupant monitor 22 functions as an occupant state detection unit that detects the state of an occupant in the vehicle. The occupant monitor 22 particularly functions as a driver state detection unit that detects the state of the driver and various operation states. The occupant monitor 22 is configured to monitor the occupant using signals output by the occupant operating, for example, a power switch, a horn switch, a turn switch, a hazard lamp switch, an automatic control switch, etc., and an occupant state monitor, and outputs various signals and monitor results to the control unit 11. The occupant monitor 22 may also monitor the occupant using signals from a steering sensor that detects whether the driver is gripping or steering the steering wheel, a seating sensor that detects whether the driver is seated in the seat, an accelerator pedal or brake pedal depression sensor, etc.

[0032] The power switch outputs a signal corresponding to the operation when turned on by an occupant in the vehicle cabin to start the internal combustion engine or the electric motor. The occupant status monitor includes a camera that detects the status of the occupant sitting in the D seat or P seat by capturing an image of the occupant using an image sensor and outputs an image signal. The driver's occupant status monitor is called DSM, which stands for Driver Status Monitor.

[0033] The occupant status monitor acquires an image signal captured by irradiating the driver's head with near-infrared light, analyzes the image as necessary, and outputs the image to the control unit 11. The occupant status monitor is used to detect the status of occupants such as the driver, particularly during driving assistance or autonomous driving. The turn switch is provided to activate the vehicle's turn indicators and outputs a turn signal to turn right or left in response to the occupant's operation of the turn switch. The horn switch is provided to sound the horn, or alarm horn, in response to the occupant's command. The hazard lamp switch is provided to flash or turn off the hazard lamps in response to the occupant's command.

[0034] The automatic control switch is turned on by an occupant in the vehicle cabin to command automatic control of the vehicle's driving state, and outputs an automatic control signal corresponding to the operation. The control unit 11 can determine the behavior of the vehicle occupant, for example, the direction in which the eyes are facing, based on the signal from the occupant monitor 22, and can also input the operation state of the power switch, the operation state of the turn indicators and hazard lamps, command information for automatic control of the vehicle, etc.

[0035] The control unit 11 can detect the state of the driver, the passenger in the front passenger seat, or the passengers sitting in the rear seats based on the signals monitored by the passenger monitor 22. This allows the control unit 11 to realize the functions of a driver state determination unit that determines the state of the driver, and an occupant state determination unit that determines the state of each occupant, such as the passenger in the front passenger seat.

[0036] The peripheral cameras 23 constitute peripheral monitoring sensors such as a front camera that captures an image in front of the vehicle, a back camera that captures an image behind the vehicle, a corner camera that captures an image of the front or rear of the vehicle, or a side camera that captures an image of the side of the vehicle, and an electronic mirror, which are output to the control unit 11 via the I / O 15 as image signals of a front guide monitor, a back guide monitor, a corner view monitor, a side guide monitor, and an electronic mirror, respectively, and stored in the storage unit 6. The communication control unit 7 is connected to an in-vehicle network 25 such as CAN or LIN, and controls data communication with another ECU 5. The vehicle may also be equipped with an inter-vehicle communication system that allows the vehicle to communicate data with another vehicle S, or a roadside communication system that allows the vehicle to communicate data with a roadside device.

[0037] The vehicle is also equipped with a distance detection sensor 24 as a perimeter monitoring sensor that detects the distance to an obstacle. The distance detection sensor 24 is composed of a clearance sonar, LiDAR, millimeter-wave radar, etc., and can detect vehicles, people, animals, fallen objects on the road, guardrails, curbs, trees, etc. that are in front of, on the front side of, the rear side of, or near the rear or side of the vehicle. It can also detect the direction to and the distance to the obstacle. The perimeter monitoring sensor can also detect road markings on the roads around the vehicle, such as lane markings, stop lines, crosswalks, signs such as "Stop" written on the road, and stop lines displayed at the boundaries of intersections. The control unit 11 can obtain the status of vehicles in the vicinity of the vehicle itself, such as departure information when the preceding vehicle has departed, approach information when a surrounding vehicle such as a following vehicle approaches the vehicle itself, and information on the distance between the vehicle itself and surrounding vehicles such as the preceding vehicle and the following vehicle, by using image information from the surrounding camera 23, communication information from the vehicle-to-vehicle communication system and the road-to-vehicle communication system, and sensor information from the distance detection sensor 24.

[0038] Furthermore, a wireless communication unit 27 is connected to the control unit 11 of the display system ECU 5x. The wireless communication unit 27 is configured to be wirelessly connected to a mobile terminal 28 carried by a vehicle occupant using short-range wireless communication technology such as wireless LAN or Bluetooth (registered trademark). The mobile terminal 28 is a so-called smartphone, tablet terminal, or mobile phone, and is equipped with a display device that displays various information and a function for inputting information. The mobile terminal 28 is also equipped with a telephone function for making calls to the outside, and is capable of making calls to and from outside the vehicle, as well as data communication.

[0039] The wireless communication unit 27 is communicatively linked with a portable terminal 28 carried by a vehicle occupant by short-range wireless communication technology. The display system ECU 5x can communicate with the external center 50 via wireless communication and a wired communication network (not shown) from the portable terminal 28 carried by the vehicle occupant via the wireless communication unit 27. The display system ECU 5x can communicate various information with the external center 50 by wireless communication via the portable terminal 28 linked by short-range wireless communication technology.

[0040] The driver or passenger may also wear a wearable device 29 on their body. The wearable device 29 may be what is called a smart watch or smart glasses, and may be worn on the body in various forms, such as a wristwatch, eyeglasses, ring, earphones, clothing, or shoes. The wearable device 29 is operated by a microcomputer and has a short-range wireless communication function, and when communication is established with the mobile terminal 28 or the wireless communication unit 27, data can be transmitted between the wearable device 29 and the mobile terminal 28. Furthermore, the wearable device 29 has an audio input / output function, and when the driver inputs audio through the wearable device 29, communication with the mobile terminal 28 can be performed to make an external call. When audio is input from an external device, the wearable device 29 can output the audio to the driver.

[0041] Furthermore, the wearable device 29 is equipped with various sensors, and when worn on the body of a driver, passenger, or other occupant, it can measure the vital information of each occupant. Examples of vital information that can be measured here include heart rate (pulse), body temperature, blood concentration, and blood pressure. Other information that can be measured include the number of steps taken, exercise records, calories burned, and sleep time. Because the wearable device 29 is always worn on the body, vital information can be constantly obtained even if the driver experiences some kind of abnormality while driving or takes a break from driving during autonomous driving. Furthermore, even if a passenger falls asleep during manual driving or if the driver falls asleep during autonomous driving, the wearable device 29 can detect this sleep information.

[0042] 3B shows an example of the hardware and software configuration of the information processing device 10. Each of the ECUs 5, 5a constituting the information processing device 10 is equipped with an SoC 30, 31, respectively, and the above-mentioned microcomputer is incorporated into the equipped SoC 30, 31. The microcomputer incorporated into the SoC 30, 31 of the ECU 5 is configured to run a variety of multiple applications 33 (hereinafter abbreviated as apps 33) on a pre-installed general-purpose OS 32, for example, Linux OS (Linux is a registered trademark). SoC is an abbreviation for System On Chip.

[0043] The applications 33 include an image processing application 34, a navigation application, and other applications. In response to a drawing request from the image processing application 34, the SoC 30 performs drawing processing on the display screen of each display 2a of the PtoP display 2.

[0044] On the other hand, ECU 5a is provided for meter drawing purposes and is therefore designated by the reference numeral 5a. A real-time OS 35 capable of processing with higher real-time performance than a general-purpose OS 32 is embedded in the microcomputer built into SoC 31 of ECU 5a, and a meter application 36 is configured to run on the real-time OS 35. Note that although various functions are realized by hardware executing software, the following explanation may focus on the software of application 33, such as image processing application 34 and meter application 36, for ease of understanding.

[0045] The meter application 36 is software for notifying occupants, particularly the driver, of the vehicle speed, engine revolutions, warnings, etc., and draws meter image content in a specific display area of ​​the PtoP display 2. For example, the meter application 36 draws image content such as a speedometer, a tachometer, the shift range position status, or warning lights. The speedometer includes a speed image whose display needs to be updated in real time to show changes in vehicle speed. Similarly, the tachometer is also included in the meter image because its display needs to be updated in real time to show changes in engine revolutions.

[0046] The content drawn by the meter application 36 requires relatively higher real-time performance than the content drawn by other applications. The applications 33 include a navigation application. The navigation application realizes the navigation function described above and also draws image content such as a navigation screen including a map and the current vehicle position on the PtoP display 2.

[0047] The applications 33 also include an image synthesis application. The image synthesis application specifies the size and type of various image contents to be displayed on the PtoP display 2, synthesizes parts of the image contents within one frame, and outputs this synthesized mixed image to the PtoP display 2. The image synthesis application realizes the functions of an image synthesis unit, also known as a compositor, and also the functions of an image output unit.

[0048] A display layer for drawing the image content is assigned to each of the applications 33 and 36 that draw image content. These display layers are allocated in the storage unit 6 in a size that allows the necessary image content to be drawn.

[0049] Furthermore, the image content displayed on the PtoP display 2 can be animated. Here, animation refers to a display mode in which the position or size of the image showing the content gradually changes, the image rotates, the entire user interface moves in response to a swipe operation, the image gradually fades in or out, or the color of the image changes. The control unit 11 executes the application 33 based on the navigation application and performs various processes to realize the function of the position identification unit 11a. The control unit 11 executes the application 33 and performs various processes to realize the functions of the autonomous driving level determination unit 11c, the use determination unit 11d, the vicinity determination unit 11e, and the tire display determination unit 11f according to the present application.

[0050] The autonomous driving level determination unit 11c has a function of determining whether the autonomous driving level is autonomous driving or manual driving. The usage determination unit 11d has a function of determining whether the P seat front display 2c is being used. The P seat front display 2c is equivalent to a display device for the passenger seat. The surroundings determination unit 11e has a function of determining whether a pedestrian T, another vehicle S, or a foreign object U is present in the surroundings. The tire display determination unit 11f has a function of determining whether a tire Ta is to be displayed.

[0051] This embodiment is characterized by the content of display control when camera images are displayed on the multiple displays 2a to 2c. Specific processing will be described below with reference to FIG.

[0052] Control unit 11 controls the display of camera images by setting whether to display camera images from peripheral camera 23 in at least two or more of the display areas of multiple displays 2a to 2c in a coordinated manner, or to display them in the display area of ​​a single display 2a, 2b, or 2c, according to various input information. When coordinated display is set, control unit 11 controls the display of camera images in a coordinated manner across the display areas of multiple displays 2a to 2c.

[0053] A specific example is shown in Figure 4. While the host vehicle is traveling or stopped, the peripheral camera 23 captures camera footage in S11. When the camera footage is captured, the information processing device 10 stores the camera footage in the memory unit 6. The control unit 11 performs image analysis of the camera footage stored in the memory unit 6 to determine whether an intersection with poor visibility is shown in S12. The control unit 11 may also determine whether an intersection with poor visibility is shown by taking into account current location information acquired by executing a navigation app in the image analysis results of the camera footage.

[0054] For example, if the camera image does not show an intersection with poor visibility, the control unit 11 exits the subroutine shown in Fig. 4, but if it determines in S12 that an intersection with poor visibility is shown, it determines in S13 whether the P seat front display 2c is being used. For example, the control unit 11 determines whether entertainment information or navigation information is already being displayed on the P seat front display 2c.

[0055] The control unit 11 sets whether to output the coordinated display to multiple displays including the P seat front display 2c (for example, the entire PtoP display 2) or to display on a single display (for example, the center information display 2b) depending on whether the P seat front display 2c is being used.

[0056] For example, if the control unit 11 is already controlling the display of other content on the P seat front display 2c, it determines YES in S13, and controls the display of the camera image on the single center information display 2b in S14, while leaving the content already displayed on the P seat front display 2c as it is.

[0057] Conversely, if the P seat front display 2c is not being used, the control unit 11 determines NO in S13 and controls the coordinated display to be displayed on the entire surface of multiple displays including the P seat front display 2c, for example, the PtoP display 2. For example, the control unit 11 sets the camera image to be output in wide coordinated format on the entire surface of the PtoP display 2. As a result, the display processing unit 13 performs coordinated display processing in wide format on the entire surfaces of the D seat front display 2a, center information display 2b, and P seat front display 2c that make up the PtoP display 2.

[0058] <When your vehicle approaches an intersection with poor visibility> An example of the display is shown in Figure 5. For example, when approaching an intersection with poor visibility such as the T-junction shown in Figure 5, other vehicles S may be stopped or moving slowly on the roads passing on either side of the T-junction ahead, or pedestrians T may be walking.

[0059] In this case, the display processing unit 13 synthesizes and outputs the camera images from the peripheral cameras 23 for wide coordinated display under the control of the control unit 11. In the example shown in Fig. 5, the control unit 11 issues a command to synthesize camera images from the front camera and the corner cameras capturing the front part of the vehicle, among the camera images from the peripheral cameras 23 sequentially stored in the storage unit 6, and the display processing unit 13 outputs to each of the displays 2a to 2c. By controlling the control unit 11 to display the wide coordinated images on the entire surface of each of the displays 2a to 2c constituting the PtoP display 2, the driver and passengers can get a bird's-eye view of the entire road with poor visibility, making it easier to notice other vehicles S and pedestrians T.

[0060] In the cases shown in Figures 4 and 5, the control unit 11 controls the display of the camera images by setting whether to display the camera images from the peripheral camera 23 on at least two or more displays in a coordinated manner or on a single display, depending on the information about the situation outside the vehicle. This makes it possible to set whether to display the images on each display in a coordinated manner or on a single display depending on the situation outside the vehicle, allowing occupants to view information that is most important to them. The control unit 11 also controls the display of the camera images by setting the coordinated display on displays 2a to 2c depending on the determination result of whether the P seat front display 2c is being used. This makes it possible to set whether to display the images on each display in a coordinated manner or on a single display depending on the usage status of the P seat front display 2c.

[0061] Furthermore, as illustrated in Figure 5, the display areas of the electronic mirror display windows 2x and 2y may be arranged to be superimposed on the D seat front display 2a and the P seat front display 2c, respectively, and the control unit 11 may control the display of the camera image captured by the corner camera of the peripheral camera 23 on the inside of the display windows 2x and 2y.

[0062] Another example of processing operation is shown in Fig. 6. First, while the host vehicle is traveling or stopped, the peripheral camera 23 captures camera video in S11 of Fig. 6. When the camera video is captured, the information processing device 10 stores the camera video in the memory unit 6. The control unit 11 performs image analysis of the camera video stored in the memory unit 6 to determine whether an intersection with poor visibility is displayed in S12. The control unit 11 may determine whether an intersection with poor visibility is displayed by adding current location information acquired by executing a navigation application to the image analysis results of the camera video.

[0063] For example, if the camera image does not show an intersection with poor visibility, the control unit 11 exits the subroutine shown in Figure 6, but if it determines in S12 that an intersection with poor visibility is shown, it determines in S23 whether the vehicle is in automatic driving mode.

[0064] For example, if the autonomous driving level is level zero, I, or II, the control unit 11 determines in S23 that the driving is manual, including driving assistance, and in S24 controls the display of the camera image in wide format by cooperatively outputting the camera image to the entire surface of the PtoP display 2. Conversely, if the control unit 11 determines that the autonomous driving level is level III or higher, the control unit 11 controls the display of the camera image in the display area of ​​the center information display 2b in the middle in S25.

[0065] In the case shown in Fig. 6, the control unit 11 controls the display of the camera images by setting whether to display the images on the displays 2a to 2c in a coordinated manner or on a single display (for example, 2b) depending on the determination result of whether the vehicle is in autonomous driving. This makes it possible to set whether to display the images on each display in a coordinated manner or on a single display depending on whether the vehicle is in autonomous driving. Whether to display the camera images in a coordinated manner or on a single display was determined depending on whether the autonomous driving level is level II or lower or level III or higher, but this is not limiting and the determination may also be made depending on whether the autonomous driving level is level I or lower or level II or higher.

[0066] Another example of processing operation is shown in Fig. 7. First, while the host vehicle is traveling or stopped, the peripheral camera 23 captures camera video in S11 of Fig. 7. When the camera video is captured, the information processing device 10 stores the camera video in the memory unit 6. The control unit 11 performs image analysis of the camera video stored in the memory unit 6 to determine whether an intersection with poor visibility is displayed in S12. The control unit 11 may determine whether an intersection with poor visibility is displayed by adding current location information acquired by executing a navigation application to the image analysis results of the camera video.

[0067] For example, if the camera image does not show an intersection with poor visibility, the control unit 11 exits the subroutine shown in Figure 7, but if it determines in S12 that the camera image shows an intersection with poor visibility, it determines in S33 whether a pedestrian T or another vehicle S is present.

[0068] For example, if the control unit 11 determines through image analysis that a pedestrian T or another vehicle S is present, the control unit 11 displays and outputs the camera image in a manner that calls attention to the pedestrian T or the other vehicle S by enclosing them in a frame as shown in Fig. 5 in S34. Conversely, the control unit 11 may control the display of the camera image on a part of the center information display 2b in the middle in S35.

[0069] 7, the control unit 11 controls the display of the camera images by setting whether to display the images in a coordinated manner on the displays 2a to 2c or on a single center information display 2b, depending on the result of the determination by the periphery determination unit 11e as to whether a pedestrian T or another vehicle S is present. This makes it possible to set whether to display the images in a coordinated manner on each display or on a single display, depending on the situation around the vehicle.

[0070] <When driving slowly on a narrow road> 8 shows modified examples of the display mode according to the location. First, while the host vehicle is traveling or stopped, the peripheral camera 23 captures camera video in S11 of FIG. 8. When the camera video is captured, the information processing device 10 stores the camera video in the storage unit 6. The control unit 11 performs image analysis of the camera video stored in the storage unit 6 to determine whether the vehicle is traveling on a narrow road in S42. The control unit 11 may determine whether the vehicle is traveling on a narrow road by taking into account current location information acquired by executing a navigation application in addition to the image analysis results of the camera video.

[0071] For example, if the control unit 11 determines that the vehicle is traveling on a narrow road, it exits the subroutine shown in Figure 8, but if it determines in S42 that the vehicle is traveling on a narrow road, it determines in S43 whether the P seat front display 2c is being used.

[0072] For example, if the control unit 11 is already controlling the display of other content on the P seat front display 2c, it determines YES in S43, and controls the display of the camera image on the single center information display 2b in S44, while leaving the content already displayed on the P seat front display 2c as it is.

[0073] Conversely, for example, if the P seat front display 2c is not being used, the control unit 11 determines NO in S43 and controls the coordinated display on multiple displays including the P seat front display 2c, for example, the entire surface of the PtoP display 2. That is, the control unit 11 controls the wide coordinated display of the camera video on the entire surface of the PtoP display 2. As a result, the display processing unit 13 processes the coordinated display in wide format on the entire surfaces of the D seat front display 2a, center information display 2b, and P seat front display 2c that make up the PtoP display 2. In this case, the same effects as those of the processing operation shown in FIG. 4 can be obtained.

[0074] When the vehicle is moving slowly on a narrow road, the control unit 11 may control the display of one of the following: camera images of the front taken by the front camera, camera images under the hood of the vehicle, camera images of the side of the vehicle taken by the side camera, or a bird's-eye view created by combining multiple camera images to provide a bird's-eye view of the vehicle from above, either in conjunction with one another on multiple displays, for example, the entire surface of the PtoP display 2, or on a single display, for example, the center information display 2b.

[0075] For example, as shown in Fig. 9, while the vehicle is moving slowly on a narrow road, the control unit 11 may perform image analysis of the camera image and discover a foreign object U in the camera image showing the area under the hood. This foreign object U may be, for example, a small child playing by hiding under the vehicle, a small animal hiding under the vehicle, a step that marks the boundary between the roadway and the sidewalk, or a parking stop block. As in the example of Fig. 9, the control unit 11 displays the camera image under the hood on the entire surface of the PtoP display 2 via the display processing unit 13, making it easier for the driver and passengers to notice the foreign object U.

[0076] 9, the control unit 11 controls the display of the camera images by setting whether to display the images on the entire PtoP display 2 in a coordinated manner or on a single display, such as the center information display 2b, depending on the result of the determination by the periphery determination unit 11e as to whether or not a foreign object U is present. This makes it possible to set whether to display the images on each display in a coordinated manner or on a single display depending on the situation around the vehicle.

[0077] Furthermore, even when the control unit 11 controls the display of the camera image on the D-seat front display 2a through the display processing unit 13, it is required by law to control the display of the vehicle information SP, including the vehicle speed. When the control unit 11 displays the camera image on the entire surface of the PtoP display 2, it is preferable to superimpose the vehicle information SP on the camera image. In this case, it is preferable that the display processing unit 13 superimposes information different from the camera image (here, the vehicle information SP) on an upper layer of the display area displaying the camera image, and further controls the display by making the other information transparent so that the camera image can also be seen.

[0078] Furthermore, when the control unit 11 controls the display of other information from the upper layer of the camera image, it is desirable to display the other information in a display area of ​​the camera image where an image of low importance is displayed. The display area of ​​the image of low importance here is preferably an area of ​​the entire camera image where an object of high importance is not detected by image recognition. Examples of objects of high importance include the foreign object U, other vehicle S, pedestrian T, tire Ta, etc., as described above.

[0079] <Display mode change according to tire Ta display setting> Furthermore, when the control unit 11 controls the display of the left and right tires Ta from camera images via the display processing unit 13, the driver can get a sense of the vehicle width by checking the left and right tires Ta and comparing the vehicle width with the road width. For this reason, it is desirable for the control unit 11 to unconditionally display the left and right tires Ta and the camera images of their surroundings on the entire surface of the PtoP display 2. Even if there are fences or gutters on the side of a narrow road, it becomes easier to drive safely by avoiding contact with the fence or running off into the gutters.

[0080] The control unit 11 may allow a user to select and input, by operating the operation input unit 21, whether to display the camera image from the front camera showing the tire Ta or not to display the tire Ta in conjunction with the image. When a selection is made to display the tire Ta, the control unit 11 controls the display of the camera image as is. When a selection is made not to display the tire Ta, the control unit 11 may control the display by cutting out the image of the tire Ta from the camera image or by using an image erasing technique to partially erase the tire Ta and blend it into the background. Alternatively, the control unit 11 may control the display by using an image captured by a peripheral camera 23 that is installed so as not to capture the tire Ta.

[0081] Drivers who are inexperienced in driving can avoid contact with obstacles and drive safely by driving while displaying tire Ta, as mentioned above. Conversely, drivers who are experienced in driving can drive based on their own intuition by hiding tire Ta.

[0082] An example of processing is shown in Fig. 10. While the host vehicle is traveling or stopped, in S11 of Fig. 10, the peripheral camera 23 captures camera images and stores the camera images in the storage unit 6. The control unit 11 performs image analysis of the camera images stored in the storage unit 6, thereby determining in S42 whether the vehicle is traveling on a narrow road. The control unit 11 may determine whether the vehicle is traveling on a narrow road by adding current location information acquired by executing a navigation application to the image analysis results of the camera images.

[0083] For example, if the control unit 11 determines that the vehicle is traveling on a narrow road, it exits the subroutine shown in FIG. 10, but if it determines in S42 that the vehicle is traveling on a narrow road, it determines in S46 whether the tire Ta is set to be superimposed and displayed.

[0084] If the superimposed display setting for the tire Ta is enabled, the control unit 11 determines YES in S46, and in S47 displays the tire Ta in a wide, coordinated manner across the entire surface of the PtoP display 2. Conversely, if the superimposed display setting for the tire Ta is not enabled, in S48 the control unit 11 controls the display of the camera image without displaying the tire Ta on a single display, for example, the center information display 2b.

[0085] As a result, the control unit 11 can control the display of the camera images by setting whether to display the images on the multiple displays 2a to 2c in a coordinated manner or on a single display, such as the center information display 2b, depending on the determination result of the tire display determination unit 11f. This allows the display mode to be changed depending on the determination result of whether to display the tire Ta.

[0086] When the linked display is performed on the entire surface of the PtoP display 2, the control unit 11 may control the display of the camera image showing the rear side on the P seat front display 2c using the wipe display windows 2x, 2y, etc. When the linked display is performed on the entire surface of the PtoP display 2, navigation information such as a map may be wipe-displayed on the P seat front display 2c.

[0087] <Display mode changes according to the autonomous driving level> In addition, the control unit 11 may control the display of the camera images by setting whether to display them in conjunction with the displays 2a to 2c or on a single display, such as the center information display 2b, depending on the result of the autonomous driving level determination by the autonomous driving level determination unit 11c.

[0088] An example of processing is shown in Fig. 11. While the host vehicle is traveling or stopped, in S11 of Fig. 11, the peripheral camera 23 captures camera images and stores the camera images in the storage unit 6. The control unit 11 performs image analysis of the camera images stored in the storage unit 6, thereby determining in S42 whether the vehicle is traveling on a narrow road. The control unit 11 may determine whether the vehicle is traveling on a narrow road by adding current location information acquired by executing a navigation application to the image analysis results of the camera images.

[0089] If the control unit 11 determines that the vehicle is traveling on a narrow road, it exits the subroutine shown in FIG. 11. However, if it determines in S42 that the vehicle is traveling on a narrow road, it determines in S49 whether the vehicle is traveling autonomously. The "autonomous driving" referred to here does not only refer to an autonomous driving level such as Level III, where the driver does not need to check the road ahead. It also includes cases where the vehicle autonomously drives or steers autonomously, even though the driver is responsible for driving, such as Level II. From the perspective of driving technology, "autonomous driving" includes cases where the driving control system ECU 5z automatically controls only the drive function of driving the wheels or only the steering function in the autonomous driving function.

[0090] When considering the autonomous driving level, if the level is Level 0 or I, the control unit 11 determines that the vehicle is being driven manually, including driving assistance, in S49, and displays a wide, linked image across the entire PtoP display 2 in S50. If the driver operates either the accelerator or the steering wheel, or both, the vehicle is being driven manually, and a wide, linked image is displayed. At this time, the content displayed on the P-seat front display 2c, such as navigation information, may be wiped or erased. Conversely, if the autonomous driving level is Level II or higher, the control unit 11 controls the camera image to be displayed on a single display, such as the center information display 2b, without a linked image in S48. This is because, in the case of autonomous driving at Level II or higher, the driver does not need to operate the accelerator or steering wheel, and therefore there is less need for a wide, linked image.

[0091] Specifically, considering the autonomous driving level, for example, when traveling on a narrow road at level 0 or I, the driver is required to perform at least some driving operations, so it is advisable to control the displays 2a to 2c to display images in a coordinated manner to make it easier to recognize the actual road width. If the autonomous driving level is level II or higher, the cruise control system ECU 5z executes control for autonomous driving, so the control unit 11 controls the camera images around the vehicle to be displayed on a single center information display 2b rather than on the displays 2a to 2b in a coordinated manner. In this case, it is also possible to control the display of the tire Ta captured by the peripheral camera 23 or auxiliary lines to a parking space. While it is determined whether the autonomous driving level is level I or lower or level II or higher whether to display the camera images in a coordinated manner or on a single display, this is not limiting, and it is also possible to determine whether the autonomous driving level is level 0 or level I or higher, or whether the autonomous driving level is level II or lower or level III or higher.

[0092] <Change in display mode when backing into a parking space> In addition, the control unit 11 may control the display of the camera images by setting whether to display the images in coordination on each of the displays 2a to 2c or on a single display, such as the center information display 2b, depending on whether the P seat front display 2c is being used when backing into a parking space.

[0093] An example of processing is shown in Fig. 12. While the host vehicle is traveling or stopped, in S11 of Fig. 12, the peripheral camera 23 captures camera images and stores the camera images in the storage unit 6. In S42a, the control unit 11 detects whether the vehicle is being reversed into parking by using a shift position sensor or the like.

[0094] When the control unit 11 determines that the vehicle is backing into a parking space, it determines in S43 whether the P seat front display 2c is being used. If it is being used, it outputs the camera image to the single center information display 2b while leaving the image on the P seat front display 2c. Conversely, when the P seat front display 2c is not being used, the control unit 11 controls the display by outputting the camera image in wide coordinated format across the entire PtoP display 2. This makes it easier for the driver to view and understand the camera image from the backup camera while backing into a parking space. Figure 13 shows an example of the wide coordinated display while backing into a parking space. Because the rear of the vehicle is displayed across the entire PtoP display 2, the driver can easily grasp the distance to another vehicle S parked adjacently, making parking easier.

[0095] Another processing example is shown in Fig. 14. While the host vehicle is traveling or stopped, in S11 of Fig. 13, the peripheral camera 23 captures camera images and stores the camera images in the storage unit 6. In S42a, the control unit 11 detects whether the vehicle is being reversed into parking by using a shift position sensor or the like.

[0096] When the control unit 11 determines that the vehicle is being reversed into a parking space, it determines in S51 whether a foreign object U or another vehicle S is present. If it determines that neither a foreign object U nor another vehicle S is present, it outputs the camera image to, for example, a single center information display 2b without any linked display in S48. Conversely, if a foreign object U or another vehicle S is present, it controls the display by linking and outputting the camera image in wide format across the entire surface of the PtoP display 2 in S50. This makes it easier for the driver to view and understand the camera image from the backup camera while backing into a parking space. As shown in FIG. 13, because the rear of the vehicle is displayed across the entire surface of the PtoP display 2, the driver can easily grasp the distance to the other vehicle S parked adjacently, making parking easier.

[0097] Another processing example is shown in Fig. 15. While the host vehicle is traveling or stopped, in S11 of Fig. 15, the peripheral camera 23 captures camera images and stores the camera images in the storage unit 6. In S42a, the control unit 11 detects whether the host vehicle is being reversed into parking by using a shift position sensor or the like.

[0098] If it is determined that the vehicle is being reversed into parking, the control unit 11 determines in S52 whether the setting for superimposed display of the tire Ta is enabled. If the setting for superimposed display of the tire Ta is enabled, the control unit 11 controls the display of an image in which the camera image is output in wide coordinated format on the entire surface of the PtoP display 2 and the tire Ta is superimposed. Conversely, if the setting for superimposed display of the tire Ta is not enabled, in S48 the camera image is output without coordinated display, for example, on a single center information display 2b without superimposed display of the tire Ta. This makes it possible to change the display mode of the displays 2a to 2c depending on whether or not the tire Ta is to be displayed.

[0099] Another processing example is shown in Fig. 16. While the host vehicle is traveling or stopped, in S11 of Fig. 16, the peripheral camera 23 captures camera images and stores the camera images in the storage unit 6. In S42a, the control unit 11 detects whether the vehicle is being reversed into parking by using a shift position sensor or the like.

[0100] If the control unit 11 determines that the vehicle is backing into a parking space, it determines in S53 whether driving assistance for automatic parking is being performed. If driving assistance for automatic parking is being performed, the control unit 11 controls the display of the image by outputting the camera image in wide coordinated format on the entire surface of the PtoP display 2 in S50. Conversely, if driving assistance for automatic parking is not being performed, the control unit 11 outputs the camera image to, for example, a single center information display 2b without coordinated display in S48. This makes it possible to change the display mode of the displays 2a to 2c depending on whether driving assistance for automatic parking is being performed.

[0101] <Change in display mode when a foreign object U is detected on the travel route> Another processing example is shown in Fig. 17. While the host vehicle is traveling or stopped, in S11 of Fig. 17, the peripheral camera 23 captures camera images and stores the camera images in the memory unit 6. The control unit 11 performs image analysis of the camera images stored in the memory unit 6, and determines in S42b whether or not a foreign object U can be detected on the traveling route.

[0102] When the control unit 11 detects a foreign object U, it determines in S43 whether the P seat front display 2c is in use. If the P seat front display 2c is not in use, the control unit 11 controls the display of the image by outputting the camera image in wide coordinated format on the entire surface of the PtoP display 2 in S45. Conversely, if the P seat front display 2c is in use, the control unit 11 outputs the camera image to, for example, a single center information display 2b without coordinated display in S44. This makes it possible to change the display mode of the displays 2a to 2c depending on whether or not a foreign object U is present on the driving route. As shown in FIG. 18, when the control unit 11 detects a foreign object U under the hood in the image captured by the peripheral camera 23, it is preferable to control the display of the camera image capturing the area under the hood on the entire surface of the PtoP display 2, provided that the P seat front display 2c is not in use.

[0103] Another processing example is shown in Fig. 19. While the host vehicle is traveling or stopped, in S11 of Fig. 19, the peripheral camera 23 captures camera images and stores the camera images in the memory unit 6. The control unit 11 performs image analysis of the camera images stored in the memory unit 6, and determines in S42b whether or not a foreign object U can be detected on the traveling route.

[0104] When the control unit 11 detects a foreign object U, it determines in S46 whether the setting for superimposed display of the tire Ta is enabled. If the setting for superimposed display of the tire Ta is enabled, the control unit 11 controls the display of an image in which the camera image is linked to the entire PtoP display 2 in wide format and the tire Ta is superimposed on the entire PtoP display 2 in S47. Conversely, if the setting for superimposed display of the tire Ta is not enabled, the control unit 11 controls the display of the camera image without linked display, that is, the camera image is output without superimposed display of the tire Ta on, for example, a single center information display 2b in S48. This makes it possible to change the display mode of the displays 2a to 2c depending on whether or not the tire Ta is to be displayed. If the setting for linked display is enabled on the entire PtoP display 2, the control unit 11 may control the display of the camera image showing the rear side on the P seat front display 2c by using a wipe or the like.

[0105] The control unit 11 may allow a user to select and input, by operating the operation input unit 21, whether to display a camera image from a front camera that captures the tire Ta or not to display the tire Ta and not to display the tire Ta in conjunction with the image. When a selection is made to display the tire Ta, the control unit 11 controls the display of the camera image as is. When a selection is made not to display the tire Ta, the control unit 11 may control the display by cutting out an image of the tire Ta from the camera image, or by using a partial image erasing technique to partially erase the tire Ta and blend it into the background. Alternatively, the control unit 11 may control the display by using an image captured by a peripheral camera 23 that is installed so as not to capture the tire Ta. This allows for a display that is comfortable and easy for the occupant to understand.

[0106] Although specific scenes have been given as examples above, the present invention is not limited to the above-described scenes. For example, even if the control unit 11 detects an object such as a foreign object U, a pedestrian T, or another vehicle S in another driving scene, the control unit 11 may control the display so that the detected object U, pedestrian T, or another vehicle S is included in the camera image. In other words, when caution is required or when there is a risk, it is preferable to display the camera image including the detected object. When it is determined that there is a particular risk, it is desirable to control the display to cover the entire surface of the PtoP display 2.

[0107] <Display changes according to the situation inside the vehicle> The control unit 11 may control the display of the camera image by setting whether to display the display area of ​​the camera image on at least two or more of the display areas of the multiple displays 2a-2c in coordination with one another, or on a single display, such as the center information display 2b, depending on the situation inside the vehicle. For example, the control unit 11 may determine whether the passenger in the P seat is watching the P seat front display 2c from the monitor signal of the occupant monitor 22, and, depending on the result of this determination, control the display of the camera image by selecting whether to display the camera image on the multiple displays 2a-2c or on a single display (for example, 2b). When the control unit 11 determines that the passenger in the P seat is watching the P seat front display 2c, it may control the display of the camera image by displaying the camera image on the D seat front display 2a and the center information display 2b in coordination with one another, excluding the P seat front display 2c, or by displaying the camera image on the single center information display 2b. Conversely, when the control unit 11 determines that the passenger in the P seat is not viewing the P seat front display 2c, it may control the P seat front display 2c and the center information display 2b to display a wide camera image in cooperation with each other. It may also be controlled to display the entire PtoP display 2 in cooperation with each other.

[0108] In addition, for example, while the vehicle is driving at autonomous driving level III or higher, the driver may watch some entertainment content, check destination information or email, or even hold a remote conference. When the driver uses private content in this way, the control unit 11 may control the display of only the D-seat front display 2a. When the control unit 11 detects that the vehicle is experiencing the above-described conditions based on the operation input information from the operation input unit 21, the sensor signal from the passenger monitor 22, or the content display, it may control the display of various content only on the D-seat front display 2a in front of the driver. In the case of manual driving or manual driving, a wide variety of driving-related information is displayed, so there is a high need to display camera footage in a coordinated manner across multiple displays (e.g., 2a to 2c). Conversely, in the case of autonomous driving or automatic driving, the amount of driving-related information to be displayed is reduced, so the priority of displaying camera footage across multiple displays (e.g., 2a to 2c) is lowered. For this reason, if some content is being controlled to be displayed on some displays (for example, the display 2c in front of the P seats) and is already in use, it is advisable to avoid full-screen display on the PtoP display 2 by controlling the display to be displayed on other displays (for example, the display 2a in front of the D seats and the center information display 2b).

[0109] <Description of the display to be displayed> In the above description, when a single display is the display control target, the example has been given in which the center information display 2b is the main display target, as shown by the hatching in the upper part of Fig. 20. Alternatively, when the control unit 11 controls the display of a single display as the display target, the display 2c in front of the P seat or the display 2a in front of the D seat may be the display target. Furthermore, when the control unit 11 controls the display of multiple displays in a linked manner, it is desirable to set the entire surface of the PtoP display 2 as the display target, as shown by the hatching in the lower part of Fig. 20.

[0110] Additionally, when displaying information in a coordinated manner on multiple displays, only the center information display 2b and the P seat front display 2c may be the objects of the coordinated display. In other words, the D seat front display 2a does not have to be the object of the coordinated display. Also, only the D seat front display 2a and the center information display 2b may be the objects of the coordinated display. In other words, the P seat front display 2c does not have to be the object of the coordinated display. Also, only the D seat front display 2a and the P seat front display 2c may be the objects of the coordinated display. In other words, the center information display 2b does not have to be the object of the coordinated display.

[0111] <Summary of this embodiment> As described above, according to this embodiment, the control unit 11 of the display system ECU 5x is able to display camera images in cooperation with the display areas of the multiple displays 2a to 2c that make up the PtoP display 2. In this case, the control unit 11 controls the display of the camera images by setting whether to display the camera images in cooperation with at least two or more of the display areas of the displays 2a to 2c, or to display the camera images on a single center information display 2b, depending on the autonomous driving level and the conditions inside or outside the vehicle.

[0112] When controlling the display of camera images on the displays 2a to 2c, the control unit 11 can change whether to display the camera images on some or all of the multiple displays 2a to 2c in cooperation with one another or on a single display depending on the situation. This allows the camera images and other displays to be displayed appropriately to the occupants in various scenes where camera images are displayed.

[0113] In particular, when displaying multiple displays 2a to 2c in a coordinated manner, the control unit 11 displays a single camera image in a coordinated manner on the PtoP display 2, which is configured by continuously connecting the D seat front display 2a, which serves as the meter display area, the center information display 2b, and the P seat front display 2c, making it easier for occupants to check the various contents displayed in a coordinated manner.

[0114] (Second embodiment) The second embodiment will be described with reference to Figures 21 and 22. The configuration of the in-vehicle display is not limited to the configuration of the PtoP display 2 described in the first embodiment. For example, as shown in Figure 21, a meter display 2m in front of the D seat may be configured independently, and a display 2g may be configured in which the center information display 2b and the P seat front display 2c are integrated.

[0115] In the display configuration shown in Fig. 21, when the control unit 11 controls the display of a single display, the center information display 2b may be the display target, as shown by the hatching in the upper part of Fig. 22. The display 2c in front of the passenger seat may also be the display target. When the control unit 11 controls the display of multiple displays in a coordinated manner, the entire surface of the display 2g may be the display target in a coordinated manner, as shown by the hatching in the lower part of Fig. 22. This embodiment also achieves the same effects as the above-described embodiment.

[0116] (Third embodiment) The third embodiment will be described with reference to Figures 23 and 24. The configuration of the in-vehicle display may be, for example, a display 2h shown in Figure 23. As shown in Figure 23, the display 2h is configured by integrating a center information display 2b with a D-seat front display 2a.

[0117] In the display configuration shown in Fig. 23, when the control unit 11 controls display of a single display, it is preferable to set the display 2a in front of the D seat as the display target, as shown by the hatching in the upper part of Fig. 24. It is also possible to set the center information display 2b as the display target. Furthermore, when the control unit 11 controls linked display on multiple displays, it is preferable to set the entire surface of the display 2h as the linked display target, as shown by the hatching in the lower part of Fig. 24. This embodiment also achieves the same effects as the above-described embodiment.

[0118] (Fourth embodiment) A fourth embodiment will be described with reference to Figs. 25 and 26. As shown in Fig. 25, the same application as in the above-described embodiments can be applied to a case where the PtoP display 2 is composed of five displays 2a to 2e. As shown in Fig. 25, the PtoP display 2 is provided with a display 2a for displaying vehicle information and an electronic mirror display 2d in front of the D seat. As explained in the above-described embodiments, the D seat front display 2a is a display device having a meter display area for displaying vehicle information SP. The electronic mirror display 2d is a display device that generally shows an image of the left rear of the vehicle.

[0119] The PtoP display 2 also includes a P-seat front display 2c and an electronic mirror display 2e in front of the P seat. The P-seat front display 2c is typically used to display entertainment information. The electronic mirror display 2e is used to display an image of the right rear of the vehicle.

[0120] In the display configuration shown in Fig. 25, when the control unit 11 controls the display of a single display, the center information display 2b may be the display target, as shown by hatching in the upper part of Fig. 26. Alternatively, any of the other displays 2a, 2b, 2d, and 2e may be the display target.

[0121] When electronic mirror displays 2d and 2e are provided, the control unit 11 may display camera images on the center information display 2b in conjunction with the electronic mirror displays 2d and 2e, as shown by hatching in the lower part of FIG. 26. The camera images may be displayed only on the electronic mirror displays 2d and 2e in conjunction with each other. However, if the camera images are displayed on the other displays 2a, 2b, and 2c in conjunction with each other, it is not necessary to display the camera images on the electronic mirror displays 2d and 2e in conjunction with each other. This embodiment also achieves the same effects as the above-described embodiment.

[0122] (Fifth embodiment) A fifth embodiment will be described with reference to FIG. 27. In the above-described embodiment, a display-system ECU 5x is configured using multiple ECUs 5. However, the display-system ECU 5x may be configured as an HCU using a single ECU 5. An example of the hardware and software configuration in this case is shown in FIG. 32. Each ECU 5 is equipped with an SoC 230, and a microcomputer is embedded in the SoC 230. The microcomputer embedded in the SoC 230 of the ECU 5 has a general-purpose OS 32 and a real-time OS 35 running on a hypervisor 231. A variety of multiple applications 33 are configured to run on the general-purpose OS 32. The real-time OS 35 has higher real-time processing performance than the general-purpose OS 32. A meter application 36 is configured to run on the real-time OS 35. Even when such a configuration is adopted, the same configuration, functions, and effects as those of the above-described embodiment can be achieved.

[0123] (Other embodiments) The present invention is not limited to the above-described embodiment, and the following modifications or extensions are possible. In addition to the contents set forth in the claims, the present disclosure also includes the following contents. [1] A display control device (5x, 10) that enables a camera image to be displayed by linking the display areas of a plurality of display devices (2, 2a to 2e), A display control device for a vehicle, comprising: a display control unit (11) that controls the display of the camera image by setting whether the display area of ​​the camera image is displayed on at least two or more of the plurality of display devices in conjunction with each other or on a single display device, depending on the autonomous driving level and the situation inside or outside the vehicle.

[0124] [2] a use determination unit (11d) that determines whether the passenger seat display is being used; The display control unit controls the display of the camera image by setting whether to display the image on the linked display or on the single display depending on the determination result of the use determination unit.

[0125] [3] an automatic driving level determination unit (11c) that determines whether the automatic driving level is automatic driving or manual driving; The display control unit controls the display of the camera image by setting whether to display the image on the linked display or on the single display depending on the judgment result of the autonomous driving level judgment unit [1] or [2].

[0126] [4] a surroundings determination unit (11e) that determines whether or not a pedestrian, another vehicle, or a foreign object is present in the surroundings; The display control unit controls the display of the camera image by setting whether to display the image on the linked display or on the single display depending on the judgment result of the periphery judgment unit.

[0127] [5] a tire display determination unit (11f) that determines whether or not to display tires; The display control unit controls the display of the camera image by setting whether to display the image on the linked display or on the single display depending on the determination result of the tire display determination unit.

[0128] [6] A plurality of display devices (2a to 2e); A display control system for a vehicle, comprising: a display control unit (11) that controls the display of the camera image by setting whether the display area of ​​the camera image is displayed in coordination with the display areas of the plurality of display devices or on a single display device depending on the autonomous driving level and the situation inside or outside the vehicle; and a display control device (5x, 10) that controls the display of the camera image by coordinating the display areas of the plurality of display devices.

[0129] [7] When the display control unit (11) is configured to be able to control the display of the camera image by setting whether to display the display area of ​​the camera image on a plurality of display devices (2a to 2e) in cooperation with each other or on a single display device depending on the autonomous driving level, the situation inside or outside the vehicle, The display control unit controls the display of the camera images in cooperation with the plurality of display devices when the cooperative display setting is performed.

[0130] [8] When the display control unit is configured to be able to control the display of the camera image by setting whether the display area of ​​the camera image is displayed on the multiple display devices in cooperation with each other or on a single display device depending on the autonomous driving level, the situation inside or outside the vehicle, a display control program for a vehicle that causes the display control unit to execute a procedure for controlling the display of the camera images in cooperation with the plurality of display devices when the cooperative display setting is performed;

[0131] The techniques described in this disclosure may be implemented by a special purpose computer configured with a processor and memory programmed to perform one or more functions embodied in a computer program. Alternatively, the techniques described in this disclosure may be implemented by a special purpose computer configured with a processor comprising one or more dedicated hardware logic circuits. Alternatively, the techniques described in this disclosure may be implemented by one or more special purpose computers configured with a processor comprising one or more hardware logic circuits in combination with a processor and memory programmed to perform one or more functions. Furthermore, a computer program may be stored as instructions executed by a computer on a computer-readable non-transitory storage medium.

[0132] Although the present disclosure has been described based on the above-described embodiment, it is understood that the present disclosure is not limited to the embodiment or the structure described in the embodiment. The present disclosure also encompasses various modifications and modifications within the equivalent range. In addition, various combinations and forms, as well as other combinations and forms including only one element, more than one element, or less than one element, are also within the scope and spirit of the present disclosure. [Explanation of symbols]

[0133] In the drawings, 2a indicates a display (display device) in front of the D seat, 2b indicates a center information display (display device), 2c indicates a display (display device, display device for the passenger seat) in front of the P seat, 5x indicates a display system ECU (vehicle display control device), 10 indicates an information processing device (vehicle display control device), 11 indicates a control unit (display control unit), 11c indicates an autonomous driving level determination unit, 11d indicates a usage determination unit, 11e indicates a surrounding determination unit, and 11f indicates a tire display determination unit.

Claims

1. A vehicle display control device (5x, 10) that enables the display areas of multiple display devices (2, 2a to 2e) to be linked to display camera images, A display control unit (11) controls the display of the camera image by setting whether to display the camera image in a coordinated manner on at least two or more of the multiple display devices or on a single display device, depending on the level of autonomous driving and the conditions inside or outside the vehicle. It includes a usage determination unit (11d) that determines whether or not the passenger seat display device (2c, 2e) is being used, The display control unit controls the display of the camera image by setting whether to display it in a linked display or on a single display device according to the determination result of the usage determination unit. Vehicle display control device.

2. A vehicle display control device (5x, 10) that enables the display areas of multiple display devices (2, 2a to 2e) to be linked to display camera images, A display control unit (11) controls the display of the camera image by setting whether to display the camera image in a coordinated manner on at least two or more of the multiple display devices or on a single display device, depending on the level of autonomous driving and the conditions inside or outside the vehicle. The system includes an automatic driving level determination unit (11c) that determines whether the automatic driving level is automatic driving or manual driving, The display control unit controls the display of the camera image by setting whether to display it in a linked display or on a single display device according to the determination result of the automatic driving level determination unit. Vehicle display control device.

3. A vehicle display control device (5x, 10) that enables the display areas of multiple display devices (2, 2a to 2e) to be linked to display camera images, A display control unit (11) controls the display of the camera image by setting whether to display the camera image in a coordinated manner on at least two or more of the multiple display devices or on a single display device, depending on the level of autonomous driving and the conditions inside or outside the vehicle. It includes a tire display determination unit (11f) that determines whether or not to display a tire, The display control unit controls the display of the camera image by setting whether to display it in a linked display or on a single display device according to the determination result of the tire display determination unit. Vehicle display control device.

4. It includes a surrounding area determination unit (11e) that determines whether or not pedestrians, other vehicles, or foreign objects are present in the surrounding area. The vehicle display control device according to any one of claims 1 to 3, wherein the display control unit controls the display of the camera image by setting whether to display it in conjunction with the surrounding area or on a single display device according to the determination result of the surrounding area determination unit.

5. Multiple display devices (2a to 2e), The system includes a display control unit (11) that controls the display of the camera image by setting whether to display the camera image in conjunction with the display areas of the multiple display devices or on a single display device, depending on the autonomous driving level and the conditions inside or outside the vehicle; a usage determination unit (11d) that determines whether or not the passenger seat display devices (2c, 2e) are being used; and a display control device (5x, 10) that controls the display of the camera image by coordinating the display areas of the multiple display devices. The display control unit controls the display of the camera image by setting whether to display it in a linked display or on a single display device according to the determination result of the usage determination unit. Vehicle display and control system.

6. Multiple display devices (2a to 2e), The system includes a display control unit (11) that controls the display of the camera image by setting whether to display the camera image in conjunction with the display areas of the multiple display devices or on a single display device, depending on the autonomous driving level and the conditions inside or outside the vehicle; an autonomous driving level determination unit (11c) that determines whether the autonomous driving level is autonomous driving or manual driving; and a display control device (5x, 10) that controls the display of the camera image by coordinating the display areas of the multiple display devices. The display control unit controls the display of the camera image by setting whether to display it in a linked display or on a single display device according to the determination result of the automatic driving level determination unit. Vehicle display and control system.

7. Multiple display devices (2a to 2e), The system includes a display control unit (11) that controls the display of the camera image by setting whether to display the camera image in conjunction with the display areas of the multiple display devices or on a single display device, depending on the autonomous driving level and the conditions inside or outside the vehicle; a tire display determination unit (11f) that determines whether or not to display the tires; and a display control device (5x, 10) that controls the display of the camera image by coordinating the display areas of the multiple display devices. The display control unit controls the display of the camera image by setting whether to display it in a linked display or on a single display device according to the determination result of the tire display determination unit. Vehicle display and control system.

8. The display control device includes an surrounding determination unit (11e) that determines whether or not there are pedestrians, other vehicles, or foreign objects in the surrounding area, The display control unit controls the display of the camera image by setting whether to display it in a linked display or on a single display device according to the determination result of the peripheral determination unit. A vehicle display control system according to any one of claims 5 to 7.

9. When the display control unit (11) is configured to control the display of the camera image by setting whether to display the camera image in conjunction with the display areas of multiple display devices (2a to 2e) or on a single display device, depending on the level of autonomous driving and the situation inside or outside the vehicle, The display control unit, The usage determination unit (11d) determines whether or not the passenger seat display devices (2c, 2e) are being used, and the determination result determines whether to display the information in conjunction with the other devices or on a single display device. When the aforementioned linked display setting is configured, the multiple display devices are linked to control the display of the camera image. A method for controlling vehicle displays.

10. When the display control unit (11) is configured to control the display of the camera image by setting whether to display the camera image in conjunction with the display areas of multiple display devices (2a to 2e) or on a single display device, depending on the level of autonomous driving and the situation inside or outside the vehicle, The display control unit, The automatic driving level determination unit (11c) determines whether the automatic driving level is automatic driving or manual driving, and the determination result of this unit determines whether to display the information in a linked display or on a single display device. When the aforementioned linked display setting is configured, the multiple display devices are linked to control the display of the camera image. A method for controlling vehicle displays.

11. When the display control unit (11) is configured to control the display of the camera image by setting whether to display the camera image in conjunction with the display areas of multiple display devices (2a to 2e) or on a single display device, depending on the level of autonomous driving and the situation inside or outside the vehicle, The display control unit, The tire display determination unit (11f) determines whether or not to display the tire, and based on the determination result, it is set whether to display it in a linked display or on a single display device. When the aforementioned linked display setting is configured, the multiple display devices are linked to control the display of the camera image. A method for controlling vehicle displays.

12. The display control unit controls the display of the camera image by setting whether to display it in conjunction with the other display device or on a single display device, according to the determination result of the surrounding determination unit (11e) which determines whether or not there are pedestrians, other vehicles, or foreign objects in the surrounding area. A vehicle display control method according to any one of claims 9 to 11.

13. When the camera image display area can be set to be displayed on multiple display devices in conjunction or on a single display device, depending on the level of autonomous driving and the conditions inside or outside the vehicle, and the camera image can be controlled by the display control unit (11), The display control unit, The usage determination unit (11d) determines whether or not the passenger seat display devices (2c, 2e) are being used, and based on the determination result, it is set whether to display the information in conjunction with the other devices or on a single display device. When the aforementioned linked display setting is performed, the procedure for coordinating the display and control of the camera image on the multiple display devices is executed. Vehicle display control program.

14. When the camera image display area can be set to be displayed on multiple display devices in conjunction or on a single display device, depending on the level of autonomous driving and the conditions inside or outside the vehicle, and the camera image can be controlled by the display control unit (11), The display control unit, The automatic driving level determination unit (11c) determines whether the automatic driving level is automatic driving or manual driving, and the determination result of this unit determines whether to display the information in a linked display or on a single display device. When the aforementioned linked display setting is performed, the procedure for coordinating the display and control of the camera image on the multiple display devices is executed. Vehicle display control program.

15. When the camera image display area can be set to be displayed on multiple display devices in conjunction or on a single display device, depending on the level of autonomous driving and the conditions inside or outside the vehicle, and the camera image can be controlled by the display control unit (11), The display control unit, The tire display determination unit (11f) determines whether or not to display the tire, and based on the determination result, it is set whether to display it in a linked display or on a single display device. When the aforementioned linked display setting is performed, the procedure for coordinating the display and control of the camera image on the multiple display devices is executed. Vehicle display control program.

16. The display control unit is configured to display the camera image on a single display device or to display it in conjunction with the surrounding area, depending on the determination result of the surrounding area determination unit (11e) which determines whether or not there are pedestrians, other vehicles, or foreign objects in the surrounding area. A vehicle display control program according to any one of claims 13 to 15.

Images