Vehicular display system, vehicular display method, and display program for vehicle

Abstract

To provide a vehicular display system by which a crewman can confirm adequate information while minimizing the number of times he shifts his sight line and shakes his neck, a vehicular display method and a display program for vehicle.SOLUTION: A determination part determines main content viewed by a vehicle crewman. A change detection part detects that status relating to the main content or status of related content relating to the main content has changed. A display processing part changes a display mode of the main content or the related content on the basis of a detection result by the change detection part, and displays the main content and the related content in a visual field range beyond the vehicle crewman's sight line.SELECTED DRAWING: Figure 2

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application is based on Japanese Application No. 2021-092992 filed on June 2, 2021, the contents of which are incorporated herein by reference. [Technical field]

[0002] The present disclosure relates to a vehicle display system, a vehicle display method, and a vehicle display program. [Background technology]

[0003] In order to supplement the visual information required for driving, many peripheral monitoring functions using peripheral cameras have been proposed. For example, see Patent Document 1. In the technology described in Patent Document 1, images captured by peripheral cameras at the rear and sides are processed into a continuous composite image, making multiple images appear as one. In addition, electronic mirror technology has been provided that makes it easier to monitor the surroundings by understanding the turning direction using blinkers and widening the angle of view in the turning direction.

[0004] In this way, by processing multiple images into one continuous image or by mechanically changing the angle of view on the camera side, the occupant can see the image that he or she is supposed to see without changing the location he or she is looking at. These issues can also be solved by framing the display while increasing the display area of ​​the display, but framing technology alone cannot provide the information that vehicle occupants want.

[0005] In addition, technology is being provided that supports driving by converting information obtained from various sensors into graphical visual information that is easy to see and understand. As technology improves, the amount of information handled within a vehicle increases, and the area of ​​the display screen and the number of displays installed tend to increase.

[0006] Although a large amount of information can be displayed to support vehicle occupants, the occupants' field of vision is limited and they cannot check all the information at once. This increases the number of times the vehicle occupants move their eyes or turn their heads, which is inconvenient. In addition, when a large amount of information is displayed, it is difficult to search for the necessary and appropriate information in a short amount of time and check it at a glance. [Prior art documents] [Patent documents]

[0007] [Patent Document 1] JP 2019-118016 A Summary of the Invention

[0008] An object of the present disclosure is to provide a vehicle display system, a vehicle display method, and a vehicle display program that enable occupants to check appropriate information while minimizing the number of times they move their eyes or shake their heads.

[0009] According to one aspect of the present disclosure, the determination unit determines the primary content being viewed by the vehicle occupant, the change detection unit detects a change in a situation related to the primary content or a change in a situation of related content related to the primary content, and the display processing unit changes a display mode of the primary content or the related content based on a detection result by the change detection unit, and displays the primary content and the related content in a visual field ahead of the line of sight of the vehicle occupant.

[0010] The driving load determination unit determines the driving load of the driver. The display processing unit changes the amount of information of the related content based on the driving load determined by the driving load determination unit and performs display processing in a state of autonomous driving level II or lower, or in a driving mode in which the driver is obligated to monitor the surroundings.

[0011] According to one aspect of the present disclosure, the main content and the related content are displayed in the field of view of the vehicle occupant's line of sight based on the detection result by the change detection unit, so that the vehicle occupant can check the related content as well as the main content. Therefore, the number of times the occupant moves their line of sight or shakes their head is minimized, and appropriate information can be confirmed. In addition, the amount of information of the related content is changed and displayed based on the driver's driving load. [Brief description of the drawings]

[0012] The above and other objects, features and advantages of the present disclosure will become more apparent from the following detailed description taken in conjunction with the accompanying drawings, in which: [Figure 1] FIG. 1 is a diagram showing a control image of the first embodiment; [Diagram 2] FIG. 2 is a block diagram illustrating a vehicle display system; [Diagram 3] FIG. 3 is a flowchart showing the outline of the processing contents. [Figure 4] FIG. 4 is an illustration of a display as seen by a vehicle occupant; [Diagram 5] FIG. 5 shows one example of a display mode. [Figure 6] FIG. 6 is a bird's-eye view showing the positional relationship between the host vehicle and an obstacle. [Figure 7] FIG. 7 shows a second example of the display mode. [Figure 8] FIG. 8 is an explanatory diagram of a display as seen by a vehicle occupant; [Figure 9] FIG. 9 shows a third example of the display mode. [Figure 10] FIG. 10 shows the fourth example of the display mode. [Figure 11] FIG. 11 is a first explanatory diagram of the change in display mode. [Figure 12] FIG. 12 is a second explanatory diagram of the change in the display mode. [Figure 13] FIG. 13 is a fifth example of a display mode in the second embodiment. [Figure 14]FIG. 14 is a sixth example of the display mode. [Figure 15] FIG. 15 is a seventh example of a display mode. [Figure 16] FIG. 16 is the eighth example of the display mode. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] Hereinafter, several embodiments of the vehicular display system 1, the vehicular display method, and the vehicular display program will be described with reference to the drawings. Note that the same reference numerals will be used to denote substantially common parts in the respective embodiments.

[0014] 1, the vehicle display system 1 is configured by a cockpit system equipped with multiple displays such as a center display 3 and an electronic mirror 4. However, the number, installation, and configuration of the displays are merely examples and are not limited to these.

[0015] The center display 3 is configured, for example, by a liquid crystal display or an organic EL display, and is installed between the driver's seat and the passenger seat.

[0016] The center display 3 is located near the center console where it is easily visible to both the driver and the passenger in the front passenger seat, and is capable of displaying various contents. An operation panel 21 is provided above the center display 3, and allows the selection of contents to be displayed on the center display 3, as well as input operations for operating the air conditioning, audio, and navigation functions.

[0017] The electronic mirror 4 is configured with a door mirror camera 4a and a display 4b. See Fig. 2. The door mirror camera 4a is a peripheral camera 23 that captures an image of the rear of the host vehicle Ca, that is, the side rear, from the location where the door mirror of the host vehicle Ca is installed.

[0018] The display 4b of the electronic mirror 4 is, for example, a liquid crystal display or an organic EL display. The displays 4b are installed on the left and right sides of the driver, for example, at the bottom of the A-pillars inside the vehicle.

[0019] The display 4b of the electronic mirror 4 is provided in a position that allows the driver to easily check the area behind the vehicle, and is configured to be able to display information about the surroundings of the vehicle captured by the door mirror camera 4a.

[0020] As shown in Figs. 1 and 2, many ECUs 5 are configured in the vehicle and connected to an in-vehicle network 25. The ECUs 5 include a display system ECU, a surroundings monitoring system ECU, a driving control system ECU, and a DCM that communicates with the outside of the vehicle. DCM stands for Data Communication Module. The driving control system ECUs are well-known vehicle control ECUs, engine control ECUs, motor control ECUs, brake control ECUs, steering control ECUs, and integrated control ECUs. The driving control system ECUs include an autonomous driving ECU. The autonomous driving ECU is an Autonomous Driving Electric Control Unit.

[0021] When the autonomous driving ECU receives an automatic control signal, it drives the driving actuator to execute a corresponding predetermined level of driving assistance and autonomous driving. For example, in level I driving assistance, the vehicle can execute automatic braking to avoid collisions with obstacles, following the vehicle in front, or lane prevention driving to prevent the vehicle from going out of the lanes on either side. In level II autonomous driving, the vehicle can execute a combination of level I driving assistance, or autonomous driving under specific conditions, such as an autonomous driving mode that automatically overtakes slower vehicles on a highway or automatically merges into and out of the highway. In level III or higher autonomous driving, the system executes all driving tasks while monitoring the vehicle.

[0022] Each ECU 5 is mainly composed of a microcomputer equipped with various storage units 6 such as a processor, a cache memory, a RAM, and a ROM, an I / O, and a bus connecting these. Each ECU 5 is connected to other ECUs 5 provided in the vehicle so as to be able to communicate with them via a communication control unit 7 and an in-vehicle network 25.

[0023] In this embodiment, as shown in FIG. 2, a plurality of display-system ECUs 5 constitute an HCU as a vehicle device 10, and the processing power of the internal physical resources of the display-system ECUs 5 is shared to perform display processing on the center display 3 and the display 4b of the electronic mirror 4. The display-system ECUs 5 may be connected to each other by a dedicated line, or may be constituted by a single ECU 5. HCU is an abbreviation for Human Machine Interface Control Unit and indicates an information presentation control device. The storage unit 6 described above indicates a non-transient substantial storage medium that non-temporarily stores programs and data that can be read by a computer. The non-transient substantial storage medium is realized by a semiconductor memory or the like.

[0024] As shown in FIG. 2, the vehicular device 10 includes a control device 11, a calculation device 12, a memory unit 6, a display processing unit 13, a sound processing unit 14, an I / O control unit 15 that manages inputs and outputs from various devices, and a communication control unit 7 that manages communications with other ECUs 5.

[0025] In addition, the output signals of the position detector 19, the operation panel 21, the occupant monitor 22, the peripheral camera 23, and the distance detection sensor 24 shown in FIG. 2 will be described as being input via the I / O control unit 15, but the signals may also be input from other ECUs 5, such as an ECU for a peripheral monitoring system or an ECU for a driving control system, via the in-vehicle network 25.

[0026] Based on the control of the control device 11, the calculation device 12 calculates the display area to be displayed on the display screen of the center display 3 and the display 4b of the electronic mirror 4 for content such as images, sentences, characters, etc. (hereinafter referred to as images, etc.) stored in the memory unit 6, calculates in which area of ​​the display screen of the center display 3 and the display 4b of the electronic mirror 4 the content such as images will be displayed, and on which area the content such as images will be superimposed, and outputs the display area together with the content such as images to the display processing unit 13 via the control device 11.

[0027] The display processing unit 13, under the control of the control device 11, processes and displays content such as images in the above-mentioned display areas on the display screens of the center display 3 and the display 4b of the digital mirror 4. This allows content such as images to be displayed in a superimposed manner for each display layer on the display screens of the displays 3 and 4b. The audio processing unit 14, under the control of the control device 11, inputs received voice input from the microphone 17 and outputs transmitted voice from the speaker 18. When text or character content is input from the control device 11, the audio processing unit 14 converts it into voice and reads it out through the speaker 18.

[0028] The position detector 19 detects the position with high accuracy using a well-known GNSS receiver such as a GPS, and an inertial sensor such as an acceleration sensor or a gyro sensor, not shown. The position detector 19 outputs a position detection signal to the control device 11 through the I / O control unit 15. The position identification unit 11a of the control device 11 realizes a function as an ADAS locator that sequentially locates the current position of the vehicle with high accuracy based on map information input from a map data input device and the position detection signal of the position detector 19. ADAS is an abbreviation for Advanced Driver Assistance Systems.

[0029] In this case, the vehicle position is represented by a coordinate system consisting of latitude and longitude, and in this coordinate system, for example, the X-axis indicates longitude and the Y-axis indicates latitude. The vehicle position may be measured using, for example, a vehicle speed sensor mounted on the host vehicle Ca. The vehicle speed sensor detects the speed of the vehicle and outputs a speed signal to the control device 11. The position may be measured based on information on a travel distance obtained based on a sensing result by the vehicle speed sensor, and various configurations may be adopted as long as the position of the host vehicle Ca can be specified. The control device 11 can perform so-called navigation processing based on the current position of the host vehicle Ca.

[0030] The operation panel 21 is a touch panel configured on the center display 3, and when an operation input is made by a passenger, the I / O control unit 15 accepts the operation input and outputs it to the control device 11. The control device 11 executes control based on the operation signal from the operation panel 21.

[0031] The occupant monitor 22 detects the state or operation state of an occupant in the vehicle. The occupant monitor 22 is configured using, for example, a power switch, an occupant state monitor, a turn switch, an automatic control switch, etc., and outputs various signals to the control device 11. The occupant monitor 22 may include a steering sensor that detects whether the steering wheel is being gripped or steered by the driver, a seating sensor that detects whether the driver is seated in the seat, an accelerator pedal or brake pedal depression sensor, etc.

[0032] The power switch outputs a signal corresponding to an operation performed by an occupant in the vehicle cabin to start the internal combustion engine or the electric motor. The occupant status monitor includes a camera that detects the status of an occupant in the D or P seat by capturing an image of the occupant using an image sensor and outputs an image signal. The driver's occupant status monitor is called DSM. DSM is an abbreviation for Driver Status Monitor. The occupant status monitor irradiates near-infrared light onto the driver's head, captures the captured image signal, performs image analysis as necessary, and outputs the image to the control device 11.

[0033] The occupant status monitor is used to detect the status of occupants such as the driver during driving assistance or automatic driving, in particular. When the turn switch is turned on by an occupant in the vehicle cabin when the turn indicator of the host vehicle Ca is activated, the turn switch outputs a turn signal to turn right or left in response to the turn.

[0034] When an occupant in the vehicle turns on the automatic control switch in order to command a predetermined driving assistance or automatic control for the driving state of the vehicle, the automatic control switch outputs an automatic control signal corresponding to the operation. This automatic control signal is output to the ECU of the driving control system, so that a corresponding predetermined level of driving assistance or automatic driving control can be executed.

[0035] The control device 11 can determine the behavior of the vehicle occupant, for example, which direction the eye is looking, based on the sensor signal from the occupant monitor 22, and can also input the operation status of the power switch, the operation status of the turn signal, command information for the automatic control of the vehicle, and speed information from the vehicle's speed sensor.

[0036] The peripheral cameras 23 include a forward visibility camera that images the area in front of the vehicle, a rear camera that images the area behind the vehicle, a corner camera that images the front and rear parts of the vehicle, a side camera that images the side of the vehicle, and a door mirror camera 4a of the electronic mirror 4, and these are output to the control device 11 via the I / O control unit 15 as respective imaging signals of the front guide monitor, back guide monitor, corner view monitor, side guide monitor, and electronic mirror 4, and stored in the memory unit 6.

[0037] The display 4b of the electronic mirror 4 displays the image captured by the door mirror camera 4a. The electronic mirror 4 reproduces a rearview mirror. Therefore, the electronic mirror 4 displays a mirror-inverted image of the image captured by the door mirror camera 4a on the display 4b for the electronic mirror 4.

[0038] The vehicle is also provided with a distance detection sensor 24 as a perimeter monitoring sensor for detecting the distance to an obstacle. The distance detection sensor 24 is composed of a clearance sonar 24a, LiDAR, or a radar using millimeter waves, and detects the distance to an obstacle such as a vehicle or a person approaching the front, front side, rear side, rear, or side of the vehicle.

[0039] Each ECU 5 is equipped with an SoC. SoC stands for System On Chip. The above-mentioned microcomputer is built into the mounted SoC. The microcomputer built into the SoC of the ECU 5 is configured to run a variety of applications (hereinafter referred to as apps) on a pre-installed general-purpose OS and real-time OS, which enables the control device 11 to realize the functions described below. The apps include an image processing app and other apps. In response to a drawing request from the image processing app, a processor built into the SoC performs drawing processing on the display screen of the center display 3 and the display 4b of the electronic mirror 4.

[0040] Furthermore, the content rendered by the app can also be displayed on another display, here the center display 3. The apps include a navigation app, etc. The navigation app realizes the navigation function and renders a navigation screen including a map image, the current position of the vehicle, etc., that is displayed mainly on the center display 3 or the display 4b of the electronic mirror 4.

[0041] The applications also include an image synthesis application. The image synthesis application specifies the size and type of various image contents to be displayed on the display, synthesizes the images of the image contents within one frame, and outputs this synthesized mixed image to the center display 3 or the display 4b of the electronic mirror 4. The image synthesis application realizes the function of an image synthesis unit, also called a compositor, and the function of an image output unit.

[0042] The applications also include a detection application. The detection application detects the gestures of the occupants in the vehicle based on the image signal from the occupant monitor 22, detects finger operations and their line of sight, and detects facial expressions and gestures of the occupants in the vehicle. It also detects the operator of the touch panel of the operation panel 21.

[0043] Among the applications, an application that renders image content is assigned a display layer for rendering the image content. These display layers are allocated in the storage unit 6 in a size that allows the necessary image content to be rendered.

[0044] Various contents are stored in the storage unit 6. The contents are classified into entertainment contents such as movies and television images, meter screens, images captured by the door mirror camera 4a of the electronic mirror 4, images captured by the peripheral cameras 23 (for example, a front imaging camera, a corner camera, a side camera, a rear camera, the door mirror camera 4a, etc.), around-view image contents created by synthesizing the images captured by the peripheral cameras 23, and the like.

[0045] The above-mentioned contents are associated with each other, and are registered in a database in the storage unit 6 with a relationship between the main content and the related content. For example, contents indicating the detection conditions Sa and Sb of the clearance sonar 24a are associated with the captured image of the electronic mirror 4. Furthermore, the captured images of the above-mentioned peripheral cameras 23, i.e., the captured images of the front capturing camera, corner camera, side camera, rear camera, and door mirror camera 4a, are associated with each other.

[0046] Furthermore, the map image and meter image related to navigation are stored in association with the captured image of the electronic mirror 4. Furthermore, the navigation map image is stored in the storage unit 6 in association with the content of supplementary information. The supplementary information is, for example, the content of the operation instruction button display screen such as destination, current location, navigation setting, environment setting, map detail display, display change, map display off, destination setting, point registration, wide area map change, etc. Furthermore, the map image related to navigation is stored in the storage unit 6 in association with information of facilities such as restaurants near the map location.

[0047] When a certain content stored in the storage unit 6 is determined as the main content, the control device 11 can read and refer to the related content related to the main content by referring to the storage unit 6. This allows the display processing unit 13 to simultaneously display the main content and the related content on the display 3 or 4b.

[0048] Furthermore, the content displayed on each of the displays 3 and 4b can be animated. Here, animation refers to a display mode in which the position or size of an image showing the content gradually changes, the image rotates, the user interface moves as a whole in response to a swipe operation, the image gradually fades in or out, or the color of the image changes. However, animation is not limited to these, and any display mode that changes continuously or intermittently from a certain point in time is included in the animation.

[0049] The control device 11 shown in Fig. 2 executes various applications stored in the storage unit 6. In this embodiment, the control device 11 executes a navigation application stored in the storage unit 6 to realize a navigation function. The control device 11 also executes a detection application stored in the storage unit 6 to detect the presence or absence of an occupant sitting in the driver's seat (hereinafter, D seat), passenger seat (hereinafter, P seat), or rear seat in the vehicle based on a monitor signal from the occupant monitor 22. The control device 11 also functions as a gaze detection unit 11b that detects the gaze of an occupant sitting in the D seat, P seat, or rear seat.

[0050] The control device 11 executes an application stored in the storage unit 6 to realize a function as a determination unit 11c that determines the main content being viewed by the vehicle occupant. The control device 11 executes an application stored in the storage unit 6 to realize a function as a change detection unit 11d that detects a change in a situation related to the main content, a change in a situation of related content related to the main content, or a change in the vehicle occupant viewing the main content. The control device 11 also executes an application stored in the storage unit 6 to realize a function as a driving load determination unit 11e that determines the driving load of the vehicle occupant, particularly the driver.

[0051] The display processing unit 13 changes the display mode of the main content or the related content based on the detection result by the change detection unit 11d, and displays the main content and the related content in a predetermined visual field ahead of the line of sight of the vehicle occupant.

[0052] The above operation will be described with reference to Fig. 3 and subsequent drawings. As shown in Fig. 3, when the power switch is operated to start the vehicle, the control device 11 starts an application such as a navigation application in S1. When the application is started, the passenger monitor 22 is used to monitor the line of sight of the vehicle passenger in S2, and the display processing unit 13 outputs and displays an image on each of the displays 3 and 4b.

[0053] In S4, the control device 11 determines the content being viewed by the driver and recognizes it as the main content. The driver corresponds to a first vehicle occupant. Here, the control device 11 can determine the content being viewed by detecting the driver's line of sight and determining which display 3, 4b the driver is looking at.

[0054] The control device 11 then determines in S5 whether a situation related to the main content has changed, determines in S6 whether a situation of the related content has changed, and determines in S7 whether a vehicle occupant viewing the main content has changed.

[0055] The determination in S5 "the situation related to the main content has changed" indicates, for example, when the image captured by the electronic mirror 4 is recognized as the main content, and a change is detected in which an obstacle begins to be recognized in the image captured by the electronic mirror 4 as the vehicle backs up. In addition, when the main content seen by the driver is the detection situation Sa of the clearance sonar 24a, it indicates a case where some change is detected, such as an obstacle being detected around the vehicle Ca by the clearance sonar 24a. In other words, when some change occurs in the main content, YES is determined in S5.

[0056] In addition, the judgment result of S6 "the status of related content has changed" indicates, for example, when the main content being viewed by the driver is the image captured by the electronic mirror 4, there is a change in the detection status Sa of the clearance sonar 24a, which is related content related to the image captured by the electronic mirror 4, such as when an obstacle begins to be detected.

[0057] The determination in S6 "the status of related content has changed" indicates a case where, when the main content being viewed by the driver is the detection status Sa of the clearance sonar 24a, an obstacle begins to be detected in the captured image of the electronic mirror 4 related to the detection status Sa of the clearance sonar 24a, causing a change in the image recognition status of the captured image. In other words, when any change occurs in the related content related to the main content, YES is determined in S6.

[0058] When the control device 11 determines in S5 that the situation related to the main content has changed, or in S6 that the situation of the related content has changed, it changes the display mode in S9 and then identifies the display 3, 4b to be displayed on in S10, and the display processing unit 13 outputs the image to the identified display 3, 4b for display.

[0059] The determination in S7 "the passenger viewing the main content has changed" indicates, for example, a case where the driver's line of sight has shifted from the main content and the passenger in the P seat has started viewing the same main content. When the passenger viewing a specific main content has changed, the result in S7 is YES. This is equivalent to a vehicle occupant.

[0060] When the control device 11 determines in S7 that the vehicle occupant viewing the main content has changed, it detects the line of sight of the second vehicle occupant viewing the main content using the occupant monitor 22 in S8, changes the display mode of the main content or related content in S9, identifies the display (e.g., 3, 4b, etc.) on which to display in S10, and outputs and displays the image in S11.

[0061] An example of application of the present invention to actual vehicle operation will be described below. For example, when a driver drives his / her vehicle Ca backward to park in a parking lot, the driver looks at the detection status Sa of the clearance sonar 24a displayed on the center display 3 as shown in Figs. 4 and 5.

[0062] The control device 11 detects that the driver is looking at the detection status Sa of the clearance sonar 24a. At this time, the detection status Sa of the clearance sonar 24a is recognized as the main content. When there is no obstacle around the vehicle Ca and the clearance sonar 24a does not react to proximity, the display processing unit 13 displays the detection status Sa of the clearance sonar 24a on the center display 3 as shown in FIG. 5.

[0063] Thereafter, when the driver reverses the host vehicle Ca, assuming that an obstacle Ob is present to the left rear of the host vehicle Ca as shown in FIG. 6, the clearance sonar 24a detects the obstacle Ob to the left rear of the host vehicle Ca.

[0064] At this time, the display processing unit 13 displays the detection direction of the obstacle reacting to the clearance sonar 24a and the captured image of the peripheral camera 23 capturing the surroundings of the detection direction as related content in the visual field. Specifically, the display processing unit 13 may display a warning in the direction in which the obstacle was detected together with the detection status Sa of the clearance sonar 24a, and may further display the captured image of the peripheral camera 23 corresponding to the direction in which the clearance sonar 24a reacted adjacent to the display area of ​​the detection status Sa. In the case shown in FIG. 6, it is preferable to display the captured image of the door mirror camera 4a of the electronic mirror 4 capturing the image of the left rear as shown in FIG. 7. At this time, it is preferable for the display processing unit 13 to display the captured image so as to be within the visual field of the vehicle occupant.

[0065] As a result, before a situation change is detected by the change detection unit 11d of the control device 11, the display processing unit 13 displays the image captured by the door mirror camera 4a of the electronic mirror 4, which is the related content, on the display 4b of the electronic mirror 4, which is the default predetermined position, and after a situation change is detected, the related content can be moved to the field of view of the center display 3. At this time, the display processing unit 13 may erase the related content that has been displayed on the display 4b of the electronic mirror 4 as the default, or may copy and display it on the center display 3 while leaving it displayed on the display 4b of the electronic mirror 4. It is desirable to determine in advance which direction (forward, backward, front-side, rear-side, etc.) of the image content captured by the peripheral camera 23 should be displayed for the reaction position of the clearance sonar 24a, and to store the correspondence in advance in the memory unit 6.

[0066] Moreover, it is desirable that the display processing unit 13 changes the position where the related content is displayed according to the relative positional relationship between the position of the vehicle occupant viewing the main content and the main content. For example, when the related content is related to the right side of the driver viewing the main content, it is preferable to display the related content to the right of the main content, and conversely, when the related content is related to the left side of the driver viewing the main content as shown in Figs. 6 and 7, it is preferable to display the related content to the left of the main content. Also, for example, when it is detected that the occupant in the P seat has viewed the content, it may be displayed on the side of the occupant in the P seat. When a dangerous place is shown by a captured image, it is preferable to display the main content as the center and the relative position of the dangerous place.

[0067] Furthermore, even if the driver subsequently moves his / her gaze to the side of the center of a predetermined center display 3, the control device 11 detects this change in the gaze movement, and the display processing unit 13 displays an image captured by the peripheral camera 23 capturing an image in the direction in which the clearance sonar 24a reacts, for the specified center display 3. If the gaze is moved toward the center of the center display 3, an image captured by the backup camera should be displayed. The display processing unit 13 should display image content so that it is within the field of view of the destination of the gaze.

[0068] Even if the line of sight is not moved after the clearance sonar 24a has finished reacting to an obstacle, it is desirable to keep the image captured by the peripheral camera 23 displayed for a certain period of time. This allows the vehicle occupant to check the situation around the vehicle Ca and recognize that there are no problems. This reduces the frequency with which the vehicle occupant changes the direction of their line of sight, and reduces the number of times they turn their head.

[0069] Also, consider a case where the driver drives the vehicle Ca backward and parks it in a home parking space located to the left rear while checking the display 4b of the left electronic mirror 4 as shown in Figures 8 and 9. In this way, when the driver is looking at the image captured by the left door mirror camera 4a, the control device 11 recognizes the image captured by the electronic mirror 4 as the main content.

[0070] In normal driving, not automatic driving, the driver must always check the driving direction, so it is difficult to continue driving while continuously looking at the display 4b of the electronic mirror 4. Also, when passing an oncoming vehicle while driving on a narrow road, or when passing obstacles on the left and right, it is better to drive while paying attention to the left and right electronic mirrors 4 as in the past. For this reason, when the control device 11 detects that the driver is looking at the display 4b of the electronic mirror 4 by the passenger monitor 22, it is preferable that the detection situation Sa of the clearance sonar 24a is set as related content, and the display processing unit 13 displays the detection situation Sa in which the clearance sonar 24a reacts to another obstacle in a part of the display area of ​​the electronic mirror 4, as shown in FIG. 10. As shown in FIG. 10, even if another vehicle Ce is present to the side and rear of the host vehicle Ca, the direction can be recognized.

[0071] In summary, as shown in FIG. 11, when the display processing unit 13 displays the detection status Sa of the clearance sonar 24a on the center display 3, and further determines that the driver is looking at the detection status Sa of the clearance sonar 24a, the display processing unit 13 displays the image captured by the electronic mirror 4 together with the normal image of the detection status Sa of the clearance sonar 24a on the same display 3.

[0072] In addition, when the display processing unit 13 displays the captured image on the display 4b of the electronic mirror 4 and determines that the driver is looking at the electronic mirror 4, the display processing unit 13 displays the detection status Sa of the clearance sonar 24a on the same display 4b together with the normal captured image of the electronic mirror 4. In other cases, the normal display is continued. This makes it possible to draw the driver's attention.

[0073] When there are multiple images that the driver should check depending on the driving environment or traffic conditions, the display processing unit 13 may display them together in one place as main content and related content. It is also preferable to display them in a way that is optimized for the location where the passenger is looking.

[0074] Moreover, it is preferable that the display processing unit 13 performs display processing by changing the related contents or changing the display mode of the related contents depending on the display priority set for the main content and the related contents according to the driving environment or traffic conditions. Specifically, the control device 11 sets display priorities for the images captured by the peripheral camera 23 and the images captured by the electronic mirror 4 according to the driving environment or traffic conditions that change from moment to moment, and displays the images on the displays 3 and 4b using the display priorities. At this time, it is preferable to display the images on the displays 3 and 4b in the order of the contents with the highest display priorities.

[0075] In this case, if the display priority of the related content is changed to be higher than that of the main content, it is preferable to display the related content so that it stands out against the main content. If the situation of the related content changes and the display priority is changed to be higher, it is preferable to change the related content itself or the display mode of the related content.

[0076] The control device 11 detects the images captured by the peripheral camera 23 and the behavior of the host vehicle Ca, and thereby can determine the situation in which the host vehicle Ca is placed in the driving environment or traffic conditions including location and time.

[0077] For example, a case will be described in which the position identification unit 11a determines that the vehicle Ca is traveling on an expressway. When the control device 11 detects many other vehicles Ce around the vehicle Ca while traveling on the expressway and detects that the traveling speed of the vehicle Ca has rapidly decreased within a predetermined time, it can determine the driving environment and traffic conditions of the vehicle Ca, such as the vehicle Ca being caught in a traffic jam while traveling on the expressway.

[0078] If such driving environment and traffic conditions are stored in a database in the memory unit 6, the control device 11 can refer to the database in the memory unit 6 and detect using the surrounding camera 23 and distance detection sensor 24 to determine how densely packed other vehicles Ce are around the host vehicle Ca, or whether the number of other vehicles Ce is increasing around the host vehicle Ca, thereby predicting whether traffic congestion will occur or how severe it will be.

[0079] For this reason, when it is recognized that the vehicle has decelerated more rapidly than a predetermined amount while traveling on an expressway, the control device 11 detects that a traffic jam has occurred by referring to the database in the storage unit 6, and further predicts the extent of the traffic jam, and may display, for example, an image captured by a backup camera, among the surrounding cameras 23, which shows the situation of the vehicle following the vehicle Ca, as related content on the center display 3, which displays a map screen as the main content. By performing display processing in this manner, the driver's attention can be drawn.

[0080] In addition, for example, a case will be described in which the position identification unit 11a determines that the vehicle is traveling near the driver's home address. When the vehicle is determined to be traveling near the driver's home address, the control device 11 determines whether the shift range is in the R range, and by referring to the database stored in the storage unit 6, estimates events that are likely to cause accidents when parking the vehicle in the home parking space, and reflects the results in the display priority of the image to be displayed on the display 3 or 4b.

[0081] For example, when the driver reverses the vehicle Ca while turning the steering wheel to the left, the driver reverses the vehicle Ca while checking the image on the display 4b of the left electronic mirror 4, as shown in Fig. 10. However, if the driver pays too much attention to the display 4b of the left electronic mirror 4, he or she is likely to overlook another obstacle (not shown) on the right side. If the accident event is registered in the database of the storage unit 6, the control device 11 can refer to the accident event.

[0082] When an obstacle is detected to the right rear by the clearance sonar 24a, the display processing unit 13 may display the right detection situation Sb on the display 4b of the electronic mirror 4 instead of the left detection situation Sa displayed as related content, as shown in Fig. 12. Also, as shown in Fig. 12, the right detection situation Sb may be displayed larger than the detection situation Sa to alert the driver. In the example shown in Fig. 12, only the right detection situation Sb is displayed as related content, but the left detection situation Sa may also be displayed together.

[0083] According to this embodiment, the display mode of the main content or the related content is changed based on the detection result by the change detection unit 11d, and the main content and the related content are displayed in the field of view of the vehicle occupant who is viewing the main content and the related content. Therefore, the vehicle occupant can check the related content together with the main content at the same time. This allows the vehicle occupant to check appropriate information while minimizing the number of times they move their eyes or shake their head.

[0084] Second embodiment The second embodiment will be described with reference to Figures 13 to 16. In the second embodiment, the relationship between the main content and the related content when the passenger views a map screen drawn according to a navigation application will be described.

[0085] The map screen B1 shown in Fig. 13 is drawn on the display 3 or the like when a navigation application is executed. Normally, it is preferable for the display processing unit 13 to display the map screen B1 large and filling the display screen of the center display 3. This is because the driver can easily check the map screen B1 even if he or she glances at it while paying attention to the road ahead during normal driving. In other words, when the control device 11 detects through the passenger monitor 22 that the driver is looking at the map screen B1, it is preferable for the display processing unit 13 to display the map screen B1 large and filling the display screen of the center display 3, without displaying any unnecessary items.

[0086] Since it would be inconvenient if the necessary information was not displayed without the occupant's operation, it is also possible to display the necessary information in advance, such as map direction information B2 and VICS (registered trademark) information acquisition time B3, as shown in FIG. 14.

[0087] Furthermore, on condition that the control device 11 detects that the map screen B1 has been viewed by the occupant for a certain period of time, the display processing unit 13 may display supplementary information for the map screen B1 as related content, as shown in Fig. 15. The supplementary information shown in Fig. 15 is, for example, the contents of an operation instruction button display screen B8 such as destination setting B4, current location setting B5, other navigation settings B6, environment setting B7, map detail display, display change, map display off, destination setting, point registration, wide area map change, etc.

[0088] That is, when the control device 11 judges that the driver has not looked at the screen or has looked at the screen for less than a certain period of time, the display processing unit 13 does not display the supplementary information, and when the driver continues to look at the screen for a certain period of time, the display processing unit 13 displays the supplementary information in a superimposed manner. This allows the information to be provided while gradually increasing it, and information according to the situation can be provided. The driver can intuitively confirm more information with a delay, and the cognitive processing burden on the driver is also reduced.

[0089] Furthermore, the control device 11 and the display processing unit 13 may change the content displayed on the center display 3 depending on whether the display is being viewed by the driver or by the passenger in seat P. That is, when the control device 11 determines in S7 of Fig. 3 that the passenger viewing the map screen has changed and that the passenger in seat P is now viewing the map information for the navigation system, the display processing unit 13 may detect the line of sight of the passenger in seat P in S8, and then perform processing in S9 to S11 to display, as related content, entertainment information being held near the current location displayed on the map screen B1, or recommended stores in the vicinity B9, as shown in Fig. 16.

[0090] At this time, the display processing unit 13 may display the related content on the side of the passenger in seat P who is looking at the map screen B1. At this time, it may be displayed within the visual field of the passenger in seat P. This allows the passenger in seat P to reduce the frequency of moving his or her line of sight and the number of times he or she turns his or her head.

[0091] In addition, it is preferable that the control device 11 judges the driving load of the driver by the driving load judgment unit 11e, and the display processing unit 13 changes the amount of information of the related content according to the driving load of the driver and performs display processing. The driving load can be calculated by detecting the driving actuator operated by the driver, the operation status of the navigation display screen, the autonomous driving level, or the state of the occupant by the occupant monitor 22 using well-known technology.

[0092] For example, when the control device 11 determines that the driver's driving load is within the standard range by the driving load determination unit 11e, the amount of information is displayed as a normal predetermined amount, but when the control device 11 determines that the driving load is lower than the standard, the amount of information may be increased. Conversely, when the control device 11 determines that the driver's driving load is higher than the standard, it is preferable to make the expression less noticeable than the standard or to reduce the amount of information. In other words, it is preferable to change the amount of information for the related content according to the driver's driving load and display it.

[0093] For example, when the vehicle is stopped or when the navigation destination or current location is set while the vehicle is stopped, the driving load is low, so the amount of information on the related content may be displayed in a larger amount than the standard amount. Also, during automated driving of level III or higher, the driving load on the driver may be determined to be low, and the amount of information on the related content may be displayed in a larger amount than the standard amount. Also, during following driving by ACC or steady driving by lane keep assist (LAS), the driving load may be determined to be low, and the amount of information on the related content may be displayed in a larger amount than the standard amount.

[0094] Conversely, if the control device 11 determines using the perimeter monitoring sensor that there are more obstacles around the vehicle than a predetermined amount, or determines by referring to traffic information from VICS (registered trademark) or the like that the vehicle is traveling on a highway where the traffic volume in the vicinity is greater than a predetermined amount, or determines using the perimeter monitoring sensor that another vehicle Ce is entering the vehicle's lane while traveling on a highway, or determines that people are present or moving around the vehicle even while traveling on a general road, it may determine that the driving load is high and display the amount of information in the related content less than standard or in a less prominent manner.

[0095] Furthermore, when lane change assist is being performed in level II driving assistance, the driver is legally obligated to monitor, especially while steering the vehicle. The lane change assist is called Lane Change Assist. The control device 11 may determine that the driving load is high while lane change assist is being performed, and may reduce the amount of information of the related content below the standard amount or display it in an inconspicuous manner.

[0096] It is desirable for the display processing unit 13 to determine whether or not to display related content or change the display mode depending on whether or not the related content is being checked by one or more vehicle occupants, because safety can be improved by preventing vehicle occupants such as the driver from checking the related content depending on the situation.

[0097] For example, if the display processing unit 13 displays useful video content targeted at the passenger in the P seat as related content, it becomes an unsafe distraction for the driver. It is preferable to stop displaying the video content when it is detected that the driver has viewed the related content. In addition, when there are multiple passengers viewing the video content, it is preferable for the control device 11 to determine which passenger is viewing it, and to stop displaying the video content when the driver is viewing it and there is a problem with the safety of vehicle operation.

[0098] When the vehicle changes from a stopped state to a driving state, it is desirable to stop the video content if it is detected that the driver is watching entertainment video content. However, during autonomous driving at a predetermined level or higher, for example, during autonomous driving at level III or higher, the driver may be treated as a passenger in the P seat, and the display processing unit 13 may continue to display the video content.

[0099] (Other embodiments) The present disclosure is not limited to the above-described embodiment, and can be implemented in various modifications and variations without departing from the spirit and scope of the present disclosure. For example, the following modifications or extensions are possible.

[0100] When the change detection conditions are satisfied by the change detection unit 11d, the main content and related content are displayed in the field of view ahead of the vehicle occupant's line of sight, but conversely, a polarizing filter or the like may be used to adjust the width of the visible range and display the main content and related content in the field of view.

[0101] In the drawing, 11c denotes a determination section, 11d denotes a change detection section, 11e denotes a driving load determination section, and 13 denotes a display processing section.

[0102] The control device 11 and the display processing unit 13 described in the present disclosure may be realized by a dedicated computer provided by configuring a processor and a memory programmed to execute one or more functions embodied in a computer program. Alternatively, the control device 11, the display processing unit 13, and the method described in the present disclosure may be realized by a dedicated computer provided by configuring a processor with one or more dedicated hardware logic circuits. Alternatively, the control device 11, the display processing unit 13, and the method described in the present disclosure may be realized by one or more dedicated computers configured by combining a processor and a memory programmed to execute one or more functions and a processor configured with one or more hardware logic circuits. In addition, the computer program may be stored in a computer-readable non-transient tangible recording medium as instructions executed by a computer.

[0103] Although the present disclosure has been described based on the above-described embodiment, it is understood that the present disclosure is not limited to the embodiment or structure. The present disclosure also includes various modifications and modifications within the equivalent range. In addition, various combinations and forms, as well as other combinations and forms including one, more, or less than one element, are also within the scope and concept of the present disclosure.

Claims

1. A determination unit (11c) for determining a main content being viewed by a vehicle occupant; a change detection unit (11d) for detecting a change in a situation related to the primary content or a change in a situation of related content related to the primary content; a display processing unit (13) that changes a display mode of the primary content or related content related to the primary content based on a detection result by the change detection unit, and displays the primary content and the related content in a visual field ahead of a line of sight of the vehicle occupant; A driving load determination unit (11e) that determines a driving load of a driver, The display processing unit is a display system for a vehicle that changes the amount of information of the related content and processes the display based on the driving load determined by the driving load determination unit in a state of autonomous driving level II or below, or in a driving mode in which the driver is obligated to monitor the surroundings.

2. 2. The vehicle display system according to claim 1, wherein the driving load determination unit determines that the driving load is low when the vehicle is stopped or when a navigation destination or current location is being set while the vehicle is stopped, and the display processing unit displays a larger amount of information of the related content than standard.

3. 2. The vehicle display system according to claim 1, wherein the driving load determination unit determines that the driving load is low during following driving using ACC or steady driving using lane keep assist (LAS), and the display processing unit displays an amount of information of the related content that is greater than the standard amount.

4. 2. The vehicle display system according to claim 1, wherein the driving load determination unit determines that the driving load is high while lane change assist is being performed, and the display processing unit displays the related content in a manner that reduces the amount of information thereof below standard or makes it less noticeable.

5. 5. A vehicle display system as described in any one of claims 1 to 4, wherein before a change in situation is detected by the change detection unit, the display processing unit displays the related content at a default specified position, and after the change in situation is detected, the display processing unit moves or copies the related content into the field of view range.

6. The vehicular display system according to claim 1 , wherein the display processing unit changes a position at which the related content is displayed depending on a relative positional relationship between a position of the vehicle occupant viewing the primary content and the primary content.

7. The vehicle display system according to claim 5, wherein the display processing unit changes the related content or changes the display mode of the related content using display priorities set for the main content and the related content depending on the driving environment or traffic conditions.

8. The display processing unit determines whether to display or changes the display mode depending on whether or not one or more of the vehicle occupants are checking the related content.

9. A determination unit (11c) determines the main content being viewed by the vehicle occupant; A change detection unit (11d) detects a change in a situation related to the primary content or a change in a situation of related content related to the primary content; a display processing unit (13) that changes a display mode of the primary content or related content related to the primary content based on a detection result by the change detection unit, and displays the primary content and the related content in a visual field ahead of the line of sight of the vehicle occupant; A driving load determination unit (11e) determines the driving load of the driver, A display method for a vehicle in which the display processing unit changes the amount of information of the related content based on the driving load determined by the driving load determination unit and processes the display when the autonomous driving level is II or lower, or when the driver is in a driving mode where he or she is obligated to monitor the surroundings.

10. A procedure for causing a determination unit (11c) to determine a primary content being viewed by a vehicle occupant; a step of causing a change detection unit (11d) to detect a change in a situation related to the primary content or a change in a situation of related content related to the primary content; a step of causing a display processing unit (13) to change a display mode of the primary content or related content related to the primary content based on a detection result by the change detection unit, and displaying the primary content and the related content in a visual field ahead of a line of sight of the vehicle occupant; A procedure for causing a driving load determination unit (11e) to determine a driving load of a driver. The procedure for displaying in the display processing unit includes a vehicle display program that changes the amount of information on the related content based on the driving load determined by the driving load determination unit and processes the display when the autonomous driving level is II or lower, or when the driving mode is such that the driver is obligated to monitor the surroundings.

Images