Display control device, head-up display device, program, and on-vehicle agent system

Abstract

Provided is a display control device and the like that are suitable for an on-vehicle agent system which, as a result of changing the representation form of an agent image in accordance with the driving load, does not hinder driving.　A display control device 40 that controls a display device 700 for displaying an agent image capable of exchanging information with an occupant riding in a vehicle comprises: a storage unit 42 that stores the agent image; and a control unit 41 that determines the driving load from at least one of the state of the vehicle, the driving environment, and the state of an occupant who is driving, and that performs control for changing the representation form of the agent image in accordance with the determined driving load and displaying the same on the display device.

Description

Display control device, head-up display device, program, and in-vehicle agent system

[0001] The present invention relates to a display control device suitable for use in, for example, an in-vehicle agent system that provides driving assistance to a vehicle occupant.

[0002] In the past, there has been a known in-vehicle agent system in which an agent appears in the vehicle cabin and its actions are determined based on the vehicle's condition and the situation of the occupant driving the vehicle, and the agent is able to convey various information and exchange information with the occupant. In this case, the agent is displayed on the display as a personified character and outputs voice, making the vehicle seem more human and lovable.

[0003] For example, Patent Document 1 describes an in-vehicle agent system that places an agent as a three-dimensional character image linked to in-vehicle equipment and provides assistance to the occupant. The technology described in Patent Document 1 has an assist means that allows the three-dimensional character image to move freely within the vehicle cabin and provides assistance at an appropriate position related to the assistance, and a sound generation means for the character image, and can localize the sound image at an appropriate position related to the assistance.

[0004] See Japanese Patent Application Laid-Open No. 2006-284454 ([Abstract])

[0005] However, the technology described in Patent Document 1 has problems such as the driver's gaze being drawn to instructions and dialogue from the agent (their gaze being glued to the vehicle), which can hinder safe driving.

[0006] On the other hand, the surrounding circumstances of the vehicle, including the state of the vehicle, change from moment to moment as the vehicle travels, and the occupant (the driver seated in the driver's seat) is subjected to various psychological and physical stresses (so-called driving load) depending on the surrounding circumstances. If such driving load becomes too large, it may have a negative effect on driving operation, so attempts have been made to estimate the driving load and reflect the driving load in various processes of the in-vehicle system.

[0007] The present invention has been made to solve the above-mentioned problems, and aims to provide a display control device, etc. suitable for an in-vehicle agent system that does not interfere with driving, for example, by changing the representation form of an agent image according to driving load.

[0008] Other objects of the present invention will become apparent to those skilled in the art by referring to the following exemplary aspects and best modes, as well as the accompanying drawings.

[0009] In order to facilitate an understanding of the outline of the present invention, the following examples are given of embodiments according to the present invention.

[0010] The first aspect is a display control device that controls a display device that displays an agent image that can exchange information with a passenger on board a vehicle, and includes a memory unit that stores the agent image, and a control unit that determines the driving load from at least one of the state of the vehicle, the driving environment, or the state of the passenger driving, and changes the representation form of the agent image in accordance with the determined driving load and controls the display on the display device.

[0011] Here, "driving load" refers to the psychological and physical load imposed on the occupant due to the state of the vehicle, the driving environment, the state of the occupant driving the vehicle, etc., for example, the degree of strain required for cognition, judgment, and driving operation. For example, as shown in Fig. 3A, driving load is classified into "A" (high driving load), "B" (medium driving load), and "C" (low driving load), and behavior is defined for each classification. Furthermore, the "representation form of the agent image" refers to, for example, a form in which the agent image is represented as a monochrome geometric image whose shape changes according to the agent's speech, as shown as "A" in Figure 3A, in which only information necessary for driving (information to call attention, navigation information, etc.) is presented; or a form in which the agent image is represented as an anthropomorphized character, as shown as "B" or "C" in Figure 3A, in which the character is represented as an image with gestures such as blinking, nodding, and looking in the direction of the occupant, in addition to the information necessary for driving, for example, entertainment information (tourist information, news, weather, recommendations (information on television, music, movies, etc. that do not affect driving operations), etc.) as shown in Figure 3B is presented.

[0012] In the first aspect, the control unit determines the driving load from at least one of the vehicle state, the driving environment, and the state of the driving occupant, and controls to change the representation of the agent image in accordance with the determined driving load. Therefore, according to the first aspect, for example, by displaying a geometrical image that does not interfere with the occupant's driving and presenting only driving-related information, the occupant can concentrate on driving, and by displaying an anthropomorphic character, the occupant can feel more attached to the vehicle and feel it is more human. In this way, by switching the representation of the agent image depending on the driving load, it is possible to both prevent interference with driving and increase the likability of the agent to the occupant (the degree to which the agent is perceived as human and attached).

[0013] Here, the driving load is determined based on at least one of the vehicle state, the driving environment, and the state of the driving occupant. When determining the driving load based on the vehicle state, it can be estimated from, for example, the level of driving assistance, the level of autonomous driving, the steering angle, the accelerator and brake operation amounts, etc. When determining the driving load based on the driving environment, it can be estimated from map information or detection information from an in-vehicle monitoring device. In the case of map information, it is possible to estimate the driving load from the area in which the vehicle is traveling based on information on areas where the driving load varies, such as urban areas, suburban areas, bypasses, expressways, intersections, and the number of lanes on a road. In the case of detection information from an in-vehicle monitoring device, the driving load can be estimated from surrounding vehicles, obstacles, pedestrians, road width, etc. detected by LiDAR (Light Detection and Ranging) or the like. When determining the driving load based on the state of the occupant, it can be estimated from, for example, biometric information such as heart rate and pulse wave obtained from a biometric information sensor, or facial expressions obtained by image recognition of the occupant's facial image captured by a camera. For example, if the driver is estimated to be in a tense state based on the heart rate, it can be determined that the driving load is high, and if the driver is estimated to be relaxed, it can be determined that the driving load is low.

[0014] In a second aspect dependent on the first aspect, the control unit may, when it is determined that the driving load is high, display the agent image on the display device in a first representation form in which the agent image is geometrically represented, and when it is determined that the driving load is low, display the agent image on the display device in a second representation form in which the agent image is represented by an anthropomorphized character.

[0015] In the second aspect, the control unit switches between displaying the agent image in a first representation style in which the agent is represented geometrically (e.g., polygons, arcs, lines, etc.) and a second representation style in which the agent is represented by an anthropomorphic character, depending on the driving load. Therefore, according to the second aspect, it is possible to prevent interference with driving and to increase the likeability of the agent to the occupant (the degree to which the agent is perceived as a human being that the occupant can feel attached to).

[0016] In a third aspect dependent on the second aspect, when the agent image is displayed on the display device in the second representation form, the control unit may control the display range of the body parts of the person represented by the character in accordance with the driving load.

[0017] In the third aspect, when the control unit displays the agent image in the second representation mode, the control unit controls the display range of the body parts of the person represented by the character in accordance with the driving load. Therefore, according to the third aspect, for example, when the driving load is high, by performing control to reduce the number of body parts displayed (from (b) to (a) in FIG. 5A), the amount of information contained in the agent image is reduced, thereby reducing the possibility of the driver focusing on unnecessary parts and improving safety. Also, for example, the same effect can be obtained by displaying only the upper body parts (from (b) to (a) in FIG. 5B), as shown in FIG. 5B.

[0018] In a fourth aspect dependent on the second aspect, when the agent image is displayed on the display device in the second representation form, the control unit may control the movement of the body parts of the person represented by the character in accordance with the driving load.

[0019] In the fourth aspect, when the control unit displays the agent image in the second representation mode, the control unit controls, for example, the movement of the body parts of the person represented by the character in accordance with the driving load. Therefore, according to the fourth aspect, for example, when the driving load is high, the amount of movement of the agent image (body parts of the character) can be reduced (e.g., from FIG. 6( a) to FIG. 6( b)), thereby reducing the attractiveness of the agent image. Here, "attractiveness" refers to the degree to which the agent image attracts a person's attention. This prevents further unnecessary attention during high driving loads, thereby improving safety. For example, in FIGS. 6( a) and 6( b), this is achieved by reducing the amount of movement of one of the body parts, the arm. However, this is not limited to this; for example, the movement speed of the other arm may be reduced. On the other hand, when the driving load is low, the control unit may change the character's facial expression, for example, by causing the character to smile near the destination. In this case, when the driving load is high, the character's facial expression becomes stiff.

[0020] In a fifth aspect dependent on the second aspect, the control unit may control the display size of the character in accordance with the driving load when displaying the agent image on the display device in the second representation form.

[0021] In the fifth aspect, when the control unit displays the agent image in the second representation mode, for example, the control unit controls the display size of the character in accordance with the driving load. Therefore, according to the fifth aspect, for example, when the driving load is high, the display size of each body part of the agent image can be increased to enable a rough grasp of information, and when the driving load is high, it is assumed that it is difficult to take one's eyes off the road, but when the display size is increased, the rough movement can be grasped even with a "side glance," and therefore, "looking away" can be reduced.

[0022] In a sixth aspect dependent on the second aspect, the control unit may control the degree of confrontation of the character with the occupant in accordance with the driving load when displaying the agent image on the display device in the second representation form.

[0023] In a sixth aspect, when the control unit displays the agent image in the second representation mode, the control unit controls, for example, the degree of facing of the character to the occupant in accordance with the driving load. Here, "facing" refers to the degree to which the agent image (the line of sight of the agent image) faces the occupant (mainly the occupant sitting in the driver's seat), and is defined as, for example, an angle θ shown in FIG. 8( a), where a smaller angle θ indicates a lower degree of facing. According to the sixth aspect, for example, when the driving load is high, the feeling of being watched can be reduced by reducing the facing of the agent image to the occupant. For example, people can feel the gaze of others even when looking at a painting with a person in it, and performing the above-described control can reduce distractions and unnecessary attention.

[0024] In a seventh aspect dependent on the second aspect, the control unit may control the number of color schemes used for the geometric representation in accordance with the driving load when displaying the agent image on the display device in the first representation form.

[0025] In a seventh aspect, when the control unit displays the agent image in the first representation mode, the control unit controls the number of colors used for geometric representation in accordance with the driving load. In the seventh aspect, for example, when the driving load is high, the number of colors used for the agent image is reduced, thereby reducing the possibility that the occupant will find more meaning in the colors than the information presented by the character.

[0026] In an eighth aspect dependent on the second aspect, the control unit may control the degree of the geometric representation in accordance with the driving load when displaying the agent image on the display device in the first representation form.

[0027] In the eighth aspect, when an agent image is displayed in the first representation mode, the order of geometric representation is controlled according to the driving load, for example. For example, when the driving load is high, the order used for the geometric representation of the agent image is controlled to be lowered. For example, as shown in FIG. 9( a), lines are used in the first-order case (high driving load), polygons or circles are used in the second-order case (medium driving load) as shown in FIG. 9( b), and three-dimensional representation is used in the third-order case (low driving load) as shown in FIG. 9( c). Therefore, the lower the order, the less information is used for representation, and the cognitive load can be kept low. On the other hand, in situations where the driving load is low and there is relatively more time, the representation can be made more rich, making the product more appealing.

[0028] In a ninth aspect dependent on the second aspect, the control unit may control dynamic elements of the geometric representation in accordance with the driving load when displaying the agent image on the display device in the first representation form.

[0029] In the ninth aspect, when an agent image is displayed in the first representation mode, for example, dynamic elements of geometric representation are controlled in accordance with the driving load. Therefore, according to the ninth aspect, for example, when the driving load is high, by reducing the dynamic elements (e.g., shape changes and animations) of the geometric representation of the agent image, specifically by reducing the amount of shape change or the update cycle of the dynamic representation, unnecessary attention to the occupant can be reduced, thereby contributing to safe driving.

[0030] In a tenth aspect dependent on the first to ninth aspects, the control unit may switch the display of the agent image between the first representation form and the second representation form at a timing when the occupant is not viewing the agent image or is not interacting with the agent.

[0031] In a tenth aspect, when the control unit switches the agent image between the first representation mode and the second representation mode, it executes the switching at a timing when the occupant is not viewing the agent image or not having a conversation with the agent image. Therefore, according to the tenth aspect, by executing the switching at a timing when the occupant is not viewing the agent image or not having a conversation with the agent image, the amount of change in the image caused by the switching is reduced, so that the occupant does not become aware of the sudden switching.

[0032] Whether or not the occupant is viewing the agent image can be determined, for example, by calculating the gaze direction from the occupant's viewpoint position detected by the viewpoint sensor 305 of the in-vehicle monitoring device 30 shown in Fig. 1 and then calculating the gaze point on the screen of the display device 70 from the obtained gaze direction of the occupant. Alternatively, if the display device 70 is configured with a touch panel, it can also be estimated from actions such as whether or not the screen is operated. On the other hand, with regard to the timing when no conversation is taking place, for example, speech can be estimated by image recognition of the movement of the mouth of the occupant's face image captured by the camera 301 of the in-vehicle monitoring device 30 as shown in Fig. 1.

[0033] In an eleventh aspect dependent on the first to ninth aspects, when the control unit switches the display between the agent images displayed in the first representation form or the agent images displayed in the second representation form due to an increase or decrease in the driving load, the control unit may continuously change the angle of view and display the image on the display device.

[0034] In the eleventh aspect, when switching between the agent images in the first and second representation modes, the control unit performs control to continuously change the angle of view. Therefore, according to the eleventh aspect, for example, a transition may occur between agent images displayed in the second representation mode due to an increase or decrease in driving load. In this case, the angle of view is continuously changed by zooming in and out, for example, as shown in FIG. 10 . For example, when the driving load increases, zoom-in control is performed (transition from FIG. 10( a) to FIG. 10( b)), and when the driving load decreases, zoom-out control is performed (transition from FIG. 10( b) to FIG. 10( a)). This allows the agent image to display an appropriate amount of information depending on the driving load, and prevents the occupant from being aware of the sudden change. The same applies when a transition occurs between agent images displayed in the first representation mode due to an increase or decrease in driving load.

[0035] In a twelfth aspect dependent on the first to eleventh aspects, when the control unit determines the driving load based on the state of the vehicle, the control unit may make the determination based on at least one of the degree of driving assistance, the steering angle, the amount of accelerator operation, or the amount of brake operation.

[0036] In a twelfth aspect, when the control unit determines the driving load based on the vehicle state, the determination is based on at least one of, for example, the level of driving assistance, the steering angle, the accelerator operation amount, or the brake operation amount. Therefore, the vehicle load can be determined from detection information detected by a driving assistance device equipped in a vehicle capable of driving assistance or an on-board monitoring device 30 originally equipped in the vehicle, making it easy and inexpensive to implement. For example, when determining the driving load based on the level of driving assistance provided by an ADAS (Advanced Driver Assistance System), the level of intervention of the driving assistance is scored, and the driving load is determined based on a total score value. In this case, for example, a score of 1 is assigned for lane departure warning, a score of 2 for lane departure correction, a score of 3 for lane keeping, a score of 1 for maintaining the vehicle speed at a set value, a score of 2 for matching the speed to the vehicle ahead, and a score of 3 for automating moving forward / stopping in response to driving conditions such as traffic lights. The driving load is determined to be low when the total score value is 5 or greater (a threshold value). In addition, in the case of autonomous driving, driving load may be judged to be high for foot-free level 1 (driving assistance in only one direction) or below, in which the occupant is the main driver, while driving load may be judged to be low for hands-free level 2 (driving assistance in both vertical and horizontal directions) and eyes-free level 3 (autonomous driving under specific conditions).

[0037] In a thirteenth aspect dependent on the first to eleventh aspects, when the control unit determines the driving load based on the driving environment of the vehicle, the control unit may make the determination from map information having information on an area where the driving load changes, or from environmental information around the vehicle monitored while driving by a sensor mounted on the vehicle.

[0038] In a thirteenth aspect, when the control unit determines the driving load based on the driving environment of the vehicle, the control unit makes the determination based on, for example, map information having information on areas where the driving load changes or environmental information around the vehicle detected while traveling by a sensor mounted on the vehicle. The "map information" here refers to, for example, a digital map stored in the map information DB 200 of the navigation device 20 shown in Fig. 1 or acquired by communicating with an external center (not shown), and the "sensor mounted on the vehicle" refers to, for example, the camera 301, the GPS (Global Positioning System) 302, the behavior sensor 303, the LiDAR 304, etc., of the on-board monitoring device 30 shown in Fig. 1.

[0039] Therefore, according to the thirteenth aspect, for example, when determining the driving load based on map information, the driving load may be determined based on the curvature, frequency, and inclination of the curves on which the vehicle travels. For example, a driving environment in which the vehicle travels on a road with an incline and many sharp curves, such as a mountain pass, may be determined to be high, while a driving environment in which the vehicle travels on a flat expressway without curves may be determined to be low. The driving load may also be determined based on the number of people, pedestrians, bicycles, vehicles, and the like in the vicinity. Furthermore, the driving load may be determined based on the number of buildings, for example, a densely built area, such as a residential area, may be determined to be high, and a building-free area, such as a footpath, may be determined to be low. The driving load may also be determined based on the number of traffic lights, lanes, intersections, crosswalks, signs, and other signs that affect driving operations. For example, when determining the driving load based on information detected by a sensor, the driving load may be determined to be low in a driving environment in which the vehicle travels during the day, and high in a driving environment in which the vehicle travels at night, when more attention is required. As in the case of determining the driving load based on the vehicle state, the driving environment may be scored and the total score may be compared with a threshold value.

[0040] In a fourteenth aspect that is dependent on the first to eleventh aspects, when the control unit determines the driving load based on the state of the occupant driving the vehicle, the control unit may make the determination based on the occupant's biometric information or the occupant's analyzed facial expression monitored by a sensor worn by the occupant or mounted on the vehicle.

[0041] In a fourteenth aspect, when the control unit determines the driving load based on the state of the occupant driving the vehicle, the determination is made from the occupant's biometric information monitored by a sensor or the occupant's analyzed facial expression. For example, the biometric information is determined from the heart rate, brain waves, sweating, etc. In the case of the heart rate, the determination is made from the heart rate within a certain period of time and frequency analysis values. If the heart rate is high, it can be determined that the user is in a state of tension (stress) and the driving load is high. In the case of brain waves, frequency analysis is performed, and if the beta wave band, which indicates concentration and stress, is high, it can be determined that the driving load is high, and if the alpha wave, which indicates relaxation, is high, it can be determined that the driving load is low. Furthermore, if there is a lot of sweating, it can be determined that there is a lot of tension and stress, and the driving load can be determined to be high.

[0042] Furthermore, the determination may be made based on the occupant's movements, rather than on biometric information. In this case, if the occupant is facing forward or looking, the driving load may be determined to be high, and if the occupant is facing in a direction wider than the lane width, the driving load may be determined to be low. Alternatively, the determination may be made based on the steering wheel grip position, and if the occupant grips the steering wheel in a high position, it may be estimated that a steering angle operation is necessary depending on the environment around the vehicle. In this case, the driving load may be determined to be high, and if the occupant grips the steering wheel in a low position, the driving load may be determined to be low. Note that, as in the case of determining based on the vehicle state and driving environment, the above-mentioned occupant state may be scored and the total value may be compared with a threshold value to make the determination.

[0043] A fifteenth aspect is a head-up display device that projects an agent image onto an image plane that is virtually set in front of a vehicle, superimposed on the forward field of view of the vehicle, and includes an image storage unit that displays the agent image, and a control unit that determines the driving load from at least one of the state of the vehicle, the driving environment, or the state of the occupant driving the vehicle, and changes the representation form of the agent image in accordance with the determined driving load and controls the image to be displayed on the image display unit.

[0044] In a fifteenth aspect, the control unit determines the driving load from at least one of the vehicle state, the driving environment, and the state of the driving occupant, and controls the image display unit to change the representation of the agent image in accordance with the determined driving load. Therefore, according to the fifteenth aspect, for example, a head-up display device can be provided that displays only driving-related information in a geometrical manner that does not interfere with the occupant's driving, allowing the occupant to concentrate on driving, and that displays an anthropomorphic character that makes the vehicle appear more human and lovable. In this way, by switching the representation of the agent image and displaying it in accordance with the driving load, it is possible to achieve both preventing interference with driving and increasing the likability of the agent in the occupant (the degree to which the agent appears human and lovable).

[0045] In a sixteenth aspect that is dependent on the fifteenth aspect, the control unit may, when it is determined that the driving load is high, represent the agent image in a color that is close to the target area of ​​the forward field of view on which it is superimposed, and when it is determined that the driving load is low, represent the agent image in a color that conforms to the character or design of the agent image.

[0046] In a sixteenth aspect, when the driving load is determined to be high, the control unit displays the agent image in a color similar to the target area of ​​the forward field of view to be superimposed, and when the driving load is determined to be low, the control unit displays the agent image in a color consistent with the character or design of the agent image. Therefore, according to the sixteenth aspect, the head-up display device displays the agent image superimposed on the vehicle's forward field of view. Therefore, when the driving load is high, the agent image is displayed in the same color as the superimposed area of ​​the forward field of view (e.g., gray if superimposed on the road, or the color of the vehicle if superimposed on the vehicle ahead) so as not to interfere with driving. When the driving load is low, the agent image is displayed in a color consistent with the character and design of the agent, thereby preventing interference with driving and increasing the likeability of the agent. Furthermore, the head-up display device can display the image projected on the image surface according to the display distance (from near to far), so that the image can be superimposed on a corner of a crossroads such as an intersection. When the driving load is low, the agent appears to be standing at the corner, guiding (pointing) the way to turn, thereby increasing the likeability of the agent.

[0047] A seventeenth aspect is a program for a display control device that controls a display device that displays an agent image that can exchange information with a passenger on board a vehicle, and causes a processor of the display control device to execute a process of determining the driving load from at least one of the state of the vehicle, the driving environment, or the state of the passenger driving the vehicle, and a process of changing the representation form of the agent image in accordance with the determined driving load and controlling the display on the display device.

[0048] In the seventeenth aspect, the processor of the display control device sequentially reads and executes programs stored in a partial area of ​​the storage unit 42 shown in Fig. 1, for example, to provide a geometric display that does not interfere with the occupant's driving and present only driving-related information, thereby enabling the occupant to concentrate on driving, and to use an anthropomorphized character representation to make the vehicle seem more human and lovable. In this way, by switching the representation form of the agent image depending on the driving load, it is possible to achieve both prevention of driving interference and the likability of the agent to the occupant (the degree to which the agent seems to be human and lovable).

[0049] An eighteenth aspect is an in-vehicle agent system that places an agent image within the vehicle cabin space to provide driving assistance to an occupant, and includes: a head-up display device that projects the agent image onto an image plane that is virtually set in front of the vehicle, superimposing the agent image onto the front field of view of the vehicle; a display control device that determines the driving load from at least one of the state of the vehicle, the driving environment, or the state of the occupant driving the vehicle, and controls the head-up display device to change the representation form of the agent image in accordance with the determined driving load; and a driving assistance device that cooperates with the display control device to make the agent image appear within the vehicle cabin, communicates information via the agent image, and assists the occupant in driving the vehicle by exchanging information with the occupant.

[0050] In an eighteenth aspect, the display control device cooperates with the driving assistance device to determine the driving load based on at least one of the vehicle state, the driving environment, and the state of the occupant driving the vehicle, and controls the display of the agent image on the head-up display device to change the representation of the agent image according to the determined driving load, and supports the occupant in driving the vehicle by communicating information through the agent image and exchanging information with the occupant. Therefore, according to the eighteenth aspect, for example, an in-vehicle agent system can be provided that displays the agent image geometrically so as not to interfere with the occupant's driving and presents only driving-related information, allowing the occupant to concentrate on driving, and that displays the agent image as an anthropomorphic character, making the occupant feel more attached to the vehicle and more human-like. In this way, by switching the representation of the agent image according to the driving load, it is possible to achieve both preventing interference with driving and increasing the occupant's likeability (the degree to which the agent image is perceived as human-like and lovable).

[0051] Those skilled in the art will easily understand that the exemplified embodiments according to the present invention can be further modified without departing from the spirit of the present invention.

[0052] FIG. 1 is a block diagram illustrating an example of the configuration of an in-vehicle agent system to which a display control device according to an embodiment of the present invention is applied. FIG. 2 is a flowchart illustrating basic processing steps of the display control device according to an embodiment of the present invention. FIG. 3A is a table used to explain the behavior of an agent generated by a display control device according to an embodiment of the present invention. FIG. 3B is a diagram illustrating an example display of entertainment information generated by a display control device according to an embodiment of the present invention. FIG. 4 is a flowchart illustrating detailed steps of the "driving load determination process" in FIG. 2. FIG. 5A is a diagram illustrating an example of an agent image (1) represented by an anthropomorphized character generated by a display control device according to an embodiment of the present invention. FIG. 5B is a diagram illustrating an example of an agent image (2) represented by an anthropomorphized character generated by a display control device according to an embodiment of the present invention. FIG. 6 is a diagram illustrating an example of an agent image (3) represented by an anthropomorphized character generated by a display control device according to an embodiment of the present invention. FIG. 7 is a diagram illustrating an example of an agent image (4) represented by an anthropomorphized character generated by a display control device according to an embodiment of the present invention. FIG. 8 is a diagram illustrating the confrontation between an occupant and an agent displayed on a display device. FIG. 9 is a diagram illustrating geometric representations that differ depending on the order generated by a display control device according to an embodiment of the present invention. 10 is a screen transition diagram used to explain switching between agent images by a display control device according to an embodiment of the present invention. FIG. 11 is a block diagram showing an example of the configuration of a head-up display device according to an embodiment of the present invention.

[0053] The best mode described below is used to facilitate understanding of the present invention, and therefore, those skilled in the art should be aware that the present invention is not unduly limited by the mode described below (hereinafter referred to as the present mode).

[0054] 1 is a block diagram showing an example of the configuration of an in-vehicle agent system 100 to which a display control device 40 of this embodiment is applied. The in-vehicle agent system 100 is a system that makes an agent appear inside the vehicle cabin, whose actions are determined based on the status of the vehicle and the occupant driving the vehicle, and that transmits various information through the agent and is also capable of exchanging information with the occupant (the driver sitting in the driver's seat).

[0055] According to FIG. 1, the in-vehicle agent system 100 includes a driving assistance device 10, a navigation device 20, an in-vehicle monitoring device 30, a biometric information sensor 90, a display control device 40 of this embodiment, and an audio input / output control device 50, all of which are configured to be capable of two-way communication via an I / O interface 60.

[0056] The driving assistance device 10 has at least one of a driving assistance function using ADAS such as a Lane Keeping Assist System (LKAS) or an Adaptive Cruise Control System (ACC) that assists the occupant in driving the vehicle, and an automatic driving function that can act as a substitute for the occupant in driving the vehicle. The driving assistance device 10 recognizes or detects the driving environment and behavior ahead of the vehicle based on information acquired from the in-vehicle monitoring device 30, and controls the vehicle (including the drive system, braking system, and steering system, not shown) based on the analysis results of the information obtained. In addition, the driving assistance device 10 can transfer the analysis results of the information to the navigation device 20 and the display control device 40 via the I / O interface 60. In addition, the driving assistance device 10 can cooperate with the display control device 40 to display an agent image in the vehicle cabin, communicate information through the agent image, and exchange information with the occupant to assist the occupant in driving the vehicle.

[0057] The navigation device 20 uses a Global Navigation Satellite System (GNSS) such as GPS and a gyro sensor to obtain map information from map information or via wireless communication with the outside of the vehicle, and provides guidance to facilities around the vehicle, route guidance, etc. The navigation device 20 transfers a signal prompting display output based on this guidance information at an appropriate timing to the display control device 40 of this embodiment via the I / O interface 60.

[0058] The navigation device 20 has a map information database (map information DB 200). The map information DB 200 can acquire and store the latest map information, for example, by communicating with an external center (not shown) via a V2X (Vehicle to X) type communication system (not shown). The map information stored in the map information DB 200 is mapping data that has been digitized to represent the vehicle's driving environment. The mapping data is preferably digital data of a particularly high-precision dynamic map. Here, a "dynamic map" refers to a digital map that combines a huge amount of dynamic information that changes every moment, such as traffic regulations, construction information, accidents, congestion, pedestrians, and traffic lights, with static information such as high-precision three-dimensional position information (road surface information, slope information, three-dimensional structures).

[0059] The on-board monitoring device 30 is a set of sensors necessary for recognizing the surrounding driving environment, including the area ahead of the vehicle, and includes a camera 301, a GPS 302, a behavior sensor 303, a LiDAR 304, a viewpoint sensor 305, etc. The camera 301 captures at least the vehicle's forward field of view (actual view) and the occupant's eyes (pupil image), and the LiDAR 304 uses near-infrared light, visible light, and ultraviolet light to irradiate, for example, an obstacle or the like in front of the vehicle photographed by the camera 301, capture the reflected light with an optical sensor, and determine the distance to the obstacle based on the time difference. The behavior sensor 303 includes an IMU (Inertial Measurement Unit), a vehicle speed sensor, etc. The IMU measures the vehicle's driving situation and attitude (acceleration [m / s]) using a three-axis acceleration sensor and a three-axis angular velocity sensor (gyro sensor). ２The viewpoint sensor 305 can detect the viewpoint position of the occupant with high accuracy by recognizing the image of the eye (pupil image) of the occupant sitting in the driver's seat of the vehicle captured by the camera 301.

[0060] The biometric information sensor 90 is a wearable device that can acquire the occupant's heart rate and breathing information using, for example, radio wave sensing technology, and can detect periodic body movement components such as breathing rate, heart rate, and pulse rate fluctuations by analyzing the radio waves reflected from the occupant.

[0061] The display control device 40 of this embodiment can display an agent image that can exchange information with a vehicle occupant on a connected display device 70. The display device 70 may be, for example, a head-up display device that projects an agent image onto an image plane virtually set in front of the vehicle, superimposed on the vehicle's forward field of view, or alternatively, a center information display (CID) provided in the center of the vehicle or a head-mounted display device (HMD device) mounted on the head of the occupant.

[0062] The display control device 40 of this embodiment includes a control unit 41 and a storage unit 42. The control unit 41 determines the driving load from at least one of the vehicle state, the driving environment, and the state of the driver, and controls the display device 70 to change the representation form of the agent image according to the determined driving load.

[0063] The "representation form of the agent image" refers to, for example, a first representation form in which the agent image is represented as a monochrome geometric image whose shape changes depending on the agent's speech, as shown by "A" in Figure 3A, and only information necessary for driving (such as information to call attention or navigation information) is presented; or a second representation form in which the agent image is represented as an anthropomorphized character, as shown by "B" or "C" in Figure 3A, and the character is represented as an image that moves with gestures such as blinking, nodding, and looking in the direction of the occupant, and in addition to the information necessary for driving, entertainment information (tourist information, news, weather, recommendations (information on television, music, movies, etc. that do not affect driving operations), etc.) as shown in Figure 3B is presented.

[0064] Here, the control unit 41 determines the driving load from at least one of the vehicle state, the driving environment, and the state of the driving occupant. When the control unit 41 determines the driving load from the vehicle state, the driving load is estimated from, for example, the degree of driving assistance, the level of autonomous driving, the steering angle, the accelerator and brake operation amounts, etc. When the control unit 41 determines the driving load from the driving environment, the driving load is estimated from map information and detection information from the in-vehicle monitoring device 30. In the case of map information, the driving load can be estimated from the area in which the vehicle is traveling based on information on areas where the driving load changes, such as urban areas, suburban areas, bypasses, expressways, intersections, and the number of lanes on roads. In the case of detection information from the in-vehicle monitoring device 30, the driving load can be estimated from surrounding vehicles, obstacles, pedestrians, road width, etc. detected by the LiDAR 304 or the like.

[0065] Furthermore, when the control unit 41 determines the driving load from the state of the occupant, the driving load can be estimated from, for example, biological information such as heart rate and pulse wave obtained from the biological information sensor 90 worn by the occupant, or facial expression obtained by image recognition of the occupant's facial image captured by the camera 301 of the in-vehicle monitoring device 30. For example, if a state of tension is estimated from the heart rate or the occupant's facial expression, the driving load can be determined to be high, and if the occupant is estimated to be relaxed, the driving load can be determined to be low.

[0066] The storage unit 42 is a memory that is allocated a program area and a work area and that may be implemented with, for example, static RAM, dynamic RAM, flash memory, etc. Here, the program area stores a program for the display control device 40 that controls the display device 70 that displays an agent image that enables information exchange with a vehicle occupant, and the work area stores information generated in the execution process of the above-mentioned program, agent images, etc.

[0067] Furthermore, when the control unit 41 determines that the driving load is high, it can display the agent image on the display device 70 in a first representation form in which the agent image is geometrically represented, and when it determines that the driving load is low, it can display the agent image on the display device 70 in a second representation form in which the agent image is represented by an anthropomorphized character.

[0068] Furthermore, when the agent image is displayed on the display device 70 in the second representation mode, the control unit 41 can control the display range of the body parts of the person represented by the character in accordance with the driving load. For example, when the driving load is high, the amount of information contained in the agent image can be reduced by performing control to reduce the number of body parts to be displayed ((b) → (a) in FIG. 5A), or, for example, as shown in FIG. 5B, only the upper body parts can be displayed ((b) → (a) in FIG. 5B).

[0069] Furthermore, when the agent image is displayed on the display device 70 in the second representation format, the control unit 41 can control the movement of the body parts of the person represented by the character in accordance with the driving load. For example, when the driving load is high, control is performed to reduce the amount of movement of the agent image (body parts of the character) (for example, from FIG. 6(a) to FIG. 6(b)).

[0070] Furthermore, when the agent image is displayed on the display device 70 in the second representation mode, the control unit 41 can control the display size of the character in accordance with the driving load. For example, when the driving load is high, control is performed to increase the display size of each body part of the agent image.

[0071] Furthermore, when displaying the agent image on the display device 70 in the second representation mode, the control unit 41 can control the degree of facing of the character to the occupant in accordance with the driving load. Here, "facing" refers to the degree to which the agent image (the line of sight of the agent image) faces the occupant (mainly the occupant seated in the driver's seat). For example, this is represented by the angle θ shown in FIG. 8A , where a smaller angle θ indicates a lower degree of facing. For example, when the driving load is high, control is performed to reduce the facing of the agent image to the occupant. It is also possible to avoid eye contact, and by not directing the line of sight of the agent image, which is represented by a personified character, toward the occupant, the distance between the occupant and the agent can be reduced or eliminated. To this end, it is necessary to control facing not only based on the position of the occupant's eyes, but also based on the position of the driver's seat where the occupant is seated and the position of the occupant's head.

[0072] Here, the head and gaze directions of the occupants are measured by image recognition from images or videos of the occupants captured by the camera 301 of the in-vehicle monitoring device 30. Alternatively, the head position of the occupants can be estimated from the position of the driver's seat or headrest. Furthermore, if the display device 70 is configured with a touch panel, it is possible to determine that the occupants are looking toward the screen if they are operating the touch panel. Furthermore, when the driving load is high, control may be performed assuming that the occupants are facing forward.

[0073] Furthermore, when the agent image is displayed on the display device 70 in the first representation mode, the control unit 41 can control the number of colors used for geometric representation in accordance with the driving load. For example, when the driving load is high, the control unit 41 performs control to reduce the number of colors used for the agent image.

[0074] Furthermore, when the agent image is displayed on the display device 70 in the first representation format, the control unit 41 can control the order of geometric representation in accordance with the driving load. For example, when the driving load is high, the control is performed to lower the order used for the geometric representation of the agent image (line for first order, polygon or circle for second order, and three-dimensional representation for third order).

[0075] Furthermore, when the agent image is displayed on the display device 70 in the first representation mode, the control unit 41 can control the dynamic elements of the geometric representation in accordance with the driving load. For example, when the driving load is high, control is performed to reduce the dynamic elements (e.g., shape changes and animations) of the geometric representation of the agent image.

[0076] Furthermore, when switching the agent image, the control unit 41 can change from the first representation mode to the second representation mode or from the second representation mode to the first representation mode at a timing when the occupant is not viewing or interacting with the agent image. Whether the occupant is viewing the agent image can be estimated, for example, by calculating the gaze direction from the occupant's viewpoint position detected by the viewpoint sensor 305 of the in-vehicle monitoring device 30 and then calculating the gaze point on the screen of the display device 70 from the obtained gaze direction (see, for example, WO 2012 / 077713). Alternatively, if the display device 70 is configured as a touch panel, it can be estimated from the occupant's actions, such as whether or not they operate the screen.

[0077] On the other hand, the timing when the occupant is not conversing with the agent can be estimated by image recognition of the movement of the mouth of the occupant's face image captured by the camera 301 of the in-vehicle monitoring device 30. In addition, in order to acquire voice data using the voice input / output device 80 (microphone) and distinguish it from a conversation with a fellow passenger, if a fellow passenger is on board, the presence or absence of a conversation with the fellow passenger may be measured in a similar manner, and if the driver and the fellow passenger are speaking alternately, it may be determined that the agent and the occupant (the occupant sitting in the driver's seat) are not conversing.

[0078] Furthermore, when switching between agent images displayed in the first representation mode or between agent images displayed in the second representation mode, the control unit 41 can switch (change) the agent images by continuously changing the angle of view and display them on the display device 70. For example, a transition may occur between agent images displayed in the second representation mode due to an increase or decrease in driving load. In this case, the control unit 41 performs control to cause the transition by continuously changing the angle of view, for example, by zoom-in control, zoom-out control, etc.

[0079] Furthermore, when determining the driving load based on the state of the vehicle, the control unit 41 can make the determination based on at least one of the level of driving assistance, the steering angle, the accelerator operation amount, or the brake operation amount. For example, when determining the driving load based on the level of driving assistance of the ADAS, the level of intervention of the driving assistance can be scored, and the driving load can be determined based on the total score value.

[0080] For example, in driving assistance, a score of 1 is given for lane departure warning, a score of 2 for lane departure correction, a score of 3 for lane keeping, a score of 1 for maintaining vehicle speed at a set value, a score of 2 for matching speed to the vehicle ahead, and a score of 3 for automating moving forward / stopping in response to driving conditions such as traffic lights, and if the total score is 5 or more (threshold), the driving load can be determined to be low. Also, in the case of autonomous driving, a foot-free level 1 (one-way driving assistance only) or lower in which the occupant is the main driver may be determined to be a high driving load, and a hands-free level 2 (longitudinal and lateral driving assistance) and an eyes-free level 3 (autonomous driving under specific conditions) may be determined to be a low driving load.

[0081] Furthermore, when determining the driving load based on the driving environment of the vehicle, the control unit 41 can make the determination based on map information containing information on areas where the driving load changes, or environmental information around the vehicle monitored by sensors mounted on the vehicle while the vehicle is traveling. Note that "map information" refers to, for example, a digital map stored in the map information DB 200 of the navigation device 20 or acquired by communicating with an external center (not shown). Furthermore, "sensors mounted on the vehicle" refer to, for example, the camera 301, GPS 302, behavior sensor 303, LiDAR 304, etc. of the on-board monitoring device 30.

[0082] For example, when determining the driving load from map information, the driving load may be determined based on the curvature, frequency, and inclination of curves, etc., and a driving environment where the vehicle is traveling on a road with an incline and many sharp curves, such as a mountain pass, may be determined to be high, while a driving environment where the vehicle is traveling on a flat expressway without curves may be determined to be low. The driving load may also be determined based on the number of people, pedestrians, bicycles, vehicles, etc. in the vicinity. Furthermore, the driving load may be determined based on the number of buildings, for example, a densely built area, such as a residential area, may be determined to be high, and a building-free area, such as a footpath, may be determined to be low. The driving load may also be determined based on the number of traffic lights, lanes, intersections, crosswalks, signs, and other signs that affect driving operations. Furthermore, when determining the driving load from information detected by a sensor, a driving load may be determined to be low in a driving environment where the vehicle is traveling during the day, and high in a driving environment where the vehicle is traveling at night, when more attention is required. As in the case of determining the driving load based on the vehicle condition, the driving environment may be scored and the total score may be compared with a threshold value.

[0083] Furthermore, when determining the driving load based on the state of the occupant driving the vehicle, the control unit 41 can make the determination based on the occupant's biometric information monitored by a sensor mounted on the vehicle or worn by the occupant, or the occupant's facial expression analyzed. For example, the biometric information is determined based on the heart rate, brain waves, sweating, etc. measured by the biometric information sensor 90 shown in FIG. 1 . In the case of the heart rate, the determination is made based on the heart rate and frequency analysis value within a certain period of time. If the heart rate is high, the control unit 41 determines that the user is in a state of tension (stress) and determines that the driving load is high. In the case of brain waves, frequency analysis is performed. If the beta wave band, which indicates concentration and stress, is high, the driving load is determined to be high, and if the alpha wave band, which indicates relaxation, is high, the driving load is determined to be low. Furthermore, if the occupant is sweating a lot, the control unit 41 determines that the user is in a state of tension and stress and determines that the driving load is high.

[0084] Furthermore, rather than relying on biometric information, the determination may be made based on the movement of the occupant captured by the camera 301 of the in-vehicle monitoring device 30. In this case, if the occupant is facing forward or looking, the driving load may be determined to be high, and if the occupant is facing in a direction wider than the lane width, the driving load may be determined to be low. Alternatively, the determination may be made based on the grip position of the steering wheel, and if the occupant is gripping the steering wheel in a high position, it may be estimated that a steering angle operation is necessary depending on the environment around the vehicle, and in this case, the driving load may be determined to be high, and if the occupant is gripping the steering wheel in a low position, the driving load may be determined to be low. Note that, as in the case of determining based on the vehicle state and driving environment, the above-mentioned state of the occupant may be scored and the total value may be compared with a threshold value to make the determination.

[0085] For this reason, the control unit 41 is configured to include a vehicle information acquisition unit 411, an environmental information acquisition unit 412, an occupant information acquisition unit 413, a driving load determination unit 414, an image switching unit 415, an image generation unit 416, and a display control unit 417.

[0086] The vehicle information acquisition unit 411 acquires, for example, captured images, position information, vehicle behavior, obstacle detection information, etc. detected by the camera 301, GPS 302, behavior sensor 303, and LiDAR 304 of the in-vehicle monitoring device 30, and outputs the information to the display control device 40 via the I / O interface 60. Furthermore, the environmental information acquisition unit 412 acquires, for example, map information held by the navigation device 20 (map information DB 200) and information on surrounding vehicles, obstacles, pedestrians, road width, etc. detected by the LiDAR 304 of the in-vehicle monitoring device 30, and outputs the information to the display control device 40 via the I / O interface 60.

[0087] The occupant information acquisition unit 413 acquires, for example, the facial expression of the occupant photographed by the camera 301 of the in-vehicle monitoring device 30, or biometric information such as the occupant's heart rate and pulse wave measured by the biometric information sensor 90 worn by the occupant, and outputs the information to the display control device 40 via the I / O interface 60.

[0088] The driving load determination unit 414 determines the driving load from at least one of the vehicle state, the driving environment, and the state of the driver. When determining the driving load from the vehicle state, the driving load can be estimated from, for example, the degree of driving assistance, the level of autonomous driving, the steering angle, the accelerator and brake operation amounts, etc. When determining the driving load from the driving environment, the driving load can be estimated from map information (the map information DB 200 of the navigation device 20) or detection information acquired by the in-vehicle monitoring device 30. When map information is used, the driving load can be estimated from the area in which the vehicle is traveling based on information on areas where the driving load changes, such as urban areas, suburban areas, bypasses, expressways, intersections, and the number of lanes on roads. When detection information from the in-vehicle monitoring device 30 is used, the driving load can be estimated from, for example, surrounding vehicles, obstacles, pedestrians, road width, etc. detected by the LiDAR 304. Furthermore, when determining the driving load from the state of the occupant, the driving load can be estimated from, for example, biological information such as heart rate and pulse wave obtained from the biological information sensor 90, or facial expression obtained by image recognition of the occupant's face image captured by the camera 301. For example, if the occupant is estimated to be in a tense state based on the heart rate, the driving load can be determined to be high, and if the occupant is estimated to be relaxed, the driving load can be determined to be low.

[0089] When switching the display of the agent image between a geometric representation (first representation form) and a personified character representation (second representation form), the image switching unit 415 can execute the switching at a timing when the occupant is not viewing the agent image or not interacting with the agent. Furthermore, when switching the display between agent images displayed in the first representation form or the second representation form due to an increase or decrease in driving load, the image switching unit 415 can continuously change the angle of view and display the image on the display device 70.

[0090] The image generation unit 416 generates display information including an agent image and controls writing of the information in synchronization with the update cycle of the display information to a VRAM (Video RAM) area allocated to a partial area of ​​the storage unit 42. The display control unit 416 also reads out the display information written to the VRAM area in synchronization with the display timing of the display device 70 and outputs it to the display device 70, thereby obtaining a desired display including an agent image.

[0091] In order to perform the above-described control, the control unit 41 is equipped with, for example, a processor having built-in memory (ROM / RAM) or external memory, and the processor reads and sequentially executes programs recorded in the memory (which may be the storage unit 42), thereby executing the functions of the vehicle information acquisition unit 411, the environmental information acquisition unit 412, the occupant information acquisition unit 413, the driving load determination unit 414, the image switching unit 415, the image generation unit 416, and the display control unit 417. That is, the control unit 41 determines the driving load from at least one of the vehicle state, the driving environment, and the state of the driving occupant, and controls the display of the agent image on the display device 70 to change the representation format in accordance with the determined driving load. Furthermore, at least some of the above-described functions can be realized not by a processor but by hardware such as an FPGA (Field Programmable Gate Array) or a logic circuit.

[0092] The voice input / output control device 50 controls the interface between the connected voice input / output devices 80 such as microphones and speakers and the I / O interface 60, and performs "voice synthesis processing" to mechanically create the content of speech by the agent and output it to the speaker, and "voice recognition processing" to mechanically understand the content of speech by the occupants collected through the microphone and transmit it to the agent.

[0093] (Operation of the Embodiment) FIG. 2 is a flowchart showing the basic processing procedure of the display control device 40 according to this embodiment, FIG. 3A is a table used to explain the behavior of an agent, FIG. 3B is a diagram showing an example of the display of entertainment information, and FIG. 4 is a flowchart showing the detailed procedure of the "driving load determination process" shown in FIG. 2. Also, FIGS. 5A, 5B, 6, and 7 are diagrams showing examples (1) to (4) of agent images represented by anthropomorphized characters (second representation form), respectively. Also, FIG. 8 is a diagram used to explain the confrontation between the occupant and the agent displayed on the display device 70, FIG. 9 is a diagram used to explain geometric representations that differ depending on the order, and FIG. 10 is a screen transition diagram used to explain switching between agent images. The operation of the display control device 40 according to this embodiment shown in FIG. 1 will now be described in detail with reference to FIGS. 2 to 10.

[0094] First, in the display control device 40 (control unit 41) of this embodiment, the vehicle information acquisition unit 411 acquires vehicle state information, the environmental information acquisition unit 412 acquires driving environment information, and the occupant information acquisition unit 413 acquires occupant state information, and outputs the acquired information to the driving load determination unit 414 (steps ST101 to ST103). Here, the vehicle state information is, for example, the degree of driving assistance intervention or the level of autonomous driving acquired from the driving assistance device 10, or the steering angle, accelerator, and brake operation amounts detected by the behavior sensor 303 of the in-vehicle monitoring device 30. Furthermore, the driving environment information is, for example, information on surrounding vehicles, obstacles, pedestrians, road width, etc. detected or output by the LiDAR 304 of the in-vehicle monitoring device 30 or the graphic information DB 200 of the navigation device 20. In addition, the occupant's condition information is, for example, biometric information such as heart rate and pulse wave obtained from the biometric information sensor 90, or information regarding the occupant's facial expression obtained by image recognition of the occupant's facial image captured by the camera 301 of the in-vehicle monitoring device 30.

[0095] The driving load determination unit 414 determines the driving load on the occupant driving the vehicle from at least one of the acquired vehicle state information, driving environment information, and driving occupant state information (step ST104). When determining the driving load from the vehicle state information, for example, the driving load is estimated from the level of driving assistance, the level of autonomous driving, the steering angle, the accelerator and brake operation amount, etc. For example, when the driving assistance device 10 performs driving assistance by ADAS and the driving load is determined based on the level of driving assistance intervention, for example, as shown in FIG. 4 , when the driving assistance by the driving assistance device 10 intervenes (step ST201 “A”), the control unit 41 (driving load determination unit 414) scores the level of the intervention (step ST202) and determines the driving load based on the total score value (step ST203).

[0096] Here, when the driving load determination unit 414 scores the degree of intervention of driving assistance, the control unit 41 (driving load determination unit 414) assigns, for example, a score of 1 for lane departure warning, a score of 2 for lane departure correction, a score of 3 for lane keeping, a score of 1 when the vehicle speed is maintained at a set value, a score of 2 when the speed is matched to that of the vehicle ahead, and a score of 3 when automation of moving forward / stopping in response to the driving environment such as traffic lights is performed. Then, the driving load determination unit 414 determines that the driving load is low (step ST204) when the total score value is, for example, 5 or more (threshold value) (step ST203 "YES"), and determines that the driving load is high (step ST205) when the total score is less than 5 (step ST203 "NO").

[0097] On the other hand, if it is determined in step ST201 that autonomous driving is being performed (step ST201 "B"), the driving load determination unit 414 determines whether the driving load is at or below level 1 (driving assistance in only one direction) of feet-free driving, in which the occupant is the main driver (step ST206). If it is determined that the driving load is at or below level 1 (step ST206 "YES"), the driving load determination unit 414 determines that the driving load is high (step ST207). If it is at level 2 of hands-free driving (driving assistance in both longitudinal and lateral directions) or level 3 of eyes-free driving (autonomous driving under specific conditions) (step ST206 "NO"), the driving load determination unit 414 determines that the driving load is low (step ST208).

[0098] When the driving load determination unit 414 determines the driving load from driving environment information, it can estimate the driving load from map information (acquired from the map information DB 200 of the navigation device 20) or various detection information obtained by the in-vehicle monitoring device 30. When map information is used, the driving load can be estimated from the area in which the vehicle is traveling based on information on areas where the driving load varies, such as urban areas, suburban areas, bypasses, expressways, intersections, and the number of lanes on a road. When detection information from the in-vehicle monitoring device 30 is used, the driving load can be estimated from, for example, surrounding vehicles, obstacles, pedestrians, road width, and the like detected by the LiDAR 304. When the driving load determination unit 414 determines the driving load from occupant state information, it can estimate the driving load from, for example, biometric information such as heart rate and pulse wave acquired from the biometric information sensor 90, or facial expressions obtained by image recognition of facial images of the occupant captured by the camera 301. For example, if the driving load is estimated to be high based on the heart rate, it can be determined that the driving load is high, and if the driving load is estimated to be relaxed, it can be determined that the driving load is low.

[0099] The explanation will be returned to the basic processing procedure of the display control device 40 of this embodiment shown in Fig. 2. When the vehicle load is determined as described above and the driving load determination unit 414 determines that the driving load is high ("YES" in step ST105), the control unit 41 (image generation unit 416) generates an agent image and writes it into the VRAM area of ​​the storage unit 42, and the display control unit 416 reads out the agent image from the VRAM area in synchronization with the display timing and outputs it to the display device 70, thereby obtaining a desired display. The agent image generated in this case and displayed on the display device 70 is, for example, a geometric representation (first representation form) of the agent image, as shown in the upper part of the table in Fig. 3A. In this case, the agent image is represented in a single color with a relatively simple pattern whose shape changes in response to the agent's utterance (step ST106).

[0100] On the other hand, if the driving load determination unit 414 determines that the driving load is not high (step ST105 "NO"), the control unit 41 (image generation unit 416) generates an agent image and writes it to the VRAM area of ​​the storage unit 42. The display control unit 416 then reads the agent image from the VRAM area in synchronization with the display timing and outputs it to the display device 70 to obtain the desired display. In this case, the agent image generated and displayed on the display device 70 is represented, for example, as an anthropomorphized character (second expression form) as shown in the lower part of the table in FIG. 3A. The character may be represented with gestures, such as blinking, nodding, and looking in the direction of the occupant while speaking, to express linguistic communication (step ST107). The background may also be changed depending on the character's emotional changes or the content. Furthermore, in addition to information necessary for driving, entertainment information (tourist information, news, weather, recommendations (information on television, music, movies, etc. that do not affect driving operation), etc., may also be presented, as shown in FIG. 3B.

[0101] As shown in the interruption of FIG. 3A , in addition to the "high" and "low" driving loads, there may be multiple modes such as a "medium" mode in which the character performs actions with little change, such as blinking, nodding, and turning toward the passenger, and the character's communication expression may be switched in stages depending on the driving load level.

[0102] Figures 5A, 5B, 6(a) and 6(b), and 7(a) and 7(b) show examples of agent images (1) to (4) represented by personified characters generated by the control unit 41 (image generation unit 416).

[0103] In the agent image (1) shown in FIG. 5A , when the driving load determination unit 414 determines that the driving load is high (here, the driving load is (a) > (b)), the image generation unit 416 controls the generation of an agent image by reducing the number of body parts displayed. Accordingly, the display control unit 417 controls the transition of the display content from (b) of FIG. 5A to that shown in (a) of FIG. 5A on the display device 70. This reduces the amount of information contained in the agent image, reduces the possibility of the driver focusing on unnecessary parts, and enhances safety. Incidentally, in (b) of FIG. 5A , the number of body parts (display range) of the character is "10," but in (a) of FIG. 5A , it is reduced to six, thereby reducing the amount of information. Furthermore, for example, the same effect can be achieved by displaying only the upper body parts (transition from (b) to (a) of FIG. 5B ), as shown in the agent image (2) shown in FIG. 5B . Incidentally, in (b) of FIG. 5B , the number of body parts (display range) is reduced from "6" to "3," thereby reducing the amount of information.

[0104] In addition, in the agent image (3) shown in FIGS. 6( a) and 6(b), when the driving load determination unit 414 determines that the driving load is high (here, the driving load is (a) > (b)), the image generation unit 416 controls the image generation by reducing the amount of movement (indicated by arrows) of the agent image (body parts of the character). Accordingly, the display control unit 417 controls the display content to transition from FIG. 6( a) to FIG. 6(b) and display it on the display device 70. This reduces the attractiveness of the agent image. Here, "attractiveness" refers to the degree to which the image attracts a person's attention. This prevents further unnecessary attention from being generated during high driving loads, thereby improving safety. Note that in FIGS. 6( a) and 6(b), this is achieved by reducing the amount of movement of the arm, which is one of the body parts. However, this is not limited to this. For example, the movement speed of the other arm may be reduced. On the other hand, when the driving load is low, the character's facial expression may be changed, for example, by smiling when the character is near the destination. In this case, when the driving load is high, the character's facial expression becomes stiff.

[0105] In addition, in the agent image (4) shown in Figures 7(a) and (b), the image generation unit 416 controls the image generation by changing the display size of the character according to the driving load, for example, so that when the driving load is high (here, the driving load is (a) > (b)), the display size of each body part of the agent image is increased (Figure 7(b) → Figure 7(a)), making it possible to grasp the general information. In this case, it is assumed that it is difficult to take your eyes off the road, but if the display size is increased, the general movement can be grasped even with a "side glance," and therefore it is possible to reduce "looking away."

[0106] In addition, when the image generation unit 416 generates and displays an agent image using an anthropomorphized character (second representation), the degree of facing between the character and the occupant may be controlled according to the driving load. Here, "facing" refers to the degree to which the agent image AG (the line of sight of the agent image) and the occupant DR (primarily the occupant seated in the driver's seat) face each other (the dashed line in FIG. 6), as shown in FIG. 8, for example. This is indicated by the angle θ shown in FIG. 8(a), where the smaller the angle θ, the lower the degree of facing. (FIG. 8(b) shows a state in which the angle θ is 0°, resulting in complete facing.) For example, when the driving load is high, the feeling of being watched can be reduced by reducing the facing of the agent image to the occupant. For example, even when looking at a painting with a person on it, people may feel gazes. Therefore, the above-described control can reduce distractions and unnecessary attention. In FIGS. 8(a) and 8(b), 70a denotes the display surface 70a of the display device 70.

[0107] On the other hand, when the image generation unit 416 displays an agent image generated in a geometric shape (first representation mode) on the display device 70, the number of colors used for the geometric representation may be controlled in accordance with the driving load. In this case, for example, when the driving load is high, the number of colors used for the agent image is reduced, thereby reducing the possibility that the occupant will find more meaning in the information presented by the character than what is conveyed by the colors.

[0108] Furthermore, when the image generation unit 416 generates an agent image using a geometric shape (first representation form) and the display control unit 417 displays it on the display device 70, the order of the geometric representation may be controlled according to the driving load. In this case, when the driving load is high, the order used for the geometric representation of the agent image is controlled to be lowered. For example, as shown in FIG. 9( a), in the first-order case (high driving load), a line (waveform) is used as a motif; as shown in FIG. 9( b), in the second-order case (medium driving load), a polygon or circle is used as a motif; and as shown in FIG. 9( c), in the third-order case (low driving load), a three-dimensional shape is used as a motif. Therefore, the lower the order, the less information is used for the representation, thereby reducing the cognitive load. On the other hand, in a scene where the driving load is low and there is relatively more time, the representation can be made more rich, thereby making the product more appealing.

[0109] Furthermore, when the image generation unit 416 generates an agent image in a geometric shape (first representation form) and the display control unit 417 displays it on the display device 70, for example, the dynamic elements of the geometric representation may be controlled in accordance with the driving load. In this case, for example, when the driving load is high, the dynamic elements of the geometric representation of the agent image (for example, shape changes and animations) can be reduced, specifically, the amount of shape change can be reduced, the update cycle of the dynamic representation can be reduced, or other measures can be taken to reduce unnecessary attention of the occupant, thereby contributing to safe driving.

[0110] Returning to the basic processing procedure of the display control device 40 of this embodiment shown in Fig. 2 , as described above, after the agent image is displayed on the display device 70 using a geometric representation (first representation mode) or a personified character (second representation mode), the control unit 41 (driving load determination unit 414) determines whether or not there is a change in the driving load (step ST108). Specifically, when there is a change in the driving load and the image is switched between the first representation mode (displaying the agent image using a geometric representation) and the second representation mode (displaying the agent image using a personified character) ("YES" in step ST108), the control unit 41 (image switching unit 415) determines whether or not the occupant is viewing the agent image (step ST109).

[0111] Whether or not the occupant is viewing the agent image can be determined, for example, by calculating the line-of-sight direction from the occupant's viewpoint position detected by the viewpoint sensor 305 of the in-vehicle monitoring device 30 in Fig. 1, and then calculating the gaze point on the screen of the display device 70 from the obtained line-of-sight direction of the occupant. Alternatively, if the display device 70 is configured as a touch panel, it is also possible to estimate whether or not the occupant is viewing the agent image from their actions, such as whether or not they are operating the screen.

[0112] Here, if it is determined that the occupant is not viewing the agent image ("YES" in step ST109), the image switching unit 415 changes the representation format of the agent image (switches the display) and controls the display on the display device 70 (step ST111). On the other hand, if it is determined that the occupant is viewing the agent image ("NO" in step ST109), the image switching unit 415 further determines whether or not there is a dialogue with the agent (step ST110), and if there is no dialogue with the agent ("YES" in step ST110), switches the agent image (changes the representation format) and controls the display on the display device 70 (step ST111). Whether or not a conversation with the agent is occurring can be estimated, for example, by image recognition of the movement of the occupant's mouth in the video captured by the camera 301 of the in-vehicle monitoring device 30. Also, audio data is acquired by the audio input / output device 80 (microphone), and in order to distinguish this from a conversation with a passenger, if a passenger is on board, the presence or absence of the passenger's speech can be measured in a similar manner, and if the driver and passenger are speaking alternately, it can be determined that the agent and occupant (the occupant sitting in the driver's seat) are not conversing.

[0113] In this way, the control unit 41 (image switching unit 415) switches the display when the occupant is not viewing the agent image or interacting with the agent. This reduces the amount of image change that occurs when the display is switched, and the occupant does not feel any abruptness in the image switching. Although not shown in the flowchart of FIG. 2, a transition between images represented by personified characters may occur depending on the increase or decrease in driving load. In this case, as shown in FIG. 10, for example, the image switching unit 415 performs zoom-in control, zoom-out control, or the like to continuously change the angle of view to transition. For example, by dynamically performing zoom-in control (FIG. 10(a) → FIG. 10(b)) when the driving load increases and zoom-out control (FIG. 10(b) → FIG. 10(a)) when the driving load decreases, the agent image can be adjusted to an appropriate amount of information depending on the driving load, and the occupant does not feel any abruptness in the image switching.

[0114] (Modification) According to the display control device 40 of the above-described embodiment, the display control device 40 determines the driving load from at least one of the vehicle state, the driving environment, and the state of the driver, and controls the display device 70 to change the representation format of the agent image according to the determined driving load. However, by incorporating these functions into, for example, a head-up display device (HUD device 70A in FIG. 11 ), the functions of the display control device 40 can be realized by the HUD device 70A alone. In this case, the processing load on the display control device 40 can be reduced. The HUD device 70A can project an agent image onto an image plane that is virtually set in front of the vehicle, superimposed on the vehicle's forward field of view.

[0115] In this case, the HUD device 70A is configured by a control unit 71 and an image display unit 72, as shown in an example of the configuration in FIG.

[0116] the control unit 71 determines the driving load from at least one of the vehicle state, the driving environment, and the state of the driving occupant, and controls the display of the agent image on the display device 70 to change the representation form of the agent image according to the determined driving load. In order to do this, the control unit 71 has, for example, a vehicle information acquisition unit 711 that acquires vehicle state information from the in-vehicle monitoring device 30 shown in FIG. 1 , an environmental information acquisition unit 712 that acquires vehicle environment information, an occupant information acquisition unit 713 that acquires occupant (particularly, the driver sitting in the driver's seat) state information such as bioinformation acquired from a bioinformation sensor worn by a superior, a driving load determination unit 714 that determines the driving load from at least one of the vehicle state, the driving environment, and the state of the driving occupant, an image switching unit 715 that switches the display of the agent image between a geometric representation (first representation form) and a personified character representation (second representation form), an image generation unit 716 that generates display information including the agent image, and a display control unit 717 that controls the display of the display information including the generated agent image on the image display unit 73.

[0117] The HUD device 70A of this embodiment further includes a program for the control unit 71 to determine the driving load from at least one of the vehicle state, the driving environment, or the state of the driving occupant, and change the representation form of the agent image according to the determined driving load and display it on the display device 70. In addition, the agent image and VRA area are assigned to the work area and are each stored therein.

[0118] The image display unit 73 is mainly composed of a projection unit 731 including a light source formed by a light-emitting diode mounted on a wiring board and a relay optical system, and a TFT (Thin Film Transistor Liquid Crystal) type liquid crystal display element 732 located on the emission side (directly above) of the light source so as to transmit illumination light from the light source to form display light, and can output display light by transmitting light emitted from the light source through the liquid crystal display element 732. The liquid crystal display element 742 is a display device that forms a desired image based on display information (drive signal) generated under the control of the control unit 71, and displays the image formed on an imaging plane (display area) virtually set in front of the vehicle so as to be superimposed on the viewer's forward field of vision, allowing the viewer to view it.

[0119] According to the HUD device 70A of this embodiment, the control unit 71 determines the driving load from at least one of the vehicle state, the driving environment, and the state of the driver, and controls the display unit 73 to change the representation of the agent image according to the determined driving load. For example, a head-up display device can be provided in which a geometric representation (first representation) that does not interfere with the driver's driving is used to present only driving-related information, allowing the driver to concentrate on driving, and a personified character representation (second representation) is used to make the driver feel more attached to the vehicle and more human. In this way, by switching the representation of the agent image depending on the driving load, it is possible to achieve both preventing interference with driving and increasing the likability of the agent to the driver (the degree to which the agent is perceived as human and lovable).

[0120] Furthermore, when the control unit 71 determines that the driving load is high, for example, the control unit 71 displays the agent image in a color similar to the target area of ​​the forward field of view to be superimposed; when the control unit 71 determines that the driving load is low, for example, the control unit 71 displays the agent image in a color consistent with the character or design of the agent image. Since the HUD device 70A displays the agent image superimposed on the vehicle's forward field of view, when the driving load is high, the agent image is displayed in the same color as the superimposed area of ​​the forward field of view (e.g., gray if the image is superimposed on a road, or the color of the vehicle if the image is superimposed on a vehicle ahead) so as not to interfere with driving. When the driving load is low, the agent image is displayed in a color consistent with the character and design of the agent, thereby preventing interference with driving and increasing the likeability of the agent. Furthermore, the HUD device 70A can display the image projected on the image surface according to its display distance (from near to far), so that the image can be superimposed on a corner of a crossroads such as an intersection. When the driving load is low, the agent image can be displayed as if it is standing at the corner, guiding (pointing) the way to turn, thereby increasing the likeability of the agent.

[0121] (Effects of the embodiment) As explained above, the display control device 40 of this embodiment is, for example, a display control device 40 that controls a display device 70 that displays an agent image that allows information exchange with a vehicle occupant, as shown in Fig. 1. The display control device 40 includes a storage unit 42 that stores the agent image, and a control unit 41 that determines the driving load from at least one of the vehicle state, the driving environment, and the state of the driving occupant, and changes the representation form of the agent image in accordance with the determined driving load and controls the display of the agent image on the display device 70.

[0122] In the display control device 40 of this embodiment, the control unit 41 determines the driving load from at least one of the vehicle state, the driving environment, and the state of the driver, and controls changing the representation of the agent image in accordance with the determined driving load. Therefore, according to the display control device 40 of this embodiment, for example, by displaying a geometrical image that does not interfere with the driver's driving and presenting only driving-related information, the driver can concentrate on driving. Furthermore, by displaying an anthropomorphic character, the driver can feel that the vehicle is more human-like and lovable. In this way, by switching the representation of the agent image depending on the driving load, it is possible to prevent interference with driving and increase the likability of the agent in the driver (the degree to which the agent is perceived as human-like and lovable).

[0123] Furthermore, in the display control device of this embodiment, when the control unit 41 determines that the driving load is high, it displays the agent image in a first representation form in which it is geometrically expressed on the display device 70, and when it determines that the driving load is low, it displays the agent image in a second representation form in which it is expressed as a personified character on the display device 70. In this way, the agent image is displayed by switching between the first representation form in which it is geometrically expressed and the second representation form in which it is expressed as a personified character depending on the driving load, so that it is possible to both prevent interference with driving and increase the likeability of the agent to the occupant (the degree to which the agent is perceived as a human being that the occupant can feel attached to).

[0124] Furthermore, in the display control device of this embodiment, when the control unit 41 displays the agent image on the display device 70 in the second representation mode, the control unit 41 controls the display range of the body parts of the person represented by the character in accordance with the driving load, so that, for example, when the driving load is high, the number of body parts to be displayed is reduced (for example, from (b) to (a) in FIG. 5A), thereby reducing the amount of information contained in the agent image and thereby reducing the possibility of focusing on unnecessary parts, thereby improving safety. Also, for example, the same effect can be obtained by displaying only the upper body parts as shown in FIG. 5B (from (b) to (a) in FIG. 5B).

[0125] Furthermore, in the display control device of this embodiment, when the control unit 41 displays the agent image on the display device 70 in the second representation mode, the control unit 41 controls the movement of the body parts of the person represented by the character in accordance with the driving load. For example, when the driving load is high, the control unit 41 reduces the amount of movement of the agent image (body parts of the character) (e.g., from FIG. 6( a) to FIG. 6( b)), thereby reducing the attention to the agent image. This prevents further unnecessary attention from being generated during high driving loads, thereby improving safety. For example, in FIGS. 6( a) and 6( b), this is achieved by reducing the amount of movement of one of the body parts, the arm. However, this is not limited to this; for example, the movement speed of the other arm may be reduced. On the other hand, when the driving load is low, the control unit 41 may change the character's facial expression, for example, by making the character smile when approaching the destination. In this case, when the driving load is high, the character's facial expression becomes stiff.

[0126] Furthermore, in the display control device of this embodiment, when the control unit 41 displays the agent image on the display device 70 in the second representation form, the display size of the character is controlled in accordance with the driving load. For example, when the driving load is high, the display size of each body part of the agent image is increased, making it possible to grasp the general information. Also, when the driving load is high, it is assumed that it is difficult to take your eyes off the road, but if the display size is increased, the general movement can be grasped even with a side glance, so it is possible to reduce looking away.

[0127] Furthermore, in the display control device of this embodiment, when the control unit 41 displays the agent image on the display device 70 in the second representation mode, the control unit 41 controls the degree of facing of the character to the occupant in accordance with the driving load, so that, for example, when the driving load is high, the facing of the agent image to the occupant can be reduced to reduce the feeling that someone is watching. For example, people can feel someone's gaze even when looking at a painting with a person in it, and by performing the control described above, it is possible to reduce distraction and unnecessary attention.

[0128] Furthermore, in the display control device of this embodiment, when the control unit 41 displays the agent image on the display device 70 in the first representation form (geometric representation), the number of color combinations used for the geometric representation is controlled according to the driving load. For example, when the driving load is high, the number of color combinations for the agent image is reduced, thereby reducing the possibility that the occupant will find meaning beyond the information presented by the character from the meaning of the colors.

[0129] Furthermore, in the display control device of this embodiment, when the control unit 41 displays the agent image on the display device 70 in the first representation mode, the control unit 41 controls the order of geometric representation in accordance with the driving load. For example, when the driving load is high, the control unit 41 controls to lower the order used for the geometric representation of the agent image. For example, as shown in FIG. 9( a), a line is used in the first order (high driving load), a polygon or circle is used in the second order (medium driving load) as shown in FIG. 9( b), and a three-dimensional representation is used in the third order (low driving load) as shown in FIG. 9( c). As a result, the lower the order, the less information is used for representation, and the cognitive load can be kept low. On the other hand, in situations where the driving load is low and there is relatively more time, the representation can be made more rich, making the product more appealing.

[0130] Furthermore, in the display control device 40 of this embodiment, when the control unit 41 displays the agent image on the display device in the first representation form, the control unit 41 controls the dynamic elements of the geometric representation in accordance with the driving load, so that, for example, when the driving load is high, the dynamic elements of the geometric representation of the agent image (for example, shape changes and animations) can be reduced, specifically, by reducing the amount of shape change, reducing the update cycle of the dynamic representation, etc., thereby reducing unnecessary attention to the occupants and contributing to safe driving.

[0131] Furthermore, in the display control device of this embodiment, when the control unit 41 switches the display of the agent image between the first representation form and the second representation form, it performs the switching at a time when the occupant is not viewing the agent image or not interacting with the agent, thereby reducing the amount of change in the image caused by the switching, so that the occupant does not have to be aware of the sudden switching.

[0132] Furthermore, in the display control device of this embodiment, when the control unit 41 switches the display between agent images displayed in the first representation form or between agent images displayed in the second representation form due to an increase or decrease in driving load (a transition between agent images displayed in the second representation form may occur due to an increase or decrease in driving load), the angle of view is continuously changed and displayed on the display device 70, for example, as shown in Figure 10, by continuously changing and transitioning the angle of view by zooming in, zooming out, etc., for example, when the driving load increases, zoom-in control is performed (transition from Figure 10(a) to Figure 10(b)), and when the driving load decreases, zoom-out control is performed (transition from Figure 10(b) to Figure 10(a)), so that the agent image can have an appropriate amount of information according to the driving load.

[0133] Furthermore, in the display control device of this embodiment, when the control unit 41 determines the driving load based on the state of the vehicle, the determination is made based on at least one of the degree of driving assistance, the steering angle, the amount of accelerator operation, or the amount of brake operation, thereby making it possible to easily and at low cost determine the driving load based on information obtained from a driving assistance device 10 (see Figure 1) possessed by a vehicle with a driving assistance function, or based on detection information obtained from a behavior sensor 303 of an on-board monitoring device 30 that the vehicle originally possesses.

[0134] Furthermore, in the display control device of this embodiment, when the control unit 41 determines the driving load based on the driving environment of the vehicle, the determination is made from map information having information on areas where the driving load changes, or environmental information around the vehicle monitored while driving by a sensor mounted on the vehicle. For example, the driving load can be determined easily and at low cost based on detection information obtained from a digital map stored in the map information DB 200 of the navigation device 20 shown in Figure 1 or obtained by communicating with an external center not shown, or from the camera 301, GPS 302, behavior sensor 303, LiDAR 304, etc. of the on-board monitoring device 30 originally possessed by the vehicle.

[0135] Furthermore, in the display control device of this embodiment, when the control unit 41 determines the driving load based on the state of the occupant driving the vehicle, the driving load can be determined easily and with low coherence by making the determination from the occupant's biometric information or analyzed facial expression of the occupant monitored by a sensor worn by the occupant or mounted on the vehicle.

[0136] The head-up display device (HUD device 70A) of this embodiment is a head-up display device (HUD device 70A) that projects an agent image superimposed on the forward field of view of the vehicle on an image plane that is virtually set in front of the vehicle, as shown in Fig. 11. The HUD device 70A includes an image display unit 73 that displays the agent image, and a control unit 71 that determines the driving load from at least one of the vehicle state, the driving environment, and the state of the occupant driving the vehicle, and changes the representation form of the agent image to be displayed on the image display unit 73 in accordance with the determined driving load.

[0137] According to the HUD device 70A of this embodiment, the control unit 71 determines the driving load from at least one of the vehicle state, the driving environment, and the state of the occupant driving the vehicle, and controls the image display unit 73 to change the representation of the agent image in accordance with the determined driving load. For example, by displaying the agent image geometrically so as not to interfere with the occupant's driving and presenting only driving-related information, the occupant can concentrate on driving. Furthermore, by representing the agent image as, for example, an anthropomorphized character, the HUD device 70A can be provided, which makes the occupant feel more attached to the vehicle and more human. In this way, by switching the representation of the agent image and displaying it in accordance with the driving load, it is possible to achieve both prevention of driving interference and the likability of the agent in the occupant (the degree to which the agent is perceived as an lovable human).

[0138] Furthermore, in the HUD device 70A of this embodiment, when the control unit 71 determines that the driving load is high, the agent image is displayed in a color close to the target area of ​​the forward field of view to be superimposed, and when the control unit 71 determines that the driving load is low, the agent image is displayed in a color that conforms to the character or design of the agent image.As a result, the HUD device 70A displays the agent image superimposed on the vehicle's forward field of view, and therefore when the driving load is high, the agent image is displayed in the same color as the area of ​​the forward field of view to be superimposed (for example, gray if superimposed on a road, the color of the vehicle if superimposed on a vehicle in front) so that the agent does not interfere with driving, and when the driving load is low, the agent image is displayed in a color that conforms to the character and design of the agent, thereby preventing interference with driving and making the agent more likable. Furthermore, since the HUD device 70A can display the image projected onto the image surface according to the display distance (from far to near), it can display a display superimposed on a corner of a crossroads such as an intersection, and when the driving load is low, it can increase the likeability of the agent by making it appear as if the agent is standing at the corner and guiding (pointing) the way to turn.

[0139] The program of this embodiment is, for example, a program for a display control device 40 that controls a display device 70 that displays an agent image that allows information exchange with a vehicle occupant, as shown in Fig. 1. The program causes a processor included in the display control device 40 to execute, for example, a process of determining a driving load from at least one of the vehicle state, the driving environment, and the state of the occupant driving the vehicle (steps ST101 to ST104), and a process of changing the representation format of the agent image in accordance with the determined driving load and controlling the display of the agent image on the display device 70 (steps ST105 to ST111), as shown in Fig. 2.

[0140] According to the program of this embodiment, the processor of the display control device 40 sequentially reads and executes programs stored in a partial area of ​​the storage unit 42 shown in Fig. 1, for example, to provide a geometric display that does not interfere with the occupant's driving and present only driving-related information, thereby allowing the occupant to concentrate on driving, and to use an anthropomorphized character representation, making the occupant feel more attached to the vehicle and more human-like. In this way, by switching the representation form of the agent image depending on the driving load, it is possible to achieve both prevention of driving interference and the likability of the agent in the occupant (the degree to which the agent is felt to be human and lovable).

[0141] 1, an in-vehicle agent system 100 of this embodiment is an in-vehicle agent system 100 that provides driving assistance to an occupant by placing an agent image within the vehicle cabin. The in-vehicle agent system 100 includes a head-up display device (display device 70) that projects an agent image onto a virtual image plane set in front of the vehicle so that the agent image is superimposed on the vehicle's forward field of view, a display control device 40 that determines the driving load from at least one of the vehicle state, the driving environment, and the state of the occupant driving the vehicle, and controls the display of the agent image on the head-up display device 70 to change the representation format of the agent image in accordance with the determined driving load, and a driving assistance device 10 that cooperates with the display control device 40 to make the agent image appear within the vehicle cabin, communicates information via the agent image, and exchanges information with the occupant to assist the occupant in driving the vehicle.

[0142] According to the in-vehicle agent system 100 of this embodiment, the display control device 40 cooperates with the driving assistance device 10 to determine the driving load based on at least one of the vehicle state, the driving environment, and the state of the occupant driving the vehicle, and controls the display of the agent image on the head-up display device (display device 70) to change the representation of the agent image according to the determined driving load. The agent image communicates information and exchanges information with the occupant, thereby assisting the occupant in driving the vehicle. Therefore, for example, by displaying the agent image in a geometrical form that does not interfere with the occupant's driving and presenting only driving-related information, the occupant can concentrate on driving. Furthermore, by using an anthropomorphic representation of the agent image, for example, the in-vehicle agent system 100 can be provided, which makes the vehicle appear more human and lovable. In this way, by switching the representation of the agent image depending on the driving load, it is possible to achieve both preventing driving interference and increasing the occupant's likability (the degree to which the agent appears human and lovable).

[0143] The present invention is not limited to the above-described exemplary embodiments, and those skilled in the art will be able to easily modify the above-described exemplary embodiments to the extent that they fall within the scope of the claims.

[0144] 10... driving assistance device, 20... navigation device, 30... in-vehicle monitoring device, 40... display control device, 41... control unit, 42... memory unit, 50... audio input / output control device, 60... I / O interface, 70... display device, 70A... head-up display device (HUD device), 71 control unit (HUD device side), 72 memory unit (HUD device side), 73... image display unit, 80... audio input / output device, 90... biometric information sensor, 100... in-vehicle agent system, 200... map information DB, 301... camera, 302... GPS, 303... behavior sensor, 304... LiDAR, 305... viewpoint sensor, 411... vehicle information acquisition unit, 412... environmental information acquisition unit, 413... occupant information acquisition unit, 414... driving load determination unit, 415... image switching unit, 416... image generation unit, 417... display control unit, 711... vehicle information acquisition unit, 712... environmental information acquisition unit, 713... occupant information acquisition unit, 714... driving load determination unit, 715... image switching unit, 716... image generation unit, 717... display control unit, 731... projection unit, 732... liquid crystal display element

Claims

1. A display control device that controls a display device that displays an agent image capable of exchanging information with a passenger in a vehicle, comprising: a memory unit that stores the agent image; and a control unit that determines a driving load from at least one of the state of the vehicle, the driving environment, or the state of the passenger driving, and changes the representation form of the agent image in accordance with the determined driving load and controls the display on the display device.

2. A display control device as described in claim 1, wherein the control unit, when it is determined that the driving load is high, displays the agent image on the display device in a first representation form in which the agent image is geometrically represented, and when it is determined that the driving load is low, displays the agent image on the display device in a second representation form in which the agent image is represented by an anthropomorphized character.

3. A display control device as described in claim 2, wherein the control unit controls the display range of the human body parts represented by the character in accordance with the driving load when the agent image is displayed on the display device in the second representation form.

4. A display control device as described in claim 2, wherein the control unit controls the movement of the body parts of the person represented by the character in accordance with the driving load when the agent image is displayed on the display device in the second representation form.

5. A display control device as described in claim 2, wherein the control unit controls the display size of the character in accordance with the driving load when the agent image is displayed on the display device in the second representation form.

6. A display control device as described in claim 2, wherein the control unit controls the degree of confrontation of the character with the occupant in accordance with the driving load when the agent image is displayed on the display device in the second representation form.

7. A display control device as described in claim 2, wherein the control unit controls the number of color schemes used for the geometric representation in accordance with the driving load when the agent image is displayed on the display device in the first representation form.

8. A display control device as described in claim 2, wherein the control unit controls the degree of the geometric representation in accordance with the driving load when the agent image is displayed on the display device in the first representation form.

9. A display control device as described in claim 2, wherein the control unit controls dynamic elements of the geometric representation in accordance with the driving load when the agent image is displayed on the display device in the first representation form.

10. A display control device as described in any one of claims 2 to 9, wherein the control unit switches the display of the agent image between the first representation form and the second representation form at a timing when the occupant is not viewing the agent image or not interacting with the agent.

11. A display control device as described in any one of claims 2 to 9, wherein the control unit, when switching the display between the agent images displayed in the first representation form or switching the display between the agent images displayed in the second representation form due to an increase or decrease in the driving load, continuously changes the angle of view and displays it on the display device.

12. A display control device as described in claim 1, wherein the control unit, when determining the driving load based on the state of the vehicle, makes a determination based on at least one of the degree of driving assistance, the steering angle, the amount of accelerator operation, or the amount of brake operation.

13. A display control device as described in claim 1, wherein the control unit, when determining the driving load based on the driving environment of the vehicle, makes the determination from map information having information on the area where the driving load changes, or from environmental information around the vehicle monitored while driving by a sensor mounted on the vehicle.

14. A display control device as described in claim 1, wherein the control unit, when determining the driving load based on the state of the occupant driving the vehicle, makes the determination from the occupant's biometric information or the occupant's analyzed facial expression monitored by a sensor worn by the occupant or mounted on the vehicle.

15. A head-up display device which projects an agent image onto an image plane virtually set in front of a vehicle, superimposed on the forward field of view of the vehicle, comprising: an image display unit which displays the agent image; and a control unit which determines a driving load from at least one of the state of the vehicle, the driving environment, or the state of the driving occupant, and controls the display of the agent image on the image display unit to change its representation form in accordance with the determined driving load.

16. A head-up display device as described in claim 15, wherein the control unit, when it is determined that the driving load is high, displays the agent image in a color close to the target area of ​​the forward field of view on which it is superimposed, and when it is determined that the driving load is low, displays the agent image in a color conforming to the character or design of the agent image.

17. A program for a display control device that controls a display device that displays an agent image capable of exchanging information with a passenger on board a vehicle, the program causing a processor of the display control device to execute a process of determining the driving load from at least one of the state of the vehicle, the driving environment, or the state of the driving passenger, and a process of controlling the display device to change the representation form of the agent image and display it in accordance with the determined driving load.

18. An in-vehicle agent system that provides driving assistance to an occupant by placing an agent image within the vehicle cabin space, comprising: a head-up display device that projects the agent image onto an image plane that is virtually set in front of the vehicle, superimposed on the forward field of view of the vehicle; a display control device that determines the driving load from at least one of the state of the vehicle, the driving environment, or the state of the occupant driving, and controls the head-up display device to change the representation form of the agent image in accordance with the determined driving load; and a driving assistance device that cooperates with the display control device to make the agent image appear within the vehicle cabin, convey information through the agent image, and assist the occupant in driving the vehicle by exchanging information with the occupant.

Images