Display control devices, head-up displays, software products, and in-vehicle intelligent systems

Abstract

This invention provides a display control device, etc. It is applicable to an in-vehicle intelligent agent system that achieves driving-free operation by changing the display mode of an intelligent agent image according to the driving load. A display control device (40) controls a display device (700) that displays an intelligent agent image capable of exchanging information with occupants of a vehicle. The display control device (40) includes: a storage unit (42) that stores intelligent agent images; and a control unit (41) that performs the following control: determines the driving load based on at least one of the vehicle's state, the driving environment, or the state of the occupant driving; changes the display mode of the intelligent agent image according to the determined driving load; and displays the intelligent agent image on the display device in the changed display mode.

Description

Technical Field

[0001] This invention relates to, for example, display and control devices for in-vehicle intelligent systems that assist occupants of driving vehicles. Background Technology

[0002] Previously, there was a known in-vehicle intelligent agent system that brought an intelligent agent into the vehicle interior to determine its behavior based on the vehicle's state or the condition of the occupants. This intelligent agent conveyed various information and could exchange information with the occupants. In this case, because the intelligent agent was displayed as a human-like figure on the screen and provided voice output, the vehicle was perceived as an increasingly humanized and affectionate vehicle.

[0003] For example, Patent Document 1 describes an in-vehicle intelligent agent system that configures an intelligent agent through a three-dimensional character image in cooperation with in-vehicle equipment and assists the occupants. According to the technology described in Patent Document 1, it has an assistance unit that allows the three-dimensional character image to move freely within the vehicle interior and assist at a suitable position related to assistance, and a voice-emitting unit for the character image, which enables the sound image to be positioned at a suitable position related to assistance.

[0004] Existing technical documents Patent documents Patent Document 1: Japanese Patent Application Publication No. 2006-284454 (see (Abstract)) Summary of the Invention

[0005] The technical problem that the invention aims to solve However, according to the technology described in Patent Document 1, there are issues such as the occupant's gaze being drawn to instructions or dialogue from an intelligent agent (the gaze is locked), which hinders safe driving.

[0006] On the other hand, the vehicle's surrounding conditions, including its status, change constantly as the vehicle moves. Due to these conditions, occupants (the driver in the driver's seat) experience various psychological and physical loads (so-called driving load). Since increased driving load can negatively impact driving operations, various attempts have been made to estimate driving load and reflect it in the vehicle's processing systems.

[0007] The present invention was made to solve the above-mentioned problems, and its object is to provide a display control device suitable for an in-vehicle intelligent agent system that does not interfere with driving, for example by changing the way the intelligent agent image is displayed according to the driving load.

[0008] Other objects of the invention will be apparent to those skilled in the art from the following illustrated schemes and preferred embodiments, as well as the accompanying drawings.

[0009] Technical solutions adopted to solve technical problems Hereinafter, in order to facilitate understanding of the content of the present invention, the solutions of the present invention will be illustrated.

[0010] The first solution is a display control device that controls a display device that displays an intelligent agent image capable of exchanging information with occupants of a vehicle. It includes: a storage unit that stores the intelligent agent image; and a control unit that performs the following control: determining a driving load based on at least one of the vehicle's state, the driving environment, or the state of the occupant driving; changing the display mode of the intelligent agent image based on the determined driving load; and displaying the intelligent agent image on the display device in the changed display mode.

[0011] Here, "driving load" refers to the psychological and physical burden on occupants based on the vehicle's condition, driving environment, and the occupants' condition; for example, it refers to the degree of burden required for cognition, judgment, and driving operations. Figure 3A As shown, driving load is categorized into "A" (high driving load), "B" (medium driving load), and "C" (low driving load), and the behavior for each category is defined. Additionally, "the representation of the agent image" refers to, for example, in... Figure 3A As shown in "A", the image of the intelligent agent is represented by a monochrome geometric image whose shape changes according to the agent's voice, and only the information needed for driving (information to prompt attention or navigation information, etc.) is presented; or as in Figure 3A As indicated by "B" or "C", the intelligent agent image is represented by an anthropomorphic character. This character is depicted using images with physical actions such as blinking, nodding, and looking in the direction of the occupants. In addition to the information needed for driving, it also provides prompts such as... Figure 3B The entertainment information shown includes sightseeing information, news, weather, and recommendations (such as TV shows, music, and movies that match the passengers' interests) that do not affect driving operations.

[0012] In the first approach, the control unit performs the following control: it determines the driving load based on at least one of the vehicle's state, the driving environment, or the state of the occupant driving, and changes the representation of the agent image based on the determined driving load. Therefore, according to the first approach, for example, by displaying a geometric representation that does not interfere with the occupant's driving and only providing driving-related information, the occupant can concentrate on driving; furthermore, by displaying a human-like character, the vehicle can be perceived as a humanized entity that evokes feelings of attachment. In this way, by switching and displaying the agent image based on the driving load, it is possible to balance preventing interference with driving with the positive feelings the agent evokes in the occupant (the degree to which the vehicle is perceived as a humanized entity that evokes feelings of attachment).

[0013] Furthermore, driving load is determined based on at least one of the following: vehicle status, driving environment, and the status of the occupant driving. When determining driving load based on vehicle status, it can be estimated, for example, based on the level of driver assistance or autonomous driving, steering wheel angle, accelerator pedal input, and brake pedal input. When determining driving load based on the driving environment, it can be estimated based on map information or detection information from an onboard monitoring device. When using map information, for example, driving load can be estimated from the area where the vehicle is currently driving, based on information about driving load variations such as urban areas, suburbs, bypasses, highways, intersections, and the number of lanes on the road. When using detection information from an onboard monitoring device, driving load can be estimated based on surrounding vehicles, obstacles, pedestrians, or road width detected by devices such as LiDAR (Light Detection and Ranging). When determining driving load based on the occupant's status, driving load can be estimated, for example, based on biometric information such as heart rate or pulse waves obtained from biosensors, or facial expressions obtained through image recognition of occupant facial images captured by a camera. For example, if a state of tension is inferred based on heart rate, the driving load can be determined to be high; if a state of relaxation is inferred, the driving load can be determined to be low.

[0014] In the second scheme, which is subordinate to the first scheme, the control unit may display the agent image on the display device in a first representation of geometry when it determines that the driving load is high, and display the agent image on the display device in a second representation of an anthropomorphic character when it determines that the driving load is low.

[0015] In the second approach, the control unit switches between and displays a first representation of the agent's image using geometric representations (e.g., polygons, arcs, lines) and a second representation using an anthropomorphic character, depending on the driving load. Therefore, the second approach balances preventing obstruction of driving with enhancing the occupant's positive impression of the agent (the degree to which the vehicle is perceived as a humanized vehicle that evokes a sense of attachment).

[0016] In a third embodiment subordinate to the second embodiment, the control unit may control the display range of the body parts of the person represented by the character based on the driving load when displaying the image of the intelligent agent on the display device in the second representation mode.

[0017] In the third approach, when the control unit displays the agent image in the second presentation mode, it controls the display range of body parts represented by the character based on the driving load. Therefore, according to the third approach, for example, under high driving load conditions, control is performed to reduce the number of body parts displayed. Figure 5A (b) → (a)), reducing the amount of information in the agent's image, thereby reducing the likelihood of fixation on unnecessary parts and improving security. Additionally, for example, Figure 5B As shown, only the upper body portion is displayed. Figure 5B (b) → (a) can also achieve the same effect.

[0018] In the fourth embodiment, which is subordinate to the second embodiment, the control unit may control the movements of the body parts of the person represented by the character based on the driving load when the image of the intelligent agent is displayed on the display device in the second representation mode.

[0019] In the fourth approach, when the control unit displays the agent image in a second manner, it controls the movements of body parts represented by the character, for example, based on the driving load. Therefore, according to the fourth approach, for example, under high driving load, control is performed to reduce the amount of movement of the agent image (the character's body parts) (e.g., ...). Figure 6 (a) → Figure 6 (b) can reduce the eye-catching nature of images of intelligent agents. Here, "eye-catching nature" refers to the degree to which an image attracts human attention. Therefore, no further unnecessary salience is generated under high driving loads, improving safety. For example, in Figure 6 In (a) and (b), this is achieved by reducing the amount of movement of the arm, which is one of the body parts, but it is not limited to this; for example, the movement speed of the other arm can also be reduced. On the other hand, under low driving load conditions, it is also possible to control the character's expression, such as making the character smile near the destination. In this case, under high driving load conditions, the character's expression will become frozen.

[0020] In the fifth embodiment, which is subordinate to the second embodiment, the control unit may control the display size of the character based on the driving load when displaying the image of the intelligent agent on the display device in the second representation.

[0021] In the fifth approach, when the control unit displays the agent image in the second display mode, it controls the display size of the character based on, for example, the driving load. Therefore, according to the fifth approach, for example, under high driving load, by increasing the display size of each body part of the agent image, general information can be grasped. In addition, under high driving load, it is assumed that the driver's gaze is difficult to take off, but if the display size is increased, the general movement can be grasped by "glancing to the side", thus reducing "gaze deviation".

[0022] In the sixth embodiment, which is subordinate to the second embodiment, the control unit may control the degree of antagonism between the character and the occupant based on the driving load when displaying the image of the intelligent agent on the display device in the second representation.

[0023] In the sixth embodiment, when the control unit displays the agent image in a second representation, it controls the degree of orientation between the character and the occupant, for example, based on the driving load. Here, "orientation" refers to the degree to which the agent image (its line of sight) is face-to-face with the occupant (primarily the occupant sitting in the driver's seat), for example, by... Figure 8 (a) shows the angle θ, which is defined as a smaller angle θ indicating a lower degree of antagonism. According to the sixth scheme, for example, under high driving load conditions, by reducing the antagonism of the agent's image to the occupant, the feeling of being watched can be alleviated. For example, even in paintings with figures, people sometimes feel gazes; by performing the above control, inattentiveness or unnecessary conspicuousness can be reduced.

[0024] In the seventh embodiment, which is subordinate to the second embodiment, the control unit may control the number of colors used for the geometric representation based on the driving load when displaying the agent image on the display device in the first representation manner.

[0025] In the seventh embodiment, when the control unit displays the agent image in the first presentation mode, it controls the number of colors used for geometric representation based on the driving load. In the seventh embodiment, for example, under high driving load conditions, by reducing the number of colors in the agent image, the possibility of occupants interpreting meanings beyond those provided by the character based on the inherent meanings of colors can be reduced.

[0026] In the eighth embodiment, which is subordinate to the second embodiment, the control unit may control the number of times the geometric representation is performed based on the driving load when the image of the intelligent agent is displayed on the display device in the first representation manner.

[0027] In the eighth approach, when the agent image is displayed in the first representation mode, the number of geometric representations is controlled, for example, based on the driving load. For instance, under high driving load, the number of geometric representations used for the agent image is reduced. Figure 9 As shown in (a), in the case of a single instance (high driving load), the line is used, such as Figure 9 As shown in (b), in the secondary case (moderate driving load), polygons or circles are used, such as Figure 9 As shown in (c), stereoscopic representation is performed in three scenarios (low driving load). Therefore, the lower the resolution, the less information is used for representation, thus suppressing cognitive load. On the other hand, in scenarios with low driving load and sufficient capacity, the representation can be richer to allow the product to be perceived.

[0028] In the ninth embodiment, which is subordinate to the second embodiment, the control unit may control the dynamic elements of the geometric representation based on the driving load when the intelligent agent image is displayed on the display device in the first representation manner.

[0029] In the ninth approach, when the agent image is displayed in the first representation, the dynamic elements of the geometric representation are controlled, for example, according to the driving load. Therefore, according to the ninth approach, for example, under high driving load, by reducing the dynamic elements of the geometric representation of the agent image (e.g., shape changes or animations), specifically by reducing the amount of shape changes, reducing the update cycle of the dynamic representation, etc., unnecessary distractions for the occupants are reduced, thereby contributing to safe driving.

[0030] In the tenth embodiment, which is subordinate to the first to ninth embodiments, the control unit may switch the display of the agent image between the first and second display modes when the occupant has not visually confirmed the agent image or has not engaged in dialogue with the agent.

[0031] In the tenth embodiment, when the control unit switches the agent image between the first and second display modes, the switching is performed when the passenger has not visually confirmed the agent image or has not engaged in dialogue. Therefore, according to the tenth embodiment, by performing the switching when the passenger has not visually confirmed the agent image or has not engaged in dialogue, the amount of image change caused by the switching is reduced, thus preventing the passenger from being aware of the abrupt switching.

[0032] Furthermore, regarding whether occupants visually confirm the agent's image, for example, it can be based on... Figure 1The vehicle monitoring device 30, as shown, uses the viewpoint sensor 305 to detect the occupant's viewpoint position, calculates the gaze direction, and then calculates the gaze point on the display device 70 screen from the obtained occupant gaze direction to make a determination. Alternatively, if the display device 70 is a touchscreen, it can be estimated from whether there is screen operation or other actions. On the other hand, regarding the timing of no interaction, for example... Figure 1 As shown, vocalization can be inferred by image recognition of the mouth movements in the facial images of occupants captured by the camera 301 of the vehicle monitoring device 30.

[0033] In the eleventh scheme, which belongs to the first to ninth schemes, the control unit may continuously change the viewing angle and display it on the display device when switching between the intelligent agent images displayed in the first presentation mode or between the intelligent agent images displayed in the second presentation mode according to the increase or decrease of the driving load.

[0034] In the eleventh solution, the control unit performs the following control: continuously changing the viewing angle when switching between displaying agent images in the first or second display mode. Therefore, according to the eleventh solution, although there may be cases where the view shifts between agent images displayed in the second display mode depending on the increase or decrease in driving load, in such cases, for example... Figure 10 As shown, the movement is achieved by continuously changing the viewing angle, such as zooming in and out. For example, when the driving load increases, zoom control is implemented (…). Figure 10 (a) → Figure 10 (b) migration), under reduced driving load, by implementing reduction control ( Figure 10 (b) → Figure 10 (a) The migration not only enables the agent image to have an appropriate amount of information corresponding to the driving load, but also avoids making the occupants aware of the abrupt switch. The same applies to the migration that occurs between agent images displayed in the first representation according to the increase or decrease of driving load.

[0035] In the twelfth embodiment, which belongs to the first to eleventh embodiments, the control unit determines the driving load based on at least one of the following when determining the driving load according to the state of the vehicle: the degree of driving assistance, the steering angle of the steering wheel, the amount of throttle operation, or the amount of brake operation.

[0036] In the twelfth embodiment, when the control unit determines the driving load based on the vehicle's state, it does so based on at least one of the following: the degree of driving assistance, the steering wheel angle, the amount of throttle input, or the amount of brake input. Therefore, since the vehicle load can be determined based on detection information from driving assistance devices provided by the vehicle or the vehicle's original onboard monitoring device 30, it can be implemented in a simple and low-cost manner. For example, when determining the driving load based on the degree of driving assistance from ADAS (Advanced Driver Assistance Systems), the degree of driving assistance intervention is scored, and the driving load is determined by a total score. In this case, for example, a score of 1 is assigned for lane departure warning, a score of 2 for lane departure correction, a score of 3 for lane keeping, a score of 1 is assigned for maintaining a set speed, a score of 2 is assigned for matching the speed of the vehicle ahead, and a score of 3 is assigned for automatic forward / stop response to traffic signals or other driving conditions. If the total score is 5 or higher (threshold), the driving load is determined to be low. In addition, in the case of autonomous driving, it can be determined that Level 1 (driving assistance in only one direction) with the occupant as the main driver is considered to have high driving load, while Level 2 (driving assistance in both longitudinal and lateral directions) with hand freedom and Level 3 (autonomous driving under specific conditions) with eye freedom are considered to have low driving load.

[0037] In the thirteenth embodiment, which belongs to the first to eleventh embodiments, the control unit may determine the driving load based on map information of the area where the driving load changes, or based on environmental information of the vehicle's surroundings monitored by sensors mounted on the vehicle during driving, when determining the driving load based on the vehicle's driving environment.

[0038] In the thirteenth embodiment, the control unit determines the driving load based on the vehicle's driving environment, for example, using map information showing areas with varying driving loads, or using environmental information about the vehicle's surroundings detected by sensors mounted on the vehicle during driving. The "map information" mentioned here is, for example, stored in... Figure 1 The navigation device 20 shown contains a map information database 200, or a digital map acquired through communication with an external center (omitted from the diagram). Additionally, "vehicle-mounted sensors" are, for example... Figure 1 The vehicle-mounted monitoring device 30 shown includes a camera 301, a GPS (Global Positioning System) 302, a behavior sensor 303, a LiDAR 304, etc.

[0039] Therefore, according to the thirteenth scheme, for example, when determining driving load based on map information, the determination can be made based on factors such as the curvature of curves, the frequency of curves, and the gradient of the road. Driving on roads with steep inclines and many sharp curves, like mountain roads, is considered a high driving load, while driving on flat highways without curves is considered a low driving load. Alternatively, the determination can be based on the number of people, pedestrians, bicycles, and vehicles in the surrounding area. Furthermore, the determination can be based on the number of buildings; for example, densely built-up areas like residential areas are considered high driving loads, while areas without buildings, like country lanes, are considered low driving loads. Additionally, the determination can be based on the number of traffic lights, lanes, intersections, pedestrian crossings, signs, and billboards that affect driving operations. Furthermore, driving load can also be determined based on information detected by sensors; driving during the day is considered a low driving load, while driving at night, when more attention is required, is considered a high driving load. Moreover, similar to the determination based on vehicle status, the determination can also be made by scoring the above driving environments and comparing the total score with a threshold.

[0040] In the fourteenth embodiment, which belongs to the first to eleventh embodiments, the control unit may determine the driving load based on the state of the occupant driving the vehicle, or based on the occupant's biometric information monitored by sensors worn on the occupant or mounted on the vehicle, or by analyzing the occupant's facial expressions.

[0041] In the fourteenth scheme, the control unit determines the driving load based on the occupants' states, and further determines it based on the occupants' biometric information monitored by sensors or analyzed facial expressions. For example, biometric information is determined based on heart rate, brain waves, and sweating. Regarding heart rate, the system analyzes the heart rate and frequency over a certain period; a high heart rate indicates the user is in a state of tension (stress), thus indicating a high driving load. Regarding brain waves, frequency analysis identifies a high driving load when the beta wave band (indicating concentration and stress) is high, and a low driving load when the alpha wave band (indicating relaxation) is high. Additionally, excessive sweating indicates high tension and stress, also indicating a high driving load.

[0042] Alternatively, judgments can be made based on occupant actions rather than biometrics. In this case, a driver facing forward or looking straight ahead could be considered to have a high driving load, while a driver facing a direction wider than the lane width could be considered to have a low driving load. Alternatively, the steering wheel grip position could be used for judgment; a high grip position presumes the need for steering angle adjustments based on the vehicle's surroundings, thus indicating a high driving load, while a low grip position indicates a low driving load. Furthermore, similar to judgments based on vehicle status and driving environment, a judgment can also be made by scoring the aforementioned occupant status and comparing the total score to a threshold.

[0043] The fifteenth solution is a head-up display device that overlays an agent image onto the forward field of view of the vehicle and projects it onto a virtual image plane set in front of the vehicle. It includes: an image storage unit that displays the agent image; and a control unit that performs the following control: determining the driving load based on at least one of the vehicle's state, the driving environment, or the state of the occupant driving the vehicle; changing the display mode of the agent image based on the determined driving load; and displaying the agent image on the image display unit in the changed display mode.

[0044] In the fifteenth embodiment, the control unit performs the following control: it determines the driving load based on at least one of the vehicle's state, the driving environment, or the state of the occupant driving; it then changes the representation of the agent image based on the determined driving load and displays the agent image on the image display unit in the changed representation. Therefore, according to the fifteenth embodiment, for example, by providing a geometric representation that does not interfere with the occupant's driving and only displaying driving-related information, the occupant can concentrate on driving. Furthermore, by providing a human-like character representation, a head-up display device can be provided that makes the vehicle feel like a humanized carrier that evokes a sense of attachment. Thus, by switching the representation of the agent image according to the driving load and displaying it, it is possible to balance preventing interference with driving with the positive impression the agent gives to the occupant (the degree to which the vehicle feels like a humanized carrier that evokes a sense of attachment).

[0045] In the sixteenth embodiment, which is subordinate to the fifteenth embodiment, the control unit may use a color close to the color of the object area of ​​the overlapping forward field of view to represent the agent image when it determines that the driving load is high, and use a color based on the role or design of the agent image when it determines that the driving load is low.

[0046] In the sixteenth solution, when the control unit determines that the driving load is high, it displays the agent image using a color close to the color of the object area overlapping the forward field of vision. When the driving load is low, it displays the image using a color based on the role or design of the agent. Therefore, according to the sixteenth solution, since the head-up display device displays the agent image overlapping the vehicle's forward field of vision, under high driving load conditions, in order to prevent the agent from interfering with driving, it displays the agent image using the same color as the overlapping forward field of vision area (e.g., gray when overlapping with the road, the color of the vehicle when overlapping with the vehicle in front, etc.). Under low driving load conditions, by setting the color based on the role or design of the agent, it is possible to balance preventing interference with driving and improving the user's perception of the agent. In addition, since the head-up display device can display the image projected onto the image plane according to its display distance (from far to near), it can overlay the image at intersections such as corners. Under low driving load conditions, by making the agent appear to be standing at the corner guiding (pointing out) the turning route, the user's perception of the agent can be improved.

[0047] The seventeenth solution is a program product for a display control device that controls a display device displaying an agent image capable of exchanging information with occupants of a vehicle. The program product causes a processor in the display control device to perform the following processing: determining a driving load based on at least one of the vehicle's state, the driving environment, or the state of the occupant driving the vehicle; and performing the following control: changing the display mode of the agent image based on the determined driving load and displaying the agent image on the display device in the changed display mode.

[0048] In the seventeenth embodiment, the display control device has a processor, for example, that reads and executes records sequentially. Figure 1 The programs in certain areas of the storage unit 42 shown can, for example, provide driving-related information using geometric representations that do not interfere with driving, allowing occupants to concentrate on driving. Furthermore, by employing anthropomorphic character portrayals, the vehicle can be perceived as a more humanized and endearing entity. Thus, by switching the display method of the intelligent agent image according to driving load, it is possible to balance preventing interference with driving with increasing occupant affinity for the intelligent agent (the degree to which the vehicle is perceived as a humanized and endearing entity).

[0049] The eighteenth embodiment is an in-vehicle intelligent agent system that configures an intelligent agent image within the vehicle cabin space and provides driving assistance to the occupants. This in-vehicle intelligent agent system comprises: a head-up display (HUD) that overlays the intelligent agent image onto the vehicle's forward field of view and projects it onto a virtual image plane positioned in front of the vehicle; a display control device that performs the following controls: determines the driving load based on at least one of the vehicle's state, the driving environment, or the state of the occupants driving the vehicle; changes the display method of the intelligent agent image based on the determined driving load; and displays the intelligent agent image on the HUD in the changed display method; and a driving assistance device that cooperates with the display control device to make the intelligent agent image appear in the vehicle cabin, conveys information through the intelligent agent image, and assists the occupants in driving operations by exchanging information with them.

[0050] In the eighteenth embodiment, the display control device, in cooperation with the driving assistance device, performs the following control: It determines the driving load based on at least one of the vehicle's state, the driving environment, or the state of the occupants driving the vehicle; it then changes the representation of the intelligent agent image according to the determined driving load and displays the intelligent agent image on the head-up display device in the changed representation; it conveys information through the intelligent agent image; and it assists the occupants in driving operations by exchanging information with them. Therefore, according to the eighteenth embodiment, for example, by using a geometric representation that does not hinder the occupants' driving and only providing driving-related information, the occupants can concentrate on driving. Furthermore, by providing a human-like character representation, an in-vehicle intelligent agent system can be provided that makes the vehicle feel increasingly like a humanized carrier that fosters attachment. Thus, by switching the representation of the intelligent agent image according to the driving load and displaying it, it is possible to balance preventing obstruction to driving with increasing occupants' positive feelings towards the intelligent agent (the degree to which the vehicle feels like a humanized carrier that fosters attachment).

[0051] Those skilled in the art will readily understand that further modifications can be made to the illustrated embodiments of the invention without departing from the spirit of the invention. Attached Figure Description

[0052] Figure 1 This is a block diagram illustrating a structural example of an in-vehicle intelligent agent system to which the display control device according to an embodiment of the present invention is applied.

[0053] Figure 2 This is a flowchart illustrating the basic processing steps of a display control device according to an embodiment of the present invention.

[0054] Figure 3A This table is cited to illustrate the behavior of the intelligent agent generated by the display control device according to embodiments of the present invention.

[0055] Figure 3B This is a diagram illustrating an example of displaying entertainment information generated by a display control device according to an embodiment of the present invention.

[0056] Figure 4 It is shown Figure 2 A flowchart detailing the steps of "Driving Load Determination and Processing".

[0057] Figure 5A This is a diagram showing an example (1) of an intelligent agent image generated by a display control device according to an embodiment of the present invention, which is represented by an anthropomorphic character.

[0058] Figure 5B This is a diagram showing an example (2) of an intelligent agent image generated by a display control device according to an embodiment of the present invention, which is represented by an anthropomorphic character.

[0059] Figure 6 This is a diagram showing an example (3) of an intelligent agent image generated by a display control device according to an embodiment of the present invention, which is represented by an anthropomorphic character.

[0060] Figure 7 This is a diagram showing an example (4) of an intelligent agent image generated by a display control device according to an embodiment of the present invention, which is represented by an anthropomorphic character.

[0061] Figure 8 This diagram is used to illustrate the orientation between the occupant and the intelligent agent displayed on the display device.

[0062] Figure 9 The figure is cited to illustrate the geometric representation that varies with the number of iterations generated by the display control device according to the embodiments of the present invention.

[0063] Figure 10 This is a screen transition diagram used to illustrate the switching between agent images performed by the display control device according to embodiments of the present invention.

[0064] Figure 11 This is a block diagram illustrating a structural example of a head-up display device according to an embodiment of the present invention. Detailed Implementation

[0065] The preferred embodiments described below are intended to facilitate understanding of the invention. Therefore, those skilled in the art should note that the invention is not unduly limited to the embodiments described below (hereinafter referred to as "this embodiment").

[0066] (Structure of the implementation method) Figure 1This is a block diagram illustrating a structural example of an in-vehicle intelligent agent system 100 to which the display control device 40 of this embodiment is applied. The in-vehicle intelligent agent system 100 is a system that enables an intelligent agent to determine behavior based on the vehicle, the condition of the occupants driving the vehicle, etc., to appear in the vehicle interior, to convey various information through the intelligent agent, and to exchange information with the occupants (the driver sitting in the driver's seat).

[0067] according to Figure 1 The vehicle-mounted intelligent system 100 includes a driving assistance device 10, a navigation device 20, a vehicle-mounted monitoring device 30, a bio-information sensor 90, a display control device 40 according to this embodiment, and a voice input / output control device 50, all of which are configured to enable bidirectional communication via an I / O interface 60.

[0068] The driver assistance device 10 possesses at least one of the following: ADAS-based driver assistance functions such as LKAS (Lane Keeping Assist System) and ACC (Adaptive Cruise Control System) that assist occupants in driving the vehicle; and an autonomous driving function that can perform driving operations on behalf of the occupants. Furthermore, based on information obtained from the onboard monitoring device 30, the driver assistance device 10 identifies or detects the driving environment and behaviors ahead of the vehicle. In addition to controlling the vehicle (the drive system, braking system, and steering system are omitted in the illustration) according to the analysis results of the information obtained therein, it can also transmit the analysis results of this information to the navigation device 20 and the display control device 40 via the I / O interface 60. Moreover, by cooperating with the display control device 40 (cooperative action), the driver assistance device 10 can display an intelligent agent image inside the vehicle, convey information through the intelligent agent image, and assist occupants in driving the vehicle by exchanging information with them.

[0069] The navigation device 20 uses a Global Navigation Satellite System (GNSS) such as GPS or a gyroscope sensor to obtain map information from map information or through wireless communication with the outside of the vehicle, and provides guidance on surrounding facilities or routes. Based on this guidance information, the navigation device 20 transmits a signal that causes the display output to be sent to the display control device 40 of this embodiment via the I / O interface 60 at an appropriate time.

[0070] Furthermore, the navigation device 20 has a map information database (map information database 200). The map information database 200 can acquire and store the latest map information, for example, through communication with an external center (not shown) via a V2X (Vehicle to X) type communication system (not shown). Here, the map information stored in the map information database 200 is mapping data digitized in a way that represents the vehicle's driving environment. As mapping data, high-precision dynamic map digital data is particularly preferred. Here, "dynamic map" refers to a digital map that combines a vast amount of constantly changing dynamic information, such as traffic control, construction information, accidents or congestion, pedestrian or signal information, with static information such as high-precision three-dimensional location information (road surface information, diagonal line information, three-dimensional structures).

[0071] The vehicle monitoring device 30 includes sensors necessary for identifying the surrounding driving environment, including the area in front of the vehicle, such as a camera 301, GPS 302, behavior sensor 303, LiDAR 304, and viewpoint sensor 305. The camera 301 captures at least the vehicle's forward field of view (real-world view) and the occupants' eyes (pupil image). The LiDAR 304 uses near-infrared, visible, and ultraviolet light, for example, illuminating an obstacle located in front of the vehicle as captured by the camera 301, and uses a light sensor to capture the reflected light, determining the distance to the obstacle based on the time difference. Furthermore, the behavior sensor 303 includes an IMU (Inertial Measurement Unit) or a vehicle speed sensor. The IMU uses a three-axis accelerometer and a three-axis angular velocity sensor (gyroscope sensor) to measure the vehicle's driving status or attitude (detecting translational motion in three axes based on acceleration [m / s²] and rotational motion based on angular velocity [deg / s]). The viewpoint sensor 305 can detect the viewpoint position of the occupant with high precision by performing image recognition on the eyes (pupil image) of the occupant sitting in the driver's seat of the vehicle, which is captured by the camera 301.

[0072] In addition, the bio-information sensor 90 is, for example, a wearable device that uses radio wave-based sensing technology to acquire the occupant's heart rate or respiratory information. By analyzing the reflected radio waves from the occupant, it can detect respiratory rate, heart rate, pulse fluctuations, etc., as periodic body movement components.

[0073] The display control device 40 of this embodiment can display an image of an agent that can exchange information with the occupants of the vehicle on the connected display device 70. The display device 70 may be, for example, a head-up display that overlays the agent image onto the forward field of view of the vehicle and projects it onto a virtual image plane set in front of the vehicle. Alternatively, a central information display (CID) located in the center of the vehicle or a helmet display (HMD) mounted on the occupant's head may also be used.

[0074] The display control device 40 of this embodiment includes a control unit 41 and a storage unit 42. The control unit 41 performs the following control: it determines the driving load based on at least one of the vehicle's state, the driving environment, or the state of the occupant driving; it changes the display mode of the agent image based on the determined driving load; and displays the agent image on the display device 70 in the changed display mode.

[0075] Furthermore, "the way an intelligent agent's image is represented" refers to, for example, in... Figure 3A The image shown is represented by "A," depicting an agent image in a monochrome geometric form whose shape changes according to the agent's voice. This is a primary representation that only provides information necessary for driving (information to prompt attention or navigation, etc.), or as shown in... Figure 3A The image, indicated by "B" or "C", uses anthropomorphic characters to represent the intelligent agent. These characters are depicted with gestures such as blinking, nodding, and looking towards the occupants. In addition to the information needed for driving, it also provides prompts such as... Figure 3B The entertainment information shown is presented in a second way (sightseeing information, news, weather, and recommendations (TV, music, movies, etc. that match the passengers' interests) that does not affect driving operation).

[0076] Here, the control unit 41 determines the driving load based on at least one of the following: the vehicle's state, the driving environment, and the state of the occupant driving. When the control unit 41 determines the driving load based on the vehicle's state, it may estimate it based on factors such as the level of driver assistance, the level of automated driving, the steering wheel angle, and the amount of throttle and brake input. Alternatively, when the control unit 41 determines the driving load based on the driving environment, it may estimate it based on map information or detection information from the onboard monitoring device 30. In the case of map information, for example, the driving load can be estimated from the area where the vehicle is currently driving based on information about driving load variations, such as urban areas, suburbs, bypasses, highways, intersections, and the number of lanes on the road. In the case of detection information from the onboard monitoring device 30, the driving load can be estimated based on surrounding vehicles, obstacles, pedestrians, or road width detected by devices such as LiDAR 304.

[0077] Furthermore, when the control unit 41 determines the driving load based on the occupant's state, it can estimate the driving load based on biometric information such as heart rate or pulse wave obtained from the biometric sensor 90 worn by the occupant, or facial expressions obtained through image recognition of the occupant's face captured by the camera 301 of the vehicle monitoring device 30. For example, if the occupant is estimated to be in a state of tension based on their heart rate or facial expression, the driving load can be determined to be high; if they are estimated to be in a state of relaxation, the driving load can be determined to be low.

[0078] Storage unit 42 is a memory, such as one equipped with static RAM, dynamic RAM, or flash memory, which has a program area and a working area. Here, the program area contains a program product for controlling the display control device 40 of the display device 70, which displays an agent image capable of exchanging information with the occupants of the vehicle. The working area contains information or agent images generated during the execution of the program product.

[0079] In addition, when the control unit 41 determines that the driving load is high, it can display the intelligent agent image on the display device 70 in a first representation mode of geometric representation, and when it determines that the driving load is low, it can display the intelligent agent image on the display device 70 in a second representation mode of anthropomorphic character representation.

[0080] Furthermore, when the control unit 41 displays the image of the intelligent agent on the display device 70 in a second representation mode, it can control the display range of body parts of the person represented by the character according to the driving load. For example, under high driving load, it can reduce the number of body parts displayed by executing a control (…). Figure 5A (b) → (a)), reducing the amount of information in the agent's image, or for example, as Figure 5B As shown, only the upper body portion is displayed. Figure 5B (b) → (a)).

[0081] Furthermore, when the control unit 41 displays the agent image on the display device 70 in a second representation mode, it can control the movements of the body parts represented by the character according to the driving load. For example, under high driving load, it can control the amount of movement of the agent image (the character's body parts) to be reduced (e.g., Figure 6 (a) → Figure 6 (b)).

[0082] Furthermore, when the control unit 41 displays the agent image on the display device 70 in a second representation mode, it can control the display size of the character according to the driving load. For example, when the driving load is high, it controls the display size of each body part of the agent image to be increased.

[0083] Furthermore, when the control unit 41 displays the agent image on the display device 70 in a second representation, it can control the degree of orientation between the character and the occupant based on the driving load. Here, "orientation" refers to the degree to which the agent image (its line of sight) is face-to-face with the occupant (primarily the occupant sitting in the driver's seat), for example, by... Figure 8(a) shows the angle θ, which is defined as a smaller angle θ indicating a lower degree of antagonism. For example, under high driving load conditions, antagonism of the agent image towards the occupant can be controlled to reduce the antagonism. Alternatively, it can be considered to prevent eye contact; by not directing the gaze of the agent image, which is presented as an anthropomorphic character, towards the occupant, the distance at which the occupant feels watched by the agent can be reduced or eliminated. For this purpose, antagonism needs to be controlled not only based on the position of the occupant's eyes but also based on the position of the driver's seat or the position of the occupant's head.

[0084] Here, based on images or videos of the occupants captured by the camera 301 or similar device in the vehicle monitoring unit 30, the occupant's head or gaze direction is measured through image recognition. Alternatively, the occupant's head position can be estimated based on the position of the driver's seat or headrest. Furthermore, if the display device 70 is a touchscreen, if the occupant is operating the touchscreen, it can be determined that the occupant is looking in the direction of the screen. Additionally, under high driving load conditions, it can be assumed that the occupant is facing forward when making controls.

[0085] Furthermore, when the control unit 41 displays the agent image on the display device 70 in a first representation mode, it can control the number of colors used for geometric representation based on the driving load. For example, under high driving load conditions, it controls the number of colors in the agent image to be reduced.

[0086] Furthermore, when the control unit 41 displays the agent image on the display device 70 in a first representation mode, it can control the number of times the geometric representation is applied based on the driving load. For example, when the driving load is high, it controls the number of times the geometric representation of the agent image is reduced (one time is a line, two times is a polygon or circle, and three times is a 3D representation).

[0087] Furthermore, when the control unit 41 displays the agent image on the display device 70 in a first representation mode, it can control the dynamic elements of the geometric representation according to the driving load. For example, under high driving load, it can control the dynamic elements (such as shape changes or animations) that reduce the geometric representation of the agent image.

[0088] Furthermore, when switching the agent image, the control unit 41 can perform the switch when the occupant has not visually confirmed the agent image or engaged in dialogue, whether switching from the first display mode to the second display mode or vice versa. Additionally, regarding whether the occupant has visually confirmed the agent image, for example, the gaze direction can be calculated based on the occupant's viewpoint position detected by the viewpoint sensor 305 of the vehicle monitoring device 30, and the gaze point on the display device 70 screen can be calculated from the obtained gaze direction (see, for example, Publication No. WO2012 / 077713). Alternatively, if the display device 70 is a touchscreen, the estimation can also be made based on whether the occupant performs any screen operations.

[0089] On the other hand, the timing of the lack of dialogue with the intelligent agent can be estimated by image recognition of the mouth movements in the occupant's facial image captured by the camera 301 of the vehicle monitoring device 30. Alternatively, voice data can be acquired through the voice input / output device 80 (microphone). To distinguish it from conversations with fellow passengers, the same method is used to measure whether there is dialogue with fellow passengers. If the driver and fellow passengers alternate in speaking, it can be determined that the intelligent agent is not conversing with the occupant (the occupant sitting in the driver's seat).

[0090] Furthermore, when switching between agent images displayed in the first display mode or between agent images displayed in the second display mode, the control unit 41 switches (changes) the agent images by continuously changing the viewing angle and displays them on the display device 70. For example, although the migration between agent images displayed in the second display mode may occur depending on the increase or decrease of driving load, in this case, the control unit 41 controls the migration by continuously changing the viewing angle, for example, by zooming in or zooming out.

[0091] Furthermore, when determining driving load based on the vehicle's condition, the control unit 41 can make the determination based on at least one of the following: the degree of driving assistance, the steering wheel angle, the amount of throttle input, or the amount of brake input. For example, when determining driving load based on the degree of driving assistance provided by ADAS, the control unit 41 can score the degree of driving assistance intervention and determine the driving load based on the total score.

[0092] For example, in driver assistance systems, a score of 1 is assigned to lane departure warning, 2 to lane departure correction, 3 to lane keeping, 1 to maintaining a set speed, 2 to matching the speed of the vehicle ahead, and 3 to automating forward / stop actions in response to traffic signals and other driving conditions. A total score of 5 or higher (the threshold) indicates low driver load. Furthermore, in autonomous driving, levels below Level 1 (one-way driver assistance) with minimal foot movement (where the occupant is the primary driver) are considered high driver load, while Level 2 (longitudinal and lateral driver assistance) with minimal hand movement and Level 3 (automatic driving under specific conditions) with minimal eye movement are considered low driver load.

[0093] Furthermore, when determining the driving load based on the vehicle's driving environment, the control unit 41 can make the determination based on map information showing areas where the driving load changes, or on environmental information about the vehicle's surroundings monitored during driving by sensors mounted on the vehicle. The "map information" could be, for example, a map information database 200 stored in the navigation device 20, or a digital map obtained through communication with an external center (without illustrations). The "sensors mounted on the vehicle" could be, for example, the camera 301, GPS 302, behavior sensor 303, and LiDAR 304 of the vehicle monitoring device 30.

[0094] For example, when determining driving load based on map information, factors such as curve curvature, curve frequency, and gradient can be considered. Driving on steep, winding roads like mountainous terrain is considered high driving load, while driving on smooth, flat highways is considered low. Alternatively, the number of people, pedestrians, bicycles, and vehicles in the surrounding area can be used. The number of buildings can also be considered; densely built-up areas like residential areas have high driving load, while areas without buildings like country lanes have low driving load. Furthermore, the number of traffic lights, lanes, intersections, pedestrian crossings, signs, and billboards that affect driving can also be used. When determining driving load based on sensor data, daytime driving conditions are considered low driving load, while nighttime driving conditions requiring greater caution are considered high driving load. Similarly, as with vehicle status-based determinations, the driving environment can be scored, and the total score compared to a threshold.

[0095] Furthermore, when determining the driving load based on the state of the occupants of the vehicle, the control unit 41 can make the determination based on occupant biometric information monitored by sensors mounted on the vehicle or worn by the occupants, or on analyzed occupant facial expressions. For example, biometric information may be based on... Figure 1 The bio-information sensor 90 measures heart rate, brain waves, and sweating to make judgments. Regarding heart rate, the system analyzes heart rate and frequency over a certain period. A high heart rate indicates the user is in a state of tension (stress), thus determining a high driving load. Regarding brain waves, frequency analysis is performed. A high beta wave frequency (indicating concentration and stress) indicates a high driving load, while a high alpha wave frequency (indicating relaxation) indicates a low driving load. Furthermore, excessive sweating indicates high tension and stress, also resulting in a high driving load.

[0096] Alternatively, the determination can be made based on occupant movements captured by the camera 301 of the vehicle monitoring device 30, rather than on biometric information. In this case, a high driving load is determined when facing forward or looking straight ahead, while a low driving load is determined when facing a direction wider than the lane width. Alternatively, the determination can be based on the steering wheel grip position. A high grip position presupposes that steering angle adjustments are needed based on the vehicle's surroundings, thus determining a high driving load, while a low grip position determines a low driving load. Furthermore, similar to determinations based on vehicle status and driving environment, the determination can also be made by scoring the aforementioned occupant status and comparing the total score with a threshold.

[0097] Therefore, the control unit 41 is configured to include a vehicle information acquisition unit 411, an environmental information acquisition unit 412, an occupant information acquisition unit 413, a driving load determination unit 414, an image switching unit 415, an image generation unit 416, and a display control unit 417.

[0098] The vehicle information acquisition unit 411 acquires, for example, images, location information, vehicle behavior, obstacle detection information, etc., detected by the camera 301, GPS 302, behavior sensor 303, and LiDAR 304 of the vehicle monitoring device 30, and outputs them to the display control device 40 via the I / O interface 60. Additionally, the environmental information acquisition unit 412 acquires, for example, map information from the navigation device 20 (map information database 200) and information on surrounding vehicles, obstacles, pedestrians, or road width detected by the LiDAR 304 of the vehicle monitoring device 30, and outputs it to the display control device 40 via the I / O interface 60.

[0099] The occupant information acquisition unit 413 acquires, for example, occupant facial expressions captured by the camera 301 of the vehicle monitoring device 30, or occupant biometric information such as heart rate and pulse wave measured by the biometric sensor 90 worn on the occupant, and outputs it to the display control device 40 via the I / O interface 60.

[0100] The driving load determination unit 414 determines the driving load based on at least one of the following: vehicle status, driving environment, and the status of the occupant driving. When determining the driving load based on vehicle status, it can be estimated based on factors such as the level of driver assistance, the level of automated driving, the steering wheel angle, and the amount of throttle and brake input. Alternatively, when determining the driving load based on the driving environment, it can be estimated based on map information (map information database 200 of navigation device 20) or detection information obtained by the vehicle monitoring device 30. When using map information, the driving load can be estimated from the area where the vehicle is traveling, for example, based on information about areas where driving load changes, such as urban areas, suburbs, bypasses, highways, intersections, and the number of lanes on roads. When using detection information from the vehicle monitoring device 30, the driving load can be estimated based on factors such as surrounding vehicles, obstacles, pedestrians, or road width detected by LiDAR 304. Furthermore, when determining driving load based on the occupant's state, driving load can be estimated, for example, based on biometric information such as heart rate or pulse wave acquired by biometric sensor 90, or facial expressions obtained through image recognition of the occupant's face captured by camera 301. For instance, if the heart rate indicates a state of tension, the driving load can be determined to be high; if the heart rate indicates a state of relaxation, the driving load can be determined to be low.

[0101] When the image switching unit 415 switches between displaying an agent image in a geometric representation (first representation mode) and an anthropomorphic character representation (second representation mode), the switching can be performed when the occupant has not visually confirmed the agent image or has not engaged in dialogue with the agent. Furthermore, when the image switching unit 415 switches between displaying agent images in the first representation mode or between displaying agent images in the second representation mode based on changes in driving load, it can continuously change the viewing angle and display the image on the display device 70.

[0102] The image generation unit 416 generates display information containing an agent image and performs control to write to the VRAM (Video RAM) area allocated in a portion of the storage unit 42 in sync with the update cycle of the display information. Furthermore, the display control unit 416 reads the display information written to the VRAM area in sync with the display timing of the display device 70 and outputs it to the display device 70, thereby enabling the desired display containing the agent image to be obtained.

[0103] Furthermore, in the control unit 41, in order to perform the aforementioned control, a processor, for example, equipped with built-in memory (ROM / RAM) or external memory, is installed. The processor reads and executes sequentially the programs recorded in the memory (or storage unit 42), enabling it to perform the functions of each of the vehicle information acquisition unit 411, environmental information acquisition unit 412, occupant information acquisition unit 413, driving load determination unit 414, image switching unit 415, image generation unit 416, and display control unit 417. Specifically, it performs the following control: determining the driving load based on at least one of the vehicle's state, the driving environment, or the state of the occupant driving; changing the display method of the intelligent agent image based on the determined driving load; and displaying the intelligent agent image on the display device 70 in the changed display method. Additionally, at least some of the above functions can also be implemented without a processor, but rather through hardware such as an FPGA (Field Programmable Gate Array) or logic circuits.

[0104] The voice input / output control device 50 manages the interface between the connected voice input / output device 80 (such as a microphone or speaker) and the I / O interface 60, and performs "speech synthesis processing" which mechanically generates speech content based on the intelligent agent and outputs it to the speaker, and "speech recognition processing" which mechanically understands the speech content of the occupants collected through the microphone and transmits it to the intelligent agent.

[0105] (The actions of the implementation method) Figure 2 This is a flowchart illustrating the basic processing steps of the display control device 40 according to this embodiment. Figure 3A This is a table referenced to illustrate the behavior of the intelligent agent. Figure 3B This is an example of how entertainment information is displayed. Figure 4 It is shown Figure 2 A flowchart detailing the steps of "Driving Load Determination Processing". Additionally, Figure 5A , Figure 5B , Figure 6 , Figure 7 These are examples (1) to (4) of intelligent agent images presented in anthropomorphic roles (second representation). Additionally, Figure 8 This diagram is used to illustrate the orientation between the occupants and the intelligent agent displayed on the display device 70. Figure 9 The diagram was cited to illustrate the geometric representation that varies with degree. Figure 10 This is a screen transition diagram used to illustrate the switching between images of an intelligent agent. See below for reference. Figures 2 to 10 right Figure 1 The operation of the display control device 40 of this embodiment will be described in detail.

[0106] First, in the display control device 40 (control unit 41) of this embodiment, the vehicle information acquisition unit 411 acquires vehicle status information, the environment information acquisition unit 412 acquires driving environment information, and the occupant information acquisition unit 413 acquires occupant status information, and outputs the acquired information to the driving load determination unit 414 (steps ST101 to ST103). Here, the vehicle status information includes, for example, the degree of intervention of driving assistance or the level of autonomous driving acquired from the driving assistance device 10, or the steering wheel angle, throttle, and brake operation amount detected by the behavior sensor 303 of the vehicle monitoring device 30. In addition, the driving environment information includes, for example, information related to surrounding vehicles, obstacles, pedestrians, or road width detected or output by the LiDAR 304 of the vehicle monitoring device 30 or the map information database 200 of the navigation device 20. In addition, the occupant status information includes, for example, biological information such as heart rate or pulse wave acquired from the bio-information sensor 90, or information related to occupant expressions obtained by image recognition based on the occupant facial image captured by the camera 301 of the vehicle monitoring device 30.

[0107] The driving load determination unit 414 determines the driving load on the occupants of the vehicle based on at least one of the acquired vehicle status information, driving environment information, and occupant status information (step ST104). When determining the driving load based on the vehicle status information, it may be estimated based on factors such as the degree of driving assistance, the level of autonomous driving, the steering wheel angle, and the amount of throttle and brake input. For example, when the driving assistance device 10 performs ADAS driving assistance, and the driving load is determined based on the degree of driving assistance intervention, it may be estimated based on factors such as... Figure 4 As shown in the detailed steps, when the driving assistance provided by the driving assistance device 10 is intervened (step ST201 "A"), the control unit 41 (driving load determination unit 414) scores the degree of intervention (step ST202) and determines the driving load based on its total score (step ST203).

[0108] Here, when the driving load determination unit 414 scores the level of intervention of driving assistance, the control unit 41 (driving load determination unit 414) assigns a score of 1 for lane departure warning, a score of 2 for lane departure correction, a score of 3 for lane keeping, a score of 1 for maintaining the vehicle speed at a set value, a score of 2 for matching the speed of the vehicle in front, and a score of 3 for automatically moving forward / stopping in response to traffic signals or other driving environments. Then, if the total score reaches, for example, 5 or more (threshold) (step ST203 is "Yes"), the driving load determination unit 414 determines that the driving load is low (step ST204), and if the total score is less than 5 (step ST203 is "No"), it determines that the driving load is high (step ST205).

[0109] On the other hand, if it is determined in step ST201 that autonomous driving is in progress (step ST201 "B"), the driving load determination unit 414 determines whether it is below Level 1 (driving assistance in only one direction) where the occupant is the driving subject and the foot freedom is limited (step ST206). Here, if it is determined to be below Level 1 (step ST206 is "Yes"), the driving load determination unit 414 determines that the driving load is high (step ST207). If it is Level 2 (longitudinal and lateral driving assistance) where the hand freedom is limited or Level 3 (autonomous driving under specific conditions) where the eye freedom is limited (step ST206 is "No"), the driving load determination unit 414 determines that the driving load is low (step ST208).

[0110] Furthermore, when the driving load determination unit 414 determines the driving load based on driving environment information, it can make inferences based on map information (obtained from the map information database 200 of the navigation device 20) or various detection information from the vehicle monitoring device 30. When using map information, for example, the driving load can be inferred from the area where the vehicle is traveling, based on driving load variation information such as urban areas, suburbs, bypasses, highways, intersections, and the number of lanes on the road. When using detection information from the vehicle monitoring device 30, for example, the driving load can be inferred based on surrounding vehicles, obstacles, pedestrians, or road width detected by the LiDAR 304. Additionally, when the driving load determination unit 414 determines the driving load based on occupant status information, it can infer the driving load based on biometric information such as heart rate or pulse wave obtained by the biometric sensor 90, or facial expressions obtained through image recognition of occupant facial images captured by the camera 301. For example, if the heart rate indicates a state of tension, the driving load can be determined to be high; if the heart rate indicates a state of relaxation, the driving load can be determined to be low.

[0111] Return to the instructions Figure 2The basic processing steps of the display control device 40 in this embodiment are shown below. As described above, the vehicle load is determined. If the driving load determination unit 414 determines that the driving load is high (step ST105 is "Yes"), the control unit 41 (image generation unit 416) generates an agent image and writes it to the VRAM area of ​​the storage unit 42. The display control unit 416 reads the agent image from the VRAM area synchronously with the display timing and outputs it to the display device 70 to obtain the desired display. The agent image generated and displayed on the display device 70 in this case is, for example, as shown below. Figure 3A As shown in the upper part of the table, the image of the agent is represented in a geometric representation (first representation method). In this case, it is represented by a relatively simple pattern in monochrome with a shape that changes according to the agent's voice (step ST106).

[0112] On the other hand, in the driving load determination unit 414, if the driving load is determined to be low (step ST105 is "No"), the control unit 41 (image generation unit 416) generates an agent image and writes it into the VRAM area of ​​the storage unit 42. The display control unit 416 reads the agent image from the VRAM area synchronously with the display timing and outputs it to the display device 70 to obtain the desired display. The agent image generated and displayed on the display device 70 in this case is, for example, as shown in the image below. Figure 3A The lower section of the table shows a personified character (second mode of expression), accompanied by blinking, nodding, looking in the direction of the occupants while speaking, and using images of actions caused by body language to allow the character to communicate verbally (step ST107). Additionally, the background can be changed according to changes in the character's emotions or the content. Furthermore, in addition to information needed for driving, prompts can also be provided, such as... Figure 3B The entertainment information shown includes sightseeing information, news, weather, and recommendations (such as TV shows, music, and movies that match the passengers' interests) that do not affect driving operations.

[0113] In addition, such as Figure 3A As shown in the middle section, in addition to the "high" and "low" driving load, there are also multiple modes such as "medium" which involves small changes in actions such as blinking, nodding, and facing the occupants. The communication performance of the role can also be switched in stages according to the driving load level.

[0114] Figure 5A , Figure 5B , Figure 6 (a)(b) Figure 7 (a)(b) show an example (1) to (4) of an agent image generated by the control unit 41 (image generation unit 416) in the form of an anthropomorphic character.

[0115] exist Figure 5AIn the agent image (1) shown, when the driving load determination unit 414 determines that the driving load is high (here, the driving load is (a) > (b)), the image generation unit 416 controls the generation of the agent image by reducing the number of displayed body parts, and then the display control unit 417 performs the following: Figure 5A (b) is transferred to the display content shown in Figure 5(a) and displayed under the control of the display device 70. This reduces the amount of information in the agent's image, decreases the likelihood of focusing on unnecessary parts, and thus improves security. Incidentally, in Figure 5A In (b), the number of body parts (display area) of the character is "10", while Figure 5A In (a), the value is 6, indicating a reduction in information content. Additionally, for example... Figure 5B The image of the intelligent agent shown (2) is illustrated by displaying only the upper body portion. Figure 5B The migration from (b) to (a) can also achieve the same effect. Incidentally, in Figure 5B In (b), the number of body parts (display area) that was "6" is changed to "3", and the amount of information is reduced.

[0116] In addition, Figure 6 In the agent image (3) shown in (a)(b), when the driving load determination unit 414 determines that the driving load is high (here, the driving load is (a) > (b)), the image generation unit 416 controls the generation of the image by reducing the amount of motion (arrow marker) of the agent image (body parts of the character), and then displays the control unit 417 as shown. Figure 6 (a)→ Figure 6 As shown in (b), control is performed to migrate and display the displayed content on the display device 70. This reduces the eye-catching nature of the agent's image. Here, "eye-catching nature" refers to the degree to which it attracts human attention. Therefore, no further unnecessary eye-catching occurs under high driving loads, improving safety. Furthermore, in Figure 6 In (a) and (b), while this is achieved by reducing the amount of movement of the arm, which is a body part, it is not limited to this; for example, the movement speed of the other arm could also be reduced. On the other hand, under low driving load conditions, it is also possible to control the character's expression, such as making the character smile near the destination. In this case, the character's expression becomes frozen under high driving load conditions.

[0117] In addition, Figure 7 In the agent image (4) shown in (a)(b), the image generation unit 416 controls the image generation, for example, by changing the display size of the character according to the driving load. Thus, for example, in the case of a high driving load (where the driving load is (a) > (b)), the display size of each body part of the agent image is increased. Figure 7(b)→ Figure 7 (a) Being able to grasp the general information. In this case, it is assumed that it is difficult to leave the field of vision from the front, but if the display size is increased, the general movement can be grasped by "looking to the side", thus reducing "eye movement deviation".

[0118] Furthermore, when the image generation unit 416 generates and displays an image of an intelligent agent in the form of an anthropomorphic character (second representation mode), the degree of antagonism between the character and the occupant can be controlled, for example, based on the driving load. Here, "antagonism" is, for example, as... Figure 8 The figure shown refers to the degree to which the intelligent agent image AG (line of sight) is face-to-face with the occupant DR (mainly the occupant sitting in the driver's seat). Figure 6 (dashed lines in the text), for example, by Figure 8 (a) shows the angle θ, which is defined as the smaller the angle θ, the lower the degree of directionality. Figure 8 (b) is the state where the angle θ is 0° and the view is completely opposite. For example, under high driving load, by reducing the opposite orientation of the agent's image to the occupant, the feeling of being watched can be alleviated. For example, even in paintings with figures, people sometimes feel gazes; by performing the above control, inattentiveness or unnecessary distractions can be reduced. Furthermore, in Figure 8 In (a) and (b), 70a represents the display surface 70a of the display device 70.

[0119] On the other hand, when the image generation unit 416 displays an agent image generated in a geometric shape (first representation mode) on the display device 70, the number of colors used for geometric representation can be controlled according to the driving load. In this case, for example, when the driving load is high, by reducing the number of colors in the agent image, the possibility that the occupant interprets the meaning of colors beyond the meaning of the information provided by the character can be reduced.

[0120] Furthermore, when the image generation unit 416 generates an agent image in a geometric shape (first representation mode), and the display control unit 417 displays the agent image on the display device 70, the number of times the geometric representation is performed can be controlled according to the driving load. In this case, when the driving load is high, the number of times the geometric representation of the agent image is reduced. For example, Figure 9 As shown in (a), in a single scenario (high driving load), the representation is based on a line (waveform) theme, such as... Figure 9 As shown in (b), in the secondary case (moderate driving load), the representation is based on polygons or circles, such as Figure 9As shown in (c), the presentation focused on three-dimensional shapes in three scenarios (low driving load). Therefore, the lower the level of presentation, the less information is used to represent it, thus suppressing cognitive load. On the other hand, in scenarios with low driving load and more leeway, the presentation can be richer, thereby allowing the product to be perceived.

[0121] Furthermore, when the image generation unit 416 generates an agent image in a geometric shape (first representation mode) and the display control unit 417 displays the agent image on the display device 70, the dynamic elements of the geometric representation can be controlled according to the driving load, for example. In this case, for example, when the driving load is high, by reducing the dynamic elements of the agent image's geometric representation (e.g., shape changes or animations), specifically by reducing the amount of shape changes and reducing the update cycle of the dynamic representation, unnecessary visual distractions for the occupants can be reduced, thereby contributing to safe driving.

[0122] Return to the instructions Figure 2 The basic processing steps of the display control device 40 in this embodiment are shown below. As described above, after displaying the agent image on the display device 70 in either geometric representation (first representation mode) or anthropomorphic character representation (second representation mode), the control unit 41 (driving load determination unit 414) determines whether the driving load has changed (step ST108). Specifically, if the driving load has changed and an image switch occurs between the first representation mode (displaying the agent image in geometric representation) and the second representation mode (displaying the agent image in anthropomorphic character representation) (step ST108 is "yes"), the control unit 41 (image switching unit 415) determines whether the occupant is visually confirming the agent image (step ST109).

[0123] Regarding whether the occupants are visually confirming the agent's image, for example, it can be based on... Figure 1 The vehicle-mounted monitoring device 30 uses a viewpoint sensor 305 to detect the occupant's viewpoint position, calculates the direction of gaze, and then calculates the point of gaze on the display device 70 screen based on the obtained occupant's gaze direction, thereby making a determination. Alternatively, if the display device 70 is composed of a touchscreen, the determination can also be made based on whether there are screen operations or other actions.

[0124] Here, if it is determined that the occupant has not visually confirmed the agent image (step ST109 is "Yes"), the image switching unit 415 performs the following control: changes the display mode of the agent image (display switching) and displays the agent image on the display device 70 in the changed display mode (step ST111). On the other hand, if it is determined that the occupant is visually confirming the agent image (step ST109 is "No"), the image switching unit 415 further determines whether there is a dialogue with the agent (step ST110). If there is no dialogue with the agent (step ST110 is "Yes"), it controls the switching of the agent image (changing the display mode) and displays it on the display device 70 (step ST111). Furthermore, whether or not there is dialogue with the intelligent agent can be inferred by image recognition of the mouth movements of the occupant in the image captured by the camera 301 of the vehicle monitoring device 30. Alternatively, voice data can be obtained through the voice input / output device 80 (microphone). In order to distinguish it from the dialogue of the passenger, the same method is used to measure whether the passenger is speaking when there is a passenger in the vehicle. If the driver and the passenger speak alternately, it can be determined that the intelligent agent is not talking to the occupant (the occupant sitting in the driver's seat).

[0125] In this way, the control unit 41 (image switching unit 415) performs display switching when the occupant has not visually confirmed the agent's image or has not interacted with the agent. Since the amount of image change caused by the display switching is reduced, the occupant will not be aware of the abrupt switch. Furthermore, although in Figure 2 The flowchart omits illustrations, but sometimes the images representing anthropomorphic characters shift according to changes in driving load, in which case, for example, Figure 10 As shown, the image switching unit 415 continuously changes the viewing angle through zoom-in control, zoom-out control, etc., to move the image. For example, when the driving load increases, zoom-in control is performed (…). Figure 10 (a)→ Figure 10 (b) Under reduced driving load, dynamic reduction control is implemented ( Figure 10 (b)→ Figure 10 (a) enables the intelligent agent image to have an appropriate amount of information corresponding to the driving load, so that the occupants are not aware of the abrupt switch.

[0126] (Modified example) According to the display control device 40 of this embodiment described above, the display control device 40 is configured to perform the following control: determine the driving load based on at least one of the vehicle's state, the driving environment, or the state of the occupant driving; change the display mode of the agent image based on the determined driving load; and display the agent image on the display device 70 in the changed display mode. However, these functions can be incorporated into, for example, a head-up display device. Figure 11 In the HUD device 70A, the functions of the display control device 40 are implemented by the HUD device 70A alone. In this case, the processing load of the display control device 40 can be reduced. The HUD device 70A can overlay the image of the intelligent agent in the forward field of view of the vehicle and project it onto a virtual image plane set in front of the vehicle.

[0127] In this case, the HUD device 70A is, for example, Figure 11 As shown in the example structure, it consists of a control unit 71 and an image display unit 72.

[0128] The control unit 71 performs the following control: determining the driving load based on at least one of the vehicle's state, the driving environment, or the state of the occupant driving; changing the display mode of the agent image based on the determined driving load; and displaying the agent image on the display device 70 in the changed display mode. For example, it includes a vehicle information acquisition unit 711, which acquires information from... Figure 1 The vehicle monitoring device 30 shown acquires vehicle status information; an environmental information acquisition unit 712 acquires vehicle environmental information; an occupant information acquisition unit 713 acquires occupant (especially driver sitting in the driver's seat) status information such as bio-information from bio-information sensors worn on the occupants; a driving load determination unit 714 determines driving load based on at least one of the vehicle status, driving environment, and the status of the occupant driving; an image switching unit 715 switches the display of an agent image between geometric representation (first representation mode) and an anthropomorphic character representation (second representation mode); an image generation unit 716 generates display information containing an agent image; and a display control unit 717 controls the display of the generated display information containing the agent image on the image display unit 73.

[0129] In the HUD device 70A of this embodiment, in addition to the program product for the control unit 71 to perform the following control, namely, to determine the driving load based on at least one of the vehicle's state, the driving environment, or the state of the occupant driving, and to change the display mode of the agent image according to the determined driving load and display the agent image on the display device 70 in the changed display mode, the agent image and the VRA area are allocated in the working area and stored respectively.

[0130] The image display unit 73 mainly consists of a projection unit 731 and a TFT (Thin Film Transistor Liquid Crystal) type liquid crystal display element 732. The projection unit 731 includes a light source composed of light-emitting diodes mounted on a wiring substrate and a relay optical system. The liquid crystal display element 732 is located on the emission side (directly above) of the light source so that illumination light from the light source is transmitted to form display light. The liquid crystal display element 732 can output display light by transmitting light emitted from the light source. The liquid crystal display element 742 is a display device that forms a desired image based on display information (drive signals) generated under the control of the control unit 71, and overlays the image formed on a virtual imaging surface (display area) set in front of the vehicle into the forward field of vision so that the visual observer can visually confirm it.

[0131] According to the HUD device 70A of this embodiment, the control unit 71 performs the following control: it determines the driving load based on at least one of the vehicle's state, the driving environment, or the state of the occupant driving; it changes the display mode of the agent image based on the determined driving load and displays the agent image on the display unit 73 in the changed display mode. Therefore, for example, by displaying geometrically oriented information that does not interfere with the occupant's driving (first display mode) and only providing driving-related information, the occupant can concentrate on driving. Furthermore, by displaying anthropomorphic character information (second display mode), a head-up display device can be provided that makes the vehicle feel more like a humanized carrier that fosters a sense of attachment. In this way, by switching the display mode of the agent image according to the driving load, it is possible to balance preventing interference with driving with the positive feelings the agent brings to the occupant (the degree to which the vehicle feels like a humanized carrier that fosters a sense of attachment).

[0132] Furthermore, when the control unit 71 determines that the driving load is high, it displays the agent image using a color close to the color of the object area overlapping the forward field of vision. When the driving load is low, it displays the image using a color based on the agent's role or design. Since the HUD device 70A displays the agent image overlapping the vehicle's forward field of vision, it can avoid the agent interfering with driving when the driving load is high by using the same color as the overlapping forward field of vision area (e.g., gray when overlapping with the road, the color of the vehicle when overlapping with the vehicle in front). When the driving load is low, by setting the color based on the agent's role or design, it can balance preventing obstruction of driving with increasing the agent's appeal. Additionally, since the HUD device 70A can display the projected image on the image surface according to its display distance (from far to near), it can overlap the image at intersections like corners. When the driving load is low, by making the agent appear to be standing at the corner guiding (pointing out) the turning route, it can improve the agent's appeal.

[0133] (Effects of the implementation method) As described above, the display control device 40 of this embodiment is, for example, Figure 1 The diagram shows a display control device 40 that controls a display device 70 that displays an agent image capable of exchanging information with occupants of the vehicle. The display control device 40 includes a storage unit 42 for storing agent images and a control unit 41. The control unit 41 performs the following control: it determines the driving load based on at least one of the vehicle's state, the driving environment, or the state of the occupant driving; it then changes the display mode of the agent image based on the determined driving load and displays the agent image on the display device 70 in the changed display mode.

[0134] In the display control device 40 of this embodiment, the control unit 41 performs the following control: it determines the driving load based on at least one of the vehicle's state, the driving environment, or the state of the occupant driving, and changes the display mode of the agent image according to the determined driving load. Therefore, according to the display control device 40 of this embodiment, for example, by displaying a geometric representation that does not interfere with the occupant's driving and only displaying driving-related information, the occupant can concentrate on driving. In addition, by displaying an anthropomorphic character, the vehicle can be perceived as a humanized carrier that evokes feelings of attachment. In this way, by switching the display mode of the agent image according to the driving load and displaying it, it is possible to balance preventing interference with driving and increasing the occupant's positive feelings towards the agent (the degree to which the vehicle is perceived as a humanized carrier that evokes feelings of attachment).

[0135] Furthermore, in the display control device of this embodiment, when the control unit 41 determines that the driving load is high, it displays the agent image on the display device 70 in a first representation using geometric representation; when it determines that the driving load is low, it displays the agent image on the display device 70 in a second representation using an anthropomorphic character. Thus, by switching between the first representation using geometric representation and the second representation using an anthropomorphic character based on the driving load, it is possible to balance preventing hindrance to driving with the positive impression the agent gives to the occupants (the degree to which the vehicle is perceived as a humanized vehicle that evokes feelings of attachment).

[0136] Furthermore, in the display control device of this embodiment, when the intelligent agent image is displayed on the display device 70 in a second display mode, the control unit 41 controls the display range of the body parts of the person represented by the character according to the driving load. For example, when the driving load is high, it controls the number of body parts displayed to be reduced (e.g., when the driving load is high). Figure 5A (b) → (a)), reducing the amount of information in the agent's image, thereby reducing the likelihood of fixation on redundant parts and thus improving security. Additionally, for example, Figure 5B As shown, only the upper body portion is displayed. Figure 5B (b) → (a) can also achieve the same effect.

[0137] Furthermore, in the display control device of this embodiment, when the intelligent agent image is displayed on the display device 70 in a second display mode, the control unit 41 controls the movement of the body parts of the person represented by the character according to the driving load. For example, when the driving load is high, it controls the amount of movement of the intelligent agent image (the body parts of the character) by reducing the amount of movement (e.g., when the driving load is high). Figure 6 (a)→ Figure 6 (b) This reduces the eye-catching nature of images of intelligent agents. Therefore, no further unnecessary eye-catching is generated under high driving loads, improving safety. For example, in Figure 6 In (a) and (b), while this is achieved by reducing the amount of movement of the arm, which is a body part, it is not limited to this; for example, the movement speed of the other arm could also be reduced. On the other hand, under low driving load conditions, it is also possible to control the character's expression, such as making the character smile near the destination. In this case, the character's expression becomes frozen under high driving load conditions.

[0138] Furthermore, in the display control device of this embodiment, when the control unit 41 displays the intelligent agent image on the display device 70 in a second display mode, it controls the display size of the character according to the driving load. For example, when the driving load is high, by increasing the display size of each body part of the intelligent agent image, the general information can be grasped. In addition, when the driving load is high, it is assumed that the eyes are difficult to leave the front, but if the display size is increased, the general movement can be grasped by "looking to the side", thus reducing "eye movement deviation".

[0139] Furthermore, in the display control device of this embodiment, when the control unit 41 displays the agent image on the display device 70 in a second display mode, it controls the degree of orientation between the character and the occupant based on the driving load. For example, when the driving load is high, by reducing the orientation of the agent image towards the occupant, the feeling of being watched can be alleviated. For example, even in paintings with figures, people sometimes feel gazes; by performing the above control, the state of inattention or unnecessary distraction can be reduced.

[0140] Furthermore, in the display control device of this embodiment, when the control unit 41 displays the agent image in a first representation (geometric representation) on the display device 70, it controls the number of colors used for geometric representation according to the driving load. For example, when the driving load is high, by reducing the number of colors in the agent image, the possibility that the occupant interprets the meaning of the colors as beyond the meaning of the information provided by the character can be reduced.

[0141] Furthermore, in the display control device of this embodiment, when the intelligent agent image is displayed on the display device 70 in a first display mode, the control unit 41 controls the number of geometric representations based on the driving load. For example, when the driving load is high, it reduces the number of geometric representations used for the intelligent agent image. Figure 9 As shown in (a), in the case of 1 time (high driving load), the line is used, such as Figure 9 As shown in (b), in the secondary case (moderate driving load), polygons or circles are used, such as Figure 9 As shown in (c), stereoscopic representation is performed in three scenarios (low driving load). Therefore, the lower the resolution, the less information is used for representation, thus suppressing cognitive load. On the other hand, in scenarios with low driving load and sufficient capacity, the representation can be richer, thereby allowing the product to be perceived.

[0142] Furthermore, in the display control device 40 of this embodiment, when the control unit 41 displays the agent image on the display device in a first display mode, it controls the dynamic elements of the geometric representation according to the driving load. For example, when the driving load is high, it reduces the dynamic elements of the geometric representation of the agent image (e.g., shape changes or animations). Specifically, it reduces unnecessary visual distractions to the occupants by reducing the amount of shape changes and the update cycle of the dynamic representation, thereby contributing to safe driving.

[0143] Furthermore, in the display control device of this embodiment, when the control unit 41 switches the display of the agent image between the first display mode and the second display mode, the switching is performed when the occupant has not visually confirmed the agent image or has not interacted with the agent. Since the amount of image change caused by the switching is reduced, the occupant will not be aware of the abrupt switching.

[0144] Furthermore, in the display control device of this embodiment, when the control unit 41 switches between display images of an agent displayed in the first display mode or between display images of an agent displayed in the second display mode according to the increase or decrease of driving load (which sometimes causes migration between agent images displayed in the second display mode), it continuously changes the viewing angle and displays it on the display device 70, for example, as... Figure 10 As shown, the migration is achieved by continuously changing the viewing angle, such as zooming in and out. For example, zoom control is implemented when the driving load increases. Figure 10 (a)→ Figure 10 (b) migration), under conditions of reduced driving load, by implementing reduction control ( Figure 10 (b)→ Figure 10 (a) migration) enables the agent image to have an appropriate amount of information corresponding to the driving load.

[0145] Furthermore, in the display control device of this embodiment, when the control unit 41 determines the driving load based on the vehicle's state, it can make the determination based on at least one of the following: the degree of driving assistance, the steering wheel angle, the amount of throttle operation, or the amount of brake operation. This determination is based on the driving assistance device 10 (see reference 10) provided by the vehicle with driving assistance functions. Figure 1 The driving load can be easily and cost-effectively determined by information obtained from the vehicle's original onboard monitoring device 30, such as the behavior sensor 303.

[0146] Furthermore, in the display control device of this embodiment, when the control unit 41 determines the driving load based on the vehicle's driving environment, it does so by using map information containing information about areas where the driving load changes, or by using environmental information about the vehicle's surroundings monitored by sensors mounted on the vehicle during driving. For example, it can make the determination based on data stored in... Figure 1 The navigation device 20 shown can easily and cost-effectively determine the driving load by using the map information database 200, or by using a digital map obtained through communication with an external center (the diagram is omitted), or by using detection information obtained from the vehicle's original onboard monitoring device 30, such as the camera 301, GPS 302, behavior sensor 303, and LiDAR 304.

[0147] Furthermore, in the display control device of this embodiment, when the control unit 41 determines the driving load based on the state of the occupants of the vehicle, it can easily and cost-effectively determine the driving load by using occupant biometric information monitored by sensors worn on the occupants or mounted on the vehicle, or by analyzing occupant facial expressions.

[0148] The head-up display device (HUD device 70A) of this embodiment is, for example, as follows: Figure 11 The image shown is a head-up display (HUD device 70A) that overlays an agent image onto the field of view in front of the vehicle and projects it onto a virtual image plane set in front of the vehicle. Furthermore, the HUD device 70A includes an image display unit 73 for displaying the agent image and a control unit 71. The control unit 71 performs the following control: it determines the driving load based on at least one of the vehicle's state, the driving environment, or the state of the occupants driving the vehicle; it then changes the display mode of the agent image based on the determined driving load and displays the agent image on the image display unit 73 in the changed display mode.

[0149] According to the HUD device 70A of this embodiment, the control unit 71 performs the following control: it determines the driving load based on at least one of the vehicle's state, the driving environment, or the state of the occupants driving the vehicle; it changes the display mode of the agent image based on the determined driving load and displays the agent image on the image display unit 73 in the changed display mode. Thus, for example, by displaying geometric representations that do not interfere with the occupants' driving and only providing driving-related information, the occupants can concentrate on driving. Furthermore, by representing the agent image with an anthropomorphic character, for example, a HUD device 70A can be provided that makes the vehicle feel like a humanized carrier that evokes a sense of attachment. In this way, by switching and displaying the agent image's display mode according to the driving load, it is possible to balance preventing interference with driving with the positive feelings the agent evokes in the occupants (the degree to which the vehicle feels like a humanized carrier that evokes a sense of attachment).

[0150] Furthermore, in the HUD device 70A of this embodiment, when the driving load is high, the control unit 71 displays the agent image using a color close to the color of the object area overlapping the forward field of vision. When the driving load is low, it displays the agent image using a color based on the agent's role or design. Since the HUD device 70A displays the agent image overlapping the vehicle's forward field of vision, when the driving load is high, to avoid the agent obstructing driving, the agent image is displayed using the same color as the overlapping forward field of vision area (e.g., gray when overlapping with the road, the color of the vehicle when overlapping with the vehicle in front, etc.). When the driving load is low, by setting the color based on the agent's role or design, it is possible to balance preventing obstruction of driving and improving the agent's appeal. In addition, the HUD device 70A can display the image projected onto the image surface according to its display distance (from far to near), so it can be overlaid at intersections such as corners. When the driving load is low, by making the agent appear to be standing at the corner guiding (pointing out) the turning road, the appeal of the agent can be improved.

[0151] The program product of this embodiment is, for example, Figure 1 The image shown is a program product for a display control device 40 that controls a display device 70 that displays an image of an intelligent agent capable of exchanging information with occupants of the vehicle. Furthermore, this program product enables the display control device 40 to utilize a processor, for example... Figure 2 The following processes are performed: determining the driving load based on at least one of the vehicle's state, the driving environment, or the state of the occupants driving the vehicle (steps ST101 to ST104); and changing the representation of the agent image based on the determined driving load and displaying the agent image on the display device 70 in the changed representation (steps ST105 to ST111).

[0152] According to the program product of this embodiment, the display control device 40 has a processor, for example, that reads and executes data recorded in [the program]. Figure 1 The program products in certain areas of the storage unit 42 shown can, for example, allow occupants to concentrate on driving by using geometric representations that do not interfere with driving and only providing information about driving. Furthermore, by employing anthropomorphic character representations, the vehicle can be perceived as a humanized entity that evokes feelings of attachment. Thus, by switching the representation of the intelligent agent image according to driving load and displaying it, it is possible to balance preventing interference with driving with the positive feelings the intelligent agent evokes in the occupants (the degree to which the vehicle is perceived as a humanized entity that evokes feelings of attachment).

[0153] The in-vehicle intelligent agent system 100 of this embodiment is, for example, as follows: Figure 1The image shown is an in-vehicle intelligent agent system 100 that displays intelligent agent images within the vehicle cabin space and provides driving assistance to occupants. This in-vehicle intelligent agent system 100 includes: a head-up display (display device 70) that overlays the intelligent agent image onto the vehicle's forward field of view and projects it onto a virtual image plane positioned in front of the vehicle; a display control device 40 that performs the following controls: determines the driving load based on at least one of the vehicle's state, the driving environment, or the state of the occupants driving the vehicle; changes the display mode of the intelligent agent image based on the determined driving load; and displays the intelligent agent image on the head-up display (display device 70) in the changed display mode; and a driving assistance device 10 that cooperates with the display control device 40 to display the intelligent agent image within the vehicle cabin, conveys information through the intelligent agent image, and assists the occupants in driving operations by exchanging information with them.

[0154] According to the in-vehicle intelligent agent system 100 of this embodiment, the display control device 40, in cooperation with the driving assistance device 10, performs the following control: it determines the driving load based on at least one of the vehicle's state, the driving environment, or the state of the occupants driving the vehicle; it changes the representation of the intelligent agent image based on the determined driving load and displays the changed representation on the head-up display device (display device 70); it conveys information through the intelligent agent image and assists the occupants in driving by exchanging information with them. Therefore, for example, by using a geometric representation that does not hinder the occupants' driving and only providing information about driving, the occupants can concentrate on driving. Furthermore, for example, by setting the representation of the intelligent agent image to a human-like character representation, an in-vehicle intelligent agent system 100 can be provided that makes the vehicle feel like a humanized carrier that evokes a sense of attachment. In this way, by switching the representation of the intelligent agent image according to the driving load and displaying it, it is possible to balance preventing obstruction to driving with the positive impression the intelligent agent gives to the occupants (the degree to which the vehicle feels like a humanized carrier that evokes a sense of attachment).

[0155] This invention is not limited to the exemplary embodiments described above. Furthermore, those skilled in the art should be able to easily modify the exemplary embodiments described above within the scope of the claims.

[0156] Explanation of reference numerals in the attached figures 10: Driving assistance device; 20: Navigation device; 30: Vehicle monitoring device; 40: Display control device; 41: Control unit; 42: Storage unit; 50: Voice input / output control device; 60: I / O interface; 70: Display device; 70A: Head-up display (HUD device); 71: Control unit (HUD device side); 72: Storage unit (HUD device side); 73: Image display unit; 80: Voice input / output device; 90: Biometric sensor; 100: Vehicle intelligent system; 200: Map information database; 301: Camera; 302: GPS; 3 03: Behavior sensor; 304: LiDAR; 305: Viewpoint sensor; 411: Vehicle information acquisition unit; 412: Environmental information acquisition unit; 413: Occupant information acquisition unit; 414: Driver load determination unit; 415: Image switching unit; 416: Image generation unit; 417: Display control unit; 711: Vehicle information acquisition unit; 712: Environmental information acquisition unit; 713: Occupant information acquisition unit; 714: Driver load determination unit; 715: Image switching unit; 716: Image generation unit; 717: Display control unit; 731: Projection unit; 732: Liquid crystal display element.

Claims

1. A display control device for controlling a display device that displays an image of an intelligent agent capable of exchanging information with occupants of a vehicle, the display control device being characterized by comprising: Storage unit, which stores the image of the intelligent agent; and, The control unit performs the following control: determining the driving load based on at least one of the vehicle's state, the driving environment, or the state of the occupant driving; changing the display mode of the agent image based on the determined driving load; and displaying the agent image on the display device in the changed display mode.

2. The display control device according to claim 1, characterized in that, When the control unit determines that the driving load is high, it displays the intelligent agent image on the display device in a first representation of geometry; when it determines that the driving load is low, it displays the intelligent agent image on the display device in a second representation of an anthropomorphic character.

3. The display control device according to claim 2, characterized in that, When the image of the intelligent agent is displayed on the display device in the second representation mode, the control unit controls the display range of the body parts of the person represented by the character according to the driving load.

4. The display control device according to claim 2, characterized in that, When the image of the intelligent agent is displayed on the display device in the second representation mode, the control unit controls the movement of the body parts of the person represented by the character according to the driving load.

5. The display control device according to claim 2, characterized in that, When the image of the intelligent agent is displayed on the display device in the second representation mode, the control unit controls the display size of the character according to the driving load.

6. The display control device according to claim 2, characterized in that, When the control unit displays the image of the intelligent agent on the display device in the second representation mode, it controls the degree of antagonism between the character and the occupant based on the driving load.

7. The display control device according to claim 2, characterized in that, When the control unit displays the image of the intelligent agent on the display device in the first representation mode, it controls the number of colors used in the geometric representation according to the driving load.

8. The display control device according to claim 2, characterized in that, When the control unit displays the image of the intelligent agent on the display device in the first representation mode, it controls the number of times the geometric representation is performed according to the driving load.

9. The display control device according to claim 2, characterized in that, When the control unit displays the image of the intelligent agent on the display device in the first representation mode, it controls the dynamic elements of the geometric representation according to the driving load.

10. The display control device according to any one of claims 2 to 9, characterized in that, When the control unit switches the display of the agent image between the first display mode and the second display mode, the switching is performed when the occupant has not visually confirmed the agent image or has not engaged in dialogue with the agent.

11. The display control device according to any one of claims 2 to 9, characterized in that, When the control unit switches between displaying the agent images displayed in the first presentation mode or between displaying the agent images displayed in the second presentation mode according to the increase or decrease of the driving load, it continuously changes the viewing angle and displays it on the display device.

12. The display control device according to claim 1, characterized in that, When the control unit determines the driving load based on the vehicle's status, The determination is made based on at least one of the following: the level of driving assistance, the steering wheel angle, the amount of throttle input, or the amount of brake input.

13. The display control device according to claim 1, characterized in that, When the control unit determines the driving load based on the driving environment of the vehicle, The determination is made based on map information showing changes in driving load in the area, or on environmental information about the vehicle's surroundings monitored during driving by sensors mounted on the vehicle.

14. The display control device according to claim 1, characterized in that, When the control unit determines the driving load based on the state of the occupant driving the vehicle, The determination is made based on the occupant's biometric information monitored by sensors worn on the occupant or mounted on the vehicle, or by the analysis of the occupant's facial expressions.

15. A head-up display device that overlays an image of an intelligent agent onto the forward field of view of a vehicle and projects it onto a virtual image plane set in front of the vehicle, the head-up display device being characterized by comprising: An image display unit displays an image of the intelligent agent; and, The control unit performs the following control: determining the driving load based on at least one of the vehicle's state, the driving environment, or the state of the occupant driving; changing the display mode of the agent image based on the determined driving load; and displaying the agent image on the display device in the changed display mode.

16. The head-up display device according to claim 15, characterized in that, When the control unit determines that the driving load is high, it uses a color close to the color of the object area in the overlapping forward field of view to represent the agent image; when it determines that the driving load is low, it uses a color based on the role or design of the agent image to represent the agent image.

17. A program product for a display control device, the display control device controlling a display device that displays an image of an intelligent agent capable of exchanging information with occupants of a vehicle, the program product causing a processor of the display control device to perform the following processing: The driving load is determined based on at least one of the vehicle's state, the driving environment, or the state of the occupant driving; and the following control is performed: the representation of the agent image is changed according to the determined driving load, and the agent image is displayed on the display device in the changed representation.

18. A vehicle-mounted intelligent agent system, which configures intelligent agent images within the vehicle cabin space and provides driving assistance to occupants, characterized in that it comprises: A head-up display device that overlays the image of the intelligent agent onto the vehicle's forward field of view and projects it onto a virtual image plane set in front of the vehicle. The display control device performs the following control: determining the driving load based on at least one of the vehicle's state, the driving environment, or the state of the occupant driving; changing the display mode of the agent image based on the determined driving load; and displaying the agent image on the head-up display device in the changed display mode; and... A driving assistance device that cooperates with the display control device to display the intelligent agent image in the vehicle interior, conveys information through the intelligent agent image, and assists the occupant in driving the vehicle by exchanging information with the occupant.

Images