Information processing method, information processing device, and program

Abstract

To increase a sense of unity of a driver and an agent by adjusting the presence of the agent according to necessity, and increase a capability of transmitting information to be transmitted from the agent to the driver.SOLUTION: An information processing method includes: determination processing (steps S516, S517) of determining whether predetermined information to be presented to a driver D1 is present; and control processing of, when the predetermined information is present, in a state in which an agent (agent images AG1, AG2) is visible from the driver D1 utilizing a first output device 200 (or a second output device 210) located at a place that is easily noticeable to the driver D1, causing the predetermined information to be presented using the agent images AG1, AG2 (steps S518 to S521), and when the predetermined information is not present, executing presentation for making the driver feel the presence of the agent in an aspect (heat generation) which is less easily recognized by the driver D1 utilizing a peripheral device 220 (steps S525 to S527).SELECTED DRAWING: Figure 12

Description

Technical Field

[0001] The present invention relates to an information processing method, an information processing apparatus, and a program for controlling an agent capable of communicating with a user.

Background Art

[0002] Conventionally, techniques for transmitting various types of information to a user in a vehicle have been proposed. For example, a technique has been proposed in which a character image (agent image) is displayed on a display unit to assist the user in driving the vehicle (see, for example, Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In the above-described conventional technology, the agent image continues to be displayed not only when guiding information to the user but also during standby. In this way, in order to make the user feel as if the agent is with them, if the agent image is kept appearing without being moved significantly, the user may feel bothered by the presence of the agent image. Therefore, it is conceivable to erase the agent image when not guiding information to the user. However, if the agent image is erased, the user will not be able to recognize the existence of that agent, making it difficult for the user to feel as if that agent is with them. In this case, the reliability, affection, etc. for that agent may decrease, and there is a possibility that the transmission ability of the information transmitted from that agent to the user may decrease.

[0005] An object of the present invention is to enhance the sense of unity between a driver and an agent and improve the transmission ability of information transmitted from the agent to the driver by adjusting the presence of the agent as needed.

Means for Solving the Problems

[0006] One aspect of the present invention is an information processing method for executing control for communicating with a driver by using a plurality of devices in a vehicle. This information processing method includes a determination process for determining whether there is predetermined information to be presented to the driver, and when there is the predetermined information, using a first device located in a place where it is easy to enter the driver's vision among the plurality of devices, presenting the predetermined information using an agent in a state where the agent for communication is visible to the driver, and when there is no predetermined information, using a second device other than the first device among the plurality of devices, and a control process for executing an effect that makes the driver feel the presence of the agent in a mode that is difficult to be recognized by the driver.

Effects of the Invention

[0007] According to the present invention, by adjusting the presence of the agent as needed, it is possible to enhance the sense of unity between the driver and the agent and improve the transmission ability of information transmitted from the agent to the driver.

Brief Description of the Drawings

[0008]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

MODE FOR CARRYING OUT THE INVENTION

[0009] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings.

[0010] [Display Example of Agent Image] FIGS. 1 to 4 are diagrams showing a simplified configuration example of the interior of the vehicle C1. In FIGS. 1 to 4, the interior of the vehicle C1 when viewed from the rear side in the front-rear direction of the vehicle C1, which is in front of the driver's seat and the passenger seat (not shown), is shown in a simplified manner. In FIG. 1, for ease of explanation, illustrations other than the dashboard 2, the steering wheel 3, the front window 4, the rearview mirror 5, the shift lever 6, the first output device 200, and the second output device 210 are omitted. In the examples shown in FIGS. 1 to 4, for ease of explanation, only the upper part of the shift lever 6 is shown in a simplified manner as the configuration of the shift lever 6.

[0011] The first output device 200 and the second output device 210 are output devices installed inside the vehicle C1, and execute various output operations based on instructions from the information processing device 110 (see FIG. 3). The first output device 200 is a device installed near the center of the dashboard 2, and includes a display unit 201 and a sound output unit 202 (see FIG. 5). Further, the second output device 210 is a device installed on the steering wheel 3, and includes a display unit 211 and a sound output unit 212 (see FIG. 5).

[0012] For example, the first output device 200 and the second output device 210 are in-vehicle systems composed of one or more devices capable of providing various information. As the first output device 200 and the second output device 210, for example, at least one of a navigation device, an audio device, a DVD device, a TV tuner device, an IVI (In-Vehicle Infotainment), etc. can be used. Regarding the images to be displayed on each of the first output device 200 and the second output device 210, it is also possible to display them using a HUD (Head Up Display) realized on the front windshield 4 (see FIG. 4). Further, a portable device (or a device that can be installed in the vehicle C1) that the driver D1 can possess, for example, an information processing device such as a smartphone, a tablet terminal, a portable personal computer, etc. may be used.

[0013] In addition, the first output device 200 and the second output device 210 perform output operations capable of communicating with a user (including driver D1) riding in the vehicle C1. For example, on the display unit 201 of the first output device 200, an agent image AG1 (see FIG. 1) capable of communicating with the user is displayed, and various effects using the agent image AG1 are executed. Also, for example, on the display unit 211 of the second output device 210, an agent image AG2 (see FIG. 2) capable of communicating with the user is displayed, and various effects using the agent image AG2 are executed. Also, for example, on the front windshield 4 (HUD display area), an agent image AG3 (see FIG. 4) capable of communicating with the user is displayed, and various effects using the agent image AG3 are executed. Note that for the image displayed on the front windshield 4 (HUD display area), a third output device (not shown) executes various output operations based on an instruction from the information processing device 110 (see FIG. 3). The third output device is a display device for realizing HUD display using a known HUD technology. That is, the first output device 200, the second output device 210, and the front windshield 4 (HUD display area) function as agent devices capable of communicating with the user. Note that hereinafter, mainly an example of causing the first output device 200 and the second output device 210 to function as agent devices is shown, but the front windshield 4 (HUD display area) can also function as an agent device in the same manner.

[0014] Agent images AG1 to AG3 are images representing objects capable of communicating with a user. In the present embodiment, an example is shown in which images imitating a human face are used as agent images AG1 to AG3. However, agent images AG1 to AG3 are not limited to this. For example, images imitating the whole body or a part (e.g., the upper body) of a human may be used as agent images AG1 to AG3, or images imitating animals such as rabbits or pigs (or the faces of such animals), or images imitating virtual organisms (e.g., the faces of anime characters) (or their faces) may be used as agent images AG1 to AG3. Thus, it is possible to use anthropomorphic images as agent images AG1 to AG3.

[0015] The first output device 200 and the second output device 210 execute various operations related to driving support (e.g., operations related to agent images AG1 and AG2) based on instructions from the information processing device 110 (see FIG. 3). For example, the first output device 200 and the second output device 210 output various information related to driving support and various information related to surrounding facilities when the driver D1 performs a driving operation under the control of the information processing device 110. In the present embodiment, when the driving load of the driver D1 is equal to or greater than a threshold value, various information is output using the agent image AG1 displayed on the first output device 200, and when the driving load of the driver D1 is less than the threshold value, an example is shown in which various information is output using the agent image AG2 displayed on the second output device 300. Note that when the driving load of the driver D1 is equal to or greater than the threshold value, for example, the vehicle C1 is in motion. Also, when the driving load of the driver D1 is less than the threshold value, for example, the vehicle C1 is stopped. Note that the method for detecting the driving load of the driver D1 will be described later.

[0016] In addition, as driving support using the agent image AG1 or the like, notification of a moving object in front or behind is assumed. For example, as notification of a moving object in front, it is possible to output a voice such as "Be careful because there is a railroad crossing ahead" or "There is a person ahead". It is possible to execute various operations using the agent image AG1 or the like at the time of each of these outputs. That is, it is possible to execute driving support using the agent image AG1 or the like. Further, for example, the first output device 200 and the second output device 210 execute various processes using the agent image AG1 or the like based on the control of the information processing device 110. For example, the first output device 200 and the second output device 210 execute communication processing for performing various exchanges with the driver D1 using the agent images AG1 and AG2. For example, the first output device 200 and the second output device 210 perform various conversations with the driver D1 or provide information preferred by the driver D1 using the agent images AG1 and AG2.

[0017] Here, in the present embodiment, when the driving load of the driver D1 is equal to or greater than the threshold value and there is no predetermined information to be presented to the driver D1, agent images are not displayed on both the first output device 200 and the second output device 210, and an effect for making the driver feel the presence of the agent is executed using other devices. As the other device, for example, a peripheral device installed around the driver D1 and capable of being touched by the driver D1 can be used. As this device, for example, at least one of the shift lever 6, the steering wheel 3, the seating seat, the armrest, and the seat belt can be used. Note that FIG. 3 shows an example of using the shift lever 6. Further, using these devices, an effect for making the driver feel the presence of the agent in a mode that is difficult to be recognized by the driver D1 is executed. Among the peripheral devices installed around the driver D1, it is also possible to use a device (for example, a seating seat, an armrest, a seat belt) located in a place where it is difficult to enter the driver D1's vision.

[0018] The effect of making the driver feel the presence of the agent means, for example, making the driver feel the presence of the agent by using the tactile sensation at the contact part of the peripheral device 220. For example, it is possible to adjust the temperature of the device to make the driver feel the presence of the agent at the temperature of the device. For example, a heater part 221 (see FIG. 5) can be provided on the shift lever 6, and the heater part 221 can be made to generate heat. Thereby, it is possible to make the driver D1 who touches the shift lever 6 feel a biological warmth, and it is possible to generate a sense of presence of the agent at the shift lever 6. Note that the temperature when the shift lever 6 generates heat is preferably set to a temperature that can make the driver feel a biological warmth (for example, around 30 to 40 degrees). Also, these temperatures can be appropriately set based on experiments, simulations, or the like.

[0019] Similarly, for each of the steering wheel 3, the seating seat, the armrest, and the seat belt, it is also possible to adjust the temperature of each device to make the driver feel the presence of the agent at the temperature of the device. For example, for the steering wheel 3, a heater part 221 (for example, a heated seat) can be provided at the part where the driver D1 grips. Also, for the seating seat, a heater part 221 (for example, a heated seat) can be provided at the part where the driver D1's body comes into contact (for example, the part where the back and buttocks come into contact). Also, for the armrest, a heater part 221 (for example, a heated seat) can be provided at the part where the driver D1's body comes into contact (for example, the part where the elbow comes into contact). Also, for the seat belt, a heater part 221 (for example, a heated seat) can be provided at the part where the driver D1's body comes into contact.

[0020] Here, when expressing the presence of an agent in peripheral devices (e.g., shift lever 6, steering wheel 3, seating seat, armrest, seat belt), after a movement effect from another output device, an effect (i.e., heat emission) that continuously expresses the presence of the agent may be executed, or an effect (i.e., heat emission) that intermittently expresses the presence of the agent may be executed as necessary. For example, for the part touched by driver D1, an effect (i.e., heat emission) that expresses the presence of the agent may be executed. For example, when driver D1 holds the steering wheel 3 with both hands, it is possible to execute an effect (i.e., heat emission) that expresses the presence of the agent at the part of the steering wheel 3 held by the hands of driver D1. Also, for example, when one hand of driver D1 holds the shift lever 6, it is possible to execute an effect (i.e., heat emission) that expresses the presence of the agent at the part of the shift lever 6 held by the hand of driver D1.

[0021] Note that for the part in contact with driver D1, it can be detected based on the contact sensors provided in each device. Also, based on the image of driver D1 acquired by the driver situation acquisition unit 122, the part in contact with driver D1 may be detected. In this detection process, it is possible to use known image recognition techniques.

[0022] Also, if necessary, the presence of the agent may be moved. For example, after the driver D1 holds the steering wheel 3 with both hands, if one hand of the driver D1 holds the shift lever 6, it is possible to generate heat at the portion of the shift lever 6 held by the hand of the driver D1 along with the heat generation in the steering wheel 3. That is, by instantaneously warming the newly touched portion by the driver D1, it is possible to express the presence of the agent. In this way, the presence of the agent may be expressed by following the movement of the driver D1. In this case, following the movement of the driver D1, the device at the destination of the movement may be heated, and the heat generation of the device at the source of the movement may be stopped. For example, after the driver D1 holds the steering wheel 3 with both hands, if one hand of the driver D1 holds the shift lever 6, it is possible to stop the heat generation in the steering wheel 3 and generate heat at the portion of the shift lever 6 held by the hand of the driver D1.

[0023] In the above, as an effect for making the driver feel the presence of the agent, an example of adjusting the temperature of the peripheral device 220 (for example, the shift lever 6) to make the driver feel the presence of the agent by the temperature of the device has been shown, but the present embodiment is not limited to this. For example, by performing at least one of a vibration process for vibrating the peripheral device 220, a sound output process for outputting a predetermined sound capable of recognizing the presence of the agent from the peripheral device 220 or its direction, a light output process for outputting a predetermined light capable of recognizing the presence of the agent from the peripheral device 220 or its direction, a smell generation process for generating a predetermined smell capable of recognizing the presence of the agent from the peripheral device 220 or its direction, a reaction process for changing the reaction from the peripheral device 220 or its direction, and a movement process for moving the agent from another device that has expressed the agent immediately before to the peripheral device 220, the effect may be executed. In these cases, an effect for making the driver feel the presence of the agent is executed using at least one of the five senses (vision, hearing, touch, taste, smell) of the driver D1.

[0024] For example, by providing a vibrating device with a vibration function in the peripheral device 220, it is possible to execute a vibration process for vibrating the peripheral device 220. For example, by generating vibrations with the vibrating device to such an extent that the presence of the agent can be recognized, it is possible to execute an effect that expresses the presence of the agent. Note that the effect of expressing the presence of the agent by heat generation, vibration, etc. is an example of an expression method using the sense of touch among the five senses of the driver D1.

[0025] Also, for example, by providing a speaker (sound output device) in the peripheral device 220 or in its direction (the direction based on the driver D1), it is possible to execute a sound output process for outputting a predetermined sound that can recognize the presence of the agent from the peripheral device 220 or in its direction. In this case, it is not necessary to provide a speaker in each peripheral device 220, and if it is possible to make the driver D1 feel as if sound is emitted from the peripheral device 220, it is possible to provide a speaker near each peripheral device 220. For example, by outputting sound from the speaker to such an extent that the presence of the agent can be recognized, it is possible to execute an effect that expresses the presence of the agent. As the sound to be output in this case, for example, a sound related to the agent (for example, the voice quality of the agent, the entrance sound or exit sound of the agent) can be used. Note that the effect of expressing the presence of the agent by sound, etc. is an example of an expression method using the sense of hearing among the five senses of the driver D1.

[0026] Also, for example, by providing a light-emitting device (e.g., an LED (Light Emitting Diode)) in the peripheral device 220 or in its direction (the direction based on the driver D1), it is possible to execute a light output process that outputs predetermined light capable of recognizing the presence of the agent from the peripheral device 220 or its direction. In this case, it is not necessary to provide a light-emitting device in the peripheral device 220. As long as it is possible to make the driver D1 feel as if light is emitted from the peripheral device 220, it is possible to provide a light-emitting device near each peripheral device 220. For example, by outputting light from the light-emitting device to such an extent that the presence of the agent can be recognized, it is possible to execute an effect that represents the presence of the agent. As the light to be output in this case, for example, light of a color related to the agent (e.g., the color of the agent itself, the color of the agent's ornaments, the background color of the agent) can be used. Note that the effect of representing the presence of the agent with light or the like is an example of a method of expression using the sense of sight among the five senses of the driver D1.

[0027] Also, for example, by providing an odor generating device in the peripheral device 220 or in its direction (the direction based on the driver D1), it is possible to execute an odor generating process that generates a predetermined odor that can recognize the presence of the agent from the peripheral device 220 or its direction. In this case, it is not necessary to provide an odor generating device in the peripheral device 220, and if it is possible to make the driver D1 feel as if an odor is emitted from the peripheral device 220, it is possible to provide an odor generating device near the peripheral device 220. Here, it is often difficult to determine the position where the odor is generated based on the odor that reaches humans. Therefore, when using odor, it is possible to adjust the distance, direction, etc. from the driver D1 to the agent by the strength, direction, etc. of the odor. For example, an odor generator is provided in the air conditioner (aircon) installed in the vehicle C1 so that the wind blows from the direction indicating the presence of the agent. Or, the wind may be sent in the direction indicating the presence of the agent. Also, when setting the position indicating the presence of the agent to a position close to the driver D1, it is possible to send a strong wind from the headrest side and generate a strong odor. On the other hand, when setting the position indicating the presence of the agent to a position far from the driver D1, it is possible to send a weak wind from the dashboard 2 and generate a faint odor. Thus, by outputting an odor to such an extent that the presence of the agent can be known from the odor generating device, it is possible to execute an effect that expresses the presence of the agent. As the odor to be output in this case, for example, an odor related to the agent (for example, the odor of perfume that matches the impression in the case of an agent with a feminine image, the odor of sweat in the case of an agent with a sporty image, the odor of food and drink (for example, ginger grilled) in the case of an agent with an image of having a strong appetite) can be used. Note that the effect of expressing the presence of the agent by odor or the like is an example of a method of expression using the sense of smell among the five senses of the driver D1.

[0028] Also, for example, by providing a device that changes the reaction from the peripheral device 220 or its direction (the direction based on the driver D1), it is possible to execute a reaction process that changes the reaction from the peripheral device 220 or its direction. The reaction shown here means, for example, changing the firmness of the seating seat to be firmer, changing the tightening feeling of the seat belt to be firmer, etc. Note that the effect that expresses the presence of the agent by reaction or the like is an example of a method of expression using the sense of touch among the five senses of the driver D1.

[0029] Also, for example, it is possible to execute a movement process in which the agent moves from another device that has expressed the agent immediately before to the peripheral device 220. This movement process will be described in detail with reference to FIGS. 7 to 10.

[0030] Also, for example, the presence of the agent may be expressed by using the air pressure of the air conditioner. For example, when the agent moves, it is possible to execute an effect in which the air pressure of the air conditioner is temporarily increased and the agent moves onto the air conditioner.

[0031] Here, when expressing the presence of the agent by light, it may be emitted gently and intermittently by blinking or the like, or weak light that only makes the presence felt may be continuously emitted. Also, when expressing the presence of the agent by sound, it is not necessary to continuously produce sound, and it is preferable to output sound as needed. For example, when moving the agent, in order to make the movement felt, it is possible to output voice information such as "I'm going down now" and "I'm going to move horizontally now". Also, when expressing the presence of the agent by heat, it is preferable to continuously generate heat.

[0032] Here, if the agent images AG1 and AG2 are left visible in order to make the driver D1 feel as if the agent is with him / her, the driver D1 may find the presence of the agent images AG1 and AG2 bothersome. Therefore, when not guiding information to the driver D1, it is conceivable to erase the agent images AG1 and AG2. However, if the agent images AG1 and AG2 are erased, the driver D1 will not be able to recognize the presence of the agent, making it difficult for the driver D1 to feel as if the agent is with him / her. In this case, the reliability and affection towards the agent may decrease, and the transmission ability of the information transmitted from the agent to the driver D1 may decrease.

[0033] Therefore, in this embodiment, by adjusting the presence of the agent as needed, the sense of unity between the driver D1 and the agent is enhanced, and the transmission ability of the information transmitted from the agent to the driver D1 is enhanced.

[0034] [Configuration Example of Information Processing System] FIG. 5 is a block diagram showing an example of the system configuration of an information processing system 100 installed in a vehicle C1.

[0035] The information processing system 100 includes a sound acquisition unit 101, a driver image acquisition unit 102, a passenger compartment image acquisition unit 103, an outside vehicle image acquisition unit 104, an information processing device 110, a first output device 200, a second output device 210, and peripheral devices 220. The information processing device 110 is an example of a device that controls the first output device 200, the second output device 210, and the peripheral devices 220 capable of communicating with the driver D1.

[0036] Note that the information processing apparatus 110, the first output device 200, the second output device 210, and the peripheral device 220 are connected by a communication method using wired communication or wireless communication. Further, the information processing apparatus 110 is connected to the network 20 by a communication method using wireless communication. The network 20 is a network such as a public switched telephone network or the Internet. Note that the first output device 200, the second output device 210, and the peripheral device 220 may also be connected to the network 20 by a communication method using wireless communication. In FIG. 5, an example is shown in which the information processing apparatus 110, the first output device 200, and the second output device 210 are configured as separate bodies, but the information processing apparatus 110, the first output device 200, and the second output device 210 may be configured as an integrated device.

[0037] The sound acquisition unit 101 is provided inside the vehicle C1 and acquires the sound inside the vehicle C1, and outputs sound information regarding the acquired sound to the information processing apparatus 110. As the sound acquisition unit 101, for example, one or a plurality of microphones or sound acquisition sensors can be used.

[0038] The driver image acquisition unit 102 captures an image of the driver D1 riding in the vehicle C1 to generate an image (image data), and outputs image information regarding the generated image to the information processing apparatus 110. The driver image acquisition unit 102 is provided at least inside the vehicle C1 and captures an image of the driver D1 riding in the vehicle C1 to generate an image (image data). The driver image acquisition unit 102 is configured by, for example, one or a plurality of camera devices or image sensors capable of capturing an image of the driver D1. For example, one driver image acquisition unit 102 can be provided in the front (for example, the ceiling) inside the vehicle C1 to capture an image of the driver D1 from the front of the vehicle C1 and generate an image (image data). For example, the driver image acquisition unit 102 can be provided above the front windshield 4, that is, above the rearview mirror 5. Note that the driver image acquisition unit 102 and the passenger compartment image acquisition unit 103 may use the same device or may be configured as different devices.

[0039] The passenger compartment image acquisition unit 103 captures an object inside the vehicle C1 to generate an image (image data), and outputs image information regarding the generated image to the information processing device 110. The passenger compartment image acquisition unit 103 is provided at least inside (for example, on the ceiling) of the vehicle C1, and captures an object inside the vehicle C1 to generate an image (image data). The passenger compartment image acquisition unit 103 is constituted by, for example, one or a plurality of camera devices and image sensors capable of capturing an object. For example, one passenger compartment image acquisition unit 103 may be provided in front of the vehicle C1 to capture an object from the front to generate an image (image data), and another passenger compartment image acquisition unit 103 may be provided behind the vehicle C1 to capture an object from the rear to generate an image (image data).

[0040] The outside vehicle image acquisition unit 104 captures an object outside the vehicle C1 to generate an image (image data), and outputs image information regarding the generated image to the information processing device 110. Note that two or more outside vehicle image acquisition units 104 may be provided, and all or some of the images of these outside vehicle image acquisition units 104 may be used. For example, one outside vehicle image acquisition unit 104 may be provided in front of the vehicle C1 to capture an object from the front of the vehicle C1 to generate an image (image data), and another outside vehicle image acquisition unit 104 may be provided behind the vehicle C1 to capture an object behind the vehicle C1 to generate an image (image data). Also, one or a plurality of devices capable of acquiring an object existing in all directions around the vehicle C1 and an object inside the vehicle C1, for example, a 360-degree camera, may be used.

[0041] Note that the driver image acquisition unit 102, the passenger compartment image acquisition unit 103, and the outside vehicle image acquisition unit 104 are constituted by, for example, an image sensor (image sensor) that receives light from an object condensed by a lens, and an image processing unit that performs predetermined image processing on the image data generated by the image sensor. As the image sensor, for example, a CCD (Charge Coupled Device) type or CMOS (Complementary Metal Oxide Semiconductor) type image sensor can be used.

[0042] The information processing apparatus 110 includes a control unit 120, a storage unit 130, and a communication unit 140. The communication unit 140 exchanges various types of information with other devices using wired communication or wireless communication based on the control of the control unit 120.

[0043] The control unit 120 controls each unit based on various programs stored in the storage unit 130. The control unit 120 is realized by a processing device such as a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit), for example. Note that the vehicle ECU (Electronic Control Unit) of the vehicle C1 may also be used as the control unit 120, or a processing device different from the vehicle ECU may be provided as the control unit 120.

[0044] The control unit 120 executes various controls based on the information output from the sound acquisition unit 101, the driver image acquisition unit 102, the passenger compartment image acquisition unit 103, the outside vehicle image acquisition unit 104, the communication unit 140, etc., and the information acquired by the vehicle information acquisition unit 125. For example, the control unit 120 executes control processing for controlling the operating states of the first output device 200, the second output device 210, and the peripheral device 220. Specifically, the control unit 120 includes a speech acquisition unit 121, a driver situation acquisition unit 122, a passenger compartment situation acquisition unit 123, an outside vehicle situation acquisition unit 124, a vehicle information acquisition unit 125, a communication content determination unit 126, an output mode determination unit 127, and an agent control unit 128.

[0045] The speech acquisition unit 121 acquires speech information regarding the speech of each user (including the driver D1) included in the sound information by performing predetermined sound analysis processing on the sound information output from the sound acquisition unit 101, and outputs the speech information to the communication content determination unit 126. This sound analysis processing can use known sound analysis processing.

[0046] The driver status acquisition unit 122 executes a predetermined image analysis process on the image information output from the driver image acquisition unit 102 to acquire various types of information about the driver D1 included in the image information, and outputs each piece of information to the communication content determination unit 126. This image analysis process can use known image analysis processes. As various types of information about the driver D1, for example, it is possible to acquire each action of the driver D1 regarding the expression of the driver D1, the line of sight of the driver D1, and the movements of the hands, face, body, etc. By these, for example, it is possible to detect the boarding and alighting of the driver D1 to / from the vehicle C1, the presence or absence of seating in each seat in the vehicle C1, the action of fastening the seat belt, etc. That is, the driver status information regarding the state of the driver D1 is acquired by the driver status acquisition unit 122.

[0047] The passenger compartment status acquisition unit 123 executes a predetermined image analysis process on the image information output from the passenger compartment image acquisition unit 103 to acquire various types of information regarding the interior of the vehicle C1 included in the image information, and outputs each piece of information to the communication content determination unit 126. This image analysis process can use known image analysis processes. As various types of information regarding the interior of the vehicle C1, for example, it is possible to acquire each action of each user regarding the expression of each user, the line of sight of each user, and the movements of the hands, face, body, etc. That is, the user status information regarding the state of each user boarding the vehicle C1 and the vehicle status information regarding the state of the vehicle C1 are acquired by the passenger compartment status acquisition unit 123.

[0048] The external situation acquisition unit 124 executes predetermined image analysis processing on the image information output from the external image acquisition unit 104 to acquire various types of information regarding the exterior of the vehicle C1 included in the image information, and outputs each piece of this information to the communication content determination unit 126. This image analysis processing can use known image analysis processing. As various types of information regarding the exterior of the vehicle C1, for example, it is possible to detect whether the vehicle C1 is traveling on the road, whether the vehicle C1 is parked on the road, signals, signs, etc. existing in front of the vehicle C1. That is, the vehicle state information regarding the state of the vehicle C1 is acquired by the external situation acquisition unit 124.

[0049] The vehicle information acquisition unit 125 acquires various types of information (vehicle state information) regarding the vehicle state of the vehicle C1, and outputs the acquired vehicle state information to the communication content determination unit 126. The vehicle state information can be acquired from, for example, a CAN (Controller Area Network) signal. Note that the vehicle state information is, for example, vehicle speed, acceleration, the position of the shift lever (for example, P range, D range), the depression amount of the accelerator pedal, the depression amount of the brake pedal, position information, etc. For example, it is possible to determine whether the vehicle C1 is stopped or traveling based on the vehicle speed, acceleration, etc.

[0050] In addition, the vehicle information acquisition unit 125 may acquire sensor detection information output from various sensors installed in the vehicle C1. Examples of the sensors include LiDAR (Light Detection And Ranging), RADAR (Radio Detection And Ranging), Sonar, vehicle speed sensor, acceleration sensor, steering sensor, accelerator position sensor, position information acquisition sensor (position information acquisition unit), seating sensor, seat belt sensor, etc. Known sensors can be used for each of these sensors. Note that LiDAR, RADAR, Sonar, etc. are examples of sensors that detect the situation around the vehicle C1. Also, the vehicle speed sensor, acceleration sensor, steering sensor, accelerator position sensor, etc. are examples of sensors that detect the situation of the driving operation of the driver D1. Note that these are just examples, and other sensors may be used. Also, only some of these sensors may be used.

[0051] The position information acquisition unit acquires position information regarding the position where the vehicle C1 is located. For example, it can be realized by a GNSS receiver that acquires position information using GNSS (Global Navigation Satellite System). The position information includes each data regarding the position such as latitude, longitude, altitude, etc. at the time of receiving the GNSS signal. Also, position information may be acquired by other methods of acquiring position information. For example, position information may be derived using information from access points and base stations existing around. Also, position information may be acquired using a beacon.

[0052] The seating sensor (or seat sensor) is a sensor that detects the presence or absence of an occupant sitting on each seat of the vehicle C1. The seat belt sensor is a sensor that detects whether or not an occupant sitting on each seat of the vehicle C1 is wearing a seat belt. For example, regarding the presence or absence of wearing the seat belt of the driver D1 sitting on the driver's seat, it can be detected using a seating sensor, a seat belt sensor, etc.

[0053] The communication content determination unit 126 determines the content of communication with any user riding in the vehicle C1 based on each piece of information output from the utterance acquisition unit 121, the driver status acquisition unit 122, the vehicle interior status acquisition unit 123, the vehicle exterior status acquisition unit 124, the vehicle information acquisition unit 125, and the like. Then, the communication content determination unit 126 outputs those pieces of information and the determined communication content to the output mode determination unit 127 and the agent control unit 128. For example, the communication content determination unit 126 acquires at least one of user status information regarding the status of each user (including the driver D1) riding in the vehicle C1 and vehicle status information regarding the status of the vehicle C1. Then, the communication content determination unit 126 determines the content of communication with any user based on at least one of the acquired user status information and vehicle status information. For example, when support information to be conveyed to the driver D1 is detected based on the user status information or the vehicle status information, the communication content determination unit 126 determines to convey the support information to the driver D1. Also, for example, when it is detected that a predetermined communication (for example, a conversation) is to be executed with any user riding in the vehicle C1 based on the user status information or the vehicle status information, the communication content determination unit 126 determines to execute the communication.

[0054] The output mode determination unit 127 determines the mode for outputting the agent (the expression mode of the agent) based on the content of the communication determined by the communication content determination unit 126 and the driving load of the driver D1. Then, the output mode determination unit 127 outputs the determined expression mode of the agent to the agent control unit 128. For example, the output mode determination unit 127 can detect the driving load of the driver D1 based on at least one of the environmental information regarding the environment outside the vehicle C1 (obtainable by the vehicle exterior situation acquisition unit 124) and the driving behavior information regarding the driving behavior of the driver D1 (obtainable by the driver situation acquisition unit 122 and the vehicle information acquisition unit 125). The method for detecting the driving load of the driver D1 will be described later. The expression mode of the agent will be described in detail with reference to FIG. 6.

[0055] The agent control unit 128 controls the operating states of the first output device 200, the second output device 210, and the peripheral device 220 based on the content of the communication determined by the communication content determination unit 126 and the expression mode of the agent determined by the output mode determination unit 127. Each of these operations will be described in detail with reference to FIGS. 6 to 10 and the like.

[0056] The storage unit 130 is a storage medium that stores various types of information. For example, the storage unit 130 stores various types of information (e.g., control programs, agent information DB 131, agent management DB 132, map information DB) necessary for the control unit 120 to perform various processes. Also, the storage unit 130 stores various types of information acquired via the communication unit 140. As the storage unit 130, for example, ROM (Read Only Memory), RAM (Random Access Memory), SRAM (Static Random Access Memory), HDD (Hard Disk Drive), SSD (Solid State Drive), or a combination thereof can be used.

[0057] The agent information DB 131 stores various types of information necessary to implement various operations of the first output device 200, the second output device 210, and the peripheral device 220. For example, the image information regarding the agent images AG1 and AG2 to be displayed on the display unit 201 of the first output device 200 and the display unit 211 of the second output device 210, and the expression information for expressing the agent in the peripheral device 220 are stored in the agent information DB 131. Also, for example, the operation information for operating the agent in the first output device 200 or the second output device 210 when executing various communications is stored in the agent information DB 131.

[0058] The agent management DB 132 stores agent management information for managing the expression location of the agent. For example, information indicating whether the agent is expressed at any one of the first to third positions (see FIG. 6) is managed as the agent management information.

[0059] The first output device 200 is a device capable of displaying the agent image AG1 based on an instruction from the information processing device 110 and transmitting various types of information to the driver D1 etc. using the agent image AG1. Also, the second output device 210 is a device capable of displaying the agent image AG2 based on an instruction from the information processing device 110 and transmitting various types of information to the driver D1 etc. using the agent image AG2.

[0060] The first output device 200 includes a display unit 201 and a sound output unit 202. Note that the display unit 201 and the sound output unit 202 are controlled based on a control unit (not shown) provided in the first output device 200. Also, the second output device 210 includes a display unit 211 and a sound output unit 212. Note that the display unit 211 and the sound output unit 212 are controlled based on a control unit (not shown) provided in the second output device 210.

[0061] The display units 201 and 211 are display units that display various images based on instructions from the information processing apparatus 110. Note that the display units 201 and 211 may be configured as touch panels that enable operation input when the user touches or brings their finger close to the display surface, or may be configured as separate user interfaces.

[0062] The sound output units 202 and 212 output various sounds based on instructions from the information processing apparatus 110. As the sound output units 202 and 212, for example, one or more speakers can be used. Note that the display unit 201, the sound output units 212 and 202 are examples of user interfaces, and a part of these may be omitted, or other user interfaces may be used.

[0063] The peripheral device 220 is a device capable of expressing the presence of an agent based on instructions from the information processing apparatus 110. The peripheral device 220 is, for example, at least one of the shift lever 6, the steering wheel 3, the seating seat, the armrest, and the seat belt. The peripheral device 220 includes a heater unit 221. Note that the heat generation of the heater unit 221 is controlled based on the information processing apparatus 110. In FIG. 5, an example in which the peripheral device 220 executes a heat generation process is shown. However, as described above, when other processes (for example, vibration process, sound output process, light output process, odor generation process, shape change process, movement process) are executed, a processing unit corresponding to each process is provided in the peripheral device 220 (or its vicinity). For example, when the peripheral device 220 executes a vibration process, a vibration unit is provided in the peripheral device 220. Also, for example, when the peripheral device 220 executes a sound output process, a sound output device is provided in the peripheral device 220 or in its direction. Also, for example, when the peripheral device 220 executes a light output process, a light emitting device is provided in the peripheral device 220 or in its direction. Also, for example, when the peripheral device 220 executes an odor generation process, an odor generation device is provided in the peripheral device 220 or its vicinity. Also, for example, when the peripheral device 220 executes a shape change process, the peripheral device 220 itself is a device whose shape can be changed.

[0064] [Example of detecting driving load] Regarding the degree of driving load, it is possible to make a determination based on whether the vehicle C1 is stationary or in motion. For example, when the vehicle C1 is stationary, it can be determined that the driving load is low (for example, the driving load is less than the threshold value). On the other hand, when the vehicle C1 is in motion, it can be determined that the driving load is high (for example, the driving load is equal to or greater than the threshold value). Regarding whether the vehicle C1 is stationary or in motion, it can be determined based on the vehicle information (for example, vehicle speed, acceleration, position of the shift lever (for example, P range, D range), amount of depression of the accelerator pedal, amount of depression of the brake pedal) acquired by the vehicle information acquisition unit 125.

[0065] Also, the driving load can be determined based on the signal in front of the vehicle C1. For example, when the signal in front of the vehicle C1 is red and the vehicle C1 is stationary, it can be determined that the driving load is low (for example, the driving load is less than the threshold value). On the other hand, when the vehicle C1 is stationary but the signal in front of the vehicle C1 changes to green, it can be determined that the driving load is high (for example, the driving load is equal to or greater than the threshold value). In addition to these determinations, it is possible to make the same determination as based on whether the vehicle C1 is stationary or in motion.

[0066] Note that the driving load of the driver D1 may be determined based on the driving operation of the driver D1, the surrounding situation of the vehicle C1, etc.

[0067] For example, the operating status of driver D1, the status around vehicle C1, etc. can be acquired by a driver status acquisition unit 122, an outside vehicle status acquisition unit 124, a vehicle information acquisition unit 125, etc. For example, when the number of accelerator operations within a predetermined time is large and the number of steering operations is also large, it can be estimated that the driving load on driver D1 is high. Also, based on the status around vehicle C1, etc., the driving load on driver D1 can be estimated. For example, when vehicle C1 is traveling on a winding road, a narrow road, a highly congested road, a road with many people, etc., it can be estimated that the driving load on driver D1 is high. Also, based on the facial expression of the user, the voice emitted by driver D1, etc., the driving load on driver D1 can be estimated.

[0068] For example, as the driving operation of driver D1, it is possible to use steering entropy. The steering entropy method is a measurement method for measuring and estimating the driver's load based on the smoothness of the driver's steering angle, and it is possible to use a known calculation method. Also, in this steering entropy method, it is possible to use the measurement result (measurement value) quantified as the information entropy value calculated based on the time-series steering angle data.

[0069] Also, for example, as the surrounding situation of vehicle C1, it is possible to use the number of traffic participants around vehicle C1, the weather around vehicle C1, the darkness around vehicle C1, the road shape around vehicle C1, etc.

[0070] It is possible to obtain the driving load on driver D1 using each of the above-described values. For example, it is possible to perform a predetermined calculation (for example, addition) on each of the above-described values and use the calculation result to obtain the driving load on driver D1. Note that the driving load on driver D1 may be obtained using at least one of the above-described values.

[0071] [Regarding the importance of information presented to the driver] In this embodiment, when the driving load of driver D1 is equal to or higher than a threshold value (for example, when vehicle C1 is in motion), when information to be presented to driver D1 is generated, the timing of presenting the information to driver D1 is adjusted according to the importance of the information. Note that the information presented to driver D1 is, for example, assistance information for assisting the driving operation of driver D1. This assistance information is, for example, information necessary for the driving operation of driver D1, such as navigation information, landmark guidance information, sightseeing guidance information, famous place guidance information, and the like.

[0072] For example, the importance of the information presented to driver D1 is determined based on the safety of vehicle C1 and the like. For example, for information that may affect safety if not quickly conveyed to driver D1, a high importance level is set. That is, an importance level equal to or higher than the threshold value is set. Information with an importance level equal to or higher than the threshold value is, for example, information for realizing the navigation function. As this information, for example, information notifying the presence of an intersection, a railroad crossing, etc. ahead, and information notifying a right or left turn at the intersection ahead can be considered.

[0073] On the other hand, for information that is considered to have little impact on safety, a low importance level is set. That is, an importance level lower than the threshold value is set. Information with an importance level lower than the threshold value is, for example, information for guiding landmarks, sightseeing spots, famous places, etc. existing around vehicle C1. Note that when information with an importance level lower than the threshold value is generated as information to be presented to driver D1 when the driving load of driver D1 is equal to or higher than the threshold value, the information is stored. Then, when the driving load of driver D1 becomes less than the threshold value, it is possible to use an agent to convey the information to driver D1 as casual conversation.

[0074] [Example of the relationship between each situation in the vehicle, the role of the agent, and the devices used] FIG. 6 is a diagram showing an example of the relationship between each situation in vehicle C1, the role of the agent, and the devices used. Note that the relationship example shown in FIG. 6 is an example, and other relationships may also be used.

[0075] Here, regarding the sense of the existence of life felt by humans, it is possible to indicate it at each level of feeling such as existence, attention, intention, and empathy. Therefore, in this embodiment, the states of each agent are set so that the driver D1 can feel each of those levels of feeling. Also, inside the vehicle C1, the psychological distance is considered to correspond to the physical distance. Therefore, in this embodiment, by having the agent move back and forth along the physical distance, the states of each agent are set such that communication is carried out in a state of approaching the driver D1 or only the sense of existence is shown in a state of moving away from the driver D1. For example, while the vehicle C1 is in motion (while the driver D1 is driving), it is possible to set the agent to two states (the first state (the first position), the second state (the second position)). Also, while the vehicle C1 is stationary, it is possible to set the agent to one state (the third state (the third position)). Note that the second state is an example of the expression mode of the agent at a position that comes into the field of view of the driver D1 during driving and can be considered as the normal state. Also, the first state is an example of the expression mode of an agent without an appearance (for example, heat, sound, light) and can be considered as a state of moving away from the driver D1. Also, the third state can be considered as a state of approaching the driver D1. Note that the third state can also be referred to as a state for enjoying interaction. In this way, the first position is set as the position where the distance between the agent and the driver D1 is the farthest, and the third position is set as the position where the distance between the agent and the driver D1 is the closest. That is, as the first position, the second position, and the third position are reached, the physical distance of the agent approaches the driver D1.

[0076] As described above, in this embodiment, an example of using three states for the agent existing in the vehicle C1 is shown. The first state indicates a state in which the agent does not execute presentation of support information, communication, etc. when the driving load of the driver D1 is high. In this first state, since the driving load of the driver D1 is high, it is necessary to reduce distraction of the driver D1. Therefore, in the first state, an effect is executed to make the driver D1 feel the presence of the agent in a manner that is difficult to recognize by the driver D1. For example, it is possible to express the presence of the agent by utilizing the behavior of the parts of the vehicle C1 that the driver D1 touches. For example, it is possible to express the presence of the agent by heating the shift lever 6 that the driver D1 is likely to touch.

[0077] The second state indicates a state in which the agent needs to present support information. In this second state, regardless of the driving load of the driver D1, since the agent needs to present support information, the agent appears at a position within the field of view of the driver D1, and support information is presented using the agent. Regardless of the driving load of the driver D1, the support information that needs to be presented is, for example, support information with a high importance. As this support information, for example, navigation information for executing navigation (for example, information notifying intersections, etc.) is assumed.

[0078] The third state indicates a state in which the agent executes communication with the driver D1 when the driving load of the driver D1 is low (for example, when the vehicle is stopped). In this third state, since the driving load of the driver D1 is low, the agent appears at a position close to the driver D1, and the distance between the driver D1 and the agent is reduced to execute active communication. For example, it is possible to display the agent image AG2 by using the second output device 210 installed on the steering wheel 3 existing in front of the driver D1.

[0079] Next, the above-described three states will be described in detail. For example, assume a situation in vehicle C1 where the driving load on driver D1 is equal to or greater than a threshold value and there is no presentation information to be presented to driver D1. In this case, in order to prioritize the driving operation of driver D1, it is the role of the agent to show a presence in the same space as driver D1 (the space inside vehicle C1). Also, expression means for the agent to play such a role are used. For example, as shown in FIG. 3, by heating the shift lever 6, it is possible for the agent to show its presence inside vehicle C1 even when the agent is not visible. Further, for example, as shown in FIG. 4, by displaying the agent image AG3 on the front windshield 4 (HUD display area), it is possible to show the presence of the agent. Such a position of the agent is referred to as the first position. In this way, by showing the presence of the agent, it is possible to prevent driver D1 from feeling bothered by the presence of the agent. Also, since driver D1 can recognize the presence of the agent, it is possible to make driver D1 feel as if the agent is with him / her. For this reason, it is possible to maintain the degree of trust, attachment, etc. towards the agent, and it is possible to enhance the transmission ability of the information transmitted from the agent to driver D1.

[0080] Also, for example, as each situation in vehicle C1, assume a case where the driving load of driver D1 is equal to or greater than a threshold value and there is presentation information (importance is equal to or greater than a threshold value) to be presented to driver D1. In this case, it is the role of the agent to provide important presentation information without increasing the driving load of driver D1. Also, an expression means for the agent to play such a role is used. For example, as shown in FIG. 1, an agent image AG1 is displayed on the display unit 201 of the first output device 200, and by outputting presentation information using the agent image AG1 (for example, outputting voice information S1 "Turn right at the next intersection"), it is possible to provide important presentation information. Such a position of the agent is referred to as the second position. In this way, by moving the agent to a position where information can be presented without the driver D1 changing the line of sight during driving as much as possible, it is possible to appropriately express what vehicle C1 wants to convey to driver D1. Thereby, it is possible to enhance the transmission ability of the information transmitted from the agent to driver D1.

[0081] Here, an example is shown using the first output device 200 as a position where the agent can present information without the driver D1 changing their line of sight as much as possible during driving. However, other positions may also be used. For example, based on the position information of the vehicle C1 and the map information DB, the line of sight of the driver D1 can be predicted, and an agent image can be displayed in the direction of the predicted line of sight. For example, it is possible to display the agent image at a position where display is possible using the display device 720 (see FIG. 13(C)) or the HUD on the front windshield 4 (see FIG. 13(D)). For example, when a predetermined landmark or the like exists, it is assumed that the driver D1 will look in the direction of that landmark. For example, when the vehicle C1 is traveling at a position where the Tokyo Tower can be seen diagonally in front on the right, it is assumed that the driver D1 will look at the Tokyo Tower at least temporarily. Therefore, based on the relative position of the landmark information stored in the map information DB with respect to the vehicle C1, the line of sight direction of the driver D1 can be estimated. Also, for example, when a road with a predetermined shape exists, it is assumed that the driver D1 will look straight ahead along that shape. For example, when the vehicle C1 is traveling on a road that curves significantly to the right, it is assumed that the driver D1 will look to the right. Therefore, based on the relative position of the map information stored in the map information DB with respect to the vehicle C1, the line of sight direction of the driver D1 can be estimated. Note that a line of sight detection process for detecting the line of sight of the driver D1 and a line of sight direction detection process for detecting the line of sight direction of the driver D1 may be executed based on an image including the eyes of the driver D1 (for example, which can be acquired by the driver situation acquisition unit 122). For example, for this method of detecting the line of sight direction, it is possible to use a known line of sight detection technique.

[0082] Also, for example, as each situation in vehicle C1, assume a case where the driving load of driver D1 is less than the threshold value. In this case, since the driving load of driver D1 is low, it is the role of the agent to execute communication at a position close to driver D1 (that is, communication within the personal space). Also, use an expression means for the agent to play such a role. For example, as shown in FIG. 2, an agent image AG2 is displayed on the display unit 211 of the second output device 210, and it is possible to execute a predetermined communication (for example, output of voice information S2 "Let's talk") using the agent image AG2. Such a position of the agent is referred to as the third position. In this way, by moving the agent to a position close to driver D1 and having the agent communicate with driver D1, it is possible to make driver D1 enjoy the communication. Thereby, it is possible to increase the reliability, attachment, etc. to the agent, and it is possible to increase the transmission ability of the information transmitted from the agent to driver D1.

[0083] [Transition example when moving the agent from the first output device to the shift lever] FIG. 7 is a diagram showing a transition example when moving the agent from the first output device 200 to the shift lever 6. Note that regarding the configuration of the vehicle interior of vehicle C1 shown in FIG. 7, since it is the same as the configuration of the vehicle interior of vehicle C1 shown in FIGS. 1 to 4, the same reference numerals are given to the parts common to FIGS. 1 to 4 and the descriptions thereof are omitted. Also, similarly for the examples shown in FIGS. 8 to 10, the same reference numerals are given to the parts common to FIGS. 1 to 4 and the descriptions thereof are omitted.

[0084] FIG. 7(A) shows an example where the agent image AG1 is displayed on the display unit 201 of the first output device 200. For example, when the driving load of the driver D1 is equal to or higher than the threshold value (for example, when the vehicle C1 is running), and there is predetermined information (importance is equal to or higher than the threshold value) to be presented to the driver D1, the agent image AG1 is displayed on the display unit 201 of the first output device 200, and the predetermined information is output using the agent image AG1. Then, when the driving load of the driver D1 is equal to or higher than the threshold value and the output of the predetermined information is completed, a movement effect of moving the agent from the first output device 200 to the shift lever 6 is executed.

[0085] Specifically, as shown in FIG. 7(B), on the display unit 201 of the first output device 200, a movement effect is executed in which the agent image AG1 moves in the direction of the shift lever 6 and the display mode is changed so that the agent image AG1 becomes smaller in accordance with the movement.

[0086] Subsequently, as shown in FIG. 7(C), on the display unit 201 of the first output device 200, a movement effect is executed in which the agent image AG1 further moves in the direction of the shift lever 6 and the display mode is changed so that the agent image AG1 becomes even smaller in accordance with the movement. After the execution of this movement effect, or during the execution of this movement effect, a movement effect of gradually heating the heater unit 221 of the shift lever 6 to set the shift lever 6 to a predetermined temperature is executed.

[0087] Note that when the agent image AG1 moves in the direction of the shift lever 6 on the display unit 201 of the first output device 200, the transition of the movement may be expressed by animation processing so that the trajectory of the movement of the agent image AG1 can be understood.

[0088] As shown in FIGS. 7(A) to 7(C), when moving the presence of the agent image AG1 displayed on the display unit 201 of the first output device 200 to the shift lever 6, the agent image AG1 is moved in the direction of the destination (shift lever 6) on the display unit 201. Thereby, the driver D1 can easily grasp by visual sense (an example of the five senses) that the presence of the agent image AG1 has moved to the shift lever 6. Further, at the timing of the movement or immediately after the movement, the heater unit 221 of the shift lever 6 is gradually heated to set the shift lever 6 to a predetermined temperature, so that the driver D1 can easily grasp by tactile sense (an example of the five senses) that the presence of the agent image AG1 has moved to the shift lever 6.

[0089] [Transition example when moving an agent from the shift lever to the first output device] FIG. 8 is a diagram showing a transition example when moving an agent from the shift lever 6 to the first output device 200.

[0090] FIG. 8(A) shows an example of indicating the presence of an agent by setting the shift lever 6 to a predetermined temperature. For example, when the driving load of the driver D1 is equal to or higher than a threshold value (for example, when the vehicle C1 is running) and there is predetermined information (importance is equal to or higher than a threshold value) presented to the driver D1, the agent image AG1 is displayed on the display unit 201 of the first output device 200, and the predetermined information is output using the agent image AG1.

[0091] Specifically, as shown in FIGS. 8(A) and 8(B), on the display unit 201 of the first output device 200, a movement effect is executed in which the agent image AG1 moves from the direction of the shift lever 6 and the display mode is changed so that the agent image AG1 becomes larger in accordance with the movement.

[0092] Subsequently, as shown in FIG. 8(C), a movement effect is executed in which the agent image AG1 is displayed on the display unit 201 of the first output device 200. Before the execution of this movement effect or during the execution of this movement effect, a movement effect is executed to gradually stop the heat generation process in the heater unit 221 of the shift lever 6.

[0093] Note that when the agent image AG1 moves from the direction of the shift lever 6 on the display unit 201 of the first output device 200, the transition of the movement may be expressed by animation processing so that the trajectory of the movement of the agent image AG1 can be understood.

[0094] As shown in FIGS. 8(A) to (C), when moving the presence of the agent from the shift lever 6 where the presence of the agent is expressed to the display unit 201 of the first output device 200, the agent image AG1 is moved on the display unit 201 from the direction of the movement source (shift lever 6). Thereby, it becomes possible for the driver D1 to easily grasp by the driver's vision (an example of the five senses) that the presence of the agent image AG1 has moved from the shift lever 6. Further, by gradually stopping the heat generation process of the heater unit 221 of the shift lever 6 at the timing of the movement or immediately before the movement, it becomes possible for the driver D1 to easily grasp by the driver's sense of touch (an example of the five senses) that the presence of the agent at the shift lever 6 has moved to the first output device 200.

[0095] [Transition example when moving an agent from the second output device to the shift lever] FIG. 9 is a diagram showing a transition example when moving an agent from the second output device 210 to the shift lever 6.

[0096] FIG. 9(A) shows an example where the agent image AG2 is displayed on the display unit 211 of the second output device 210. For example, when the driving load of the driver D1 is less than the threshold value (for example, when the vehicle C1 is stopped), the agent image AG2 is displayed on the display unit 211 of the second output device 210, and a predetermined communication is executed using the agent image AG2. Then, when the driving load of the driver D1 becomes equal to or greater than the threshold value, a movement effect for moving the agent from the second output device 210 to the shift lever 6 is executed.

[0097] Specifically, as shown in FIG. 9(B), on the display unit 211 of the second output device 210, a movement effect is executed in which the agent image AG2 moves in the direction of the shift lever 6 and the display mode is changed so that the agent image AG2 becomes smaller in accordance with the movement.

[0098] Subsequently, as shown in FIG. 9(C), on the display unit 211 of the second output device 210, a movement effect is executed in which the agent image AG2 further moves in the direction of the shift lever 6 and the display mode is changed so that the agent image AG2 becomes even smaller in accordance with the movement. After the execution of this movement effect, or during the execution of this movement effect, a movement effect for gradually heating the heater unit 221 of the shift lever 6 and setting the shift lever 6 to a predetermined temperature is executed.

[0099] Note that when the agent image AG2 moves in the direction of the shift lever 6 on the display unit 211 of the second output device 210, the transition of the movement may be expressed by an animation process so that the trajectory of the movement of the agent image AG2 can be understood.

[0100] As shown in FIGS. 9(A) to 9(C), when moving the presence of the agent image AG2 displayed on the display unit 211 of the second output device 210 to the shift lever 6, the agent image AG2 is moved in the direction of the destination (shift lever 6) on the display unit 211. Thereby, it becomes possible for the driver D1 to easily grasp by visual sense (an example of the five senses) that the presence of the agent image AG2 has moved to the shift lever 6. Further, at the timing of the movement or immediately after the movement, the heater unit 221 of the shift lever 6 is gradually heated, and the shift lever 6 is set to a predetermined temperature, so that it becomes possible for the driver D1 to easily grasp by tactile sense (an example of the five senses) that the presence of the agent image AG2 has moved to the shift lever 6.

[0101] [Transition example when moving an agent from the shift lever to the second output device] FIG. 10 is a diagram showing a transition example when moving an agent from the shift lever 6 to the second output device 210.

[0102] FIG. 10(A) shows an example of showing the presence of an agent by setting the shift lever 6 to a predetermined temperature. For example, when the driving load of the driver D1 is less than the threshold value (for example, when the vehicle C1 is stopped), the agent image AG2 is displayed on the display unit 211 of the second output device 210, and a predetermined communication is executed using the agent image AG2.

[0103] Specifically, as shown in FIGS. 10(A) and 10(B), on the display unit 211 of the second output device 210, a movement effect is executed in which the agent image AG2 moves from the direction of the shift lever 6 and the display mode is changed so that the agent image AG2 becomes larger according to the movement.

[0104] Subsequently, as shown in FIG. 10(C), a movement effect is executed in which the agent image AG2 is displayed on the display unit 211 of the second output device 210. Before the execution of this movement effect or during the execution of this movement effect, a movement effect is executed to gradually stop the heat generation process in the heater unit 221 of the shift lever 6.

[0105] In addition, when the agent image AG2 moves in the direction of the shift lever 6 on the display unit 211 of the second output device 210, the transition of the movement may be expressed by animation processing so that the trajectory of the movement of the agent image AG2 can be understood.

[0106] As shown in FIGS. 10(A) to (C), when moving the presence of the agent from the shift lever 6 where the presence of the agent is expressed to the display unit 211 of the second output device 210, the agent image AG2 is moved on the display unit 211 from the direction of the movement source (shift lever 6). Thereby, it becomes possible for the driver D1 to easily grasp by the driver's vision (an example of the five senses) that the presence of the agent image AG2 has moved from the shift lever 6. Further, by gradually stopping the heat generation process of the heater unit 221 of the shift lever 6 at the timing of the movement or immediately before the movement, it becomes possible for the driver D1 to easily grasp by the driver's sense of touch (an example of the five senses) that the presence of the agent on the shift lever 6 has moved to the second output device 210.

[0107] As described above, in the present embodiment, when referring to the movement of the agent, it means erasing the agent image displayed on a certain display device and newly displaying the agent image on another display device. Further, when referring to the movement of the agent, it means erasing the agent image displayed on a certain display device and newly executing on another device an effect that makes the driver feel the presence of the agent corresponding to the agent image. Further, when referring to the movement of the agent, it means stopping the effect that makes the driver feel the presence of the agent being executed on a certain device and newly displaying the agent image corresponding to the agent on another display device. Note that, during these movements, it may be expressed such that the agent exists simultaneously at the movement source and the movement destination, or it may be expressed such that the agent exists only at either the movement source or the movement destination.

[0108] [Operation Example of Information Processing System] FIG. 11 and FIG. 12 are flowcharts showing an example of agent representation processing in the information processing system 100. Further, this agent representation processing is executed by the control unit 120 (see FIG. 3) based on a program stored in the storage unit 130 (see FIG. 3). Further, this agent representation processing is constantly executed every control cycle. Further, in this agent representation processing, FIGS. 1 to 10 will be appropriately referred to and described.

[0109] In step S501, the communication content determination unit 126 determines whether or not the driving load of the driver D1 of the vehicle C1 is less than the threshold value. Note that the method for detecting the driving load is the same as the detection method described above. Further, the threshold value shown here is determination information for determining whether the driving load of the driver D1 is high or low. This threshold value can be appropriately set based on experiments or simulations, etc. Note that here, an example of determining whether or not the driving load of the driver D1 is less than the threshold value is shown, but it may be determined whether the vehicle C1 is running or stopped. If the driving load of the driver D1 is less than the threshold value, the process proceeds to step S502. On the other hand, if the driving load of the driver D1 is greater than or equal to the threshold value, the process proceeds to step S516 (see FIG. 12).

[0110] In step S502, the communication content determination unit 126 determines whether or not the agent exists at the third position (see FIG. 6) based on the agent management DB 132. Specifically, as shown in FIG. 2, it is determined whether or not the agent image AG2 is displayed on the display unit 211 of the second output device 210. If the agent exists at the third position, the process proceeds to step S507. On the other hand, if the agent does not exist at the third position, the process proceeds to step S503.

[0111] In step S503, the communication content determination unit 126 determines whether there is information to be presented to driver D1. For example, as information to be presented to driver D1, navigation information, information on landmarks existing around vehicle C1, etc. are assumed. If there is information to be presented to driver D1, the process proceeds to step S504. On the other hand, if there is no information to be presented to driver D1, the process proceeds to step S513.

[0112] In step S504, the output mode determination unit 127 determines to execute an effect in which the agent moves from another position to the third position. Then, the agent control unit 128 executes an effect in which the agent moves from another position to the third position. For example, as shown in FIG. 10, a movement effect of moving the agent existing on the shift lever 6 to the display unit 211 of the second output device 210 is executed. For example, an effect is executed in which the once-warm shift lever 6 becomes cold and the agent image AG2 appears from below the screen of the display unit 211 (that is, from the direction of the shift lever 6). In this case, voice information such as "Hey" may be output from the direction of the shift lever 6. After the agent image AG2 appears on the display unit 211, the speaker for voice output may be changed so that the speech of the agent can be heard from the second output device 210. As described above, the line-of-sight direction of driver D1 may be predicted and the agent image AG2 may be displayed at a position close to the line-of-sight direction.

[0113] In step S505, the agent control unit 128 executes a process of causing the agent to appear at the third position. For example, as shown in FIG. 10(C), a process of displaying the agent image AG2 on the display unit 211 of the second output device 210 is executed.

[0114] In step S506, the agent control unit 128 executes a process of causing the agent that has appeared at the third position to present information. For example, as shown in FIG. 10(C), information for the driver D1 is provided using the agent image AG2 displayed on the display unit 211 of the second output device 210. For example, it is possible to provide information by voice information emitted by the agent image AG2.

[0115] In step S507, the communication content determination unit 126 determines whether or not the agent is in the process of communicating with the driver D1. For example, as shown in FIG. 2, when some interaction (for example, conversation) is taking place between the agent image AG2 displayed on the display unit 211 of the second output device 210 and the driver D1, it is determined that the agent is in the process of communicating with the driver D1. If the agent is in the process of communicating with the driver D1, the process proceeds to step S512. On the other hand, if the agent is not in the process of communicating with the driver D1, the process proceeds to step S508.

[0116] In step S508, the communication content determination unit 126 determines to propose to the driver D1 to execute a predetermined communication by the agent. Then, the agent control unit 128 uses the agent image AG2 displayed on the display unit 211 of the second output device 210 to propose to the driver D1 to execute a predetermined communication. For example, as the predetermined communication, it is possible to propose that the agent and the driver D1 have a conversation such as small talk, play a simple game, provide the information held in step S524 (see FIG. 12), and the like. For example, as shown in FIG. 2, it is possible to make the proposal by the agent image AG2 displayed on the display unit 211 of the second output device 210 emitting the voice information S2 "Let's talk".

[0117] In step S509, the communication content determination unit 126 determines whether the driver D1 has rejected the proposal executed in step S508 (that is, whether the driver D1 has declined the proposal). For example, it is possible to determine that the driver D1 has rejected the proposal based on whether the driver D1 has made a rejecting statement (for example, "I'm sorry, another time") or not. Also, for example, it is possible to determine that the driver D1 has rejected the proposal based on whether the driver D1 has made a rejecting movement (for example, a head shaking movement, a hand waving movement) or not. Regarding the voice emitted by the driver D1, it can be recognized using known voice recognition technology based on the voice information acquired by the speech acquisition unit 121. Also, regarding the movement of the driver D1, it can be recognized using known object recognition technology based on the image information of the driver D1 acquired by the driver status acquisition unit 122. If the driver D1 has rejected the proposal, the process proceeds to step S510. On the other hand, if the driver D1 has not rejected the proposal (if the driver D1 has accepted the proposal), the process proceeds to step S512. Note that when the proposal has been rejected by the driver D1, the rejection time is held in the storage unit 130.

[0118] In step S510, the output mode determination unit 127 determines to execute an effect in which the agent moves from the third position to the first position. Then, the agent control unit 128 executes an effect in which the agent moves from the third position to the first position. For example, an effect in which the agent returns to the shift lever 6 is executed. For example, as shown in FIG. 9, a movement effect of moving the agent image AG2 displayed on the display unit 211 of the second output device 210 to the shift lever 6 is executed. Note that when the proposal from the agent image AG2 has been rejected by the driver D1, a movement effect of moving the agent image AG2 to the shift lever 6 may be executed after the agent image AG2 has emitted voice information such as "Let's go back to the shift lever." Thus, when the proposal from the agent image AG2 has been rejected by the driver D1, an effect of making the presence of the agent disappear from the driver D1's vision and an effect of making the driver D1 feel the presence of the agent are executed.

[0119] In step S511, the agent control unit 128 executes a process of presenting the presence of the agent at the first position. For example, as shown in FIG. 9(C), a process of heating the shift lever 6 is executed.

[0120] In step S512, the agent control unit 128 executes control for the agent that appears at the third position to perform predetermined communication with the driver D1. For example, as shown in FIG. 2, control is executed for the agent image AG2 displayed on the display unit 211 of the second output device 210 to have a conversation between the agent image AG2 and the driver D1.

[0121] In step S513, the communication content determination unit 126 determines whether or not a predetermined time has elapsed since the timing when the proposal was rejected in step S509. Note that the rejected timing can be acquired based on the rejection time held in the storage unit 130 in step S509. Also, the predetermined time can be set to a time (for example, about several minutes to several tens of minutes) that is sufficient for the mood of the driver D1 who rejected the agent's proposal to change. Further, this predetermined time can be appropriately set based on experiments, simulations, or the like. If a predetermined time has elapsed since the timing when the proposal was rejected, the process proceeds to step S514. Note that the process also proceeds to step S514 in the case where the proposal was not rejected in step S509. On the other hand, if a predetermined time has not elapsed since the timing when the proposal was rejected, the operation of the agent expression process ends.

[0122] Since the processes in steps S514 and S515 correspond to the processes in steps S504 and S505, the description thereof is omitted here.

[0123] In step S516 shown in FIG. 12, the communication content determination unit 126 determines whether there is information to be presented to driver D1. The information to be presented to driver D1 is the same as the information shown in step S503. If there is information to be presented to driver D1, the process proceeds to step S517. On the other hand, if there is no information to be presented to driver D1, the process proceeds to step S525.

[0124] In step S517, the communication content determination unit 126 determines whether the importance level of the information to be presented to driver D1 is less than the threshold value. The importance level of the information to be presented to driver D1 is the same as the importance level described above. If the importance level of the information to be presented to driver D1 is less than the threshold value, the process proceeds to step S524. On the other hand, if the importance level of the information to be presented to driver D1 is equal to or greater than the threshold value, the process proceeds to step S518.

[0125] Note that in the present embodiment, an example is shown in which the position of the agent is changed based on whether the importance level of the information to be presented to driver D1 is less than the threshold value (that is, whether the importance level is high or low). However, the importance level of the information to be presented to driver D1 may be determined using a threshold value of 2 or more, and the distance between the agent and driver D1 may be gradually decreased as the importance level of the information increases. For example, when a long display device 720 (see FIG. 13(C)) in the left-right direction of the vehicle C1 is provided on the dashboard 2 and an agent image is displayed on the display device 720, the agent image may be displayed so as to approach driver D1 as the importance level of the information increases.

[0126] In step S518, the communication content determination unit 126 determines whether the agent exists at the second position (see FIG. 6) based on the agent management DB132. Specifically, as shown in FIG. 1, it is determined whether an agent image AG1 is displayed on the display unit 201 of the first output device 200. If the agent exists at the second position, the process proceeds to step S521. On the other hand, if the agent does not exist at the second position, the process proceeds to step S519.

[0127] In step S519, the output mode determination unit 127 determines to execute an effect in which the agent moves from another position to the second position. Then, the agent control unit 128 executes an effect in which the agent moves from another position to the second position. For example, as shown in FIG. 8, a movement effect of moving the agent existing on the shift lever 6 to the display unit 201 of the first output device 200 is executed.

[0128] In step S520, the agent control unit 128 executes a process of causing the agent to appear at the second position. For example, as shown in FIG. 8(C), a process of displaying the agent image AG1 on the display unit 201 of the first output device 200 is executed.

[0129] In step S521, the agent control unit 128 executes a process of presenting information to the agent that has appeared at the second position. For example, as shown in FIG. 1, information for the driver D1 is provided using the agent image AG1 displayed on the display unit 201 of the first output device 200. For example, it is possible to provide the driver D1 with the information "Turn right at the next intersection" by the voice information S1 emitted from the agent image AG1.

[0130] In step S522, the output mode determination unit 127 determines to execute an effect in which the agent moves from the second position to the first position. Then, the agent control unit 128 executes an effect in which the agent moves from the second position to the first position. For example, as shown in FIG. 7, a movement effect of moving the agent image AG1 displayed on the display unit 201 of the first output device 200 to the shift lever 6 is executed. For example, an effect is executed in which the agent image AG1 that was displayed on the display unit 201 moves downward (in the direction of the shift lever 6) of the display unit 201, disappears from the screen of the display unit 201, and the cold shift lever 6 becomes warm. Note that an effect may be executed in which the agent image AG1 that was displayed on the display unit 201 outputs voice information such as "I've conveyed the necessary information, so I'm going back to the shift lever", moves downward (in the direction of the shift lever 6) of the display unit 201, disappears from the screen of the display unit 201, and then voice information "I've returned to the shift lever" is output from the direction of the shift lever 6. In this way, an effect is executed to eliminate the presence of the agent from the driver D1's vision, and an effect is executed so that the driver D1 can feel the presence of the agent.

[0131] In step S523, the agent control unit 128 executes a process of giving a sense of presence of the agent at the first position. For example, as shown in FIG. 7(C), a process of generating heat in the shift lever 6 is executed.

[0132] In step S524, the communication content determination unit 126 causes the storage unit 130 to hold the information determined to have an importance less than the threshold value in step S517 in order to present the information to the driver D1 at the next timing. In this way, for information determined to have an importance less than the threshold value, it can be stored in the database and provided to the driver D1 as a topic of conversation or the like when the driving load on the driver D1 becomes less than the threshold value.

[0133] In step S525, the communication content determination unit 126 determines whether the agent exists at the first position (see FIG. 6) based on the agent management DB 132. Specifically, as shown in FIG. 3, it is determined whether the presence of the agent is expressed by the heat generation in the shift lever 6. If the agent exists at the first position, the operation of the agent expression process ends. On the other hand, if the agent does not exist at the first position, the process proceeds to step S526.

[0134] In step S526, the output mode determination unit 127 determines to execute an effect in which the agent moves from another position to the first position. Then, the agent control unit 128 executes an effect in which the agent moves from another position to the first position. For example, as shown in FIG. 7, a moving effect of moving the agent image AG1 displayed on the display unit 201 of the first output device 200 to the shift lever 6 is executed. Also, for example, as shown in FIG. 9, a moving effect of moving the agent image AG2 displayed on the display unit 211 of the second output device 210 to the shift lever 6 is executed. For example, an effect is executed in which the agent image AG2 that was displayed on the display unit 211 moves downward (in the direction of the shift lever 6) of the display unit 211, disappears from the screen of the display unit 211, and the cold shift lever 6 becomes warm.

[0135] Here, for example, when the front signal is a red signal and the vehicle C1 is stopped, since it is determined in step S501 that the driving load of the driver D1 is less than the threshold value, the agent image AG2 is often displayed on the display unit 211 of the second output device 210. Assuming that in such a state where the agent image AG2 is displayed on the display unit 211, the front signal changes to a green signal. In this case, the agent may inform the driver D1 that the front signal has changed to green. For example, an effect may be executed in which the agent image AG2 that was displayed on the display unit 211 outputs the voice information "It's green", moves downward (in the direction of the shift lever 6) of the display unit 211, disappears from the screen of the display unit 211, and then the voice information "It's green" is output from the direction of the shift lever 6.

[0136] In step S527, the agent control unit 128 executes a process of presenting the presence of the agent at the first position. For example, as shown in FIGS. 7(C) and 9(C), a process of heating the shift lever 6 is executed.

[0137] In addition, in FIGS. 11 and 12, mainly, control processes for changing the expression mode of the agent are shown based on the results of the determination process of the level of the driving load of the driver D1 (step S501) and the determination process of the presence or absence of information presented to the driver D1 (steps S503, S516, and S517). However, the present embodiment is not limited to this, and it may be realized by other control examples. For example, the determination process of the level of the driving load of the driver D1 (step S501) may be omitted, and based on the results of the presence or absence of information presented to the driver D1 (steps S516 and S517), a control process (for example, steps S516 to S527) for changing the expression mode of the agent may be used.

[0138] In addition, an example of executing a control process for changing the expression mode of the agent based on the driving load of the driver D1, the presence or absence of information presented to the driver D1, etc. has been shown. However, the control process may be executed by other methods. For example, it is possible to execute the control process using artificial intelligence (AI). For example, it is possible to pre-learn each situation regarding the driver D1, the vehicle C1, etc., and the expression mode of the agent to be executed corresponding to those situations, and use this learning data for the control process. For example, for each situation that occurs regarding the driver D1, the vehicle C1, etc., the expression mode of the agent can be determined using the learning data, and the determined expression mode can be executed by the agent.

[0139] [Examples of other expression modes of the agent] FIG. 13 is a diagram showing another exemplary representation of the agent. In FIGS. 13(A) and (B1), an example of using a robot as the agent is shown. In FIGS. 13(C) and (D), an example of using an agent image to be displayed on a display device that is long in the left-right direction is shown.

[0140] FIG. 13(A) shows an example of a moving structure in which the agent device 600 moves on the rail RL1. As shown in FIG. 13(A), the rail RL1 composed of two rail members extending in the left-right direction is installed on the dashboard 2. Further, wheels capable of traveling on the rail RL1 are installed at the lower part of the agent device 600. Then, the drive unit (not shown) of the agent device 600 moves the agent device 600 in the left-right direction (arrow A11 direction) on the rail RL1 based on the control from the agent control unit 128. Note that for the rail structure of the rail RL1 and the wheel structure of the agent device 600, known rail technologies can be adopted.

[0141] In FIG. 13(A), the agent device arranged at the second position is denoted as the agent device 600a, and the agent device arranged at the third position is denoted as the agent device 600b. Here, an example of presenting the agent device at two positions is shown, but the agent device may be presented at three or more positions.

[0142] Note that in FIG. 13(A), an example of the agent device 600 moving on the rail RL1 is shown, but the rail RL1 may be omitted, and an agent device 600 that can move autonomously may be used. For example, at least a part of the dashboard 2 is made flat so that the agent device 600 can move autonomously. Then, by moving the agent device 600 on the flat dashboard 2, the position of the agent device 600 can be changed.

[0143] FIG. 13(B1) shows an example of a structure in which a plurality of holes HO1 and HO2 are provided on the dashboard 2, and a mode in which the agent devices 611 and 612 move using the plurality of holes HO1 and HO2 is realized. FIG. 13(B2) shows a configuration example of the hole HO1. Note that the configuration of the other hole HO2 can be the same as that of the hole HO1. Also, the agent device 612 is the same as the agent device 611.

[0144] The hole HO1 is a cylindrical hole having a size capable of accommodating the agent device 611, and the bottom BO2 is provided with a lifting mechanism UD1 of a rod-shaped body capable of raising or lowering the agent device 611. The upper end of the lifting mechanism UD1 is connected to the lower part of the agent device 611. Also, the lifting mechanism UD1 is controlled to rise or fall by a driving unit (not shown).

[0145] For example, when the agent device 612 is arranged at the third position, as shown in FIG. 13(B1), only the agent device 612 in the hole HO2 rises, and the agent device 611 in the hole HO1 is in a state of being attached to each bottom in a lowered state. Also, for example, when the agent device 611 is arranged at the second position, only the agent device 611 in the hole HO1 rises, and the agent device 612 in the hole HO2 is in a state of being attached to each bottom in a lowered state.

[0146] Although not shown in FIGS. 13(B1) and (B2), lids may be installed in each of the plurality of holes HO1 and HO2, and only the lid of the hole in which the agent device is in the raised state may be opened, and the lids of the other holes may be closed. In this way, by installing lids in each of the plurality of holes HO1 and HO2, only one agent device can be visually recognized by the driver D1, and the identity of each agent device can be enhanced. Here, an example in which two agent devices appear using two holes is shown, but three or more agent devices may appear using three or more holes.

[0147] Note that the examples in FIGS. 13(A), (B1), and (B2) may be combined. For example, similar to the examples shown in FIGS. 13(B1) and (B2), a plurality of holes HO1 and HO2 are provided in the dashboard 2, and a horizontal hole connecting the bottoms of the respective holes HO1 and HO2 is separately provided inside the dashboard 2. That is, a concave hole is provided inside the dashboard 2. Then, a rail is installed in the horizontal hole, and the agent device may be movable on the rail. Also, depending on each expression mode, the agent device is raised from any of the holes.

[0148] FIG. 13(C) shows an example of a case where agent images 722 and 723 are displayed on a display unit 721 of a display device 720 installed in the dashboard 2 of the vehicle C1. Note that the display state of the display device 720 is controlled by an agent control unit 128. For example, it is possible to realize the agent images 722 and 723 by CG (Computer Graphics).

[0149] The agent image 722 is an image displayed at the second position. Also, the agent image 723 is an image displayed at the third position.

[0150] FIG. 7(D) shows an example of a case where agent images 731 and 732 are displayed on the HUD (Head Up Display) of the vehicle C1. In this case, an HUD display device for realizing the HUD is provided near the boundary with the front window 4 above the dashboard 2.

[0151] The HUD display device is a display device, such as a projector or an optical system, that projects light onto the display area 4a of the front windshield 4 and uses the reflected light to show a virtual image to the driver D1. That is, the light projected from the HUD display device onto the display area 4a of the front windshield 4 is reflected by the front windshield 4, and the reflected light travels towards the eyes of the driver D1. Then, the reflected light that is projected onto the display area 4a and enters the eyes of the driver D1 is displayed superimposed on the actual object visible through the front windshield 4. In this way, the HUD display device realizes HUD display by displaying a virtual image using the front windshield 4.

[0152] For example, based on the control of the information processing device 110 (see FIG. 5), the HUD display device causes the agent images 731 and 732 to be displayed in the display area 4a. Also, the front windshield 4 functions as a display medium for the HUD of the vehicle C1.

[0153] The agent image 731 is an image displayed at the second position. Also, the agent image 732 is an image displayed at the third position.

[0154] In FIGS. 13(C) and (D), when transitioning the agent image to each mode, the movement trajectory of the agent image from the position of the original mode to the position of the new mode may be expressed by animation processing or the like.

[0155] Also, for each of the above-described modes (for example, FIGS. 1 to 4, FIG. 13), any of the modes may be combined. Also, when arranging the agent at the second position and the third position, as described above, the line-of-sight direction of the driver D1 may be predicted, and the agent may be arranged at a position corresponding to the line-of-sight direction.

[0156] [Example considering vehicle occupants] In addition, in the present embodiment, an example in which the expression mode of the agent is changed based on the driving load of the driver D1 riding in the vehicle C1, the importance of the information presented to the driver D1, etc. has been shown. However, the expression mode of the agent may be changed in consideration of other passengers riding in the vehicle C1. For example, when the driving load of the driver D1 is equal to or higher than a threshold value (for example, when the vehicle C1 is running), it is conceivable that the passenger in the passenger seat executes a predetermined communication with the agent. In this case, it is conceivable that the agent image AG1 is displayed on the display unit 201 of the first output device 200, and the agent image AG1 and the passenger in the passenger seat execute a predetermined communication. In this case, since the driver D1 can understand that the agent is executing a predetermined communication with the passenger in the passenger seat, the process of showing the presence of the agent using the peripheral device 220 (for example, the heat generation process of the shift lever 6) is not executed. If the agent image AG1 is displayed on the display unit 201 of the first output device 200 and the agent image AG1 and the passenger in the passenger seat execute a predetermined communication, and the process of showing the presence of the agent using the peripheral device 220 (for example, the heat generation process of the shift lever 6) is executed, the driver D1 may feel that there are two agents. In this case, since the reliability of the agent and the like may decrease and the transmission ability of the information transmitted from the agent to the driver D1 may decrease, it is important to make the driver D1 feel that there is one agent.

[0157] [Effect Example in the Present Embodiment] Here, for example, in order to make the driver D1 feel as if the agent is with him / her, if the agent image is left as it appears without being moved significantly, driver D1 may find the presence of the agent image bothersome. Therefore, when not guiding information to driver D1, it is conceivable to erase the agent image. However, if the agent image is erased, driver D1 will not be able to recognize the presence of the agent, making it difficult for the driver to feel as if the agent is boarding vehicle C1 with him / her. That is, if the agent appears when necessary and the presence of the agent disappears when not needed, there is a risk that the driver D1 will recognize the agent as an operator that appears when necessary and disappears when not needed. In this case, the reliability and affection towards the agent may decrease, and the transmission ability of the information transmitted from the agent to driver D1 may decrease.

[0158] Therefore, in this embodiment, by adjusting the presence of the agent as needed, it becomes possible to give the impression that the agent is boarding vehicle C1 with driver D1, and it becomes possible to enhance the feeling of driver D1 being with the agent. That is, it becomes possible to enhance the sense of unity between driver D1 and the agent. As a result, it is possible to increase the reliability, affection, etc. towards the agent, and it is possible to increase the transmission ability of the information transmitted from the agent to driver D1.

[0159] Also, in order for the driver D1 to feel that the agent is an entity that is receptive to information from the vehicle C1, the agent needs to be an entity that is familiar to the driver D1. Therefore, in order to make the agent more familiar, it is conceivable to always display the agent and increase the number of contacts and the contact time between the agent and the driver D1 (for example, the mere exposure effect). However, depending on the driving situation, there is also a possibility that the driver D1 may feel annoyed by the agent being always displayed. Therefore, in order to prevent the annoyance and distraction (being distracted) caused by the agent always being present, it is conceivable to hide the appearance of the agent when it is not necessary. In this case, the above-mentioned annoyance and distraction can be prevented, but there is a possibility that the sense of unity between the agent and the vehicle C1 may be reduced due to the agent disappearing and reappearing.

[0160] Therefore, in the present embodiment, an effect is executed such that the appearance of the agent disappears when it is not necessary, but the presence of the agent remains as it is. That is, although the appearance of the agent is difficult to recognize, it is possible to give the driver D1 the feeling that the agent exists in the same space as the driver D1.

[0161] As a result, it is possible to execute an effect of the agent that reacts to the environment around the driver D1. That is, it is possible to execute an effect that gives the impression that the agent has intelligence. Also, it is possible to execute an effect that makes the driver D1 feel a sense of camaraderie and enhances the empathy with the driver D1.

[0162] [Examples of causing other devices and other systems to execute processing] Note that in the above, an example in which the determination process, detection process, control process, etc. are executed in the information processing apparatus 110 (or the information processing system 100) has been shown, but all or part of each of these processes may be executed by other devices. In this case, the information processing system is configured by each device that executes part of each of these processes. For example, at least part of each process can be executed using various information processing devices such as in-vehicle devices, devices that can be used by the user (for example, smartphones, tablet terminals, personal computers, car navigation devices, IVI), servers that can be connected via a predetermined network such as the Internet, and various electronic devices.

[0163] Also, part (or all) of the information processing system capable of executing the functions of the information processing apparatus 110 (or the information processing system 100) may be provided by an application that can be provided via a predetermined network such as the Internet. This application is, for example, SaaS (Software as a Service).

[0164] [Configuration Example and Effects of the Present Embodiment] The information processing method according to this embodiment is an information processing method for performing control for communication with the driver D1 by using the first output device 200, the second output device 210, and the peripheral device 220 (an example of a plurality of devices) in the vehicle C1. This information processing method includes a determination process (steps S516 and S517) for determining whether there is predetermined information to be presented to the driver D1, and when there is predetermined information, among the plurality of devices, the first output device 200 (an example of a first device) (or the second output device 210) located in a place where it is easy to enter the driver D1's vision is used to make an agent (for example, agent image AG1) for communication visible to the driver D1, and the predetermined information is presented using the agent (for example, agent image AG1) (steps S518 to S521). When there is no predetermined information, among the plurality of devices, the peripheral device 220 (an example of a second device) other than the first output device 200 is used to execute an effect that makes the driver D1 feel the presence of the agent in a manner that is difficult to recognize (for example, heat generation) (steps S525 to S527). Further, the program according to this embodiment is a program for causing a computer to execute each of these processes. In other words, the program according to this embodiment is a program for causing a computer to realize each function that the information processing apparatus 110 can execute.

[0165] According to this configuration, by adjusting the sense of presence of the agent according to the presence or absence of the predetermined information presented to the driver D1, it becomes possible to give the impression that the agent is riding in the vehicle C1 together with the driver D1, and it becomes possible to enhance the feeling of the driver D1 being together with the agent. That is, it becomes possible to enhance the sense of unity between the driver D1 and the agent. As a result, it is possible to enhance the reliability, affection, etc. for the agent, and it is possible to enhance the transmission ability of the information transmitted from the agent to the driver D1.

[0166] In the information processing method according to this embodiment, the first output device 200 (an example of the first device) is a display device. Further, in the control process, when predetermined information exists, an agent image AG1 representing an agent is displayed on the first output device 200 (steps S518 to S521). When the predetermined information does not exist, without displaying the agent image on both the first output device 200 and the peripheral device 220 (an example of the second device), the peripheral device 220 executes an effect that makes the driver D1 feel the presence of the agent by using senses other than vision among the five senses of the driver D1 (steps S525 to S527).

[0167] According to this configuration, the appearance of the agent is shown when necessary to present the predetermined information, and an effect is executed such that the appearance of the agent disappears when not necessary. Thereby, regardless of the presence or absence of the predetermined information presented to the driver D1, the presence of the agent in the vehicle C1 can be kept in the same state. That is, when there is no predetermined information presented to the driver D1, although the appearance of the agent is in a state where it is difficult to recognize, it is possible to give the driver D1 a feeling of existing in the same space as the driver D1.

[0168] In the information processing method according to this embodiment, the peripheral device 220 (an example of the second device) is a peripheral device installed around the driver D1. In the control process, at least one of a temperature adjustment process for adjusting the temperature of the peripheral device 220, a vibration process for vibrating the peripheral device 220, a sound output process for outputting a predetermined sound capable of recognizing the presence of the agent from the peripheral device 220 or its direction, a light output process for outputting a predetermined light capable of recognizing the presence of the agent from the peripheral device 220 or its direction, a smell generation process for generating a predetermined smell capable of recognizing the presence of the agent from the peripheral device 220 or its direction, a reaction process for changing the reaction from the peripheral device 220 or its direction, and a movement process for the agent to move from the first output device 200 (or the second output device 210) to the peripheral device 220 is executed to execute an effect that makes the driver feel the presence of the agent (steps S525 to S527).

[0169] According to this configuration, even when there is no predetermined information to be presented to the driver D1, although the appearance of the agent is difficult to recognize, it is possible to give the driver D1 a feeling of existing in the same space as the driver D1.

[0170] In the information processing method according to the present embodiment, the peripheral device 220 is a device installed around the driver D1 and can be touched by the driver D1, and is at least one of a shift lever, a steering wheel, a seating seat, an armrest, and a seat belt.

[0171] According to this configuration, by using a device installed around the driver D1 and capable of being touched by the driver D1 during driving, it is possible to give the driver D1 a feeling that the agent exists in the same space as the driver D1.

[0172] In the information processing method according to the present embodiment, a line-of-sight direction detection process for detecting the line-of-sight direction of the driver D1 is further included by performing at least one of a prediction process of the line of sight of the driver D1 based on the position information of the vehicle C1 and map information (obtainable from a map information DB or externally) and a detection process of the line of sight of the driver D1 based on an image including the eyes of the driver D1. In the control process, when predetermined information exists, among a plurality of devices, a device that enters the vision of the driver D1 obtained based on the line-of-sight direction of the driver D1 is used as the first device (steps S518 to S521). For example, in step S519, it is possible to execute the line-of-sight direction detection process.

[0173] According to this configuration, by using, as the first device, a device that enters the vision of the driver D1 obtained based on the line-of-sight direction of the driver D1 during driving, an increase in the driving load of the driver D1 can be suppressed, and the predetermined information can be appropriately presented to the driver D1.

[0174] In the information processing method according to this embodiment, the first output device 200 is a display device. Further, the information processing method further includes a driving load determination process (step S501) for determining the driving load of the driver D1 based on at least one of environmental information regarding the environment outside the vehicle C1 (obtainable by the vehicle exterior situation acquisition unit 124) and driving behavior information regarding the driving behavior of the driver D1 (obtainable by the driver situation acquisition unit 122 and the vehicle information acquisition unit 125). In the control process, when the driving load is less than a predetermined value, regardless of the presence of predetermined information, an agent image AG2 representing an agent is displayed on the second output device 210 (an example of the first device), and communication is executed using the agent image AG2 (steps S504 to S509, S512).

[0175] According to this configuration, when the driving load of the driver D1 is less than a predetermined value, by executing communication using the agent image AG2, it is possible to enhance the attachment to the agent image AG2 and improve the transmission ability of the information transmitted from the agent image AG2 to the driver D1.

[0176] In the information processing method according to this embodiment, the first device is a plurality of display devices including the second output device 210 (an example of the first display unit) and the first output device 200 (an example of the second display unit) located at a position farther from the driver D1 than the second output device 210. In the control process (steps S502 to S508, S512, S518 to S521), when the driving load of the driver D1 is less than a predetermined value, the agent image AG2 is displayed on the second output device 210, and when the driving load of the driver D1 is greater than or equal to the predetermined value and predetermined information exists, the agent image AG1 is displayed on the first output device 200, and the predetermined information is presented using the agent image AG1.

[0177] According to this configuration, the agent images AG1 and AG2 can be displayed using an appropriate display device according to the driving load of the driver D1. Thereby, it is possible to suppress an increase in the driving load of the driver D1 and appropriately present the predetermined information to the driver D1.

[0178] In the information processing method according to this embodiment, in the control process, when switching between the first control for presenting predetermined information using an agent by using the first output device 200 or the second output device 210 (an example of the first device) and the second control for executing an effect that makes the driver feel the presence of the agent by using the peripheral device 220 (an example of the second device), a movement effect in which the agent moves between the first output device 200 (or the second output device 210) and the peripheral device 220 is executed (steps S504, S510, S514, S519, S522, S526). For example, as shown in FIGS. 7 to 10, the movement effect is executed.

[0179] According to this configuration, since it becomes easier for the driver D1 to grasp the destination and the source of movement, it is possible to give the driver D1 a feeling that the agent exists in the same space as the driver D1.

[0180] In the information processing method according to this embodiment, the predetermined information is support information presented to the driver D1 and having an importance level equal to or higher than a threshold value.

[0181] According to this configuration, since the expression mode of the agent can be determined based on the support information having an importance level equal to or higher than the threshold value among the support information presented to the driver D1, an increase in the driving load of the driver D1 can be suppressed, and the predetermined information can be appropriately presented to the driver D1.

[0182] In the information processing method according to this embodiment, the peripheral device 220 may be a device located in a place that is difficult to enter the driver D1's vision.

[0183] According to this configuration, it becomes possible to appropriately execute an effect that hides the appearance of the agent when it is not necessary.

[0184] The information processing device 110 is an information processing device that executes control for communicating with the driver D1 by using the first output device 200, the second output device 210, and the peripheral device 220 (an example of a plurality of devices) in the vehicle C1. The information processing device 110 includes a communication content determination unit 126 (an example of a determination unit) that determines whether there is predetermined information to be presented to the driver D1, and when there is predetermined information, among the plurality of devices, the first output device 200 (an example of a first device) (or the second output device 210) located in a place where it is easy for the driver D1 to visually recognize is used, and an agent (for example, agent image AG1) for communicating is made visible to the driver D1, and the predetermined information is presented using the agent (for example, agent image AG1). When there is no predetermined information, among the plurality of devices, the peripheral device 220 (an example of a second device) other than the first output device 200 is used to execute an effect that makes the driver D1 feel the presence of the agent in a mode that is difficult to recognize (for example, heat generation). The information processing device 110 includes an output mode determination unit 127 and an agent control unit 128 (an example of a control unit). Note that the information processing device 110 may be a device built in any of the first output device 200, the second output device 210, and the peripheral device 220, or may be a device different from the first output device 200, the second output device 210, and the peripheral device 220. Further, instead of the information processing device 110, an information processing system configured by a plurality of devices capable of executing each process realized by the information processing device 110 may be used.

[0185] According to this configuration, by adjusting the presence of the agent according to the presence or absence of the predetermined information presented to the driver D1, it becomes possible to give the impression that the agent is riding in the vehicle C1 with the driver D1, and it becomes possible to enhance the feeling that the driver D1 is with the agent. That is, it becomes possible to enhance the sense of unity between the driver D1 and the agent. As a result, it is possible to enhance the reliability, affection, etc. for the agent, and it is possible to enhance the transmission ability of the information transmitted from the agent to the driver D1.

[0186] Note that each processing procedure shown in this embodiment is an example for realizing this embodiment. Within the range where this embodiment can be realized, the order of some of the processing procedures may be swapped, and some of the processing procedures may be omitted or other processing procedures may be added.

[0187] Note that each process of this embodiment is executed based on a program for causing a computer to execute various processing procedures. This embodiment can also be understood as an embodiment of a program that realizes the functions of executing those respective processes and a recording medium that stores that program. For example, by an update process for adding a new function to an information processing apparatus, the program can be stored in the storage device of the information processing apparatus. Thereby, it becomes possible to cause the updated information processing apparatus to execute each process shown in this embodiment.

[0188] As described above, embodiments of the present invention have been explained. However, the above embodiments merely show application examples of the present invention, and are not intended to limit the technical scope of the present invention to the specific configurations of the above embodiments.

Explanation of Reference Numerals

[0189] 20 Network, 100 Information Processing System, 101 Sound Acquisition Unit, 102 Driver Image Acquisition Unit, 103 Cabin Image Acquisition Unit, 104 Out-of-Vehicle Image Acquisition Unit, 105 Vehicle Information Input Unit, 110 Information Processing Apparatus, 120 Control Unit, 121 Speech Acquisition Unit, 122 Driver Situation Acquisition Unit, 123 Cabin Situation Acquisition Unit, 124 Out-of-Vehicle Situation Acquisition Unit, 125 Vehicle Information Acquisition Unit, 126 Communication Content Determination Unit, 127 Output Mode Determination Unit, 128 Agent Control Unit, 130 Storage Unit, 131 Agent Information DB, 132 Agent Management DB, 140 Communication Unit, 200 First Output Device, 201, 211 Display Unit, 202, 212 Sound Output Unit, 210 Second Output Device, 220 Peripheral Device, 221 Heater Unit

Claims

1. An information processing method for performing control to communicate with a driver by using a plurality of devices in a vehicle, comprising: a determination process for determining whether there is predetermined information to be presented to the driver; when the predetermined information exists, using a first device among the plurality of devices, which is located in a place where it is easy for the driver to visually notice, to make an agent for performing the communication visible to the driver, and presenting the predetermined information using the agent; when the predetermined information does not exist, a control process for performing an effect that makes the driver feel the presence of the agent in a manner that is difficult for the driver to recognize, by using a second device among the plurality of devices other than the first device; the information processing method.

2. The information processing method according to claim 1, wherein: the first device is a display device; in the control process: when the predetermined information exists, causing the first device to display an agent image representing the agent; when the predetermined information does not exist, without causing the agent image to be displayed on both the first device and the second device, performing the effect that makes the driver feel the presence of the agent using a sense other than vision among the five senses of the driver, by the second device; the information processing method.

3. The information processing method according to claim 1, wherein: the second device is a peripheral device installed around the driver; in the control process, by performing at least one of a temperature adjustment process for adjusting the temperature of the second device, a vibration process for vibrating the second device, a sound output process for outputting a predetermined sound capable of recognizing the presence of the agent from the second device or its direction, a light output process for outputting a predetermined light capable of recognizing the presence of the agent from the second device or its direction, a smell generation process for generating a predetermined smell capable of recognizing the presence of the agent from the second device or its direction, a reaction process for changing the reaction from the second device or its direction, and a movement process for moving the agent from the first device to the second device, performing the effect; the information processing method.

4. The information processing method according to claim 3, wherein: The peripheral device is installed around the driver and is a device that can be touched by the driver, and is at least one of a shift lever, a steering wheel, a seating seat, an armrest, and a seat belt. Information processing method.

5. The information processing method according to claim 1, By performing at least one of the prediction process of the driver's line of sight based on the position information and map information of the vehicle and the detection process of the driver's line of sight based on an image including the driver's eyes, a line-of-sight direction detection process for detecting the driver's line-of-sight direction is further included. In the control process, when the predetermined information exists, among the plurality of devices, the device that enters the driver's vision obtained based on the driver's line-of-sight direction is used as the first device. Information processing method.

6. The information processing method according to claim 1, The first device is a display device, Further including a driving load determination process for determining the driving load of the driver based on at least one of environmental information regarding the environment outside the vehicle and driving behavior information regarding the driving behavior of the driver. In the control process, when the driving load is less than a predetermined value, an agent image representing the agent is displayed on the first device, and the communication is executed using the agent image. Information processing method.

7. The information processing method according to claim 6, The first device is a plurality of display devices including a first display unit and a second display unit located at a position farther from the driver than the first display unit. In the control process, When the driving load is less than a predetermined value, the agent image is displayed on the first display unit. When the driving load is greater than or equal to the predetermined value and the predetermined information exists, the agent image is displayed on the second display unit, and the predetermined information is presented using the agent image. Information processing method.

8. The information processing method according to any one of claims 1 to 7, In the control process, when switching between the first control for presenting the predetermined information using the agent using the first device and the second control for performing the effect using the second device, a movement effect in which the agent moves between the first device and the second device is executed. Information processing method.

9. An information processing method according to any one of claims 1 to 7, wherein the predetermined information is support information among the support information presented to the driver and having an importance level equal to or higher than a threshold value. Information processing method.

10. An information processing method according to any one of claims 1 to 7, wherein the second device is a device located in a place difficult to enter the driver's vision. Information processing method.

11. An information processing apparatus that executes control for communicating with a driver using a plurality of devices in a vehicle, a determination unit that determines whether or not there is predetermined information to be presented to the driver, when the predetermined information exists, using a first device located in a place easy to enter the driver's vision among the plurality of devices, presenting the predetermined information using the agent with the agent for performing the communication made visible to the driver, when the predetermined information does not exist, a control unit that executes an effect for making the driver feel the presence of the agent in a mode difficult to be recognized by the driver, using a second device other than the first device among the plurality of devices. Information processing apparatus.

12. A program for causing a computer to execute control for communicating with a driver using a plurality of devices in a vehicle, a determination process for determining whether or not there is predetermined information to be presented to the driver, when the predetermined information exists, using a first device located in a place easy to enter the driver's vision among the plurality of devices, presenting the predetermined information using the agent with the agent for performing the communication made visible to the driver, when the predetermined information does not exist, a control process for executing an effect for making the driver feel the presence of the agent in a mode difficult to be recognized by the driver, using a second device other than the first device among the plurality of devices, A program for causing a computer to execute.

Images