Information processing method and information processing device

Abstract

To more appropriately transmit information by using feeling expressions that can be perceived by a user in accordance with a user's state or the like.SOLUTION: An information processing method for controlling an agent apparatus 200 includes estimation processing (steps S502, S522) for estimating a type of a feeling perceptible by a user U1, determination processing (steps S503, S523) for determining whether the user U1 can perceive a first feeling expression to be transmitted to the user U1 which has been already determined, extraction processing (steps S505, S511, S524, S531, S543) for extracting a second feeling similar to the first feeling, and control processing (steps S504, S506, S511, S512, S525) for making the agent apparatus 200 execute first performance to express the first feeling if the user U1 can perceive the expression of the first feeling, and making the agent apparatus 200 execute second performance to express a second feeling unless the user U1 can perceive the expression of the first feeling.SELECTED DRAWING: Figure 5

Description

[Technical field]

[0001] The present invention relates to an information processing method and an information processing apparatus for controlling an agent device capable of communicating with a user. [Background technology]

[0002] Conventionally, there have been proposed techniques for providing various types of information to users. For example, a technique has been proposed in which the voice quality and timing of providing information are determined using the attributes of the driver of a vehicle, the driving load, and the type of information to be provided, and the information is output to the driver (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2008-301382 A Summary of the Invention [Problem to be solved by the invention]

[0004] In the above-mentioned conventional technology, when the type of information to be provided is emergency, the information is output using the voice quality of a middle-aged man who has excellent information transmission ability. When the type of information to be provided is normal, the timing of providing the information is changed if the driving load is high, and when the driving load is low, the information is output using a voice quality that is common to the driver in terms of gender and age. In this way, when the driving load is low, the information is output using a predetermined voice quality, assuming that the driver has enough capacity to receive the information. However, for example, when the driving load is low but the driver's level of awakening is low, it is assumed that the driver is not in a state where he or she can perceive the information. In other words, when the driver is in a state where it is difficult to perceive the information, there is a risk that the information cannot be transmitted appropriately.

[0005] An object of the present invention is to more appropriately convey information using emotional expressions that the user can perceive, depending on the user's state, etc. [Means for solving the problem]

[0006] One aspect of the present invention is an information processing method for controlling an agent device capable of communicating with a user. This information processing method includes an estimation process for estimating a type of emotion perceivable by the user based on at least one of a plurality of elements related to the state of the user, a determination process for comparing the type of the estimated emotion with a first emotion to be transmitted to the user that has already been determined and determining whether the user is capable of perceiving the expression of the first emotion, an extraction process for extracting a second emotion similar to the first emotion from the types of the estimated emotions when it is determined that the user is not capable of perceiving the expression of the first emotion, and a control process for causing the agent device to execute a first performance expressing the first emotion when it is determined that the user is capable of perceiving the expression of the first emotion, and causing the agent device to execute a second performance expressing the second emotion when it is determined that the user is not capable of perceiving the expression of the first emotion. Effect of the Invention

[0007] According to the present invention, it is possible to more appropriately convey information using emotional expressions that the user can perceive, depending on the user's state, etc. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a simplified diagram showing an example of the configuration inside a vehicle cabin. [Diagram 2] FIG. 2 is a perspective view showing an example of the external configuration of an agent device in a simplified manner. [Diagram 3] FIG. 3 is a diagram illustrating an example of a system configuration of the information processing system. [Figure 4] FIG. 4 is a diagram showing a wheel of emotions defined by emotion values ​​and arousal levels. [Diagram 5] FIG. 5 is a flowchart showing an example of the control process. [Figure 6] FIG. 6 is a simplified diagram showing an example of the operation of an agent device. [Figure 7]FIG. 7 shows an emotion model table and a calculation result table. [Figure 8] FIG. 8 is a diagram showing a simplified example of an apology production operation of an agent device. [Figure 9] FIG. 9 is a flowchart showing an example of the control process. [Figure 10] FIG. 10 shows a calculation result table and a visual and auditory margin table. [Figure 11] FIG. 11 is a flowchart showing an example of the control process. [Figure 12] FIG. 12 is a diagram showing an extraction process for extracting a second emotion in a predetermined period. [Figure 13] FIG. 13 is a flowchart showing an example of the control process. [Figure 14] FIG. 14 is a diagram showing an extraction process for extracting a second emotion in a predetermined period. [Figure 15] FIG. 15 is a flowchart showing an example of the control process. [Figure 16] FIG. 16 shows the calculation result table and the primary information table. [Figure 17] FIG. 17 is a flowchart showing an example of the control process. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, an embodiment of the present invention will be described with reference to the accompanying drawings.

[0010] [First embodiment] [Agent device installation example] Fig. 1 is a simplified diagram showing an example of the configuration of the interior of a vehicle C1. Fig. 1 shows the interior of the vehicle C1 in a simplified manner, viewed from the rear in the longitudinal direction of the vehicle C1, in front of the driver's seat and passenger seat (not shown). To facilitate explanation, Fig. 1 omits illustrations of components other than a dashboard 2, a steering wheel 3, a windshield 4, a rearview mirror 5, an image acquisition unit 101, and an agent device 200.

[0011] The agent device 200 is a small robot installed on the dashboard 2 of the vehicle C1. In each embodiment, an example is shown in which the agent device 200 is a robot imitating a human face. FIG. 1 shows an example in which the agent device 200 is installed on the dashboard 2, but the present invention is not limited thereto. For example, the agent device 200 may be installed on the upper part of the windshield 4. FIG. 1 also shows an example in which the agent device 200 is a robot imitating a human, but the present invention is not limited thereto. For example, the agent device 200 may be a robot imitating an animal such as a rabbit or a pig, a robot imitating a virtual creature (for example, the face of an anime character), or a robot imitating another object (for example, a television-type device, a radio-type device). In each embodiment, an example is shown in which a face image is displayed as an image representing the likeness of a living thing (living thing display), but other images (for example, the whole body of the agent, a mechanical object, a virtual face) may be displayed as an image representing the likeness of a living thing. In this way, it is possible to use the agent device 200 as an agent that has been imitated as a living thing.

[0012] 1 shows an example in which the agent device 200 is installed on the dashboard 2 of the vehicle C1 in a direction in which the eyes of the face 210 face the inside of the vehicle. In each embodiment, an example is shown in which a face image is displayed on the display unit 230 (see FIG. 3) when the agent device 200 is in an on state. The agent device 200 can be turned on in response to an on operation of the vehicle C1. The agent device 200 can be turned on in response to a user operation on the agent device 200. The on / off operation of the vehicle C1 refers to the on or off operation of a start key related to starting or stopping the vehicle C1.

[0013] The agent device 200 is a device that is composed of a substantially spherical face 210 and a body 220, and the body 220 is provided with left and right arms 221 and left and right legs 222. A display unit 230 (see FIG. 3) is provided on at least a part of the surface of each of these parts (for example, the face 210). For example, the display unit 230 may be provided on the entire surface of the face 210 and the body 220 (including the arms 221 and the legs 222), or the display unit 230 may be provided on a part of the surface of the face 210 and the body 220. When the display unit 230 is provided on a part of the surface of the face 210 and the body 220, each physical part (for example, a mouth, eyes, nose, fingers) that constitutes the body, face, etc. may be provided on the part where the display unit 230 is not provided.

[0014] The display unit 230 can be configured as a display medium capable of displaying various images. As this display medium, various display media such as a light-emitting sheet, a liquid crystal display, an organic EL (Electro Luminescence), etc. For example, the display unit 230 of the face 210 displays eyes, a mouth, etc.

[0015] The body part 220 rotates in the front-rear direction of the agent device 200 with the lower part of the agent device 200 as the center of rotation. Similarly, the body part 220 rotates in the left-right direction of the agent device 200 with the lower part of the agent device 200 as the center of rotation. Similarly, the body part 220 rotates in the left-right direction with the lower part of the agent device 200 as the center of rotation and the up-down direction of the body part 220 as the axis. In this way, a drive unit 250 (see FIG. 3) for driving the body part 220 is provided at the lower part of the agent device 200. Then, it becomes possible to rotate the body part 220 of the agent device 200 in the left-right direction, rotate the body part 220 in the front-rear direction, swing the body part 220 in the left-right direction, and swing the body part 220 in the front-rear direction. The operation mode of the agent device 200 will be described in detail with reference to FIG. 2.

[0016] 1 shows an example of agent device 200 having face portion 210 and body portion 220, but agent device 200 having only face portion 210 may also be used. In this case, each emotion (for example, irritation, anger) may be realized by changing face portion 210.

[0017] 1 shows an example in which the agent device 200 is configured with a substantially spherical face portion 210 and a body portion 220, but is not limited to this. For example, the agent device 200 may be a device in which a flat body portion (e.g., a planar display panel) functions as a face portion. For example, the agent device 200 may be a device in which the face portion and the body portion are configured as an integrated housing. For example, the agent device 200 may be a device in which the face portion and the body portion are configured as separate bodies. For example, the agent device 200 may be a display device that displays the face portion and the body portion.

[0018] The agent device 200 executes various operations based on instructions from the information processing device 110 (see FIG. 3). For example, the agent device 200 outputs various information related to driving support when the user U1 drives and surrounding facilities under the control of the information processing device 110. As the driving support, a notification of a moving object in front or behind is assumed. For example, as a notification of a moving object in front, a voice output of "Be careful because there is a railroad crossing ahead" or a voice output of "There is a person ahead" can be used. In this way, the agent device 200 executes driving support. Also, for example, the agent device 200 executes various processes under the control of the information processing device 110. For example, the agent device 200 executes a communication process for carrying out various exchanges with the user U1. For example, the agent device 200 exchanges various conversations with the user U1 and provides information that the user U1 likes. FIG. 1 shows an example of providing store information introducing nearby restaurants that serve delicious steaks to a meat-loving user U1.

[0019] The image acquisition unit 101 is provided on the ceiling inside the vehicle C1, and captures an image of a subject inside the vehicle C1 to generate an image (image data). The image acquisition unit 101 is, for example, configured with one or more camera devices or image sensors capable of capturing an image of a subject. For example, the image acquisition unit 101 can be provided on the upper part of the windshield 4, that is, above the rearview mirror 5. Although FIG. 1 shows an example in which at least one image acquisition unit 101 is provided, two or more imaging devices may be provided, and all or some of the images from these imaging devices may be used. The installation locations of the imaging devices are not limited to the example shown in FIG. 1, and may be changed as appropriate. One or more devices capable of capturing subjects present in all directions of the vehicle C1 and subjects inside the vehicle C1, for example, a 360-degree camera, may be used.

[0020] Here, the agent device 200 has a function of expressing and conveying emotions to the user U1 in order to ensure smooth communication when communicating with the user U1. For example, if the agent device 200 wants to communicate properly with the user U1, it is considered that the agent device 200 expresses an irritated emotion and conveys it to the user U1. Also, for example, when the user U1 is talking with the agent device 200 and the user U1 says, "I don't understand what you're saying," the agent device 200 may express an irritated emotion by informing the user U1 that the words of the user U1 are irritating. Also, it is considered that the agent device 200 expresses emotions when providing directions, announcing danger, etc. For example, when the agent device 200 informs the user U1 that there has been an accident 10 km ahead, it is considered that the agent device 200 expresses a slightly uneasy emotion. Note that the agent device 200 may use voice, image, vibration, movement, etc. as an expression means for conveying emotions to the user U1. Note that specific expressions of emotions will be described with reference to FIG. 2.

[0021] Here, in a situation where the user U1 is driving the vehicle C1, the user U1 may not see the entire agent device 200. For example, when the user U1 is driving the vehicle C1 and there are many traffic participants (e.g., other vehicles, pedestrians) around the vehicle C1, the user U1 needs to concentrate his / her attention on the surroundings of the vehicle C1, and the driving load of the user U1 increases. In this case, it is assumed that the user U1 will be in a situation where he / she can vaguely see the agent device 200 with his / her peripheral vision. Similarly, when the user U1 has a low level of awakening, it is assumed that the user U1 will be in a situation where he / she can vaguely see the agent device 200. Note that the level of awakening is an index indicating the degree to which the user is awake and conscious, and the higher the value, the more awake and conscious the user is. In other words, when the driving load of the user U1 is high or when the user U1 has a low level of alertness, the user U1 cannot pay attention to the agent device 200. Therefore, even if the agent device 200 expresses an emotion with a subtle facial expression such as irritation, it is expected that the user U1 will have difficulty perceiving the emotional expression.

[0022] However, even when the driving load of the user U1 is high, it is assumed that extreme and easy-to-understand emotions, such as anger, laughter, sadness, etc., are easily perceived by the user U1. However, it is assumed that emotions with a low expression level, such as irritation, are difficult to be perceived by the user U1 when the driving load of the user U1 is high. Therefore, in each embodiment, when the driving load of the user U1 is high or the awakening level of the user U1 is low, it is unavoidable to temporarily express a tentative emotion that is easy to understand, so that mutual communication can be smoothly performed. In each embodiment, the human ability to see, hear, and recognize something through the human senses is described as perception. However, instead of perception, it may be called cognition.

[0023] [Example of agent device appearance configuration] 2 is a perspective view showing, in a simplified form, an example of the external configuration of the agent device 200. In each embodiment, an example is shown in which, when the agent device 200 is in an on state, the agent device 200 is in a basic state or in a state in which various emotions are expressed. That is, when the agent device 200 executes various communications with the user U1, the information processing device 110 (see FIG. 5) executes control to switch the emotional expression of the agent device 200 according to the content of the communication.

[0024] Fig. 2(A) shows a front view of agent device 200 in a basic state. Fig. 2(B) to (F) show front views of agent device 200 in an irritated emotional state, an upset emotional state, an angry emotional state, a fearful emotional state and an apologetic emotional state, respectively. Note that agent device 200 shown in Fig. 2(A) corresponds to agent device 200 shown in Fig. 1.

[0025] Agent device 200 is a robot device in which a roughly spherical face 210 is provided on the top of a prolate ellipsoid body 220. Rod-shaped arms 221 are provided on the left and right sides of body 220, and rod-shaped legs 222 are provided on the left and right lower part of body 220. Various images, such as a mouth and eyes, are displayed on the surface of face 210.

[0026] For example, as shown in Fig. 2(A), normal eyes and mouth can be displayed. Also, for example, as shown in Fig. 2(B) to (F), eyes and mouth corresponding to the emotion that the user U1 wants to convey can be displayed on the face 210.

[0027] The face unit 210 rotates in the up-down and left-right directions of the agent device 200 with the lower part of the face unit 210 as the center of rotation. For example, as shown in Fig. 2(F), the face unit 210 rotates downward in the up-down direction. In this manner, the agent device 200 is provided with a drive unit 250 for driving the face unit 210. It is then possible to rotate and move the face unit 210 of the agent device 200 in the left-right direction, to rotate and move the face unit 210 in the up-down direction, to rock the face unit 210 in the left-right direction, and to rock the face unit 210 in the up-down direction.

[0028] The arm 221 rotates in the front-rear, left-right, up-down, etc. directions of the agent device 200 around the attachment part of the arm 221 on the body part 220. The central part of the arm 221 rotates in the front-rear, left-right, up-down, etc. directions of the agent device 200 around the central part.

[0029] In this way, a driving unit 250 for driving the arm unit 221 is provided at the attachment portion and central portion of the arm unit 221. It is then possible to move the arm unit 221 up and down, move the arm unit 221 back and forth, swing the arm unit 221 up and down, and swing the arm unit 221 back and forth. The driving unit 250 that drives each portion of the agent device 200 performs each operation based on the control of the information processing device 110 (see FIG. 3).

[0030] For example, as shown in Fig. 2(B) to (F), the eyes and mouth corresponding to the emotion to be conveyed to the user U1 are displayed on the face 210, and the arm 221 is moved corresponding to the emotion, so that the emotion can be expressed. Also, as shown in Fig. 2(D) to (F), the additional images 211 to 213 corresponding to the emotion to be conveyed to the user U1 are displayed on the display unit 230, so that the emotion can be emphasized and expressed. For example, by changing at least a part of the surface of the face 210 to a predetermined color (e.g., blue, red, black) corresponding to the emotion to be conveyed to the user U1, the emotion can be further emphasized and expressed. Also, as shown in Fig. 2(E), it is also possible to express a state in which the agent device 200 is trembling by vibrating at least a part of the face 210 and the body 220.

[0031] In this way, in each embodiment, it is possible to make the agent device 200 express various emotions by changing at least some of the components of the agent device 200. Furthermore, the user U1 of the vehicle C1 who sees the various emotions can perceive that the agent device 200 is trying to communicate something. Note that these shapes and display modes of the agent device 200 are merely examples, and each emotion may be expressed by other shapes and display modes.

[0032] Furthermore, these operation modes of the agent device 200 are merely examples, and other operation modes are also possible. For example, the agent device 200 can be slid within a predetermined range on the dashboard 2. For example, the agent device 200 can be slid in the forward / backward and left / right directions. Also, for example, it is possible to produce an effect in which the agent device 200 is temporarily floating.

[0033] 2 shows an example in which each facial expression is expressed two-dimensionally by changing the display mode of the display unit 230, but each facial expression may be expressed three-dimensionally by changing each physically installed member such as the eye, nose, mouth, etc. In this case, for example, the movement of the eye may be expressed by moving the eye member or displaying a black image or a white image on the surface of the eye member.

[0034] [Example of information processing system configuration] FIG. 3 is a diagram showing an example of a system configuration of an information processing system 100 installed in a vehicle C1.

[0035] The information processing system 100 includes an image acquisition unit 101, a position information acquisition unit 102, a sound input unit 103, sensors 104, an information processing device 110, and an agent device 200. The information processing device 110 is an example of a device that controls the agent device 200 that is capable of communicating with a user U1.

[0036] The information processing device 110 and the agent device 200 are connected by a communication method using wired communication or wireless communication. The information processing device 110 is connected to the network 20 by a communication method using wireless communication. The network 20 is a network such as a public line network or the Internet. The agent device 200 may also be connected to the network 20 by a communication method using wireless communication. Although FIG. 3 shows an example in which the information processing device 110 and the agent device 200 are configured as separate entities, the information processing device 110 and the agent device 200 may also be configured as an integrated device.

[0037] The image acquisition unit 101 captures an image of a subject to generate an image (image data), and outputs image information related to the generated image to the information processing device 110. The image acquisition unit 101 is provided at least inside the vehicle C1, and captures an image of a subject inside the vehicle C1 to generate an image (image data). Note that FIG. 1 shows the image acquisition unit 101 provided inside the vehicle C1. As described above, the image acquisition unit 101 is configured, for example, with one or more camera devices or image sensors capable of capturing an image of a subject. For example, one image acquisition unit 101 may be provided in front of the vehicle C1 to capture an image of a subject from in front of the vehicle C1 to generate an image (image data), and another image acquisition unit 101 may be provided behind the vehicle C1 to capture an image of a subject behind the vehicle C1 to generate an image (image data).

[0038] The location information acquisition unit 102 acquires location information related to the location where the vehicle C1 is located, and outputs the acquired location information to the information processing device 110. For example, it can be realized by a GNSS receiver that acquires location information using a Global Navigation Satellite System (GNSS). The location information includes data related to the position such as latitude, longitude, and altitude at the time of receiving the GNSS signal. The location information may also be acquired by other methods of acquiring location information. For example, the location information may be derived using information from access points or base stations present in the vicinity. The location information may also be acquired using a beacon.

[0039] The sound input unit 103 is provided inside the vehicle C1, acquires sounds inside the vehicle C1, and outputs sound information related to the acquired sounds to the information processing device 110. As the sound input unit 103, for example, one or more microphones or sound acquisition sensors can be used.

[0040] The sensors 104 are various sensors installed in the vehicle C1, and output detection information acquired by each sensor to the information processing device 110. The sensors are, for example, LiDAR (Light Detection And Ranging), RADAR (Radio Detection And Ranging), Sonar, a vehicle speed sensor, an acceleration sensor, a steering sensor (steering force angle meter), an accelerator position sensor, an illuminance sensor, a temperature sensor, a non-contact temperature sensor, and the like. For each of these sensors, a known sensor can be used. Note that LiDAR, RADAR, Sonar, and the like are examples of sensors that detect the situation around the vehicle C1. Also, the vehicle speed sensor, the acceleration sensor, the steering sensor, the accelerator position sensor, and the like are examples of sensors that detect the situation of the driving operation of the user U1. Note that these are only examples, and other sensors may be used. Also, only some of these sensors may be used.

[0041] The information processing device 110 includes a control unit 120, a storage unit 130, and a communication unit 140. The communication unit 140 exchanges various types of information with other devices based on the control of the control unit 120 by using wired communication or wireless communication.

[0042] The control unit 120 controls each unit based on various programs stored in the storage unit 130. The control unit 120 is realized by a processing device such as a central processing unit (CPU) or a graphics processing unit (GPU). Note that the vehicle ECU (electronic control unit) of the vehicle C1 may be used as the control unit 120, or a processing device different from the vehicle ECU may be provided as the control unit 120.

[0043] The control unit 120 executes various controls based on the information output from the image acquisition unit 101, the position information acquisition unit 102, the sound input unit 103, the sensors 104, the communication unit 140, etc. For example, the control unit 120 executes a control process for controlling the operation state of the agent device 200. Specifically, the control unit 120 includes a feeling estimation unit 121, a determination unit 122, an extraction unit 123, a certainty estimation unit 124, and an agent control unit 125.

[0044] The emotion estimation unit 121 estimates the type of emotion that the user U1 can perceive based on at least one of a plurality of elements related to the state of the user U1. Then, the emotion estimation unit 121 outputs the estimation result to the determination unit 122, the extraction unit 123, the confidence factor estimation unit 124, and the agent control unit 125. For example, the emotion estimation unit 121 estimates the type of emotion that the user U1 can perceive based on each piece of information output from the image acquisition unit 101, the position information acquisition unit 102, the sound input unit 103, the sensors 104, the communication unit 140, and the like, using spatial data (coordinate data) that takes into account the circle of emotions (see FIG. 4), Ekman's six basic emotions (see FIG. 7), and the like. These estimation methods will be described in detail with reference to FIG. 4, FIG. 7, and the like.

[0045] The determination unit 122 compares the type of emotion that the user U1 can perceive, estimated by the emotion estimation unit 121, with a first emotion to be conveyed to the user U1, and determines whether the user U1 can perceive the expression of the first emotion. Then, the determination unit 122 outputs the determination result to the determination unit 122, the extraction unit 123, the confidence estimation unit 124, and the agent control unit 125. Here, the first emotion is an emotion that the agent device 200 should convey to the user U1. The method of making these determinations will be described in detail with reference to Figs. 4, 7, etc.

[0046] When the determination unit 122 determines that the user U1 cannot perceive the expression of the first emotion, the extraction unit 123 extracts a second emotion similar to the first emotion from among the types of emotions that the user U1 can perceive, which are estimated by the emotion estimation unit 121. The extraction unit 123 then outputs the extraction result to the agent control unit 125. That is, the second emotion is an example of an emotion that can be expressed instead of the first emotion, and can also be called an emotion that is highly related to the first emotion, an emotion related to the first emotion, an emotion that is closely related to the first emotion, or the like. For example, the second emotion similar to the first emotion is extracted using the wheel of emotions (see FIG. 4) and Ekman's six basic emotions (see FIG. 7). The extraction method will be described in detail with reference to FIG. 4, FIG. 7, and the like.

[0047] The certainty estimation unit 124 estimates the certainty of transmission for each emotion that the agent device 200 can express when transmitting the emotion to the user U1, based on the user U1's visual composure, the user U1's auditory composure, etc. Then, the certainty estimation unit 124 outputs the estimation result to the extraction unit 123. The certainty of transmission is an index showing the likelihood of information transmission. These estimation methods will be described in detail with reference to Figs. 10, 16, etc.

[0048] The agent control unit 125 controls the operating state of the agent device 200. Specifically, the agent control unit 125 controls the operating state of the agent device 200 based on each piece of information output from the emotion estimation unit 121, the determination unit 122, the extraction unit 123, the confidence estimation unit 124, etc., and each piece of information output from the image acquisition unit 101, the position information acquisition unit 102, the sound input unit 103, the sensors 104, the communication unit 140, etc. For example, the agent control unit 125 causes the display unit 230 to display each part (e.g., eyes, mouth) constituting the face unit 210 of the agent device 200. Also, for example, the agent control unit 125 causes the sound output unit 240 to output the voice of the agent device 200, the voice corresponding to the information to be notified to the user U1, etc. Also, for example, the agent control unit 125 causes the agent device 200 to execute various operations. For example, the agent control unit 125 determines an emotion (first emotion) to be conveyed to the user U1, and causes the agent device 200 to express the first emotion.

[0049] Furthermore, when the determination unit 122 determines that the user U1 is capable of perceiving the expression of the first emotion, the agent control unit 125 causes the agent device 200 to execute a first performance expressing the first emotion. On the other hand, when the determination unit 122 determines that the user U1 is not capable of perceiving the expression of the first emotion, the agent control unit 125 causes the agent device 200 to execute a second performance expressing the second emotion extracted by the extraction unit 123. Each of these performances will be described in detail with reference to FIG. 5 etc.

[0050] The storage unit 130 is a storage medium that stores various types of information. For example, the storage unit 130 stores various types of information (for example, a control program, an agent information DB 131, a communication information DB 132, an emotion information DB 133, and a map information DB 134) required for the control unit 120 to perform various processes. The storage unit 130 also stores various types of information acquired via the communication unit 140. As the storage unit 130, for example, a ROM (Read Only Memory), a RAM (Random Access Memory), an SRAM (Static Random Access Memory), an HDD (Hard Disk Drive), an SSD (Solid State Drive), or a combination of these can be used.

[0051] The agent information DB 131 stores various types of information required to realize various operations of the agent device 200. For example, face image information (e.g., eyes, mouth, and additional images 211 to 213) displayed on the display unit 230 and audio information output from the sound output unit 240 are stored in the agent information DB 131. Also, for example, operation information for operating the agent device 200 when various communications are performed is stored in the agent information DB 131.

[0052] The communication information DB 132 stores various types of information required when the agent device 200 executes various communications with the user U1. For example, notification information required when making some kind of notification to the user U1 (for example, traffic information, accident information) is stored in the communication information DB 132. Also, for example, response information required when making some kind of response to an utterance by the user U1 is stored in the communication information DB 132. Note that each of these pieces of information may be realized by using, for example, artificial intelligence (AI).

[0053] Various types of information required for extracting second information similar to first information are stored in emotion information DB 133. For example, information related to the wheel of emotions (see FIG. 4), Ekman's six basic emotions (see FIG. 7), and the like are stored in emotion information DB 133.

[0054] The agent device 200 is a robot device that executes various operations based on instructions from the information processing device 110. The agent device 200 comprises a display unit 230, a sound output unit 240, and a drive unit 250. The display unit 230, the sound output unit 240, and the drive unit 250 are controlled by a control unit (not shown) that the agent device 200 comprises.

[0055] The display unit 230 is a display unit that displays various images based on instructions from the information processing device 110. The display unit 230 may be configured as a touch panel that allows a user to perform operation input by touching or approaching the display surface with his / her finger, or may be configured as a separate user interface.

[0056] The sound output unit 240 outputs various sounds based on instructions from the information processing device 110. For example, one or more speakers can be used as the sound output unit 240. Note that the display unit 230 and the sound output unit 240 are examples of user interfaces, and some of them may be omitted, or other user interfaces may be used.

[0057] The driving unit 250 drives the body part 220 of the agent device 200 and each part of the agent device 200 based on instructions from the information processing device 110. For example, the driving unit 250 is a driving device that realizes a mechanism for rotating the face part 210 and the arms and vibrating the body part 220, as shown in Figures 2(B) to (F). For example, the driving unit 250 is composed of a motor, a servo motor, or the like that can move the agent device 200 and rotate the face part 210, the body part 220, and the arms.

[0058] [Example of extracting perceptible emotions using emotion wheels] 4 is a diagram showing spatial data (coordinate data) relating to an emotion wheel defined by emotion value and arousal level. The emotion wheel is a two-dimensional coordinate system defined by arousal level, which indicates a person's arousal state as a numerical value, and emotion value, which indicates a person's emotion as a numerical value. The emotion wheel is an example of a two-dimensional emotion model.

[0059] In the two-dimensional coordinate system TD1, the vertical axis indicates arousal level, with higher arousal levels indicated toward the top and lower arousal levels indicated toward the bottom (i.e., calm state). The horizontal axis indicates valence levels, with higher valence levels indicated toward the right (i.e., pleasant state) and lower valence levels indicated toward the left (i.e., unpleasant state). Each emotion is located at a predetermined coordinate in the two-dimensional coordinate system TD1. For example, the corresponding emotion (e.g., enthusiasm) is located at a coordinate with both high arousal level and high valence. Also, for example, the emotion is often easier to understand as it is expressed with larger (or more exaggerated) movements as it is positioned on the outside of the circle of emotions.

[0060] In addition, in FIG. 4, anger AG1, frustration AN1, agitation DY1, and fear KF1 are illustrated as examples of each emotion. Note that the coordinate positions of each emotion shown in FIG. 4 are examples for ease of explanation, and the coordinate positions of each emotion in the two-dimensional coordinate TD1 can be changed as appropriate depending on the creator, etc., and are not limited to the positions shown in FIG. 4. In addition, in FIG. 4, for ease of explanation, an example is shown in which each emotion is arranged at each coordinate (point) in the two-dimensional coordinate TD1, but this is not limiting. For example, each emotion may be arranged in each area (not a point) in the two-dimensional coordinate TD1 and used.

[0061] [Example of operation of information processing device] Fig. 5 is a flow chart showing an example of a control process of the agent device 200 by the information processing device 110. This control process is executed by the control unit 120 (see Fig. 2) based on a program stored in the storage unit 130 (see Fig. 3). This control process shows an example of an extraction process that uses a wheel of emotions (see Fig. 4) to extract emotions that can be perceived by the user U1. This control process is executed constantly at every control cycle. This control process will be described with appropriate reference to Figs. 1 to 4.

[0062] In step S501, the agent control unit 125 determines a first emotion to be conveyed from the agent device 200 to the user U1. For example, when some conversation is taking place between the agent device 200 and the user U1, the agent control unit 125 determines the response content to the content uttered by the user U1 and the emotion at that time. For example, as shown in FIG. 1, assume that the user U1 responds to the utterance content S1 uttered by the agent device 200 (There is a restaurant up ahead that serves popular steaks!) with the utterance content S2 (I can't eat steak in the morning.). In this case, the user U1 utters irritating words to the agent device 200. In such a case, the agent control unit 125 can determine silence as the response content to the utterance content S2 uttered by the user U1 and determine "irritation" as the emotion at that time in response to the cold reply given to the meat-loving user U1 despite providing useful information. The utterance content uttered by the user U1 can be acquired based on sound information from the sound input unit 103.

[0063] In this example, an example of determining an emotion in response to the contents of an utterance made by the user U1 is shown, but an emotion can be determined in a similar manner when the agent device 200 makes some kind of response to the attitude of the user U1. For example, if the user U1 has an irritating attitude toward the agent device 200, the agent control unit 125 can determine "irritation" as the emotion toward that attitude. The attitude of the user U1 can be acquired based on image information from the image acquisition unit 101. Also, a known image recognition technology can be used for the determination process of determining the attitude of the user U1 based on an image including the face of the user U1, etc.

[0064] Also, assume that the agent device 200 proactively notifies the user U1 of some information. In this case, the agent control unit 125 determines the voice output corresponding to the notification information to be notified to the user U1, and can determine the emotion corresponding to the notification information as the emotion at that time. For example, if the notification information to be notified to the user U1 is related to traffic safety, a serious emotion can be determined. The emotion, voice information, etc. of the agent device 200 determined by the agent control unit 125 are stored in the communication information DB 132 (see FIG. 3).

[0065] In step S502, the emotion estimation unit 121 estimates the type of emotion that the user U1 can perceive based on at least one of a plurality of elements related to the state of the user U1. Here, the plurality of elements related to the state of the user U1 are, for example, the awakening level of the user U1, the driving load of the user U1, and the like. The awakening level of the user U1 is an index that indicates the awakening state of the user U1 as a numerical value, and the driving load of the user U1 is an index that indicates the degree of the load caused to the user U1 when driving the vehicle C1 as a numerical value. Each of these pieces of information can be acquired based on each piece of information acquired from the image acquisition unit 101, the position information acquisition unit 102, the sound input unit 103, and the sensors 104. In addition, based on each of these pieces of information, it is determined whether the user U1 has the mental capacity to perceive the agent device 200.

[0066] [Example of detecting user's driving load] Next, a method for detecting the driving load of the user U1 will be described. The driving load of the user U1 can be detected based on the driving operation of the user U1, the surrounding conditions of the vehicle C1, and the like.

[0067] For example, the operation status of the user U1, the situation around the vehicle C1, etc. can be acquired by the image acquisition unit 101, the sensors 104, etc. For example, if the number of accelerator operations and the number of steering operations within a predetermined time are large, it can be estimated that the driving load of the user U1 is high. In addition, the driving load of the user U1 can be estimated based on the situation around the vehicle C1, etc. For example, if the vehicle C1 is traveling on a winding road, a narrow road, a highly congested road, a road with many people, etc., it can be estimated that the driving load of the user U1 is high. In addition, the driving load of the user U1 can be estimated based on the facial expression of the user, the voice of the user U1, etc. In the following, a specific example of a method for detecting the driving load of the user U1 will be described.

[0068] For example, steering entropy can be used as the driving operation of the user U1. The steering entropy method is a measurement method that measures and estimates the load of the driver based on the smoothness of the driver's steering angle, and a known calculation method can be used. In addition, in this steering entropy method, it is possible to use a measurement result (measurement value) that is digitized as an information entropy value calculated based on time-series steering angle data.

[0069] The surrounding conditions of the vehicle C1 may include the number of traffic participants around the vehicle C1, the weather around the vehicle C1, the darkness around the vehicle C1, the shape of the road around the vehicle C1, and the like.

[0070] The traffic participants refer to various moving objects such as automobiles, motorcycles, bicycles, pedestrians, etc. that exist within a predetermined range based on the vehicle C1. The number of traffic participants around the vehicle C1 can be acquired based on an image of the surroundings of the vehicle C1 (e.g., an image in front of the vehicle C1) acquired by the image acquisition unit 101 and various sensors (e.g., LiDAR, RADAR, Sonar) included in the sensors 104. The number of traffic participants around the vehicle C1 may be estimated by combining these.

[0071] For example, it is assumed that the driving load of the user U1 increases as the number of traffic participants around the vehicle C1 increases. Therefore, it is possible to calculate a detection value that increases as the number of traffic participants around the vehicle C1 increases.

[0072] The weather around the vehicle C1 and the darkness around the vehicle C1 can be acquired based on the image of the vehicle C1 acquired by the image acquisition unit 101, weather information that can be acquired via the communication unit 140, a raindrop sensor, and the like. For example, the weather around the vehicle C1 (for example, sunny, cloudy, rain, snow, strong wind) and the brightness around the vehicle C1 can be determined based on the image of the vehicle C1 acquired by the image acquisition unit 101. For example, the weather and brightness can be determined using a known image recognition technology. In addition, if weather information can be acquired from an external device (for example, a weather information providing server) via the communication unit 140, the weather around the vehicle C1 can be acquired based on the weather information and the current location of the vehicle C1 acquired by the position information acquisition unit 102. In this case, the darkness around the vehicle C1 can be estimated based on the weather around the vehicle C1 and the current time. In addition, if a raindrop sensor is installed in the vehicle C1, the raindrop sensor can detect that there is rain at the current location of the vehicle C1. Moreover, these may be combined to estimate the weather and darkness around the vehicle C1.

[0073] For example, if the weather around the vehicle C1 worsens or it becomes dark, the driving load on the user U1 is expected to increase depending on the degree of the weather. Therefore, it is possible to calculate a detection value that increases depending on the degree of the weather around the vehicle C1 or the degree of darkness around the vehicle C1.

[0074] The road shape around the vehicle C1 can be acquired based on the image of the surroundings of the vehicle C1 acquired by the image acquisition unit 101, map information (acquired via the communication unit 140 or stored in the storage unit 130), an acceleration sensor, and the like. For example, the narrowness of the road ahead of the vehicle C1, the bad road condition, and the like can be determined based on the image of the surroundings of the vehicle C1 acquired by the image acquisition unit 101. For example, the road shape can be determined using a known image recognition technique. In addition, when the map information DB 134 can be used, information on the road at the current location of the vehicle C1 (narrowness, traffic volume, degree of congestion) can be acquired based on the map information DB 134 and the current location of the vehicle C1 acquired by the position information acquisition unit 102. In addition, the bad road condition, and the like of the road around the vehicle C1 can be determined by acquiring the actual shaking, and the like of the vehicle C1 using various sensors (for example, acceleration sensors) installed in the vehicle C1.

[0075] For example, if the roads around the vehicle C1 become narrower or the roads around the vehicle C1 become rough, the driving load on the user U1 is expected to increase depending on the degree of narrowness or roughness of the roads around the vehicle C1. Therefore, it is possible to calculate a detection value that increases depending on the degree of narrowness or roughness of the roads around the vehicle C1.

[0076] The driving load of the user U1 can be obtained by using each of the above-mentioned detection values. For example, a predetermined calculation (e.g., addition) is performed on each of the above-mentioned detection values, and the driving load of the user U1 can be obtained by using the calculation result. Note that the driving load of the user U1 may be obtained by using at least one of the above-mentioned detection values.

[0077] [Example of detecting user alertness] Next, a method for detecting the awakening level of the user U1 will be described. The awakening level of the user U1 can be detected based on the state of the user U1, the voice uttered by the user U1, and the like.

[0078] As the state of the user U1, for example, the facial expression of the user U1, the frequency of blinking of the user U1, the duration of blinking of the user U1, whether the neck of the user U1 is tilted forward, the angle of the neck of the user U1, etc. These pieces of information can be acquired by a known image recognition technology based on the face of the user U1 and its surroundings included in the image acquired by the image acquisition unit 101.

[0079] In addition, when the user U1 has a sleepy facial expression, it is assumed that the wakefulness level of the user U1 is lowered accordingly (i.e., the wakefulness level of the user U1 is low). Therefore, it is possible to calculate a detection value that is lowered as the user U1 has a sleepy facial expression.

[0080] In addition, when the blink frequency of the user U1 tends to increase, it is assumed that the wakefulness level of the user U1 decreases according to the degree of the increase. Therefore, it is possible to calculate a detection value that decreases according to the blink frequency of the user U1 tends to increase.

[0081] In addition, when the blink duration of the user U1 becomes longer, it is assumed that the wakefulness level of the user U1 decreases accordingly. Therefore, it is possible to calculate a detection value that decreases as the blink duration of the user U1 becomes longer.

[0082] In addition, when the user U1 tilts his / her neck forward, it is assumed that the wakefulness of the user U1 decreases according to the degree (angle) of the tilt. Therefore, it is possible to calculate a detection value that decreases according to the angle at which the user U1 tilts his / her neck forward.

[0083] The voice uttered by the user U1 may be, for example, the volume of the voice uttered by the user U1, the content of the voice uttered by the user U1, the frequency at which the user U1 utters the voice, the time at which the user U1 utters the voice, etc. These pieces of information may be acquired by a known voice recognition technology based on the voice of the user U1 contained in the sound information acquired by the sound input unit 103.

[0084] In addition, when the volume of the voice of the user U1 becomes smaller, it is assumed that the awakening level of the user U1 becomes lower according to the degree of the decrease. Therefore, it is possible to calculate a detection value that becomes lower according to the decrease in the volume of the voice of the user U1.

[0085] In addition, when the voice of the user U1 becomes incomprehensible, it is assumed that the awakening level of the user U1 decreases according to the degree of incomprehensibility. Therefore, it is possible to calculate a detection value that decreases according to the content of the voice of the user U1 becoming incomprehensible.

[0086] In addition, when the frequency with which the user U1 speaks becomes lower, it is assumed that the awakening level of the user U1 becomes lower accordingly. Therefore, it is possible to calculate a detection value that becomes lower as the frequency with which the user U1 speaks becomes lower.

[0087] In addition, when the time during which the user U1 speaks becomes shorter, it is assumed that the awakening level of the user U1 becomes lower in accordance with the degree of the shortening of the time during which the user U1 speaks. Therefore, it is possible to calculate a detection value that becomes lower as the time during which the user U1 speaks becomes shorter.

[0088] The wakefulness of the user U1 can be obtained by using each of the above-mentioned detection values. For example, a predetermined calculation (e.g., addition) can be performed on each of the above-mentioned detection values, and the wakefulness of the user U1 can be obtained by using the calculation result. Note that the wakefulness of the user U1 may be obtained by using at least one of the above-mentioned detection values.

[0089] [Example of estimating the types of emotions that a user can perceive] Next, a method for estimating the type of emotion that the user U1 can perceive will be described. Specifically, the method estimates which emotion the user U1 can perceive by using the above-mentioned awakening level of the user U1 and the driving load of the user U1.

[0090] For example, it is possible to set in advance a relationship between a plurality of elements related to the state of the user U1 (for example, the awakening level of the user U1 and the driving load of the user U1) and the type of emotion that the user U1 can perceive. For example, an experiment or a simulation is performed on the state of a person and the type of emotion that the person can perceive in that state, and based on the experiment or simulation, it is possible to associate the state of a person with the type of emotion that the person can perceive in that state. That is, it is possible to define in advance to what range of emotions in the two-dimensional coordinate TD1 (see FIG. 4) a person can perceive when he / she is at a certain driving load and a certain awakening level. For example, it is possible to define in advance the relationship between the awakening level of a person, the driving load of a person, and the thick circle CE1 (see FIG. 4). Then, each piece of information is stored in the emotion information DB 133.

[0091] FIG. 4 shows an example of types of emotions that user U1 can perceive, estimated based on multiple elements related to the state of user U1. In FIG. 4, the types of emotions that user U1 can perceive are shown outside the thick circle CE1. In other words, emotions shown within the thick circle CE1 are types of emotions that user U1 cannot perceive. For example, assume that the circle of emotions is defined as a circle with a radius of 1 in two-dimensional coordinates TD1. In this case, if the value that quantifies the alertness and driving load of user U1 is a radius a, then the thick circle CE1 can be expressed as 0≦a≦1. Also, if the area (x, y) that includes the types of emotions that user U1 can perceive is defined as a radius a. <x 2 +y 2 It is possible to define it as ≦1.

[0092] For example, when the driving load of the user U1 increases, it is estimated that the user U1 is more focused on driving, and the size of the thick circle CE1 increases. On the other hand, when the level of arousal of the user U1 increases, it is estimated that the user U1 is clearly awake, and the size of the thick circle CE1 decreases. Note that in FIG. 4, for ease of explanation, the types of emotions that the user U1 can perceive are shown as thick circles CE1, but are not limited to this. The types of emotions that the user U1 can perceive may be defined by shapes other than circles (e.g., ellipses, winding closed shapes), or may be defined for each individual emotion without being classified by closed shapes.

[0093] Here, the influence of the driving load of the user U1 and the degree of awakening of the user U1 will be described. For example, it is preferable to make the bold circle CE1 different in size and shape when the degree of awakening of the user U1 is low and the driving load of the user U1 is high, and when the degree of awakening of the user U1 is high and the driving load of the user U1 is low.

[0094] 4, emotions within the thick circle CE1 are presumed to be emotions that the user U1 cannot perceive, and therefore are not used to express emotions in the agent device 200. On the other hand, emotions outside the thick circle CE1 are presumed to be emotions that the user U1 can perceive, and therefore are used to express emotions in the agent device 200.

[0095] In step S503, the determination unit 122 compares the type of emotion that the user U1 can perceive, estimated in step S502, with the first emotion determined in step S501, and determines whether the user U1 can perceive the expression of the first emotion. Specifically, the determination unit 122 determines whether the first emotion determined in step S501 exists within a thick-lined circle CE1 (see FIG. 4). If the first emotion exists within the thick-lined circle CE1 (including a circular shape), the determination unit 122 determines that the user U1 can perceive the expression of the first emotion. On the other hand, if the first emotion exists outside the thick-lined circle CE1, the determination unit 122 determines that the user U1 cannot perceive the expression of the first emotion. If it is determined that the user U1 can perceive the expression of the first emotion, the process proceeds to step S504. On the other hand, if it is determined that the user U1 cannot perceive the expression of the first emotion, the process proceeds to step S505.

[0096] In step S504, the agent control unit 125 controls the agent device 200 to execute a first performance expressing the first emotion determined in step S501. For example, if frustration is determined as the first emotion, the agent control unit 125 controls the agent device 200 to be in the frustrated emotional state shown in Fig. 2(B).

[0097] In step S505, the extraction unit 123 extracts the second emotion closest to the first emotion determined in step S501 from among the types of emotions that the user U1 can perceive, estimated in step S502. FIG. 5 shows an example in which the closeness of distance in the two-dimensional coordinate TD1 (see FIG. 4) is used as the similarity of emotions. That is, the emotion closest to the first emotion in the two-dimensional coordinate TD1 is extracted as the second emotion similar to the first emotion. Specifically, the extraction unit 123 extracts the emotion that is the shortest distance from the first emotion from among the emotions present outside the thick circle CE1 in the two-dimensional coordinate TD1 (see FIG. 4) as the second emotion. Here, a known distance calculation method in the two-dimensional coordinate TD1 can be adopted for calculating the distance. In this case, a known distance such as Euclidean distance or Manhattan distance can be adopted as the distance. In addition, when at least one of the arousal axis and the valence axis is far apart for each coordinate of the first emotion and the candidate second emotion, both the Euclidean distance and the Manhattan distance become large. In this case, the emotion with the smallest difference value on one axis may be selected as the second emotion.

[0098] For example, when frustration AN1 is determined as the first emotion, anger AG1, which has the shortest Euclidean distance from frustration AN1, is extracted from among the emotions present outside the thick circle CE1. In other words, the second emotion can be set as a coordinate that satisfies the following formula 1. Note that argmin means the argument that gives the minimum value. Note that Δx means the difference value between the first emotion and the second emotion on the horizontal axis, and Δy means the difference value between the first emotion and the second emotion on the vertical axis. 2nd emotion=argmin(√(Δx 2 +Δy 2 )) …Formula 1

[0099] In step S506, the agent control unit 125 controls the agent device 200 to execute a second performance expressing the second emotion extracted in step S505. For example, if anger is determined as the second emotion, the agent control unit 125 controls the agent device 200 to enter the angry emotion state shown in Fig. 2(D). An example of this is shown in Fig. 6.

[0100] [Example of agent device operation] Fig. 6 is a simplified diagram showing an example of the operation when agent device 200 executes a second performance expressing a second emotion. Note that the example shown in Fig. 6 is obtained by partially modifying agent device 200 in the configuration example shown in Fig. 1, and the parts other than agent device 200 are common to Fig. 1.

[0101] As shown in Fig. 4 and Fig. 5, if the user U1 is not able to perceive the expression of the first emotion (e.g., irritation), a second emotion (e.g., anger) that is closest to the first emotion and that the user U1 is able to perceive is extracted. In this case, the agent device 200 executes a second performance to express anger towards the user U1. For example, it is possible to express anger towards the user U1 by voice output or movement of the agent device 200. Fig. 6 shows an example of executing a second performance to express anger in response to the speech content S2 of the user U1.

[0102] In this way, in a situation where the user U1 cannot perceive the expression of the first emotion (e.g., irritation), the second emotion (e.g., anger) that is closest to the first emotion and that the user U1 can perceive is searched for and expressed. In other words, when the driving load of the user U1 is high, when the user U1's level of arousal is low, etc., it is unavoidable to temporarily express a tentative emotion (e.g., anger) that is easy to understand, thereby maintaining the continuity of communication and making communication smooth.

[0103] [Second embodiment] In the first embodiment, an example is shown in which a second emotion similar to a first emotion is searched for on a two-axis space of valence and arousal (wheel of emotions), but it is also possible to search for a second emotion using an emotion model other than the wheel of emotions. Therefore, in the second embodiment, an example is shown in which a second emotion similar to a first emotion is searched for on a six-axis space.

[0104] [Example of extracting emotions that users can perceive using Ekman's six emotions] 7 is a diagram showing an example (emotion model table 300) of the relationship between Ekman's six basic emotions and emotional expressions that can be expressed by agent device 200, and a calculation result table 330 used when calculating the similarity of emotions using this. Note that the emotion model table 300 and the calculation result table 330 are stored in emotion information DB 133 of storage unit 130.

[0105] The emotion model table 300 is a table in which the horizontal axis defines the six basic emotions of Ekman in six dimensions, and the vertical axis defines all emotions that the agent device 200 can express. Specifically, anger 301, disgust 302, fear 303, happiness 304, sadness 305, and surprise 306 are defined as the six basic emotions of Ekman. In addition to these six emotions, irritation 317, interest 318, earnestness 319, enjoyment 320, suspicion 321, contempt 322, etc. are defined as all emotions that the agent device 200 can express. Note that these emotions are merely examples, and other emotions may be defined. Also, while FIG. 7 shows an example in which emotions are defined in six dimensions, emotions in three, four, seven or more dimensions may be defined and used.

[0106] In the emotion model table 300, the parameters of each emotion (vertical axis) that the agent device 200 can express are defined using Ekman's six basic emotions. For example, for each emotion (vertical axis) corresponding to anger 301, disgust 302, fear 303, happiness 304, sadness 305, and surprise 306, "1.0" is defined in the corresponding column, and "0" is defined in the other boxes. In addition, for each emotion other than these (irritation 317, interest 318, earnestness 319, enjoyment 320, suspicion 321, and contempt 322 on the vertical axis), a numerical value (0 to 1) indicating the degree of similarity with other emotions is defined. For this numerical value, 1 is defined for the same emotion, and 0 is defined for completely different emotions. In addition, a value of 0 to 1 is set according to the degree of similarity between the emotions.

[0107] For example, irritation 317 is considered to be an emotion relatively distant from each emotion of happiness 304, sadness 305, and surprise 306. Therefore, "0" indicating a low degree of similarity is defined in the columns of happiness 304, sadness 305, and surprise 306 of irritation 317. On the other hand, irritation 317 is considered to be different from each emotion of anger 301, disgust 302, and fear 303, but is not a distant emotion. Therefore, "0.6", "0.3", and "0.1" are defined in the columns of anger 301, disgust 302, and fear 303 of irritation according to the degree of similarity. In this example, irritation 317 indicates that it has the highest degree of similarity with anger 301. Note that each numerical value shown in FIG. 7 is a simplified value for ease of explanation, and each numerical value can be appropriately set based on an experiment, a simulation, or the like.

[0108] In addition, in the emotion model table 300, a threshold value (0 to 1) of the user's arousal level required for each emotion that the agent device 200 can express is set in advance. This threshold value is defined in the required user state 307. For emotions that can be perceived when the driving load of the user U1 is close to the maximum value or when the arousal level of the user U1 is close to the minimum value (for example, when the user is sleeping), a value close to 0 is defined. On the other hand, for emotions that can be perceived when the driving load of the user U1 is close to the minimum value and when the arousal level of the user U1 is close to the maximum value, a value close to 1 is defined. In addition, a value of 0 to 1 is set according to the driving load of the user U1 and the level of the arousal level of the user U1. In other words, as the value approaches 0, it can be considered as an emotion that can be perceived even if the user is in a daze.

[0109] Here, a case will be described in which the detection value of the emotion of the user U1 is 0.3 when frustration 317 (dotted rectangle 341) is determined as the first emotion to be conveyed to the user U1. In this case, the emotion estimation unit 121 estimates, from among the necessary user states 307, emotions that are equal to or less than the detection value of the emotion of the user U1, 0.3, as emotions that the user U1 can perceive. Specifically, six emotions enclosed in a dotted rectangle 342 are estimated as emotions that the user U1 can perceive.

[0110] Next, the determination unit 122 compares the types of emotions (six emotions) that the user U1 can perceive, estimated by the emotion estimation unit 121, with the first emotion (irritation 317) determined in step S501, and determines whether the user U1 can perceive the expression of the first emotion. In the example shown in Fig. 7, since the first emotion (irritation 317) does not exist within the dotted rectangle 342, the determination unit 122 determines that the user U1 cannot perceive the expression of the first emotion.

[0111] Next, the extraction unit 123 extracts a second emotion that is closest to the first emotion determined as the emotion to be conveyed to the user U1 from among the types of emotions that the user U1 can perceive and that are estimated by the emotion estimation unit 121. Fig. 7 shows an example in which the distance in a six-dimensional space is calculated between each emotion that the user U1 can perceive and the first emotion, and the second emotion is extracted based on this distance.

[0112] Specifically, as shown by distance 331 in emotion model with respect to the first emotion, the extraction unit 123 calculates the distance in six-dimensional space between each emotion that the user U1 can perceive and the first emotion. For example, the Euclidean distance calculation method can be used as the method for calculating this distance. For example, the distance in six-dimensional space between frustration 317 (first emotion) and anger 311 (emotion that the user U1 can perceive) is set to "0.5099..." (=√{(0.6-1.0) 2 +(0.3-0) 2 +(0.1-0) 2 +(0-0) 2 +(0-0) 2 +(0-0) 2 In this case, the rounded value “0.51” is stored as distance 331 on the emotion model from the first emotion.

[0113] Note that emotion similarity 332 with the first emotion is a value calculated based on distance 331 from the first emotion on the emotion model, and is an index showing the emotion similarity with the first emotion. Specifically, it is calculated using the following formula 2. That is, emotion similarity 332 with the first emotion is a value that increases the closer the distance from the first emotion on the emotion model, and decreases as the value of distance 331 from the first emotion on the emotion model increases. Note that FIG. 7 shows an example in which a value in the range of 0 to 1 is used as emotion similarity 332 with the first emotion. Similarity of emotions = 1 - (distance from the first emotion in the emotion model (331) / √6) ... Equation 2

[0114] Then, the extraction unit 123 extracts, from among the emotions present in the dotted rectangle 342, the emotion with the smallest distance 331 on the emotion model from the first emotion as the second emotion. Specifically, the emotion with the smallest distance 331 on the emotion model from the first emotion (0.51) is extracted as the second emotion (anger 311). Note that the emotion (anger 311) may be extracted as the second emotion using the largest emotion similarity 332 with the first emotion (0.79).

[0115] In this way, each embodiment can be applied not only to a model of two-dimensional coordinates TD1 of emotion value and arousal level, but also to higher dimensional emotion models, and if it is possible to define the emotional distance based on the definition of that emotion, it is possible to extract the second emotion.

[0116] The control process of the agent device 200 in the second embodiment is similar to the example shown in Fig. 5. However, in each process of steps S502, S503, and S505, a six-axis space of Ekman's six emotions is used instead of the two-dimensional coordinate TD1 of the emotion wheel, which is different from Fig. 5.

[0117] [Third embodiment] In the first and second embodiments, an example is shown in which the agent device 200 executes a second performance to express a second emotion similar to the first emotion when the user U1 cannot perceive the expression of the first emotion to be transmitted to the user U1. In this way, by expressing the second emotion instead of the first emotion, it is possible to increase the possibility that the emotion expression of the agent device 200 is perceivable by the user U1. However, there is a risk that expressing the second emotion different from the first emotion may upset the user U1. In addition, it is assumed that the user U1 who sees the expression of the second emotion different from the first emotion may feel a sense of discomfort toward the agent device 200 or feel distrust toward the agent device 200. In these cases, it is assumed that the user U1 may feel uncomfortable. Therefore, in the third embodiment, an example is shown in which the agent device 200 executes a follow-up such as expressing gratitude for expressing an emotion different from his / her true intention. In other words, an example is shown in which the agent device 200 is made to apologize for expressing a second emotion different from the first emotion originally desired to be transmitted to the user U1.

[0118] [Example of apologizing for expressing an emotion different from the first emotion] Fig. 8 is a simplified diagram showing an example of the operation when the agent device 200 executes an apology performance. Note that the example shown in Fig. 8 is obtained by modifying part of the agent device 200 in the configuration example shown in Fig. 1, and the parts other than the agent device 200 are the same as in Fig. 1.

[0119] For example, an apology to the user U1 can be expressed by voice output or movement of the agent device 200. Fig. 8 shows an example of executing an apology performance in which apologetic voice information S11 is output in response to the utterance content S12 of the user U1.

[0120] [Example of operation of information processing device] Fig. 9 is a flow chart showing an example of a control process of the agent device 200 by the information processing device 110. Note that this control process is a partial modification of the control process shown in Fig. 5, and differs from Fig. 5 in that steps S511 and S512 are added, but the rest is common to Fig. 5. Therefore, the processes common to Fig. 5 are given the same reference numerals and their explanations are omitted. This control process will be explained with reference to Figs. 1 to 8 as appropriate.

[0121] In step S511, the agent control unit 125 determines the strength of the regret according to the magnitude of the difference between the first emotion determined in step S501 and the second emotion expressed in step S506. Here, the difference between the first emotion and the second emotion can be calculated based on the distance between the first emotion and the second emotion, the similarity between the first emotion and the second emotion, and the like. The distance between the first emotion and the second emotion can be, for example, the distance in the two-dimensional coordinate TD1 (see FIG. 4) or the distance in the six-dimensional space (see FIG. 7). Also, the strength of the regret is increased according to the magnitude of the difference between the first emotion and the second emotion. For example, the strength of the regret can be determined in proportion to the magnitude of the difference between the first emotion and the second emotion.

[0122] In step S512, the agent control unit 125 causes the agent device 200 to execute an apology performance that expresses an apology according to the strength of the apologetic feeling determined in step S511. For example, the agent device 200 may apologize to the user U1 through various apology performances, such as an apology performance in which the agent device 200 bows its head, an apology performance in which the agent device 200 makes an apologetic face, an apology performance in which the agent device 200 utters apologetic words, etc. Examples of such apology performances are shown in FIG.

[0123] Also, a performance related to the second emotion expressed to the user U1 (the second emotion that caused the apology) may be executed. For example, in a scene in which the agent device 200 is made to express an emotion of irritation, if the agent device 200 is made to express extreme anger, an apology performance may be executed by transitioning from an angry emotion state (see FIG. 2(D)) to an apologizing state (see FIG. 2(F)), and then transitioning to an embarrassed irritated emotion state (see FIG. 2(D)).

[0124] In this way, in a scene in which the agent device 200 is made to express irritation, if the agent device 200 is made to express extreme anger, this may cause discomfort to the user U1. In such a case, therefore, the agent device 200 is made to apologize to the user U1 for the difference between irritation and anger. In this case, the greater the difference (similarity, distance) between irritation and anger, the greater the discomfort caused to the user U1, so the degree of error in expressing the apology is made greater. For example, it is possible to increase the degree of error by making the movement of the agent device 200 larger, changing the words of apology uttered by the agent device 200, or changing the facial expression of the agent device 200.

[0125] [Fourth embodiment] In the first to third embodiments, an example is shown in which it is determined whether the expression of a first emotion to be conveyed to the user U1 is perceptible or imperceptible by the user U1. However, it is also possible to estimate the probability that the user U1 can perceive the expression of the first emotion as 80%, for example. Therefore, in the fourth embodiment, an example is shown in which a second emotion is extracted taking into account the degree of certainty of the conveyance of the emotion.

[0126] [Example of extracting user-perceivable emotions using emotion similarity and transmission confidence] FIG. 10 is a diagram showing emotion model table 300 (see FIG. 7), calculation result table 330 used when using this to find emotion similarity, an example of the relationship between vision and hearing required to perceive each emotion (visual and auditory ease table 350), and calculation result table 360 ​​used when using these to extract a second emotion. Note that emotion model table 300 is the same as that shown in FIG. 7, and therefore is not shown. Also, calculation result table 330 is the same as that shown in FIG. 7, and therefore a detailed description here is omitted. Also, emotion model table 300, calculation result table 330, visual and auditory ease table 350, and calculation result table 360 ​​are stored in emotion information DB 133 of storage unit 130.

[0127] The visual and auditory margin table 350 is a table that defines the visual and auditory margins required to perceive each emotion. Fig. 10 shows an example in which the required visual margin 351 and the required auditory margin 352 are defined as 0 to 1. For example, when the visual margin required to perceive a certain emotion is high (i.e., there is margin), a value close to 1 (maximum value 1) is defined, and when the visual margin required to perceive a certain emotion is low (i.e., there is no margin), a value close to 0 (minimum value 0) is defined.

[0128] In the fourth embodiment, the visual margin of the user U1 and the auditory margin of the user U1 are detected and used. Here, the visual margin of the user U1 is an index indicating how much strain is placed on the visual of the user U1. The auditory margin of the user U1 is an index indicating how much strain is placed on the auditory of the user U1. For example, when the visual load of the user U1 is low (i.e., there is a margin), a value close to 1 (maximum value 1) is detected, and when the visual load of the user U1 is high (i.e., there is no margin), a value close to 0 (minimum value 0) is detected. Note that a value between 0 and 1 is also detected for the auditory margin of the user U1.

[0129] A publicly known margin detection technique can be used to estimate the visual margin of the user U1 and the auditory margin of the user U1. For example, the confidence estimation unit 124 can estimate the visual margin of the user U1 based on the eye movement of the user U1, the viewing angle of the user U1, the driving situation of the user U1, and the like. The eye movement of the user U1 and the viewing angle of the user U1 can be acquired based on the image acquired by the image acquisition unit 101. Also, the driving situation of the user U1 can be acquired in the same manner as in the above-mentioned example.

[0130] For example, as the movement of the user U1's eyes increases, it can be estimated that the load on the user U1's eyes is large. Therefore, as the movement of the user U1's eyes increases, it can be estimated that the visual margin of the user U1 is low. Also, for example, as the viewing angle of the user U1 narrows, it can be estimated that the load on the user U1's eyes is large. Therefore, as the viewing angle of the user U1 narrows, it can be estimated that the visual margin of the user U1 is low. Also, for example, based on the driving situation of the user U1, it can be estimated that the load on the user U1's eyes is large. For example, as the number of other vehicles, pedestrians, etc. around the vehicle C1 increases, it can be estimated that the load on the user U1's eyes is large. Therefore, as the number of other vehicles, pedestrians, etc. around the vehicle C1 increases, it can be estimated that the visual margin of the user U1 is low. Also, for example, as the road on which the vehicle C1 travels is narrow or winding, it can be estimated that the load on the user U1's eyes is large. Therefore, depending on whether the road on which the vehicle C1 is traveling is narrow or winding, it is possible to estimate the visual margin of the user U1 to be low.

[0131] Also, for example, as the sound around the user U1 (for example, siren, music) gets louder, it can be estimated that the load on the user U1's ears is large. Therefore, as the sound around the user U1 gets louder, it is possible to estimate the hearing margin of the user U1 to be low. Also, for example, it is possible to estimate the hearing margin of the user U1 based on the driving situation of the user U1. For example, it can be estimated that the hearing margin will suddenly increase when an ambulance approaches the vehicle C1. Therefore, it is possible to estimate the visual margin of the user U1 to be low when an ambulance approaches the vehicle C1.

[0132] In addition, it is possible to calculate the confidence k of the transmission of each emotion based on the visual margin of the user U1 and the auditory margin of the user U1. This confidence k of the transmission is an index indicating the probability that each emotion can be transmitted to the user U1. Specifically, it is possible to calculate the confidence k of the transmission of each emotion according to the following formula 3 using the visual margin a0 of the user U1, the auditory margin b0 of the user U1, the required visual margin a (351), and the required auditory margin b (352). However, the confidence k of the transmission is a value between 0 and 1 (i.e., the minimum value is 0 and the maximum value is 1). Confidence of transmission k=(a0-a)+(b0-b) …Equation 3

[0133] FIG. 10 shows an example in which "0.4" is detected as the visual margin of user U1, and "0.3" is detected as the auditory margin of user U1. In this case, as shown in the transmission certainty 361, 0.2 (=(0.4-0.3)+(0.3-0.2)) is calculated as the transmission certainty k for anger 311. Also, for example, 0 is calculated as the transmission certainty k for insult 322. In this case, when calculation is performed using formula 3, the result is -0.3 (=(0.4-0.3)+(0.3-0.7)), so the minimum value 0 is set.

[0134] Next, using the similarity with the first emotion and the confidence of emotion transmission, an emotion that is most likely to be transmitted to user U1 is extracted. Specifically, the confidence estimation unit 124 obtains a calculation result (calculation result 362 of emotion similarity and confidence of emotion transmission) by multiplying the emotion similarity 332(j) with the first emotion by the confidence of emotion transmission 361(k). For example, for anger 311, "0.16" is calculated. In this case, the result is 0.158 (=0.79×0.2), so "0.16" is set by rounding off the third decimal place.

[0135] Next, the emotion with the highest calculation result (calculation result 362 of emotion similarity and transmission certainty) obtained by multiplying emotion similarity 332(j) with the first emotion by transmission certainty 361(k) is extracted as the second emotion. However, the highest emotion is extracted from among the emotions that user U1 can perceive. For example, assume that irritation 317 (dotted rectangle 371) is determined as the first emotion to be transmitted to user U1, and the detection value of the state of user U1 is 0.3. In this case, the emotion with the highest calculation result 362 of emotion similarity and transmission certainty (fear 313) is extracted (shown by dotted rectangle 373) from among the six emotions shown enclosed in dotted rectangle 372. That is, the second emotion is extracted using the following formula 4. Note that argmax f(X) means the argument that gives the maximum value. Also, X means the type of emotion, j(X) means emotion similarity, and k(X) means transmission certainty. Second emotion = argmax f(X) = argmax(j(X) · k(X)) ...Formula 4

[0136] [Example of operation of information processing device] Fig. 11 is a flow chart showing an example of a control process of the agent device 200 by the information processing device 110. Note that this control process is a partial modification of the control process shown in Fig. 5, and differs from Fig. 5 in that step S511 is executed instead of step S505, but the rest is common to Fig. 5. Therefore, the processes common to Fig. 5 are given the same reference numerals and their explanations are omitted. Also, this control process will be explained with appropriate reference to Figs. 1 to 10.

[0137] In step S511, the extraction unit 123 extracts the second emotion from among the emotions that the user U1 can perceive, based on the emotion similarity and the emotion transmission confidence. Specifically, the confidence estimation unit 124 estimates the confidence k of the transmission of each emotion (the emotion transmission confidence 361) based on the user U1's visual margin and the user U1's auditory margin. Then, the extraction unit 123 obtains a calculation result of the emotion similarity (emotion similarity with the first emotion 332) and the emotion transmission confidence k (the emotion transmission confidence 361) from among the emotions that the user U1 can perceive. Then, the extraction unit 123 extracts the emotion with the highest calculation result 362 of the emotion similarity and the emotion transmission confidence as the second emotion from among the emotions that the user U1 can perceive.

[0138] In this way, the confidence in the transmission of an emotion is calculated using the visual and auditory composures of user U1, and the second emotion is extracted using the confidence in the transmission of an emotion. This makes it possible to extract a more appropriate second emotion that takes into account factors such as the probability that the emotion can be transmitted.

[0139] [Fifth embodiment] In the first to fourth embodiments, an example is shown in which a second emotion similar to the first emotion is extracted and expressed at the timing when the first emotion to be conveyed to the user U1 is determined. Here, it is assumed that a relatively long time is set as the timing for expressing the first emotion to be conveyed to the user U1. In such a case, it is also conceivable that the second emotion similar to the first emotion is expressed at any timing during the period during which the first emotion is expressed. Therefore, in the fifth embodiment, an example is shown in which the second emotion is extracted taking into consideration the period during which the first emotion is expressed. That is, in a situation in which emotions are to be conveyed within a certain predetermined time, emotions that can be perceived during the predetermined time are continuously estimated, a second emotion that is closest to the first emotion during the predetermined time is selected, and the agent device 200 expresses the second emotion at the timing when the second emotion is perceived.

[0140] [Example of extracting the second emotion at a given time when the first emotion is expressed] Fig. 12 is a diagram showing an example of an extraction process for extracting a second emotion similar to a first emotion during a predetermined period P1 during which the first emotion is transmitted. Fig. 12 shows an example of extracting the second emotion using emotion circles EC1 to EC4. Note that emotion circles EC1 to EC4 are the same as the example shown in Fig. 4. The extraction process shown in Fig. 12 will be described in detail with reference to Fig. 13.

[0141] [Example of operation of information processing device] Fig. 13 is a flow chart showing an example of a control process of the agent device 200 by the information processing device 110. Note that this control process is a partial modification of the control process shown in Fig. 5, and differs from Fig. 5 in that steps S521 to S525 are executed instead of steps S502, S503, S505, and S506, but the rest is common to Fig. 5. Therefore, the processes common to Fig. 5 are given the same reference numerals and their explanations are omitted. Also, this control process will be explained with reference to Figs. 1 to 12 as appropriate.

[0142] In step S521, the agent control unit 125 determines a time t1 for transmitting the first emotion determined in step S501 to the user U1. This time t1 is a time indicating by when the first emotion should be transmitted to the user U1. For example, the time t1 can be determined based on the type of the first emotion determined in step S501 and the state of communication between the agent device 200 and the user U1. For example, in the case of an emotion such as anger, a relatively short time is set. On the other hand, in the case of an emotion such as irritation or agitation, a relatively long time is set. Also, for example, when the state of communication between the agent device 200 and the user U1 is smooth, it is considered that the expression of the agent device 200 is easily conveyed to the user U1, so a relatively short time is set. On the other hand, when the state of communication between the agent device 200 and the user U1 is not so smooth, it is considered that the expression of the agent device 200 is not easily conveyed to the user U1, so a relatively long time is set. Note that these methods of determining the time t1 are merely examples, and the time t1 may be determined by other methods.

[0143] In step S522, the feeling estimation unit 121 estimates the type of feeling that the user U1 can perceive based on at least one of a plurality of elements related to the state of the user U1 during a period P1 from time (current time) t0 to time t1. The current estimation method is the same as that in FIG. 5.

[0144] Here, a method for estimating the future will be described. Here, an example is shown in which the wakefulness of user U1 and the driving load of user U1 are used as multiple elements related to the state of user U1. For example, it is assumed that the wakefulness of user U1 does not change relatively much during period P1. However, if user U1 takes the initiative in performing some action (e.g., music operation, air conditioner operation), it is possible to increase the wakefulness of user U1 thereafter.

[0145] The driving load of the user U1 can be estimated based on the driving situation, the surrounding situation, and the like. For example, when an intersection exists on the road on which the vehicle C1 is traveling, the driving load of the user U1 increases near the intersection. When the road narrows suddenly on the road on which the vehicle C1 is traveling, the driving load of the user U1 increases near the narrowing point. Note that each of these road conditions can be determined based on the state ahead of the vehicle C1 included in the image included in the image acquisition unit 101 (see FIG. 3) provided in front of the vehicle C1. Similarly, the driving load can be determined based on the state ahead of the vehicle C1 that can be acquired based on the vehicle C1 acquired by the position information acquisition unit 102 and the map information DB 134 (see FIG. 3).

[0146] Furthermore, when there are many other vehicles and pedestrians around the vehicle C1, the driving load of the user U1 increases in the vicinity of that location. The surrounding conditions can be determined based on the surrounding conditions of the vehicle C1 contained in the image included in the image acquisition unit 101 (see FIG. 3) provided around the vehicle C1. Similarly, the determination can be made by acquiring traffic congestion information from another device. For example, based on the expected passing time and position of the vehicle C1, traffic congestion information for the road corresponding to that time and position can be acquired from another device via a predetermined network, for example, the Internet.

[0147] Furthermore, when the weather around the vehicle C1 worsens, the driving load of the user U1 increases in the vicinity of that location. The surrounding weather can be determined based on the situation around the vehicle C1 contained in the image included in the image acquisition unit 101 (see FIG. 3) provided around the vehicle C1. Similarly, the weather can be determined by acquiring weather information from another device. For example, based on the predicted passing time and position of the vehicle C1, weather information for the road corresponding to that time and position can be acquired from another device via a predetermined network, for example, the Internet.

[0148] 12 shows bold circles CE11 to CE14 corresponding to the types of emotions that the user U1 can perceive, which were estimated in step S522. For ease of explanation, only the estimation results at the timings of t0, ta, tb, and t1 in the period P1 are shown in FIG. 12, but estimation may be performed similarly at other timings in the period P1.

[0149] In step S523, the determination unit 122 compares the type of emotion that the current user U1 can perceive, estimated in step S522, with the first emotion determined in step S501, and determines whether the current user U1 can perceive the expression of the first emotion. Specifically, the determination unit 122 determines whether the first emotion determined in step S501 exists within the thick circle CE11 (see FIG. 12). Then, if the first emotion exists within the thick circle CE11 (including a circular shape), the determination unit 122 determines that the current user U1 can perceive the expression of the first emotion. On the other hand, if the first emotion exists outside the thick circle CE11, the determination unit 122 determines that the current user U1 cannot perceive the expression of the first emotion. If it is determined that the current user U1 can perceive the expression of the first emotion, the process proceeds to step S504. On the other hand, if it is determined that the current user U1 cannot perceive the expression of the first emotion, the process proceeds to step S524. In this example, whether or not the current user U1 can perceive the expression of the first emotion is determined, but whether or not the user U1 can perceive the expression of the first emotion may be determined at any timing during period P1. In this case, in step S504, the agent device 200 may be caused to express the first emotion at that timing, or the agent device 200 may be caused to continue expressing the first emotion during period P1.

[0150] In step S524, the extraction unit 123 extracts a second emotion that is closest to the first emotion determined in step S501 and a time t2 from among the types of emotions that the user U1 can perceive in the period P1 estimated in step S522. Specifically, the extraction unit 123 extracts, as the second emotion, the emotion that is the closest to the first emotion among the emotions that exist outside the thick circles CE11 to CE14 on the circle of emotions EC1 to EC4 (see FIG. 12). The extraction method is the same as the extraction method in step S505 (see FIG. 5).

[0151] In FIG. 12, it is assumed that frustration AN1 is determined as the first emotion to be conveyed to user U1. In this case, frustration AN1 does not exist within the thick circles CE11 to CE14 at the timings of t0, ta, tb, and t1. Therefore, the extraction unit 123 extracts anger AG1 as the emotion that is the shortest distance from frustration AN1 among the emotions that exist outside the thick circles CE11 to CE14 in the circle of emotions EC1 to EC4, and extracts the timing (time tb (= t2)). Note that it is also possible to sequentially select emotions that are the shortest distance from frustration AN1 at each of the timings of times t0, ta, tb, and t1, and finally select the emotion that is closest to frustration AN1 as the second emotion from each of the selected emotions.

[0152] In step S525, the agent control unit 125 controls the agent device 200 to execute a second performance expressing the second emotion at the timing of time t2 extracted in step S524. For example, if anger AG1 is extracted as the second emotion at the timing of time tb (= t2), the agent control unit 125 controls the agent device 200 to enter the angry emotion state shown in Fig. 2(D) at the timing of time tb (= t2).

[0153] In the example shown in FIG. 13, the second emotion is expressed only at time t2 during period P1. However, it is assumed that user U1 may be in a state in which he or she can perceive the first emotion during period P1 for some reason. Therefore, the first emotion may be expressed at times other than time t2, and the second emotion may be expressed at time t2. In this case, when it is determined that the first emotion can be perceived up to time t2, the second emotion is not expressed, and only the first emotion is expressed.

[0154] In this way, in a situation where the current user U1 cannot perceive the expression of the first emotion (e.g., irritation), a second emotion (e.g., anger) that is closest to the first emotion and that the user U1 can perceive is searched for and expressed during the period P1 in which the first emotion should be conveyed. That is, when the driving load of the user U1 is high, when the user U1's level of arousal is low, etc., it is unavoidable to temporarily express a tentative emotion (e.g., anger) that is easy to understand during the period P1, thereby maintaining the continuity of communication and making communication smooth.

[0155] [Sixth embodiment] In the fifth embodiment, an example is shown in which a second emotion similar to a first emotion is extracted using an emotion circle during a predetermined period P1 during which a first emotion is transmitted. However, it is also possible to extract a second emotion using an emotion model other than an emotion circle. Therefore, in the sixth embodiment, an example is shown in which a second emotion is searched for in a six-axis space during a predetermined period P1.

[0156] [Example of extracting secondary emotions using emotion similarity and transmission confidence] Fig. 14 is a diagram showing an example of an extraction process for extracting a second emotion similar to a first emotion during a predetermined period P11 during which the first emotion is transmitted. Fig. 14 shows an example of extracting the second emotion using the calculation results of the similarity j with the first emotion and the confidence k of transmission. The extraction process shown in Fig. 14 will be described in detail with reference to Fig. 15.

[0157] [Example of operation of information processing device] Fig. 15 is a flow chart showing an example of a control process of the agent device 200 by the information processing device 110. This control process is a partial modification of the control process shown in Fig. 13, and differs from Fig. 13 in that step S531 is executed instead of step S524, "t1" in steps S521 and S522 is changed to "t11", and "t2" in step S525 is changed to "t12", but the rest is the same as Fig. 13. Therefore, the processes common to Fig. 13 are denoted by the same reference numerals and their explanations are omitted. This control process will be explained with reference to Figs. 1 to 14 as appropriate.

[0158] FIG. 15 shows an example of extracting a second emotion using emotion model table 300 (see FIG. 7), calculation result table 330 (see FIG. 10), visual and auditory leisure level table 350 (see FIG. 10), and calculation result table 360 ​​(see FIG. 10).

[0159] Here, a method of estimating the future visual margin of the user U1 and the future auditory margin of the user U1 used in calculating the transmission certainty factor k will be described. These can be estimated based on the driving situation of the user U1, the surrounding situation, etc. The method of estimating the future driving situation of the user U1 and the surrounding situation is the same as the above example.

[0160] In step S522, the determination unit 122 estimates, from among the necessary user states 307 (see FIG. 7), emotions that are equal to or less than the detection value of the detected state of the current user U1 as emotions that the user U1 can perceive. Similarly, the estimation method for the period up to time t11 can also be performed based on the necessary user states 307 and the future states of the user U1.

[0161] In step S523, the determination unit 122 compares the type of emotion that the current user U1 can perceive, estimated in step S522, with the first emotion determined in step S501, and determines whether the current user U1 can perceive the expression of the first emotion. Specifically, the determination unit 122 determines whether the necessary user state 307 corresponding to the first emotion determined in step S501 is equal to or lower than the detection value of the state of the current user U1. Then, when the necessary user state 307 corresponding to the first emotion is equal to or lower than the detection value of the state of the current user U1, the determination unit 122 determines that the current user U1 can perceive the expression of the first emotion. On the other hand, when the necessary user state 307 corresponding to the first emotion exceeds the detection value of the state of the current user U1, the determination unit 122 determines that the current user U1 cannot perceive the expression of the first emotion. When it is determined that the current user U1 can perceive the expression of the first emotion, the process proceeds to step S504. On the other hand, if it is determined that the current user U1 is unable to perceive the expression of the first emotion, the process proceeds to step S531.

[0162] In step S531, the extraction unit 123 extracts a second emotion similar to the first emotion determined in step S501 and a time t12 from among the types of emotions perceivable by the user U1 in the period P11 estimated in step S522. In this case, the second emotion and the time t12 are extracted using the calculation results of the similarity j to the first emotion determined in step S501 and the confidence k of transmission.

[0163] Specifically, the certainty estimation unit 124 calculates the certainty k of transmission (certainty 361 of transmission (see FIG. 10)) for the period P11 using the visual margin of user U1 estimated for the period P11 and the auditory margin of user U1. The calculation method is similar to the example shown in FIG. 10. Then, the extraction unit 123 executes a predetermined calculation (e.g., multiplication) on the similarity j with the first emotion determined in step S501 and the certainty k of transmission to obtain the calculation result. An example of the calculation result obtained in this manner is shown in FIG. 14.

[0164] FIG. 14 shows simplified calculation results obtained for irritation X1 (=j(X1)·k(X1,t)), seriousness X2 (=j(X2)·k(X2,t)), anger X3 (=j(X3)·k(X3,t)), and doubt X4 (=j(X4)·k(X4,t)). In this manner, the calculation results of the similarity j with the first emotion and the confidence k of the transmission in the period P11 are obtained. Note that although irritation X1 is the first emotion, it is illustrated for comparison. In this case, the extraction unit 123 extracts anger X3, which has the largest value, from the calculation results of seriousness X2, anger X3, and doubt X4 in the period P11, and extracts the timing (time t12). That is, the second emotion is extracted using the above-mentioned formula 4.

[0165] In this way, in a situation where the current user U1 cannot perceive the expression of the first emotion (e.g., irritation), a second emotion (e.g., anger) that is closest to the first emotion and that the user U1 can perceive is searched for and expressed during the period P11 in which the first emotion should be transmitted. That is, the confidence level of emotion transmission is calculated using the visual and auditory margins of the user U1 during the period P11, and the second emotion is extracted using the confidence levels of emotion transmission, making it possible to extract a more appropriate second emotion that takes into account the probability of emotion transmission during the period P11, etc.

[0166] [Seventh embodiment] In the first to sixth embodiments, an example was shown in which emotions perceivable by the user U1 are estimated, and a second emotion similar to the first emotion to be conveyed is searched for from among these emotions. Here, each part of the agent device 200, a voice emitted by the agent device 200, and the like are assumed as expression means (transmission means) used by the agent device 200 to convey some information to the user U1. Furthermore, the emotional expression of the agent device 200 often changes according to the emotion to be conveyed to the user U1. For example, an emotional expression of irritation may be expressed by a minute change in the facial expression of the face 210, a small movement of the body 220, and the like. On the other hand, the more complex the emotion, the more parts of the face 210 and the body 220 may be used. For example, an emotional expression of anger may be expressed by a facial expression of the face 210, large movements of the limbs of the body 220, a loud voice, and the like.

[0167] For this reason, for example, in the case of the agent device 200 having the face 210 and the body 220, depending on the state and type of emotion of the user U, a situation is assumed in which the facial expression of the face 210 can be seen but the body 220 cannot be seen. On the other hand, depending on the state and type of emotion of the user U, a situation is assumed in which the facial expression of the face 210 cannot be seen but the body 220 can be seen. Also, depending on the state and type of emotion of the user U, a situation is assumed in which the facial expression of the face 210 and the body 220 cannot be seen but the voice emitted by the agent device 200 can be heard. On the other hand, depending on the state and type of emotion of the user U, a situation is assumed in which the voice emitted by the agent device 200 cannot be heard but the facial expression of the face 210 and the body 220 can be seen. For example, when the user U1 has a good hearing, emotions that can be understood only from the voice are easily perceived. On the other hand, even if the user U1 has a good hearing, irritation is not easily perceived.

[0168] In this way, depending on the type of emotion, a part or voice that can be perceived by the user U1 may be used, and a part or voice that is difficult for the user U1 to perceive may be used. In other words, it is assumed that the probability of conveying the emotion to the user U1 varies depending on the state of the user U1 and the expression of the emotion to be conveyed to the user U1. Therefore, in the seventh embodiment, an example is shown in which the second emotion is extracted for the expression means of the agent device 200, taking into consideration the probability of conveying the emotion to the user U1.

[0169] [Example of estimating the confidence of primary information transmission] First, the components of the primary information of the emotional expression that are used as a reference when estimating the degree of certainty of the transmission of the primary information will be described. Here, the degree of certainty of the transmission of the primary information means the probability that the expression means of the agent device 200 can transmit the emotional expression by the agent device 200 to the user U1, etc. In other words, it means the probability that the agent device 200 can transmit the emotional expression to the user U1 using its own expression means, etc.

[0170] First, the components of the primary information of each emotional expression used when estimating the degree of certainty of the transmission of primary information for two types of emotions, irritation and anger, will be described. For example, components of the primary information of an emotional expression may include components related to the agent device 200 itself and components related to the surrounding environment of the agent device 200.

[0171] Concerning the components of the primary information of the emotional expression of irritation, the components related to the agent device 200 itself include body movement (small) (i1), eyebrow tilt (i2), mouth tilt (i3), speed of movement (large) (i4), and duration of movement (large) (i5). Also, the components related to the surrounding environment of the agent device 200 (obstructing factors) include surrounding noise (h1), brightness (h2), and temperature (h3).

[0172] Concerning the components of the primary information of the emotional expression of anger, the components related to the agent device 200 itself include body movement (big) (i1), inclination of eyebrows (i2), inclination of mouth (i3), speed of movement (big) (i4), duration of movement (small) (i5), tone of voice (strong) (i6), volume of voice (loud) (i7), movement of hitting something (i8), and volume of sound of hitting something (i9). Concerning the components (obstructive factors) related to the surrounding environment of the agent device 200, the surrounding noise (h1), brightness (h2), and temperature (h3) are considered, as in the case of the emotional expression of irritation. The calculation method of i1 to i9 and h1 to h3 will be described with reference to FIG. 16.

[0173] [Example of calculating the confidence level of primary information transmission] FIG. 16 is a diagram showing emotion model table 300 (see FIG. 7), calculation result table 330 (see FIG. 10), an example of the relationship between the visual, auditory, and inhibition indices (primary information tables 410, 420) required when calculating the confidence level of transmission of primary information, and calculation result table 360 ​​(see FIG. 10). Note that emotion model table 300 is the same as that shown in FIG. 7, and therefore is not shown. Also, calculation result table 330 and calculation result table 360 ​​are the same as those shown in FIG. 10, and therefore are not shown. Also, each of these tables is stored in emotion information DB 133 of storage unit 130.

[0174] The primary information table 410 is a table that defines the visual and auditory margins and the inhibition index required when calculating the confidence level of the transmission of primary information about the emotional expression of frustration. Fig. 16(A) shows an example in which the required visual margin 411 and the required auditory margin 412 are defined as 0 to 1. The definition of 0 to 1 is the same as the example shown in Fig. 10.

[0175] The primary information table 420 is a table that defines the visual and auditory margins and the inhibition index required when calculating the confidence of the transmission of primary information about the emotional expression of anger. Fig. 16(B) shows an example in which the required visual margin 421 and the required auditory margin 422 are defined as 0 to 1. The definition of 0 to 1 is the same as the example shown in Fig. 10.

[0176] The detection method for the visual margin of the user U1 and the auditory margin of the user U1 is the same as the detection method described above. The surrounding noise (inhibition index 414) can be acquired based on sound information acquired by the sound input unit 103 provided outside the vehicle C1. The brightness (inhibition index 414) can be acquired based on brightness information acquired by the sensors 104 (e.g., an illuminance sensor). The temperature (inhibition index 414) can be acquired based on temperature information acquired by the sensors 104 (e.g., a temperature sensor).

[0177] Here, in the seventh embodiment, unlike the fourth embodiment, the confidence level of emotion transmission is not calculated based only on the visual and auditory margins, but is calculated as the total value of the confidence level of the transmission of each piece of primary information required for each emotion expression. In other words, in the seventh embodiment, the confidence level of the transmission of primary information is calculated taking into consideration the parts used as the expression means of the agent device 200, sounds, and impeding factors other than the expression means of the agent device 200. Specifically, using the visual margin a0 of the user U1, the auditory margin b0 of the user U1, the required visual margin a (411, 421), and the required auditory margin b (412, 422), it is possible to calculate the confidence level k1 (413, 423) of the transmission of each piece of primary information according to the following formula 5. However, the confidence level k1 of the transmission of primary information is set to a value between 0 and 1 (i.e., the minimum value is 0 and the maximum value is 1). Confidence of the transmission of primary information k1 = (a0-a) + (b0-b) … Equation 5

[0178] FIG. 16 illustrates an example in which the visual margin of the user U1 is detected as "0.4" and the auditory margin of the user U1 is detected as "0.3," similarly to the example illustrated in FIG.

[0179] That is, the transmission certainty 413, 423 shown in Fig. 16 means an index showing to what extent each component of the agent device 200 can be perceived by the user U1. Therefore, the transmission certainty 413, 423 is calculated for each component of the agent device 200. That is, the transmission certainty of the primary information is calculated. Next, the sum of the transmission certainty 413, 423 and the inhibition index 414, 424 is calculated.

[0180] Specifically, it is possible to calculate the confidence of transmission of the secondary information of the emotional expression of irritation using the confidence of transmission 413 and the inhibition index 414 and the following formula 6. Confidence of secondary information transmission k2=max(i1+i2+i3+i4+i5-h1-h2-h3,100%) …Equation 6

[0181] In addition, it is possible to calculate the confidence k3 of the transmission of the secondary information of the emotional expression of anger by using the confidence 423 of the transmission, the inhibition index 424, and the following formula 7. Confidence of secondary information transmission k3=max(i1+i2+i3+i4+i5+i6+i7+i8+i9-h1-h2-h3,100%) …Equation 7

[0182] The degree of certainty of the transmission of secondary information calculated in this manner represents an index showing to what extent the means of expression of the agent device 200 for each emotion can be perceived.

[0183] In the example shown in Fig. 16(A), the confidence level k2 of the transmission of secondary information of an emotional expression of irritation is calculated as 0.5 using formula 6. Also, in the example shown in Fig. 16(B), the confidence level k3 of the transmission of secondary information of an emotional expression of anger is calculated as 0.4 using formula 7. In this way, based on the state of user U1, the surrounding circumstances, etc., it is possible to estimate the extent to which user U1 can perceive agent device 200 using the confidence level of the transmission of primary information and the confidence level of the transmission of secondary information. This makes it possible to improve the accuracy of estimating the confidence level of transmission.

[0184] Furthermore, similar to the example shown in FIG. 10, if the first emotion is irritation and the state of user U1 is 0.3, the emotion similarity 332 (see FIG. 10) between irritation and the first emotion is 1.00, and the emotion similarity 332 between anger and the first emotion is 0.79.

[0185] In this case, as the calculation result 362 (see FIG. 10) of the emotion similarity and the transmission certainty, 0.5 (1.00 (emotion similarity 332 with the first emotion)×0.5 (secondary information transmission certainty k2)) is calculated for the emotion expression of irritation. Also, 0.316 (0.79 (emotion similarity 332 with the first emotion)×0.4 (secondary information transmission certainty k3)) is calculated for the emotion expression of anger. The method of extracting the second emotion using these is the same as the example shown in FIG. 10.

[0186] In this way, it is possible to estimate the confidence k1 of the transmission of primary information by taking into consideration the type of primary information that the user U1 can perceive (parts and sounds used as the means of expression of the agent device 200, and obstructive factors). Then, it is possible to estimate the confidence k2, k3 of the transmission of secondary information, which are used when estimating the emotions (secondary information) that the user U1 can perceive, by using the confidence k1 of the transmission of primary information.

[0187] [Example of operation of information processing device] Fig. 17 is a flow chart showing an example of a control process of the agent device 200 by the information processing device 110. Note that this control process is a partial modification of the control process shown in Fig. 5, and differs from Fig. 5 in that steps S541 to S543 are executed instead of step S505, but the rest is common to Fig. 5. Therefore, the processes common to Fig. 5 are given the same reference numerals and their explanations are omitted. Also, this control process will be explained with reference to Figs. 1 to 16 as appropriate.

[0188] In step S541, the confidence estimation unit 124 estimates the confidence of the transmission of primary information for the emotion perceptible by the user U1. For example, when estimating the confidence of the transmission of primary information for the emotion of irritation, the confidence estimation unit 124 uses the visual composure of the user U1, the auditory composure of the user U1, and the primary information table 410 to calculate the confidence k1 of the transmission of primary information using the above-mentioned formula 5.

[0189] In step S542, the confidence estimation unit 124 estimates the confidence of the transmission of the secondary information by using the confidence of the transmission of the primary information estimated in step S541. For example, when estimating the confidence of the transmission of secondary information for the feeling of irritation, the confidence estimation unit 124 calculates the confidence k2 of the transmission of the secondary information by using the confidence of the transmission 413 (see FIG. 16) and the inhibition index 414 (see FIG. 16) and the above-mentioned formula 6.

[0190] In step S543, the extraction unit 123 extracts a second emotion from among the emotions that the user U1 can perceive, based on the similarity of the emotions and the confidence of the transmission of the secondary information. Specifically, the extraction unit 123 obtains a calculation result of the similarity of the emotions (emotion similarity 332 with the first emotion (see FIG. 10)) and the confidence of the transmission of the secondary information (confidence of transmission 361 (see FIG. 10)) from among the emotions that the user U1 can perceive. Then, the extraction unit 123 extracts, from among the emotions that the user U1 can perceive, the emotion with the highest calculation result 362 (see FIG. 10) of the similarity of the emotions and the confidence of the transmission as the second emotion.

[0191] In this way, the certainty of the transmission of primary information is estimated taking into consideration the type of primary information that the user U1 can perceive (parts and sounds used as expression means of the agent device 200, and obstructive factors), and the certainty of the transmission of secondary information is estimated using this certainty of the transmission of primary information. In other words, since the certainty of the transmission is estimated in two stages, the certainty of the transmission of primary information and the certainty of the transmission of secondary information, it is possible to improve the accuracy of extraction of a more appropriate second emotion taking into consideration the probability that an emotion can be transmitted, etc.

[0192] In the above, examples have been shown in which various emotion models are used to perform an estimation process of the type of emotion that the user U1 can perceive, a determination process of whether the user U1 can perceive the expression of the first emotion, an extraction process of the second emotion, and the like. However, these processes may be performed by other methods. For example, each process can be performed using artificial intelligence (AI). For example, it is possible to learn in advance a plurality of elements related to the human state (e.g., the human arousal level and the human driving load) and the type of emotion that the human can perceive, and use this learning data for the estimation process and the determination process. In addition, for example, it is possible to learn in advance a plurality of elements related to the human state (e.g., the human arousal level and the human driving load), the type of the second emotion expressed to the human, and the human's reaction to the second emotion, and use this learning data for the extraction process.

[0193] [Examples of Effects in Each Embodiment] As described above, when the user U1 is driving the vehicle C1, he / she may not be able to see the entire agent device 200. Furthermore, when the driving load of the user U1 is high or the user U1 has a low level of alertness, the user U1 cannot pay attention to the agent device 200. Therefore, even if the agent device 200 expresses an emotion such as irritation with a subtle facial expression, it is expected that the user U1 will have difficulty perceiving the emotion.

[0194] Therefore, in each embodiment, when the driving load of the user U1 is high, when the awakening level of the user U1 is low, etc., the agent device 200 is forced to temporarily express a tentative emotion that is easy to understand, so that mutual communication can be smoothly performed. For example, in a situation where the agent device 200 wants to convey a first emotion, it is possible for the agent device 200 to select a second emotion that is closest to the first emotion from among the emotions that the agent device 200 can express and the emotions that the user U1 can perceive, and express the second emotion. For example, when the first emotion is an irritation emotion expressed with small movements, it is possible to select and express an anger emotion that is closest to irritation as the second emotion. This anger emotion is, for example, expressed with large (exaggerated) movements, and is therefore considered to be easy to understand and to be easily perceived by the user U1. As a result, even when the driving load of the user U1 is high, when the awakening level of the user U1 is low, etc., it is possible to establish information transmission from the agent device 200 to the user U1, and it is possible to increase the continuity of mutual communication. That is, information can be more appropriately conveyed using emotional expressions that the user U1 can perceive, depending on the state of the user U1, etc. Also, it is possible to maintain the quality required to realize an attachment to the agent device 200, and to realize an agent device 200 that gives the impression of intelligence. Also, it is possible to realize an agent device 200 that is responsive to the user U1 and the surrounding environment, and has the independence to appear to maintain its own goals, plans, and intentions.

[0195] [Example for multiple users] In each embodiment, an example in which an exchange takes place between the user U1 and the agent device 200 has been shown, but it is also assumed that an exchange takes place between a plurality of users and the agent device 200. In this case, the states of all users may be detected, and an inference process, a determination process, an extraction process, etc. may be performed based on the states of each of these users. Alternatively, the states of some of the users may be detected, and an inference process, a determination process, an extraction process, etc. may be performed based on the states of the some of the users. For example, in a situation in which emotions are conveyed to a plurality of users at the same time, the agent device 200 can express the second emotion by selecting a second emotion that is closest to the first emotion from among emotions that all of the users can perceive. Also, for example, in a situation in which emotions are conveyed to a plurality of users at the same time, the agent device 200 can express the second emotion by selecting a second emotion that is closest to the first emotion from among emotions that some of the users can perceive.

[0196] [Examples of agent devices that can be installed outside of vehicles] In the above, an example of the agent device 200 installed in the vehicle C1 has been shown. However, each embodiment can also be applied to an agent image displayed on a display device, an agent device removable from the vehicle C1, an agent device that can be installed outside the vehicle, and the like. For example, it is assumed that a user who possesses a portable agent device will install the agent device on the dashboard 2 of the vehicle C1 when getting into the vehicle C1, and will carry the agent device when getting off the vehicle C1. It is also assumed that the user will use the agent device at home. It is also assumed that various devices (for example, a game device, a glasses-type display, AR (Augmented Reality) glasses) capable of displaying an agent image (corresponding to the agent device 200) will be used.

[0197] For example, in a home, an agent device may execute control over each device installed in the home. For example, it may execute control over heating the bath, cooking with cooking utensils, outputting music, opening the front door and windows, and the like. In these cases, it may be expected to provide notification information about the bath, cooking utensils, doors, windows, audio output, and the like. Also, for example, it may be expected that an agent device or an agent image may have various interactions (e.g., conversations, questions) with a user U1 in a home. Each embodiment may be applied when the agent device or agent image expresses emotions at the timing of providing such notification information and at the timing of various interactions.

[0198] The load of the user U1 can be determined based on whether the user U1 is moving, whether the user U1 is working (for example, whether the user U1 is operating an appliance, whether the user U1 is cooking), and the surrounding conditions of the user U1. For example, when the user U1 is moving, it is possible to determine that the load of the user U1 is high, and when the user U1 is stationary, it is possible to determine that the load of the user U1 is low. The movement of the user U1 can be acquired based on an acceleration sensor of an electronic device (for example, a smartphone, a wearable device) that the user U1 possesses. When the user U1 is operating an electronic device (for example, a smartphone, a game device), it is possible to determine that the load of the user U1 is high, and when the user U1 is not operating an electronic device, it is possible to determine that the load of the user U1 is low. Incidentally, whether the electronic device is operated can be determined based on control information that can be acquired from the electronic device. When the user U1 is talking to another person, it is possible to determine that the load of the user U1 is high, and when the user U1 is not talking to another person, it is possible to determine that the load of the user U1 is low. The presence or absence of conversation of the user U1 can be determined based on voice information acquired by a voice input device. The load of the user can also be determined based on the facial expression, speed of movement, etc. of the user U1. The awakening level of the user U1 can be detected in the same manner as in the embodiments.

[0199] [Example of executing processing on other devices or systems] In the above, examples have been shown in which the control processes and the like are executed in the agent device 200 and the information processing device 110 (or the information processing system 100), but all or part of each of these processes may be executed in other devices. In this case, the information processing system is made up of each device that executes part of each of these processes. For example, at least part of each process can be executed using various information processing devices and various electronic devices, such as in-vehicle devices, devices that can be used by users (e.g., smartphones, tablet terminals, personal computers, car navigation devices, IVI), and servers that can be connected via a predetermined network such as the Internet.

[0200] In addition, a part (or the whole) of the information processing system capable of executing the functions of the information processing device 110 (or the information processing system 100) may be provided by an application that can be provided via a predetermined network such as the Internet. This application is, for example, SaaS (Software as a Service).

[0201] [Configuration Example of This Embodiment and Its Effects] The information processing method according to this embodiment is an information processing method for controlling an agent device 200 capable of communicating with a user U1. This information processing method includes an estimation process (steps S502, S522) for estimating the type of emotion that the user U1 can perceive based on at least one of a plurality of elements related to the state of the user U1, a determination process (steps S503, S523) for comparing the type of the estimated emotion with a first emotion that has already been determined to be conveyed to the user U1 and judging whether the user U1 can perceive the expression of the first emotion, and a determination process (steps S504, S525) for, when it is determined that the user U1 cannot perceive the expression of the first emotion, transmitting the estimated type of emotion. The program includes an extraction process (steps S505, S511, S524, S531, S543) for extracting a second emotion similar to the first emotion from the first emotion, and a control process (steps S504, S506, S511, S512, S525) for causing the agent device 200 to execute a first performance expressing the first emotion when it is determined that the user U1 can perceive the expression of the first emotion, and causing the agent device 200 to execute a second performance expressing the second emotion when it is determined that the user U1 cannot perceive the expression of the first emotion. The program according to this embodiment is a program for causing a computer to execute each of these processes. In other words, the program according to this embodiment is a program for causing a computer to realize each function executable by the information processing device 110.

[0202] According to this configuration, even when the driving load of the user U1 is high or the awakening level of the user U1 is low, it is possible to establish information transmission from the agent device 200 to the user U1, and it is possible to increase the continuity of mutual communication. That is, it is possible to more appropriately transmit information using emotional expressions that the user U1 can perceive, depending on the state of the user U1, etc.

[0203] In the information processing method according to this embodiment, the extraction process (step S505) extracts, from among the types of estimated emotions, an emotion that is closest to the first emotion as the second emotion on two-dimensional coordinates TD1 defined by arousal level indicating a person's arousal state as a numerical value and valence indicating a person's emotion as a numerical value.

[0204] According to this configuration, the emotion that is closest to the first emotion in the two-dimensional coordinate TD1 expressed by two axes can be extracted as the second emotion that is similar to the first emotion, which makes it possible to extract a more appropriate second emotion.

[0205] In the information processing method according to this embodiment, the extraction process (step S505) extracts, in six-dimensional coordinates in which each of the emotions constituting Ekman's six emotions is defined as a parameter value on a coordinate axis, an emotion whose parameter value corresponding to the first emotion is closest to the parameter value corresponding to the six emotions for each estimated emotion type as the second emotion.

[0206] According to this configuration, it is possible to extract the emotion that is closest to the first emotion in the space represented by the six axes as the second emotion that is similar to the first emotion, thereby making it possible to extract a more appropriate second emotion.

[0207] In the information processing method of this embodiment, the estimation process (steps S502, S522) uses at least one of the following as an element related to the state of user U1: alertness, which is an index indicating the alertness state of user U1, and user load, which is an index indicating the degree of load placed on user U1.

[0208] According to this configuration, it is possible to use appropriate information (user U1's alertness, user load of user U1) as elements related to the state of user U1, thereby making it possible to improve the accuracy of estimating the types of emotions that user U1 can perceive.

[0209] In the information processing method according to the present embodiment, in the estimation process (steps S502, S522), it is assumed that the user U1 is in a state in which it is easier to perceive emotions as the awakening level of the user U1 increases, and the number of types of emotions that the user U1 can perceive is increased.

[0210] According to this configuration, it is possible to change the type of emotion that the user U1 can perceive depending on the level of awakening of the user U1, thereby making it possible to improve the accuracy of estimating the type of emotion that the user U1 can perceive.

[0211] In the information processing method according to this embodiment, in the estimation process (steps S502, S522), it is assumed that the user U1 is in a state where he or she can easily perceive emotions as the user load becomes smaller, and the number of emotions that the user U1 can perceive is increased.

[0212] According to this configuration, it is possible to change the type of emotion that the user U1 can perceive depending on the magnitude of the user load, and therefore it is possible to improve the accuracy of estimating the type of emotion that the user U1 can perceive.

[0213] In the information processing method according to the present embodiment, the agent device 200 is capable of communicating with a plurality of users. In the estimation process (steps S502, S522), the type of emotion perceivable by the plurality of users is estimated for each user, in the determination process (steps S503, S523), it is determined whether or not the expression of the first emotion is perceivable by all of the plurality of users, in the extraction process (steps S505, S511, S524, S531, S543), if it is determined that the expression of the first emotion is not perceivable by all of the plurality of users, a second emotion similar to the first emotion is extracted from the estimated types of emotions, and in the control process (steps S504, S506, S525), if it is determined that the expression of the first emotion is perceivable by all of the plurality of users, the agent device 200 is caused to execute a first performance, and if it is determined that the expression of the first emotion is not perceivable by all of the plurality of users, the agent device 200 is caused to execute a second performance.

[0214] With this configuration, even when the agent device 200 communicates with a plurality of users, it is possible to establish information transmission from the agent device 200 to each user, thereby improving the continuity of mutual communication. In other words, information can be transmitted more appropriately using emotional expressions that each user can perceive, depending on the state of each user, etc.

[0215] In the information processing method according to this embodiment, the first emotion is an emotion that has been determined to be conveyed to the user U1 within a predetermined time. In the estimation process (step S522), a future state of the user U1 is estimated in time series based on the current state of the user U1 and the circumstances surrounding the user U1, and the type of emotion that the user U1 can perceive within the predetermined time is estimated in time series based on the current state of the user and the future state of the user. In the determination process (step S523), it is determined whether the user U1 can perceive an expression of the first emotion at at least a part of the predetermined time based on the type of emotion estimated in time series. In the extraction process (steps S524, S531), it is determined whether the user U1 can perceive an expression of the first emotion within the predetermined time. If it is determined that the user U1 cannot perceive the expression of the first emotion, a second emotion similar to the first emotion within the specified time period and a first timing at which the second emotion is perceivable are extracted from the types of emotions estimated in time series, and in the control process (steps S504, S525), if it is determined that the user U1 can perceive the expression of the first emotion at least at a part of the timing within the specified time period, the agent device 200 is caused to execute a first performance at least at that timing, and if it is determined that the user U1 cannot perceive the expression of the first emotion within the specified time period, the agent device 200 is caused to execute a second performance at least at the first timing.

[0216] With this configuration, even in a situation where the current user U1 is not able to perceive the expression of a first emotion (e.g., irritation), it is possible to search for and express a second emotion (e.g., anger) that is closest to the first emotion and that the user U1 can perceive during the period P1 in which the first emotion should be conveyed.

[0217] In the information processing method according to this embodiment, in the control process (steps S511, S512), when the second performance is executed by the agent device 200, the agent device 200 is caused to execute an apology performance expressing an apology for expressing an emotion different from the first emotion. For example, it is possible to execute an apology performance as shown in FIG. 8.

[0218] According to this configuration, if the difference between the first emotion (e.g., irritation) and the second emotion (e.g., anger) causes discomfort to the user U1, it is possible to apologize to the user U1 for the difference. This makes it possible to reduce the distrust and discomfort felt by the user U toward the agent device 200, and to increase the continuity of mutual communication.

[0219] In the information processing method according to the present embodiment, in the control process (steps S511, S512), the strength of the apology in the apology performance is changed based on the magnitude of the similarity (for example, distance) between the first emotion and the second emotion.

[0220] According to this configuration, the greater the similarity between the first emotion and the second emotion, the greater the discomfort caused to the user U1, so the degree of apology is increased. This makes it possible to reduce the distrust and discomfort felt by the user U toward the agent device 200, and to increase the continuity of mutual communication.

[0221] The information processing method according to this embodiment further includes a confidence estimation process (steps S511, S531) for estimating, for each emotion, a confidence of transmission indicating the likelihood of information transmission when transmitting a plurality of emotions that can be expressed by the agent device 200 to the user U1, based on a margin indicating the degree of load on at least one of the user U1's vision and hearing, and in the extraction process (steps S511, S531), a second emotion is extracted from the estimated emotion types based on a similarity indicating the degree of similarity between each emotion corresponding to that type and the first emotion, calculated for each emotion, and the confidence of transmission estimated for each emotion.

[0222] According to this configuration, the confidence of emotion transmission is estimated using the visual comfort of user U1, the auditory comfort of user U1, etc., and the second emotion is extracted using the confidence of emotion transmission, making it possible to extract a more appropriate second emotion that takes into account the probability that the emotion can be transmitted, etc.

[0223] In the information processing method according to the present embodiment, the agent device 200 can express emotions by a plurality of transmission means (e.g., each body part, sound, etc.), and the transmission means is realized by one or a plurality of components (e.g., body parts, sound, inhibition factors) included in the agent device 200, and one or a plurality of components used when the agent device 200 expresses each emotion are set for each emotion. The method further includes a confidence estimation process (step S541) for estimating, for each emotion, a confidence of transmission indicating the likelihood of information transmission when transmitting a plurality of emotions that the agent device 200 can express to the user U1 based on a margin indicating the degree of load on at least one of the senses of vision and hearing of the user U1 and the components, and in the extraction process (steps S542, S543), a second emotion is extracted from the types of estimated emotions based on a similarity indicating the degree of similarity between each emotion corresponding to the type and the first emotion calculated for each emotion, and the confidence of transmission estimated for each emotion.

[0224] According to this configuration, it is possible to estimate the certainty of the transmission of primary information by taking into consideration the type of primary information that the user U1 can perceive (parts and sounds used as a means of transmission of the agent device 200, and obstructive factors), and to estimate the certainty of the transmission of secondary information using this certainty of the transmission of primary information. In other words, since the certainty of the transmission is estimated in two stages, that is, the certainty of the transmission of primary information and the certainty of the transmission of secondary information, it is possible to improve the accuracy of extraction of a more appropriate second emotion by taking into consideration the probability that an emotion can be transmitted, etc.

[0225] The information processing device 110 is an information processing device that controls an agent device 200 capable of communicating with a user U1. The information processing device 110 includes an emotion estimation unit 121 (an example of an estimation unit) that estimates the type of emotion that the user U1 can perceive based on at least one of a plurality of elements related to the state of the user U1, a determination unit 122 that compares the type of the estimated emotion with a first emotion to be transmitted to the user U1 that has already been determined and determines whether the user U1 can perceive the expression of the first emotion, an extraction unit 123 that extracts a second emotion similar to the first emotion from the types of the estimated emotions when it is determined that the user U1 cannot perceive the expression of the first emotion, and an agent control unit 125 that causes the agent device 200 to execute a first performance expressing the first emotion when it is determined that the user U1 can perceive the expression of the first emotion, and causes the agent device 200 to execute a second performance expressing the second emotion when it is determined that the user U1 cannot perceive the expression of the first emotion. The information processing device 110 may be a device built into the agent device 200, or may be a device different from the agent device 200. Moreover, instead of the information processing device 110, an information processing system configured with a plurality of devices capable of executing each process realized by the information processing device 110 may be used.

[0226] According to this configuration, even when the driving load of the user U1 is high or the awakening level of the user U1 is low, it is possible to establish information transmission from the agent device 200 to the user U1, and it is possible to increase the continuity of mutual communication. That is, it is possible to more appropriately transmit information using emotional expressions that the user U1 can perceive, depending on the state of the user U1, etc.

[0227] Note that each processing procedure shown in each embodiment is an example for realizing each embodiment, and the order of some of the processing procedures may be changed to the extent that each embodiment is realizable, and some of the processing procedures may be omitted or other processing procedures may be added.

[0228] Each process in each embodiment is executed based on a program for causing a computer to execute various processing procedures. Each embodiment can also be understood as an embodiment of a program for realizing a function for executing each process, and a recording medium for storing the program. For example, an update process for adding a new function to an information processing device can store the program in a storage device of the information processing device. This makes it possible to cause the updated information processing device to execute each process shown in each embodiment.

[0229] Although the embodiments of the present invention have been described above, the above-mentioned embodiments merely show application examples of the present invention, and it is not intended that the technical scope of the present invention be limited to the specific configurations of the above-mentioned embodiments. [Explanation of symbols]

[0230] 20 network, 100 information processing system, 101 image acquisition unit, 102 position information acquisition unit, 103 sound input unit, 104 sensors, 110 information processing device, 120 control unit, 121 emotion estimation unit, 122 judgment unit, 123 extraction unit, 124 certainty estimation unit, 125 agent control unit, 130 memory unit, 131 agent information DB, 132 communication information DB, 133 emotion information DB, 134 map information DB, 140 communication unit, 200 agent device, 230 display unit, 240 sound output unit, 250 drive unit

Claims

1. 1. An information processing method for controlling an agent device capable of communicating with a user, comprising: An estimation process for estimating a type of emotion that the user can perceive based on at least one of a plurality of factors related to the state of the user; a determination process of comparing the type of the estimated emotion with a first emotion that has already been determined to be conveyed to the user, and determining whether or not the user can perceive the expression of the first emotion; an extraction process for extracting a second emotion similar to the first emotion from the estimated emotion types when it is determined that the expression of the first emotion is not perceivable by the user; a control process for causing the agent device to execute a first performance expressing the first emotion when it is determined that the user is capable of perceiving the expression of the first emotion, and for causing the agent device to execute a second performance expressing the second emotion when it is determined that the user is not capable of perceiving the expression of the first emotion. Information processing methods.

2. 2. The information processing method according to claim 1, In the extraction process, an emotion that is closest to the first emotion among the types of estimated emotions is extracted as the second emotion in two-dimensional coordinates defined by an arousal level that indicates a person's arousal state as a numerical value and an emotional valence that indicates a person's emotion as a numerical value. Information processing methods.

3. 2. The information processing method according to claim 1, In the extraction process, in a six-dimensional coordinate system in which each of the emotions constituting Ekman's six emotions is defined as a parameter value on a coordinate axis, an emotion having a closest distance between each parameter value corresponding to the six emotions and each parameter value corresponding to the first emotion for each type of estimated emotion is extracted as the second emotion. Information processing methods.

4. 4. An information processing method according to claim 1, further comprising: In the estimation process, at least one of an alertness level, which is an index indicating an alertness state of the user, and a user load, which is an index indicating a level of load imposed on the user, is used as the element related to the state of the user. Information processing methods.

5. 5. The information processing method according to claim 4, In the estimation process, as the user's level of arousal increases, it is determined that the user is in a state in which it is easy to perceive emotions, and the number of types of emotions that the user can perceive is increased. Information processing methods.

6. 5. The information processing method according to claim 4, In the estimation process, it is determined that the user is in a state in which it is easy to perceive emotions as the user load becomes smaller, and the number of types of emotions that the user can perceive is increased. Information processing methods.

7. 4. An information processing method according to claim 1, further comprising: the agent device is capable of communicating with a plurality of the users; In the estimation process, a type of emotion perceivable by each of the plurality of users is estimated; The determination process includes determining whether or not the expression of the first emotion is perceivable by all of the plurality of users; In the extraction process, when it is determined that the expression of the first emotion is not perceivable by all of the plurality of users, a second emotion similar to the first emotion is extracted from the estimated types of emotions; the control process causes the agent device to execute the first representation when it is determined that the expression of the first emotion is perceivable by all of the plurality of users, and causes the agent device to execute the second representation when it is determined that the expression of the first emotion is not perceivable by all of the plurality of users. Information processing methods.

8. 4. An information processing method according to claim 1, further comprising: the first emotion is an emotion determined to be conveyed to the user within a predetermined time; In the estimation process, a future state of the user is estimated in time series based on a current state of the user and a situation around the user, and a type of emotion that the user can perceive within the predetermined time is estimated in time series based on the current state of the user and the future state of the user; the determining process determines whether or not the user can perceive an expression of the first emotion at at least a part of the predetermined time based on the type of the emotion estimated in time series; in the extraction process, when it is determined that the user is not able to perceive an expression of the first emotion within the predetermined time, a second emotion similar to the first emotion and a first timing at which the second emotion becomes perceivable within the predetermined time are extracted from the types of emotions estimated in time series; in the control process, when it is determined that the user is capable of perceiving the expression of the first emotion at at least a part of the timing within the predetermined time, the agent device is caused to execute the first representation at least at the timing, and when it is determined that the user is not capable of perceiving the expression of the first emotion within the predetermined time, the agent device is caused to execute the second representation at least at the first timing. Information processing methods.

9. 4. An information processing method according to claim 1, further comprising: in the control process, when the second representation is executed by the agent device, the control process causes the agent device to execute an apology representation expressing an apology for having caused the agent device to express an emotion different from the first emotion. Information processing methods.

10. 10. The information processing method according to claim 9, In the control process, a strength of the apology in the apology performance is changed based on a degree of similarity between the first emotion and the second emotion. Information processing methods.

11. 2. The information processing method according to claim 1, and a certainty estimation process for estimating a certainty of transmission, which indicates a certainty of information transmission when transmitting a plurality of emotions that can be expressed by the agent device to the user, based on a margin indicating a degree of load on at least one of the user's vision and hearing, In the extraction process, the second emotion is extracted from the estimated emotion types based on a similarity degree calculated for each emotion corresponding to the type and the first emotion, the similarity degree being indicative of a degree of similarity between the first emotion and the emotion, and the confidence degree of transmission estimated for each emotion. Information processing methods.

12. 2. The information processing method according to claim 1, the agent device is capable of expressing emotions through a plurality of communication means; the transmission means is realized by one or more components included in the agent device, the one or more components used when the agent device expresses each emotion are set for each emotion, and a certainty estimation process for estimating, for each emotion, a degree of certainty of information transmission indicating the certainty of information transmission when transmitting a plurality of emotions that can be expressed by the agent device to the user, based on a margin indicating a degree of load on at least one of the user's vision and hearing and on the components, In the extraction process, the second emotion is extracted from the estimated emotion types based on a similarity degree calculated for each emotion corresponding to the type and the first emotion, the similarity degree being indicative of a degree of similarity between the first emotion and the emotion, and the confidence degree of transmission estimated for each emotion. Information processing methods.

13. An information processing device for controlling an agent device capable of communicating with a user, an estimation unit that estimates a type of emotion that the user can perceive based on at least one of a plurality of elements related to the state of the user; a determination unit that compares the type of the estimated emotion with a first emotion that has already been determined to be conveyed to the user, and determines whether or not the user can perceive an expression of the first emotion; an extraction unit that extracts a second emotion similar to the first emotion from among the types of the estimated emotions when it is determined that the user cannot perceive the expression of the first emotion; a control unit which, when it is determined that the expression of the first emotion is perceivable by the user, causes the agent device to execute a first performance expressing the first emotion, and, when it is determined that the expression of the first emotion is not perceivable by the user, causes the agent device to execute a second performance expressing the second emotion. Information processing device.

Images