Information processing method and information processing apparatus

Abstract

To reduce a continual feeling of discomfort of a user after an apology of an agent device, and increase a sense of trust from the user.SOLUTION: An information processing method for controlling an agent device 200 includes: a discomfort level estimation process (step S504) which estimates a level of discomfort of a user U1 on the basis of at least one of biometric information, a content of speech and an action of the user U1; and a control process (steps S505 to S516) which changes a mode of the agent device 200 to at least one of a normal mode, an apology mode and a standby mode. When a predetermined condition based on the level of discomfort of the user U1 is satisfied after notification information is provided to the user U1, the control process shifts the mode of the agent device 200 from the normal mode to the apology mode. When the level of discomfort of the user U1 does not fall below a first threshold TH3 after the agent device 200 enters the apology mode, the mode of the agent device 200 is shifted from the apology mode to the standby mode.SELECTED DRAWING: Figure 10

Description

Technical Field

[0001] The present invention relates to an information processing method and an information processing apparatus for controlling an agent device capable of communicating with a user.

Background Art

[0002] Conventionally, there are devices capable of communicating with a user. For example, a technology for making a phone call between a bot (automatic or semi-automatic system) and a user has been proposed (see, for example, Patent Document 1). In this conventional technology, during a phone call, the conversation is analyzed to determine an increase or decrease in the tension of the conversation between the bot and the user. When it is determined that the tension exceeds a threshold value, the person on the other end of the phone is switched from the bot to a human operator.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In the above-described conventional technology, since it is determined that the tension between the bot and the user increases at the timing when the bot apologizes, the person on the other end of the phone can be switched from the bot to a human operator, and this operator can apologize to the user. However, even if a human operator apologizes to the user, there is a possibility that the user still feels angry with the bot and the unpleasant state continues. In such a state, if the person on the other end of the phone is switched from a human operator to the bot and the conversation between the bot and the user is resumed, the user's discomfort level may be increased, and there is a risk of losing the user's trust.

[0005] An object of the present invention is to reduce the continuation of the user's unpleasant state after the agent device apologizes and enhance the user's trust. [Means for solving the problem]

[0006] One aspect of the present invention is an information processing method for controlling an agent device capable of communicating with a user. This information processing method includes: a discomfort estimation process that estimates the user's level of discomfort based on at least one of the user's biometric information, the user's speech content, and the user's actions; and a control process that changes the mode of the agent device to at least one of the following: a normal mode that provides notification information to the user, an apology mode that shows an apology to the user, and a standby mode that is less likely to attract the user's attention than the normal mode. In the control process, after providing notification information to the user, if a predetermined condition based on the user's level of discomfort is met, the agent device is transitioned from the normal mode to the apology mode; and after transitioning the agent device to the apology mode, if the user's level of discomfort does not fall below a first criterion, the agent device is transitioned from the apology mode to the standby mode. [Effects of the Invention]

[0007] According to the present invention, it is possible to reduce the persistence of the user's discomfort after an apology from the agent device and to increase the user's trust in the device. [Brief explanation of the drawing]

[0008] [Figure 1] Figure 1 is a simplified diagram showing an example of the interior layout of a vehicle. [Figure 2] Figure 2 is a simplified diagram illustrating an example of an apology issued by an agent device after it has outputted notification information. [Figure 3] Figure 3 is a simplified front view showing an example of the external configuration of an agent device. [Figure 4] Figure 4 shows an example of the system configuration of an information processing system installed in a vehicle. [Figure 5] Figure 5 shows an example of a method for determining the confidence level of a sensing result. [Figure 6]Figure 6 shows an example of a method for determining user discomfort and an example of the relationship between user discomfort and the performance of the agent device. [Figure 7] Figure 7 shows an example of the transition when changing the location of the agent device. [Figure 8] Figure 8 shows an example of the transition of an agent device when the user's level of displeasure decreases after an apology for the information provided. [Figure 9] Figure 9 shows an example of the transition of an agent device when the user's level of displeasure does not decrease after an apology for the information provided. [Figure 10] Figure 10 is a flowchart showing an example of control processing in an information processing device. [Modes for carrying out the invention]

[0009] Embodiments of the present invention will be described below with reference to the attached drawings.

[0010] [Example of agent device installation] Figure 1 is a simplified diagram showing an example of the interior configuration of vehicle C1. Note that Figure 1 shows an example of the exterior configuration as viewed from the rear of vehicle C1, specifically the area in front of the driver's seat and passenger seat (not shown). Furthermore, for the sake of clarity, Figure 1 omits illustrations of anything other than the dashboard 2, steering wheel 3, front windshield 4, rearview mirror 5, camera 101, and agent equipment 200.

[0011] The agent device 200 is a small robot installed on the dashboard 2 of the vehicle C1. In this embodiment, an example is shown in which a robot modeled after a human is used as the agent device 200. Note that Figure 1 shows an example in which the agent device 200 is installed on the dashboard 2, but it is not limited to this. For example, the agent device 200 may be installed on the top of the front windshield 4. Also, Figure 1 shows an example in which a robot modeled after a human face is used as the agent device 200, but it is not limited to this. For example, the agent device 200 may be a robot modeled after a whole human body, a robot modeled after an animal such as a rabbit or a pig, a robot modeled after a fictional creature (for example, the face of an anime character), or a robot modeled after another object (for example, a television-type device or a radio-type device). In this way, it is possible to use an anthropomorphic agent as the agent device 200.

[0012] The agent device 200 performs various operations based on instructions from the information processing device 110 (see Figure 4). For example, based on the control of the information processing device 110, the agent device 200 outputs various information such as driving assistance and information about surrounding facilities when the user is operating the vehicle. This driving assistance could include notifying the driver of moving objects in front of or behind the vehicle. For example, as notification of a moving object in front, the agent device 200 could output a voice message such as, "There's a railroad crossing ahead, so be careful," or "There's a person ahead." In this way, the agent device 200 performs driving assistance. Figure 1 shows an example where voice information S1, which displays an advertisement for XYZ Coffee Shop, is output as notification information.

[0013] Here, "notification" means conveying or informing someone of some kind of information. Furthermore, information that should be notified to the user of vehicle C1 (the occupant of vehicle C1) will be referred to as notification information. Notification information may be conveyed to the user by image display or by audio output. In this embodiment, an example of conveying notification information to the user by audio output is mainly shown. Note that the notification shown in this embodiment may also be referred to as notification, transmission, etc.

[0014] In addition, as notification information, for example, information about the vehicle C1, information about the driving support of the vehicle C1, advertisement information about commercial facilities, tourist facilities, etc. existing around the vehicle C1, etc. are assumed. The information about the vehicle C1 is, for example, information about the vehicle C1 itself such as notification of the remaining battery level, notification of the half door, etc. Further, the information about the driving support of the vehicle C1 is, for example, various information necessary when the user U1 drives such as notification of wearing a seat belt, notification of a traffic signal, route guidance, etc. Further, the advertisement information about commercial facilities, tourist facilities, etc. existing around the vehicle C1 is, for example, an advertisement of a shopping mall, a guide to a tourist destination, etc.

[0015] The camera 101 is provided on the ceiling inside the vehicle C1 and captures a subject inside the vehicle C1 to generate an image (image data). Note that the camera 101 is composed of, for example, one or a plurality of camera devices or image sensors capable of capturing a subject. For example, the camera 101 can be provided above the front windshield 4, that is, above the rearview mirror 5. In FIG. 1, an example including at least one camera 101 is shown, but two or more imaging devices may be provided and all or part of the images of these imaging devices may be used. Further, the installation location of each imaging device is not limited to the example shown in FIG. 1 and can be changed as appropriate. Also, one or a plurality of devices capable of acquiring subjects existing in all directions of the vehicle C1 and subjects inside the vehicle C1, for example, a 360-degree camera, may be used.

[0016] [Example of apology for notification by agent device] FIG. 2 is a diagram schematically showing an example of an apology when the agent device 200 apologizes for the notification information after the agent device 200 outputs the notification information. Note that the example shown in FIG. 2 is a modification of a part of the agent device 200 in the configuration example shown in FIG. 1, and the parts other than the agent device 200 are common to FIG. 1.

[0017] As shown in FIG. 2, after the agent device 200 outputs notification information, if it is determined that the user U1 is made uncomfortable by the notification information or the mode of the notification, the agent device 200 apologizes to the user U1. For example, it is possible to express an apology to the user U1 by voice output or the movement of the agent device 200. In FIG. 2, an example of expressing an apology to the user U1 by outputting voice information S2 with the face of the agent device 200 facing the direction of the user U1 is shown. In the present embodiment, the mode in which the agent device 200 executes an apology performance is referred to as an apology mode and will be described.

[0018] Note that the mode of notification is, for example, the volume of voice output at the time of notification, the timing of notification, the mannerisms of the agent device 200 at the time of notification, and the like. For example, if the volume of voice output is too large or too small, there is a risk of giving the user U1 a sense of discomfort. Also, if the timing of notification is late for the target to be notified, there is a risk of giving the user U1 a sense of discomfort. For example, if the target to be notified is a guide to XYZ Coffee Shop and the notification timing is after the vehicle C1 has passed by XYZ Coffee Shop, there is a risk of giving the user U1 a sense of discomfort. Also, for example, as shown in FIG. 1, if the target to be notified is a breakfast set at XYZ Coffee Shop from 7:00 to 10:00 and the notification timing is 10:24, there is a risk of giving the user U1 a sense of discomfort. Also, if the movement or expression of the agent device 200 does not match the target to be notified, there is a risk of giving the user U1 a sense of discomfort. For example, when the user U1 likes coffee and the target to be notified is XYZ Coffee Shop, if the agent device 200 notifies with a sad expression, there is a risk of giving the user U1 a sense of discomfort.

[0019] As mentioned above, if user U1 is in a state of displeasure and the object of that displeasure is agent device 200, it is conceivable that agent device 200 could alleviate the displeasure caused to user U1 by apologizing to user U1. However, it is also conceivable that user U1's displeasure towards agent device 200 may persist. In such a case, if agent device 200 continues to apologize to user U1 or provides notification information to user U1, it may cause user U1 to feel even more displeasure. In other words, if agent device 200 apologizes and then resumes communication with user U1 while user U1 is still angry, it is possible that user U1's trust in agent device 200 may be undermined. For example, if user U1 is angry about an event executed by agent device 200, even if agent device 200 temporarily apologizes for that event, it is conceivable that user U1 may feel that agent device 200 has not repented for the apology. In this situation, if agent device 200 resumes the next event, user U1 may feel that agent device 200 has not learned its lesson at all, potentially undermining trust in agent device 200.

[0020] Furthermore, if agent device 200 continues to apologize to user U1 or provides information to user U1 even though user U1 is feeling uncomfortable, it may give the impression that agent device 200 is a thoughtless robot. In this case, user U1 may feel that agent device 200 is not thinking about anything, as it is behaving in a way that is unrelated to user U1's discomfort, making it difficult to alleviate user U1's feelings of discomfort.

[0021] Therefore, in this embodiment, if user U1 continues to experience discomfort with the agent device 200, the agent device 200 is set to a mode that is less likely to attract user U1's attention (standby mode) until user U1's discomfort decreases. Furthermore, if user U1's discomfort with the agent device 200 is resolved after transitioning to the standby mode, the agent device 200 is returned to the normal mode. In this embodiment, the mode of the agent device 200 that is less likely to attract user U1's attention than the normal mode is referred to as the standby mode. Additionally, a mode that is less likely to attract user U1's attention than the normal mode but more likely to attract user U1's attention than the standby mode is referred to as the intermediate mode.

[0022] Thus, when an apology from agent device 200 is deemed ineffective, agent device 200, which is the object of user U1's discomfort, is temporarily removed from user U1's view. At this point, if user U1's discomfort is resolved, for example, if user U1's anger subsides, it can be considered that agent device 200 has been removed from user U1's object of discomfort. Therefore, once user U1's discomfort is resolved, agent device 200, which was removed from user U1's view, is gradually made to reappear. This makes it possible to have agent device 200 perform actions that show it values ​​its relationship with user U1, thereby deepening the bond between user U1 and agent device 200. It also allows user U1 to deepen their affection for agent device 200. As a result, user U1 can appropriately accept the information communicated by agent device 200. Furthermore, for user U1 who has experienced discomfort, it is possible to alleviate their discomfort, increase the trust relationship with user U1, and facilitate smooth communication with user U1.

[0023] The degree of discomfort of user U1 can be estimated based on at least one of the following: user U1's biometric information (e.g., heart rate, body temperature, facial surface temperature, blood flow, information related to excitement level), user U1's speech content (e.g., predetermined keywords such as "you bastard"), and user U1's actions (e.g., magnitude, speed, and facial expression of body movements). For example, the degree of discomfort of user U1 can be detected based on user U1's facial expression, user U1's movements, and the voice that user U1 makes. User U1's facial expression and user U1's movements can be obtained based on images acquired by camera 101. User U1's facial expression also includes user U1's gaze. The voice that user U1 makes can be obtained based on voice information acquired by voice input unit 103 (see Figure 4). The method for estimating user U1's degree of discomfort will be explained in detail with reference to Figure 6, etc.

[0024] Furthermore, the factors causing discomfort for user U1 can be estimated based on the circumstances surrounding user U1. The method for estimating the factors causing discomfort for user U1 will be described later.

[0025] In this embodiment, provided that the sensing confidence level is higher than the standard, when user U1 experiences discomfort, the agent device 200 is instructed to perform an apology to user U1 for that discomfort, as well as a standby sequence (transition to standby mode and intermediate mode). Here, sensing confidence level is an index representing the degree of certainty regarding the sensing. Furthermore, sensing confidence level can be calculated based on sensing stability, CPU utilization rate, etc. Figure 5 shows an example of calculating sensing confidence level based on sensing stability and CPU utilization rate.

[0026] The confidence level of the sensing corresponds to the confidence level of the processing in the equipment used to control the agent device 200. The equipment used to control the agent device 200 includes, for example, the parts shown in Figure 4. For example, as the confidence level of the processing in each of these devices increases, it becomes possible to acquire information about the user U1 and information about the vehicle C1 more accurately.

[0027] In this embodiment, we will use user U1, who is seated in the driver's seat of vehicle C1, as an example. However, this embodiment can also be applied when there are multiple passengers in vehicle C1. In this case, the target of the apology and waiting animations may be user U1 seated in the driver's seat, or other passengers. Furthermore, the passengers who are the target of the notification information may also be the targets of the apology and waiting animations.

[0028] [Example of external configuration of agent device] Figure 3 is a simplified front view showing an example of the external configuration of the agent device 200. In this embodiment, when the agent device 200 is ON and in its normal mode, a facial image is displayed on the display unit 210 (see Figure 4). The agent device 200 can be turned ON in response to the ON operation of the vehicle C1. The agent device 200 can also be turned ON in response to user operation related to the agent device 200. Note that the ON / OFF operation of the vehicle C1 refers to the ON or OFF operation of the start key related to starting or stopping the vehicle C1.

[0029] Figure 3(A) shows a perspective view of the agent device 200. Figure 3(B) shows a front view of the agent device 200. Note that the agent device 200 shown in Figure 3(B) corresponds to the agent device 200 shown in Figure 1. Figure 3(C) shows a top view of the agent device 200. Figure 3(D) shows a side view of the agent device 200. In this embodiment, the state of the agent device 200 shown in Figures 3(A) to (D) will be described as the normal state. Figures 3(E) and 3(F) show examples of the external configuration when the agent device 200 performs an apology presentation to user U1 as part of its operation mode. Figures 3(G) and 3(H) show examples of the external configuration when the agent device 200 performs a relief presentation to reduce user U1's level of discomfort as part of its operation mode. Note that the apology presentation is performed when the agent device 200 transitions to the apology mode. Furthermore, the relief animation is executed when the agent device 200 transitions from an apologetic mode to a normal mode, or when it transitions from a standby mode (or intermediate mode) to a normal mode.

[0030] The agent device 200 is a device consisting of a roughly spherical main body 201, and a display unit 210 is provided on the surface of the main body 201. Various images, such as eye parts E1 and E2, are displayed on the display unit 210. The display unit 210 may be provided on the entire surface of the main body 201, or on only a part of the surface of the main body 201. Furthermore, if the display unit 210 is provided on only a part of the surface of the main body 201, the parts other than the area where the display unit 210 is provided may be provided with physical parts that constitute a face (for example, a mouth and a nose).

[0031] In this embodiment, an example is shown in which the agent device 200 is installed on the dashboard 2 of the vehicle C1, with the eye sections E1 and E2 facing inwards. For this reason, one surface of the agent device 200 that includes the portion where the eye sections E1 and E2 are displayed will be referred to as the front or face section. Furthermore, the front of the agent device 200 will be referred to as the front side of the agent device 200, and the side opposite the front of the agent device 200 will be referred to as the rear side of the agent device 200. In addition, in the agent device 200, the right side in Figure 3 will be referred to as the left side of the agent device 200, and the left side in Figure 3 will be referred to as the right side of the agent device 200.

[0032] The main body 201 rotates in the front-to-back direction of the agent device 200, i.e., in the direction of arrow A1 (see Figure 3(A) and (D)), with the lower part of the agent device 200 as the pivot point. Similarly, the main body 201 rotates in the left-to-right direction of the agent device 200, i.e., in the direction of arrow A2 (see Figure 3(B)), with the lower part of the agent device 200 as the pivot point. Similarly, the main body 201 rotates in the direction of rotation around the rigidity axis, i.e., in the direction of arrow A3 (see Figure 3(C)), with the lower part of the agent device 200 as the pivot point. Thus, a drive unit 230 (see Figure 4) for driving the main body 201 is provided at the lower part of the agent device 200. This makes it possible to rotate the main body 201 of the agent device 200 in the left-to-right direction, rotate the main body 201 in the front-to-back direction, swing the main body 201 in the left-to-right direction, and swing the main body 201 in the front-to-back direction.

[0033] Furthermore, the display unit 210 can be configured as a display medium capable of displaying various images. Various display media such as light-emitting sheets, liquid crystal displays, and organic EL (Electro-Luminescence) can be used as this display medium. For example, as shown in Figures 3(A) to (D), the normal eye areas E1 and E2 can be displayed.

[0034] Furthermore, as shown in Figure 3(E), for example, a sweat image SW1 can be displayed along with the downcast eye area E1 and E2. Also, as shown in Figure 3(F), for example, the upper part of the face B1 can be set to a predetermined color (e.g., blue) to indicate an anxious expression, and a vertical line image V1 indicating an anxious expression can be displayed on a part of that B1. In addition, as shown in Figures 3(E) and (F), the anxious attitude can be further emphasized by shaking the main body 201 from side to side (arrows A4 and A5). Note that the expressions shown in Figures 3(E) and (F) are examples of apology performances.

[0035] Furthermore, as shown in Figure 3(G), for example, a smiling mouth M1 can be displayed, along with a cheek image RS1 that appears to have red cheeks. Also, as shown in Figure 3(H), a smiling mouth M1, eyes E1, and E2 can be displayed. Note that the facial expressions shown in Figures 3(G) and (H) are examples of expressions of relief.

[0036] In this way, by setting the color of at least a part of the display unit 210 to a specific color, deforming the eye parts E1 and E2, or displaying the mouth part M1, it is possible to display the agent device 200 in a way that makes it appear as if it is anxious, relieved, or otherwise. Note that the display modes shown here are just examples, and each expression may be realized by other display modes. For example, an anxious expression may be realized by rapidly moving the eye parts E1 and E2 from side to side, or by making the eye parts E1 and E2 look around, or by rapidly moving the main body 201 from side to side, or by making the main body 201 look around from side to side.

[0037] Thus, in this embodiment, the agent device 200 can be changed by an operating mode that changes at least a part of each component of the agent device 200, and by a display mode that changes the display state of at least a part of the surface of the agent device 200. Furthermore, the occupants of the vehicle C1 who observe these changes will be able to recognize that the agent device 200 is making some kind of movement. For this reason, in this embodiment, these changes in the agent device 200 will be described as operating modes of the agent device 200.

[0038] The operating modes of these agent devices 200 are examples only, and other operating modes are possible. For example, it is possible to move the agent device 200 within a predetermined range on the dashboard 2. An example of moving the agent device 200 on the rail RL1 installed on the dashboard 2 is shown in Figure 7(A). Furthermore, it is also possible to create a presentation in which the agent device 200 is housed in a hole. An example of housing the agent device 200 in a hole is shown in Figure 7(B).

[0039] Figure 3 shows an example of representing each facial expression in two dimensions by changing the display mode of the display unit 210. However, each facial expression may also be represented in three dimensions by changing the physically installed components such as the eyes, nose, and mouth. In this case, for example, the movement of the eyes may be represented by moving the eye component or by displaying a black image or a white image on the surface of the eye component.

[0040] Furthermore, although this embodiment shows an example of an agent device 200 that only has a face, other body parts (e.g., hands, feet) may also be included in the agent device 200. In this case, each of these parts (e.g., hands, feet) may be changed to realize each attitude (e.g., an anxious attitude, an relieved attitude). For example, each attitude may be expressed by changing the hands to touch the head or face, or by flapping the feet or hands.

[0041] The shapes, display modes, and audio output modes of these agent devices 200 are examples only, and other shapes, display modes, and audio output modes are also possible. Furthermore, Figure 3 shows an example of an agent device 200 in which a substantially spherical main body 201 functions as a face, but the present invention is not limited to this. For example, an agent device 200 may be a device in which a flat main body (for example, a planar display panel) functions as a face. Alternatively, an agent device 200 may be a device in which the face and body are configured as an integrated housing. Alternatively, an agent device 200 may be a device in which the face and body are configured as separate components.

[0042] Furthermore, while this embodiment shows an example of displaying a face image as an image representing a living creature (living creature representation), other images (for example, the agent's whole body, a mechanical object, or a virtual face) may also be displayed as images representing a living creature.

[0043] [Example of an information processing system configuration] Figure 4 shows an example of the system configuration of the information processing system 100 installed in vehicle C1.

[0044] The information processing system 100 comprises a camera 101, a location information acquisition sensor 102, an audio input unit 103, sensors 104, an information processing device 110, and an agent device 200. The information processing device 110 and the agent device 200 are connected by a communication method using wired communication or wireless communication. The information processing device 110 is also connected to the network 20 by a communication method using wireless communication. The network 20 is a network such as a public telephone network or the Internet. The agent device 200 may also be connected to the network 20 by a communication method using wireless communication. Although Figure 3 shows an example in which the information processing device 110 and the agent device 200 are configured as separate units, the information processing device 110 and the agent device 200 may also be configured as a single unit.

[0045] Camera 101 captures an image of a subject and generates an image (image data) based on the control of the information processing device 110, and outputs image information related to the generated image to the information processing device 110. Camera 101 is installed inside at least one part of the vehicle C1 and captures an image of a subject inside the vehicle C1 and generates an image (image data). In Figure 1, camera 101 is shown installed inside the vehicle C1. As described above, camera 101 is composed of, for example, one or more camera devices or image sensors capable of capturing an image of a subject. For example, one camera 101 may be installed in front of the vehicle C1 to capture an image of a subject from the front of the vehicle C1 and generate an image (image data), or another camera 101 may be installed in the rear of the vehicle C1 to capture an image of a subject behind the vehicle C1 and generate an image (image data).

[0046] The position information acquisition sensor 102 acquires position information regarding the location of vehicle C1 and outputs the acquired position information to the information processing device 110. For example, this can be implemented using a GNSS receiver that acquires position information using GNSS (Global Navigation Satellite System). The position information includes various position data such as latitude, longitude, and altitude at the time of receiving the GNSS signal. Position information may also be acquired by other methods. For example, position information may be derived using information from access points or base stations in the vicinity. Position information may also be acquired using beacons. For example, based on the information acquired by the position information acquisition sensor 102, it is possible to determine the state of vehicle C1, such as driving, stopped, or reversing.

[0047] Furthermore, for example, it is possible to determine, based on the location information acquired by the location information acquisition sensor 102, whether vehicle C1 is located near a facility outside of vehicle C1, such as a coffee shop.

[0048] The audio input unit 103 is located inside the vehicle C1 and, based on the control of the information processing device 110, acquires sounds from inside the vehicle C1 and outputs sound information related to the acquired sounds to the information processing device 110. For example, one or more microphones or sound acquisition sensors can be used as the audio input unit 103.

[0049] The sensors 104 are various sensors installed on the vehicle C1, and the detection information acquired by each sensor is output to the information processing device 110. Examples of sensors include LiDAR (Light Detection and Ranging), RADAR (Radio Detection and Ranging), Sonar, human presence sensor, vehicle speed sensor, acceleration sensor, seat occupancy sensor, seat belt sensor, door sensor, battery sensor, non-contact temperature sensor, etc. Note that these are just examples, and other sensors may be used. Also, only some of these sensors may be used.

[0050] The human presence sensor is a sensor that detects the presence, number, location, and status of people inside the vehicle C1. For example, it is possible to use a human presence sensor that utilizes infrared light, ultrasound, visible light, or an image sensor. For example, if a person is seated in the driver's seat, passenger seat, or rear seat, their presence can be detected by the human presence sensor. Furthermore, by using both a human presence sensor and a seating sensor, it is possible to improve the accuracy of detecting the seating status of each seat. In addition, this makes it possible for the agent device 200 to identify the location of the target person.

[0051] The seat occupancy sensor (or seat sensor) is a sensor that detects whether or not there is an occupant sitting in each seat of vehicle C1. The seat belt sensor is a sensor that detects whether or not an occupant sitting in each seat of vehicle C1 is wearing a seat belt. The door sensor is a sensor that detects whether or not each door of vehicle C1 is ajar. The battery sensor is a sensor that measures the remaining charge of the battery installed in vehicle C1.

[0052] The non-contact temperature sensor is a sensor that detects the body temperature of user U1 riding in vehicle C1 without contact. For example, the temperature of user U1's face can be measured using the non-contact temperature sensor. For each of these sensors, known sensors can be used.

[0053] The information processing device 110 comprises a control unit 120, a storage unit 130, and a communication unit 140. The communication unit 140 exchanges various types of information with other devices using wired or wireless communication based on the control of the control unit 120.

[0054] The control unit 120 controls each part based on various programs stored in the memory unit 130. The control unit 120 is implemented by a processing unit such as a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit). For example, the processing speed can be increased by performing image processing using a GPU. Furthermore, the processing speed can be increased even further by performing parallel processing using a GPU. The vehicle ECU (Electronic Control Unit) of vehicle C1 may be used as the control unit 120, or a processing unit different from the vehicle ECU may be provided as the control unit 120. The control unit 120 also includes a conversion function that converts text to speech. This conversion function is implemented by, for example, TTS (Text to Speech).

[0055] The control unit 120 performs control processing to control the operating state of the agent device 200 based on the information output from the camera 101, location information acquisition sensor 102, voice input unit 103, sensors 104, communication unit 140, etc. Specifically, the control unit 120 includes a discomfort level determination unit 121, a discomfort factor estimation unit 122, a confidence level determination unit 123, and an agent control unit 124.

[0056] The discomfort level determination unit 121 determines the level of discomfort of user U1 riding in vehicle C1 based on the information output from the camera 101, location information acquisition sensor 102, voice input unit 103, sensors 104, communication unit 140, etc., and outputs the determination result to the agent control unit 124. The method for determining the level of discomfort of user U1 will be explained in detail with reference to Figure 6(A).

[0057] The discomfort factor estimation unit 122 estimates the discomfort factors of user U1 riding in vehicle C1 based on the information output from the camera 101, location information acquisition sensor 102, voice input unit 103, sensors 104, communication unit 140, etc., and outputs the estimation result to the agent control unit 124. The method for estimating the discomfort factors of user U1 will be described later.

[0058] The confidence level determination unit 123 determines the confidence level of the sensing based on the information output from the camera 101, position information acquisition sensor 102, voice input unit 103, sensors 104, communication unit 140, etc., and outputs the determination result to the agent control unit 124. The method for determining the confidence level of the sensing will be explained in detail with reference to Figure 5, etc.

[0059] The agent control unit 124 controls the operating state of the agent device 200 based on the judgment results from the discomfort level determination unit 121, the discomfort factor estimation unit 122, and the confidence level determination unit 123, as well as the information output from the camera 101, the position information acquisition sensor 102, the voice input unit 103, the sensors 104, the communication unit 140, etc. For example, the agent control unit 124 displays the parts (eyes E1, E2) that make up the face of the anthropomorphized agent on the display unit 210. The agent control unit 124 also displays information to be notified to the occupants of vehicle C1 (notification information) on the display unit 210. Furthermore, the agent control unit 124 outputs the voice of the anthropomorphized agent, voices corresponding to the information to be notified to the occupants of vehicle C1, etc., from the sound output unit 220.

[0060] For example, the agent control unit 124 determines the notification information to be displayed on the display unit 210 and the notification information to be output from the sound output unit 220, and executes control to output that notification information. Notification information concerning objects located inside the vehicle C1 is stored in the notification information DB 132. Examples of output for each of these notification information are shown in Figures 1 and 8(A). For example, in the case of notification information regarding seat belt fastening, audio information regarding seat belt fastening is output as notification information. Alternatively, an image representing a seat belt may be displayed on the display unit 210 as notification information. In the case of notification information regarding battery level, audio information indicating the battery level may be output as notification information, or an image representing the battery may be displayed on the display unit 210 as notification information. In the case of notification information regarding a door ajar, audio information indicating a door ajar may be output as notification information, or an image representing the door may be displayed as notification information.

[0061] Furthermore, for example, if the output of the notification information causes discomfort to the user U1, the agent control unit 124 will instruct the agent device 200 to apologize for the output of that notification information. For example, as shown in Figures 2 and 8(C), the agent device 200 will be instructed to perform an apology. This apology can be performed through voice output, gestures, etc.

[0062] Furthermore, for example, if the user U1's discomfort persists after the agent device 200 apologizes, the agent control unit 125 sequentially transitions the agent device 200 to standby mode, intermediate mode, and normal mode according to the user U1's level of discomfort. For example, as shown in Figure 9(E), the agent device 200 may be transitioned to standby mode, or as shown in Figure 9(F), the agent device 200 may be transitioned to intermediate mode. The method of transitioning between the modes of the agent device 200 will be explained in detail with reference to Figures 8 to 10, etc. The agent control unit 125 also functions as a response method determination unit that determines the response method for transitioning to each mode according to the user U1's level of discomfort. Thus, in this embodiment, the agent device 200 is transitioned to one of the following modes: normal mode, apology mode, standby mode, or intermediate mode, according to the user U1's level of discomfort.

[0063] The memory unit 130 is a storage medium for storing various types of information. For example, the memory unit 130 stores various types of information necessary for the control unit 120 to perform various processes (e.g., control program, agent information DB 131, notification information DB 132, map information DB 133, surrounding conditions DB 134, user status DB 135). The memory unit 130 also stores various types of information acquired via the communication unit 140. As the memory unit 130, for example, ROM (Read Only Memory), RAM (Random Access Memory), SRAM (Static Random Access Memory), HDD (Hard Disk Drive), SSD (Solid State Drive), or a combination thereof can be used.

[0064] The agent information DB131 stores information necessary to realize various operations of the agent device 200. For example, the face image information displayed on the display unit 210 (e.g., eye area E1, E2, sweat image SW1, vertical line image V1, cheek image RS1, mouth area M1) and the audio information output from the sound output unit 220 are stored in the agent information DB131. In addition, for example, operation information for operating the agent device 200 when an apology is performed is stored in the agent information DB131. Furthermore, for example, transition information for transitioning the agent device 200 to standby mode and intermediate mode is stored in the agent information DB131.

[0065] The notification information DB132 stores information necessary for outputting notification information about objects located inside or outside the vehicle C1. For example, in the case of notification information about a door being ajar, image information showing the door and audio information for notifying that the door is ajar are stored as notification information in the notification information DB132. Also, for example, in the case of notification information about wearing a seat belt, image information showing the seat belt and audio information for notifying that the seat belt should be worn are stored as notification information in the notification information DB132. Also, for example, in the case of notification information about the battery level, image information showing the battery and audio information for notifying that the seat belt should be worn are stored as notification information in the notification information DB132.

[0066] Map information DB133 stores map information such as road information necessary for route guidance of vehicle C1. This map information includes road gradient, road intersections, number of road lanes, road width information, road topography information, and road condition. The map information also includes road signs indicating speed limits, one-way streets, etc., road markings such as pedestrian crossings and lane markings. Furthermore, the map information may also include information on road structures (e.g., traffic lights, utility poles), facilities such as buildings, and surrounding tourist information. Agent information DB131, notification information DB132, and map information DB133 may be stored and used in the memory unit 130 of vehicle C1, or they may be obtained and used from external devices via the network 20.

[0067] The surrounding conditions DB134 stores information about the conditions around vehicle C1. This information is acquired and stored based on information output from the camera 101, position information acquisition sensor 102, voice input unit 103, sensors 104, communication unit 140, etc.

[0068] The information stored in the surrounding conditions DB134 can be used for training to improve the detection accuracy when detecting information about the surrounding conditions of vehicle C1.

[0069] The user status DB135 stores information about the status of user U1. This information is acquired and stored based on information output from the camera 101, audio input unit 103, sensors 104, communication unit 140, etc.

[0070] The user state DB135 can be used for training to improve the detection accuracy when detecting information about the state of user U1.

[0071] The agent device 200 is a robotic device that performs various operations based on instructions from the information processing device 110. The agent device 200 includes a display unit 210, a sound output unit 220, and a drive unit 230. The display unit 210, the sound output unit 220, and the drive unit 230 are controlled by a control unit (not shown) provided in the agent device 200.

[0072] The display unit 210 is a display unit that displays various images based on instructions from the information processing device 110. For example, the display unit 210 can be an OLED (Electroluminescence) panel, an LCD (Liquid Crystal Display) panel, or other display panel. The display unit 210 may be configured as a touch panel that allows the user to perform input by touching or bringing their finger close to the display surface, or it may be configured as a separate user interface.

[0073] The sound output unit 220 outputs various sounds based on instructions from the information processing device 110. For example, one or more speakers can be used as the sound output unit 220. Note that the display unit 210 and the sound output unit 220 are examples of a user interface, and some of them may be omitted, or other user interfaces may be used.

[0074] The drive unit 230 drives the main body 201 of the agent device 200 and other parts of the agent device 200 based on instructions from the information processing device 110. For example, the drive unit 230 is a drive device that realizes a mechanism for rotating the main body 201, as shown in Figures 3(A) to (H). Also, for example, the drive unit 230 is a drive device that realizes a mechanism for moving the agent device 200 on the rail RL1, as shown in Figure 7(A). For example, the drive unit 230 is composed of a motor, servo motor, etc., capable of moving the agent device 200 or rotating the main body 201.

[0075] [Example of determining the confidence level of sensing] Next, we will explain how to determine the confidence level of the sensing. As mentioned above, when the confidence level of the sensing is high, the accuracy of acquiring information about the surrounding conditions of vehicle C1 and information about user U1 will also be high, and it is expected that the confidence level of processing in the equipment used to control agent equipment 200 will be high.

[0076] [Example of determining the confidence level of sensing based on sensing stability] Figure 5 shows an example of a method for determining the confidence level of sensing. Figure 5(A) shows the relationship between the detected values ​​of each sensor (camera 101, position information acquisition sensor 102, audio input unit 103, and sensors 104) and time. In this embodiment, for the sake of simplicity, the image sensor constituting the camera 101 is used as an example. In this embodiment, when the pixel value (sensor element value) output from the image sensor becomes 0 or the maximum value (for example, 255), it is referred to as the state in which the sensor element value is fixed. In this embodiment, an example is shown in which the sensing stability is determined based on the percentage of the state in which the sensor element value is fixed during a predetermined period based on the timing of the determination process.

[0077] For example, consider an image sensor that outputs pixel values ​​from 0 to 255. A pixel value of 0 corresponds to black, and a pixel value of 255 corresponds to white. In this image sensor, if the incident light is too strong, for instance, it may output a value that cannot be detected, such as 0 or 255, as the pixel value. This state, where the image sensor outputs an undetectable value, is sometimes referred to as a "lost state." For example, in backlit situations or when there is a sudden change in light, the image sensor may enter a lost state. In this case, the pixel value will output 0 or 255. That is, the pixel value will be stuck at 0 or 255. When an image sensor enters a lost state and outputs a pixel value of 0 or 255, it is presumed that the image sensor is unstable.

[0078] Therefore, in this embodiment, we will show an example of calculating sensing stability based on the proportion of time during which the image sensor is in a lost state and the pixel value is output as 0 or 255. For example, it is possible to determine the proportion of time during which the pixel value is output as 0 or 255 (the proportion of time during which the pixel value is fixed) within a specified period (for example, a few seconds (for example, about 5 seconds)), and calculate the sensing stability based on this proportion. Specifically, as shown below, the fixed ratio is calculated, and the value obtained by subtracting this fixed ratio from 1 is calculated as the sensing stability. Sticking ratio = Time the pixel value is stuck / Specified period t Sensing stability = 1 - adhesion ratio

[0079] For example, if the adhesion rate is 100%, the sensing stability will be 0%, and if the adhesion rate is 0%, the sensing stability will be 100%. Also, for example, if the adhesion rate is 60%, the sensing stability will be 40%, and if the adhesion rate is 30%, the sensing stability will be 70%.

[0080] For example, if the pixel value 255 is output continuously for 5 seconds, the sticking ratio becomes 1, and the sensing stability becomes 0%. In other words, if strong light is continuously detected, the sensing stability becomes 0%, and it is presumed that proper detection using camera 101 is not being performed. Therefore, in such cases, the confidence level of the sensing can be considered low.

[0081] Generally, an image sensor is composed of a large number of pixels. Therefore, when calculating the sensing stability described above, only some of the pixels constituting the image sensor may be used, or all of the pixels may be used. For example, if at least one pixel among the pixels constituting the image sensor is stuck, the sensing stability may be calculated using the time that one pixel is stuck. Alternatively, for example, one or more pixels among the pixels constituting the image sensor may be set to be used for calculating the sensing stability, and the sensing stability may be calculated using the pixel value of those pixels.

[0082] In the graph shown in Figure 5(A), the horizontal axis represents the time axis, and the vertical axis represents the sensor detected value (pixel value). Furthermore, the sensor detected value from the image sensor is assumed to be a value ranging from 0 to the maximum value MAX.

[0083] In the example shown in Figure 5(A), the period from the time of confidence determination t3 to t seconds prior (i.e., the period from time t1 to time t3) is defined as the specified period t. Time t2 is set to half the value of the period from time t1 to time t3. Note that t can be set to a value of approximately 5 seconds, for example. Note that t can be set appropriately based on the type of sensor, the accuracy of the sensing stability, etc.

[0084] In the example shown in Figure 5(A), we will explain assuming that the sensor detected value SE1 up to time t1 is a value between 0 and the maximum value MAX, the sensor detected value SE2 from time t1 to time t2 is 0, and the sensor detected value SE3 from time t2 onward is a value between 0 and the maximum value MAX. In this case, the state in which the sensor detected value is fixed is the time period from time t1 to time t2 when the sensor detected value is 0. Therefore, in the specified period t based on the confidence level determination time t3, the proportion of the period in which the sensor detected value is fixed (fixed ratio) is 1 / 2. In this case, the sensing stability is 50% (1-1 / 2).

[0085] In the graph shown in Figure 5(B), the horizontal axis represents the time axis, and the vertical axis represents the sensing stability. The example in Figure 5(B) also shows an example of setting a threshold TH1 between 50% and 100% to determine the confidence level of the sensing. The threshold TH1 can be set appropriately based on experimental data, etc.

[0086] For example, in the example shown in Figure 5(A), the sensing stability is 50%, which is below the threshold TH1. Therefore, the confidence level of the sensing is determined to be low. On the other hand, if the sensing stability is equal to or greater than the threshold TH1, the confidence level of the sensing is determined to be high.

[0087] [Example of calculating the confidence level of sensing based on CPU utilization] Based on the sensing stability described above, it is possible to determine the confidence level of the sensing. However, even if appropriate detection values ​​are output from each sensor (i.e., the sensing stability is high), it is conceivable that calculation processing using those detection values ​​may not be executed properly when the operating rate of the control unit 120 is high. Therefore, the confidence level of the sensing may be determined using the operating rate of the control unit 120. In this embodiment, an example is shown in which the confidence level of the sensing is determined using the operating rate of the CPU constituting the control unit 120. The operating rate of the CPU can be calculated using a known calculation method.

[0088] In the graph shown in Figure 5(C), the horizontal axis represents time, and the vertical axis represents CPU utilization. The example in Figure 5(C) shows how to set a threshold TH2 for determining the confidence level of sensing between 50% and 100%. It is possible to set the threshold TH2 to a value close to 100%. The threshold TH2 can be set appropriately based on experimental data, etc.

[0089] For example, the CPU utilization rate at the time of confidence assessment t21, or the average CPU utilization rate over a predetermined period based on that time t21, is determined. If the CPU utilization rate (or its average value) is equal to or greater than the threshold TH2, the sensing confidence is determined to be low. On the other hand, if the CPU utilization rate (or its average value) is less than the threshold TH2, the sensing confidence is determined to be high.

[0090] In addition, the above describes methods for determining sensing confidence based on sensing stability and methods for determining sensing confidence based on CPU utilization. However, both of these methods may be used to determine sensing confidence, or either one of them may be used. For example, it is possible to determine that sensing confidence is high if both the sensing confidence based on sensing stability and the sensing confidence based on CPU utilization are determined to be high. In this case, if at least one of the sensing confidence based on sensing stability and the sensing confidence based on CPU utilization is determined to be low, the sensing confidence is determined to be low. Other methods for determining sensing confidence may also be used.

[0091] [Examples of determining user discomfort levels] Next, we will explain the method for determining the level of discomfort of user U1. The level of discomfort of user U1 can be calculated based on user U1's biometric information (e.g., heart rate, blood flow), sounds emitted by user U1, and user U1's appearance (e.g., facial expressions, movements).

[0092] For example, when user U1 becomes uncomfortable, it is assumed that user U1 will become agitated and their heart rate will increase. Therefore, it is possible to quantify user U1's heart rate and calculate user U1's level of discomfort. Also, when user U1 becomes uncomfortable, it is assumed that user U1's face may become sullen, angry, or their eyes may narrow. Therefore, by analyzing an image including user U1's face, it is possible to obtain changes in user U1's facial expression and calculate user U1's level of discomfort based on the degree of change. For example, if user U1's facial expression changes rapidly and the resulting expression is a predetermined expression (e.g., sullen, angry, or narrowed eyes), it is possible to determine that user U1's level of discomfort is high. It is also possible to determine that user U1's level of discomfort is high if user U1's face is in a predetermined expression. Images including user U1's face can be obtained by camera 101 (see Figure 4). Furthermore, for determining facial expressions, it is possible to use publicly known facial expression recognition technologies (e.g., facial expression recognition programs).

[0093] Furthermore, the degree of discomfort of user U1 may be determined using other information related to the face. For example, if user U1 is angry, it is expected that the surface temperature of user U1's face will rise. Therefore, it is possible to estimate the surface temperature of user U1's face based on the image acquired by camera 101 (including user U1's face), a non-contact temperature sensor, etc., and to determine user U1's degree of discomfort based on that temperature. For example, user U1's degree of discomfort can be calculated based on the change in the surface temperature of user U1's face. For example, if the surface temperature of user U1's face changes rapidly and the temperature after the change exceeds a predetermined value, it is possible to determine that user U1 is highly uncomfortable. Furthermore, known temperature estimation techniques can be used for temperature estimation and determination.

[0094] Furthermore, when user U1 becomes uncomfortable, it is anticipated that user U1 may become agitated and utter specific keywords (e.g., "you bastard"), or that user U1 may raise their voice and increase the volume of their voice. Therefore, by analyzing the voice emitted by user U1, specific keywords uttered by user U1 (e.g., "you bastard") and the volume of the voice emitted by user U1 are obtained. Then, if user U1 utters a specific keyword (e.g., "you bastard"), or if the volume of the voice emitted by user U1 exceeds a threshold, it is possible to determine that user U1 is highly uncomfortable. The voice emitted by user U1 can be obtained by the voice input unit 103 (see Figure 4). Furthermore, known sound recognition technology (e.g., a sound recognition program) can be used for voice judgment. It is also possible to use known estimation technology that estimates emotions by combining facial expressions and voice. For example, it is possible to estimate emotions by combining facial expressions and voice and quantify those emotions as the level of discomfort of user U1. An example of using this quantified level of discomfort for user U1 is shown in Figure 6(A). Alternatively, other numerical values ​​for user U1's discomfort level that can be calculated based on at least one of the following may be used: user U1's biometric information, user U1's speech content, and user U1's actions.

[0095] Figure 6(A) shows an example of a method for determining the level of discomfort of user U1. In the graph shown in Figure 6(A), the horizontal axis represents the time axis, and the vertical axis represents the level of discomfort of user U1. This level of discomfort can be calculated using the calculation method described above. The first threshold TH3 is shown as a dotted line. The first threshold TH3 is a threshold used to determine whether user U1 is in a state of discomfort. The first threshold TH3 is also used as a threshold to determine whether user U1 has forgiven agent device 200 after the apology performance. For example, if user U1 says things like "You made a mistake again," "I don't understand what you're saying," "You bastard," or "I can't believe it" to agent device 200 after the apology performance, it is estimated that user U1's level of discomfort is high and that user U1 has not forgiven agent device 200.

[0096] In this embodiment, the same first threshold TH3 is used for determining whether user U1 is in an unpleasant state and for determining whether user U1 has forgiven the agent device 200 after the apology performance. However, these thresholds may be set to different values.

[0097] The first threshold TH3' is a threshold set in place of the first threshold TH3, depending on the increase in the number of times user U1 determines that they will not tolerate agent device 200 after the apology performance within a predetermined period (for example, during one trip). That is, the more times user U1 determines that they will not tolerate agent device 200 after the apology performance within a predetermined period, the lower the first threshold TH3 is set. Note that one trip means a period of time with some meaning. For example, one trip may be the period from when user U1 boards vehicle C1 until when they disembark, or a predetermined time (for example, several minutes to several tens of minutes, several hours, or one day) may be considered one trip. Note that the number of times user U1 determines that they will not tolerate agent device 200 after the apology performance and the number of times agent device 200 transitions to standby mode are the same value. Alternatively, instead of the number of times user U1 determines that they will not tolerate agent device 200 after the apology performance, the number of times agent device 200 apologizes may be used to set the first threshold TH3'.

[0098] The first thresholds TH3 and TH3' may be fixed values ​​or variable values. If the first thresholds TH3 and TH3' are fixed values, they can be set appropriately based on experimental data, etc. If the first thresholds TH3 and TH3' are variable values, it is possible to set the first thresholds TH3 and TH3' based on the level of discomfort of user U1 within a predetermined period prior to the notification information output time (or the apology time of agent device 200) t31. For example, it is possible to calculate the average level of discomfort of user U1 within a predetermined period prior to time t31 (for example, a few seconds to tens of seconds), and set the first thresholds TH3 and TH3' by adding a predetermined value to this average value. In this case, the predetermined value may be variable based on the average value, or it may be a fixed value set based on experimental data, etc.

[0099] Thus, in this embodiment, if the agent device 200 repeatedly transitions to standby mode or repeatedly apologizes within a relatively short period, the first threshold TH3 is changed to a lower value (first threshold TH3') in accordance with the increase in the number of such occurrences. This makes it easier for the agent device 200 to transition to standby mode and harder for it to transition to normal mode. In other words, it becomes possible to keep the agent device 200 in standby mode for a relatively long period.

[0100] In the graph shown in Figure 6(A), the time of output of the notification information (or the time of the agent device 200's apology) is indicated by time t31. Line UH1 shows an example of the user U1's level of discomfort before time t31, and lines UH2 and UH3 show examples of the user U1's level of discomfort after time t31. Line UH2 shows an example of the transition when user U1's level of discomfort increases and remains high even after agent device 200 performs an apology. Line UH3 shows an example of the transition when user U1's level of discomfort decreases and falls below the first threshold TH3.

[0101] Figure 6(B) shows an example of the relationship between user U1's level of discomfort and the performance of agent device 200. In the graph shown in Figure 6(B), the horizontal axis represents user U1's level of discomfort, and the vertical axis represents the degree of performance by agent device 200. User U1's level of discomfort can be calculated using the calculation method described above. The first threshold TH3 (or first threshold TH3') is the same as in Figure 6(A). The degree of performance by agent device 200 is a value that indicates the distance between user U1 and agent device 200, the amount of movement of agent device 200, the frequency of agent device 200's speech (e.g., the number of times it makes small talk), and the loudness of the output sound of agent device 200. Figure 6(B) shows an example where the distance between user U1 and agent device 200 changes linearly, while Figure 7 shows an example where the distance between user U1 and agent device 200 changes in three stages.

[0102] For example, consider a scenario where the agent device 200 is switched to standby mode after an apology. In this case, if the user U1's level of discomfort remains above the first threshold TH3 for an extended period, the standby mode of the agent device 200 is maintained, and therefore, as shown in line OL1, the agent device 200 does not perform any actions.

[0103] Furthermore, for example, if the user U1's level of discomfort falls below the first threshold TH3 after the agent device 200 has transitioned to a standby state, the behavior of the agent device 200 may be increased in accordance with the decrease in the user U1's level of discomfort, as shown in line OL1. For example, it is possible to increase the size of the agent device 200's movements, increase its speed, or increase the volume of the sound emitted by the agent device 200.

[0104] As described above, when user U1's discomfort is resolved, the agent device 200 is switched from standby mode to normal mode. When the agent device 200 is switched from standby mode to normal mode in this way, as shown by line OL1 in Figure 6(B), it is possible to increase the agent device 200's interaction with user U1 as the level of discomfort of user U1 decreases. In this case, as described above, the interaction may be increased linearly or in three stages.

[0105] [Examples of estimated factors causing user discomfort] Next, we will explain the estimation method for estimating the factors causing discomfort for user U1. The factors causing discomfort for user U1 can be estimated based on the surrounding environment of user U1, information about user U1, etc. As information about user U1, it is possible to use the information used in the process of determining the degree of discomfort for user U1, for example, the biometric information of user U1 (e.g., heart rate, blood vessels), sounds emitted by user U1, and the appearance of user U1 (e.g., facial expressions, movements). For example, it is possible to estimate the factors based on specific events that occurred around user U1 at the time when the degree of discomfort for user U1 reached the first threshold TH3 or higher.

[0106] For example, based on the surrounding conditions of user U1, it is possible to estimate unpleasant factors such as congestion (traffic jam) in vehicle C1, the heat inside the vehicle, sudden cutting in from behind by other vehicles, exhaust fumes from trucks ahead, and agent equipment 200.

[0107] For example, based on images of the area around vehicle C1 acquired by camera 101 (e.g., images in front of vehicle C1), the voice of user U1 acquired by voice input unit 103, various sensors included in sensors 104 (e.g., LiDAR, RADAR, Sonar), and traffic congestion information obtainable via communication unit 140, other vehicles present around vehicle C1 can be detected. Based on this information, the discomfort factor estimation unit 122 determines whether or not the road on which vehicle C1 is traveling is congested. For example, if it is detected that many vehicles are lined up and stopped or traveling slowly in front of vehicle C1 based on images in front of vehicle C1, detection information from sensors such as LiDAR, RADAR, and Sonar, it can be determined that the road on which vehicle C1 is traveling is congested. Also, for example, if user U1 repeatedly says things like "It's so crowded," it can be determined that the road on which vehicle C1 is traveling is congested.

[0108] Furthermore, the discomfort factor estimation unit 122 can estimate that the discomfort factor for user U1 is congestion (or crowding) if the road on which vehicle C1 is traveling is congested. Congestion information can be obtained from other devices (for example, a road information provision server) via the communication unit 140. Based on the congestion information obtained in this way and the current location of vehicle C1 obtained by the location information acquisition sensor 102, the congestion situation around vehicle C1 can be understood.

[0109] Furthermore, the temperature inside the vehicle C1 can be detected based on, for example, images of user U1 acquired by camera 101 (e.g., images including user U1's face, images including user U1's whole body or a part thereof), various sensors included in sensors 104 (e.g., temperature sensor, non-contact temperature sensor), and weather information obtainable via communication unit 140. Based on this information, the discomfort factor estimation unit 122 determines whether the temperature inside the vehicle C1 is hot or not. The discomfort factor estimation unit 122 can then estimate that the discomfort factor for user U1 is the heat inside the vehicle if the temperature inside the vehicle C1 is higher than what user U1 perceives as hot. Weather information can be obtained from other devices (e.g., a weather information server) via communication unit 140. Based on the weather information obtained in this way and the current location of vehicle C1 acquired by location information acquisition sensor 102, the temperature conditions around vehicle C1 can be understood. The temperature sensor can also detect the temperature inside vehicle C1. Furthermore, the non-contact temperature sensor can acquire the user U1's body temperature (e.g., the surface temperature of their face).

[0110] Furthermore, it is possible to detect the facial expression of user U1 based on an image that includes user U1's face. Based on this facial expression, it is possible to determine whether the temperature inside vehicle C1 is hot or not. For example, if user U1 is sweating on their face, if user U1 is wiping sweat from their face, or if user U1's face is red, it can be determined that the temperature inside vehicle C1 is hot.

[0111] Furthermore, based on an image including the entire body or part thereof of user U1, it is possible to detect the clothing user U1 is wearing and user U1's movements. Therefore, based on the clothing user U1 is wearing and user U1's movements, it is possible to determine whether the temperature inside vehicle C1 is hot or not. For example, if user U1 is wearing light clothing, fanning their face, or frequently operating the air conditioner, it can be determined that the temperature inside vehicle C1 is hot.

[0112] For example, based on images of the area around vehicle C1 acquired by camera 101 (e.g., images of the area including the front of vehicle C1, images including the face of user U1), and various sensors included in sensors 104 (e.g., LiDAR, RADAR, Sonar), it is possible to detect sudden cut-ins by other vehicles from behind vehicle C1 to the front of vehicle C1. Based on this information, the discomfort factor estimation unit 122 determines whether or not there was a sudden cut-in by another vehicle from behind vehicle C1 to the front of vehicle C1. If there was a sudden cut-in by another vehicle from behind vehicle C1 to the front of vehicle C1, the discomfort factor estimation unit 122 can estimate that the cause of discomfort for user U1 was that cut-in.

[0113] For example, it is possible to detect the facial expression of user U1 based on an image that includes the user U1's face. Based on this facial expression, it is possible to determine whether or not an unexpected action (e.g., a sudden interruption) has occurred. For example, if the user has a surprised expression in response to an unexpected action, it can be determined that an unexpected action has occurred.

[0114] For example, based on images of the area around vehicle C1 acquired by camera 101 (e.g., images of the area in front of vehicle C1), the voice of user U1 acquired by voice input unit 103, and various sensors included in sensors 104 (e.g., LiDAR, RADAR, Sonar, odor sensor), exhaust fumes from other vehicles present around vehicle C1 can be detected. Based on this information, the discomfort factor estimation unit 122 determines whether or not there are large amounts of exhaust fumes from other vehicles present around vehicle C1. For example, based on images of the area around vehicle C1 and detection information from sensors such as LiDAR, RADAR, and Sonar, it is possible to detect the presence of a vehicle (e.g., a large truck) that may emit a large amount of exhaust fumes around vehicle C1. Also, for example, if a large truck is present around vehicle C1 and user U1 says something like "It smells like exhaust fumes," it is possible to determine that a large truck emitting a large amount of exhaust fumes is present around vehicle C1. Furthermore, if, for example, user U1 repeatedly utters phrases such as "The exhaust fumes stink," it can be determined that there is a vehicle (such as a large truck) emitting a large amount of exhaust fumes around vehicle C1.

[0115] Furthermore, since the odor sensor can detect odors inside vehicle C1, it is possible to determine, based on the detection results of the odor sensor, that there is a vehicle emitting exhaust gas (e.g., a large truck) around vehicle C1. In addition, it is possible to detect the facial expression of user U1 based on an image that includes user U1's face. Therefore, it is possible to determine whether or not user U1 has a face that suggests they smell something. For example, if user U1 has slanted eyes and an angry expression towards a smelly truck, if user U1 is frowning towards a smelly truck, if user U1 is making a gesture as if pinching their nose, or if user U1 is fanning their nose with their hand, it is possible to determine that there is a large truck emitting exhaust gas around vehicle C1.

[0116] For example, the state of user U1 can be detected based on the image of user U1's face acquired by camera 101, user U1's voice acquired by voice input unit 103, etc. Based on this information, the discomfort factor estimation unit 122 determines whether or not user U1 is feeling uncomfortable with the agent device 200. For example, if user U1's facial expression is sullen and their gaze is directed towards the agent device 200, it can be determined that user U1 is feeling uncomfortable with the agent device 200. Also, for example, if user U1 is saying something like "Agent, what are you talking about?", it can be determined that user U1 is feeling uncomfortable with the agent device 200.

[0117] Furthermore, the discomfort factor estimation unit 122 can estimate that the cause of discomfort for user U1 is the agent device 200 if user U1 finds the agent device 200 unpleasant.

[0118] It should be noted that multiple factors may occur as causes of discomfort for user U1, or as part of the circumstances surrounding user U1. In other words, it is conceivable that there may be multiple sources of discomfort. For example, it is conceivable that the car may smell bad while stuck in traffic, and there may also be sudden cut-offs. In such cases, multiple causes of discomfort may be judged, or only the cause of discomfort that is expected to cause the highest degree of discomfort may be judged.

[0119] [Example of changing the location of agent equipment] Figure 7 shows an example of transitions when changing the position of the agent device 200. Figure 7(A) shows an example of the movement of the agent device 200 in this embodiment. Figures 7(B1) to (D) show modified examples of the movement example shown in Figure 7(A).

[0120] Furthermore, Figures 7(A) to (D) show examples of transitioning the agent device 200 between a normal mode in which notification information is provided to user U1, a mode that is less likely to attract user U1's attention than the normal mode (standby mode), and a mode that is more likely to attract user U1's attention than the standby mode (intermediate mode). The position of the mode showing an apology to user U1 (apology mode) is the same as the position of the normal mode. The position of the standby mode is preferably a position that is farther away from user U1, out of user U1's line of sight, or less likely to attract user U1's attention. For example, the position of the standby mode can be near the A-pillar on the passenger side of vehicle C1. The position of the intermediate mode can be near the midpoint between the position of the normal mode and the position of the standby mode.

[0121] Figure 7(A) shows an example of a movement structure in which the agent device 200 moves on rail RL1. As shown in Figure 7(A), rail RL1, which consists of two rail members extending in the left-right direction, is installed on the dashboard 2. Wheels capable of running on rail RL1 are installed on the lower part of the agent device 200. The drive unit 230 of the agent device 200 (see Figure 4) moves the agent device 200 in the left-right direction (direction of arrow A11) on rail RL1 based on control from the agent control unit 124. It is possible to employ known rail technology for the rail structure of rail RL1 and the wheel structure of the agent device 200.

[0122] Figure 7(A) shows an example where the agent device 200 is positioned in the normal configuration. The appearance of the agent device 200 when it is positioned in the standby configuration is shown by the dotted line 200a. The appearance of the agent device 200 when it is positioned in the intermediate configuration is shown by the dotted line 200b.

[0123] In Figure 7(A), an example is shown in which the agent device 200 moves along rail RL1. However, rail RL1 may be omitted, and an autonomously moving agent device 200 may be used. For example, at least a portion of the dashboard 2 can be made flat so that the agent device 200 can move autonomously. Then, by having the agent device 200 move along this flat surface of the dashboard 2, the position of the agent device 200 can be changed.

[0124] Figure 7(B1) shows an example of a structure in which multiple holes HO1 to HO3 are installed on the dashboard 2, and the agent device 200 moves using these multiple holes HO1 to HO3. Figure 7(B2) shows an example of the configuration of hole HO1. The configuration of the other holes HO2 and HO3 can be the same as that of hole HO1. Agent devices 202 and 203 are the same as those of agent device 200.

[0125] The hole HO1 is a cylindrical hole large enough to accommodate the agent device 200, and the bottom BO2 is equipped with a rod-shaped lifting / lowering mechanism UD1 capable of raising or lowering the agent device 200. The upper end of the lifting / lowering mechanism UD1 is connected to the lower part of the agent device 200. The lifting / lowering mechanism UD1 is controlled to rise or fall by a drive unit 230 (see Figure 4).

[0126] For example, when agent devices 200, 202, and 203 are in their normal or apologetic state, as shown in Figure 7(B1), only agent device 200 in hole HO1 will be in an elevated state, while agent devices 202 in hole HO2 and 203 in hole HO3 will be in a lowered state and attached to their respective bottoms. Also, for example, when agent devices 200, 202, and 203 are in a standby state, only agent device 203 in hole HO3 will be in an elevated state, while agent devices 200 in hole HO1 and 202 in hole HO2 will be in a lowered state and attached to their respective bottoms. Also, for example, when agent devices 200, 202, and 203 are in an intermediate state, only agent device 202 in hole HO2 will be in an elevated state, while agent devices 200 in hole HO1 and 203 in hole HO3 will be in a lowered state and attached to their respective bottoms.

[0127] Although not shown in Figures 7(B1) and 7(B2), a cover may be placed over each of the multiple holes HO1 to HO3, with only the cover of the hole in which the agent device is in the raised position open and the covers of the other holes closed. By placing a cover over each of the multiple holes HO1 to HO3 in this way, only one agent device can be visually identified by the user U1, thereby increasing the identity of each agent device.

[0128] Furthermore, the examples in Figures 7(A), 7(B1), and 7(B2) may be combined. For example, similar to the examples shown in Figures 7(B1) and 7(B2), multiple holes HO1 to HO3 may be installed in the dashboard 2, and a separate horizontal hole may be provided inside the dashboard 2 to connect the bottoms of each hole HO1 to HO3. That is, a V-shaped hole may be provided inside the dashboard 2. Then, a rail may be installed in the horizontal hole, and the agent device 200 may be made movable along the rail. In addition, depending on the configuration, the agent device 200 may be raised from one of the holes.

[0129] Figure 7(C) shows an example of displaying agent images 205, 205a, and 205b on the display unit 251 of the display device 250 installed on the dashboard 2 of vehicle C1. Agent images 205, 205a, and 205b correspond to agent devices 200. The display state of the display device 250 is controlled by the agent control unit 124. For example, agent images 205, 205a, and 205b can be realized using computer graphics (CG).

[0130] Agent image 205 is displayed in the normal mode and apology mode positions. Agent image 205a is displayed in the standby mode position. Agent image 205b is displayed in the intermediate mode position.

[0131] Figure 7(C) shows an example using a display device 250 equipped with a horizontally elongated display unit 251, but multiple display devices may be used to display agent images for each mode. For example, display device A may be installed at the positions of the normal mode and the apology mode, display device B at the position of the standby mode, and display device C at the position of the intermediate mode. In this case, it is possible to display agent image 205 on display device A, agent image 205a on display device B, and agent image 205b on display device C.

[0132] Figure 7(D) shows an example of displaying agent images 206, 206a, and 206b on the Head-Up Display (HUD) of vehicle C1. In this case, a HUD display device is installed at the top of the dashboard 2, near the boundary with the front windshield 4, to realize the HUD.

[0133] A HUD display device is a display device, such as a projector or optical system, that projects light onto the display area 4a of the front windshield 4 and uses the reflected light to show a virtual image to the user U1, thereby realizing a HUD display. In other words, the light projected from the HUD display device onto the display area 4a of the front windshield 4 is reflected by the front windshield 4, and this reflected light is directed towards the user U1's eyes. The reflected light projected onto the display area 4a and entering the user U1's eyes is displayed superimposed on the actual object visible through the front windshield 4. In this way, the HUD display device realizes a HUD display by displaying a virtual image using the front windshield 4.

[0134] For example, the HUD display device displays agent images 206, 206a, and 206b in the display area 4a based on the control of the information processing device 110 (see Figure 4). The front windshield 4 also functions as a display medium for the vehicle C1's HUD.

[0135] Agent image 206 is displayed in the normal mode and apology mode positions. Agent image 206a is displayed in the standby mode position. Agent image 206b is displayed in the intermediate mode position.

[0136] Figures 7(C) and 7(D) show an example of moving an agent image of the same size horizontally, but the system is not limited to this. For example, the standby mode may be realized by deleting the agent image or reducing its size to a smaller size than the normal mode, or the intermediate mode may be realized by reducing the size of the agent image to a smaller size than the normal mode. Furthermore, the standby mode and intermediate mode may be realized by increasing the transparency of the agent image to a higher level than the normal mode, or by decreasing the brightness of the agent image to a lower level than the normal mode. Additionally, the standby mode and intermediate mode may be realized by decreasing the saturation of the agent image to a lower level than the normal mode, or by decreasing the contrast of the agent image to a lower level than the normal mode. When transitioning the agent image to each mode, the movement trajectory of the agent image from the position of the original mode to the position of the new mode may be represented by animation processing or the like.

[0137] Furthermore, while Figures 7(A) to (B2) show examples of realizing each mode by physically moving the agent device or changing the agent device visible to user U1 to a different agent device, the system is not limited to these examples. For instance, the standby mode and intermediate mode may be realized by restricting the audio output from the agent device, or the standby mode and intermediate mode may be realized by keeping the agent device stationary.

[0138] Furthermore, while Figures 7(C) and 7(D) show examples of realizing each mode by moving the agent image, the system is not limited to this. For example, the standby mode and intermediate mode may be realized by restricting the audio output from the agent image, or the standby mode and intermediate mode may be realized by keeping the agent image stationary.

[0139] For example, if the agent device or agent image is flickering in the corner of user U1's field of vision, user U1 is more likely to notice it. However, if the agent device or agent image is completely still and makes no sound, user U1 is unlikely to notice it. Thus, by keeping the agent device or agent image still and making it silent, it is possible to achieve a state where it is visible to user U1, but user U1 does not notice it.

[0140] In this way, by having the agent device or agent image stop speaking to user U1 or become immobile, it is possible to put the agent device or agent image into a state where it does not attract user U1's attention. In other words, it is possible to put the agent device or agent image into a standby state.

[0141] Furthermore, the above-described embodiments may be combined with corresponding embodiments. For example, the standby and intermediate embodiments may be realized by moving the agent device to a location further away from the user U1 than in the normal embodiment, and by restricting the audio output from the agent device or keeping the agent device stationary.

[0142] [Example of changing the various modes of the agent device according to the user's discomfort level] Next, we will describe examples of transitions when the various modes of the agent device 200 are changed according to the discomfort state of user U1.

[0143] [Example of a transition when user dissatisfaction decreases after an apology from the agent device] Figure 8 shows an example of the transition of the agent device 200 when the user U1's level of discomfort decreases after an apology for the notification information. Specifically, it shows an example of the transition when the user U1's level of discomfort increases due to the output of notification information while waiting for a signal, so the agent device 200 apologizes to user U1, and the user U1's level of discomfort decreases after this apology.

[0144] Figure 8(A) shows a vehicle C1 stopped in front of traffic light SG1, which is illuminated red, on road R1. Figure 8(A) also shows an example where other vehicles C2 and C3, also stopped at a red light, are located in front of vehicle C1. A parent and child pedestrian W1 is assumed to be located outside road R1. User U1 is assumed to be moving their head from side to side, as indicated by arrow A21, to check the area around vehicle C1. In the example shown in Figure 8(A), based on instructions from the information processing device 110 (see Figure 4), an audio message S11, "Watch out for pedestrians," is output as notification information to inform the user of the presence of the parent and child pedestrian W1.

[0145] Figure 8(B) shows the state immediately after traffic light SG1 switches from red to green on road R1. Figure 8(B) also shows an example where vehicle C2 in front of vehicle C1 begins to move, and another vehicle C4 passes in the adjacent lane. Note that the parent and child pedestrians W1 on the outside of road R1 are waiting for the red light without approaching road D1.

[0146] Here, let's consider a scenario where the agent device 200 outputs notification information that is different from what user U1 intended. For example, suppose user U1 wanted to know as soon as possible that the traffic light SG1 had turned green, and the voice notification S11, "Watch out for pedestrians," was output. In this case, user U1 might think, "Why didn't you tell me it was green earlier? The car behind me started moving first!" and thus feel annoyed with the agent device 200.

[0147] Furthermore, consider a scenario where, for example, user U1 wanted to know the status of surrounding vehicles other than pedestrians, and the voice information S11 "Watch out for pedestrians" was output as notification information. In this case, user U1 might think, "A car just passed by on the road, shouldn't I be paying attention to that?" and it is expected that this would cause displeasure towards agent device 200. Thus, if notification information that differs from user U1's expectations is output from agent device 200, it is expected that user U1 will feel displeasure towards agent device 200. Note that in Figures 8(B) and (C), user U1's displeasure is indicated by an exclamation mark on the back of user U1's head.

[0148] Thus, if notification information that differs from what user U1 intended is output from the agent device 200, and user U1 experiences discomfort towards the agent device 200, it is conceivable that user U1's level of discomfort may exceed the first threshold TH3. In this case, the discomfort determination unit 121 can determine that user U1's level of discomfort has exceeded the first threshold TH3. The discomfort factor estimation unit 122 can also estimate that the object of user U1's discomfort is the agent device 200. Thus, if user U1's level of discomfort is above the first threshold TH3 and the object of that discomfort is the agent device 200, the agent device 200 transitions to an apology mode and performs an apology to user U1. An example of this is shown in Figure 8(C).

[0149] Figure 8(C) shows an example of conveying an apology to user U1, who is feeling uncomfortable, by outputting voice information S12. For example, agent device 200 outputs voice information S12, "I'm sorry, did I do something wrong?", and also conveys an apology to user U1 by making predetermined gestures such as a sweaty, anxious expression, a downcast expression, or an upward glance at user U1. In this case, it is also possible to further emphasize the anxious attitude by, for example, shaking the main unit 201 in the forward and backward direction (arrow A22 direction).

[0150] In this embodiment, an apology is conveyed by having the agent device 200 perform predetermined actions. The predetermined actions refer to a form of apology to user U1 through at least one of the following: outputting a message from the agent device 200 (including speech and display) or the actions of the agent device 200 (including facial expressions).

[0151] As shown in Figure 8(C), it is conceivable that the user U1's level of discomfort may decrease as a result of the agent device 200 apologizing. For example, it is conceivable that the user U1's level of discomfort may fall below the first threshold TH3. In this case, the discomfort determination unit 121 can determine that the user U1's level of discomfort has fallen below the first threshold TH3. Thus, when the user U1's level of discomfort falls below the first threshold TH3, the agent device 200 transitions to normal mode and continues to execute normal processing. An example of this is shown in Figure 8(D1).

[0152] Figure 8(D1) shows an example where user U1, whose discomfort has been resolved by an apology from agent device 200, speaks voice S13 to agent device 200. When the discomfort is resolved by an apology from agent device 200, it is conceivable that user U1 might say to agent device 200, for example, "It's not your fault" or "Well, I don't mind." In response, agent device 200 will make a gesture of relief and then continue with normal communication.

[0153] On the other hand, as shown in Figure 8(C), even if the agent device 200 apologizes, it is conceivable that the user U1's level of discomfort may not decrease. For example, it is conceivable that the user U1's level of discomfort may remain at or above the first threshold TH3. In this case, the discomfort determination unit 121 can determine that the user U1's level of discomfort is at or above the first threshold TH3. Thus, if the user U1's level of discomfort remains at or above the first threshold TH3, the agent device 200 transitions to standby mode. An example of this is shown in Figure 9(D2).

[0154] [Example of a transition when user dissatisfaction does not decrease after an apology from the agent device] Figure 9 shows an example of the transition of the agent device 200 when the user U1's level of discomfort does not decrease after an apology for the notification information. Specifically, as shown in Figure 8(C), it shows an example of the transition when the agent device 200 apologizes to user U1 for the output of notification information while in a signal waiting state, but the user U1's level of discomfort does not decrease after this apology.

[0155] Figure 9(D2) shows an example where user U1, whose discomfort is not resolved by an apology from agent device 200, makes a voice command S21 to agent device 200. If the discomfort is not resolved by an apology from agent device 200, it is conceivable that user U1 might turn towards agent device 200 and say something like, "Watch out!" In this example, it is assumed that user U1's level of discomfort is above the first threshold TH3. In this case, agent device 200 transitions to standby mode. Specifically, agent device 200 moves to a position where it is not in user U1's line of sight. This example is shown in Figure 9(E).

[0156] Figure 9(E) shows an example where the agent device 200 is moved to a position out of the user U1's line of sight. Specifically, as shown in Figure 9(D2), the agent device 200, which was positioned near the center of the dashboard 2 (near the boundary between the passenger seat and the driver's seat), moves in the direction of arrow A31. The agent device 200 may be moved along the rail RL1 as shown in Figure 7(A), or it may move autonomously on the dashboard 2. Alternatively, the agent device 200 may be moved by other means of movement.

[0157] Furthermore, when transitioning to standby mode, that is, when disappearing from user U1's line of sight, it is preferable to move the agent device 200 to the standby position while maintaining the same facial expression as immediately before transitioning to standby mode. In other words, when transitioning to standby mode, it is preferable to make the presence of the agent device 200 disappear instantaneously. By making the presence of the agent device 200 disappear instantaneously in this way, it is possible to transition the agent device 200 to standby mode without attracting the attention of user U1. On the other hand, if the presence of the agent device 200 disappears gradually, it may attract the attention of user U1, potentially increasing user U1's discomfort by making them wonder what the agent device 200 is doing.

[0158] However, if, for example, user U1 says "Watch out!", the agent device 200 may be moved while displaying a predetermined gesture such as a downcast expression, a slightly dejected expression after being scolded, or a downward-looking expression.

[0159] For example, it is conceivable that an apology from agent device 200 may not be effective. In such cases, if agent device 200, which is the source of user U1's discomfort, takes any action in response to user U1's discomfort, it may further increase user U1's level of discomfort. Therefore, when an apology from agent device 200 is deemed ineffective, agent device 200 is temporarily removed from user U1's field of vision, and then gradually reappears after user U1's discomfort subsides. Alternatively, instead of moving agent device 200, as described above, its facial expression may be removed, its movement may be stopped, or agent device 200 may cease to speak, making it difficult for user U1 to notice its presence.

[0160] As shown in Figure 9(E), by moving the agent device 200 to a position where it is out of the user U1's line of sight, it is conceivable that the user U1's interest in the agent device 200 will decrease, and the user U1's level of discomfort with the agent device 200 will decrease. For example, it is conceivable that the user U1's level of discomfort will fall below the second threshold (but above the third threshold). In this case, the discomfort determination unit 121 can determine that the user U1's level of discomfort is below the second threshold. Thus, when the user U1's level of discomfort falls below the second threshold, the agent device 200 transitions to an intermediate state. An example of this is shown in Figure 9(F).

[0161] Here, the second threshold is a threshold for determining whether or not to transition the agent device 200 from standby mode to intermediate mode, and is set to a value smaller than the first threshold. The second threshold may be a fixed value or a variable value. When a fixed value is used for the second threshold, it can be set appropriately based on experimental data, etc. When a variable value is used for the second threshold, it can be set based on the level of discomfort of user U1 in a calm state. For example, it is possible to set the second threshold based on the average value of user U1's discomfort level during a predetermined period prior to the determination that user U1's discomfort level has reached or exceeded the first threshold TH3. The average value refers to the value when user U1 was in a normal state during that predetermined period. For example, this average value can be set as the second threshold. Alternatively, instead of using the determination that user U1's discomfort level has reached or exceeded the first threshold TH3 as the basis, the timing of the agent device 200 transitioning to the apology mode, or the timing of the agent device 200 outputting notification information, may be used as the basis. Furthermore, the specified period can be set to a period of, for example, several minutes to several tens of minutes.

[0162] Figure 9(F) shows an example where the agent device 200 moves to a position that is in the corner of user U1's field of vision. Specifically, after the agent device 200 moves to the position shown in Figure 9(E), if user U1's level of discomfort drops below the second threshold, the agent device 200 moves from the position shown in Figure 9(E) in the direction of arrow A32. However, it moves to an intermediate position, not the normal position. In this way, when user U1's level of discomfort decreases to a certain extent, the agent device 200 gradually appears from a distance away from user U1. Also, as shown in Figure 6(B), the agent device 200 may move gradually or gradually speak to user U1. Furthermore, the agent device 200's facial expression may be one of impatience.

[0163] As shown in Figure 9(F), it is conceivable that user U1's interest in the agent device 200 may increase as the agent device 200 moves to a position that is in the corner of user U1's field of vision. In this case, for example, if the standby state continues for a long time, it is conceivable that user U1 may forget the discomfort with the agent device 200, and user U1's level of discomfort may decrease. For example, it is conceivable that user U1's level of discomfort may fall below the third threshold. In this case, the discomfort determination unit 121 can determine that user U1's level of discomfort is below the third threshold. Thus, when user U1's level of discomfort falls below the third threshold, the agent device 200 transitions to the normal state. An example of this is shown in Figure 9(G).

[0164] Here, the third threshold is a threshold for determining whether or not to transition the agent device 200 from the intermediate state to the normal state, and is set to a value smaller than the second threshold. The third threshold may be a fixed value or a variable value. When a fixed value is used for the third threshold, it can be set appropriately based on experimental data, etc. When a variable value is used for the third threshold, it can be set based on the level of discomfort of user U1 in a calm state. For example, it is possible to set the third threshold based on the lowest value of user U1's discomfort within a predetermined period prior to the time when it is determined that user U1's discomfort level has reached or exceeded the first threshold TH3. The lowest value refers to the value when user U1 was in the calmest state within that predetermined period. For example, this lowest value can be set as the third threshold. Alternatively, instead of using the time when user U1's discomfort level has reached or exceeded the first threshold TH3 as the basis, the timing of the agent device 200 transitioning to the apology state, or the timing when the agent device 200 outputs notification information, may be used as the basis. Furthermore, the specified period can be set to a period of, for example, several minutes to several tens of minutes.

[0165] Here, while user U1 is driving, there are fluctuations in emotions, and the level of discomfort changes sequentially. Therefore, the average and minimum values ​​within a predetermined period prior to the timing when the agent device 200 transitions to an apologetic state, the timing when the agent device 200 outputs notification information, etc., are used to determine whether or not to transition the agent device 200 back to the normal state. This makes it possible to appropriately determine whether or not user U1 has become calm.

[0166] Thus, in this embodiment, the third threshold for transitioning the agent device 200 from the intermediate state to the normal state is set lower than the second threshold for transitioning the agent device 200 from the standby state to the intermediate state. This makes it possible to set an appropriate timing for returning the agent device 200 to the normal state. In this way, when the user U1's level of discomfort decreases to a normal level, the agent device 200 is returned from the intermediate state to the normal state. When returning the agent device 200 from the intermediate state to the normal state, the agent device 200 may gradually speak to the user U1, or it may gradually move or make various facial expressions.

[0167] Figure 9(G) shows an example of when the agent device 200 moves to the normal position. Specifically, if the user U1's level of discomfort decreases to below the third threshold after the agent device 200 has moved to the position shown in Figure 9(F), the agent device 200 moves from the position shown in Figure 9(F) in the direction of arrow A33. This returns the agent device 200 to the normal position. In this way, when the user U1's discomfort is resolved, the agent device 200 is returned to the normal position. Alternatively, as shown in Figure 6(B), the movement of the agent device 200 may be made larger, or the user U1's speaking frequency may be increased. However, it is preferable that the agent device 200's facial expression be one of remorse during the process of returning the agent device 200 to the normal position, or immediately after returning it to that position. This remorseful expression could be, for example, a downcast gaze or an anxious expression. This makes it possible to make the user U1 feel that the agent is reflecting on its past actions. The expression on the agent device 200's face may be changed from a remorseful expression to a relieved expression. An example of this transition to a relieved expression is shown in Figure 9(H).

[0168] In this embodiment, if user U1 shows displeasure at the agent device 200's apology, a predetermined response method is executed. Specifically, if the agent device 200's apology is deemed ineffective and user U1's anger has not subsided, the agent device 200 is temporarily removed from the system. That is, the agent device 200 is transitioned from an apologetic state to a standby state. It is also assumed that as user U1's anger subsides and they gradually calm down, user U1 will become more composed and their level of discomfort will decrease. In this case, the agent device 200 is gradually brought back into the system in accordance with the decrease in user U1's level of discomfort. In this case, the agent device 200 may be brought back into the system gradually while behaving in a way that suggests it is concerned about user U1. In this way, the agent device 200 is sequentially transitioned from a standby state to an intermediate state and then to a normal state.

[0169] [Example of information processing device operation] Figure 10 is a flowchart illustrating an example of control processing in the information processing device 110. This control processing is executed by the control unit 120 based on a program stored in the memory unit 130. This control processing is also executed at the timing when the agent device 200 outputs notification information. This control processing will be explained with reference to Figures 1 to 9 as appropriate.

[0170] In step S501, the agent control unit 124 performs output processing to output notification information to user U1 from the agent device 200.

[0171] Here, the timing for outputting notification information is determined based on the results of an analysis process that analyzes input information such as information input to the information processing device 110 from the camera 101, position information acquisition sensor 102, voice input unit 103, sensors 104, etc., and information acquired from the outside via the communication unit 140 and input to the information processing device 110. For example, the agent control unit 124 determines whether or not there is information to be notified to the occupants of vehicle C1 (notification information) based on the results of the input information analysis.

[0172] For example, the battery level of vehicle C1 can be detected based on the battery sensor. If it is determined that the battery level falls below a threshold, it is determined that there is notification information available to alert the vehicle that the battery level is low.

[0173] For example, based on a seat belt sensor that detects whether a seat belt is being worn and a seat occupancy sensor (or seat sensor) that detects the presence or absence of an occupant in each seat of vehicle C1, it is possible to determine whether an occupant in each seat of vehicle C1 is wearing a seat belt. If it is determined that an occupant is seated but not wearing a seat belt, it is determined that there is notification information available for that occupant.

[0174] Furthermore, for example, a partially closed door on vehicle C1 can be detected based on a partially closed door sensor that detects each door of vehicle C1 being partially closed. If a partially closed door is detected, it is determined that there is notification information for the partially closed door.

[0175] Furthermore, for example, based on information obtained from an external device, such as an information provision server, via the communication unit 140, and the current location of vehicle C1 obtained by the location information acquisition sensor 102, it is possible to determine whether there are any landmarks, commercial facilities, etc., around vehicle C1. For example, if it is determined that there is a commercial facility, such as a coffee shop, around vehicle C1, it is determined that there is information available as notification information about that coffee shop. Also, for example, if it is determined that there is a tourist facility, such as ABC Castle, around vehicle C1, it is determined that there is information available as notification information about ABC Castle.

[0176] In step S502, the control unit 120 performs sensing of the surrounding conditions of the vehicle C1 and the user U1. The control unit 120 then calculates the confidence level of the sensing. Specifically, the confidence level determination unit 123 calculates the confidence level of the sensing based on the information obtained from each sensor. For example, as shown in Figure 5, it is possible to calculate the confidence level of the sensing based on sensing stability, CPU utilization rate, etc.

[0177] In step S503, the confidence determination unit 123 determines whether the confidence level of the sensing output in step S502 is higher than a standard (for example, threshold TH1 (see Figure 5(B)) or threshold TH2 (see Figure 5(C))). If the confidence level of the sensing is higher than the standard, the process proceeds to step S504. On the other hand, if the confidence level of the sensing is lower than the standard, the control process terminates.

[0178] Thus, when the confidence level of the sensing is high, it is assumed that the confidence level of each process of the control unit 120 based on that sensing is also high. On the other hand, when the confidence level of the sensing is low, it is assumed that the confidence level of each process of the control unit 120 based on that sensing is also low. Therefore, when the confidence level of the sensing is high, it becomes possible to appropriately determine the discomfort state of user U1, so the process proceeds to step S504 and control processing is executed to control the behavior of the agent device 200. On the other hand, when the confidence level of the sensing is low, it becomes difficult to appropriately determine the discomfort state of user U1, so control processing to control the behavior of the agent device 200 is not executed. In this way, by deciding whether or not to perform control processing to control the behavior of the agent device 200 based on the confidence level of the sensing, it is possible to branch off the risk of the agent device 200 taking actions that user U1 cannot understand.

[0179] In step S504, the discomfort determination unit 121 estimates the discomfort level of user U1 based on the information acquired from each sensor. For example, as shown in Figure 6(A), it is possible to calculate the discomfort level of user U1 based on user U1's biometric information (e.g., heart rate, blood flow), sounds emitted by user U1, and user U1's appearance (e.g., facial expressions, movements). The discomfort determination unit 121 continues the discomfort level estimation process.

[0180] In step S505, the discomfort determination unit 121 determines whether the discomfort level of user U1 estimated in step S503 is equal to or greater than the first threshold TH3. If the discomfort level of user U1 is equal to or greater than the first threshold TH3, the process proceeds to step S506. On the other hand, if the discomfort level of user U1 is less than the first threshold TH3, the control process terminates. Alternatively, the process may be configured to proceed to step S506 only if the state in which the discomfort level of user U1 is equal to or greater than the first threshold TH3 continues for a predetermined time (for example, several seconds to several minutes). This makes it possible to prevent frequent transitions to the apology mode and the waiting mode, which would damage the reliability of user U1.

[0181] As explained in Figure 6(A), the first threshold TH3 may be changed to a lower value, and the first threshold TH3' may be set. In this case, when the first threshold TH3' is set, the determination process in step S505 should use the first threshold TH3'. The same applies to the determination process in step S509.

[0182] In step S506, the discomfort factor estimation unit 122 estimates the discomfort factors of user U1 based on the information obtained from each sensor. For example, as described above, it is possible to estimate the discomfort factors of user U1.

[0183] In step S507, the discomfort factor estimation unit 122 determines whether the object of user U1's discomfort is the agent device 200. That is, the discomfort factor estimation unit 122 determines whether the object of user U1's discomfort estimated in step S506 is the agent device 200 or something else. If the object of user U1's discomfort is the agent device 200, the process proceeds to step S508. On the other hand, if the object of user U1's discomfort is something other than the agent device 200, causing the agent device 200 to perform an apology may result in an apology that user U1 cannot understand. Therefore, if the object of user U1's discomfort is something other than the agent device 200, the control process is terminated.

[0184] In step S508, the agent control unit 125 causes the agent device 200 to transition to an apology mode. Specifically, the agent control unit 125 causes the agent device 200 to perform an apology sequence to apologize for making user U1 uncomfortable. For example, as shown in Figures 2 and 8(C), it is possible to have the agent device 200 apologize to user U1 for making user U1 uncomfortable through voice output and predetermined gestures.

[0185] In step S509, the discomfort determination unit 121 determines whether user U1 has allowed the agent device 200. Specifically, the discomfort determination unit 121 determines whether user U1's level of discomfort is less than the first threshold TH3. If user U1's level of discomfort is less than the first threshold TH3, it determines that user U1 has allowed the agent device 200 and proceeds to step S510. On the other hand, if user U1's level of discomfort is equal to or greater than the first threshold TH3, it determines that user U1 has not allowed the agent device 200 and proceeds to step S511. Here, it is assumed that some people will resolve their discomfort quickly, while others will experience prolonged discomfort. Therefore, the determination process in step S509 may be performed after a predetermined time (for example, several seconds to several minutes) has elapsed. This makes it possible to prevent frequent transitions to the standby mode from undermining the reliability of user U1.

[0186] In step S510, the agent control unit 125 causes the agent device 200 to perform a relief animation and transition the agent device 200 back to its normal state. If the user proceeds to step S510 via steps S511, S514, etc., the agent device 200 is positioned in the corner of the user U1's field of vision. Therefore, if the user proceeds to step S510 via steps S511, S514, etc., the agent device 200 is returned to its normal position before the relief animation is performed. If the user proceeds to step S510 without going through steps S511, S514, etc., the agent device 200 is positioned in its normal position and has performed an apology animation. Therefore, if the user proceeds to step S510 without going through steps S511, S514, etc., the position of the agent device 200 is not changed, and the apology animation is completed before the relief animation is performed.

[0187] In step S511, the agent control unit 125 transitions the agent device 200 to standby mode. Specifically, the agent control unit 125 moves the agent device 200 to a position where it is not in the user U1's line of sight. For example, as shown in Figure 9(E), the agent device 200 is moved in the direction of arrow A31 and positioned near the A-pillar on the passenger side of vehicle C1.

[0188] In step S512, the discomfort determination unit 121 determines whether the discomfort level of user U1 is below the second threshold. If the discomfort level of user U1 is below the second threshold, the process proceeds to step S514. On the other hand, if the discomfort level of user U1 is equal to or greater than the second threshold, the process proceeds to step S513. Since it is estimated in step S507 that the cause of discomfort for user U1 is the agent device 200, the discomfort level of user U1 that is to be determined in step S512 is estimated to be the discomfort level towards the agent device 200. The same applies to step S515.

[0189] In step S513, the agent control unit 125 maintains the standby state of the agent device 200. Specifically, the agent control unit 125 places the agent device 200 in a position that is not in the line of sight of user U1. For example, the agent device 200 is placed in the position shown in Figure 9(E) and placed in standby mode.

[0190] In step S514, the agent control unit 125 transitions the agent device 200 to an intermediate state. Specifically, the agent control unit 125 makes the agent device 200 appear in the corner of the user U1's field of view. For example, as shown in Figure 9(F), the agent device 200 is moved in the direction of arrow A32 and positioned so that it is in the corner of the user U1's field of view.

[0191] In step S515, the discomfort determination unit 121 determines whether the discomfort level of user U1 is less than the third threshold. If the discomfort level of user U1 is less than the third threshold, the process proceeds to step S510. On the other hand, if the discomfort level of user U1 is equal to or greater than the third threshold, the process proceeds to step S516.

[0192] In step S516, the agent control unit 125 maintains the intermediate configuration of the agent device 200. Specifically, the agent control unit 125 places the agent device 200 on standby at a location in the corner of the user U1's field of vision. For example, the agent device 200 is placed on standby at the location shown in Figure 9(F).

[0193] Here, it is conceivable that user U1's level of discomfort temporarily falls below the second threshold, but then rises above the second threshold before falling below the third threshold. Therefore, in step S515, it is determined that user U1's level of discomfort is above the third threshold, and after the intermediate state is maintained in step S516, the process proceeds to step S512, where it is determined again whether user U1's level of discomfort is below the second threshold. This makes it possible to realize an appropriate mode for the agent device 200 according to user U1's discomfort state.

[0194] [Examples of agent devices that can be installed outside of vehicles] The above describes examples of agent equipment 200 installed in vehicle C1 and agent images displayed in vehicle C1. However, this embodiment can also be applied to agent equipment that can be removed from vehicle C1, agent equipment that can be installed outside of a vehicle, etc. For example, it is conceivable that a user who possesses a portable agent device may place the agent device on the dashboard 2 of vehicle C1 when getting into vehicle C1, and then carry the agent device with them when getting out of vehicle C1. It is also conceivable that a user may use the agent device at home. Furthermore, it is conceivable that various devices capable of displaying agent images (corresponding to agent equipment 200) (for example, game consoles, glasses-type displays, AR (Augmented Reality) glasses) may be used.

[0195] For example, within a home, it is conceivable that notification information could be output regarding various appliances installed in the home. For instance, it is envisioned that notifications could be issued when the bath is ready or when cooking is finished. It is also envisioned that notifications could be issued if the front door or windows are open. Furthermore, it is envisioned that notifications could be issued if the gas stove is left on. In these cases, it is envisioned that notification information regarding the bath, cooking appliances, doors, windows, etc., would be provided.

[0196] For example, if user U1 becomes uncomfortable with the notification information provided to user U1, or the manner in which the notification is presented, the system will transition to an appropriate mode such as an apology mode, a waiting mode, or an intermediate mode, based on the degree of discomfort experienced by user U1. The degree of discomfort experienced by user U1 can be calculated, as in the example described above, based on user U1's biometric information (e.g., heart rate, blood flow), sounds emitted by user U1, and user U1's appearance (e.g., facial expressions, movements). The apology mode, waiting mode, and intermediate mode are similar to those shown in the examples in Figures 8 and 9.

[0197] For example, consider an agent device equipped with movable wheels at its base, which can be configured as a standby mode and an intermediate mode. In this case, the agent device can be configured as a standby mode and an intermediate mode by moving its position away from the user. For example, if an agent device that can move along rails is configured as a standby mode and an intermediate mode, the agent device can be configured as a standby mode and an intermediate mode by moving the agent device away from the user. Also, for example, if an autonomously moving agent device is configured as a standby mode and an intermediate mode, the agent device can be configured as a standby mode and an intermediate mode by moving the agent device away from the user.

[0198] Furthermore, consider a case where an agent image is displayed using a device capable of displaying the agent image (e.g., a game console, glasses-type display, AR glasses), and the agent image is set to a standby state and an intermediate state. In this case, similar to the examples shown in Figures 7(C) and 7(D), the agent image can be set to a standby state and an intermediate state by moving the position of the agent image away from the user. Alternatively, the standby state can be achieved by gradually fading and erasing the agent image. In this case, the intermediate state and the normal state can be achieved by gradually darkening and displaying the agent image. As mentioned above, each state may also be achieved by changing the transparency, brightness, and saturation of the agent image, or by restricting audio output from the agent image, or by keeping the agent image still, etc. In these cases, the device capable of displaying the agent image corresponds to the agent device 200.

[0199] [Examples of the effects of this embodiment] Here, it is important to apologize promptly and at an appropriate time for any errors made by agent device 200, as this is crucial for maintaining a good relationship with user U1. In other words, instead of doing nothing when agent device 200 makes a mistake, it is important to take some kind of follow-up action quickly and at an appropriate time. This will allow the bond between user U1 and agent device 200 to deepen even if some kind of error occurs in agent device 200. Furthermore, if some kind of error occurs in agent device 200 and causes displeasure to user U1, it will be possible to quickly reduce that displeasure.

[0200] Therefore, in this embodiment, if the output of notification information, or the manner of notification, causes discomfort to user U1, the agent device 200 will issue an apology at an appropriate time. This appropriate time is preferably as early as possible.

[0201] However, even if agent device 200 apologizes to user U1, user U1 may still feel angry towards agent device 200 and remain in an unpleasant state. In this state, if agent device 200 resumes any communication with user U1, it may increase user U1's level of displeasure and risk causing a loss of trust in user U1.

[0202] Therefore, in this embodiment, if user U1 shows an unpleasant reaction to the agent device 200's apology, the agent device 200 is put into a state that is difficult for user U1 to notice. Then, after user U1's unpleasant state subsides, the agent device 200 is gradually brought back into play. Specifically, if user U1's level of unpleasantness remains above the first threshold TH3 even after the agent device 200's apology, the agent device 200 is transitioned from the apologetic state to the standby state. After the agent device 200 has transitioned to the standby state, if user U1's level of unpleasantness is above the third threshold but below the second threshold, the agent device 200 is transitioned from the standby state to the intermediate state. Furthermore, after the agent device 200 has transitioned to the intermediate state, if user U1's level of unpleasantness falls below the third threshold, the agent device 200 is transitioned from the intermediate state to the normal state. In this case, it is preferable to have the agent device 200 perform a gesture of relief after transitioning to the normal state.

[0203] Thus, if user U1's discomfort persists after the agent device 200's apology, the agent device 200 is switched to a standby mode that is less likely to be noticed by user U1 until user U1's discomfort subsides. In this standby mode, the movement of the agent device 200 is reduced, making it possible to reduce power consumption and reduce the computational processing of the control unit 120, etc. Then, as user U1's discomfort gradually subsides, the system is sequentially switched back to the standby mode, intermediate mode, and normal mode. In other words, as user U1's discomfort subsides, the agent device 200 is gradually brought back into service. In this way, this embodiment makes it possible to reduce the persistence of user U1's discomfort after the agent device 200's apology and to increase user U1's trust in the system.

[0204] Thus, in this embodiment, when outputting notification information or apologizing for the manner of notification, the apology is made promptly and at an appropriate time, thereby preventing further discomfort to user U1 and preventing a loss of user U1's trust. Furthermore, considering the user's discomfort after the apology, the system appropriately determines and executes the normal mode of showing relief, the waiting mode, and the intermediate mode. Therefore, it is possible to prevent further discomfort to user U1 and prevent a loss of user U1's trust.

[0205] Thus, in this embodiment, the agent device 200 can take actions that indicate it values ​​its relationship with user U1. This deepens the bond between user U1 and the agent device 200, and enables user U1 to appropriately receive notification information from the agent device 200.

[0206] [Example of executing processing on other devices or systems] In the above, we have shown an example in which control processing and the like are performed in the agent device 200 and the information processing device 110 (or information processing system 100), but all or part of each of these processes may be performed in other devices. In this case, the information processing system is composed of each device that performs part of each of these processes. For example, at least part of each process can be performed using various information processing devices and various electronic devices such as in-vehicle devices, user-accessible devices (e.g., smartphones, tablet terminals, personal computers, car navigation systems, IVI), and servers that can be connected via a predetermined network such as the Internet.

[0207] Furthermore, some (or all) of the information processing system capable of executing the functions of the information processing device 110 (or information processing system 100) may be provided by an application that can be provided via a predetermined network such as the Internet. This application may be, for example, SaaS (Software as a Service).

[0208] [Example configuration and effects of this embodiment] The information processing method according to this embodiment is an information processing method for controlling an agent device 200 capable of communicating with user U1. This information processing method includes an unpleasantness estimation process (step S504) that estimates the degree of discomfort of user U1 based on at least one of the following: the user U1's biometric information (e.g., heart rate, body temperature, facial surface temperature, blood flow, information related to the degree of excitement), the user U1's speech content (e.g., predetermined keywords such as "you bastard"), and the user U1's actions (e.g., magnitude, speed of body movements, facial expressions), and at least one of the following modes: a normal mode that provides notification information to user U1, an apology mode that shows an apology to user U1, and a standby mode that is less likely to attract user U1's attention than the normal mode, and the agent device 200 is controlled by the agent device 200. The system includes control processing (steps S505 to S516) to change the mode of the device 200, and in the control processing (steps S505 to S511), after providing notification information to user U1, if a predetermined condition based on user U1's level of discomfort is met (for example, if user U1's level of discomfort becomes equal to or greater than the first threshold TH3), the agent device 200 is transitioned from normal mode to apologetic mode. After the agent device 200 is transitioned to apologetic mode, if user U1's level of discomfort does not become less than the first threshold TH3 (an example of the first criterion), the agent device 200 is transitioned from apologetic mode to standby mode. For example, as shown in Figure 8(C), the agent device 200 is transitioned from normal mode to apologetic mode. Also, for example, as shown in Figure 9(E), the agent device 200 is transitioned from apologetic mode to standby mode. Furthermore, the program according to this embodiment is a program that causes a computer to execute each of these processes. In other words, the program according to this embodiment is a program that causes a computer to realize each of the functions that the information processing device 110 can execute.

[0209] With this configuration, if user U1's discomfort persists after agent device 200's apology, agent device 200 can be switched to a standby mode that is less likely to be noticed by user U1. This reduces the persistence of user U1's discomfort after agent device 200's apology and increases user U1's trust in the system.

[0210] In the information processing method according to this embodiment, the agent device 200 is a movable device, and the standby mode includes at least one of the following: a first mode in which the agent device 200 is moved to a first position farther from the user U1 than in the normal mode; a second mode in which the agent device 200 is moved to a second position outside the user U1's field of view; a third mode in which the audio output from the agent device 200 is restricted; and a fourth mode in which the agent device 200 is kept stationary.

[0211] With this configuration, by transitioning the agent device 200 to at least one of the first to fourth modes, it is possible to realize a standby mode in which the agent device 200 is less likely to be noticed by the user U1.

[0212] In the information processing method according to this embodiment, the agent device is a display device 250 (an example of a device) that displays agent images 205, 205a, and 205b for communication with user U1 on a display unit 251, and the standby mode includes at least one of the following: a first mode in which agent image 205a is displayed on the display unit 251 at a position further from user U1 than in the normal mode; a second mode in which the transparency of agent image 205a on the display unit 251 is made higher than in the normal mode; a third mode in which the brightness of agent image 205a on the display unit 251 is made lower than in the normal mode; a fourth mode in which the saturation of agent image 205a on the display unit 251 is made lower than in the normal mode; a fifth mode in which audio output from agent image 205a is restricted; and a sixth mode in which agent image 205a on the display unit 251 is kept in a static state. The same can also be done when agent images 206, 206a, and 206b are displayed in the display area 4a using a HUD display device, as shown in Figure 7(D).

[0213] With this configuration, by transitioning the agent image to at least one of the first to sixth embodiments, it is possible to realize a standby mode in which the agent image is less likely to be noticed by user U1.

[0214] In the information processing method according to this embodiment, in the control processing (steps S511 to S516), after transitioning the agent device 200 to standby mode, if the user U1's level of discomfort falls below a third threshold (an example of a second criterion), which is a value smaller than the first threshold TH3 (an example of a first criterion), the agent device 200 is transitioned from standby mode to normal mode. For example, as shown in Figure 9(G), the agent device 200 is transitioned to normal mode.

[0215] With this configuration, after the agent device 200 apologizes and transitions to standby mode, if user U1's discomfort subsides, it can be returned to normal mode. In other words, by not returning to normal mode until user U1's discomfort subsides, and only returning the agent device 200 to normal mode once user U1's discomfort subsides, it is possible to increase user U1's trust.

[0216] In the information processing method according to this embodiment, in the control processing (steps S511 to S516), when transitioning the agent device 200 from standby mode to normal mode, the transition is made via an intermediate mode, which is more likely to attract the user U1's attention than the standby mode, to the normal mode. For example, as shown in Figures 9(F)(G), the agent device 200 is transitioned to the intermediate mode, and then transitioned to the normal mode.

[0217] With this configuration, after the agent device 200 apologizes, it can be switched to standby mode, and then gradually returned to normal mode as user U1's discomfort subsides. This makes it possible to increase user U1's trust.

[0218] In the information processing method according to this embodiment, in the control processing (steps S511 to S516), after transitioning the agent device 200 to standby mode, if the user U1's level of discomfort becomes less than a second threshold (example of a third criterion), which is a value smaller than a first threshold TH3 (example of a first criterion) and larger than a third threshold (example of a second criterion), the agent device 200 is transitioned from standby mode to intermediate mode. After transitioning the agent device 200 to intermediate mode, if the user U1's level of discomfort becomes less than the third threshold, the agent device 200 is transitioned from intermediate mode to normal mode. For example, as shown in Figures 9(F)(G), after transitioning the agent device 200 to intermediate mode, the agent device 200 is transitioned to normal mode.

[0219] This configuration makes it possible to sequentially execute transitions from standby mode to intermediate mode and from intermediate mode to normal mode at appropriate timings using appropriate criteria.

[0220] In the information processing method according to this embodiment, the third threshold (an example of the second criterion) is set based on the lowest value of the user U1's discomfort level within a predetermined period prior to the time of determining the level of discomfort when the agent device 200 transitions to an apologetic state. For example, that lowest value can be set as the third threshold.

[0221] This configuration makes it possible to set an appropriate third threshold based on the user U1's normal level of discomfort, and to sequentially execute the transition from standby mode to normal mode at the appropriate timing.

[0222] In the information processing method according to this embodiment, the second threshold (an example of a third criterion) is set based on the average value of the user U1's level of discomfort during a predetermined period prior to the time of determination of the level of discomfort when the agent device 200 transitions to an apologetic state. For example, the average value can be used as the second threshold.

[0223] This configuration makes it possible to set an appropriate second threshold based on the user U1's normal level of discomfort, and to execute the transition from standby mode to intermediate mode at the appropriate timing.

[0224] In the information processing method according to this embodiment, in the control processing (steps S508 to S516), after transitioning the agent device 200 to an apologetic state, if the user U1's level of discomfort falls below a first threshold TH3 (an example of a first criterion) within a predetermined period, the agent device 200 is transitioned from an apologetic state to a normal state, and the agent device 200 is instructed to perform a predetermined action showing a relieved gesture in that normal state. For example, as shown in Figure 8(D1), after transitioning the agent device 200 to a normal state, the agent device 200 is instructed to show an expression of relief.

[0225] With this configuration, if user U1's discomfort has been resolved immediately after the agent device 200's apology, the agent device 200 can be quickly returned to its normal state, and in that normal state, it can project an air of reassurance. In this way, because the agent device 200 can be quickly returned to its normal state after the apology, it is possible to increase user U1's trust in it.

[0226] In the information processing method according to this embodiment, in the control process, if the transition to the apology mode occurs a predetermined number of times or more within a predetermined period, the first threshold TH3 (an example of the first criterion) is changed to a smaller value. The second predetermined period may be, for example, the period from when user U1 boards vehicle C1 until when they disembark, or it may be a predetermined time (for example, several minutes to several tens of minutes, several hours, or one day). The first threshold TH3 (first threshold TH3') that has been changed to a smaller value is used in the determination processes of steps S505 and S509. When the first threshold TH3 is changed to a smaller value, at least one of the second threshold (an example of the third criterion) and the third threshold (an example of the second criterion) may also be changed to a smaller value in accordance with that change.

[0227] With this configuration, if the agent device 200 repeatedly apologizes incorrectly within a predetermined period, it becomes easier to transition to standby mode and more difficult to return to normal mode. This prevents frequent repetition of transitions between normal mode and standby mode. Therefore, it is possible to prevent unpleasant situations for user U1 and increase user U1's trust.

[0228] The information processing device 110 is an information processing device that controls an agent device 200 capable of communicating with user U1. The information processing device 110 includes a discomfort determination unit 121 (an example of a discomfort estimation unit) that estimates the degree of discomfort of user U1 based on at least one of the following: the user U1's biometric information (e.g., heart rate, body temperature, facial surface temperature, blood flow, and information related to the degree of excitement), the user U1's speech content (e.g., predetermined keywords such as "you bastard"), and the user U1's actions (e.g., magnitude, speed, and facial expression of body movements); and an agent control unit 124 (an example of a control unit) that changes the mode of the agent device 200 to at least one of the following: a normal mode that provides notification information to user U1, an apology mode that shows an apology to user U1, and a standby mode that is less likely to attract user U1's attention than the normal mode. After providing notification information to user U1, the agent control unit 124 transitions the agent device 200 from normal mode to apologetic mode if a predetermined condition based on user U1's level of discomfort is met (for example, if user U1's level of discomfort becomes equal to or greater than the first threshold TH3). After transitioning the agent device 200 to apologetic mode, if user U1's level of discomfort does not fall below the first threshold TH3 (an example of the first criterion), the agent device 200 transitions from apologetic mode to standby mode. For example, as shown in Figure 9(E), the agent device 200 is transitioned from normal mode to apologetic mode. Alternatively, as shown in Figure 9(F), the agent device 200 is transitioned from apologetic mode to standby mode. The information processing device 110 may be a device built into the agent device 200, or it may be a device different from the agent device 200. Alternatively, instead of the information processing device 110, an information processing system consisting of multiple devices capable of executing each process realized by the information processing device 110 may be used.

[0229] With this configuration, if user U1's discomfort persists after agent device 200's apology, agent device 200 can be switched to a standby mode that is less likely to be noticed by user U1. This reduces the persistence of user U1's discomfort after agent device 200's apology and increases user U1's trust in the system.

[0230] The processing steps shown in this embodiment are merely examples of how to implement this embodiment. The order of some of the processing steps may be changed, some of the processing steps may be omitted, or other processing steps may be added, as long as the embodiment is feasible.

[0231] Furthermore, each process shown in this embodiment is executed based on a program that causes a computer to execute each processing procedure. For this reason, this embodiment can also be understood as an embodiment of a program that realizes the function of executing each of these processes, and a recording medium that stores that program. For example, an update process to add a new function to an information processing device can cause that program to be stored in the storage device of the information processing device. This makes it possible to have the updated information processing device perform each of the processes shown in this embodiment.

[0232] Although embodiments of the present invention have been described above, these embodiments only represent a part of the application examples of the present invention, and are not intended to limit the technical scope of the present invention to the specific configurations of the above embodiments. [Explanation of Symbols]

[0233] 100 Information processing system, 110 Information processing device, 120 Control unit, 121 Discomfort level determination unit, 122 Discomfort factor estimation unit, 123 Confidence level determination unit, 124 Agent control unit, 130 Storage unit, 131 Agent information DB, 132 Notification information DB, 133 Map information DB, 134 Surrounding conditions DB, 135 User status DB, 140 Communication unit, 200 Agent device, 210 Display unit, 220 Sound output unit, 230 Drive unit

Claims

1. An information processing method for controlling an agent device capable of communicating with a user, Discomfort estimation process that estimates the user's level of discomfort based on at least one of the user's biometric information, the user's speech content, and the user's actions, The control process includes changing the mode of the agent device to at least one of the following: a normal mode in which notification information is provided to the user; an apology mode in which an apology is shown to the user; and a standby mode in which a mode less likely to attract the user's attention than the normal mode. In the aforementioned control process, After providing the notification information to the user, if a predetermined condition based on the level of discomfort is met, the agent device will transition from the normal mode to the apology mode. If, after the agent device has transitioned to the apology mode, the level of discomfort does not fall below the first criterion, the agent device is transitioned from the apology mode to the standby mode. Information processing methods.

2. The information processing method according to claim 1, The aforementioned agent device is a portable device. The standby mode includes at least one of the following: a first mode in which the agent device is moved to a first position further from the user than the normal mode; a second mode in which the agent device is moved to a second position outside the user's field of vision; a third mode in which the audio output from the agent device is restricted; and a fourth mode in which the agent device is kept stationary. Information processing methods.

3. The information processing method according to claim 1, The agent device is a device that displays an agent image for communication with the user on a display unit. The standby mode includes at least one of the following: a first mode in which the agent image is displayed on the display unit at a position further from the user than in the normal mode; a second mode in which the transparency of the agent image on the display unit is made higher than in the normal mode; a third mode in which the brightness of the agent image on the display unit is made lower than in the normal mode; a fourth mode in which the saturation of the agent image on the display unit is made lower than in the normal mode; a fifth mode in which the audio output from the agent image is restricted; and a sixth mode in which the agent image on the display unit is kept still. Information processing methods.

4. The information processing method according to claim 1, In the control process described above, if, after transitioning the agent device to the standby mode, the level of discomfort becomes less than a second criterion which is a value smaller than the first criterion, the agent device is transitioned from the standby mode to the normal mode. Information processing methods.

5. The information processing method according to claim 4, In the control process described above, when transitioning the agent device from the standby mode to the normal mode, the transition is made via an intermediate mode, which is more likely to attract the user's attention than the standby mode, to the normal mode. Information processing methods.

6. The information processing method according to claim 5, In the aforementioned control process, If, after the agent device has transitioned to the standby mode, the level of discomfort falls below a third criterion which is a value smaller than the first criterion and larger than the second criterion, the agent device is transitioned from the standby mode to the intermediate mode. If, after transitioning the agent device to the intermediate mode, the level of discomfort becomes less than the second criterion, the agent device is transitioned from the intermediate mode to the normal mode. Information processing methods.

7. An information processing method according to any one of claims 4 to 6, The second criterion is set based on the lowest value of the user's discomfort during a predetermined period prior to the time of determination of the level of discomfort when the agent device transitions to the apology mode. Information processing methods.

8. The information processing method according to claim 6, The third criterion is set based on the average value of the user's level of discomfort during a predetermined period prior to the time of determination of the level of discomfort when the agent device transitions to the apology mode. Information processing methods.

9. An information processing method according to any one of claims 1 to 6, In the control process described above, if, after transitioning the agent device to the apologetic mode, the level of discomfort falls below the first criterion within a predetermined period, the agent device is transitioned from the apologetic mode to the normal mode, and the agent device is instructed to perform a predetermined action in the normal mode that shows a relieved gesture. Information processing methods.

10. An information processing method according to any one of claims 1 to 6, In the control process described above, if the transition to the apology mode occurs a predetermined number of times or more within a predetermined period, the first criterion is changed to a smaller value. Information processing methods.

11. An information processing device that controls an agent device capable of communicating with a user, A discomfort estimation unit estimates the user's level of discomfort based on at least one of the user's biometric information, the user's speech content, and the user's actions. The control unit changes the mode of the agent device to at least one of the following modes: a normal mode that provides notification information to the user, an apology mode that shows an apology to the user, and a standby mode that is less likely to attract the user's attention than the normal mode. The control unit, After providing the notification information to the user, if a predetermined condition based on the level of discomfort is met, the agent device will transition from the normal mode to the apology mode. If, after the agent device has transitioned to the apology mode, the level of discomfort does not fall below the first criterion, the agent device is transitioned from the apology mode to the standby mode. Information processing device.

Images