Information processing method and information processing apparatus

Abstract

To more appropriately notify a user of information.SOLUTION: An information processing method provides a user U1 of notification information using agent apparatuses 200, 700 having a display unit 210 that displays a face image. The information processing method includes: determination processing (step S504) for determining whether the user U1 is watching the display unit 210 based on operating state information of the user U1; and control processing (steps S505 to S510) for controlling a display mode of the display unit 210, based on a result of determination by the determination processing. In the control processing, if it is determined that the user U1 is not watching the display unit 210, the display unit 210 is caused to execute a first performance so as to call the user U1's attention and when it is determined that the user U1 is watching the display unit 210, the display unit 210 is caused to display the notification information.SELECTED DRAWING: Figure 7

Description

Technical Field

[0001] The present invention relates to an information processing method and an information processing apparatus for notifying a user of information.

Background Art

[0002] Conventionally, there are robots capable of communicating with users. For example, a device has been proposed that displays information such as a character string on a display device that displays a face image to communicate with the user (see, for example, Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In the above-described conventional technology, it is possible to gradually change a face image and completely hide the face image to display information such as a character string on a display device. However, there is a possibility that the information to be notified is displayed on the display device when the user is not looking at the display device. In such a case, it becomes difficult to appropriately notify the user of the information. Therefore, it is important to attract the user's attention when displaying the information to be notified and to more appropriately notify the user of the information.

[0005] An object of the present invention is to more appropriately notify a user of information.

Means for Solving the Problems

[0006] One aspect of the present invention uses an agent device having a display device that displays a face image , relating to a specified objectThis is an information processing method for providing notification information to a user. This information processing method includes a determination process that determines whether the user is looking at the display device based on the user's operating state information, and a control process that controls the display mode of the display device based on the determination result of the determination process. In the control process, if it is determined that the user is not looking at the display device, the display device is made to execute a first effect to guide the user's attention, and if it is determined that the user is looking at the display device, After the agent device executes a second effect, which is a performance intended to guide the user's attention toward the object and displays a facial image on the display device that is looking toward the object, Display notification information on the display device. [Effects of the Invention]

[0007] According to the present invention, information can be communicated to the user more appropriately. [Brief explanation of the drawing]

[0008] [Figure 1] Figure 1 is a simplified diagram showing an example of the interior layout of a vehicle. [Figure 2] Figure 2 is a simplified front view showing an example of the external configuration of an agent device. [Figure 3] Figure 3 shows an example of the system configuration of an information processing system installed in a vehicle. [Figure 4] Figure 4 shows an example of a notification system used to inform passengers in a vehicle to wear their seat belts. [Figure 5] Figure 5 shows an example of a notification system used to inform passengers in a vehicle to wear their seat belts. [Figure 6] Figure 6 shows an example of a notification system used to inform passengers in a vehicle to wear their seat belts. [Figure 7] Figure 7 is a flowchart showing an example of broadcast information output processing in an information processing device. [Figure 8] Figure 8 shows an example of the system configuration of an information processing system installed in a vehicle. [Figure 9] Figure 9 shows an example of the external configuration of an agent device whose face can rotate left and right. [Figure 10]Figure 10 shows an example of the external configuration of an agent device whose face portion can rotate vertically. [Figure 11] Figure 11 shows an example of the external configuration of an agent device equipped with a rotatable arm. [Modes for carrying out the invention]

[0009] Embodiments of the present invention will be described below with reference to the attached drawings.

[0010] [Example of agent device installation] Figure 1 is a simplified diagram showing an example of the interior configuration of vehicle C1. Note that Figure 1 shows an example of the exterior configuration as viewed from the rear of vehicle C1, specifically the area in front of the driver's seat and passenger seat (not shown). Furthermore, for the sake of clarity, Figure 1 omits illustrations of anything other than the dashboard 2, steering wheel 3, front windshield 4, rearview mirror 5, camera 101, and agent equipment 200.

[0011] The agent device 200 is a small robot installed on the dashboard 2 of the vehicle C1. In this embodiment, an example is shown in which a robot modeled after a human is used as the agent device 200. Note that Figure 1 shows an example in which the agent device 200 is installed on the dashboard 2, but it is not limited to this. For example, the agent device 200 may be installed on the top of the front windshield 4. Also, Figure 1 shows an example in which a robot modeled after a human is used as the agent device 200, but it is not limited to this. For example, the agent device 200 may be a robot modeled after an animal such as a rabbit or a pig, a robot modeled after a virtual creature (for example, the face of an anime character), or a robot modeled after another object (for example, a television-type device, a radio-type device). In this way, it is possible to use an anthropomorphic agent as the agent device 200.

[0012] The agent device 200 executes various operations based on instructions from the information processing device 110 (see FIG. 3). For example, the agent device 200 outputs various types of information related to driving assistance when the user U1 performs a driving operation under the control of the information processing device 110. As this driving assistance, notification of a moving object in front or behind is assumed. For example, as notification of a moving object in front, it is possible to output an audio such as "There is a railroad crossing ahead, so be careful" or "There is a person ahead". Thus, the agent device 200 executes driving assistance.

[0013] Here, notification means conveying or informing some kind of information. Note that the notification shown in this embodiment may also be referred to as notification, transmission, etc.

[0014] The camera 101 is provided on the ceiling inside the vehicle C1 and captures a subject inside the vehicle C1 to generate an image (image data). Note that the camera 101 is constituted by, for example, one or a plurality of camera devices or image sensors capable of capturing a subject. For example, the camera 101 can be provided above the front windshield 4, that is, above the rearview mirror 5. In FIG. 1, an example including at least one camera 101 is shown, but two or more imaging devices may be provided, and all or part of the images of these imaging devices may be used. Also, the installation location of each imaging device is not limited to the example shown in FIG. 1 and can be changed as appropriate. Further, one or a plurality of devices capable of acquiring a subject existing in all directions of the vehicle C1 and a subject inside the vehicle C1, for example, a 360-degree camera, may be used.

[0015] [Example of the external configuration of the agent device] FIG. 2 is a front view schematically showing an example of the external configuration of the agent device 200. In the present embodiment, an example is shown in which face images, notification information, etc. are displayed on the display unit 210 when the agent device 200 is in the on state. Note that the agent device 200 can be turned on in response to the on operation of the vehicle C1. Further, the agent device 200 can be turned on in response to a user operation related to the agent device 200. The on / off operation of the vehicle C1 means an on operation or an off operation of a start key related to the start or stop of the vehicle C1.

[0016] FIG. 2(A) shows a front view of the agent device 200 in a normal state. The agent device 200 is configured by arranging a substantially box-shaped body part 202 and a substantially box-shaped face part 201 vertically via a connecting part 203. Further, a display unit 210 for displaying various images is provided on the face part 201. A body image indicating the body is displayed on the surface of the body part 202. As the display unit 210, for example, various display media such as a light-emitting sheet, a liquid crystal display, and an organic EL (Electro Luminescence) can be used. The connecting part 203 is recognized as a part corresponding to the neck of a living being.

[0017] On the display unit 210 of the face part 201, each part constituting the face of the agent, for example, an eye part 211, a nose part 212, and a mouth part 213 are displayed. In the present embodiment, an example is shown in which the agent device 200 is installed on the dashboard 2 of the vehicle C1 such that the eye part 211 looks inside the vehicle. Further, the side on which the display unit 210 of the agent device 200 is provided is referred to as the front side of the agent device 200, and the side opposite to the front side of the agent device 200 is referred to as the rear side of the agent device 200 for explanation. Further, in the agent device 200, the left side in FIG. 2 is referred to as the right side of the agent device 200, and the right side in FIG. 2 is referred to as the left side of the agent device 200 for explanation.

[0018] Figure 2(B) shows a front view of the agent device 200 with each part of the face 201 facing left. Figure 2(C) shows a front view of the agent device 200 with each part of the face 201 facing right. As shown in Figure 2(B), by moving the eye 211 to the left, shifting the pupil portion of the eye 211 to the left, and making the mouth 213 a vertically elongated oval, it is possible to create the expression of the face 201 facing left. Also, as shown in Figure 2(C), by moving the eye 211 to the right, shifting the pupil portion of the eye 211 to the right, and making the mouth 213 a vertically elongated oval, it is possible to create the expression of the face 201 facing right.

[0019] Figure 2(D) shows a front view of the agent device 200 with its face 201 in an angry state. As shown in Figure 2(D), by making the eye area 211 a vertically elongated oval, shifting the pupil portion of the eye area 211 downwards, and making the mouth area 213 a vertically elongated oval, it is possible to create an angry facial expression for the face 201. Furthermore, by displaying an exclamation mark image 214 to attract attention or outputting angry audio information S1, it is possible to express anger further. Also, by displaying an exclamation mark image 214, it is possible to express surprise. In other words, the facial expression of the face 201 shown in Figure 2(D) can also be understood as a surprised facial expression for the face 201.

[0020] In this way, by performing effects using various images such as the exclamation mark image 214, it is possible to further attract the attention of user U1. In addition, the attention of user U1 may be attracted by performing effects using means other than images such as the exclamation mark image 214, for example, effects using sound, effects using the movements of various parts of the agent device 200, etc. Effects using the movements of various parts of the agent device 200 will be explained in detail with reference to Figures 8 to 11.

[0021] Note that the shape, display mode, and audio output mode of these agent devices 200 are examples only, and other shapes, display modes, and audio output modes are also possible. Furthermore, although Figure 2 shows an example in which the face part 201 and the body part 202 are configured as separate components, the face part 201 and the body part 202 may also be configured as a single housing.

[0022] [Example of an information processing system configuration] Figure 3 shows an example of the system configuration of the information processing system 100 installed in vehicle C1.

[0023] The information processing system 100 comprises a camera 101, a location information acquisition sensor 102, an audio input unit 103, sensors 104, an information processing device 110, and an agent device 200. The information processing device 110 and the agent device 200 are connected by a communication method using wired communication or wireless communication. The information processing device 110 is also connected to the network 20 by a communication method using wireless communication. The network 20 is a network such as a public telephone network or the Internet. The agent device 200 may also be connected to the network 20 by a communication method using wireless communication. Although Figure 3 shows an example in which the information processing device 110 and the agent device 200 are configured as separate units, the information processing device 110 and the agent device 200 may also be configured as a single unit.

[0024] Camera 101 captures an image of a subject and generates an image (image data) based on the control of the information processing device 110, and outputs image information related to the generated image to the information processing device 110. Camera 101 is installed inside at least one part of the vehicle C1 and captures an image of a subject inside the vehicle C1 and generates an image (image data). In Figure 1, camera 101 is shown installed inside the vehicle C1. As described above, camera 101 is composed of, for example, one or more camera devices or image sensors capable of capturing an image of a subject. For example, one camera 101 may be installed in front of the vehicle C1 to capture an image of a subject from the front of the vehicle C1 and generate an image (image data), or another camera 101 may be installed in the rear of the vehicle C1 to capture an image of a subject behind the vehicle C1 and generate an image (image data).

[0025] The position information acquisition sensor 102 acquires position information regarding the location of vehicle C1 and outputs the acquired position information to the information processing device 110. For example, this can be implemented using a GNSS receiver that acquires position information using GNSS (Global Navigation Satellite System). The position information includes various position data such as latitude, longitude, and altitude at the time of receiving the GNSS signal. Position information may also be acquired by other methods. For example, position information may be derived using information from access points or base stations in the vicinity. Position information may also be acquired using beacons. For example, based on the information acquired by the position information acquisition sensor 102, it is possible to determine the state of vehicle C1, such as driving, stopped, or reversing.

[0026] Furthermore, for example, it is possible to determine, based on the location information acquired by the location information acquisition sensor 102, whether vehicle C1 is located near a facility outside of vehicle C1, such as a coffee shop.

[0027] The audio input unit 103 is located inside the vehicle C1 and, based on the control of the information processing device 110, acquires sounds from inside the vehicle C1 and outputs sound information related to the acquired sounds to the information processing device 110. For example, one or more microphones or sound acquisition sensors can be used as the audio input unit 103.

[0028] The sensors 104 are various sensors installed on the vehicle C1, and the detection information acquired by each sensor is output to the information processing device 110. Examples of sensors include a human presence sensor, distance sensor, vehicle speed sensor, acceleration sensor, seat occupancy sensor, seat belt sensor, door sensor, etc. These are just examples, and other sensors may be used. Also, only some of these sensors may be used.

[0029] A human presence sensor is a sensor that detects the presence, number, location, and status of people inside vehicle C1. For example, it is possible to use a human presence sensor that utilizes infrared light, ultrasound, visible light, or an image sensor. For instance, if a person is seated in the driver's seat, passenger seat, or rear seat, their presence can be detected by the human presence sensor. Furthermore, by using both a human presence sensor and a seating sensor, it is possible to improve the accuracy of detecting the seating status of each seat.

[0030] The distance sensor is a sensor that detects the distance to a person inside vehicle C1, the distance to an object to be notified, etc. For example, various sensors such as distance measuring sensors can be used as distance sensors.

[0031] The seat occupancy sensor (or seat sensor) is a sensor that detects whether or not there is an occupant seated in each seat of vehicle C1. The seat belt sensor is a sensor that detects whether or not an occupant seated in each seat of vehicle C1 is wearing a seat belt. The door sensor is a sensor that detects whether or not each door of vehicle C1 is ajar. For each of these sensors, known sensors can be used.

[0032] The information processing device 110 comprises a control unit 120, a storage unit 130, and a communication unit 140. The communication unit 140 exchanges various types of information with other devices using wired or wireless communication based on the control of the control unit 120.

[0033] The control unit 120 controls each part based on various programs stored in the memory unit 130. The control unit 120 is implemented by a processing unit such as a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit). For example, the processing speed can be increased by performing image processing using a GPU. Furthermore, the processing speed can be increased even further by performing parallel processing using a GPU. The vehicle ECU (Electronic Control Unit) of vehicle C1 may be used as the control unit 120, or a processing unit different from the vehicle ECU may be provided as the control unit 120. The control unit 120 also includes a conversion function that converts text to speech. This conversion function is implemented by, for example, TTS (Text to Speech).

[0034] The control unit 120 executes control processing to control the operating state of the agent device 200 based on the information output from the camera 101, position information acquisition sensor 102, voice input unit 103, sensors 104, communication unit 140, etc. Specifically, the control unit 120 comprises an operating state determination unit 121 and an agent control unit 122.

[0035] The operating state determination unit 121 determines the operating state of the occupants riding in vehicle C1 and outputs the determination result to the agent control unit 122. The operating state of the occupants is determined to include, for example, the occupants' line of sight in vehicle C1, the direction of the occupants' faces in vehicle C1, the occupants' seating position in vehicle C1, and the occupants' seating state in vehicle C1.

[0036] For example, the operating state determination unit 121 detects the gaze of the vehicle C1 occupant included in the image based on the image acquired by the camera 101 and identifies the direction of that gaze. Also, for example, the operating state determination unit 121 detects the orientation of the vehicle C1 occupant's face included in the image based on the image acquired by the camera 101 and identifies the direction of that face. Known image recognition techniques can be used for this gaze detection and face orientation detection. Alternatively, the gaze and face orientation of user U1 may be detected using other equipment capable of gaze measurement, such as a gaze measurement device. For example, the camera 101 may be provided on glasses that user U1 can wear, and the gaze of user U1 may be detected based on the image acquired by this camera 101. In this case, for example, the gaze of user U1 can be detected based on the position of user U1's pupil included in the image acquired by the camera 101. In this embodiment, an example is shown in which the gaze of the vehicle C1 occupant is used as the operating state of the vehicle C1 occupant.

[0037] The agent control unit 122 controls the operating state of the agent device 200 based on the determination result from the operating state determination unit 121 and the information output from the camera 101, position information acquisition sensor 102, voice input unit 103, sensors 104, communication unit 140, etc. For example, the agent control unit 122 displays the various parts (eyes 211, nose 212, mouth 213) that make up the face of the anthropomorphized agent on the display unit 210. The agent control unit 122 also displays information that should be notified to the occupants of vehicle C1 (notification information) on the display unit 210. Furthermore, the agent control unit 122 outputs the voice of the anthropomorphized agent, voices corresponding to the information that should be notified to the occupants of vehicle C1, etc., from the sound output unit 220.

[0038] For example, the agent control unit 122 calculates the direction the eye unit 211 should face and performs gaze control to display the eye unit 211 facing that direction on the display unit 210. The direction the eye unit 211 should face is, for example, the direction in which the object corresponding to the notification information exists. For example, if the notification information concerns wearing the driver's seat belt, the object corresponding to that notification information is the driver's seat belt. In this case, the direction the eye unit 211 should face is the direction of the driver's seat belt. In this way, the agent control unit 122 functions as a gaze control unit that controls the position of the eye unit 211.

[0039] Furthermore, the agent control unit 122 determines the notification information to be displayed on the display unit 210 and the notification information to be output from the sound output unit 220, and executes control to output that notification information. Notification information concerning objects located inside the vehicle C1 is stored in the notification information DB 132. Examples of the output of each of these notification information are shown in Figures 4 to 6 and 9 to 11. For example, in the case of notification information regarding a door being ajar, an image showing a door is displayed as notification information, and in the case of notification information regarding wearing a seat belt, an image showing a seat belt is displayed as notification information.

[0040] The memory unit 130 is a storage medium for storing various types of information. For example, the memory unit 130 stores various types of information necessary for the control unit 120 to perform various processes (e.g., control programs, agent information DB 131, notification information DB 132, map information DB 133). The memory unit 130 also stores various types of information acquired via the communication unit 140. As the memory unit 130, for example, ROM (Read Only Memory), RAM (Random Access Memory), SRAM (Static Random Access Memory), HDD (Hard Disk Drive), SSD (Solid State Drive), or a combination thereof can be used.

[0041] The agent information DB131 stores information necessary to realize various operations of the agent device 200. For example, facial image information displayed on the display unit 210 (e.g., eye area 211, nose area 212, mouth area 213) and audio information output from the sound output unit 220 are stored in the agent information DB131.

[0042] The notification information DB132 stores the information necessary to output notification information about objects located inside or outside the vehicle C1. For example, in the case of notification information about a door being ajar, image information showing the door and audio information for notifying the ajar door are stored as notification information in the notification information DB132. Also, for example, in the case of notification information about wearing a seat belt, image information showing the seat belt and audio information for notifying the seat belt to be worn are stored as notification information in the notification information DB132.

[0043] Map information DB133 stores map information such as road information necessary for route guidance of vehicle C1. This map information includes road gradient, road intersections, number of road lanes, road width information, and road topography information. The map information also includes road signs indicating speed limits, one-way streets, etc., pedestrian crossings, lane markings, etc. The map information may also include information on road structures (e.g., traffic lights, utility poles), facilities such as buildings, and surrounding tourist information. Agent information DB131, notification information DB132, and map information DB133 may be stored in the storage unit 130 of vehicle C1, or they may be obtained from external devices via the network 20.

[0044] The agent device 200 is a robotic device that performs various operations based on instructions from the information processing device 110. The agent device 200 includes a display unit 210 and a sound output unit 220. The display unit 210 and the sound output unit 220 are controlled by a control unit (not shown) provided in the agent device 200.

[0045] The display unit 210 is a display unit that displays various images based on instructions from the information processing device 110. For example, the display unit 210 can be an OLED (Electroluminescence) panel, an LCD (Liquid Crystal Display) panel, or other display panel. The display unit 210 may be configured as a touch panel that allows the user to perform input by touching or bringing their finger close to the display surface, or it may be configured as a separate user interface.

[0046] The sound output unit 220 outputs various sounds based on instructions from the information processing device 110. For example, one or more speakers can be used as the sound output unit 220. Note that the display unit 210 and the sound output unit 220 are examples of a user interface, and some of them may be omitted, or other user interfaces may be used.

[0047] Here, the agent device 200 installed on the dashboard 2 is a three-dimensional robot that has been anthropomorphized, and is therefore assumed to have a strong presence that is easily recognized by the occupants of vehicle C1. For example, if it is necessary to notify the occupants of vehicle C1 of some information, it is conceivable to use the agent device 200 to do so. For example, the notification information could be displayed on the display unit 210 to notify the occupants of vehicle C1. However, there is a possibility that the notification information may be displayed on the display unit 210 even if the occupants of vehicle C1 are not looking at it. In such a case, it becomes difficult to properly convey the notification information to the occupants of vehicle C1.

[0048] Therefore, in this embodiment, it is determined whether or not the occupant of vehicle C1 is looking at the display unit 210. If it is determined that the occupant of vehicle C1 is not looking at the display unit 210, the display unit 210 is instructed to perform an action to attract the attention of the occupant of vehicle C1, thereby drawing the occupant's attention to the display unit 210. When it is determined that the occupant of vehicle C1's attention is directed towards the display unit 210 and that the occupant of vehicle C1 is looking at the display unit 210, notification information is displayed on the display unit 210. Similarly, if it is initially determined that the occupant of vehicle C1 is looking at the display unit 210, notification information is also displayed on the display unit 210. In this way, it is possible to attract the attention of the occupant of vehicle C1 when displaying notification information on the display unit 210, so that the notification information can be conveyed to the occupant of vehicle C1 more appropriately.

[0049] [Example of a notification to fasten a seatbelt] Figures 4 to 6 show examples of notifications when informing a user U1, who is riding in vehicle C1, to wear a seat belt. User U1 is assumed to be the driver sitting in the driver's seat of vehicle C1.

[0050] [Example of notifying a user viewing a facial image on an agent device to fasten their seatbelt] Figure 4 shows an example of notification when a user U1, who is sitting in the driver's seat of vehicle C1, is notified to wear a seat belt. Figure 4 also shows an example of notification when user U1 is viewing a face image displayed on the display unit 210 of the agent device 200, as indicated by arrow A1 (see Figure 4(A)).

[0051] The operating state determination unit 121 detects the user U1's gaze A1 based on the image acquired by the camera 101 and identifies the direction of that gaze A1. In other words, the operating state determination unit 121 determines where the user U1 is looking.

[0052] Next, the agent control unit 122 determines whether the gaze A1 of user U1, as determined by the operating state determination unit 121, is directed toward the display unit 210 of the agent device 200. If the gaze A1 of user U1 is directed toward the display unit 210 of the agent device 200, the agent control unit 122 causes the agent device 200 to perform an action to guide the gaze A1 of user U1 toward the target object. In the example shown in Figure 4(A), the target object to be notified is the seat belt of user U1, so, similar to the example shown in Figure 2(B), the agent control unit 122 controls the display state of the display unit 210 so that each part of the face unit 201 faces toward user U1. Specifically, the face image (eyes 211, nose 212, mouth 213) is displayed on the display unit 210 so that the gaze A2 of the eye unit 211 faces toward user U1. In this manner, the agent control unit 122 causes the display unit 210 to perform an animation to guide the user U1's gaze towards the target object. However, if the target object to be notified is located in the direction of user U1 (for example, user U1's seat belt), and user U1's gaze A1 is directed towards the display unit 210, the animation to guide user U1's gaze towards the target object may be omitted. In this case, when it is determined that user U1's gaze A1 is directed towards the display unit 210, the animation to guide user U1's gaze towards the target object is omitted, and output processing (see Figure 4(B)) is performed to output notification information.

[0053] Next, the display control unit 123 performs output processing to output notification information. Specifically, as shown in Figure 4(B), the display control unit 123 displays a notification image 400 on the display unit 210 to notify the user of wearing a seat belt, and outputs audio information S2 to notify the user of wearing a seat belt. In this case, since the user U1's line of sight A3 is directed towards the display unit 210 of the agent device 200, the notification image 400 can be notified to the user U1 more appropriately, as shown by arrow A4.

[0054] In this way, when displaying notification information on the display unit 210 to inform user U1 to wear a seat belt, it is possible to attract user U1's attention, and thus the notification information can be conveyed to user U1 more appropriately.

[0055] [Example of notifying a user who has not seen the facial image on the agent device to fasten their seatbelt] Figure 5 shows an example of notification when a user U1, who is sitting in the driver's seat of vehicle C1, is notified to wear a seat belt. Also in Figure 5, as indicated by arrow A12 (see Figure 5(A)), an example of notification when user U1 is not looking at the face image displayed on the display unit 210 of the agent device 200.

[0056] Similar to Figure 4, the operating state determination unit 121 detects the user U1's gaze A12 based on the image acquired by the camera 101 and identifies the direction of that gaze A12.

[0057] Next, the agent control unit 122 determines whether the user U1's gaze A12, as determined by the operating state determination unit 121, is directed towards the display unit 210 of the agent device 200. If the user U1's gaze A12 is not directed towards the display unit 210 of the agent device 200, the agent control unit 122 instructs the agent device 200 to perform an action to guide the user U1's attention. An example of this is shown in Figure 5(B).

[0058] Specifically, similar to the example shown in Figure 2(D), the agent control unit 122 controls the display state of the display unit 210 so that each part of the face image represents an angry state. Specifically, the face image (eyes 211, nose 212, mouth 213) is displayed on the display unit 210 so that the gaze A15 of the eye part 211 is directed toward the user U1. The agent control unit 122 also executes control to output angry audio information S1 from the sound output unit 220. In this way, the agent control unit 122 causes the agent device 200 to perform an action to guide the user U1's attention to the display unit 210. Note that the action to guide the user U1's attention to the display unit 210 may be performed solely by display using the display unit 210, or solely by audio output using the sound output unit 220. Alternatively, it may be performed using other means, such as the movement of the face part 201 itself (see, for example, Figures 9 and 10).

[0059] As shown in Figure 5(B), it is assumed that when an angry facial image is displayed on the display unit 210 and angry voice information S1 is output, user U1 will turn towards the display unit 210 of the agent device 200. In other words, it is assumed that the presentation that guides user U1's attention to the display unit 210 will cause user U1, who was not facing the direction of the display unit 210 of the agent device 200, to face the direction of the display unit 210 of the agent device 200, as shown by arrow A13. In this case, the operation state determination unit 121 detects user U1's gaze A14 based on the image acquired by the camera 101 and identifies the direction of that gaze A14. As a result, the agent control unit 122 determines that user U1's gaze A14, as determined by the operation state determination unit 121, is facing the direction of the display unit 210 of the agent device 200.

[0060] Thus, when user U1's gaze A14 is directed towards the display unit 210 of the agent device 200, the agent control unit 122 causes the agent device 200 to perform an action to guide user U1's gaze A14 towards the target object. Specifically, similar to the example shown in Figure 4(A), the agent control unit 122 controls the display state of the display unit 210 so that the face image faces user U1. This example is shown in Figure 5(C). Note that the example shown in Figure 5(C) is the same as that in Figure 4(A), so no explanation is given.

[0061] Next, the display control unit 123 performs output processing to output notification information, as shown in Figure 5(D). Specifically, similar to the example shown in Figure 4(B), the display control unit 123 displays a notification image 400 on the display unit 210 to notify the user of wearing a seat belt, and outputs audio information S2 to notify the user of wearing a seat belt. In this case, since the user U1's line of sight A18 is directed towards the display unit 210 of the agent device 200, the notification image 400 can be notified to the user U1 more appropriately, as shown by arrow A19.

[0062] In this way, when displaying notification information on the display unit 210 to inform a user U1 who is not facing the display unit 210 to wear a seat belt, an effect is performed to guide the user U1's attention to the display unit 210. This makes it possible to attract the user U1's attention to the display unit 210, and thus the notification information can be conveyed to the user U1 more appropriately.

[0063] [Example: Notifying a user viewing a facial image on an agent device that another person is wearing a seatbelt] Figure 6 shows an example of notification when user U1, who is sitting in the driver's seat of vehicle C1, is notified that a user sitting in the rear seat behind user U1 is wearing a seat belt. Also in Figure 6, as indicated by arrow A21 (see Figure 6(A)), an example of notification is shown when user U1 is viewing a face image displayed on the display unit 210 of the agent device 200.

[0064] Similar to Figure 4, the operating state determination unit 121 detects the user U1's gaze A21 based on the image acquired by the camera 101 and identifies the direction of that gaze A21.

[0065] Next, the agent control unit 122 determines whether the user U1's gaze A21, as determined by the operation state determination unit 121, is directed towards the display unit 210 of the agent device 200. If the user U1's gaze A21 is directed towards the display unit 210 of the agent device 200, the agent control unit 122 instructs the agent device 200 to perform an action to guide the user U1's gaze A21 towards the target object. An example of this operation is shown in Figure 6(B).

[0066] In Figure 6, the object to be notified is the seat belt of another user other than user U1, so the agent control unit 122 controls the display state of the display unit 210 so that the face image faces the direction of the seat belt. In Figure 6(B), for the sake of clarity, the position of the rear seat behind user U1 (the seat belt of the rear seat behind user U1) is schematically shown by a circle above user U1 (object O1). Specifically, the agent control unit 122 displays the eyes 211, nose 212, and mouth 213 on the display unit 210 so that the line of sight A23 of the eyes 211 is in the direction of object O1. The agent control unit 122 also outputs audio information S3 from the sound output unit 220 to guide user U1's attention to object O1. This makes it possible to direct user U1's line of sight A24 towards object O1.

[0067] As shown in Figure 6(C), a facial image is displayed on the display unit 210 showing the gaze A26 of the eye unit 211 facing the direction of the object O1, and it is assumed that user U1 will turn towards the object O1 in response to the output of audio information S3 guiding user U1 towards the object O1. In other words, it is assumed that, through the guidance of user U1 towards the object O1, user U1, who was facing the direction of the display unit 210 of the agent device 200, will turn towards the direction of the object O1, as shown by arrow A25. In this case, the operation state determination unit 121 detects user U1's gaze A27 based on the image acquired by the camera 101 and identifies the direction of that gaze A27. As a result, the agent control unit 122 determines that user U1's gaze A27, as determined by the operation state determination unit 121, is facing the direction of the object O1.

[0068] Thus, when user U1's line of sight A27 is directed toward object O1, the display control unit 123 executes output processing to output notification information, as shown in Figure 6(D). Specifically, similar to the example shown in Figure 4(B), the display control unit 123 displays a notification image 400 on the display unit 210 to notify the user to wear a seat belt, and also outputs audio information S2 to notify the user to wear a seat belt. In this case, as shown in Figure 6(D), it is expected that the display content of the face image that was facing toward object O1 will change, and the audio information S2 will be output, causing user U1, who was facing toward object O1, to turn toward the display unit 210. In other words, it is expected that the output operation of the notification information will cause user U1, who was facing toward object O1, to turn toward the display unit 210 of the agent device 200, as shown by arrow A28. In this case, the operation state determination unit 121 detects the user U1's gaze A29 based on the image acquired by the camera 101 and identifies the direction of that gaze A29. As a result, the agent control unit 122 determines that the user U1's gaze A29, as determined by the operation state determination unit 121, is pointing towards the target object O1. In this case, since the user U1's gaze A29 is pointing towards the display unit 210 of the agent device 200, the notification image 400 can be notified to the user U1 more appropriately, as shown by arrow A30.

[0069] In this way, when displaying notification information on the display unit 210 to inform user U1, who is facing the display unit 210, that another person is wearing a seat belt, an effect is performed to guide user U1's attention to the location of the other person's seat belt. This makes it possible to draw user U1's attention to the location of the seat belt that needs to be worn, and thus the notification information can be conveyed to user U1 more appropriately.

[0070] [Example of information processing device operation] Figure 7 is a flowchart showing an example of the notification information output process in the information processing device 110. This notification information output process is executed by the control unit 120 based on a program stored in the memory unit 130. This notification information output process is executed continuously at each control cycle. This notification information output process will be explained with reference to Figures 1 to 6 as appropriate.

[0071] In step S501, the agent control unit 122 performs an analysis process to analyze the input information. This input information includes information input to the information processing device 110 from the camera 101, location information acquisition sensor 102, voice input unit 103, sensors 104, etc., and information acquired from the outside via the communication unit 140 and input to the information processing device 110.

[0072] In step S502, the agent control unit 122 determines, based on the results of the analysis of the input information, whether or not there is information (notification information) that should be notified to the occupants of vehicle C1.

[0073] For example, based on a seat belt sensor that detects whether a seat belt is being worn and a seat occupancy sensor (or seat sensor) that detects the presence or absence of an occupant in each seat of vehicle C1, it is possible to determine whether an occupant in each seat of vehicle C1 is wearing a seat belt. If it is determined that an occupant is seated but not wearing a seat belt, it is determined that there is notification information available for that occupant.

[0074] Furthermore, for example, a partially closed door on vehicle C1 can be detected based on a partially closed door sensor that detects each door of vehicle C1 being partially closed. If a partially closed door is detected, it is determined that there is notification information for the partially closed door.

[0075] Furthermore, for example, based on information obtained from an external device, such as an information provision server, via the communication unit 140, and the current location of vehicle C1 obtained by the location information acquisition sensor 102, it is possible to determine whether there are any landmarks, commercial facilities, etc., around vehicle C1. For example, if it is determined that there is a commercial facility, such as a coffee shop, around vehicle C1, it is determined that there is information available as notification information about that coffee shop. Also, for example, if it is determined that there is a tourist facility, such as ABC Castle, around vehicle C1, it is determined that there is information available as notification information about ABC Castle.

[0076] In step S503, the agent control unit 122 determines whether or not information has been generated that should be notified to the occupants of vehicle C1. If information has been generated that should be notified to the occupants of vehicle C1, the process proceeds to step S504. On the other hand, if no information has been generated that should be notified to the occupants of vehicle C1, the notification information output processing operation is terminated.

[0077] In step S504, the operating state determination unit 121 determines whether the occupants of vehicle C1 are facing the agent device 200. For example, if there is one occupant in vehicle C1, it is determined whether that one occupant is facing the agent device 200. Also, for example, if there are multiple occupants in vehicle C1, it is determined whether at least one of the multiple occupants, or all of them, are facing the agent device 200. The occupants of vehicle C1 can be detected based on a human presence sensor, a seating sensor, etc. If the occupants of vehicle C1 are facing the agent device 200, the process proceeds to step S505. On the other hand, if the occupants of vehicle C1 are not facing the agent device 200, the process proceeds to step S508.

[0078] In step S505, the agent control unit 122 instructs the agent device 200 to perform actions that guide the attention of the occupants of vehicle C1 to the target object. For example, the agent control unit 122 controls the display state of the display unit 210 so that the gaze of the eye portion 211 of the agent device 200, which is modeled after a living creature, is directed toward the target object. The agent control unit 122 also controls the output state of the sound output unit 220 so that sounds that draw attention to the target object are emitted from the face portion 201. For example, as shown in Figures 6(B) and 6(C), a face image (eye portion 211, nose portion 212, mouth portion 213) is displayed on the display unit 210, and sound information S3 is output from the sound output unit 220. Furthermore, as shown in Figures 9 and 10, the face unit 201 itself may be moved to guide the occupant's attention to the object, or as shown in Figure 11, the arms 751 and 752 may be used to point to the object as a way to guide the occupant's attention to the object.

[0079] Furthermore, control may be performed such that the gaze of the eye unit 211 moves back and forth between the object and the occupants of vehicle C1. That is, by repeatedly guiding the occupants to the object and drawing their attention, it is possible to guide the attention of the occupants of vehicle C1 to the object. For example, suppose that the occupants of vehicle C1 are reluctant to look at the display unit 210. In such a case, if there is some movement of the agent device 200, the occupants of vehicle C1 may sense that the agent device 200 is doing something and look at the agent device 200. For example, this control may be performed according to the importance of the notification information. That is, this control can be performed when the importance of the notification information is high. As mentioned above, if the object to be notified is in the direction of the occupants of vehicle C1, and the occupants' gaze is directed towards the display unit 210, the performance to guide the occupants' attention to the object may be omitted. In this case, if it is determined that the occupant of vehicle C1 is looking in the direction of the display unit 210, the effects that guide the occupant's attention to the object are omitted, and output processing is performed to output notification information.

[0080] In step S506, the agent control unit 122 causes the agent device 200 to perform output processing to output notification information. For example, as shown in Figures 4(B), 5(D), and 6(D), the notification image 400 is displayed on the display unit 210, and the audio information S2 is output from the sound output unit 220.

[0081] Furthermore, the system may repeatedly perform at least one of the following actions: an action to guide the attention of the occupants of vehicle C1 (step S508), an action to guide the attention of the occupants of vehicle C1 to the target object (step S505), and an output process to output notification information (step S506). This makes it possible to guide the attention of the occupants of vehicle C1 to the display unit 210 while providing notification information to the occupants of vehicle C1 more appropriately.

[0082] In step S507, the agent control unit 122 determines whether or not the occupant of vehicle C1 has taken a predetermined action corresponding to the notification information output in step S506. That is, it determines whether or not the content of the notification information has been resolved. If the occupant has taken a predetermined action corresponding to the notification information, the notification information output process is terminated. If any of the following are being performed: an effect to guide the attention of the occupant of vehicle C1 (step S508), an effect to guide the attention of the occupant of vehicle C1 to the target object (step S505), or output processing to output notification information (step S506), these operations are terminated. On the other hand, if there is no predetermined action by the occupant corresponding to the notification information, the process proceeds to step S509.

[0083] For example, if the notification information output in step S506 is notification information regarding seat belt fastening, the predetermined action is for the occupant seated in the seat that was the subject of the notification to fasten their seat belt. Whether or not this seat belt fastening action has been performed can be determined based on the information detected by the seat belt sensor.

[0084] Furthermore, for example, if the notification information output in step S506 concerns a partially closed door on vehicle C1, the predetermined action is to close the door that was the subject of the notification. Whether or not this door closing action is performed can be determined based on the detection information from the door sensor.

[0085] Furthermore, for example, if the notification information output in step S506 concerns commercial facilities, tourist facilities, etc., located around vehicle C1, then moving vehicle C1 to the location of the notification becomes a predetermined action. Whether or not to move vehicle C1 to the location of the notification can be determined based on the location information of vehicle C1 acquired by the location information acquisition sensor 102. In this case, the action of the occupant making a voice statement about the location of the notification also becomes a predetermined action. For example, if the notification information concerns commercial facilities located around vehicle C1, such as XYZ Coffee Shop, then making a voice statement about XYZ Coffee Shop, such as "XYZ Coffee is nice" or "XYZ Coffee looks delicious," can be a predetermined action. Also, for example, if the notification information concerns tourist facilities located around vehicle C1, such as ABC Castle, then making a voice statement about ABC Castle, such as "ABC Castle is nice" or "I want to see ABC Castle," can be a predetermined action. Whether or not these voice-related actions are performed can be determined based on the degree of match (or similarity) between the voice information acquired by the voice input unit 103 and predetermined keywords stored in the storage unit 130. For determining the degree of match (or similarity) of the voices, known voice recognition technologies can be employed.

[0086] In step S508, the agent control unit 122 instructs the agent device 200 to perform actions to attract the attention of the occupants of vehicle C1. For example, a face image resembling an animal can be displayed on the display unit 210, and the agent device 200 can perform animal-like movements as an action to attract the attention of the occupants of vehicle C1.

[0087] For example, as shown in Figure 5(B), an angry face image is displayed on the display unit 210, and angry voice information S1 is output. The performance to guide the attention of the occupants of vehicle C1 may be executed on the condition that a predetermined amount of time has elapsed. For example, the performance to guide the attention of the occupants of vehicle C1 may be executed when the number of times it is determined in step S504 that user U1 is not facing agent device 200 exceeds a threshold, or when a predetermined amount of time has elapsed since it was determined in step S504 that user U1 is not facing agent device 200. Similarly, the output of voice information may also be executed on the condition that a predetermined amount of time has elapsed. For example, immediately after it is determined in step S504 that user U1 is not facing agent device 200, an angry face image may be displayed on the display unit 210, and then, after a predetermined amount of time has elapsed, angry voice information S1 may be output.

[0088] Furthermore, the process in step S508 may be repeatedly executed until it is determined in step S504 that user U1 is facing agent device 200. In this case, if the process in step S508 is repeated, the content of the process in step S508 may be changed according to the increase in the number of repetitions. For example, in the first Nth process (where N is an integer greater than or equal to 2), an action using the eye unit 211 is performed; from the Nth to the Mth process (where M is an integer greater than N), an action using voice information together with the eye unit 211 is performed; and in the Mth and subsequent processes, other actions, such as those shown in Figures 9 to 11, are performed together with the eye unit 211 and voice information. In this way, if the process in step S508 is repeated, additional actions such as movement of the eye unit 211, voice output, rotation, and pointing are sequentially performed according to the increase in the number of repetitions. In other words, the initial action is to attract the user U1's attention with small movements of the creature's eyes, but if the user does not notice these small movements, additional actions such as larger eye movements, pointing, sound, and rotation are performed to further guide the occupant's attention.

[0089] In step S509, the agent control unit 122 determines whether the importance of the notification information to be output is high or low. The importance of the notification information is set high for information related to the safety of the occupants of vehicle C1. For example, the importance of notification information regarding seat belt use, ajar door status, etc., is set higher than the standard. On the other hand, the importance of notification information providing information about the surroundings of vehicle C1, tourist information, etc., is set lower than the standard. For example, the importance of notification information regarding facilities around vehicle C1 is set lower. The importance level of the notification information is stored in the notification information DB 132 (see Figure 3).

[0090] If the importance of the broadcast information to be output is high, the process returns to step S504. On the other hand, if the importance of the broadcast information to be output is low, the process proceeds to step S510. It is also possible that, for some reason, the output of the broadcast information may become unnecessary. In this case as well, the process proceeds to step S510.

[0091] For example, in the case of a seatbelt wearing notification, if seatbelt wearing is detected, the output of that notification is unnecessary. Also, for example, in the case of a door ajar notification, if the door that was ajar is detected to be fully closed, the output of that notification is unnecessary. Furthermore, for example, in the case of a commercial facility advertisement notification, if vehicle C1 has moved far away from the commercial facility, the output of that notification is unnecessary.

[0092] It is also conceivable that the occupants of vehicle C1 may not face the agent device 200 for extended periods. In particular, the driver of vehicle C1 may recognize the guidance operation of the agent device 200 but not face it. In such cases, notification information may be forcibly output, and a predetermined time may be set, after which the notification information output process may be terminated. That is, each process from steps S504 to S509 may be repeated until a timeout occurs. Alternatively, notification information may be provided to the occupants of vehicle C1 using only audio. In this way, it is important to prevent guidance effects that may interfere with the driver while driving by appropriately executing forced output of notification information, output of audio-only notification information, etc.

[0093] In step S510, the agent control unit 122 executes a termination process to end the output of the notification information determined in step S502. For example, if the notification information concerns a commercial facility or a tourist facility, it executes a deletion process to delete that notification information. If the notification information can be communicated to the occupants of vehicle C1 by voice alone, the termination process may be executed after the notification information has been output by voice. In addition, if the effects that guide the attention of the occupants of vehicle C1 (step S508) or the effects that guide the attention of the occupants of vehicle C1 to the target object (step S505) are being executed, these operations are also terminated.

[0094] In the example shown in Figure 7, it is possible that in step S504, it is determined that the occupant of vehicle C1 is facing the agent device 200, and after notification information is output in step S506, the process returns to step S504. For example, it is conceivable that after notification information is output in step S506, it is determined in step S504 that the occupant of vehicle C1 is not facing the agent device 200. In this case, in step S508, an action is performed to guide the user U1's attention so that the occupant of vehicle C1 faces the agent device 200 again.

[0095] Furthermore, in the example shown in Figure 7, if the notification information is of high importance, the processes in steps S504 to S509 are repeated and the notification information is output. Conversely, if the notification information is of low importance, the process terminates without outputting the notification information. However, it is also possible to repeat the processes in steps S504 to S509 and output the notification information regardless of its importance.

[0096] Here, it is important to prevent the attention guidance effects of the agent device 200 from interfering with the driver while they are driving. As mentioned above, for example, if the driver is in the driver's seat and driving, the driver may be aware of the attention guidance effects of the agent device 200, but may not be looking in the direction of the agent device 200. Therefore, it is preferable to perform the effects that guide the attention of the occupants of vehicle C1 to the target object, and the effects that guide the attention of the occupants of vehicle C1, while vehicle C1 is stopped. However, if the target is an occupant other than the driver in the driver's seat, it may be performed while vehicle C1 is moving.

[0097] [Example of an agent device equipped with a rotatable face and a pointing pointer] The above examples demonstrate how facial expressions and sounds displayed on the display unit 210 can be used to guide the attention of the occupants of vehicle C1 to an object, and how to perform actions to attract the attention of the occupants of vehicle C1. Here, it is possible to further attract the attention of the occupants of vehicle C1 by operating the face unit 201 itself. Furthermore, by providing the agent device with a component corresponding to a finger, hand, or arm (indicator), and operating this indicator, it is possible to further attract the attention of the occupants of vehicle C1. Therefore, the following examples show agent devices 700 equipped with a rotatable face unit and an indicator unit capable of pointing. Figures 9 and 10 show examples of agent devices 700 equipped with a rotatable face unit. Figure 11 shows an example of agent devices 700 equipped with an indicator unit capable of pointing.

[0098] [Example of an information processing system configuration] Figure 8 shows an example of the system configuration of the information processing system 600 installed in vehicle C1. Since the information processing system 600 is a modified version of the information processing system 100 shown in Figure 3, the same reference numerals are used for parts common to both the information processing system 100 and the information processing system 600, and some of their explanations are omitted.

[0099] The information processing system 600 comprises an information processing device 110 and an agent device 700. The agent control unit 122 controls the operating state of the agent device 700. The agent information DB 131 stores information necessary to realize various operations of the agent device 700. For example, rotation information for rotating the face unit 201 (see Figures 9 and 10), and operation information for operating the arm units 751 and 752 (see Figure 11) are stored in the agent information DB 131.

[0100] The agent device 700 includes a drive unit 730. The drive unit 730 is a drive device that realizes a mechanism for rotating the face unit 201 (see Figures 9 and 10). The drive unit 730 is also a drive device that realizes a mechanism for moving the arms 751 and 752 (see Figure 11).

[0101] Figure 9 shows an example in which the face portion 201 is rotated left and right (in the direction of arrows A30 and A31) using the neck portion 731 as the pivot axis. Figure 10 shows an example in which the face portion 201 is rotated up and down (in the direction of arrow A41) using the rotation support portions 732 and 733 as pivot axes. Figure 11 shows an example in which the arms 751 and 752 are used to point to an object.

[0102] [Example of an agent device with a face that can rotate left and right] Figure 9 shows an example of the external configuration of an agent device 700 whose face portion 201 can be rotated in the left-right direction.

[0103] The agent device 700 is a robotic device in which the neck portion 731 serves as a rotational support, and the face portion 201 and the body portion 202 are connected by the neck portion 731. The neck portion 731 is connected to the drive unit 730 (see Figure 8), and the rotational movement of the drive unit 730 causes the face portion 201 to rotate left and right (arrows A30, A31 direction) with the neck portion 731 as the pivot axis. The rotation range of the face portion 201 (display portion 210) can be set to a range of, for example, a few degrees to 360 degrees. For example, by rotating it about 30 degrees left and right, it is possible to create the impression that the face portion 201 is swaying.

[0104] Similar to Figure 4, the operating state determination unit 121 detects the user U1's gaze A32 based on the image acquired by the camera 101 and identifies the direction of that gaze A32.

[0105] Next, the agent control unit 122 determines whether the user U1's gaze A32, as determined by the operating state determination unit 121, is directed towards the display unit 210 of the agent device 700. If the user U1's gaze A32 is not directed towards the display unit 210 of the agent device 700, the agent control unit 122 instructs the agent device 700 to perform an action to guide the user U1's attention. An example of this is shown in Figure 9(A).

[0106] Specifically, as shown in Figure 9(A), the agent control unit 122 rotates the face unit 201 in the left-right direction (arrows A30 and A31 direction) using the neck unit 731 as the pivot axis. The agent control unit 122 also controls the display state of the display unit 210 so that each part displays an angry face image. In this case, the gaze A33 of the eye unit 211 is directed towards the user U1. The agent control unit 122 also executes control to output angry voice information S11 from the sound output unit 220.

[0107] As shown in Figure 9(B), it is assumed that user U1 will turn towards the display unit 210 of the agent device 700 in response to the angry face image being rotated left and right and the angry voice information S11 being output. In other words, it is assumed that the user U1, who was not facing the direction of the display unit 210 of the agent device 700, will turn towards the display unit 210 of the agent device 700 as shown by the arrow A35, due to the presentation that guides user U1's attention. In this case, the operation state determination unit 121 detects user U1's gaze A36 based on the image acquired by the camera 101 and identifies the direction of that gaze A36. As a result, the agent control unit 122 determines that user U1's gaze A36, as determined by the operation state determination unit 121, is facing the direction of the display unit 210 of the agent device 700. In this case, since user U1's line of sight A36 is directed towards the display unit 210 of the agent device 700, each image displayed on the display unit 210 can be shown to user U1 more appropriately, as indicated by arrow A37.

[0108] Thus, when user U1's line of sight A36 is directed towards the display unit 210 of the agent device 700, the agent control unit 122 sequentially instructs the agent device 700 to perform guidance operations to guide user U1's line of sight A36 towards the target object, and output processing to output notification information. Note that these guidance operations and output processing are the same as those shown in the examples in Figures 4 to 7, so their explanation is omitted here.

[0109] Here, it is considered that the movement of a three-dimensional object shaking or rotating is more likely to attract the user's attention than the change in the facial expression of the agent displayed on the display unit 210. Therefore, in the example shown in Figure 9, the movement of the face unit 201 shaking and rotating in the left-right direction (arrows A30 and A31 direction) with the neck unit 731 as the pivot point is performed as a visual effect to guide the user U1's attention.

[0110] [Example of an agent device with a face that can rotate vertically] Figure 10 shows an example of the external configuration of an agent device 700 whose face portion 201 can be rotated vertically.

[0111] The agent device 700 is a robotic device in which a face portion 201 is installed inside a frame portion 740 that is square-shaped when viewed from the front, and the face portion 201 is capable of rotating vertically inside the frame portion 740. Specifically, the left and right ends of the face portion 201 and the inner surface of the frame portion 740 are connected by rotation support portions 732 and 733. The positions where the rotation support portions 732 and 733 are installed correspond to the ears of the face portion 201, so the rotation support portions 732 and 733 may be configured as the ears of the face portion 201. The rotation support portions 732 and 733 are connected to a drive unit 730 (see Figure 8), and the rotational movement of the drive unit 730 causes the face portion 201 to rotate vertically (in the direction of arrow A41) with the rotation support portions 732 and 733 as the pivot axis. The rotation range of the face unit 201 (display unit 210) can be set to a range of, for example, a few degrees to 360 degrees. For example, by rotating it about 30 degrees in the vertical direction, it is possible to create the effect of the face unit 201 shaking.

[0112] Similar to Figure 4, the operating state determination unit 121 detects the user U1's gaze A42 based on the image acquired by the camera 101 and identifies the direction of that gaze A42.

[0113] Next, the agent control unit 122 determines whether the user U1's gaze A42, as determined by the operating state determination unit 121, is directed towards the display unit 210 of the agent device 700. If the user U1's gaze A42 is not directed towards the display unit 210 of the agent device 700, the agent control unit 122 instructs the agent device 700 to perform an action to guide the user U1's attention. An example of this is shown in Figure 10(A).

[0114] Specifically, as shown in Figure 10(A), the agent control unit 122 rotates the face unit 201 vertically (in the direction of arrow A41) using the rotation support units 732 and 733 as pivot axes. The agent control unit 122 also controls the display state of the display unit 210 so that each part displays an angry face image. In this case, the gaze A43 of the eye unit 211 is directed towards the user U1. The agent control unit 122 also executes control to output angry voice information S12 from the sound output unit 220.

[0115] As shown in Figure 10(B), it is assumed that user U1 will turn towards the display unit 210 of the agent device 700 in response to the angry face image being rotated vertically and the angry voice information S12 being output. In other words, it is assumed that the user U1, who was not facing the direction of the display unit 210 of the agent device 700, will turn towards the display unit 210 of the agent device 700 as shown by the arrow A45, due to the presentation that guides user U1's attention. In this case, the operation state determination unit 121 detects user U1's gaze A46 based on the image acquired by the camera 101 and identifies the direction of that gaze A46. As a result, the agent control unit 122 determines that user U1's gaze A46, as determined by the operation state determination unit 121, is facing the direction of the display unit 210 of the agent device 700. In this case, since user U1's line of sight A46 is directed towards the display unit 210 of the agent device 700, each image displayed on the display unit 210 can be shown to user U1 more appropriately, as indicated by arrow A47.

[0116] Thus, when user U1's line of sight A45 is directed towards the display unit 210 of the agent device 700, the agent control unit 122 sequentially instructs the agent device 700 to perform guidance operations to guide user U1's line of sight A46 towards the target object, and output processing to output notification information. Note that these guidance operations and output processing are the same as those shown in the examples in Figures 4 to 7, so their explanation is omitted here.

[0117] Here, it is considered that the shaking and rotating of a three-dimensional object is more likely to attract the user's attention than the movements of the agent's face displayed on the display unit 210. Therefore, in the example shown in Figure 10, the face unit 201 is made to shake and rotate in the vertical direction (direction of arrow A41) using the rotation support units 732 and 733 as pivot axes, as an effect to guide the user U1's attention.

[0118] Figures 9 and 10 show an example where the display unit 210 is provided on one side of the face unit 201. However, it is possible to rotate the face unit 201 so that either the side of the face unit 201 with the display unit 210 or the back side of the display unit 210 is visible to the user U1. Therefore, the face unit 201 may be configured as a two-sided display with a front and back surface, with the display unit 210 provided on one side (front) and another display unit on the other side (back). In this case, the display unit 210 on the front surface of the face unit 201 displays the eyes 211, nose 212, and mouth 213 to perform effects that guide the user U1's attention to an object. On the other hand, the other display unit on the back side of the face unit 201 displays notification information, for example, the notification information shown in Figures 4(B), 5(D), and 6(D), to perform notification operations to the user U1. This makes it possible to repeatedly provide images of living creatures and notification information to user U1 using the face unit 201 of the agent device 700.

[0119] In this way, it is possible to attract the attention of user U1 by rotating or swaying the face unit 201, and then perform a notification action to user U1. This allows for appropriate notification to be given to user U1. Alternatively, the display state of the face unit 201 may be changed to attract the attention of user U1, and then the face unit 201 may be rotated or swayed before the notification action to user U1 is performed. In this way, when the face unit 201 itself starts moving, it is possible to guide user U1's attention to what will be displayed.

[0120] As shown in Figures 9 and 10, a facial image mimicking a living creature is displayed on the display unit 210, and the agent device 700 can be made to move in a way that resembles a living creature.

[0121] For example, when performing a performance that guides the user's attention to an object (step S505 (see Figure 7)) or a performance that guides the user's attention (step S508 (see Figure 7)), it is possible to perform the rotational movement of the face unit 201 shown in Figures 9 and 10. In this case, each process shown in Figure 7 may be executed repeatedly. In such cases, when each process shown in Figure 7 is repeated, the processing content related to the performance may be changed according to the increase in the number of repetitions. For example, when the number of times it is determined in step S504 that user U1 is not facing agent device 700 exceeds a threshold, or when a predetermined time has elapsed since it was determined in step S504 that user U1 is not facing agent device 700, the rotational movement of the face unit 201 shown in Figures 9 and 10 may be executed.

[0122] Furthermore, control may be implemented that alternates between the display operation of the display unit 210 shown in Figures 4 and 6, and the rotational operation of the face unit 201 shown in Figures 9 and 10. In other words, by repeatedly executing the display mode of the display unit 210 and the operation mode of the face unit 201, it becomes possible to further guide the attention of the occupants of the vehicle C1.

[0123] [Example of an agent device equipped with an arm] Figure 11 shows an example of the external configuration of an agent device 700 equipped with rotatable arms 751 and 752. The arms 751 and 752 are examples of indicator units that can point in a predetermined direction through predetermined movements.

[0124] The agent device 700 is a robotic device in which rod-shaped arms 751 and 752 are provided on the left and right sides of the body 202.

[0125] The arm 751 rotates in the forward / backward, left / right, and up / down directions of the agent device 700, with the attachment point of the arm 751 on the body 202 as the pivot point. Similarly, the central part of the arm 751 rotates in the forward / backward, left / right, and up / down directions of the agent device 700, with its central part as the pivot point.

[0126] As described above, a drive mechanism for driving the arm 751 is provided in the mounting portion and central portion of the arm 751. This makes it possible to move the arm 751 vertically, move it forward and backward, swing it vertically, and swing it forward and backward. Similarly, a drive mechanism for driving the arm 752 is provided in the mounting portion and central portion of the arm 752. This makes it possible to move the arm 752 vertically, move it forward and backward, swing it vertically, and swing it forward and backward. These drive mechanisms constitute the drive unit 730 (see Figure 8).

[0127] Furthermore, the drive unit 730 (see Figure 8) that drives the arm sections 751 and 752 performs each operation based on the control of the agent control unit 122 (see Figure 3).

[0128] Figures 11(B) and 11(C) show a front view of the agent device 700 when the face 201 is facing left and the tip of the arm 751 is pointing to the left.

[0129] The example shown in Figure 11 is a modified version of the example shown in Figure 6, and illustrates an example of notification when user U1, who is sitting in the driver's seat of vehicle C1, is notified that a user sitting in the rear seat behind user U1 is wearing a seat belt. Furthermore, as indicated by arrow A21, Figure 11 shows an example of notification when user U1 is viewing a facial image displayed on the display unit 210 of the agent device 700.

[0130] Specifically, Figure 6 shows an example in which the gaze A23 of the eye unit 211 and audio information S3 that guides the user towards the object O1 are used to guide the user U1 to face the object O1. In contrast, Figure 11 shows an example in which the gaze A23 of the eye unit 211 and audio information S3 that guides the user towards the object O1 are used, along with the tip of the arm unit 751 pointing in the direction of the object O1, to guide the user U1 to face the object O1. Note that, except for the fact that the guidance action is performed using the arm unit 751, it is the same as the example shown in Figure 6, so the same reference numerals are used for parts common to Figure 6 and their explanations are omitted.

[0131] The agent control unit 122 determines whether the user U1's gaze A21, as determined by the operation state determination unit 121, is directed towards the display unit 210 of the agent device 700. If the user U1's gaze A21 is directed towards the display unit 210 of the agent device 700, the agent control unit 122 instructs the agent device 700 to perform an action to guide the user U1's gaze A21 towards the target object. An example of this operation is shown in Figure 11(B).

[0132] As shown in Figure 11(C), a facial image of the user U1 facing the object O1 is displayed on the display unit 210, the tip of the arm 751 points towards the object O1, and voice information S3 guiding the user towards the object O1 is output. In other words, it is expected that the user U1, who was facing the display unit 210 of the agent device 700, will turn towards the object O1 as indicated by arrow A25. In this case, the operation state determination unit 121 detects the user U1's gaze A27 based on the image acquired by the camera 101 and identifies the direction of that gaze A27. As a result, the agent control unit 122 determines that the user U1's gaze A27, as determined by the operation state determination unit 121, is facing the object O1.

[0133] Furthermore, in the example shown in Figure 11, for the sake of ease of explanation, an example is shown in which arm sections 751 and 752 are used as members to point to an object. However, the agent device 700 may be equipped with members that mimic a hand, fingers, and an arm, and these may be used to point to an object. Alternatively, the agent device 700 may be equipped with a member that mimics at least one of the hand, fingers, and arm, and these may be used to point to an object.

[0134] For example, when performing a presentation that guides the user's attention to an object (step S505 (see Figure 7)), it is possible to perform an action that points to the object shown in Figure 11. In this case, each process shown in Figure 7 may be executed repeatedly. In such cases where each process shown in Figure 7 is repeated, the content of the presentation processing may be changed according to the increase in the number of repetitions. For example, if the number of times it is determined in step S504 that user U1 is not facing agent device 700 exceeds a threshold, or if a predetermined time has elapsed since it was determined in step S504 that user U1 is not facing agent device 700, the action that points to the object shown in Figure 11 may be executed.

[0135] Furthermore, control may be implemented that alternates between the display operation of the display unit 210 shown in Figures 4 and 6 and the operation of pointing to an object shown in Figure 11. In other words, by repeatedly executing the display mode of the display unit 210 and the operation mode of the arm units 751 and 752, it becomes possible to further guide the attention of the occupant of the vehicle C1 to the object.

[0136] [Examples of agent devices that can be installed outside of vehicles] The above describes examples of agent devices 200 and 700 installed in vehicle C1. However, this embodiment can also be applied to agent devices that are removable from vehicle C1, agent devices that can be installed outside of a vehicle, etc. For example, it is conceivable that user U1, who possesses a portable agent device, would place the agent device on the dashboard 2 of vehicle C1 when getting into vehicle C1, and then carry the agent device with them when getting out of vehicle C1. It is also conceivable that user U1 would use the agent device within their home.

[0137] For example, within a home, it is conceivable to output notification information regarding various appliances installed in the home. For instance, notifications could be issued when the bath is ready or when cooking is finished. Furthermore, notifications could be issued if the front door or windows are open. Also, notifications could be issued if the gas stove is left on. In these cases, the target objects could be the bath, cooking appliances, doors, windows, etc. The level of importance can be set based on criteria such as safety and urgency. For example, objects with high urgency can have their importance level set higher than the standard, while objects with low urgency can have their importance level set lower. For example, objects with high urgency and those requiring safety maintenance include the gas stove (to prevent forgetting to turn off the flame), the front door, and windows (to notify if they are open). On the other hand, items with a low level of urgency include, for example, bathtubs (notifying when the bath is ready) and cooking appliances (notifying when cooking is finished).

[0138] [Example of representing agent devices in 2D] The above explanation uses three-dimensional agent devices 200 and 700 as examples. However, if it is possible to express the lifelike nature of the agent device, this embodiment may also be applied by displaying the agent image as a two-dimensional image. In this case, both the face and body parts are displayed as two-dimensional images. Furthermore, the indicator parts that mimic hands, fingers, and arms may also be displayed as two-dimensional images.

[0139] [Examples of the effects of this embodiment] Thus, it is possible to guide the user's attention using at least one of the eye area 211, nose area 212, and mouth area 213, as well as using the movements of the face area 201 itself, and using the arms 751 and 752. In other words, it is possible to attract the user's attention by performing various actions using the parts that resemble those of a living creature.

[0140] Thus, biological arms and eyes easily attract the user U1's attention. Therefore, in this embodiment, after attracting the user U1's attention using biological arms and eyes, notification information is displayed at the location where the attention was attracted, making it possible to convey the notification information to the user more appropriately.

[0141] In this way, the effectiveness can be enhanced by displaying a living creature on the display unit 210 or making the face unit 201 move, and then displaying notification information on the display unit 210. For example, if notification information is simply displayed on the display unit 210, it is conceivable that the notification information will be displayed without knowing whether the user is actually looking at it. In contrast, according to this embodiment, by attracting the user's attention using biological arms and eyes, and then displaying the notification information in the place where the attention has been attracted, it is possible to prevent the user from missing the notification information. In other words, since it is possible to attract the user's attention when displaying the notification information on the display unit 210, the notification information can be conveyed to the user more appropriately.

[0142] In this way, by utilizing the lifelike agent devices 200 and 700, it is possible to attract the user's attention through their lifelike appearance and guide their attention to provide notification information. For example, agent device 200 can attract and guide the user's attention through its lifelike appearance by looking towards the user. Also, for example, agent device 700 can attract the user's attention through its lifelike appearance by pointing to an object and guide the user's attention to the object. Furthermore, if the user does not respond or does not look at the notification information, the lifelike agent devices 200 and 700 will be used again to attract the user's attention.

[0143] In the above examples, notification information output processing and the like are shown to be performed by agent devices 200, 700 and information processing device 110 (or information processing systems 100, 600). However, all or part of each of these processes may be performed by other devices. In this case, the information processing system is composed of each device that performs part of each of these processes. For example, at least part of each process can be performed using various information processing devices such as in-vehicle devices, devices usable by user U1 (e.g., smartphones, tablet terminals, personal computers, car navigation systems, IVI), servers that can be connected via a predetermined network such as the Internet, and various electronic devices.

[0144] Furthermore, some (or all) of the information processing systems capable of executing the functions of the information processing device 110 (or information processing systems 100, 600) may be provided by an application that can be provided via a predetermined network such as the Internet. This application may be, for example, SaaS (Software as a Service).

[0145] [Example configuration and effects of this embodiment] The information processing method according to this embodiment is an information processing method that provides notification information to user U1 using agent devices 200, 700 having a display unit 210 (an example of a display device) that displays a face image. This information processing method includes a determination process (step S504) that determines whether user U1 is looking at the display unit 210 based on user U1's operating state information, and a control process (steps S505 to S510) that controls the display mode of the display unit 210 based on the determination result of the determination process. In the control process, if it is determined that user U1 is not looking at the display unit 210, the display unit 210 is made to execute a first effect to guide user U1's attention, and if it is determined that user U1 is looking at the display unit 210, the display unit 210 is made to display notification information. Furthermore, the program according to this embodiment is a program that causes a computer to execute each of these processes. In other words, the program according to this embodiment is a program that causes a computer to realize each of the functions that the information processing device 110 can execute.

[0146] With this configuration, after the first effect of displaying the face of an animal or the like on the display unit 210 is performed, notification information is displayed on the display unit 210, thereby attracting the attention of user U1, and thus the notification information can be conveyed to user U1 more appropriately.

[0147] In the information processing method according to this embodiment, the notification information is information relating to a predetermined object. Furthermore, in the control processing (steps S505, S506), if it is determined that user U1 is looking at the display unit 210, the agent devices 200 and 700 are instructed to perform a second action to guide user U1's attention toward the object, and then the notification information is displayed on the display unit 210. For example, as shown in Figure 6(B), a face image facing toward the object O1 is displayed on the display unit 210, and audio information S3 is output from the sound output unit 220.

[0148] With this configuration, when displaying notification information about a predetermined object, the notification information can be more appropriately communicated to the user U1 by first guiding the user U1's attention towards the object and then displaying the notification information on the display unit 210.

[0149] In the information processing method according to this embodiment, the control process (step S505) includes a second effect in which a face image with the eyes facing the object is displayed on the display unit 210. For example, as shown in Figure 6(B), a face image facing the object O1 is displayed on the display unit 210.

[0150] With this configuration, by using a facial image in which the eyes are facing the object, the user U1's attention is guided towards the object, and then the notification information is displayed on the display unit 210, the notification information can be communicated to the user U1 more appropriately.

[0151] In the information processing method according to this embodiment, the agent device 700 is equipped with arms 751 and 752 (an example of an indicator unit) that can point in a predetermined direction by a predetermined operation, and in the control processing (step S505), as a second performance, the agent device 700 is made to perform an operation to point to an object using the arms 751 and 752. For example, as shown in Figure 11(B), an operation is performed in which the direction of the object O1 is pointed to by the arm 751.

[0152] With this configuration, by using the arm parts 751 and 752 to point in the direction of the object, the user U1's attention is guided towards the object, and then the notification information is displayed on the display unit 210, thereby more appropriately notifying the user U1 of the notification information.

[0153] In the information processing method according to this embodiment, in the control processing (steps S508, S509), if it is determined that user U1 is not looking at the display unit 210, the first performance is continued until it is determined that user U1 is looking at the display unit 210.

[0154] With this configuration, the first presentation that guides the user U1's attention continues until the user U1 looks at the display unit 210, and by displaying the notification information on the display unit 210, the notification information can be conveyed to the user U1 more appropriately.

[0155] In the information processing method according to this embodiment, in the control processing (steps S508 to S510), if it is determined that user U1 is not looking at the display unit 210, and the importance of the notification information is higher than the standard, the first performance is continued until it is determined that user U1 has looked at the display unit 210. If the importance of the notification information is lower than the standard, the first performance is terminated and the notification of the notification information is stopped.

[0156] With this configuration, it is possible to change whether or not to continue the first performance and whether or not to output the notification information, depending on the importance of the notification information. Therefore, the notification information can be communicated to user U1 more appropriately according to its importance.

[0157] In the information processing method according to this embodiment, in the control processing (steps S506, S507), after displaying the notification information on the display unit 210, it is determined whether or not a predetermined action corresponding to the notification information has been performed by user U1. If the predetermined action has been performed, the display of the notification information is terminated. If the predetermined action has not been performed, the display of the notification information is continued until the predetermined action is performed.

[0158] With this configuration, the need to continue broadcasting information is determined based on whether or not user U1 performs a predetermined action corresponding to the broadcast information, thereby enabling the broadcast information to be communicated to user U1 more appropriately.

[0159] In the information processing method according to this embodiment, in the control processing (steps S506, S507, S509, S510), if a predetermined operation has not been performed after the notification information has been displayed on the display unit 210, and the importance of the notification information is higher than the standard, the display of the notification information will continue until the predetermined operation is performed, and if the importance of the notification information is lower than the standard, the display of the notification information will be terminated.

[0160] With this configuration, the system determines whether or not to continue broadcasting the information based on whether or not user U1 performs a predetermined action corresponding to the broadcast information, and the importance of the broadcast information. This allows the broadcast information to be communicated to user U1 more appropriately.

[0161] In the information processing method according to this embodiment, the agent device 700 is a device capable of rotating the display unit 210 in a predetermined direction around a predetermined rotation axis (neck portion 731, rotation support portions 732, 733). In the control process (step S508), if it is determined that user U1 is not looking at the display unit 210, the first effect is to perform an operation to rotate the display unit 210. For example, as shown in Figures 9(A) and 10(A), an operation is performed in which the display unit 210 rotates.

[0162] With this configuration, the display unit 210 rotates to attract the attention of the user U1, and then displays the notification information on the display unit 210, thereby more appropriately notifying the user U1 of the notification information.

[0163] In the information processing method according to this embodiment, agent devices 200 and 700 are devices that can be installed on vehicle C1, and the notification information is information relating to an object installed on vehicle C1 (e.g., seat belts, doors) or an object located outside vehicle C1 (e.g., tourist facilities, commercial facilities around vehicle C1).

[0164] With this configuration, agent devices 200 and 700 installed on vehicle C1 can be used to more appropriately notify user U1 of notification information related to vehicle C1.

[0165] The information processing device 110 is an information processing device that provides notification information to a user using agent devices 200 and 700, each having a display unit 210 (an example of a display device) that displays a face image. The information processing device 110 includes an operation state determination unit 121 (an example of a determination unit) that determines whether user U1 is looking at the display unit 210 based on the operation state information of user U1, and an agent control unit 122 (an example of a control unit) that controls the display mode of the display unit 210 based on the determination result of the determination process. If the agent control unit 122 determines that user U1 is not looking at the display unit 210, it causes the display unit 210 to execute a first effect to guide user U1's attention, and if it determines that user U1 is looking at the display unit 210, it causes the display unit 210 to display notification information.

[0166] With this configuration, after the first effect of displaying the face of an animal or the like on the display unit 210 is performed, notification information is displayed on the display unit 210, thereby attracting the attention of user U1, and thus the notification information can be conveyed to user U1 more appropriately.

[0167] The processing steps shown in this embodiment are merely examples for realizing this embodiment, and the order of some of the processing steps may be changed, some of the processing steps may be omitted, or other processing steps may be added, to the extent that this embodiment can be realized.

[0168] Furthermore, each process shown in this embodiment is executed based on a program that causes a computer to execute each processing procedure. For this reason, this embodiment can also be understood as an embodiment of a program that realizes the function of executing each of these processes, and a recording medium that stores that program. For example, an update process to add a new function to an information processing device can cause that program to be stored in the storage device of the information processing device. This makes it possible to have the updated information processing device perform each of the processes shown in this embodiment.

[0169] Although embodiments of the present invention have been described above, these embodiments only represent a part of the application examples of the present invention, and are not intended to limit the technical scope of the present invention to the specific configurations of the above embodiments. [Explanation of Symbols]

[0170] 100, 600 Information processing system, 110 Information processing device, 120 Control unit, 121 Operating state determination unit, 122 Agent control unit, 130 Storage unit, 131 Agent information DB, 132 Notification information DB, 133 Map information DB, 140 Communication unit, 200, 700 Agent equipment, 210 Display unit, 220 Sound output unit, 730 Drive unit

Claims

1. An information processing method that provides notification information about a predetermined object to a user using an agent device having a display device that displays a facial image, A determination process that determines whether the user is looking at the display device based on the user's operating status information, Includes a control process that controls the display mode of the display device based on the determination result of the determination process, In the control process described above, if it is determined that the user is not looking at the display device, the display device is instructed to perform a first effect to guide the user's attention; if it is determined that the user is looking at the display device, the agent device is instructed to perform a second effect to guide the user's attention toward the object, which involves displaying the face image of the user looking toward the object on the display device, and then the notification information is displayed on the display device. Information processing methods.

2. An information processing method for providing notification information about a predetermined object to a user using an agent device having a display device that displays a facial image and an indicator unit that can point in a predetermined direction by a predetermined operation, A determination process that determines whether the user is looking at the display device based on the user's operating status information, Includes a control process that controls the display mode of the display device based on the determination result of the determination process, In the control process described above, if it is determined that the user is not looking at the display device, the display device is instructed to perform a first action to guide the user's attention; if it is determined that the user is looking at the display device, the agent device is instructed to perform a second action to guide the user's attention toward the object, which is an action in which the indicator unit points to the object, and then the notification information is displayed on the display device. Information processing methods.

3. The information processing method according to claim 1 or 2, In the control process described above, if it is determined that the user is not looking at the display device, the first performance is continued until it is determined that the user is looking at the display device. Information processing methods.

4. The information processing method according to claim 3, In the control process described above, if it is determined that the user is not looking at the display device, and the importance of the notification information is higher than the standard, the first performance is continued until it is determined that the user has looked at the display device; and if the importance of the notification information is lower than the standard, the first performance is terminated and the notification of the notification information is stopped. Information processing methods.

5. The information processing method according to claim 1 or 2, In the control process described above, after displaying the notification information on the display device, it is determined whether or not the user has performed the predetermined action corresponding to the notification information. If the prescribed action is performed, the display of the notification information will be terminated; if the prescribed action is not performed, the display of the notification information will continue until the prescribed action is performed. Information processing methods.

6. The information processing method according to claim 5, In the control process described above, if the predetermined operation has not been performed after the notification information has been displayed on the display device, and the importance of the notification information is higher than the standard, the display of the notification information will continue until the predetermined operation is performed; and if the importance of the notification information is lower than the standard, the display of the notification information will be terminated. Information processing methods.

7. The information processing method according to claim 1 or 2, The agent device is a device capable of rotating the display device around a rotation axis, In the control process described above, if it is determined that the user is not looking at the display device, the first effect is to perform an action to rotate the display device. Information processing methods.

8. The information processing method according to claim 1 or 2, The aforementioned agent device is a device that can be installed in a vehicle. The aforementioned notification information is information relating to an object installed on the vehicle or an object located outside the vehicle. Information processing methods.

9. An information processing device that provides notification information about a predetermined object to a user using an agent device having a display device that displays a facial image, A determination unit that determines whether the user is looking at the display device based on the user's operating status information, The system comprises a control unit that controls the display mode of the display device based on the determination result from the determination unit, If the control unit determines that the user is not looking at the display device, it causes the display device to perform a first effect to attract the user's attention; if the control unit determines that the user is looking at the display device, it causes the agent device to perform a second effect to attract the user's attention towards the object, which involves displaying the face image of the user looking towards the object on the display device, and then displays the notification information on the display device. Information processing device.

10. An information processing device that provides notification information about a predetermined object to a user using an agent device having a display device that displays a face image and an indicator unit that can point in a predetermined direction by a predetermined operation, A determination unit that determines whether the user is looking at the display device based on the user's operating status information, The system comprises a control unit that controls the display mode of the display device based on the determination result from the determination unit, If the control unit determines that the user is not looking at the display device, it causes the display device to perform a first action to attract the user's attention; if the control unit determines that the user is looking at the display device, it causes the agent device to perform a second action to attract the user's attention towards the object, which is an action of pointing to the object using the indicator unit, and then displays the notification information on the display device. Information processing device.

Images