Information processing method and information processing apparatus

Abstract

To adequately indicate a position of an object to be a notification object when a notification about information related to the object is provided to a user in a vehicle with the use of an agent.SOLUTION: An information processing method for notifying a user U1 in a vehicle C1 on which an agent unit 200 and a display device group 300 for displaying an agent image can be installed of information related to an object includes control processing to control an operation state of the agent unit 200 and a display state of the display device group 300. In the control processing, when a notification about the information related to the object is provided to the user U1, switching is performed between a first notification to indicate the object with the use of the agent unit 200 so as to provide a notification about the information and a second notification to indicate the object with the use of the agent image so as to provide a notification about the information on the basis of at least one among a positional relation between the object and the agent unit 200, the number of objects, and a motion state of the object.SELECTED DRAWING: Figure 8

Description

Technical Field

[0001] The present invention relates to an information processing method and an information processing apparatus for notifying a user riding in a vehicle of information regarding an object.

Background Art

[0002] Conventionally, there are technologies for performing various notifications using robots. For example, as a small robot disposed on a dashboard, a technology has been proposed that uses an anthropomorphic agent device to prompt a driver to take an avoidance action (see, for example, Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] Here, it is known that an agent device can produce an effective effect on a user by acting as a three-dimensional robot and can attract the user's attention more. Therefore, it is conceivable to use an agent device to notify a user riding in a vehicle of information regarding some object However, depending on the operation of the agent device, it is also assumed that it is difficult to accurately indicate the position of the object to be notified.

[0005] An object of the present invention is to accurately indicate the position of an object to be notified when notifying a user riding in a vehicle of information regarding the object using an agent.

Means for Solving the Problems

[0006] One aspect of the present invention is an information processing method for informing a user riding in a vehicle that can be equipped with an agent device having a member capable of pointing in a predetermined direction by a predetermined operation, and a display device that displays an agent image corresponding to the agent device as a display mode indicating an object located inside or outside the vehicle, of an object. This information processing method includes control processing to control the operating state of the agent device and the display state of the display device, and in the control processing, when informing the user of information, the positional relationship between the object and the agent device, the number of objects, and the object Rate of change Based on at least one of the above, the system switches between a first notification, which uses an agent device to point to the object and provide information, and a second notification, which uses an agent image to point to the object and provide information. [Effects of the Invention]

[0007] According to the present invention, when an agent is used to inform a user riding in a vehicle about an object, the location of the object to be informed can be accurately indicated. [Brief explanation of the drawing]

[0008] [Figure 1] Figure 1 is a simplified diagram showing an example of the interior layout of a vehicle. [Figure 2] Figure 2 is a simplified diagram showing an example of the interior layout of a vehicle. [Figure 3] Figure 3 shows an example of the interior layout of a vehicle. [Figure 4] Figure 4 is a simplified perspective view showing an example of the external configuration of an agent device. [Figure 5] Figure 5 shows an example of the system configuration of an information processing system installed in a vehicle. [Figure 6] Figure 6 shows an example of an image of the area around a vehicle displayed on the display unit of the first display device when the vehicle is parked in a designated parking space. [Figure 7]Figure 7 shows an example of a map image displayed on the display unit of the first display device when a destination is set using the navigation device installed in the vehicle. [Figure 8] Figure 8 is a flowchart showing an example of agent switching processing in an information processing device. [Figure 9] Figure 9 is a flowchart showing an example of agent switching processing in an information processing device. [Figure 10] Figure 10 shows an example of a guidance image displayed on the display unit of the first display device when a user is searching for vehicle equipment. [Figure 11] Figure 11 shows an example of how an agent image is displayed on the display unit of the second display device when communication with a rear-seat user becomes necessary. [Figure 12] Figure 12 is a flowchart showing an example of agent switching processing in an information processing device. [Figure 13] Figure 13 is a flowchart showing an example of agent switching processing in an information processing device. [Figure 14] Figure 14 is a flowchart showing an example of agent switching processing in an information processing device. [Figure 15] Figure 15 is a flowchart showing an example of agent switching processing in an information processing device. [Figure 16] Figure 16 shows an example of display transitions when displaying information on the front windshield display area according to the number of objects in front of the vehicle. [Figure 17] Figure 17 is a flowchart showing an example of agent switching processing in an information processing device. [Figure 18] Figure 18 is a flowchart showing an example of agent switching processing in an information processing device. [Modes for carrying out the invention]

[0009] Hereinafter, embodiments of the present invention will be described while referring to the accompanying drawings.

[0010] [Installation Example of Agent Device] FIG. 1 and FIG. 2 are diagrams showing a simplified configuration example of the interior of the vehicle C1. In FIG. 1, the front side of the driver's seat 11 and the passenger seat 12 (see FIG. 2) is shown as an external configuration example when viewed from the rear side of the vehicle C1, and in FIG. 2, the front side of the rear seat is shown as an external configuration example when viewed from the rear side of the vehicle C1. Also, in FIGS. 1 and 2, for ease of explanation, illustrations other than the dashboard 2, the steering wheel 3, the front windshield 4, the rearview mirror 5, the side mirrors 6, 7, the USB (Universal Serial Bus) port 8, the in-vehicle camera 101, the agent device 200, the first display device 310, and the second display device 320 are omitted. Note that the USB port 8 means an insertion port into which a plug of an electronic device for USB connection is inserted.

[0011] The agent device 200 is a small robot installed on the dashboard 2 of the vehicle C1. In this embodiment, an example of a robot imitating a human is shown as the agent device 200. Note that in FIGS. 1 and 2, an example of installing the agent device 200 on the dashboard 2 is shown, but it is not limited thereto. For example, the agent device 200 may be installed on the upper part of the front windshield 4. Also, in FIGS. 1 and 2, an example of a robot imitating a human is shown as the agent device 200, but it is not limited thereto. For example, a robot imitating an animal such as a rabbit or a pig, a robot imitating a virtual creature (for example, the face of an anime character), or a robot imitating another object (for example, a TV-type device or a radio-type device) may be used as the agent device 200.

[0012] The agent device 200 executes various operations based on instructions from the information processing device 110 (see FIG. 5). For example, the agent device 200 outputs various types of information related to driving support when the user U1 performs a driving operation under the control of the information processing device 110. As this driving support, notification of moving objects ahead or behind is assumed. For example, as notification of a moving object ahead, it is possible to output an audio message such as "There is a railroad crossing ahead, so be careful" or "There is a person ahead". In this way, the agent device 200 executes driving support.

[0013] Here, notification means conveying or informing some kind of information. Note that the notification shown in this embodiment may also be referred to as notification, transmission, etc.

[0014] The first display device 310 is a display device provided at the central portion of the dashboard 2 of the vehicle C1 so as to be visible to the passengers sitting in the driver's seat 11 and the front passenger seat 12. The second display device 320 is a display device provided near the center of the ceiling inside the vehicle C1 so as to be visible to the passengers sitting in the rear seats of the vehicle C1. As the second display device 320, for example, a flip-down type display device, that is, a foldable display device attached to the ceiling using a support member 41 can be used. In the examples shown in FIGS. 1 and 2, an example of installing at least two first display devices 310 and second display devices 320 in the vehicle C1 is shown, but one or three or more display devices may be installed in the vehicle C1. Also, the installation locations of each display device are not limited to the examples shown in FIGS. 1 and 2 and can be changed as appropriate.

[0015] The first display device 310 and the second display device 320 display various images based on the control of the information processing device 110 (see Figure 5). For example, touch panels that can accept user operations by touch operation can be used as the first display device 310 and the second display device 320. For example, various devices equipped with a user interface such as a display unit (e.g., smartphones, tablet terminals, car navigation systems, IVI (In-Vehicle Infotainment)) can be used as the first display device 310 and the second display device 320. In addition, the first display device 310 and the second display device 320 can display agent images corresponding to the agent device 200 on the display units 311 and 321, and use these agent images to provide information to the user U1. Examples of agent image display are shown in Figures 6, 7, 10, and 11.

[0016] The in-vehicle camera 101 is installed on the ceiling inside the vehicle C1 and captures images of subjects inside the vehicle C1 to generate images (image data). The in-vehicle camera 101 is composed of, for example, one or more camera devices or image sensors capable of capturing images of subjects. For example, the in-vehicle camera 101 can be installed on the upper part of the front windshield 4, i.e., above the rearview mirror 5. In this example, an example with at least one in-vehicle camera 101 is shown, but two or more imaging devices may be used, and images from all or some of these imaging devices may be used. Furthermore, the installation location of each imaging device is not limited to the example shown in Figure 1 and can be changed as appropriate. In addition, one or more devices capable of acquiring images of subjects present in all directions around the vehicle C1 and subjects inside the vehicle C1, such as a 360-degree camera, may be used.

[0017] [Example of HUD configuration] Figure 3 shows an example of the interior configuration of vehicle C1. In Figure 3, a simplified cross-section of the interior of vehicle C1 is shown as viewed from the left side in the left-right direction of vehicle C1, and other components are omitted from the illustration except for the steering wheel 3, front windshield 4, in-cabin camera 101, and third display device 330.

[0018] As shown in Figure 3, a third display device 330 for realizing a Head-Up Display (HUD) for the vehicle C1 is provided at the top of the dashboard 2, near the boundary with the front windshield 4.

[0019] The third display device 330 is a display device, such as a projector or optical system, for realizing a HUD display that projects light onto the display area 331 of the front windshield 4 and uses the reflected light to show a virtual image to the user U1. That is, the light projected from the third display device 330 onto the display area 331 of the front windshield 4 is reflected by the front windshield 4, and the reflected light is directed towards the user U1's eyes. The reflected light projected onto the display area 331 and entering the user U1's eyes is displayed superimposed on the actual object visible through the front windshield 4. In this way, the third display device 330 realizes a HUD display by displaying a virtual image using the front windshield 4.

[0020] For example, the third display device 330 displays various vehicle information about vehicle C1 in the display area 331 based on the control of the information processing device 110 (see Figure 5). For example, as shown in Figure 16(B), agent images 461 to 463 are displayed to notify the vehicle C1 of multiple pedestrians 451 to 453 located in front of it.

[0021] The front windshield 4 is a shield (for example, a windshield) provided at the front of the vehicle C1 and functions as a display medium for the vehicle C1's HUD.

[0022] Furthermore, the front windshield 4 may be formed using glass or other materials. For example, when laminated glass consisting of a rear glass, a front glass, and an interlayer is used as the front windshield 4, the refraction of light can be controlled by shaping the cross section of the interlayer into a wedge shape. That is, the laminated glass is formed so that the rear glass and the front glass form a V-shape in cross-sectional view. This allows the front windshield 4 to function as a display medium for the HUD.

[0023] Figure 3 shows an example where the front windshield 4 functions as a HUD, but other shields may also function as HUDs. For example, display devices may be installed at predetermined positions near the rear shield and side shields, and the rear shield and side shields may function as HUDs.

[0024] [Example of external configuration of agent device] Figure 4 is a simplified perspective view showing an example of the external configuration of the agent device 200. In this embodiment, when the agent device 200 is in the ON state, an example is shown in which it switches to either the active state or the dormant state. That is, the information processing device 110 (see Figure 5) executes control to switch the operating state of the agent device 200 between the active state and the dormant state. It is possible to turn on the agent device 200 in response to the ON operation of the vehicle C1. It is also possible to turn on the agent device 200 in response to user operation related to the agent device 200. Note that the ON / OFF operation of the vehicle C1 means the ON or OFF operation of the start key related to starting or stopping the vehicle C1.

[0025] Figure 4(A) shows a front view of the agent device 200 in an active state. Figure 4(B) shows a front view of the agent device 200 in a dormant state. Note that the agent device 200 shown in Figure 4(A) corresponds to the agent device 200 shown in Figures 1 and 2.

[0026] The agent device 200 is a robotic device in which a roughly box-shaped main body 210 is provided with a roughly spherical face 220 on top of the main body 210, and rod-shaped arms 211 and 212 are provided on the left and right sides of the main body 210. Various images, such as patterns 213, are displayed on the surface of the main body 210. Various images, such as a mouth 221 and eyes 222, are displayed on the surface of the face 220. In this embodiment, an example is shown in which the agent device 200 is installed on the dashboard 2 of a vehicle C1, with the eyes 222 facing inwards. Furthermore, the side of the agent device 200 in which the mouth 221 and eyes 222 are displayed will be referred to as the front side of the agent device 200, and the side opposite to the front side of the agent device 200 will be referred to as the rear side of the agent device 200. Furthermore, in the agent device 200, the side in which the arms 211 are provided will be referred to as the right side of the agent device 200, and the side in which the arms 212 are provided will be referred to as the left side of the agent device 200.

[0027] Figure 4(C) shows a front view of the agent device 200 when the face 220 is facing to the right and the tip of the arm 211 is pointing to the right. Figure 4(D) shows a front view of the agent device 200 when the face 220 is facing to the left and the tip of the arm 212 is pointing to the left. Figure 4(E) shows a front view of the agent device 200 when the face 220 is facing upward and the tip of the arm 211 is pointing upward.

[0028] The face portion 220 rotates vertically and horizontally around the central part of the face portion 220 as the pivot point of the agent device 200. For example, as shown in Figures 4(C) and 4(D), the face portion 220 rotates horizontally. Also, for example, as shown in Figures 4(B) and 4(E), the face portion 220 rotates vertically. Thus, the agent device 200 is provided with a drive device for driving the face portion 220. This makes it possible to rotate the face portion 220 of the agent device 200 horizontally, rotate the face portion 220 vertically, swing the face portion 220 horizontally, and swing the face portion 220 vertically.

[0029] The arm 211 rotates in the forward / backward, left / right, and up / down directions of the agent device 200, with the mounting portion of the arm 211 on the main body 210 as its pivot point. Similarly, the central portion of the arm 211 rotates in the forward / backward, left / right, and up / down directions of the agent device 200, with its central portion as its pivot point.

[0030] As described above, a drive device for driving the arm 211 is provided at the mounting portion and central portion of the arm 211. This makes it possible to move the arm 211 up and down, move the arm 211 back and forth, swing the arm 211 up and down, and swing the arm 211 back and forth. Similarly, a drive device for driving the arm 212 is provided at the mounting portion and central portion of the arm 212. This makes it possible to move the arm 212 up and down, move the arm 212 back and forth, swing the arm 212 up and down, and swing the arm 212 back and forth.

[0031] Furthermore, the drive devices that operate each part of the agent device 200 perform their respective operations based on the control of the information processing device 110 (see Figure 5).

[0032] Furthermore, in this embodiment, for the sake of ease of explanation, an example is shown in which arm portions 211 and 212 are used as members for pointing to an object. However, the agent device 200 may be provided with members that mimic a hand, fingers, and an arm, and these may be used to point to an object. Alternatively, the agent device 200 may be provided with a member that mimics at least one of the hand, fingers, and arm, and these may be used to point to an object.

[0033] Furthermore, at least a portion of the surface of the main body 210 and the face portion 220 is equipped with a display medium capable of displaying various images. Various display media such as light-emitting sheets, liquid crystal displays, and organic EL (Electro-Luminescence) can be used as this display medium. For example, as shown in Figures 4(A), (C) to (E), a normal eye portion 222 can be displayed. Also, for example, as shown in Figure 4(B), a closed eye portion 222 can be displayed. Also, for example, as shown in Figures 4(A) and (B), a normal mouth portion 221 can be displayed. Also, for example, as shown in Figures 4(C) to (E), an angry-looking mouth portion 221 can be displayed. In this way, in this embodiment, the agent device 200 can be changed by an operation mode that changes at least a portion of each part of the agent device 200 and a display mode that changes the display state of at least a portion of the surface of the agent device 200. Furthermore, the user U1 of the vehicle C1 who sees the change can recognize that the agent device 200 is making some kind of movement. Furthermore, the agent device 200 in a dormant state can be set to, for example, a state where the eye portion 222 disappears and the face portion 220 is lowered, or a shutdown state.

[0034] The operating modes of these agent devices 200 are merely examples, and other operating modes are possible. For example, the agent device 200 can be slid within a predetermined range on the dashboard 2. For example, the agent device 200 can be slid in the forward, backward, left, and right directions. Furthermore, for example, it is possible to create an effect where the agent device 200 appears to be temporarily floating.

[0035] [Example of an information processing system configuration] Figure 5 shows an example of the system configuration of the information processing system 100 installed in vehicle C1.

[0036] The information processing system 100 includes an in-vehicle camera 101, a front camera 102, a rear camera 103, a vehicle speed sensor 104, a position information acquisition sensor 105, an audio input unit 106, sensors 107, an information processing device 110, an agent device 200, and a display device group 300. The information processing device 110 and the agent device 200 are connected by a communication method using wired or wireless communication. The information processing device 110 is also connected to a network using a communication method using wireless communication. The network is a public telephone network, the Internet, or other network. The agent device 200 may also be connected to the network using a communication method using wireless communication.

[0037] The in-vehicle camera 101, the front camera 102, and the rear camera 103 capture images of subjects and generate images (image data) based on the control of the information processing device 110, and output image information related to the generated images to the information processing device 110. The in-vehicle camera 101 is installed inside the vehicle C1 and captures images of subjects inside the vehicle C1 and generates images (image data). The front camera 102 is installed in front of the vehicle C1 and captures images of subjects in front of the vehicle C1 and generates images (image data). The rear camera 103 is installed behind the vehicle C1 and captures images of subjects behind the vehicle C1 and generates images (image data). The in-vehicle camera 101, the front camera 102, and the rear camera 103 are composed of, for example, one or more camera devices or image sensors capable of capturing images of subjects. At least one of these cameras may be, for example, one or more devices capable of acquiring images of subjects present in all directions around the vehicle C1 and subjects inside the vehicle C1, such as a 360-degree camera.

[0038] The vehicle speed sensor 104 is a sensor that acquires the speed of vehicle C1 and outputs the acquired speed of vehicle C1 to the information processing device 110.

[0039] The position information acquisition sensor 105 acquires position information regarding the location of vehicle C1 and outputs the acquired position information to the information processing device 110. For example, this can be implemented using a GNSS receiver that acquires position information using GNSS (Global Navigation Satellite System). The position information includes various position data such as latitude, longitude, and altitude at the time of receiving the GNSS signal. Position information may also be acquired by other methods. For example, position information may be derived using information from access points or base stations in the vicinity. Position information may also be acquired using beacons. For example, based on the information acquired by the vehicle speed sensor 104 or the position information acquisition sensor 105, the state of vehicle C1, such as driving, stopped, or reversing, can be determined.

[0040] The audio input unit 106 is located inside or outside the vehicle C1 and, based on the control of the information processing device 110, acquires sounds from the surrounding area of ​​the vehicle C1 (sounds from outside the vehicle) and sounds from inside the vehicle, and outputs sound information related to the acquired sounds to the information processing device 110. For example, one or more microphones or sound acquisition sensors can be used as the audio input unit 106.

[0041] The sensors 107 are various sensors installed on the vehicle C1, and the detection information acquired by each sensor is output to the information processing device 110. Examples of sensors include acceleration sensors and seating sensors. For example, sonar, lidar (Light Detection and Ranging), and radar can be used as sensors 107 to detect objects present around the vehicle C1. Sonar, lidar, and radar are sensors that measure objects present around the vehicle C1 and the distance to those objects. For example, the distance to an object can be measured by irradiating the area around the vehicle C1 with laser light, millimeter waves, etc., and receiving and observing the reflected waves. These are just examples, and other sensors may be used. Also, only some of these sensors may be used.

[0042] The control unit 120 controls each part based on various programs stored in the memory unit 130. The control unit 120 is implemented by a processing unit such as a CPU (Central Processing Unit). The vehicle ECU (Electronic Control Unit) of vehicle C1 may be used as the control unit 120, or a processing unit different from the vehicle ECU may be provided as the control unit 120.

[0043] The control unit 120 executes control processing to control the operating state of the agent device 200 and the display state of the display device group 300 based on the information output from the in-vehicle camera 101, front camera 102, rear camera 103, vehicle speed sensor 104, position information acquisition sensor 105, voice input unit 106, sensors 107, etc. Specifically, the control unit 120 comprises a switching determination unit 121, an agent device control unit 122, and a display control unit 123.

[0044] The switching determination unit 121 performs a switching process to switch between a first notification, in which the agent device 200 is used to point to the target object and provide information, and a second notification, in which the agent image is used to point to the target object and provide information. For example, the switching determination unit 121 performs the switching process based on information output from the in-vehicle camera 101, front camera 102, rear camera 103, vehicle speed sensor 104, position information acquisition sensor 105, voice input unit 106, sensors 107, etc. For example, the switching determination unit 121 performs the switching process based on at least one of the positional relationship between the target object and the agent device 200, the number of targets, and the operating state of the targets.

[0045] For example, the switching determination unit 121 calculates the distance between the tips of the arms 211 and 212 and the object when the position of the object is notified by the movement of the arms 211 and 212. Then, if the distance between the tips of the arms 211 and 212 and the object is greater than a predetermined value, the switching determination unit 121 performs a switching process to execute a second notification. Examples of this switching process are shown in Figures 8, 12, 14, and 17. In this way, before notifying the user U1 of information about the object, the switching determination unit 121 calculates the operation of the agent device 200 when the position of the object is notified by the movement of the arms 211 and 212 of the agent device 200, and executes a switching process based on that operation. Note that the operation of the agent device 200 shown here means a pointing operation in which the tips of the arms 211 and 212 of the agent device 200 are pointed towards the position of the object. The distance between the tip of each arm 211 or 212 and the object can be determined by subtracting the length of the arm 211 or 212 from the distance between the agent device 200 and the object. Alternatively, instead of calculating the distance between the tip of each arm 211 or 212 and the object, the distance between the agent device 200 and the object may be calculated and used.

[0046] Furthermore, for example, the switching determination unit 121 determines whether or not a component of vehicle C1 exists as an obstacle between the tips of the arms 211 and 212 and the object when the position of the object is notified by the movement of the arms 211 and 212. If an obstacle exists between the tips of the arms 211 and 212 and the object, the switching determination unit 121 performs a switching process to execute a second notification. Examples of this switching process are shown in Figures 8, 12, and 14.

[0047] Furthermore, for example, the switching determination unit 121 calculates the rate of change (i.e., movement speed) of the object to be notified, and if the rate of change of the object is greater than a predetermined value, it performs a switching process to execute a second notification. The rate of change of the object to be notified can be detected based on images acquired by the front camera 102 or detection information acquired by the sensors 107.

[0048] Furthermore, for example, if there are multiple objects to indicate, the switching determination unit 121 performs a switching process to execute a second notification. An example of this switching process is shown in Figure 17.

[0049] Furthermore, for example, if the content displayed by the display device group 300 includes the object being pointed to, the switching determination unit 121 performs a switching process to execute a second notification. Examples of this switching process are shown in Figures 8, 12, and 14.

[0050] The agent device control unit 122 controls the operating state of the agent device 200. For example, if the switching determination unit 121 performs a switching process for issuing a first notification, the agent device control unit 122 performs a first notification by pointing to the target object using the agent device 200 and notifying information.

[0051] The display control unit 123 controls the display state of each display device (first display device 310, second display device 320, third display device 330) included in the display device group 300. For example, if the switching determination unit 121 performs a switching process for providing a second notification, the display control unit 123 displays an agent image on at least one of the display devices included in the display device group 300 and performs a second notification by pointing to the target object using this agent image to provide information. Examples of this display are shown in Figures 6, 7, 10, 11, and 16.

[0052] The memory unit 130 is a storage medium for storing various types of information. For example, the memory unit 130 stores various types of information necessary for the control unit 120 to perform various processes (e.g., control programs, map information DB 131, equipment information DB 132). The memory unit 130 also stores various types of information acquired via the communication unit (not shown). As the memory unit 130, for example, ROM (Read Only Memory), RAM (Random Access Memory), HDD (Hard Disk Drive), SSD (Solid State Drive), or a combination thereof can be used.

[0053] Map information DB131 stores map information such as road information necessary for route guidance of vehicle C1. This map information includes road gradient, road intersections, number of road lanes, road width information, and road topography information. The map information also includes road signs indicating speed limits, one-way streets, etc., pedestrian crossings, lane markings, etc. The map information may also include information on road structures (e.g., traffic lights, utility poles), buildings and other facilities, and surrounding tourist information. Furthermore, facility information DB132 stores facility images showing each facility installed in vehicle C1, associated with images of the interior of the vehicle including that facility (guidance images). Note that map information DB131 and facility information DB132 may be stored in the storage unit 130 of vehicle C1, or they may be obtained from external devices via a predetermined network, such as the Internet.

[0054] Here, since the agent device 200 installed on the dashboard 2 is a three-dimensional robot that has been made into an artificial being, it is assumed that it has a greater presence that can be recognized by the user U1 riding in the vehicle C1 than the image displayed on the first display device 310. For this reason, for example, when it is necessary to notify the user U1 of an object, it is conceivable to use the agent device 200 to do so. For example, as shown in Figures 4(C) to (E), it is conceivable that the tip of the arm parts 211 and 212 could point in the direction of the object to be notified, thereby notifying the user U1 of the object's location. However, for example, when using the arm parts 211 and 212 of the agent device 200 to notify the location of an object that is far from the agent device 200, it is assumed that it will be difficult to accurately indicate the location of the object, and therefore difficult to make the user U1 recognize the object. Also, for example, if there is an obstacle between the agent device 200 and the object, it is assumed that it will be difficult to accurately indicate the location of the object, and therefore difficult to make the user U1 recognize the object. Thus, when using the agent device 200 installed on the dashboard 2 to notify of an object, there may be situations where it is difficult to notify of the object depending on the positional relationship between the object and the agent device 200, the number of objects, the operating state of the objects, etc. Therefore, in this embodiment, the display of agent images by each display device of the display device group 300 is switched depending on the positional relationship between the object and the agent device 200, the number of objects, the operating state of the objects, etc., in order to appropriately notify of the object.

[0055] [Example of an alert system that notifies the user of an object when the vehicle is reversing] Figure 6 shows an example of an image of the area around the vehicle displayed on the display unit 311 of the first display device 310 when the vehicle C1 is parked in a predetermined parking space. Figure 6 shows an example where an object is detected in the user U1's blind spot while the vehicle C1 is reversing when it is parked in a predetermined parking space. The object could be, for example, a person, a bicycle, or an animal (e.g., a cat or dog). For example, the pillar of the vehicle C1 could become an obstacle and create a blind spot for the user U1. Objects around the vehicle C1 can also be detected by the sensors included in the sensors 117, such as sonar, lidar, and radar.

[0056] Figure 6 shows an example of displaying vehicle surrounding images, specifically a top image 400 of vehicle C1 viewed from above, and a rear image 410 of vehicle C1 viewed from the rear of vehicle C1. The top image 400 can be displayed using known image processing techniques. The rear image 410 can be displayed based on an image acquired by the rear camera 103.

[0057] The top view image 400 displays a vehicle image 401 corresponding to vehicle C1, vehicle images 402 and 403 corresponding to vehicles parked on either side of vehicle C1's parking space, white lines 404 indicating the area of ​​each parking space, and an assistance image 405 to assist user U1 in driving.

[0058] The rear view image 410 displays an assistance image 411 to support user U1's driving, vehicle images 412 and 413 corresponding to vehicles parked on both sides of vehicle C1's parking space, and white lines 414 indicating the boundaries of each parking space. An agent image 415 is also displayed in the rear view image 410. The vehicle images 412 and 413 and the white lines 414 included in the rear view image 410 are displayed based on images acquired by the rear camera 103.

[0059] For example, if vehicle C1 is reversing and an object (e.g., a person or a cat) enters the area behind vehicle C1, the object can be detected by the sensors 117. In this case, the object detected by the sensors 117 can be notified to user U1. For example, the arm 211 of the agent device 200 could be used to indicate the location of the object to the left rear of vehicle C1. However, even if the arm 211 of the agent device 200 is used to indicate the location of the object to the left rear of vehicle C1, it is expected that only a rough direction will be transmitted to user U1 because of the distance between the agent device 200 and the object to the left rear of vehicle C1. In other words, even if the arm 211 of the agent device 200 is used to indicate the location of the object to the left rear of vehicle C1, it is expected that the accuracy of the object's location will be low. In this case, for example, if the object is located in a blind spot relative to user U1, user U1 may not be able to see it directly and may conclude that nothing was there. Furthermore, it is anticipated that it may be difficult to properly notify user U1 of the presence of the object. Thus, if an object enters a blind spot behind vehicle C1 while vehicle C1 is reversing, it is anticipated that it will be difficult for agent device 200 to inform user U1 of the direction from which the object entered.

[0060] Therefore, in this embodiment, if an object enters a blind spot behind vehicle C1 while vehicle C1 is reversing, the presence of that object is displayed on the first display device 310 to inform user U1. Specifically, an agent image 415 is displayed on the rear image 410, and the agent image 415 indicates the position of the object that has entered behind vehicle C1, thereby informing user U1. In other words, in this embodiment, dangerous objects present around vehicle C1 are detected, and if those objects are located in a position invisible to user U1, the presence and position of those objects are appropriately notified to user U1.

[0061] The example shown in Figure 6 illustrates a case where an object is detected on the right side of vehicle C1 (vehicle image 413). In such a case, the agent image 415 is displayed in a way that indicates the location where the object was detected. Specifically, the agent image 415 extends its left arm 416 to the left to indicate the location of the object on that side of vehicle (vehicle image 413). Alternatively, the detection of an object behind vehicle C1 may be notified by voice information S1. The location of the object behind vehicle C1 can be detected by the various sensors included in the sensors 117, such as sonar, lidar, and radar.

[0062] In this manner, if an object (e.g., a child, a cat) is detected in a location not visible from the agent device 200 installed on the dashboard 2, the agent device 200 switches to the agent image 415. Then, the agent image 415 is used to inform the user U1 of the object's location. In this case, for example, voice information S1 such as "Check the surrounding area for safety, and be careful as something has entered this blind spot" may be output.

[0063] The switching timing for agent device 200 and agent image 415 will be explained in detail with reference to Figures 8 and 9.

[0064] [Example of a navigation system notifying the destination] Figure 7 shows an example of a map image 420 displayed on the display unit 311 of the first display device 310 when a destination is set using the navigation device installed in vehicle C1. Figure 7 shows an example where the first display device 310 is equipped with a navigation function.

[0065] As shown in Figure 7, the map image 420 displays a destination image 421 indicating the destination on a map that includes the area around the destination set by user U1. Regarding the method of setting the destination, known setting techniques can be used. For example, the destination can be set by the user entering text indicating the destination. Alternatively, the destination can be set by the user specifying the destination on the map. Furthermore, if the user speaks a voice indicating the destination, the voice can be acquired using speech recognition technology to set the destination.

[0066] For example, if user U1 sets a destination by voice using voice recognition technology, it is possible to notify user U1 of the location of that destination. In this case, for example, the arm parts 211 and 212 of the agent device 200 could be used to point to the location of the destination displayed on the display unit 311 of the first display device 310. However, even if the arm parts 211 and 212 of the agent device 200 are used to point to the location of the destination displayed on the display unit 311, it is expected that the accuracy of the destination location will be low because of the distance between the agent device 200 and the display unit 311.

[0067] Therefore, in this embodiment, when user U1 sets a destination using voice recognition technology or the like, the agent image 422 indicates the location of the destination on the map to inform user U1. Specifically, the agent image 422 is displayed on the map image 420, and the location of the destination is indicated to user U1 by the left arm 423 of the agent image 422.

[0068] In the example shown in Figure 7, the display mode is such that the left arm 423 of the agent image 422 points to the destination image 421, which is overlaid on the map image 420. Specifically, the agent image 422 extends its left arm 423 to the left to point to the destination image 421. Alternatively, the setting of a destination may be notified by voice information S2.

[0069] Here, the switching timing between the agent device 200 and the agent image 422 will be explained. The switching timing from the agent device 200 to the agent image 422 can be, for example, the timing when the user has finished setting a destination and information about the destination is provided to the user for confirmation of that destination. For example, if user U1 sets a destination using voice recognition technology, the switching timing can be the timing when user U1 speaks a voice indicating the destination and that destination is recognized by the information processing device 110. In this way, the switching timing from the agent device 200 to the agent image 422 can be the timing when user U1 should be directed to an object (for example, a destination image 421) included in the content (for example, a map image 420) displayed on the display unit 311 of the first display device 310.

[0070] On the other hand, the timing of the switch from agent image 422 to agent device 200 can be, for example, when the user has finished confirming the destination. The user can confirm the destination by performing some operation, such as pressing a confirmation completion button or by providing an audio confirmation of completion.

[0071] [Example of vehicle operation when reversing] Figure 8 is a flowchart illustrating an example of agent switching processing in the information processing device 110. This agent switching processing is executed based on a program stored in the storage unit 130. Similarly, the agent switching processing shown in Figures 9, 12 to 15, 17, and 18 is also executed based on a program stored in the storage unit 130. This agent switching processing is executed when the vehicle C1 is operated in reverse, for example, when the shift lever is operated to the R range. This agent switching processing will be explained with reference to Figures 1 to 7 as appropriate.

[0072] In step S501, the switching determination unit 121 calculates the risk potential of vehicle C1 based on the information output from the rear camera 103, sensors 107, etc. This risk potential is an indicator of various potential risks of vehicle C1. Various potential risks of vehicle C1 include, for example, the possibility of collision between vehicle C1 and surrounding objects. In this case, the value indicating the possibility of collision can be used as the risk potential. For example, when vehicle C1 is reversing to park, the risk potential will be high when an obstacle, moving object, etc. is detected behind vehicle C1. Also, for example, when vehicle C1 is in motion, the risk potential will be high when a pedestrian, etc. is detected on a crosswalk in front of vehicle C1. The possibility of collision between vehicle C1 and an object detected around vehicle C1 can be calculated based on, for example, the relative speed of the object to vehicle C1, the distance between vehicle C1 and the object, etc.

[0073] In the example shown in Figure 8, the risk potential is considered high when the object is in the blind spot of user U1 in the direction of travel of vehicle C1.

[0074] In step S502, the switching determination unit 121 determines whether or not an object exists in the user U1's blind spot in the direction of travel of vehicle C1, based on the information output from the rear camera 103, sensors 107, etc. That is, it determines whether or not the risk potential calculated in step S501 is above a threshold. If the risk potential is above a threshold, it is determined that an object exists in the user U1's blind spot in the direction of travel of vehicle C1. If an object exists in the user U1's blind spot, the process proceeds to step S503. On the other hand, if no object exists in the user U1's blind spot, the agent switching process is terminated.

[0075] In step S503, the agent device control unit 122 executes a stop animation process to stop the agent device 200. For example, it executes an operation to shut down the agent device 200. Alternatively, it may sequentially or simultaneously erase each display on the surface of the agent device 200. Then, for example, it enters the dormant state shown in Figure 3(B).

[0076] In step S504, the display control unit 123 performs a display process to display the agent image on the display unit 311 of the first display device 310. In this case, for example, an appearance scene may be performed on the display unit 311 to show the agent image. It is also preferable that the agent image displayed on the display unit 311 has the same appearance (for example, the same color and shape) as the agent device 200 at the time of this switch. Furthermore, an effect may be performed in which the agent image emits the same sound that the agent device 200 was emitting at the time of this switch, or an effect may be performed in which the agent image begins to speak the same conversation that the agent device 200 was uttering at the time of this switch. Additionally, an effect may be performed to show user U1 that the agent image holds the same information as the information held by the agent device 200 at the time of this switch.

[0077] Thus, when the agent device 200 presents information to user U1, if certain predetermined conditions are met, the system switches from the agent device 200 to an agent image. When switching, it is preferable to shut down the agent device 200 before displaying the agent image. In this case, a predetermined effect may be performed. For example, an effect can be performed in which the agent image suddenly appears in a place where there was originally nothing, or in which a sleeping agent image wakes up. Alternatively, for example, an effect can be performed in which the agent device 200, which is installed above the first display device 310, shuts down and moves to the display unit 311 of the first display device 310. For example, immediately after the agent device 200 shuts down, the agent image can be displayed in a way that it descends from above the display unit 311 of the first display device 310.

[0078] In this way, by having the agent image start moving after the agent device 200 has stopped, a sense of unity and continuity between the agent device 200 and the agent image can be created. For example, by making the appearance of the agent device 200 and the agent image identical or nearly identical, the user U1 can be made to recognize them as the same, giving the impression that the agent device 200 has transformed into the agent image. This allows the reliability of the agent device 200 to be transferred to the agent image. In other words, the reliability of both the agent device 200 and the agent image can be enhanced.

[0079] In this embodiment, only one of the agent device 200 or the agent image is set to an active state, while the other is set to a dormant state. The active state of the agent image means that it is displayed on any of the display devices 300. The dormant state of the agent image means that it is not displayed on any of the display devices 300, or that it is displayed on any of the display devices 300 but in an inactive display mode. By setting only one of the agent device 200 or the agent image to an active state and the other to a dormant state, it is possible to perform an effect such as the agent's spirit being transferred, and to make the user U1 recognize that the agent device 200 and the agent image are the same person.

[0080] Furthermore, both the agent device 200 and the agent image may be in an active state, or both the agent device 200 and the agent image may be in a dormant state. In this case, it is preferable to have either the agent device 200 or the agent image in a state close to the active state.

[0081] Figure 8 shows an example where, when vehicle C1 is reversed, the agent device 200 switches to the agent image on the condition that an object exists in the blind spot of user U1 behind vehicle C1. However, the agent device 200 may also switch to the agent image on the condition that vehicle C1 is reversed. In other words, the timing of the vehicle C1 being reversed can be set as the timing of the switch from agent device 200 to the agent image. By setting the switching timing in this way, even if no object is detected in the blind spot of user U1 behind vehicle C1, user U1 can understand that agent device 200 is functioning correctly. Furthermore, it is possible to increase the sense of security regarding agent device 200 and the agent image. Thus, when the agent image is displayed on the condition that vehicle C1 is reversed, the agent device 200 may be made to perform a safety check animation. In this case, the agent device 200 may be made to output voice messages such as "Be careful."

[0082] Figures 6 and 8 show an example in which, when vehicle C1 is reversed, the agent image 415 is displayed on the first display device 310 instead of the agent device 200 to notify the user of the presence of an object behind vehicle C1. Here, it is conceivable that some users, when reversing vehicle C1, may look at the rearview mirror or directly look behind through the rear window instead of looking at the rear image displayed on the first display device 310. In such cases, it is thought that the notification effect can be enhanced by changing the position in which the agent image 415 is displayed based on the user's line of sight. Therefore, here we show an example in which the display device that shows the agent image 415 is changed based on the user's line of sight when vehicle C1 is reversed. Note that the rearview mirror shown here includes various mirrors such as the rearview mirror that can check the area behind vehicle C1.

[0083] For example, consider a case where, instead of mirrors, the rearview mirror 5 and side mirrors 6 and 7 of vehicle C1 are display devices capable of displaying images of the area behind vehicle C1 acquired by the rear camera 103. Also consider a case where the rear window (not shown) and side windows (not shown) of vehicle C1 are equipped with a HUD function that realizes HUD display by displaying virtual images.

[0084] Furthermore, when reversing vehicle C1, the switching determination unit 121 determines where user U1 is looking based on the image acquired by the in-vehicle camera 101. That is, the switching determination unit 121 detects user U1's gaze and identifies the direction of that gaze. For this gaze detection, known image recognition technology can be used. The display control unit 123 then displays an agent image on a display device located at a position corresponding to the direction of user U1's gaze to notify the user of objects located behind vehicle C1. For example, if user U1 is reversing vehicle C1 while looking in the rearview mirror 5, the agent image is superimposed on the rear image displayed in the rearview mirror 5, and this agent image is used to notify the user of objects located behind vehicle C1. Also, for example, if user U1 is reversing vehicle C1 while looking in the side mirror 6, the agent image is superimposed on the rear image displayed in the side mirror 6, and this agent image is used to notify the user of objects located behind vehicle C1. Furthermore, for example, if user U1 is reversing vehicle C1 while directly looking behind through the rear window, the agent image is superimposed on the HUD display area of ​​the rear window, and this agent image is used to notify the user of objects behind vehicle C1. Also, for example, if user U1 is reversing vehicle C1 while directly looking behind through the side window, the agent image is superimposed on the HUD display area of ​​the side window, and this agent image is used to notify the user of objects behind vehicle C1. It is assumed that when vehicle C1 is reversing, user U1's line of sight will change sequentially. In this way, if user U1's line of sight changes sequentially, the position where the agent image is displayed can be changed sequentially according to that change.

[0085] [Example of actions taken when a vehicle finishes reversing] Figure 9 is a flowchart showing an example of agent switching processing in the information processing device 110. This agent switching processing is executed after the vehicle C1 is reversed and the display processing that shows the agent image is completed. This agent switching processing will be explained with reference to Figures 1 to 8 as appropriate.

[0086] In step S511, the switching determination unit 121 detects that the shift lever has been operated.

[0087] In step S512, the switching determination unit 121 determines whether the shift lever operation detected in step S511 was an operation to put the vehicle into the P range. In other words, it determines whether parking of vehicle C1 is complete. If it was an operation to put the vehicle into the P range, the process proceeds to step S513. On the other hand, if it was not an operation to put the vehicle into the P range, the agent switching process ends.

[0088] In step S513, the display control unit 123 causes the agent image displayed on the display unit 311 of the first display device 310 to perform a departing behavior. For example, the agent image displayed on the display unit 311 performs a waving motion and executes a closing scene in which the agent image says goodbye.

[0089] In step S514, the agent device control unit 122 causes the agent device 200 to perform the actions of reviving. For example, it performs an action to activate the agent device 200. It also sequentially activates each display on the surface of the agent device 200. This results in the active state shown in Figure 3(A), for example.

[0090] Thus, when vehicle C1 has finished parking, the switch from the agent image to agent device 200 is executed.

[0091] [Example of notifying vehicle equipment status] Figure 10 shows an example of a guidance image 430 displayed on the display unit 311 of the first display device 310 when user U1 is searching for equipment in vehicle C1. Figure 10 shows an example where user U1 is searching for USB port 8 (see Figure 1). For example, it is possible that vehicle C1 is a new vehicle and user U1 is not yet familiar with the various parts of vehicle C1. In such a case, the agent device 200 can inform the user of the location of each piece of equipment in vehicle C1.

[0092] For example, if user U1 is looking for USB port 8 installed in vehicle C1, the voice input unit 106 can detect that user U1 is looking for USB port 8 based on user U1's voice. For example, if user U1 says "Where is the USB?", the voice input unit 106 will acquire that voice "Where is the USB?", and based on that voice "Where is the USB?", the switching determination unit 121 can detect that user U1 is looking for USB port 8. For this voice-based information acquisition, known voice recognition technology can be employed.

[0093] For example, the arms 211 and 212 of the agent device 200 could be used to indicate the location of the USB port 8 on vehicle C1. However, even if the arms 211 and 212 of the agent device 200 are used to indicate the location of the USB port 8, which is located below the first display device 310 of vehicle C1, it is expected that only a rough direction will be transmitted to user U1 due to the distance between the agent device 200 and the USB port 8. In other words, even if the arms 211 and 212 of the agent device 200 are used to indicate the location of the USB port 8, which is located below vehicle C1, it is expected that the location of the USB port 8 will be transmitted with low accuracy. In this case, for example, if user U1 is not familiar with vehicle C1, user U1 may not be able to know the location of the USB port 8 and may not be able to use the USB port 8. Thus, it is expected that it will be difficult for the agent device 200 to communicate information to user U1 about equipment on vehicle C1 that is located far from the agent device 200.

[0094] Furthermore, it is conceivable that, for example, the equipment that user U1 is looking for is located on the rear seat side. In such a case, even if the arm parts 211 and 212 of the agent device 200 are used to indicate the location of the equipment, it is conceivable that only a rough direction will be conveyed to user U1 due to the distance between the agent device 200 and the equipment. Also, even if the arm parts 211 and 212 of the agent device 200 are used to indicate the location of the equipment, the rear seat may obstruct the direction being indicated. In this case, user U1 may not be able to recognize whether the agent device 200 is pointing to the rear seat or to the equipment located on the rear seat side, which could cause confusion for user U1.

[0095] Therefore, in this embodiment, when guiding user U1 to equipment in vehicle C1 located at a distance from the agent device 200, the presence of that equipment is displayed on the first display device 310 to inform user U1. Specifically, an agent image 432 is displayed along with an image of the interior or exterior of the vehicle (guidance image 430) that includes an image of the equipment to be guided to user U1 (equipment image 431), and the agent image 432 points to the equipment image 431 to inform user U1. The equipment image 431 is extracted from the equipment information DB 132 (see Figure 5) based on the target equipment.

[0096] In the example shown in Figure 10, the display configuration is such that the left arm 433 of the agent image 432 points to the equipment image 431, which represents USB port 8. Specifically, the agent image 432 extends its left arm 433 to the left to point to the equipment image 431. Alternatively, the user U1 may be notified by voice information S3 that the agent image 432 is pointing to the equipment they are looking for.

[0097] Figure 10 shows an example of displaying an equipment image 431 on the first display device 310. Similarly, when displaying other information related to vehicle C1, such as the vehicle C1 instruction manual, it is possible to point to a predetermined location using an agent image.

[0098] [Example of a scenario where a user in the back seat of a vehicle wants to communicate with an agent] For example, it is conceivable that a user sitting in the rear seat of vehicle C1 (hereinafter referred to as the rear-seat user) may want to communicate with the agent device 200. Alternatively, it is conceivable that the agent device 200 may need to interact with the rear-seat user. In such cases, because of the distance between the agent device 200 and the rear-seat user, it may be difficult to achieve smooth communication between the agent device 200 and the rear-seat user. Therefore, when some kind of exchange or interaction is necessary between the agent device 200 and the rear-seat user, an agent image can be displayed on the second display device 320 installed on the rear-seat side, and communication can be made between this agent image and the rear-seat user. For example, if the rear-seat user wants to communicate with the agent device 200, the user can perform some operation, such as touching the second display device 320, or making a voice statement to that effect. In this case, based on the user's operation, the information processing device 110 can switch from the agent device 200 to the agent image.

[0099] Figure 11 shows an example of a display when an agent image 441 is displayed on the display unit 321 of the second display device 320 in response to the need for communication with a rear-seat user. In Figure 11, an example of a display is shown when the agent image 441 displayed on the display unit 321 of the second display device 320 guides the user around the vehicle C1 in response to a request from the rear-seat user. The environment around the vehicle C1 can be obtained from the map information DB 131 (see Figure 5) based on the current location of the vehicle C1 acquired by the location information acquisition sensor 105.

[0100] For example, if communication is needed between the agent device 200 and a rear-seat user, the agent device 200 can be used to send various notifications to the rear-seat user. For instance, if the agent device 200 and the rear-seat user are playing a game, the agent device 200 can point in the direction of the rear-seat user. However, if the distance between the rear-seat user and the agent device 200 is large, the rear seat may obstruct the direction being pointed. In this case, the rear-seat user may not be able to tell whether the agent device 200 is pointing to the rear seat or to themselves, which could cause confusion. Similarly, when sending various notifications to the rear-seat user using the agent device 200, the rear-seat user may not be able to properly recognize the direction the agent device 200 is pointing, which could cause confusion.

[0101] Therefore, in this embodiment, when communication is required between the agent device 200 and the rear-seat user, the agent image 441 is displayed on the display unit 321 of the second display device 320 instead of the agent device 200. Then, communication is achieved between the agent device 200 and the rear-seat user using the agent image 441. For example, it is possible to guide the user around the vehicle C1 using the agent image 441.

[0102] For example, as shown in Figure 11, an image (e.g., a tourist information image 440) requested by a rear-seat user is displayed on the display unit 321 of the second display device 320, and the agent image 441 is superimposed on the tourist information image 440. The left arm 442 of the agent image 441 is then made to point in the direction of a predetermined tourist facility (e.g., ABC Castle). Alternatively, the user may be notified by voice information S4 that the agent image 441 is pointing to a predetermined tourist facility (e.g., ABC Castle).

[0103] [Example of operation when notifying about vehicle equipment] Figure 12 is a flowchart illustrating an example of agent switching processing in the information processing device 110. This agent switching processing is performed continuously at each control cycle. This agent switching processing will be explained with reference to Figures 1 to 11 as appropriate.

[0104] In step S521, the switching determination unit 121 performs speech recognition processing to recognize the voice spoken by user U1 based on the voice information acquired by the voice input unit 106. This speech recognition processing can employ, for example, known speech recognition technology.

[0105] In step S522, the switching determination unit 121 determines whether the voice of user U1 recognized in step S521 is a voice searching for equipment in vehicle C1. That is, the switching determination unit 121 detects the equipment (target equipment) that user U1 is looking for. For example, keywords used when searching for equipment in vehicle C1 (e.g., "Where is the USB?") are stored in the storage unit 130 beforehand. Then, the switching determination unit 121 determines whether the voice of user U1 recognized in step S521 is a voice searching for equipment in vehicle C1 based on the degree of match (or similarity) between the voice of user U1 recognized in step S521 and the keywords stored in the storage unit 130. For this voice similarity determination, known speech recognition technology can be used. If the voice of user U1 is a voice searching for equipment in vehicle C1, the process proceeds to step S523. On the other hand, if the voice of user U1 is not a voice searching for equipment in vehicle C1, the agent switching process is terminated.

[0106] In step S523, the switching determination unit 121 determines whether the positional relationship between the equipment that user U1 is looking for (target equipment) detected in step S522 and the agent device 200 satisfies predetermined conditions. For example, if the distance between the target equipment and the agent device 200 is greater than or equal to a threshold, it is determined that the positional relationship between the target equipment and the agent device 200 satisfies predetermined conditions. Also, for example, if there is some kind of obstacle between the target equipment and the agent device 200, such as a rear seat, it is determined that the positional relationship between the target equipment and the agent device 200 satisfies predetermined conditions. Also, for example, if the target equipment is located behind the rear seat, it is determined that the predetermined conditions are met. In this way, if it is assumed that it will be difficult for the agent device 200 to point to the target equipment, it is determined that the predetermined conditions are met.

[0107] On the other hand, if the distance between the target equipment and the agent device 200 is less than a threshold, and the agent device 200 is able to point to the target equipment, it is determined that the positional relationship between the target equipment and the agent device 200 does not satisfy the predetermined conditions. Thus, if it is assumed that the agent device 200 is able to point to the target equipment, it is determined that the predetermined conditions are not satisfied. For example, if the target equipment is located at the feet of the agent device 200, it is determined that the agent device 200 is able to point to the target equipment, and therefore the predetermined conditions are not satisfied. Information indicating whether the equipment of vehicle C1 and the agent device 200 satisfy the predetermined conditions can be stored in the storage unit 130 in advance and used.

[0108] If the positional relationship between the target equipment and the agent device 200 satisfies the predetermined conditions, the process proceeds to step S524. On the other hand, if the positional relationship between the target equipment and the agent device 200 does not satisfy the predetermined conditions, the agent switching process is terminated.

[0109] In step S524, the agent device control unit 122 executes a stop animation process to stop the agent device 200. This stop animation process corresponds to the stop animation process shown in Figure 8 (step S503), so a detailed explanation is omitted here.

[0110] In step S525, the display control unit 123 executes a display process to display the agent image on the display unit 311 of the first display device 310. In this display process, a guidance image including the equipment image of the target equipment is displayed on the display unit 311 of the first display device 310, and the agent image is superimposed on the guidance image. For example, as shown in Figure 10, the agent image 432 is superimposed on the guidance image 430, and the equipment image 431 of the target equipment (USB port 8) is pointed to by the left arm 433 of the agent image 432. The appearance of the agent image in this display process corresponds to the display process shown in Figure 8 (step S504), so a detailed explanation is omitted here.

[0111] [Example of actions taken when the facility guidance is completed] Figure 13 is a flowchart showing an example of agent switching processing in the information processing device 110. This agent switching processing is performed after the display processing, which overlays the agent image onto the guidance image that guides the user through the equipment, has been completed. This agent switching processing will be explained with reference to Figures 1 to 12 as appropriate.

[0112] In step S531, the switching determination unit 121 detects that a predetermined operation related to equipment guidance has been performed. This predetermined operation is determined, for example, by detecting some manual operation by user U1 (for example, a touch operation of touching the display unit 311 of the first display device 310). Alternatively, it may be determined, for example, by detecting an action by the user to use the target equipment (for example, the action of user U1 inserting a USB terminal into the USB port 8). Alternatively, the predetermined operation may be determined, for example, based on the voice spoken by user U1.

[0113] In step S532, the switching determination unit 121 determines whether the equipment search has ended based on the predetermined operation detected in step S531. That is, it determines whether the predetermined operation detected in step S531 is an equipment guidance completion operation. For example, if user U1 says "USB found," it is possible to determine that an equipment guidance completion operation has occurred based on that voice. Alternatively, for example, the equipment guidance completion operation may be determined based on user U1's response to a question from the agent image. For example, if the agent image outputs the voice "USB warm," and then user U1 says "Found," or if the user nods, the equipment guidance completion operation may be determined based on these actions. In this way, the equipment guidance completion operation can be determined based on user operations, user actions, user behavior, etc.

[0114] If the equipment search is complete, proceed to step S533. If the equipment search is not complete, terminate the agent switching process.

[0115] In step S533, the display control unit 123 causes the agent image displayed on the display unit 311 of the first display device 310 to perform a walking behavior animation. This animation process corresponds to the animation process shown in Figure 9 (step S513), so a detailed explanation is omitted here.

[0116] In step S534, the agent device control unit 122 instructs the agent device 200 to perform the revival behavior animation. This animation process corresponds to the animation process shown in Figure 9 (step S514), so a detailed explanation is omitted here.

[0117] [Example of operation when notifying rear-seat passengers] Figure 14 is a flowchart illustrating an example of agent switching processing in the information processing device 110. This agent switching processing is performed continuously at each control cycle. This agent switching processing will be explained with reference to Figures 1 to 13 as appropriate.

[0118] In step S541, the switching determination unit 121 performs speech recognition processing to recognize the voice spoken by user U1 based on the voice information acquired by the voice input unit 106. This speech recognition processing corresponds to the performance processing (step S521) shown in Figure 12, so a detailed explanation is omitted here.

[0119] In step S542, the switching determination unit 121 determines whether the voice of user U1 recognized in step S541 is a specific voice coming from the rear seat of vehicle C1. For example, multiple microphones are installed inside vehicle C1 as the voice input unit 106, and the location and direction of the user who emitted the voice can be determined based on the voice picked up by these multiple microphones. The specific voice can be, for example, the voice of a rear-seat user calling out to the agent device 200 (e.g., "Hey, agent"), or the voice of a rear-seat user requesting some action from the agent device 200 (e.g., "Give me a tour").

[0120] For example, keywords corresponding to specific voices (e.g., "Hey agent, give me a tour") are pre-stored in the memory unit 130. Then, the switching determination unit 121 determines whether the voice of user U1 recognized in step S541 is a specific voice based on the degree of match (or similarity) between the voice of user U1 recognized in step S541 and the keywords stored in the memory unit 130. For this voice similarity determination, known speech recognition technology can be used. If the voice of user U1 is a specific voice from the rear seat of vehicle C1, the process proceeds to step S543. On the other hand, if the voice of user U1 is not a specific voice from the rear seat of vehicle C1, the agent switching process ends.

[0121] In step S543, the agent device control unit 122 executes a stop animation process to stop the agent device 200. This stop animation process corresponds to the stop animation process shown in Figure 8 (step S503), so a detailed explanation is omitted here.

[0122] In step S544, the display control unit 123 executes a display process to display the agent image on the display unit 321 of the second display device 320. In this display process, an image (for example, a tourist information image) in response to a request from the rear-seat user is displayed on the display unit 321 of the second display device 320, and the agent image is superimposed on that image. For example, as shown in Figure 11, the agent image 441 is superimposed on the tourist information image 440 displayed on the display unit 321 of the second display device 320, and the left arm 442 of the agent image 441 points in the direction of a predetermined tourist facility (for example, ABC Castle). The appearance scene of the agent image in this display process corresponds to the display process shown in Figure 8 (step S504), so a detailed explanation is omitted here.

[0123] [Example of operation when rear seat display ends] Figure 15 is a flowchart showing an example of agent switching processing in the information processing device 110. This agent switching processing is performed after the display processing that displays the agent image on the display unit 321 of the second display device 320 is completed. This agent switching processing will be explained with reference to Figures 1 to 14 as appropriate.

[0124] In step S551, the switching determination unit 121 detects that a predetermined operation has been performed on the display unit 321 of the second display device 320. This predetermined operation is determined, for example, by detecting some manual operation by user U1 (for example, a touch operation such as touching the display unit 321 of the second display device 320). Alternatively, the predetermined operation may be determined, for example, based on voice output from user U1.

[0125] In step S552, the switching determination unit 121 determines whether a return operation was performed based on the predetermined operation detected in step S551. That is, it determines whether the predetermined operation detected in step S551 is a return operation. For example, if user U1 says "Return the agent," it is possible to determine that a return operation was performed based on that voice. Alternatively, for example, it may be determined that a return operation was performed based on the rear-seat user's response to a question from the agent image. For example, if the agent image outputs the voice "Do you want to play with the agent any more?" and then user U1 says "No, that's enough," or if the user nods, it may be determined that a return operation was performed based on these actions. In this way, it is possible to determine whether or not a return operation was performed based on user operations, user actions, user behavior, etc.

[0126] If a revert operation is performed, the process proceeds to step S553. On the other hand, if no revert operation is performed, the agent switching process terminates.

[0127] In step S553, the display control unit 123 causes the agent image displayed on the display unit 321 of the second display device 320 to perform a walking behavior animation. This animation process corresponds to the animation process shown in Figure 9 (step S513), so a detailed explanation is omitted here.

[0128] In step S554, the agent device control unit 122 instructs the agent device 200 to perform the animation of the revival behavior. This animation process corresponds to the animation process shown in Figure 9 (step S514), so a detailed explanation is omitted here.

[0129] [Example of notifying multiple objects located in front of a vehicle] Figure 16 shows an example of display transitions when the display area 331 of the front windshield 4 is displayed according to the number of objects present in front of vehicle C1. In Figure 16, an example of display transitions is shown when vehicle C1 is stopped at an intersection, and initially there is one pedestrian 451 in front of vehicle C1, and then multiple pedestrians 451 to 453 are present in front of vehicle C1. Objects around vehicle C1 can be detected by sensors 117.

[0130] For example, when vehicle C1 is stopped at an intersection, or when vehicle C1 is leaving a garage, it is conceivable that there may be moving objects in front of vehicle C1, such as pedestrians or animals. For example, objects that could be considered include people walking on a crosswalk, people entering the path of vehicle C1, people close to the path of vehicle C1, other vehicles, other moving objects, and other objects that may come into contact with vehicle C1.

[0131] In such cases, the user U1 of vehicle C1 needs to pay attention to the moving object in front of it. In this case, it is conceivable to use the agent device 200 to notify user U1 of the moving object present in front of vehicle C1. For example, if there is one moving object in front of vehicle C1, the agent device 200 can indicate the direction of that moving object, thereby notifying the user of its location. This example is shown in Figure 16(A). However, if there are multiple moving objects in front of vehicle C1, for example, it may be difficult for the agent device 200 to simultaneously indicate the direction of each object. In other words, it is assumed that it may be difficult for a single agent device 200 to simultaneously draw the attention of user U1 to all objects of interest. Therefore, in such cases, an agent image can be used instead of the agent device 200 to simultaneously notify user U1 of all objects of interest and draw their attention to it. This example is shown in Figure 16(B).

[0132] Figure 16(A) shows an example in which the presence of an object, such as a pedestrian 451, in front of vehicle C1 is notified to user U1 using the agent device 200. For example, the arm 211 of the agent device 200 is used to point in the direction of the object. Alternatively, the detection of an object in front of vehicle C1 may be notified via voice information S5.

[0133] Figure 16(B) shows an example in which, when multiple objects, such as multiple pedestrians 451 to 453, are present in front of vehicle C1, the presence of each object is notified to user U1 using agent images 461 to 463 displayed in the display area 331 of the front windshield 4. For example, the arms of agent images 461 to 463 are used to point in the direction in which the object is located. When agent images 461 to 463 are displayed in the display area 331 of the front windshield 4 in this manner, the agent device 200 is put into a dormant state. In this case, for example, a transition image (for example, arrow images 465 to 467) indicating the transition from agent device 200 to agent images 461 to 463 may be displayed. Alternatively, the detection of multiple objects in front of vehicle C1 may be notified by voice information S6.

[0134] Figure 16 shows an example in which, when multiple objects are detected in front of vehicle C1, the presence of each object is notified to user U1 using agent images. However, even if only one object is detected in front of vehicle C1, if that object is moving quickly, it is conceivable that the movement of the arms 211 and 212 of the agent device 200 may not keep up with the movement speed of that object. For example, if control is performed to make the rate of change of the position of the tips of the arms 211 and 212 greater than a predetermined value, it is conceivable that the actual arms 211 and 212 may not move at that rate, and it is conceivable that the first notification using the tips of the arms 211 and 212 may not be properly performed. Therefore, even if only one object is detected in front of vehicle C1, if that object is moving quickly, the presence of that object may be notified to user U1 using agent images. The movement speed of the object can be detected based on images acquired by the front camera 102 or detection information acquired by the sensors 107.

[0135] For example, if the switching determination unit 121 detects an object in front of the vehicle C1, it detects the movement speed of that object. If the switching determination unit 121 detects an object in front of the vehicle C1 whose movement speed exceeds a threshold, it displays an agent image on the display area 331 of the front windshield 4, pointing to that object.

[0136] The switching determination unit 121 may also determine whether or not to display an agent image on the display area 331 of the front windshield 4 based on the rate of change of the position of the tips of the arms 211 and 212 when notifying the position of an object located in front of the vehicle C1. For example, the switching determination unit 121 calculates the rate of change of the position of the tips of the arms 211 and 212 when notifying the position of an object located in front of the vehicle C1 based on the operation of the arms 211 and 212 of the agent device 200. This rate of change of the position of the tips of the arms 211 and 212 can be calculated based on the position and movement speed of the object located in front of the vehicle C1. If the rate of change of the position of the arms 211 and 212 is greater than a threshold, the switching determination unit 121 displays an agent image on the display area 331 of the front windshield 4, pointing to each of the objects.

[0137] [Example of operation when notifying multiple objects] Figure 17 is a flowchart illustrating an example of agent switching processing in the information processing device 110. This agent switching processing is performed continuously at each control cycle. This agent switching processing will be explained with reference to Figures 1 to 16 as appropriate.

[0138] In step S561, the switching determination unit 121 detects objects present around the vehicle C1 based on the information acquired by the front camera 102 and the sensors 107.

[0139] In step S542, the switching determination unit 121 determines whether multiple objects were detected in step S541. If multiple objects are detected, the process proceeds to step S563. On the other hand, if multiple objects are not detected (i.e., if one object is detected or no objects are detected), the agent switching process is terminated.

[0140] In step S563, the agent device control unit 122 executes a stop animation process to stop the agent device 200. This stop animation process corresponds to the stop animation process shown in Figure 8 (step S503), so a detailed explanation is omitted here.

[0141] In step S564, the display control unit 123 controls the third display device 330 to perform a display process that displays agent images in the display area 331 of the front windshield 4. In this display process, agent images are displayed in association with each of the multiple objects so that they point to each of the multiple objects detected in step S541. For example, as shown in Figure 16(B), agent images 461 to 463 are superimposed and displayed in the display area 331 of the front windshield 4, so that pedestrians 451 to 453 are pointed to by the arms of agent images 461 to 463.

[0142] In this display process, as shown in Figure 16(B), transition images (for example, arrow images 465 to 467) may be displayed at the time the agent device 200 is stopped to indicate the transition from the agent device 200 to agent images 461 to 463. Furthermore, other appearances of the agent images in this display process correspond to the display process shown in Figure 8 (step S504), so a detailed explanation is omitted here.

[0143] [Example of operation when multiple object notifications have finished] Figure 18 is a flowchart showing an example of agent switching processing in the information processing device 110. This agent switching processing is performed after display processing, which displays multiple agent images associated with multiple objects. This agent switching processing will be explained with reference to Figures 1 to 17 as appropriate.

[0144] In step S571, the switching determination unit 121 determines whether a predetermined time has elapsed since the agent image was displayed in the display area 331 of the front window 4. If the predetermined time has elapsed since the agent image was displayed, the process proceeds to step S572. On the other hand, if the predetermined time has not elapsed since the agent image was displayed, the agent switching process is terminated.

[0145] In step S572, the switching determination unit 121 detects objects present around the vehicle C1 based on the information acquired by the front camera 102 and the sensors 107.

[0146] In step S573, the switching determination unit 121 determines whether multiple objects were detected in step S572. If multiple objects are detected, the agent switching process is terminated. On the other hand, if multiple objects are not detected (i.e., if one object is detected or no objects are detected), the process proceeds to step S574.

[0147] In step S574, the display control unit 123 causes the agent image displayed in the display area 331 of the front window 4 to perform a walking behavior animation. This animation process corresponds to the animation process shown in Figure 9 (step S513), so a detailed explanation is omitted here.

[0148] In step S575, the agent device control unit 122 instructs the agent device 200 to perform the revival behavior animation. This animation process corresponds to the animation process shown in Figure 9 (step S514), so a detailed explanation is omitted here.

[0149] In the example shown in Figure 18, if only one object is detected after multiple objects have been detected, the system switches from the agent image to the agent device 200. However, if at least one object continues to be detected after multiple objects have been detected, the display of the agent image may continue to track the continuously detected object.

[0150] [Example of displaying agent images in scenes requiring rich expression] Here, it is conceivable that we may want the agent to display rich expressions. For example, let's assume that vehicle C1 is an electric vehicle. In this case, if lightning appears in the background of the agent while vehicle C1 is charging, it is possible to create the effect that the agent is charging energy. Furthermore, such an effect can make user U1 aware that vehicle C1 is charging. It can also provide entertainment for user U1 during the time when vehicle C1 is charging. In such scenes where it is necessary for the agent to display rich expressions, it is conceivable that it would be difficult to achieve such expressions with a 3D agent device 200. Therefore, when it is necessary for the agent to display rich expressions, it is possible to display an agent image on the first display device 310 and make this agent image display rich expressions. In other words, it is possible to make interesting expressions by using the background of the display screen on the first display device 310. The timing of switching between the agent device 200 and the agent image in this example will be explained. For example, when it becomes necessary for the agent to display rich expressions (for example, when lightning appears in the background of the agent, creating the effect that it is charging energy), the timing of switching from the agent device 200 to the agent image can be determined. Furthermore, for example, if it becomes unnecessary to have the agent perform rich expressions, the timing for switching from the agent image to the agent device 200 can be set to a predetermined time elapsed since the display of the agent image.

[0151] In this embodiment, examples of using an agent image to point to a destination image included in a map image displayed on the first display device 310, or to point to equipment (e.g., USB port 8) included in a guidance image displayed on the first display device 310, are shown. However, in other cases where the content displayed on each display device in the display device group 300 includes an object to be pointed to, that object may also be pointed to using an agent image. For example, when pointing to a character included in a cartoon video displayed on the first display device 310, that character may be pointed to using an agent image. That is, the control unit 120 may perform notification (second notification) using an agent image when the content displayed on each display device in the display device group 300 includes a predetermined object.

[0152] Here, to avoid interfering with user U1 while driving, it is conceivable to set the default state to displaying an agent image with low presence. In this case, the agent device 200 and the agent image can be switched when certain predetermined conditions are met. For example, the agent image is displayed while vehicle C1 is in motion. Then, when vehicle C1 stops, the switch to agent device 200 is executed.

[0153] Here, a physical, three-dimensional agent device 200 can effectively present itself to user U1 through robotic behavior. This is known to attract user U1's attention. However, there are limitations to the size and shape of the arms 211 and 212 of the agent device 200, and it may be difficult to accurately indicate the position of an object depending on the movement and orientation of the arms 211 and 212. Similarly, if the agent device 200 is equipped with parts that mimic hands, fingers, and arms, it is expected that it will be difficult to accurately indicate the position of an object due to the limitations in the size and shape of the hands, fingers, and arms.

[0154] Furthermore, there are scenarios where it may be difficult to attract the attention of the agent device 200. For example, in scenarios where precise pointing is required, it would be difficult to attract the attention of user U1 using the agent device 200. In other words, the support that the agent device 200 can provide is limited if it remains installed on the dashboard 2.

[0155] Therefore, in this embodiment, when the pointing accuracy of the agent device 200 is required, the system switches from the agent device 200 to an agent image. In this way, by using both the agent device 200 and the agent image, it becomes possible to create the impression that the same agent is consistently providing support. This makes it possible to create an experience that enhances the acceptability and empathy of user U1.

[0156] Furthermore, when the agent device 200 provides information to user U1, if certain predetermined conditions are met, the agent device 200 can switch to an agent image, and the information can be provided using the agent image. Additionally, the system can switch back from the agent image to the agent device 200 at predetermined timings. In this embodiment, when using the agent to inform user U1, who is riding in vehicle C1, about an object, it becomes possible to accurately indicate the location of the object being reported.

[0157] In this embodiment, an example was shown in which an agent image is displayed using a group of display devices 300 installed in vehicle C1. However, this embodiment can also be applied to cases where an agent image is displayed on a display device that can be installed in vehicle C1 or on a display device held by a user riding in vehicle C1. The display devices that can be installed in vehicle C1 and the display devices held by a user riding in vehicle C1 are, for example, electronic devices and information processing devices such as smartphones, tablet terminals, and personal computers. For example, the agent image may be displayed in a way that makes it appear as if it is being transferred to a smartphone held by a user in the back seat of vehicle C1.

[0158] In this embodiment, an example is shown using an agent device 200 installed in vehicle C1. However, this embodiment can also be applied to cases where an agent device that can be removed from vehicle C1 is used. For example, when a user U1 who possesses a portable agent device gets into vehicle C1, the user U1 may place the agent device on the dashboard 2 of vehicle C1, and when getting out of vehicle C1, the user U1 may carry the agent device with them.

[0159] In other words, this embodiment is applicable to an information processing device provided in a vehicle C1 capable of installing agent equipment and a display device for displaying each agent image.

[0160] Furthermore, although this embodiment shows an example of displaying an agent image on a vehicle C1 equipped with a HUD function, this embodiment can also be applied to a vehicle C1 that does not have a HUD function.

[0161] In the above example, the switching process of the agent device 200 and the agent image is performed in the information processing device 110 (or information processing system 100). However, all or part of these processes may be performed in other devices. In this case, the information processing system is composed of devices that perform part of these processes. For example, at least part of each process can be performed using various information processing devices and electronic devices such as in-vehicle devices, devices usable by user U1 (e.g., smartphones, tablet terminals, personal computers, car navigation systems, IVIs), and servers that can be connected via a predetermined network such as the Internet.

[0162] Furthermore, some (or all) of the information processing system capable of executing the functions of the information processing device 110 (or information processing system 100) may be provided by an application that can be provided via a predetermined network such as the Internet. This application may be, for example, SaaS (Software as a Service).

[0163] [Example configuration and effects of this embodiment] The information processing method according to this embodiment is an information processing method for informing a user U1 riding in a vehicle C1, which is equipped with an agent device 200 having arms 211, 212 (an example of a component) capable of pointing in a predetermined direction, and a group of display devices 300 that displays agent images 415, 422, 432, 441, 461 to 463 as a display mode that points to an object located inside or outside the vehicle C1, corresponding to the agent device 200. This information processing method includes control processing (steps S501 to S504, S511 to S514, S521 to S525, S531 to S534, S541 to S544, S551 to S554, S561 to S564, S571 to S575) that controls the operating state of the agent device 200 and the display state of the group of display devices 300. In this control process, when informing a user U1 riding in vehicle C1 about an object (for example, an object behind vehicle C1, a destination, equipment on vehicle C1, a tourist information facility, or an object in front of vehicle C1), the system switches between a first notification, which uses the agent device 200 to point to the object and inform the user about it, and a second notification, which uses an agent image to point to the object and inform the user about it, based on at least one of the positional relationship between the object and the agent device 200, the number of objects, and the operating state of the objects. Furthermore, the program according to this embodiment is a program that causes a computer to execute each of these processes. In other words, the program according to this embodiment is a program that causes a computer to implement each of the functions that the information processing device 110 can execute.

[0164] With this configuration, when informing a user U1 riding in vehicle C1 about an object, it is possible to switch between a first notification using the agent device 200 and a second notification using an agent image, based on the positional relationship between the object and the agent device 200. This makes it possible to accurately indicate the location of the object to be notified using the appropriate agent from among the agent device 200 and the agent image.

[0165] In the information processing method according to this embodiment, when a first notification is performed, the position of the object (for example, an object behind vehicle C1, a destination, equipment on vehicle C1, a tourist information facility, or an object in front of vehicle C1) is notified by the movement of arms 211 and 212 (an example of components) that point to the object, and when a second notification is performed, the position of the object is notified by the display mode of an agent image that points to the object.

[0166] With this configuration, when using the agent device 200 or agent image to inform a user U1 riding in vehicle C1 about an object, the location of the object to be informed can be accurately indicated.

[0167] In the information processing method according to this embodiment, the control processing (steps S501 to S504, S521 to S525, S541 to S544, S561 to S564) calculates the distance between the tip of the arm 211, 212 and the object when the position of the object is notified by the operation of the arm 211, 212 (an example of a component), and if the distance between the tip and the object is greater than a predetermined value, a second notification is performed. Alternatively, the distance between the agent device 200 and the object when the position of the object is notified by the operation of the agent device 200 may be calculated, and if the distance between the agent device 200 and the object is greater than a predetermined value, a second notification may be performed.

[0168] With this configuration, when the distance between the tips of the arms 211 and 212 and the object is relatively large, a second notification using the agent image is performed, making it possible to accurately indicate the position of the object.

[0169] In the information processing method according to this embodiment, the control processing (steps S501 to S504, S521 to S525, S541 to S544, S561 to S564) determines whether a component of vehicle C1 exists as an obstacle between the tip of the arm 211, 212 (an example of a component) and the object when the position of the object is notified by the operation of the arm 211, 212. If an obstacle exists between the tip of the arm 211, 212 and the object, a second notification is performed. Alternatively, when the position of the object is notified by the operation of the agent device 200, it may be determined whether a component of vehicle C1 exists as an obstacle between the agent device 200 and the object. If an obstacle exists between the agent device 200 and the object, a second notification may be performed.

[0170] In this configuration, if an obstacle exists between the tip of the arm 211, 212 and the target object, the second notification using the agent image is performed because the obstacle is in the way. This makes it possible to accurately indicate the position of the target object.

[0171] In the information processing method according to this embodiment, the control process (steps S501 to S504, S521 to S525, S541 to S544, S561 to S564) calculates the rate of change of the object, and if the rate of change of the object is greater than a predetermined value, a second notification is performed.

[0172] With this configuration, if it can be estimated that the rate of change of the position of the tips of the arms 211 and 212 will be greater than a predetermined value, a second notification using the agent image will be performed, making it possible to accurately indicate the position of the object.

[0173] In the information processing method according to this embodiment, in the control processing (steps S561 to S564), if there are multiple objects, a second notification is performed.

[0174] With this configuration, if there are multiple objects, a second notification is performed using the agent image, allowing the location of each object to be accurately indicated.

[0175] In the information processing method according to this embodiment, in the control processing (steps S501 to S504, S521 to S525, S541 to S544, S561 to S564), if the content displayed on the display device group 300 includes an object, a second notification is performed.

[0176] With this configuration, if the content displayed on the display device group 300 includes an object, a second notification is performed using the agent image, which allows for a more accurate indication of the object's location than the first notification using the tips of the arms 211 and 212.

[0177] In the information processing method according to this embodiment, the control processing (steps S501 to S504, S511 to S514, S521 to S525, S531 to S534, S541 to S544, S551 to S554, S561 to S564, S571 to S575) performs control to switch the operating state of the agent device 200 between an active state and a dormant state. When the first notification is performed, the agent device 200 is set to the active state and the display state of the display device group 300 is set to a state where the agent image is not displayed. When the second notification is performed, the agent device 200 is set to the dormant state and the display state of the display device group 300 is set to a state where the agent image is displayed.

[0178] With this configuration, by setting only one of the agent device 200 or the agent image to an active state and the other to a dormant state, it is possible to perform an effect such as the agent's spirit being transferred, and make user U1 recognize that the agent device 200 and the agent image are the same. This creates a sense of unity and continuity between the agent device 200 and the agent image, and enhances the reliability of the agent device 200 and the agent image.

[0179] In the information processing method according to this embodiment, in the control processing (steps S511 to S514, S531 to S534, S551 to S554, S571 to S575), when it is determined by the second notification that the information notification has ended, the agent device 200 is switched to an active state, and the display state of the display device group 300 is switched to a state in which the agent image is not displayed.

[0180] In this configuration, in order to accurately indicate the location of the target object, after the second notification is performed using the agent image, the first notification is performed using the agent device 200. This creates a sense of unity and continuity between the agent device 200 and the agent image, thereby increasing the reliability of the agent device 200 and the agent image.

[0181] In the information processing method according to this embodiment, the agent device 200 is a bio-engineered robot, and the arm parts 211 and 212 (an example of a component) are components that mimic at least one of a hand, fingers, and arm. In the control processing (steps S514, S534, S554, S575), when the first notification is performed, the position of the target object is notified by the movement of the arm parts 211 and 212 that point to the target object.

[0182] With this configuration, the movement of the arm portions 211 and 212 of the agent device 200 indicates the position of the target object, thus enabling accurate indication of the target object's location.

[0183] The information processing device 110 is an information processing device installed in a vehicle C1 that can be equipped with an agent device 200 having arm parts 211, 212 (an example of a component) that can point in a predetermined direction, and a group of display devices 300 that displays agent images 415, 422, 432, 441, 461 to 463 as a display mode that points to an object located inside or outside the vehicle C1, corresponding to the agent device 200. The information processing device 110 includes a control unit 120 that controls the operating state of the agent device 200 and the display state of the group of display devices 300. When the control unit 120 notifies a user U1 riding in vehicle C1 of information about an object (for example, an object behind vehicle C1, a destination, equipment on vehicle C1, a tourist information facility, or an object in front of vehicle C1), it switches between a first notification, which uses the agent device 200 to point to the object and notify the information, and a second notification, which uses an agent image to point to the object and notify the information, based on at least one of the positional relationship between the object and the agent device 200, the number of objects, and the operating state of the objects.

[0184] With this configuration, when informing a user U1 riding in vehicle C1 about an object, it is possible to switch between a first notification using the agent device 200 and a second notification using an agent image, based on the positional relationship between the object and the agent device 200. This makes it possible to accurately indicate the location of the object to be notified using the appropriate agent from among the agent device 200 and the agent image.

[0185] The processing steps shown in this embodiment are merely examples of how to implement this embodiment. The order of some of the processing steps may be changed, some of the processing steps may be omitted, or other processing steps may be added, as long as the embodiment is feasible.

[0186] Furthermore, each process shown in this embodiment is executed based on a program that causes a computer to execute each processing procedure. For this reason, this embodiment can also be understood as an embodiment of a program that realizes the function of executing each of these processes, and a recording medium that stores that program. For example, an update process to add a new function to an information processing device can cause that program to be stored in the storage device of the information processing device. This makes it possible to have the updated information processing device perform each of the processes shown in this embodiment.

[0187] Although embodiments of the present invention have been described above, these embodiments only represent a part of the application examples of the present invention, and are not intended to limit the technical scope of the present invention to the specific configurations of the above embodiments. [Explanation of symbols]

[0188] 100 Information processing system, 110 Information processing device, 120 Control unit, 121 Switching determination unit, 122 Agent device control unit, 123 Display control unit, 130 Storage unit, 131 Map information DB, 132 Equipment information DB, 200 Agent device, 300 Display device group, 310 First display device, 320 Second display device, 330 Third display device

Claims

1. An information processing method for informing a user riding in a vehicle that can be equipped with an agent device having a member capable of pointing in a predetermined direction by a predetermined operation, and a display device that displays an agent image corresponding to the agent device, in a display mode that points to an object located inside or outside the vehicle, of an object, This includes control processing that controls the operating state of the agent device and the display state of the display device, In the control process described above, when the user is notified of the information, the process switches between a first notification, in which the agent device is used to point to the object and notify the information, and a second notification, in which the agent image is used to point to the object and notify the information, based on at least one of the positional relationship between the object and the agent device, the number of objects, and the rate of change of the objects. Information processing methods.

2. The information processing method according to claim 1, When the first notification is performed, the position of the object is notified by the movement of the member that points to the object, When the second notification is performed, the location of the object is notified by the display mode of the agent image that points to the object. Information processing methods.

3. The information processing method according to claim 2, In the control process described above, the distance between the tip of the member and the object is calculated when the position of the object is notified by the movement of the member. If the distance between the tip and the object is greater than a predetermined value, the second notification is performed. Information processing methods.

4. The information processing method according to claim 2, In the control process described above, it is determined whether or not a component of the vehicle exists as an obstacle between the tip of the member and the object when the position of the object is notified by the movement of the member. If an obstacle exists between the tip and the object, the second notification is performed. Information processing methods.

5. The information processing method according to claim 1, In the control process described above, the rate of change of the object is calculated, If the rate of change of the object is greater than a predetermined value, the second notification is performed. Information processing methods.

6. The information processing method according to claim 1, In the control process described above, if there are multiple objects, the second notification is executed. Information processing methods.

7. The information processing method according to claim 1, In the control process described above, if the object is included in the content displayed on the display device, the second notification is executed. Information processing methods.

8. An information processing method according to any one of claims 1 to 7, In the control process described above, control is performed to switch the operating state of the agent device between an active state and a dormant state. When the first notification is performed, the agent device is set to the active state, and the display state of the display device is set to a state where the agent image is not displayed. When the second notification is performed, the agent device is put into the idle state, and the display state of the display device is set to display the agent image. Information processing methods.

9. The information processing method according to claim 8, In the control process, when it is determined by the second notification that the notification of the information has ended, the agent device is switched to the active state, and the display state of the display device is switched to a state in which the agent image is not displayed. Information processing methods.

10. An information processing method according to any one of claims 1 to 7, The aforementioned agent device is an anthropomorphic robot, The aforementioned member is a member that mimics at least one of the hand, fingers, and arm. In the control process described above, when the first notification is performed, the position of the object is notified by the movement of the member that points to the object. Information processing methods.

11. An information processing device to be installed in a vehicle, which is capable of installing an agent device having a member capable of pointing in a predetermined direction, and a display device that displays an agent image corresponding to the agent device, in a display mode that points to an object located inside or outside the vehicle, The system includes a control unit that controls the operating state of the agent device and the display state of the display device, When the control unit notifies a user riding in the vehicle of information about an object, it switches between a first notification, in which the agent device is used to point to the object and notify the information, and a second notification, in which the agent image is used to point to the object and notify the information, based on at least one of the positional relationship between the object and the agent device, the number of objects, and the rate of change of the objects. Information processing device.

Images