Information processing method and information processing device

Abstract

To appropriately transmit information which has to be transmitted to a user to the user without making the user feel discomfort.SOLUTION: An information processing method for controlling an agent device 200 capable of performing communication with a user U1 riding in a vehicle C1 includes: acquisition processing for acquiring user state information related to a state of the user U1 and vehicle state information related to a state of the vehicle C1 (step S501); detection processing for detecting a specific event to be reported to the user U1 based on the user state information and the vehicle state information (step S502); and control processing for allowing the agent device 200 to perform a first operation to suggest the specific event when the specific event is detected, and allowing the agent device 200 to perform a second operation (the operation in an aspect different from that of the first operation) to report the specific event when the specific event is continuously detected after a predetermined time elapses from the performance of the first operation (step S503 or S507).SELECTED DRAWING: Figure 11

Description

[Technical field]

[0001] The present invention relates to an information processing method and an information processing apparatus for controlling an agent device capable of communicating with a user. [Background technology]

[0002] Conventionally, there have been proposed techniques for conveying information about each state of a vehicle to a user. For example, there has been proposed a technique for outputting a speech corresponding to an abnormality when an abnormality occurs in a vehicle (for example, see Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2019-120839 A Summary of the Invention [Problem to be solved by the invention]

[0004] In the above-mentioned conventional technology, when multiple abnormalities occur in a vehicle, speech corresponding to the abnormalities is output as many times as the number of abnormalities. Therefore, when multiple abnormalities occur in a vehicle, the number of speeches corresponding to the abnormalities increases, and the user may feel that the speeches are annoying. In addition, for example, when an abnormality occurs while a user is listening to music, radio, etc., the speeches related to the abnormality are output, and the user may feel uncomfortable because the music, radio, etc. are interrupted by the speeches.

[0005] An object of the present invention is to appropriately convey information to a user without causing discomfort to the user. [Means for solving the problem]

[0006] One aspect of the present invention is an information processing method for controlling an agent device capable of communicating with a user in a vehicle. This information processing method includes an acquisition process for acquiring at least one of user status information related to the status of the user and vehicle status information related to the status of the vehicle, a detection process for detecting a specific event to be notified to the user based on at least one of the acquired user status information and vehicle status information, and a control process for causing the agent device to execute a first operation suggesting the specific event when the specific event is detected, and, when the specific event continues to be detected after a predetermined time has elapsed since the execution of the first operation, causing the agent device to execute a second operation (an operation of a different aspect from the first operation) for notifying the user of the specific event. Effect of the Invention

[0007] According to the present invention, information to be conveyed to a user can be appropriately conveyed without causing discomfort to the user. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram showing an example of the configuration of the interior of a vehicle. [Diagram 2] FIG. 2 is a front view showing an example of the external configuration of the agent device. [Diagram 3] FIG. 3 is a block diagram illustrating an example of a system configuration of the information processing system. [Figure 4] FIG. 4 is a diagram showing an example of the configuration of the specific event response information DB. [Diagram 5] FIG. 5 is a diagram showing an example of transition of actions suggesting that the seat belt is not fastened. [Figure 6] FIG. 6 is a diagram showing an example of transition of actions suggesting that the seat belt is not fastened. [Figure 7] FIG. 7 is a diagram showing an example of the transition of the suggestion operation related to the remaining fuel amount. [Figure 8] FIG. 8 is a diagram showing an example of the transition of the suggestion operation related to the remaining fuel amount. [Figure 9]FIG. 9 is a diagram showing an example of transition when the first operation and the second operation are executed. [Figure 10] FIG. 10 is a diagram showing an example of notification of a specific event using a character image. [Figure 11] FIG. 11 is a flowchart showing an example of the specific event notification process. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, an embodiment of the present invention will be described with reference to the accompanying drawings.

[0010] [Agent device installation example] Fig. 1 is a simplified diagram showing an example of the configuration of the interior of a vehicle C1. Fig. 1 shows the interior of the vehicle C1 in a simplified manner, viewed from the rear in the longitudinal direction of the vehicle C1, in front of the driver's seat and passenger seat (not shown). To facilitate explanation, Fig. 1 omits illustrations of components other than a dashboard 2, a steering wheel 3, a windshield 4, a rearview mirror 5, an agent device 200, and a display device 300.

[0011] The agent device 200 is a small robot installed on the dashboard 2 of the vehicle C1. In this embodiment, an example is shown in which the agent device 200 is a robot that imitates a human face. FIG. 1 shows an example in which the agent device 200 is installed on the dashboard 2, but the present invention is not limited to this. For example, the agent device 200 may be installed on the upper part of the windshield 4. FIG. 1 also shows an example in which the agent device 200 is a robot that imitates a human, but the present invention is not limited to this. For example, the agent device 200 may be a robot that imitates an animal such as a rabbit or a pig, or a robot that imitates a virtual creature (for example, the face of an anime character). In this embodiment, an example is shown in which a face image is displayed as an image that represents the likeness of a living thing, but an image that represents the entire body of the agent may be displayed. In this way, it is possible to use the agent device 200 as an agent that has been made into a creature.

[0012] 1 shows an example in which the agent device 200 is installed on the dashboard 2 of the vehicle C1 in a direction in which the eyes of the face 210 face the inside of the vehicle. In this embodiment, an example is shown in which a face image is displayed on the display unit 230 (see FIG. 3) when the agent device 200 is in an on state. The agent device 200 can be turned on in response to an on operation of the vehicle C1. The agent device 200 can be turned on in response to a user operation on the agent device 200. The on / off operation of the vehicle C1 refers to the on or off operation of a start key related to starting or stopping the vehicle C1.

[0013] The agent device 200 is a device that is composed of a substantially spherical face 210 and a body 220. The body 220 is provided with a right arm 221 and a left arm 222, and left and right feet 223. A display unit 230 (see FIG. 3) is provided on at least a part of the surface of each of these parts (for example, the face 210). For example, the display unit 230 may be provided on the entire surface of the face 210 and the body 220 (including the arms 221, 222 and the feet 223), or the display unit 230 may be provided on a part of the surface of the face 210 and the body 220. When the display unit 230 is provided on a part of the surface of the face 210 and the body 220, each physical part (for example, a mouth, eyes, nose, fingers) that constitutes the body, face, etc. may be provided on the part where the display unit 230 is not provided.

[0014] The display unit 230 can be configured as a display medium capable of displaying various images. As this display medium, various display media such as a light-emitting sheet, a liquid crystal display, an organic EL (Electro Luminescence), etc. For example, the display unit 230 of the face 210 displays eyes, a mouth, etc.

[0015] A drive unit 250 (see FIG. 3) for driving the body unit 220 is provided at the bottom of the agent device 200. The body unit 220 is configured to be able to swing in the front-rear and left-right directions of the agent device 200, with the bottom of the agent device 200 fixed. The body unit 220 is also configured to be able to rotate around the up-down direction of the body unit 220 as the rotation axis. The operating mode of the agent device 200 will be described in detail with reference to FIG. 2.

[0016] 1 shows an example of agent device 200 having face portion 210 and body portion 220, but agent device 200 having only face portion 210 may also be used. In this case, various facial expressions (for example, an expression gazing in a specific direction, an expression of praise) may be realized by changing face portion 210.

[0017] 1 shows an example in which the agent device 200 is configured with a substantially spherical face portion 210 and a body portion 220, but is not limited to this. For example, the agent device 200 may be a device in which a flat body portion (e.g., a planar display panel) functions as a face portion. For example, the agent device 200 may be a device in which the face portion and the body portion are configured as an integrated housing. For example, the agent device 200 may be a device in which the face portion and the body portion are configured as separate bodies. For example, the agent device 200 may be a display device that displays the face portion and the body portion.

[0018] The agent device 200 executes various operations based on instructions from the information processing device 110 (see FIG. 3). For example, the agent device 200 outputs various information related to driving support when the user U1 drives the vehicle and surrounding facilities, etc., based on the control of the information processing device 110. As the driving support, a notification of a moving object ahead or behind is assumed. For example, as a notification of a moving object ahead, a voice output of "Be careful because there is a railroad crossing ahead" or a voice output of "There is a person ahead" can be used. In this way, the agent device 200 executes driving support. Also, for example, the agent device 200 executes various processes based on the control of the information processing device 110. For example, the agent device 200 executes a communication process for carrying out various exchanges with the user U1. For example, the agent device 200 exchanges various conversations with the user U1 and provides information that the user U1 likes.

[0019] Here, the agent device 200 has a function of expressing some gestures and facial expressions to the user U1 in order to facilitate smooth communication when communicating with the user U1. In other words, the agent device 200 is capable of appropriately conveying information to the user U1 without making the user U1 feel uncomfortable. Note that the agent device 200 may use voice, images, vibrations, movements, etc. as means of expression to convey some information to the user U1. Note that specific expression methods will be described with reference to FIG. 2.

[0020] Here, it is assumed that a specific event (specific event) that should be notified to the user U1 occurs in the vehicle C1. The specific event means an event that, among events that occur in the vehicle C1, particularly requires the user U1 to take an active response and to eliminate the event that has occurred. For example, an event in which a warning light or an indicator light is turned on in a meter panel on which indicator lights and measuring instruments are displayed can be set as the specific event. Examples of the specific event include a fuel level abnormality (fuel level is below a threshold), a battery level abnormality (battery level is below a threshold), an ABS (Anti-lock Brake System) abnormality, etc. Also, for example, a behavior of the user U1 that does not comply with traffic rules can be set as the specific event. Examples of the specific event include not wearing a seat belt, stopping at a no-parking area, turning off the emergency flasher when stopping on the road, turning off the headlights at night, etc. Also, examples of the specific event include an abnormal state of the vehicle C1 and an abnormal state caused by a malfunction of an occupant of the vehicle C1. The abnormal state of the vehicle C1 is, for example, an abnormality in the remaining fuel amount, an abnormality in the remaining battery amount, an abnormality in the ABS, etc. Furthermore, the abnormal state caused by the erroneous operation of the occupant of the vehicle C1 is, for example, a seat belt not fastened, a half-open door, an abnormality in the remaining fuel amount, no necessary indicator lamps being lit, etc. These abnormal states caused by the erroneous operation of the occupant of the vehicle C1 are detected, for example, based on the occupant's deviation from a predetermined rule. Specific specific events will be described in detail with reference to FIG. 4.

[0021] For example, each time a specific event is detected, a notification voice or a warning sound related to the specific event may be output to the user U1. For example, when it is detected that the seat belt is not fastened, a voice notification to fasten the seat belt may be output from the agent device 200. However, when the user U1 is enjoying music or radio in the vehicle C1, it is assumed that the user U1 does not want to be interrupted by the output of the notification sound or warning sound related to the specific event. It is also assumed that the user U1 finds the voice notification or warning sound annoying and uncomfortable.

[0022] Furthermore, in the case where many specific events occur, many notification sounds and warning sounds related to the specific events will be output, and it is expected that the user U1 will find these notification sounds and warning sounds annoying and uncomfortable. Furthermore, if the agent device 200 outputs these notification sounds and warning sounds, the user U1 will find the agent device 200 annoying and uncomfortable, making it difficult to build a friendly and cooperative relationship between the user U1 and the agent device 200, and there is a risk of intimacy decreasing.

[0023] Therefore, in this embodiment, when a specific event is detected, information to be transmitted to user U1 (information about the specific event) is transmitted appropriately without causing discomfort to user U1. Specifically, when a specific event is detected, information to be transmitted to user U1 is transmitted by executing an action suggested by the agent device 200. Note that the suggested action means, for example, a gesture including a gaze action or a pointing action. This makes it possible to transmit information appropriately without causing discomfort to user U1.

[0024] [Example of agent device appearance configuration] 2 is a front view showing, in a simplified form, an example of the external configuration of the agent device 200. In this embodiment, an example is shown in which, when the agent device 200 is in an on state, the agent device 200 is in a basic state or in one of various operating states. That is, when the agent device 200 carries out various communications with the user U1, the information processing device 110 (see FIG. 3) executes control to switch the operating state of the agent device 200 depending on the content of the communication.

[0025] Fig. 2(A) shows a front view of the agent device 200 in a basic state. Fig. 2(B) shows a front view of the agent device 200 executing an action suggesting an object with a line of sight. Fig. 2(C) shows a front view of the agent device 200 executing an action suggesting an object with a hand and arm. Fig. 2(D) shows a front view of the agent device 200 executing an action suggesting an object with face 210 facing downward. Figs. 2(E) and (F) show front views of the agent device 200 executing an action of praising the user. The agent device 200 shown in Fig. 2(A) corresponds to the agent device 200 shown in Fig. 1.

[0026] The agent device 200 is a robot device having a substantially spherical face 210 provided on top of a prolate ellipsoid body 220. A rod-shaped arm 221 is provided on the right side of the body 220, a rod-shaped arm 222 is provided on the left side of the body 220, and rod-shaped feet 223 are provided on the left-right lower part of the body 220. Various images, such as a mouth and eyes, are displayed on the surface of the face 210. Note that, although this embodiment shows an example in which the agent device 200 is provided with only the arms 221 and 222, hands and fingers may be provided on the tips of the arms 221 and 222, and at least some of these may be used to perform a pointing action.

[0027] For example, as shown in Fig. 2(A), normal eyes and mouth can be displayed. Also, for example, as shown in Fig. 2(B) to (D), eyes and mouth corresponding to actions suggesting an object can be displayed on the face 210. Also, for example, as shown in Fig. 2(E) and (F), eyes and mouth corresponding to actions praising the user U1 can be displayed on the face 210.

[0028] 2(D), the face unit 210 rotates downward in the up-down direction. In this manner, the agent device 200 is provided with a driving unit 250 for driving the face unit 210. It is then possible to rotate and move the face unit 210 of the agent device 200 in the up-down and left-right directions, and to swing the face unit 210 in the up-down and left-right directions.

[0029] Also, the arm 221 rotates in the front-rear, left-right, up-down, etc. directions of the agent device 200, with the attachment part of the arm 221 on the body part 220 as the rotation center. Also, the central part of the arm 221 rotates in the front-rear, left-right, up-down, etc. directions of the agent device 200, with that central part as the rotation center. The same applies to the arm 222.

[0030] In this way, a driving unit 250 for driving the arm units 221, 222 is provided at the attachment and central portions of the arm units 221, 222. This makes it possible to move the arm units 221, 222 in up / down and left / right directions and to swing them. Note that the driving unit 250 that drives each unit of the agent device 200 executes each operation under the control of the information processing device 110 (see FIG. 3).

[0031] For example, as shown in FIG. 2B, the black part of the eye of the face 210 is moved, and the line of sight G1 of the face 210 is made to look in the direction of the object to be suggested to the user U1, by having the agent device 200 execute the action of pointing the tip of the arm 222 (corresponding to the hand and fingers) in the direction of the object to be suggested to the user U1, so that the object can be suggested to the user U1. For example, as shown in FIG. 2C, the agent device 200 is made to rotate the face 210 downward in the vertical direction, and the line of sight G2 of the face 210 is made to look in the direction of the object to be suggested to the user U1, so that the object can be suggested to the user U1.

[0032] Also, for example, as shown in FIG. 2(E), the arm parts 221, 222 are rotated upward in the vertical direction, and the agent device 200 is caused to execute an operation of displaying the eyes and mouth parts corresponding to a praising expression such as a smile on the face part 210, thereby enabling a performance of praising the user U1. In this case, text information 224, image information, etc., indicating the praising such as "amazing" may be displayed on the body part 220. Also, for example, as shown in FIG. 2(F), the arm parts 221 are rotated upward in the vertical direction, and the agent device 200 is caused to execute an operation of displaying the eyes and mouth parts corresponding to a praising expression such as a smile on the face part 210, thereby enabling a performance of praising the user U1. In this case, image information 225, etc., indicating the fingers and hands indicating "amazing" may be displayed on the body part 220. 2(E) and (F), as a gesture or performance expressing praise for the user U1, the gesture may be expressed by a smiling face, a happy face, or the like, or may be expressed by a voice output such as "Well done." The text information 224, image information 225, and the like may be displayed on an in-vehicle device of the vehicle C1 (e.g., the display device 300 or another display device) or another display device (e.g., a smartphone carried by the user U1).

[0033] In this manner, in this embodiment, it is possible to cause the agent device 200 to perform various types of operations by changing at least some of the components of the agent device 200. Furthermore, upon seeing each of these operations, the user U1 of the vehicle C1 can perceive that the agent device 200 is trying to communicate something. Note that these shapes and operation modes of the agent device 200 are merely examples, and each operation state may be expressed by other shapes and display modes.

[0034] Furthermore, these operational modes of the agent device 200 are merely examples, and other operational modes are also possible. For example, the agent device 200 may be moved by sliding within a predetermined range on the dashboard 2.

[0035] 2 shows an example in which each facial expression is expressed two-dimensionally by changing the display mode of the display unit 230, but each facial expression may be expressed three-dimensionally by changing each physically installed member such as the eye, nose, mouth, etc. In this case, for example, the movement of the eye may be expressed by moving the eye member or displaying a black image or a white image on the surface of the eye member.

[0036] [Example of information processing system configuration] FIG. 3 is a block diagram showing an example of a system configuration of an information processing system 100 installed in the vehicle C1.

[0037] The information processing system 100 includes a sound acquisition unit 101, a user image acquisition unit 102, a vehicle interior image acquisition unit 103, a vehicle exterior image acquisition unit 104, an information processing device 110, an agent device 200, and a display device 300. The information processing device 110 is an example of a device that controls the agent device 200 that is capable of communicating with a user U1.

[0038] The information processing device 110 and the agent device 200 are connected by a communication method using wired communication or wireless communication. The information processing device 110 is connected to the network 20 by a communication method using wireless communication. The network 20 is a network such as a public line network or the Internet. The agent device 200 may also be connected to the network 20 by a communication method using wireless communication. Although FIG. 3 shows an example in which the information processing device 110 and the agent device 200 are configured as separate entities, the information processing device 110 and the agent device 200 may also be configured as an integrated device.

[0039] The sound acquisition unit 101 is provided inside the vehicle C1, acquires sounds inside the vehicle C1, and outputs sound information related to the acquired sounds to the information processing device 110. As the sound acquisition unit 101, for example, one or more microphones or sound acquisition sensors can be used.

[0040] The user image acquisition unit 102 captures an image of a user riding in the vehicle C1 to generate an image (image data), and outputs image information related to the generated image to the information processing device 110. The user image acquisition unit 102 is provided at least inside the vehicle C1, and captures an image of each user riding in the vehicle C1 to generate an image (image data). The user image acquisition unit 102 is configured, for example, with one or more camera devices or image sensors capable of capturing an image of a user. For example, one user image acquisition unit 102 is provided in the front (for example, the ceiling) inside the vehicle C1, and is capable of capturing an image of a user from the front of the vehicle C1 to generate an image (image data). For example, the user image acquisition unit 102 can be provided at the top of the windshield 4, that is, above the rearview mirror 5. The user image acquisition unit 102 and the vehicle interior image acquisition unit 103 may be configured as the same device or as different devices.

[0041] The vehicle interior image acquisition unit 103 captures an image of a subject inside the vehicle C1 to generate an image (image data), and outputs image information related to the generated image to the information processing device 110. The vehicle interior image acquisition unit 103 is provided at least inside (e.g., the ceiling) of the vehicle C1, and captures an image of a subject inside the vehicle C1 to generate an image (image data). The vehicle interior image acquisition unit 103 is configured, for example, with one or more camera devices or image sensors capable of capturing an image of a subject. For example, one vehicle interior image acquisition unit 103 may be provided in the front of the vehicle C1 to capture an image of a subject from the front to generate an image (image data), and another vehicle interior image acquisition unit 103 may be provided in the rear of the vehicle C1 to capture an image of a subject from the rear to generate an image (image data).

[0042] The outside-vehicle image acquisition unit 104 captures an image of a subject outside the vehicle C1 to generate an image (image data), and outputs image information related to the generated image to the information processing device 110. Two or more outside-vehicle image acquisition units 104 may be provided, and all or some of the images from these outside-vehicle image acquisition units 104 may be used. For example, one outside-vehicle image acquisition unit 104 may be provided in front of the vehicle C1 to capture an image of a subject from in front of the vehicle C1 to generate an image (image data), and another outside-vehicle image acquisition unit 104 may be provided behind the vehicle C1 to capture an image of a subject behind the vehicle C1 to generate an image (image data). In addition, one or more devices capable of capturing subjects present in all directions of the vehicle C1 and subjects inside the vehicle C1, such as a 360-degree camera, may be used.

[0043] Each of the user image acquisition unit 102, the vehicle interior image acquisition unit 103, and the vehicle exterior image acquisition unit 104 is configured with, for example, an image sensor that receives light from a subject collected by a lens, and an image processing unit that performs predetermined image processing on image data generated by the image sensor. For example, a CCD (Charge Coupled Device) type or a CMOS (Complementary Metal Oxide Semiconductor) type image sensor can be used as the image sensor.

[0044] The information processing device 110 includes a control unit 120, a storage unit 130, and a communication unit 140. The communication unit 140 exchanges various types of information with other devices based on the control of the control unit 120 by using wired communication or wireless communication.

[0045] The control unit 120 controls each unit based on various programs stored in the storage unit 130. The control unit 120 is realized by a processing device such as a central processing unit (CPU) or a graphics processing unit (GPU). Note that the vehicle ECU (electronic control unit) of the vehicle C1 may be used as the control unit 120, or a processing device different from the vehicle ECU may be provided as the control unit 120.

[0046] The control unit 120 executes various controls based on each piece of information output from the sound acquisition unit 101, the user image acquisition unit 102, the vehicle interior image acquisition unit 103, the vehicle exterior image acquisition unit 104, the communication unit 140, etc., and each piece of information acquired by the vehicle state acquisition unit 125. For example, the control unit 120 executes a control process for controlling the operation state of the agent device 200 and the display state of the display device 300. Specifically, the control unit 120 includes an utterance acquisition unit 121, a user state acquisition unit 122, a vehicle interior state acquisition unit 123, a vehicle exterior state acquisition unit 124, a vehicle state acquisition unit 125, a detection unit 126, and an agent control unit 127.

[0047] The speech acquisition unit 121 executes a predetermined sound analysis process on the sound information output from the sound acquisition unit 101 to acquire speech information related to the user's speech contained in the sound information, and outputs the speech information to the detection unit 126. This sound analysis process can be a well-known sound analysis process. That is, the speech acquisition unit 121 acquires user state information related to the state of each user riding in the vehicle C1.

[0048] The user status acquisition unit 122 acquires various pieces of information about the user contained in the image information output from the user image acquisition unit 102 by performing a predetermined image analysis process on the image information, and outputs each piece of information to the detection unit 126. This image analysis process can be performed using a known image analysis process. As various pieces of information about the user, for example, each action of the user related to the user's facial expression, the user's gaze, the movement of the hands, face, body, etc. can be acquired. From these, for example, it is possible to detect whether the user gets on and off the vehicle C1, whether each seat in the vehicle C1 is occupied, the action of fastening the seat belt, etc. That is, the user status acquisition unit 122 acquires user status information about the status of each user who gets on the vehicle C1.

[0049] The vehicle interior status acquisition unit 123 acquires various pieces of information related to the interior of the vehicle C1 contained in the image information output from the vehicle interior image acquisition unit 103 by performing a predetermined image analysis process on the image information, and outputs the pieces of information to the detection unit 126. This image analysis process can be performed using a known image analysis process. As various pieces of information related to the interior of the vehicle C1, for example, whether or not a seat belt is fastened, objects present in the vehicle interior (for example, a smartphone, a book), etc. can be detected. That is, the vehicle interior status acquisition unit 123 acquires user status information related to the status of each user riding in the vehicle C1, and vehicle status information related to the status of the vehicle C1.

[0050] The vehicle exterior situation acquisition unit 124 acquires various pieces of information related to the outside of the vehicle C1 contained in the image information output from the vehicle exterior image acquisition unit 104 by performing a predetermined image analysis process on the image information, and outputs the pieces of information to the detection unit 126. This image analysis process can be performed using a known image analysis process. As various pieces of information related to the outside of the vehicle C1, for example, it is possible to detect whether the vehicle C1 is traveling on a road, whether the vehicle C1 is stopped on the road, signs present in front of the vehicle C1, etc. In other words, the vehicle exterior situation acquisition unit 124 acquires vehicle state information related to the state of the vehicle C1.

[0051] The vehicle state acquisition unit 125 acquires information (vehicle state information) on various vehicle states related to the vehicle C1, and outputs the acquired vehicle state information to the detection unit 126. The vehicle state information can be acquired, for example, from a CAN (Controller Area Network) signal. The vehicle state information includes, for example, vehicle speed, acceleration, position of a shift lever (for example, P range, D range), accelerator pedal depression amount, brake pedal depression amount, remaining battery charge (SOC (State Of Charge)), remaining fuel amount, ABS abnormality information, position information of the vehicle C1, half-open door information, whether each indicator lamp is on or off, etc. For example, it is possible to determine whether the vehicle C1 is stopped or running based on the vehicle speed, acceleration, etc.

[0052] The vehicle state acquisition unit 125 may acquire sensor detection information output from various sensors installed in the vehicle C1. The sensors include, for example, LiDAR (Light Detection And Ranging), RADAR (Radio Detection And Ranging), Sonar, a vehicle speed sensor, an acceleration sensor, a steering sensor, an accelerator position sensor, a position information acquisition sensor (position information acquisition unit), a fuel sensor, a battery sensor, a door sensor, an illuminance sensor, a seating sensor, a seat belt sensor, and the like. For each of these sensors, a known sensor can be used. Note that LiDAR, RADAR, Sonar, and the like are examples of sensors that detect the surrounding situation of the vehicle C1. Also, the vehicle speed sensor, the acceleration sensor, the steering sensor, the accelerator position sensor, and the like are examples of sensors that detect the driving operation situation of the driver D1. Note that these are only examples, and other sensors may be used. Also, only some of these sensors may be used.

[0053] The location information acquisition unit acquires location information relating to the location where the vehicle C1 is located. For example, it can be realized by a GNSS receiver that acquires location information using a Global Navigation Satellite System (GNSS). The location information includes data relating to the location such as latitude, longitude, and altitude at the time of receiving the GNSS signal. The location information may also be acquired by other methods of acquiring location information. For example, the location information may be derived using information from access points or base stations present in the vicinity. The location information may also be acquired using a beacon.

[0054] The fuel sensor is a sensor for measuring the remaining amount of fuel for driving the vehicle C1. The battery sensor is a sensor for measuring the remaining amount of a battery installed in the vehicle C1. The door sensor is a sensor for detecting whether each door of the vehicle C1 is half-open. The illuminance sensor is a sensor for detecting the brightness around the vehicle C1. The seating sensor (or seat sensor) is a sensor for detecting the presence or absence of an occupant seated in each seat of the vehicle C1. The seat belt sensor is a sensor for detecting whether the occupant seated in each seat of the vehicle C1 is wearing a seat belt. For example, whether the user U1 seated in the driver's seat is wearing a seat belt can be detected using a seating sensor, a seat belt sensor, or the like.

[0055] The detection unit 126 detects the occurrence and disappearance of a specific event (specific event) to be notified to any user riding in the vehicle C1 based on each piece of information output from the speech acquisition unit 121, the user status acquisition unit 122, the vehicle interior status acquisition unit 123, the vehicle exterior status acquisition unit 124, the vehicle status acquisition unit 125, etc. That is, the detection unit 126 acquires at least one of user status information related to the status of each user riding in the vehicle C1 and vehicle status information related to the status of the vehicle C1. Then, the detection unit 126 detects the occurrence and disappearance of a specific event to be notified to any user based on at least one of the acquired user status information and vehicle status information. For example, when it is detected that a user deviates from a predetermined rule based on the user status information or the vehicle status information, the detection unit 126 detects that a specific event has occurred. For example, a user's deviation from the rule of wearing a seat belt (not wearing a seat belt) can be detected using a seating sensor, a seat belt sensor, etc. Detection is also possible based on information about the user (whether or not the user is wearing a seat belt on his / her body) acquired by the user status acquisition unit 122. For example, when a seat belt is present diagonally on the upper body of the user in an image including the upper body of the user, wearing of the seat belt can be detected.

[0056] Also, for example, the detection unit 126 detects the occurrence of a specific event when it is detected that a predetermined abnormality has occurred in the vehicle C1 based on the vehicle state information. For example, it is possible to detect the occurrence of a battery remaining amount abnormality, a fuel remaining amount abnormality, or an ABS abnormality based on the battery remaining amount information, the fuel remaining amount information, and the ABS abnormality information. Also, when the occurrence of a specific event that should be notified to the user is detected, the detection unit 126 generates detection information including the type of the specific event that has occurred and the position where the specific event has occurred. Also, when the detection unit 126 detects the disappearance of the specific event, it generates detection information including the type of the disappeared specific event and the position where the specific event has disappeared. Then, the detection unit 126 outputs the detection information to the agent control unit 127. Note that a method for detecting the occurrence and disappearance of a specific event will be described in detail with reference to FIG. 4.

[0057] The agent control unit 127 controls the operating state of the agent device 200. Specifically, the agent control unit 127 controls the operating state of the agent device 200 based on the detection result output from the detection unit 126. For example, the agent control unit 127 causes the display unit 230 to display each part (e.g., eyes, mouth) constituting the face unit 210 of the agent device 200. Also, for example, the agent control unit 127 causes the sound output unit 240 to output the voice of the agent device 200, the voice corresponding to the information to be suggested or notified to the user U1, and the like. Also, for example, the agent control unit 127 causes the agent device 200 to execute various operations. For example, the agent control unit 127 causes the agent device 200 to execute an operation corresponding to the information to be suggested or notified to the user U1.

[0058] For example, when a specific event is detected, the agent control unit 127 causes the agent device 200 to execute a first operation suggesting the specific event. Also, when the disappearance of the specific event is not detected after a predetermined time has elapsed since the execution of the first operation, the agent control unit 127 causes the agent device 200 to execute a new second operation for notifying the specific event. Each of these operations will be described in detail with reference to Figures 4 to 10, etc.

[0059] The storage unit 130 is a storage medium that stores various types of information. For example, the storage unit 130 stores various types of information (for example, a control program, an agent information DB 131, a specific event response information DB 400, and a map information DB) required for the control unit 120 to perform various processes. The storage unit 130 also stores various types of information acquired via the communication unit 140. As the storage unit 130, for example, a ROM (Read Only Memory), a RAM (Random Access Memory), an SRAM (Static Random Access Memory), an HDD (Hard Disk Drive), an SSD (Solid State Drive), or a combination of these can be used.

[0060] The agent information DB 131 stores various types of information required to realize various operations of the agent device 200. For example, face image information (e.g., eyes, mouth) displayed on the display unit 230 and audio information output from the sound output unit 240 are stored in the agent information DB 131. Also, for example, operation information for operating the agent device 200 when performing various communications is stored in the agent information DB 131.

[0061] The specific event response information DB 400 stores various information required when the agent device 200 executes a first action of suggesting a specific event to the user U1, a second action of notifying the user U1 of the specific event, etc. The specific event response information DB 400 will be described in detail with reference to FIG.

[0062] The agent device 200 is a robot device that executes various operations based on instructions from the information processing device 110. The agent device 200 comprises a display unit 230, a sound output unit 240, and a drive unit 250. The display unit 230, the sound output unit 240, and the drive unit 250 are controlled by a control unit (not shown) that the agent device 200 comprises.

[0063] The display unit 230 is a display unit that displays various images based on instructions from the information processing device 110. The display unit 230 may be configured as a touch panel that allows a user to perform operation input by touching or approaching the display surface with his / her finger, or may be configured as a separate user interface.

[0064] The sound output unit 240 outputs various sounds based on instructions from the information processing device 110. For example, one or more speakers can be used as the sound output unit 240. Note that the display unit 230 and the sound output unit 240 are examples of user interfaces, and some of them may be omitted, or other user interfaces may be used.

[0065] The driving section 250 drives each section of the agent device 200 based on instructions from the information processing device 110. For example, the driving section 250 is configured by a servo motor or the like.

[0066] The display device 300 is a display device that executes various output operations based on instructions from the information processing device 110. The display device 300 includes a display unit 310 and a sound output unit (not shown). The display unit 310 and the sound output unit are controlled based on a control unit (not shown) included in the display device 300. For example, the display device 300 is an in-vehicle system that includes one or more devices that can provide various information. As the display device 300, for example, at least one of a navigation device, an audio device, a DVD device, a TV tuner device, an IVI (In-Vehicle Infotainment), and the like can be used. In this embodiment, an example is shown in which the display device 300 is also used as a navigation device. Also, instead of the display device 300, a HUD (Head Up Display) realized on the front window 4 can be used. Also, other devices that can output other information, such as a display device and a sound output device, may be used. Also, a portable device that the user U1 can carry, such as a smartphone, a tablet terminal, or an information processing device such as a portable personal computer, may be used.

[0067] [Example of specific event DB configuration] FIG. 4 is a simplified diagram showing an example of the configuration of the specific event response information DB 400 stored in the storage unit 130. As shown in FIG.

[0068] The specific event response information DB400 is a database used when a specific event (specific event) that should be notified to any occupant occurs in the vehicle C1, and some processing is performed for the specific event. Specifically, first operation information 402, second operation information 403, and notification processing information 404 are stored in the specific event response information DB400 in association with the specific event information 401.

[0069] The specific event information 401 is information indicating a specific event (specific event) that should be notified to any occupant in the vehicle C1. Note that the specific event shown in Fig. 4 is an example, and other events may be set as the specific event.

[0070] The first action information 402 is information indicating an action to be executed by the agent device 200 when a specific event occurs. As this first action, the agent device 200 is caused to execute an action to suggest the specific event to the occupant. Note that since the first action is an action of directing the line of sight, fingertip, etc. of the agent device 200 toward an object related to the specific event, the first action can also be called a following action. The first action executed by the agent device 200 will be described in detail with reference to Figs. 5 to 10 etc.

[0071] Here, Fig. 4 shows an example in which the first action is an action that suggests a specific event to the passenger using the line of sight of the agent device 200, but the first action shown in Fig. 4 is only an example, and other suggesting actions may be set as the first action. For example, as shown in Fig. 2(C), the first action may be an action that points to an object related to a specific event using the arm 221 or 222 of the agent device 200. Also, for example, as shown in Fig. 9(B), an image corresponding to an agent device (agent image AE1) or other images may be displayed on the display unit 310 of the display device 300, and an action that suggests a specific event using these images may be set as the first action.

[0072] The second action information 403 is information indicating an action to be performed by the agent device 200 when a specific event occurs and the specific event does not disappear even after a certain time has elapsed since the agent device 200 performed the first action. As the second action, the agent device 200 is caused to perform an action having a larger amount of action than the first action, an action faster than the first action, an action directly pointing to an object related to the specific event, and the like. That is, the second action means an action to inform an occupant of the occurrence of the specific event using a transmission means having a higher transmission power than the first action, for an occupant who does not notice the occurrence of the specific event even after the first action is performed. In other words, the second action can be called an action that is more easily recognized by a user than the first action, an action that is more likely to be recognized by a user than the first action, and the like. The second action performed by the agent device 200 will be described in detail with reference to FIG. 9, FIG. 10, and the like.

[0073] Here, in FIG. 4, an example is shown in which the second action is an action of notifying the passenger of a specific event using a voice output from the agent device 200, but the second action shown in FIG. 4 is only an example, and another notifying action may be set as the second action. For example, some gesture of the agent device 200 (for example, a gesture of waving the arms 221, 222 toward the user U1) may be executed together with the voice output from the agent device 200. Also, for example, the second action may be set to an action that is the same as the action executed as the first action but with a larger amount of movement or faster. Also, an action different from the action executed as the first action may be set as the second action. For example, when the line of sight is used as the first action, a pointing action using the arms 221 or 222 of the agent device 200 may be set as the second action. Also, for example, an image (agent image AE1) corresponding to the agent device or other images may be displayed on the display unit 310 of the display device 300, and an action using these images and voice information may be set as the second action.

[0074] In this manner, in this embodiment, when a specific event is detected, an action that does not involve voice output from the agent device 200 is executed as a first action as a first stage of user notification processing. Then, when the user U1 does not notice the specific event even after the execution of the first stage of user notification processing, an action that serves as a means of communication with a higher degree of communication power than the first action is executed as a second action as a second stage of user notification processing. In this second action, voice output from the agent device 200 may be executed. In this manner, when a specific event is detected, an indirect suggestive action that does not use voice is executed in the first stage, and an action that is easier for the user U1 to recognize than the first action is executed in the second stage. That is, by executing a gentle action (gaze, pointing) toward the user U1 in the first stage, it is possible to reduce the possibility of giving discomfort to the user U1. Also, by executing a slightly more intense action toward the user U1 in the second stage, it is possible to increase the possibility of being recognized by the user U1.

[0075] The notification process information 404 is information indicating a notification process for notifying the occupant of a specific event when the specific event occurs and does not disappear even after a certain time has elapsed since the agent device 200 performed the second action. As this notification process, it is possible to execute a notification process in which the contents of the specific event are directly conveyed to the occupant by voice or text information. In this case, it is possible to execute the notification process without using the agent device 200.

[0076] The information stored in the first action information 402, the second action information 403, and the notification processing information 404 may be fixed information, or may be changed based on some information. For example, each piece of information may be changed based on the reaction of the user U1 after the agent device 200 executes the first action (or the second action) after the occurrence of a specific event. For example, if the user U1 cannot understand the first action (e.g., gaze action) executed in response to the detection of the seat belt not being fastened, the content of the first action can be changed to another action (e.g., pointing action with hands and arms). Also, for example, if the user U1 cannot understand the first action (e.g., gaze action) and the second action (e.g., pointing action with hands and arms) executed in response to the detection of the door being ajar, the content of the first action can be changed to another action (e.g., pointing action with hands and arms), and the content of the second action can be changed to another action (e.g., loud voice output action).

[0077] [Detection of specific events] Next, a detection method for detecting the occurrence of a specific event and a detection method for detecting the disappearance of a specific event will be described.

[0078] [Detection example regarding seat belts] Whether or not the user in the vehicle C1 is fastening the seat belt can be detected using sensor information from a seating sensor, a seat belt sensor, etc. Also, whether or not the seat belt is fastened may be detected based on images acquired by the user image acquisition unit 102, the vehicle interior image acquisition unit 103, etc. For example, whether or not the user is fastening the seat belt can be detected using a known image recognition technology.

[0079] [Detection example of a half-open door] It is possible to detect whether any door of the vehicle C1 is half-open by using a door sensor. For example, it is possible to determine that the door is half-open when a door lock signal is not detected from the door sensor. Here, the half-open state means that the door is not completely closed.

[0080] [Detection example of parking in a no parking area] Whether the vehicle C1 is stopped at a parking and stopping prohibited place can be detected based on the image of the surroundings of the vehicle C1 acquired by the vehicle exterior image acquisition unit 104 and the vehicle state information. For example, if a parking and stopping prohibited sign is included in the image in front of the vehicle C1 and the vehicle C1 is stopped near the sign, it is possible to detect that the vehicle C1 is stopped at a parking and stopping prohibited place. The stopping of the vehicle C1 can be determined based on the vehicle speed, acceleration, the position of the shift lever (position of the P range), etc. Also, it is possible to detect the stopping at the parking and stopping prohibited place based on the current position of the vehicle C1 acquired by the position information acquisition unit and the position of the parking and stopping prohibited place in the map information DB. For example, if the current position of the vehicle C1 is near the parking and stopping prohibited place and the vehicle C1 is stopped near the parking and stopping prohibited place, it is possible to detect that the vehicle C1 is stopped at the parking and stopping prohibited place.

[0081] In addition, whether the vehicle C1 is not stopped at a parking and stopping prohibited place (starting from that place) can be detected based on the fact that the starting of the vehicle C1 is detected after the stopping at the parking and stopping prohibited place described above is detected. The starting of the vehicle C1 can be determined based on the position of the shift lever (D range position), the depression amount of the accelerator pedal, the vehicle speed, the acceleration, etc.

[0082] [Detection example of no hazard flashers when a vehicle is stopped on the road] The non-illumination of the emergency flashers (hazard lamps) when the vehicle is stopped on the road can be detected based on the position information of the vehicle C1, the position of the shift lever, and the state of the emergency flashers (on / off signal of the emergency flashers). For example, when the current location of the vehicle C1 is on a road and the position of the shift lever is in the P range, if the emergency flashers are in the off state (off signal of the emergency flashers), the non-illumination of the emergency flashers when the vehicle is stopped on the road can be detected. Whether the current location of the vehicle C1 is on a road can be determined based on an image of the front (or surroundings) of the vehicle C1 acquired by the outside image acquisition unit 104, or based on a comparison between the position information acquired by the position information acquisition unit and the road in the map information DB, or the like.

[0083] In addition, the fact that vehicle C1 has started to turn on the hazard flashers when stopped on the road can be detected based on the detection of the hazard flashers being turned on (the on signal for the hazard flashers) after the above-mentioned detection that the hazard flashers are not on.

[0084] [Detection example of no lights on at night] The absence of the lights of the vehicle C1 while traveling at night can be detected based on the brightness of the surroundings of the vehicle C1 and whether the lights are on or off (lights on / off signal). For example, when the vehicle C1 is traveling and the illuminance based on the sensor information output from the illuminance sensor is detected to be equal to or lower than a threshold, if the lights are not on (lights off signal), the absence of the lights of the vehicle C1 while traveling at night can be detected. On the other hand, if the absence of the lights of the vehicle C1 while traveling at night is detected and then the lights are on (lights on signal), the presence of the lights of the vehicle C1 while traveling at night can be detected.

[0085] [Example of detection of remaining battery capacity] Whether or not the remaining battery charge has fallen below a threshold value can be detected using sensor information from a battery sensor.

[0086] [Detection example of remaining fuel amount] Whether or not the remaining fuel amount has fallen to a threshold value or less can be detected using sensor information from a fuel sensor.

[0087] [Detection example of ABS abnormality] Whether or not an ABS abnormality has occurred can be detected using ABS abnormality information acquired from a CAN signal, for example.

[0088] [Example of an agent device that suggests to the user the detection of a specific event] Next, a transition example will be described in which, when a specific event is detected in the vehicle C1, the agent device 200 executes an operation of suggesting to the user U1 an object related to the specific event.

[0089] [Example of transition when suggesting to the user that their seat belt is not fastened] 5 and 6 are diagrams showing an example of a transition in which the agent device 200 executes an operation to suggest to the user U1 to fasten the seat belt when it is detected that the seat belt is not fastened. In the examples shown in Figs. 5 and 6, for ease of explanation, only the seat belt buckle 6 of the driver's seat is illustrated as the configuration of the seat belt of the vehicle C1. Also, since the components other than the buckle 6 are the same as those in the vehicle C1 shown in Fig. 1, explanations of the parts common to Fig. 1 will be omitted. Also, in Figs. 5 and 6, the directions of the lines of sight G11 to G15 of the user U1 and the agent device 200 are shown typically by dotted lines.

[0090] FIG. 5(A) shows an example of a conversation between a user U1 seated in the driver's seat and the agent device 200. For example, the user U1, who is in the driver's seat of a vehicle C1, utters a voice S1 "It's a good day for a drive!" to the agent device 200, and the agent device 200 responds with a voice S2 "It's a fine day in Yokohama today." Note that the conversation between the user U1 and the agent device 200 is only a part of the conversation, and there are other conversations besides the above. During these conversations, the line of sight G11 and G12 of the user U1 and the agent device 200 may be in the direction of looking at each other. Assume that during or after this conversation, before fastening the seat belt, the user U1 turns on the start key of the vehicle C1, and the seat belt switch does not turn on even after a predetermined time has passed since the on operation, or the shift lever is shifted to the D range before the predetermined time has passed since the on operation. In this case, the detection unit 126 detects that the seat belt of the user U1 seated in the driver's seat is not fastened. That is, the detection unit 126 detects that the seat belt is not fastened as a specific event, and detects the driver's seat (or the seat belt and buckle thereof) as the location where the specific event occurred.

[0091] 5(B) shows an example of a case where, after it is detected that the driver's seat belt is not fastened, the agent device 200 executes an action suggesting that the driver's seat belt is not fastened. In this example, the agent device 200 directs its gaze G13 to the buckle 6 of the driver's seat belt as an action suggesting that the driver's seat belt is not fastened.

[0092] Specifically, the agent control unit 127 extracts a first action corresponding to the specific event detected by the detection unit 126 from the specific event response information DB 400 (see FIG. 4). Furthermore, the agent control unit 127 causes the agent device 200 to execute the extracted first action based on the position (position where the specific event occurred) detected by the detection unit 126. In the example shown in FIG. 5(B), since it has been detected that the seat belt in the driver's seat is not fastened, an action of directing the gaze toward the buckle 6 (target buckle) of the driver's seat is executed as the first action. For example, the agent control unit 127 controls the display state of the face unit 210 (display unit 230) so that the pupils of the eyes of the face unit 210 face the direction of the buckle 6.

[0093] When the conversation between the user U1 and the agent device 200 continues, the agent device 200 may temporarily stop speaking and direct the gaze G13 to the buckle 6, or the agent device 200 may continue speaking and direct the gaze G13 to the buckle 6. In response to such an action of the agent device 200, the user U1 may realize that the agent device 200 wants to tell him something. For example, the user U1 may utter a voice S3 "What's wrong?" to the agent device 200 that has started directing the gaze G13 to the buckle 6 during a conversation. Then, the user U1 may look at the buckle 6 to which the gaze G13 of the agent device 200 is directed and realize that the agent device 200 wants to tell him to fasten his seat belt. This example is shown in FIG. 6(A).

[0094] 6(A) shows an example of a case where the user U1 understands the content of the suggestion made by the agent device 200 in response to the execution of the action suggesting that the driver's seat belt is not fastened. For example, when the agent device 200 executes the action suggesting that the seat belt is not fastened with a line of sight G13, the user U1 may interrupt the conversation with the agent device 200 and look ahead of the line of sight G13 of the agent device 200 (line of sight G14). In this way, by looking ahead of the line of sight G13, the user U1 may understand the content suggested by the agent device 200 and fasten the seat belt. In this case, it is also assumed that the user U1 may utter a voice indicating that he or she has understood (for example, voice S4 "I see, I'll fasten it").

[0095] When the detection unit 126 detects that the user U1 has fastened the seat belt, it detects that the specific event (not fastening the seat belt) has disappeared. In this case, the user U1 understands the action (first action) suggested by the agent device 200, and performs an action to praise the user U1 for the disappearance of the specific event suggested by the agent device 200. An example of this is shown in Fig. 6(B).

[0096] FIG. 6(B) shows an example in which the user U1 puts on his / her seat belt after performing an action suggesting that the seat belt is not fastened. In this case, the user U1 understands and performs what the agent device 200 has suggested. Therefore, the agent device 200 performs an action to praise the user U1. For example, as in the example shown in FIG. 2(E), it is possible to perform an action of raising the arms 221, 222 and displaying the eyes and mouth corresponding to a praising expression such as a smile on the face 210. In this case, it is possible to visually express praise for the user U1 by displaying text information 224 and image information indicating a praise such as "amazing" on the body 220.

[0097] In addition, when the user U1 understands and executes what the agent device 200 suggests, other than the action of praising the user U1, a confirmation action to confirm that the user U1 has executed may be executed. For example, after the user U1 fastens the seat belt, the agent device 200 can execute an action of directing the gaze toward the buckle 6 (or the seat belt) again, or the agent device 200 can execute a pointing action to point to the buckle 6 (or the seat belt) using the arms 221, 222. By executing these confirmation actions by the agent device 200, it is possible to make the user feel that the agent device 200 is pretending to check the fastening of the seat belt. In this way, when the specific event disappears, it is possible to execute a confirmation action of directing at least a part of the agent device 200 toward an object related to the specific event. Then, after these confirmation actions, it is possible to execute the above-mentioned action of praising the user U1 (for example, a smile, a facial expression expressing joy, etc., a voice output such as "good job," and each of these effects).

[0098] 6 shows an example in which an action to praise the user U1 is executed on the condition that the specific event (not wearing a seat belt) suggested by the agent device 200 has disappeared. However, an action to praise the user U1 may be executed on the condition that it is estimated that the specific event suggested by the agent device 200 will disappear. For example, if the user U1 utters the voice S4 "I see, I'll fasten the seat belt" in response to an action by the agent device 200 suggesting that the driver's seat belt is not worn, it can be estimated that the user understands the content of the suggestion and will take an action to eliminate the suggested specific event.

[0099] 5 and 6 show an example in which it is detected that the seat belt of the user U1 in the driver's seat is not fastened. However, it is also possible to execute the first action suggested by the agent device 200 in the same way when it is detected that the seat belt of an occupant in another seat (passenger seat, rear seat) is not fastened. In this case, based on the position where the seat belt is not fastened, the agent device 200 executes an operation of directing the gaze to the seat belt or buckle corresponding to that position. Note that, when the agent device 200 is relatively far from the seat, a top view of each seat in the vehicle C1 may be displayed on the display unit 310 of the display device 300, and the action suggested by the agent device 200 (for example, gaze, pointing) may be executed to the corresponding seat in the top view.

[0100] [Example of when there is no object related to a specific event inside the vehicle] 5 and 6 show an example in which an object (seat belt and buckle) related to a specific event is present in the vehicle cabin. Here, in the case of a specific event such as when the remaining battery charge is below a threshold or when the remaining fuel charge is below a threshold, an object related to the specific event is not present in the vehicle cabin. In such a case, it is possible to display an image related to the specific event on the display unit 310 of the display device 300 and have the agent device 200 suggest this image. Thus, such an example is shown in FIGS. 7 and 8. Note that, although FIGS. 7 and 8 show an example in which an image related to a specific event is displayed on the display unit 310 of the display device 300, the image may be displayed on another display device (e.g., a meter panel, a HUD) and the agent device 200 may suggest this image.

[0101] [Example of transition when informing the user that the remaining fuel level is below the threshold] 7 and 8 are diagrams showing an example of a transition in which, when it is detected that the remaining fuel amount is equal to or less than a threshold, the agent device 200 executes an operation to indicate this to the user U1. Note that the example shown in Figs. 7 and 8 is similar to the interior of the vehicle C1 shown in Fig. 1, and therefore a description of the parts common to Fig. 1 will be omitted. Also, since the example shown in Figs. 7 and 8 is a modified example of the example shown in Figs. 5 and 6, a description of the parts common to Figs. 5 and 6 will be omitted. Also, in Figs. 7 and 8, the directions of the lines of sight G21 to G25 of the user U1 and the agent device 200 are shown typically by dotted lines.

[0102] FIG. 7(A) shows an example of a conversation between a user U1 seated in the driver's seat and the agent device 200. For example, the user U1 utters a voice S11 "What is tasty in Yokohama?" to the agent device 200, and the agent device 200 utters a voice S12 "When you think of Yokohama, you think of Chinese food." During this conversation, the gazes G21 and G22 of the user U1 and the agent device 200 are considered to be in the direction of looking at each other. Assume that during or after this conversation, the detection unit 126 detects that the remaining fuel amount is below a threshold. In this case, the detection unit 126 detects that the remaining fuel amount is below a threshold as a specific event, and detects the position of the fuel tank as the position where the specific event occurred. Here, since the fuel tank does not exist in the vehicle cabin, it is difficult for the agent device 200 to suggest its position. Therefore, in this case, a sign image O1 related to fuel is displayed on the display unit 310 of the display device 300, and the agent device 200 suggests the sign image O1. An example of this is shown in FIG.

[0103] 7(B) shows an example of a case where, after it is detected that the remaining fuel amount is equal to or less than the threshold, the agent device 200 executes an operation to suggest that the remaining fuel amount is equal to or less than the threshold. In this example, the agent device 200 displays a sign image O1 relating to fuel on the display unit 310 as an operation to suggest that the remaining fuel amount is equal to or less than the threshold, and directs the line of sight G23 to the sign image O1.

[0104] In the example shown in FIG. 7(B), since it is detected that the remaining fuel amount is equal to or less than the threshold, an action of directing the line of sight G23 to the sign image O1 displayed on the display unit 310 is executed as the first action. For example, the agent control unit 127 rotates the face unit 210 (display unit 230) downward so as to face the direction of the display unit 310 of the display device 300, and controls the display state of the black part of the eye of the face unit 210. Note that the first action can also be executed in the same manner when there is some obstacle between the agent control unit 127 and the display device 300. In addition, when the agent device 200 itself can move on the dashboard 2, the agent device 200 itself may be moved toward the display device 300 (to the rear of the vehicle C1) so that the agent device 200 can direct the line of sight G23 to the display unit 310 of the display device 300.

[0105] For example, the user U1 may utter the voice S13 "What's wrong?" to the agent device 200 that has begun to direct its gaze G23 at the display device 300 during a conversation. When the user U1 looks at the display device 300 at which the agent device 200's gaze G23 is directed, he or she notices that a sign image O1 is being displayed on the display unit 310 instead of map information. In this case, the user U1 may realize that the agent device 200 wants to tell him or her something about fuel. This example is shown in Fig. 8(A).

[0106] 8(A) shows an example of a case where the user U1 understands the content of the suggestion made by the agent device 200 in response to the execution of an action suggesting that the remaining fuel amount is below a threshold. For example, when the agent device 200 executes an action suggesting that the remaining fuel amount is below a threshold with a line of sight G23, the user U1 may interrupt the conversation with the agent device 200 and look ahead of the line of sight G23 of the agent device 200 (line of sight G24). In this way, the user U1 may understand the content suggested by the agent device 200 by looking ahead of the line of sight G23, and the user U1 may utter a voice S14 to the agent device 200 saying, "Right, I need to fill up with fuel." In this case, the user U1 may refuel.

[0107] When the detection unit 126 detects that the vehicle C1 has been refueled, it detects that the specific event (the remaining fuel amount is equal to or less than a threshold) has disappeared. In this case, the user U1 understands the action (first action) suggested by the agent device 200, and performs an action to praise the user U1 for the disappearance of the specific event suggested by the agent device 200. An example of this is shown in Fig. 8(B).

[0108] Here, it is assumed that the time until the vehicle C1 is refueled may become long. Therefore, for a specific event that is expected to take a long time to disappear, it is preferable to execute an action to praise the user U1 on the condition that the specific event suggested by the agent device 200 is estimated to disappear. For example, when the user U1 makes a specific utterance (e.g., the voice S16 "Thank you for letting me know," "Please show me the gas station") in response to an action that suggests that the remaining fuel is below a threshold, it is possible to estimate that the user U1 will understand the suggested content and take an action to disappear the suggested specific event. That is, based on the user state information (e.g., a specific utterance) acquired by the utterance acquisition unit 121, it is possible to estimate that the user U1 will take an action to disappear the suggested specific event. It is also assumed that the user U1 will make an expression or gesture that indicates that he or she understands the suggested content. Therefore, based on the user state information (e.g., a face image or a whole image of the user U1) acquired by the user status acquisition unit 122, it is possible to estimate that the user U1 will take an action to disappear the suggested specific event. It is also assumed that after the user U1 understands the suggested content, he or she will take some action to eliminate the specific event suggested by the agent device 200. For example, it is assumed that the user U1 searches for a gas station using a navigation device (e.g., an in-vehicle device of the vehicle C1, or another electronic device (e.g., a smartphone)) and sets the destination of the vehicle C1 to the gas station. In this case, too, it is possible to infer that the user will take an action (refueling) to eliminate the specific event suggested by the agent device 200. Note that the operation of searching for a gas station and the operation of setting a gas station as the destination can be determined based on operation information from the display device 300, operation information from the smartphone, user status information acquired by the user status acquisition unit 122 (e.g., an overall image of the user U1 operating the smartphone), etc.

[0109] FIG. 8(B) shows an example of a case where the user U1 indicates an intention to refuel after performing an action suggesting that the remaining fuel level is below a threshold. In this case, it is assumed that the user U1 understands and performs what the agent device 200 has suggested. For this reason, the agent device 200 performs an action to praise the user U1. For example, as in the example shown in FIG. 2(F), it is possible to perform an action of raising the arm 221 and displaying the eyes and mouth corresponding to a praising expression such as a smile on the face 210. In this case, by displaying image information 225 showing fingers and hands indicating "amazing" on the body 220, it is possible to visually express praise for the user U1.

[0110] Here, when the remaining fuel amount is equal to or less than the threshold, a warning light to that effect is often displayed on the meter panel. However, since meter panels are often relatively small, there is a possibility that the agent device 200 may not understand even if it directs its gaze or points at it. Therefore, as shown in Figs. 7 and 8, a sign image O1 is displayed large on a relatively large display unit 310, and the agent device 200 directs its gaze or points at the sign image O1. When map information for navigation is displayed on the display unit 310, the map information is erased and the sign image O1 is displayed. This makes it possible to create an environment in which the user U1 can easily understand a specific event suggested by the agent device 200.

[0111] [Example of behavior suggesting a specific event] 5 to 8 show examples in which a specific event is suggested by using the line of sight of the agent device 200. However, as described above, a specific event may be suggested by using the action of the arms 221, 222 of the agent device 200 pointing at an object. For example, in FIG. 5(B), instead of using the line of sight of the agent device 200, the arm 221 or 222 of the agent device 200 may point to the buckle 6 to suggest that the seat belt is not fastened. Also, for example, in FIG. 7(B), instead of using the line of sight of the agent device 200, the arm 221 or 222 of the agent device 200 may point to a sign on the display unit 310 to suggest that the remaining fuel level is below a threshold.

[0112] In addition, an agent image corresponding to the agent device 200 may be displayed on the display unit 310, and the agent image displayed on the display unit 310 may be made to perform an action (directing gaze, pointing) suggesting a specific event (e.g., buckle 6, a door that is half-open).

[0113] [Second example of operation to notify a specific event] In Fig. 4, an example is shown in which the second action is an action of notifying the user U1 of the occurrence of a specific event by audio output. However, the occurrence of a specific event may be notified to the user U1 by an action other than audio output. Thus, Figs. 9 and 10 show examples of the second action of notifying the user U1 of the occurrence of a specific event by an action other than audio output.

[0114] [Example of transition when performing the second action using an agent image] Fig. 9 is a diagram showing an example of a transition when, when it is detected that the seat belt is not fastened, a first action is executed to suggest that the user U1 has not fastened the seat belt, and a second action is executed to notify the user U1 of that fact. Fig. 9 shows an example in which, after the first action is executed using only the agent device 200, the second action is executed using the agent device 200 and the agent image AE1.

[0115] Fig. 9(A) is a modified example of Fig. 5(B), and shows an example in which the user U1 does not notice the suggestion by the first action. Since the example is the same as Fig. 5(A) and (B) except that the user U1 does not notice the suggestion by the first action, a description of the parts common to Fig. 5 will be omitted. In Fig. 9, the directions of the lines of sight G31 to G33 of the user U1 and the agent device 200 are shown typically by dotted lines.

[0116] 9(A) shows an example of a case where the agent device 200 executes a first action suggesting that the driver's seat belt is not fastened after it is detected that the driver's seat belt is not fastened. In this example, the agent device 200 directs its gaze G32 to the buckle 6 of the driver's seat belt as the first action suggesting that the driver's seat belt is not fastened.

[0117] As shown in Fig. 9(A), it is also assumed that after the agent device 200 executes the first action, the user U1 continues to talk with the agent device 200 (e.g., voice S11) without understanding the intention of the first action. In such a case, it is also conceivable that the seat belt continues to be detected as not fastened even after a certain time has passed since the first action. Fig. 9(B) shows an example of a case where the seat belt continues to be detected as not fastened even after a certain time has passed since the first action.

[0118] FIG. 9B shows an example of a case where the seat belt is still detected as not fastened even after a certain time has elapsed since the first action was executed. In this case, the agent control unit 127 causes the agent device 200 to execute the second action as an action of notifying the user that the seat belt in the driver's seat is not fastened. As described above, the second action is an action of notifying the user U1, who does not notice the occurrence of the specific event even after the execution of the first action, of the occurrence of the specific event by using a transmission means having a higher transmission power than the first action. Thus, FIG. 9B shows an example of displaying a sign image O2 and an agent image AE1 related to the seat belt on the display unit 310 as the second action of notifying the user that the seat belt is not fastened, and directing the line of sight G33 to the sign image O2 and the agent image AE1. Also, in FIG. 9B, an action of directing the line of sight G33 to the sign image O2 and the agent image AE1 displayed on the display unit 310 is executed as the second action.

[0119] For example, when the agent device 200, which has stopped talking and is looking at the buckle 6, starts to direct its gaze G33 toward the display device 300, the user U1 may utter the voice S23 "Did something happen?" to the agent device 200. Then, when the user U1 looks at the display device 300 at which the gaze G33 of the agent device 200 is directed, he / she notices that the sign image O2 and the agent image AE1 are displayed on the display unit 310 instead of map information, and that the hand and arm of the agent image AE1 are pointing at the sign image O2. He / she also notices that the eyes of the agent image AE1 are popping out and looking at the sign image O2. In this case, the user U1 may notice that the agent device 200 wants to tell him / her something about not wearing a seat belt. The rest of the process is substantially the same as the examples shown in FIG. 6 and FIG. 8, and therefore the description here is omitted.

[0120] It is also possible to have the display unit 310 display a character image other than the agent device 200, and to execute the second action of notifying a specific event using that character image. An example of this is shown in Fig. 10. It is also possible to generate these images using known image generation techniques such as CG (Computer Graphics).

[0121] [Example of transition when suggesting to the user using a character image] Fig. 10 is a diagram showing a display example in which a specific event is notified to user U1 using character images CI1 to CI10 displayed on the display unit 310 of the display device 300. Fig. 10 shows an example in which the second action is to direct the line of sight of the agent device 200 toward the display device 300 and display sign images O11 to O17 and character images CI1 to CI10 on the display unit 310. The manner in which the line of sight of the agent device 200 is directed toward the display device 300 is the same as in Fig. 9, and therefore is not shown here.

[0122] FIG. 10(A) shows a display example when an ABS abnormality is detected as a specific event. In this case, a sign image O11 representing an ABS abnormality is displayed on the display unit 310 as a sign image related to the specific event. In addition, a character image CI1 is displayed on the display unit 310 together with the sign image O11. Then, as an operation to notify the user U1 of the ABS abnormality, the image is changed so that the size of the character image CI1 is enlarged. For example, it is possible to display a character image CI2 in which the size of the character image CI1 is enlarged. For example, in the display unit 310 capable of displaying with a sense of depth, a performance is executed in which the character image CI1, which is normally present at the back in the front-to-back direction, comes to the front only when a specific event is detected. In other words, it is possible to display the character image CI1 in such a manner that the body of the character image CI1 becomes gigantic and large only when the character image CI1 senses an abnormality. As a result, in the display unit 310 in which the sign image O11 representing an ABS abnormality and the small character image CI1 were displayed, the character image CI1 is enlarged to become the character image CI2, and the user U1 who sees the unexpected change is more likely to notice the sign image O11 displayed next to the character image CI2. That is, the behavior of the character image CI1 expanding (changing drastically) can make the user U1 aware of the behavior.

[0123] If the physical size of the agent device 200 can be enlarged or reduced so that it can be seen by the user U1, a specific event may be notified to the user U1 by enlarging the size of the agent device 200, as in the example shown in Fig. 10(A). For example, if the agent device 200 itself is movable on the dashboard 2, it is possible to make the agent device 200 appear to the user U1 as if it has been enlarged in size by moving the agent device 200 itself from the front to the rear of the vehicle C1.

[0124] FIG. 10B shows a display example when the remaining fuel amount is detected as a specific event as being equal to or less than a threshold. In this case, a sign image O12 indicating that the remaining fuel amount is equal to or less than a threshold is displayed on the display unit 310 as a sign image related to the specific event. In addition, a character image CI3 is displayed on the display unit 310 together with the sign image O12. Then, as an operation to notify the user U1 that the remaining fuel amount is equal to or less than the threshold, the image is changed so that the appearance of the character image CI3 changes. For example, it is possible to display a state in which the bird character image CI3 has changed to a bird character image CI4 having a completely different appearance. As a result, the user U1 who sees the unexpected change by changing to a completely different bird character image CI4 in the display unit 310 where the sign image O12 indicating that the remaining fuel amount is equal to or less than a threshold and the docile character image CI3 were displayed is more likely to notice the sign image O12 displayed next to the character image CI4. In other words, it is possible to have the user U1 notice the character image CI3 by the behavior of the character image CI3 changing drastically.

[0125] FIG. 10C shows a display example when the battery remaining amount is detected as a specific event as being equal to or less than a threshold. In this case, a sign image O13 indicating that the battery remaining amount is equal to or less than a threshold is displayed on the display unit 310 as a sign image related to the specific event. Also, a character image CI5 is displayed on the display unit 310 together with the sign image O13. Then, as an action to notify the user U1 that the battery remaining amount is equal to or less than the threshold, the image is changed so that the character image CI5 jumps. For example, it is possible to display a state in which the dog character image CI5 has changed to a character image CI6 that jumps and kicks its legs. That is, it is possible to change to a character image CI6 that jumps violently. As a result, in the display unit 310 in which the sign image O13 indicating that the battery remaining amount is equal to or less than a threshold and the character image CI5 that was walking quietly were displayed, the character image CI5 suddenly jumps and kicks its legs, and the user U1 who sees the unexpected change is more likely to notice the sign image O13 displayed next to the character image CI6. That is, the drastic changes in behavior of the character image CI5 can catch the user U1's attention.

[0126] FIG. 10(D) shows a display example in the case where stopping at a parking prohibited place is detected as a specific event. In this case, a sign image O14 representing the parking prohibited is displayed on the display unit 310 as a sign image related to the specific event. In addition, a character image CI7 is displayed on the display unit 310 together with the sign image O14. Then, as an operation to notify the user U1 that the parking prohibited place is being stopped, the image of the character image CI7 is changed so that it looks surprised when it sees the sign image O14. As a result, the character image CI7 suddenly changes its expression when it sees the sign image O14 representing the parking prohibited place, so that the user U1 who sees the unexpected change is more likely to notice the sign image O14 that the character image CI7 is surprised about. In other words, it is possible to have the user U1 notice the character image CI7 by the behavior of the character image CI7 changing drastically.

[0127] FIG. 10(E) shows a display example in the case where the absence of lights at night is detected as a specific event. In this case, a sign image O15 representing a light is displayed on the display unit 310 as a sign image related to the specific event. In addition, a character image CI8 is displayed on the display unit 310 together with the sign image O15. Then, as an operation to notify the user U1 that the lights are not lit at night, the image of the character image CI8 is changed so that it looks surprised when it sees the sign image O15. As a result, the character image CI8 is surprised when it sees the sign image O15 representing a light and changes its facial expression and whole body, so that the user U1 who sees the unexpected change is more likely to notice the sign image O15 that the character image CI8 is surprised about. In other words, it is possible to have the user U1 notice the character image CI8 by the behavior of the character image CI8 changing drastically.

[0128] FIG. 10(F) shows a display example in the case where the non-illumination of the emergency flasher lamp during a road stop is detected as a specific event. In this case, a sign image O16 representing the emergency flasher lamp is displayed on the display unit 310 as a sign image related to the specific event. In addition, a character image CI9 is ​​displayed on the display unit 310 together with the sign image O16. Then, as an operation to notify the user U1 that the emergency flasher lamp is not lit during a road stop, the image of the character image CI9 is ​​changed so that it looks surprised when it sees the sign image O16. As a result, the character image CI9 is ​​surprised when it sees the sign image O16 representing the emergency flasher lamp and changes its facial expression and whole body, so that the user U1 who sees the unexpected change is more likely to notice the sign image O16 that the character image CI9 is ​​surprised about. In other words, the behavior of the character image CI9 changing drastically makes it possible to make the user U1 notice the sign image.

[0129] FIG. 10(G) shows a display example when a door ajar is detected as a specific event. In this case, a sign image O17 representing the door ajar is displayed on the display unit 310 as a sign image related to the specific event. In addition, a character image CI10 is displayed on the display unit 310 together with the sign image O17. Then, as an action to notify the user U1 that the door is ajar, the image of the character image CI10 is changed so that it looks impressed when it sees the sign image O17. Alternatively, it is also possible to change the image of the character image CI10 so that it looks at the sign image O17 and nods vigorously as if to encourage or has doubts. In this way, the character image CI10 looks at the sign image O17 representing the door ajar and changes its expression with admiration, so that the user U1 who sees the unexpected change is more likely to notice the sign image O17 that the character image CI10 is impressed with. In other words, the behavior of the character image CI10 changing vigorously makes it possible to make the user U1 notice the sign image O17.

[0130] As shown in Fig. 10(D) to (F), the character images CI7 to CI9 express surprised expressions, which can increase the possibility that the user U1 will notice the expressions. Also, as shown in Fig. 10(G), the character image CI10 expresses an expression of admiration, vigorously nodding as if to encourage, or a questioning expression, which can increase the possibility that the user U1 will notice the expressions.

[0131] For example, if operations such as those shown in Figures 9 and 10 are performed from the first stage, when the user U1 notices a specific event, he or she may feel distrustful and wonder whether the agent device 200 really understands him or her. Therefore, for example, in the first stage, the agent device 200 may turn toward an object related to the specific event or point in that direction to suggest the occurrence of the specific event to the user U1. Then, in the second stage, operations such as those shown in Figures 9 and 10 are performed to notify the user U1 of the occurrence of the specific event. This makes it possible to prevent the user U1 from feeling distrustful and to inform the user U1 of the specific event.

[0132] Note that the example shown in Fig. 10 is just one example, and after the first action is completed, the character image may execute another action that indicates that it senses something abnormal. For example, the second action may be an action in which the character image takes out a magnifying glass and uses the magnifying glass to stare intently at a sign image (an image related to a specific event). Also, the second action may be an action in which the character image moves from a position where the character image is normally present to a position close to the sign image (an image related to a specific event) after the occurrence of a specific event.

[0133] Also, a second operation of notifying a specific event may be executed using other character images or display modes. For example, when a seat belt is not fastened as a specific event, the line of sight of the character image displayed on the display unit 310 may be directed toward the target seat belt or buckle, or the hand and arm of the character image may be used to point to the target seat belt or buckle. Also, for example, when a door is ajar as a specific event, the line of sight of the character image displayed on the display unit 310 may be directed toward the target door, or the hand and arm of the character image may be used to point to the target door.

[0134] 10 shows an example in which the second action is an action of directing the line of sight of the agent device 200 toward the display device 300 and displaying the marker images O11 to O17 and the character images CI1 to CI10 on the display unit 310. However, instead of the action of directing the line of sight of the agent device 200 toward the display device 300, an action of pointing to the display device 300 using the arms 221, 222 of the agent device 200 may be executed. Also, the action of directing the line of sight of the agent device 200 toward the display device 300 may be omitted, and only the action of displaying the marker images O11 to O17 and the character images CI1 to CI10 on the display unit 310 may be the second action.

[0135] 9 and 10, an example is shown in which the second action is an action of directing the line of sight of the agent device 200 toward the display device 300 and displaying a sign image and an agent image (or a character image) on the display unit 310. However, some or all of these actions may be executed as the first action. In this case, a notification action using a transmission means having a stronger transmission power than the first action is further executed as the second action.

[0136] [Example of information processing system operation] Fig. 11 is a flowchart showing an example of a specific event notification process in the information processing system 100. This specific event notification process is executed by the control unit 120 (see Fig. 3) based on a program stored in the storage unit 130 (see Fig. 3). This specific event notification process is constantly executed for each control period. This specific event notification process will be described with appropriate reference to Figs. 1 to 10.

[0137] In step S501, the detection unit 126 acquires vehicle state information about the vehicle C1 from the vehicle state acquisition unit 125, and acquires user state information about the user U1 from the utterance acquisition unit 121, the user state acquisition unit 122, the vehicle interior state acquisition unit 123, and the vehicle exterior state acquisition unit 124. In addition, the detection unit 126 acquires each piece of information that can be acquired from an external device from the communication unit 140 as necessary.

[0138] In step S502, the detection unit 126 executes a detection process to detect a specific event to be notified to the user U1 based on the vehicle state information acquired in step S501. The detection method for detecting the specific event is the same as the detection method described above.

[0139] In step S503, the detection unit 126 determines whether or not a specific event was detected in step S502. If a specific event was detected, the process proceeds to step S504. On the other hand, if a specific event was not detected, the operation of the specific event notification process ends.

[0140] In step S504, the agent control unit 127 controls the agent device 200 to execute a first action suggesting the specific event detected in step S502. Specifically, the agent control unit 127 extracts a first action corresponding to the specific event detected in step S502 from the specific event response information DB 400 (see FIG. 4). The agent control unit 127 also causes the agent device 200 to execute the extracted first action based on the position related to the specific event detected in step S502 (the position where the specific event occurred). For example, as shown in FIG. 5(B), FIG. 5(B) and FIG. 9(A), the agent control unit 127 causes the agent device 200 to execute the first action.

[0141] In step S505, the detection unit 126 determines whether the specific event detected in step S502 has disappeared within a certain time after the first operation is executed. Here, the certain time generally means the time when the specific event is assumed to disappear. This certain time may be a fixed value or a variable value according to the specific event. For example, when a seat belt not fastened, a door ajar, or the like is detected as a specific event, a value of about several tens of seconds can be set as the certain time. Also, for example, when a fuel remaining amount or a battery remaining amount below a threshold is detected as a specific event, a value of about several tens of minutes to several hours can be set as the certain time.

[0142] In step S505, instead of determining whether the specific event has disappeared within a certain time, it may be determined whether the disappearance of the specific event has been estimated within a certain time, or whether the occurrence of the specific event has been understood by the user within a certain time. This estimation method is the same as the estimation method described above. In addition, it may be set for each type of specific event whether to execute the determination of disappearance, estimation, or understanding.

[0143] In step S506, the agent control unit 127 controls the agent device 200 to execute a third action of praising the user U1. For example, as shown in Fig. 6(B) and Fig. 8(B), the agent control unit 127 controls the agent device 200 to execute an action of praising the user U1 as the third action.

[0144] In step S507, the agent control unit 127 controls the agent device 200 to execute a second action for notifying the specific event detected in step S502. Specifically, the agent control unit 127 extracts a second action corresponding to the specific event detected in step S502 from the specific event response information DB 400 (see FIG. 4). The agent control unit 127 also causes the agent device 200 to execute the extracted second action based on the position related to the specific event detected in step S502 (the position where the specific event occurred). For example, the second action can be executed by audio output. Also, for example, the second action can be executed as shown in FIG. 9(B) and FIG. 10.

[0145] In step S508, the detection unit 126 judges whether the specific event detected in step S502 has disappeared within a certain time from the execution of the second action. The certain time used in step S505 and the certain time used in step S508 may be the same value or different values. These certain times may be fixed values ​​or may be variable values ​​according to the specific event. They may also be set appropriately according to the specific event. Instead of judging whether the specific event has disappeared within a certain time from the execution of the second action, it may be judged whether the disappearance has been estimated or understood within a certain time from the execution of the second action.

[0146] In step S509, the agent control unit 127 controls the agent device 200 to execute a fourth action praising the user U1. For example, as shown in Fig. 6(B) and Fig. 8(B), the agent device 200 is caused to execute an action praising the user U1 as the fourth action. Note that the fourth action is executed when the user U1 understands the specific event after executing the first and second actions, and therefore can be a more modest action than the third action. This can motivate the user U1 to try to understand the specific event after executing the first action.

[0147] In step S510, the control unit 120 executes a notification process according to the specific event detected in step S502. This notification process is a process that is preset in the vehicle C1. For example, the notification process may be performed by outputting a predetermined warning sound and displaying a warning image on the meter panel or the display device 300. For example, if the seat belt is still not fastened even after a certain time has elapsed since the second action was executed, the notification process can be executed by continuously outputting a warning sound "Please fasten your seat belt" that warns of the seat belt not being fastened and displaying an icon representing the seat belt not being fastened on the meter panel (or the display device 300). Note that the example shown in FIG. 11 shows an example in which the notification process (step S510) is executed after the suggestion process by the first action (step S504) and the notification process by the second action (step S507). However, at least a part of the notification process (step S510) (for example, displaying an icon related to the specific event on the meter panel) may be executed simultaneously with the suggestion process by the first action (step S504) and the notification process by the second action (step S507).

[0148] In the above, an example has been shown in which the control process for executing the first action and the second action is executed using the preset specific event response information DB400. However, the control process may be executed by other methods. For example, it is possible to execute the control process using artificial intelligence (AI). For example, it is possible to learn in advance the specific event, the first action or the second action executed in response to the specific event, and the response of the user U1 to the first action or the second action, and use this learning data in the control process.

[0149] [Examples of Effects of This Embodiment] Here, if an alarm is issued regarding the occurrence of a specific event with a relatively large movement and a relatively loud sound in the passenger compartment of the vehicle C1, which is relatively quiet, the user U1 may be startled by the alarm. In this case, the user U1 may feel uncomfortable, and it may become difficult to build a friendly and cooperative relationship between the user U1 and the agent device 200, resulting in a decrease in intimacy.

[0150] Therefore, in this embodiment, before outputting an alarm (e.g., an alarm sound) regarding a specific event to the user U1, it is possible to suggest (e.g., a casual behavior, a casual gesture) an object related to the specific event by using at least a part of the agent device 200 (gaze, pointing) without outputting a sound from the agent device 200. In this way, by performing a first action to suggest to the user U1 before issuing an alarm, it is possible to make the user U1 aware of the occurrence of the specific event. Therefore, it is possible to reduce the discomfort felt by the user U1. This makes it possible to build a friendly and cooperative relationship between the user U1 and the agent device 200, and to improve intimacy.

[0151] In addition, when many specific events occur, if many notification sounds and warning sounds related to the specific events are output, it is expected that the user U1 will find those notification sounds and warning sounds annoying and uncomfortable. Also, if those notification sounds and warning sounds are output from the agent device 200, it is expected that the user U1 will find the agent device 200 annoying and uncomfortable, and will not want to be monitored by the agent device 200. This makes it difficult to build a friendly and cooperative relationship between the user U1 and the agent device 200, and there is a risk of intimacy decreasing.

[0152] Therefore, in this embodiment, when a specific event is detected, the information to be transmitted to the user U1 (information about the specific event) is transmitted in the first stage by suggesting an object related to the specific event, without audio output from the agent device 200. This makes it possible to transmit information appropriately without causing discomfort to the user U1.

[0153] [Example of executing processing on other devices or systems] In the above, examples have been shown in which the detection process, control process, etc. are executed in the agent device 200 and the information processing device 110 (or the information processing system 100), but all or part of each process may be executed in other devices. In this case, the information processing system is configured by each device that executes part of each process. For example, at least part of each process can be executed using various information processing devices and various electronic devices, such as in-vehicle devices, devices that can be used by users (e.g., smartphones, tablet terminals, personal computers, car navigation devices, IVI), and servers that can be connected via a predetermined network such as the Internet.

[0154] In addition, a part (or the whole) of the information processing system capable of executing the functions of the information processing device 110 (or the information processing system 100) may be provided by an application that can be provided via a predetermined network such as the Internet. This application is, for example, SaaS (Software as a Service).

[0155] [Configuration Example of This Embodiment and Its Effects] The information processing method according to the present embodiment is an information processing method for controlling an agent device 200 capable of communicating with a user U1 who is in a vehicle C1. The information processing method includes an acquisition process (step S501) for acquiring at least one of user status information related to the status of the user U1 and vehicle status information related to the status of the vehicle C1, a detection process (step S502) for detecting a specific event to be notified to the user U1 based on at least one of the acquired user status information and vehicle status information, and a control process (steps S503 to S507) for causing the agent device 200 to execute a first operation (including not outputting voice) suggesting the specific event when the specific event is detected, and causing the agent device 200 to execute a second operation (an operation in a manner different from the first operation) for notifying the specific event when the specific event continues to be detected after a predetermined time has elapsed since the execution of the first operation. The first operation means that the specific event is not directly conveyed by voice output or the like, but is indirectly conveyed by a pointing operation, line of sight, or the like. The program according to the present embodiment is a program for causing a computer to execute each of these processes. In other words, the program according to this embodiment is a program that causes a computer to realize each function that can be executed by the information processing device 110.

[0156] According to this configuration, it is possible to convey information about the detected specific event by suggesting an object related to the specific event, without outputting a voice from the agent device 200. This makes it possible to convey information appropriately without making the user U1 feel uncomfortable. Furthermore, if the specific event continues to be detected after a predetermined time has elapsed since the execution of the first action, it is possible to more reliably notify the user U1 of the specific event, who does not notice the occurrence of the specific event by the first action, by having the agent device 200 execute a new second action.

[0157] In the information processing method according to the present embodiment, the agent device 200 is a robot that includes at least one of a member that represents an eye and a member that represents at least one of a hand and an arm. In the control process (step S504), the first action is performed to direct the gaze of the eye to an object related to a specific event, or to point to the object using at least one of the hand and the arm. For example, as shown in FIG. 2(B), FIG. 5(B), and FIG. 7(B), the first action can be performed to direct the gaze of the eye of the face unit 210 to the object. For example, as shown in FIG. 2(C), the first action can be performed to point to the object using the arm unit 222.

[0158] According to this configuration, it is possible to suggest (for example, a casual behavior or a casual gesture) an object related to a specific event using at least a part (gaze, pointing) of the agent device 200. This makes it possible to reduce the discomfort felt by the user U1, and to build a friendly cooperative relationship between the user U1 and the agent device 200, thereby improving intimacy.

[0159] In the information processing method according to the present embodiment, the agent device 200 is a robot having at least one of a member imitating an eye and a member imitating at least one of a hand and an arm. In the control process (step S504), a sign image related to a specific event is displayed on the display unit 310 installed in the vehicle C1, and an action of directing the line of sight of the eye to the sign image or an action of pointing to the sign image using at least one of the hand and the arm is executed as the first action. For example, as shown in FIG. 7(B), it is possible to display a sign image O1 on the display unit 310 and execute an action of directing the line of sight of the agent device 200 to the sign image O1 as the first action. Also, an action of pointing to the sign image O1 using at least one of the hand and the arm of the agent device 200 may be executed as the first action.

[0160] According to this configuration, it is possible to suggest (for example, a casual behavior or a casual gesture) a sign image related to a specific event by using at least a part (gaze, pointing) of the agent device 200. This makes it possible to reduce the discomfort felt by the user U1, and to build a friendly cooperative relationship between the user U1 and the agent device 200, thereby improving intimacy.

[0161] In the information processing method according to the present embodiment, instead of the agent device 200, or together with the agent device 200, it is possible to use a display device 300 that displays a character image having at least one of eyes, hands, and arms. In this case, in the control process (step S504), a sign image related to a specific event is displayed on the display device 300, and the character image is made to express a state in which the line of sight of the eyes is directed to the sign image, or a state in which the line of sight of the hands and arms is used to point to the sign image, thereby executing the first action. For example, as shown in FIG. 9(B), it is possible to display a sign image O2 on the display device 300, and to execute an action of directing the line of sight of the agent image AE1 to the sign image O2 as the first action. Also, an action of pointing to the sign image O1 using at least one of the hands and arms of the agent image AE1 may be executed as the first action. It is also possible to have the eyes of character images CI7 to CI10 directed at marker images O14 to O17, as shown in Figures 10(D) to (G), or to have at least one of the hands and arms of character image CI8 point to marker image O15, as shown in Figure 10(E).

[0162] According to this configuration, it is possible to suggest a sign image related to a specific event (for example, a casual behavior or a casual gesture) by using at least a part (gaze, pointing) of the agent image AE1 or the character images CI1 to CI10. This makes it possible to reduce the discomfort felt by the user U1, build a friendly cooperative relationship between the user U1 and the agent device 200, and improve intimacy.

[0163] In the information processing method according to the present embodiment, in the control process (step S507), an action of notifying a specific event using a transmission means having a higher transmission power than the first action is executed as the second action. For example, as shown in FIG. 9(B), it is possible to execute an action of displaying a sign image O2 on the display device 300, directing the line of sight of the agent device 200 to the sign image O2, and directing the line of sight of the agent image AE1 to the sign image O2 as the second action. Also, it is possible to execute an action of pointing to the sign image O1 using at least one of the hand and arm of the agent image AE1 as the second action. Also, as shown in FIG. 10(A) to (G), it is possible to execute an action of notifying a specific event using the sign images O11 to O17 and the character images CI1 to CI10 as the second action.

[0164] According to this configuration, by executing an action of notifying a specific event as the second action using a transmission means having a higher transmission force than the first action, it is possible to more reliably notify user U1, who does not notice the occurrence of a specific event by the first action, of the specific event.

[0165] In the information processing method according to the present embodiment, in the control process (step S507), a voice output operation for outputting voice information for notifying a specific event is executed as the second operation.

[0166] According to this configuration, by executing an action of notifying a specific event by outputting voice information having a stronger communicative power than the first action as the second action, it is possible to more reliably notify a user U1 who does not notice the occurrence of a specific event by the first action of the specific event.

[0167] In the information processing method according to this embodiment, in the control process (step S507), the second action is executed by making the character image express a state in which a transmission means having a stronger transmission power than the first action is used. For example, as shown in Fig. 9(B) and Fig. 10(A) to (G), it is possible to execute the second action by making the agent image AE1, character images CI1 to CI10, etc. express a predetermined state.

[0168] According to this configuration, by executing a second action that causes the character image to express a manner in which the transmission means is used more than the first action, it is possible to more reliably notify user U1, who does not notice the occurrence of a specific event through the first action, of the specific event.

[0169] In the information processing method according to this embodiment, the control process (steps S506, S509) causes the agent device 200 to execute a third action (or a fourth action) representing the disappearance of the specific event when the specific event is no longer detected after the execution of the first action or the second action. For example, as shown in Fig. 6(B) and Fig. 8(B), the agent device 200 can execute an action of praising the user U1 as the third action.

[0170] According to this configuration, the user U1 notices a specific event after performing the first or second action, and then performs a third action (or a fourth action) to praise the user U1 for taking an action to eliminate the specific event, which motivates the user U1 to quickly notice the suggestions of the agent device 200. This makes it possible to build a friendly and cooperative relationship between the user U1 and the agent device 200, and to improve intimacy. It also makes it possible to quickly resolve the specific event that occurs next time onwards.

[0171] In the information processing method according to this embodiment, the control process (steps S506, S509) executes, as a third action (or a fourth action), a confirmation action of directing the gaze of the eyes of the agent device 200 to an object related to a specific event, or a confirmation action of pointing to the object using at least one of the hand and arm of the agent device 200.

[0172] With this configuration, the user U1 who notices a specific event after performing the first or second action and takes action to eliminate the specific event can easily recognize that the agent device 200 has also noticed the event's disappearance. This makes it possible to build a friendly cooperative relationship between the user U1 and the agent device 200, thereby improving intimacy between them.

[0173] In the information processing method according to this embodiment, the control process (steps S506, S509) executes the third action (or fourth action) by causing the agent device 200 to execute a joy performance that expresses joy at the disappearance of the specific event. For example, as shown in Fig. 6(B) and Fig. 8(B), the agent device 200 can execute an action of praising the user U1 as the third action.

[0174] According to this configuration, by executing a joy effect that expresses joy after the execution of the first or second action, the user U1 is motivated to quickly notice the suggestion of the agent device 200. This makes it possible to build a friendly and cooperative relationship between the user U1 and the agent device 200, and to improve intimacy. In addition, it becomes possible to quickly resolve a specific event that occurs from the next time onwards.

[0175] In the information processing method of this embodiment, the detection process (step S502) detects that a specific event has occurred when it is detected that user U1 has deviated from a predetermined rule based on at least one of the user state information and the vehicle state information, or when it is detected that a specified abnormality has occurred in vehicle C1 based on the vehicle state information.

[0176] According to this configuration, when the user U1 deviates from a predetermined rule, when a specified abnormality occurs in the vehicle C1, etc., it is possible to appropriately detect the occurrence of a specific event.

[0177] The information processing device 110 is an information processing device that controls an agent device 200 capable of communicating with a user U1 who is in a vehicle C1. The information processing device 110 includes a sound acquisition unit 101, a user image acquisition unit 102, a vehicle interior image acquisition unit 103 (an example of an acquisition unit) that acquire user status information related to the status of the user U1, a sound acquisition unit 101, a vehicle interior image acquisition unit 103, a vehicle exterior image acquisition unit 104, and a vehicle status acquisition unit 125 (an example of an acquisition unit) that acquire vehicle status information related to the status of the vehicle, a detection unit 126 that detects a specific event to be notified to the user U1 based on at least one of the acquired user status information and vehicle status information, and an agent control unit 127 (an example of a control unit) that causes the agent device 200 to execute a first operation suggesting the specific event when the specific event is detected, and causes the agent device 200 to execute a second operation (an operation in a manner different from the first operation) for notifying the specific event when the specific event continues to be detected after a predetermined time has elapsed since the execution of the first operation. The information processing device 110 may be a device built into the agent device 200, or may be a device different from the agent device 200. Moreover, instead of the information processing device 110, an information processing system configured with a plurality of devices capable of executing each process realized by the information processing device 110 may be used.

[0178] According to this configuration, it is possible to convey information about the detected specific event by suggesting an object related to the specific event, without outputting a voice from the agent device 200. This makes it possible to convey information appropriately without making the user U1 feel uncomfortable. Furthermore, if the specific event continues to be detected after a predetermined time has elapsed since the execution of the first action, it is possible to more reliably notify the user U1 of the specific event, who does not notice the occurrence of the specific event by the first action, by having the agent device 200 execute a new second action.

[0179] Note that each processing procedure shown in this embodiment is an example for realizing this embodiment, and the order of some of the processing procedures may be changed within the scope that allows the realization of this embodiment, and some of the processing procedures may be omitted or other processing procedures may be added.

[0180] Each process of this embodiment is executed based on a program for causing a computer to execute various processing procedures. This embodiment can also be understood as an embodiment of a program for implementing the function of executing each process, and a recording medium for storing the program. For example, the program can be stored in the storage device of the information processing device by an update process for adding a new function to the information processing device. This makes it possible to cause the updated information processing device to execute each process shown in this embodiment.

[0181] Although the embodiments of the present invention have been described above, the above-mentioned embodiments merely show application examples of the present invention, and it is not intended that the technical scope of the present invention be limited to the specific configurations of the above-mentioned embodiments. [Explanation of symbols]

[0182] 20 network, 100 information processing system, 101 sound acquisition unit, 102 user image acquisition unit, 103 vehicle interior image acquisition unit, 104 vehicle exterior image acquisition unit, 110 information processing device, 120 control unit, 121 speech acquisition unit, 122 user status acquisition unit, 123 vehicle interior status acquisition unit, 124 vehicle exterior status acquisition unit, 125 vehicle status acquisition unit, 126 detection unit, 127 agent control unit, 130 storage unit, 131 agent information DB, 140 communication unit, 200 agent device, 230 display unit, 240 sound output unit, 250 drive unit, 300 display device, 310 display unit, 400 specific event response information DB

Claims

1. 1. An information processing method for controlling an agent device capable of communicating with a user riding in a vehicle, comprising: An acquisition process for acquiring at least one of user state information relating to a state of the user and vehicle state information relating to a state of the vehicle; A detection process for detecting a specific event to be notified to the user based on at least one of the acquired user state information and the acquired vehicle state information; a control process for causing the agent device to execute a first operation suggesting the specific event when the specific event is detected, and for causing the agent device to execute a second operation, different from the first operation, for notifying the specific event when the specific event continues to be detected after a predetermined time has elapsed since the execution of the first operation. Information processing methods.

2. 2. The information processing method according to claim 1, the agent device is a robot having at least one of a member simulating an eye and a member simulating at least one of a hand and an arm, In the control process, an action of directing the gaze of the eye to an object related to the specific event or an action of pointing to the object using at least one of the hand and the arm is executed as the first action. Information processing methods.

3. 2. The information processing method according to claim 1, the agent device is a robot having at least one of a member simulating an eye and a member simulating at least one of a hand and an arm, In the control process, a sign image related to the specific event is displayed on a display unit installed in the vehicle, and an action of directing the line of sight of the eyes to the sign image or an action of pointing to the sign image using at least one of the hand and the arm is executed as the first action. Information processing methods.

4. 2. The information processing method according to claim 1, the agent device is a display device that displays a character image having at least one of an eye, a hand, and an arm; In the control process, a sign image related to the specific event is displayed on the display device, and the character image is caused to express a state in which the line of sight of the eyes is directed to the sign image, or a state in which the hand and / or the arm is used to point to the sign image, thereby executing the first action. Information processing methods.

5. 4. The information processing method according to claim 2, further comprising: In the control process, an operation of notifying the specific event is executed as the second operation by using a transmission means having a transmission force higher than that of the first operation. Information processing methods.

6. 6. The information processing method according to claim 5, In the control process, a voice output operation for outputting voice information for notifying the specific event is executed as the second operation. Information processing methods.

7. 5. The information processing method according to claim 4, and executing the second action by causing the character image to express an action using a transmission means having a transmission force higher than that of the first action in the control process. Information processing methods.

8. 4. An information processing method according to claim 1, further comprising: and when the specific event is no longer detected after the execution of the first action or the second action, the control process causes the agent device to execute a third action representing the disappearance of the specific event. Information processing methods.

9. 9. The information processing method according to claim 8, In the control process, a confirmation action of directing the gaze of the eye to an object related to the specific event or a confirmation action of pointing to the object using at least one of the hand and the arm is executed as the third action. Information processing methods.

10. 9. The information processing method according to claim 8, and executing the third action by causing the agent device to execute a joyful performance that expresses joy at the disappearance of the specific event in the control process. Information processing methods.

11. 5. An information processing method according to claim 1, further comprising: In the detection process, when it is detected that the user deviates from a predetermined rule based on at least one of the user state information and the vehicle state information, or when it is detected that a predetermined abnormality has occurred in the vehicle based on the vehicle state information, the occurrence of the specific event is detected. Information processing methods.

12. An information processing device that controls an agent device capable of communicating with a user riding in a vehicle, an acquisition unit that acquires at least one of user state information related to a state of the user and vehicle state information related to a state of the vehicle; a detection unit that detects a specific event to be notified to the user based on at least one of the acquired user state information and the acquired vehicle state information; a control unit which, when the specific event is detected, causes the agent device to execute a first operation suggesting the specific event, and, when the specific event continues to be detected after a predetermined time has elapsed since the execution of the first operation, causes the agent device to execute a second operation, different from the first operation, for notifying the specific event, Information processing device.

Images