Information processing method and information processor

Abstract

To enable an agent to make a response in response to operation of a user during maintenance for a vehicle, thereby enhance attachment to the vehicle of the user, and raise a motivation of the user who tries to actively execute the maintenance.SOLUTION: An information processing method controls a performance using an agent image 200 capable of communicating with a user U1. The information processing method includes: operation information acquisition processing (step S501) of acquiring operation information related to operation of the user U1; estimation processing (step S502) of estimating the type of maintenance performed by operation of the user U1, on the basis of the operation information; and control processing (step S503) of executing a specific performance in response to maintenance operation of the user U1 during execution of the maintenance, on the basis of the operation information acquired during execution of the maintenance after the estimation, and the estimated maintenance type.SELECTED DRAWING: Figure 15

Description

[Technical Field]

[0001] The present invention relates to an information processing method and an information processing device for controlling an agent capable of communicating with a user. [Background technology]

[0002] Conventionally, techniques have been proposed for conveying various types of vehicle-related information to users. For example, a technique has been proposed in which a character is displayed as an agent image, and changes in the character's eyes indicate the operation of the vehicle, and changes in the character's facial color indicate the state of a target part requiring maintenance (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2021-017126 Summary of the Invention [Problem to be solved by the invention]

[0004] The above-described conventional technology allows the character's facial color to change in response to changes in the condition of a target part (e.g., tire air pressure) that requires periodic maintenance. In other words, when a user performs maintenance, the user can visually recognize the change in the character's facial color in response to the maintenance. However, with this conventional technology, the character does not respond in response to the user's actions while the maintenance is being performed, making it difficult to raise the user's awareness of maintenance. Furthermore, if the character does not respond in response to the user's actions while the maintenance is being performed, the user's attachment to the character may decrease, and this may also lead to a decrease in the user's attachment to the vehicle.

[0005] The present invention aims to increase the user's attachment to the vehicle and motivate the user to actively perform maintenance by having an agent respond according to the user's actions while maintenance is being performed on the vehicle. [Means for solving the problem]

[0006] One aspect of the present invention is an information processing method for controlling presentations of an agent capable of communicating with a user in a vehicle. This information processing method includes a process for acquiring operation information related to the user's operation, a process for estimating a type of vehicle maintenance to be performed by the user's operation based on the operation information, and a control process for executing a specific presentation corresponding to the user's maintenance operation during the maintenance, based on the operation information acquired during the maintenance after estimation in the estimation process and the estimated type of maintenance. [Effects of the Invention]

[0007] According to the present invention, by having an agent respond according to the user's actions while maintenance is being performed on a vehicle, it is possible to increase the user's attachment to the vehicle and increase the user's motivation to actively perform maintenance. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram showing an example of the configuration of the interior of a vehicle. [Figure 2] FIG. 2 is a diagram showing an example of the configuration inside the engine room. [Figure 3] FIG. 3 is a block diagram illustrating an example of the system configuration of the information processing system. [Figure 4] FIG. 4 is a diagram showing each piece of information stored in the maintenance information DB. [Figure 5] FIG. 5 is a diagram showing each piece of information stored in the appropriate action information DB. [Figure 6] FIG. 6 is a diagram showing an example of a display mode of an agent image. [Figure 7] FIG. 7 is a diagram showing a case where a vacuum cleaner is used to collect garbage. [Figure 8] FIG. 8 is a diagram showing a case where the interior of a vehicle is being disinfected. [Figure 9] FIG. 9 is a diagram showing a case where a car is being washed using a car wash machine. [Figure 10] FIG. 10 shows a case where a car is washed using a cloth. [Figure 11] FIG. 11 is a diagram showing a case where a car is washed using a car wash machine. [Figure 12] FIG. 12 is a diagram showing a case where a car is washed using a car wash machine. [Figure 13] FIG. 13 is a diagram showing a case where fuel is supplied using a fuel dispenser. [Figure 14] FIG. 14 is a diagram showing an example of outputting maintenance information. [Figure 15] FIG. 15 is a flowchart showing an example of the specific performance control process. [Figure 16] FIG. 16 is a flowchart showing an example of the specific performance control process. [Figure 17] FIG. 17 is a flowchart showing an example of the specific performance control process. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, an embodiment of the present invention will be described with reference to the accompanying drawings.

[0010] [Agent information output example] Fig. 1 is a simplified diagram showing an example of the configuration of the interior of a vehicle C1. Fig. 1 shows the interior of the vehicle C1 in front of the driver's seat and passenger seat (not shown) as viewed from the rear in the longitudinal direction of the vehicle C1. To facilitate explanation, Fig. 1 omits the illustration of components other than a dashboard 2, a steering wheel 3, a windshield 4, a rearview mirror 5, window glasses W1 and W2, and an output device 300 that displays an agent image 200.

[0011] The output device 300 is a display device installed inside the vehicle C1, and performs various output operations based on instructions from the information processing device 110 (see FIG. 3). The output device 300 includes a display unit 310 and a sound output unit 320 (see FIG. 3). The display unit 310 and the sound output unit 320 are controlled by a control unit (not shown) included in the output device 300. For example, the output device 300 is an in-vehicle system including one or more devices capable of providing various types of information. For example, the output device 300 may be at least one of a navigation device, an audio device, a DVD device, a TV tuner, an IVI (In-Vehicle Infotainment), etc. It is also possible to use a HUD (Head Up Display) implemented on the windshield 4 instead of the display unit 310 of the output device 300. Alternatively, the HUD display may be performed using the windshield 4, the rear windshield, other window glasses W1, W2, etc. as a display medium. Also, for example, by applying technology for incorporating a transparent display into window glass, it is possible to use the window glass (front window 4, rear window, other window glass W1, W2, etc.) installed in the vehicle C1 as a display medium (see Figures 9 and 10). Also, it is possible to use other parts of the exterior of the vehicle C1 other than glass (for example, the body) as a display medium. Also, other devices capable of outputting other information, such as display devices and audio output devices, may be used. Also, portable devices that can be carried by the user U1, such as information processing devices such as smartphones, tablet terminals, and portable personal computers, may be used.

[0012] Furthermore, the output device 300 executes an output operation that enables communication with a user (including user U1). For example, an agent image 200 that enables communication with a user is displayed on the display unit 310 of the output device 300, and various effects are executed using the agent image 200. In other words, the output device 300 functions as an agent device that enables communication with a user. Note that hereinafter, the term "agent information" refers to information about an agent that is expressed using the agent image 200, audio output related to the agent, and the like. Furthermore, the effects of the agent that are expressed using the agent image 200, audio output related to the agent, and the like are realized by the behavior of the agent image 200, the voice of the agent, and the like.

[0013] The agent image 200 is an image representing an object capable of communicating with the user. In this embodiment, an example is shown in which an image imitating a human or an animal is used as the agent image 200. However, the agent image 200 is not limited to this, and for example, an image imitating various animals such as a rabbit or a pig, or an image imitating a virtual living thing (for example, the face of an anime character) may be used as the agent image 200. Note that an image representing the whole body as an image representing a living thing may be used as the agent image 200, or an image representing a part of the whole body (for example, the face). In this way, a lifelike image can be used as the agent image 200.

[0014] FIG. 1 also shows an example in which the agent image 200 is displayed on the output device 300 in a manner that looks toward the inside of the vehicle. In this embodiment, an example is shown in which the agent image 200 is displayed as needed when the output device 300 is in an on state. Note that the agent image 200 can be turned on in response to a user operation related to the agent image 200 or the surrounding environment. For example, a system that outputs agent information (e.g., the agent image 200) may be configured as a system different from the system of the vehicle C1. In this case, it is possible to output the agent information even when the vehicle C1 is turned off (when the ignition key is turned off).

[0015] The output device 300 performs various operations related to driving assistance (e.g., operations related to the agent image 200) based on instructions from the information processing device 110 (see FIG. 3 ). For example, the output device 300 outputs various information related to driving assistance when the user U1 performs driving operations and surrounding facilities, etc., based on the control of the information processing device 110. This driving assistance may include, for example, a notification of a moving object ahead or behind. For example, a notification of a moving object ahead may be made by outputting a voice message such as "Be careful, there is a railroad crossing ahead" or "There is a person ahead." At the time of each of these outputs, various operations can be performed using the agent image 200. That is, driving assistance can be performed using the agent image 200. Furthermore, for example, the output device 300 performs various processes using the agent image 200 based on the control of the information processing device 110. For example, the output device 300 performs a communication process for performing various exchanges with the user U1 using the agent image 200. For example, the output device 300 uses the agent image 200 to exchange various conversations with the user U1 and provide information that the user U1 likes.

[0016] [Installation example of engine room image acquisition unit] FIG. 2 is a simplified diagram showing an example of the configuration inside the engine room 7 of the vehicle C1. That is, FIG. 2 shows an example of the configuration inside the engine room 7 with the hood 6 open. If the vehicle C1 is an internal combustion engine vehicle, various devices including an internal combustion engine are installed in the engine room 7. If the vehicle C1 is an electric vehicle, the engine room 7 is also called a motor room, and various devices including a motor are installed inside this room. If the vehicle C1 is a hybrid vehicle, various devices including an internal combustion engine and a motor are installed in the engine room 7.

[0017] 2 shows an example in which the engine room image acquisition unit 103 is provided on the rear surface of the hood 6. When the hood 6 is closed, the rear surface of the hood 6 faces the engine room 7 and cannot be seen from outside the vehicle C1.

[0018] 2 is an example and is not limited to this. For example, the installation location of the engine room image acquisition unit 103 is not limited to the underside of the hood 6, and the engine room image acquisition unit 103 can be installed at another position as long as the image capture range includes the inside of the engine room 7 and the periphery of the engine room 7 (the periphery of the vehicle C1). Furthermore, multiple engine room image acquisition units 103 may be provided.

[0019] [Example of information processing system configuration] FIG. 3 is a block diagram showing an example of the system configuration of the information processing system 100 installed in the vehicle C1.

[0020] The information processing system 100 includes a vehicle interior image acquisition unit 101, a vehicle exterior image acquisition unit 102, an engine room image acquisition unit 103, sensors 104, a sound acquisition unit 105, an information processing device 110, and an output device 300. The information processing device 110 is an example of a device that controls an agent device (the output device 300 that outputs agent information) that can communicate with a user U1.

[0021] The information processing device 110 and the output device 300 are connected by a communication method using wired communication or wireless communication. The information processing device 110 is connected to the network 20 by a communication method using wireless communication. The network 20 is a network such as a public line network or the Internet. The output device 300 may also be connected to the network 20 by a communication method using wireless communication. Although FIG. 3 shows an example in which the information processing device 110 and the output device 300 are configured as separate entities, the information processing device 110 and the output device 300 may also be configured as an integrated device.

[0022] The interior image acquisition unit 101 captures an image of a subject inside the vehicle C1 to generate an image (image data), and outputs image information related to the generated image to the information processing device 110. The interior image acquisition unit 101 is provided inside the vehicle C1 (for example, on the ceiling), and captures an image of a subject inside the vehicle C1 to generate an image (image data). The interior image acquisition unit 101 is configured, for example, with one or more camera devices or image sensors capable of capturing an image of a subject. For example, one interior image acquisition unit 101 may be provided in the front of the vehicle C1 to capture an image of a subject from the front to generate an image (image data), and another interior image acquisition unit 101 may be provided in the rear of the vehicle C1 to capture an image of a subject from the rear to generate an image (image data).

[0023] The vehicle exterior image acquisition unit 102 captures an image of a subject outside the vehicle C1 to generate an image (image data), and outputs image information related to the generated image to the information processing device 110. Two or more vehicle exterior image acquisition units 102 may be provided, and all or some of the images from these vehicle exterior image acquisition units 102 may be used. For example, one vehicle exterior image acquisition unit 102 may be provided in front of the vehicle C1 to capture an image of a subject in front of the vehicle C1 to generate an image (image data), and another vehicle exterior image acquisition unit 102 may be provided behind the vehicle C1 to capture an image of a subject behind the vehicle C1 to generate an image (image data). Furthermore, one or more devices capable of capturing subjects present in all directions around the vehicle C1 and subjects inside the vehicle C1, such as a 360-degree camera, may be used.

[0024] The engine room image acquisition unit 103 (see Figure 2) captures images of the inside of the engine room 7 and the surroundings of the engine room 7 (the surroundings of the vehicle C1) to generate images (image data), and outputs image information related to the generated images to the information processing device 110.

[0025] The vehicle interior image acquisition unit 101, the vehicle exterior image acquisition unit 102, and the engine room image acquisition unit 103 are each configured with, for example, an image sensor that receives light from a subject collected by a lens, and an image processing unit that performs predetermined image processing on image data generated by the image sensor. As the image sensor, for example, a CCD (Charge Coupled Device) type or a CMOS (Complementary Metal Oxide Semiconductor) type image sensor can be used.

[0026] The sensors 104 are various sensors installed in the vehicle C1, and output detection information acquired by each sensor to the information processing device 110. The sensors include, for example, LiDAR (Light Detection and Ranging), RADAR (Radio Detection and Ranging), sonar, a position information acquisition sensor (position information acquisition unit), seat occupancy sensor, seat belt sensor, illuminance sensor, temperature sensor, non-contact temperature sensor, pressure sensor, humidity sensor, alcohol sensor, photoelectric sensor, noise sensor, etc. Known sensors can be used for each of these sensors. Note that LiDAR, RADAR, sonar, etc. are examples of sensors that detect objects around the vehicle C1. Note that these are only examples, and other sensors may also be used. Also, only some of these sensors may be used.

[0027] The location information acquisition unit acquires location information regarding the location where the vehicle C1 is located. For example, it can be realized by a GNSS receiver that acquires location information using a Global Navigation Satellite System (GNSS). The location information includes data regarding the location, such as latitude, longitude, and altitude, at the time of receiving the GNSS signal. The location information may also be acquired by other methods of acquiring location information. For example, the location information may be derived using information from nearby access points or base stations. The location information may also be acquired using a beacon.

[0028] The sound acquisition unit 105 is provided inside and outside the vehicle C1, acquires sounds inside and outside the vehicle C1, and outputs sound information related to the acquired sounds to the information processing device 110. As the sound acquisition unit 105, for example, one or more microphones or sound acquisition sensors can be used.

[0029] The information processing device 110 includes a control unit 120, a storage unit 130, and a communication unit 140. The communication unit 140 exchanges various types of information with other devices using wired or wireless communication under the control of the control unit 120.

[0030] The control unit 120 controls each unit based on various programs stored in the storage unit 130. The control unit 120 is realized by a processing device such as a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit). The vehicle ECU (Electronic Control Unit) of the vehicle C1 may also be used as the control unit 120, or a processing device different from the vehicle ECU may be provided as the control unit 120.

[0031] The control unit 120 executes various controls based on the information output from the vehicle interior image acquisition unit 101, the vehicle exterior image acquisition unit 102, the engine room image acquisition unit 103, the sensors 104, the sound acquisition unit 105, the communication unit 140, etc., and the information acquired by the vehicle state acquisition unit 125. For example, the control unit 120 executes a control process to control the display state of the output device 300 and the audio output state from the output device 300. Furthermore, for example, the control unit 120 executes a control process to control an external device of the vehicle C1 (for example, the external device 400) via the network 20. Specifically, the control unit 120 includes the vehicle state acquisition unit 125, a detection unit 126, an estimation unit 127, and an output control unit 128.

[0032] The vehicle state acquisition unit 125 acquires information (vehicle state information) relating to various vehicle states of the vehicle C1 and outputs the acquired vehicle state information to the detection unit 126. The vehicle state information can be acquired, for example, from a CAN (Controller Area Network) signal or a vehicle ECU. The vehicle state information includes, for example, vehicle speed, wheel speed, acceleration, shift lever position (for example, P range, D range), accelerator pedal depression amount, brake pedal depression amount (brake operation), position information of the vehicle C1, and whether or not each indicator lamp is lit (for example, blinker operation, hazard lamp operation, passing operation).

[0033] The detection unit 126 detects specific objects (e.g., people, maintenance tools) present inside and outside the vehicle C1 and their movements based on the information output from each unit. The detection unit 126 then outputs the detection results of the specific objects, motion information related to the detected movements of the specific objects, and information output from each unit (e.g., sensor values) to the estimation unit 127 and the output control unit 128. For example, the detection unit 126 can detect basic human movements such as standing, getting up, running, crouching, walking, turning the head up, down, left, and right, moving the feet up, down, left, and right, and moving the hands up, down, left, and right using known detection techniques. For example, the detection unit 126 can detect the skeleton and posture of the user U1, and objects present inside and outside the vehicle C1, using images of the user U1, objects, etc. included in the images acquired by the image acquisition units (the vehicle interior image acquisition unit 101, the vehicle exterior image acquisition unit 102, and the engine room image acquisition unit 103). Furthermore, for example, the detection unit 126 can detect the color of the vehicle C1, objects around the vehicle C1, and the like, using images of the vehicle C1 and its surroundings included in each image acquired by the image acquisition unit (vehicle exterior image acquisition unit 102, engine room image acquisition unit 103). These detection methods will be described in detail with reference to Figs. 7 to 13, etc. Note that each piece of information acquired by the detection unit 126 (detection results of a specific object, operation information related to the movement of the detected specific object, and each piece of information output from each unit (e.g., sensor values)) is an example of operation information related to the operation of the user U1.

[0034] In this embodiment, the term "user's action" includes the meaning of user's actions, user's behavior, user's operation, user's behavior, and the like.

[0035] The estimation unit 127 estimates the type of maintenance of the vehicle C1 performed by the action of the user U1 based on the action information (such as the action information of the specific object) output from the detection unit 126. Then, the estimation unit 127 outputs the estimation result to the output control unit 128.

[0036] For example, when estimating interior cleaning as a type of maintenance, the estimation unit 127 can estimate various types of interior cleaning based on each object and its movement (e.g., the skeleton and posture of user U1, and the movement of each object) detected using an image, and the humidity, alcohol concentration, and pressure inside the vehicle acquired by the sensors 104. For example, if the movement of user U1's left arm (skeleton) and a vacuum cleaner (object) are detected and the pressure and noise based on the sensor values ​​exceed certain values, the movement of sucking up dirt with the vacuum cleaner can be estimated. Also, for example, if a microfiber cloth (object) is detected and the movement of user U1's left arm continues to move by more than a certain width for more than a certain period of time, the movement of scrubbing with the microfiber cloth can be estimated.

[0037] Furthermore, for example, when estimating a manual car wash outside the vehicle as a type of maintenance, the estimation unit 127 can estimate various manual car washes outside the vehicle based on each object and its movement (e.g., the skeleton and posture of the user U1, the color of the vehicle C1, and the movement of each object) detected using the image. For example, if the movement of the user U1's left arm (skeleton) and a high-pressure gun (object) are detected, the movement of spraying water on the vehicle C1 with the high-pressure gun can be estimated. For example, if the movement of the user U1's left arm (skeleton) and the high-pressure gun (object) are detected and a gradual change in color of the vehicle C1 to white is detected, the movement of shampooing the vehicle C1 with the high-pressure gun can be estimated. For example, if a microfiber cloth (object) is detected and the movement of the user U1's left arm continues to move by more than a certain width for more than a certain period of time, the movement of scrubbing with the microfiber cloth can be estimated. Note that other estimation methods by the estimation unit 127 will be described in detail with reference to Figures 7 to 13, etc.

[0038] Here, maintenance refers to, for example, removing dirt from at least one of the interior and exterior of the vehicle C1, cleaning at least one of the interior and exterior of the vehicle C1 (i.e., cleaning things), replenishing objects or liquids in the vehicle C1 (i.e., replacing or replenishing missing items), replacing deteriorated items in the vehicle C1, etc. For example, removing dirt from the exterior of the vehicle C1 and cleaning the exterior of the vehicle C1 refer to, for example, washing the body of the vehicle C1. Also, removing dirt from the interior of the vehicle C1 and cleaning the interior of the vehicle C1 refer to, for example, cleaning the interior of the vehicle C1. Also, replenishing missing items refers to replenishing fuel, windshield washers, coolant, tire air, etc. Also, replacing deteriorated items in the vehicle C1 refers to replacing headlights, tires, etc. The types of maintenance will be described in detail with reference to FIG. 4 etc.

[0039] The output control unit 128 controls the output state of the output device 300 (or a display medium such as a HUD, a window glass capable of displaying the data, an external device 400, etc.). Specifically, the output control unit 128 controls the output state of the output device 300 based on the detection result output from the detection unit 126 and the estimation result output from the estimation unit 127. For example, the output control unit 128 executes a specific effect during the maintenance operation corresponding to the maintenance operation of the user U1 (the operation of the user U1 during the maintenance operation) based on the operation information (e.g., the detection result of the specific object, the operation information, and each piece of information output from each unit) acquired during the maintenance operation estimated by the estimation unit 127 and the type of the maintenance operation estimated by the estimation unit 127. The specific effect may be, for example, an effect using the agent image 200 on the display unit 310, an effect by outputting audio information related to the agent from the sound output unit 320, an effect by displaying or outputting audio information on the external device 400, etc. The specific effect will be described in detail with reference to FIGS. 7 to 13, etc.

[0040] The storage unit 130 is a storage medium that stores various types of information. For example, the storage unit 130 stores various types of information (e.g., a control program, a maintenance information DB 150, an appropriate operation information DB 160, an agent information DB 170, and a map information DB 180) that are necessary for the control unit 120 to perform various processes. The storage unit 130 also stores various types of information acquired via the communication unit 140. The storage unit 130 can be, for example, a read-only memory (ROM), a random access memory (RAM), a static random access memory (SRAM), a hard disk drive (HDD), a solid state drive (SSD), or a combination thereof.

[0041] The maintenance information DB 150 stores various pieces of information related to maintenance performed on the vehicle C1. The maintenance information DB 150 will be described in detail with reference to FIG.

[0042] The appropriate operation information DB 160 stores appropriate operation determination information for determining whether the maintenance operation of the user U1 regarding the maintenance performed on the vehicle C1 is appropriate. The appropriate operation information DB 160 will be described in detail with reference to FIG. 5.

[0043] The agent information DB 170 stores various types of information required to realize various operations (operations related to the output of agent information) of the output device 300 or the external device 400. For example, agent image information to be displayed on the display unit 310 (e.g., image information for displaying each agent image 200 shown in FIG. 6) and audio information output from the sound output unit 320 are stored in the agent information DB 170. Also, for example, information on operations performed using the agent image 200 when various types of communication are performed is stored in the agent information DB 170.

[0044] The map information DB 180 stores map information such as road information about roads required for route guidance of the vehicle C1. The map information includes information about the home of the user U1 and facilities where maintenance of the vehicle C1 can be performed. The maintenance information DB 150, the appropriate operation information DB 160, the agent information DB 170, and the map information DB 180 may be stored in the storage unit 130 provided in the vehicle C1 and used, or may be obtained from an external device via the network 20 and used.

[0045] The output device 300 is an output device that executes various output processes based on instructions from the information processing device 110. The output device 300 includes a display unit 310 and a sound output unit 320. The display unit 310 and the sound output unit 320 are controlled based on a control unit (not shown) included in the output device 300.

[0046] The display unit 310 is a display unit that displays various images based on instructions from the information processing device 110. The display unit 310 may be configured as a touch panel that allows a user to perform operation input by touching or bringing a finger close to the display surface, or may be configured as a separate user interface.

[0047] The sound output unit 320 outputs various sounds based on instructions from the information processing device 110. For example, one or more speakers can be used as the sound output unit 320. Note that the display unit 310 and the sound output unit 320 are examples of user interfaces, and some of them may be omitted, or other user interfaces may be used.

[0048] The external device 400 includes a communication unit 410, a control unit 420, an output unit 430, and an operation unit 440. The communication unit 410 exchanges various types of information with other devices using wired or wireless communication under the control of the control unit 420. The operation unit 440 accepts various operations from the user U1 and outputs the contents of the accepted operations to the control unit 420.

[0049] The control unit 420 controls each unit based on various programs stored in a storage unit (not shown). The control unit 420 is realized by a processing device such as a CPU or a GPU.

[0050] The output unit 430 is an output unit that executes various output processes under the control of the control unit 420. The output unit 430 includes a display unit 431 and a sound output unit 432.

[0051] The display unit 431 is a display unit that displays various images based on the control of the control unit 420. The sound output unit 432 is a unit that outputs various sounds based on the control of the control unit 420. As the sound output unit 432, for example, one or more speakers can be used.

[0052] The display unit 431 and the operation unit 440 may be configured as a touch panel that allows a user to perform operation input by touching or approaching the display surface with their finger, or may be configured as a separate user interface. When configured as a separate user interface, various operation members such as buttons and a keyboard can be used as the operation unit 440. The display unit 431, the sound output unit 432, and the operation unit 440 are examples of user interfaces, and some of them may be omitted, or another user interface may be used.

[0053] In this embodiment, an example in which the external device 400 is a car wash CW1 (see FIG. 11) and a fuel supply machine TA1 (see FIG. 13) is shown.

[0054] [Example of maintenance information DB content] 4 is a diagram schematically showing each piece of information stored in the maintenance information DB 150. The maintenance information DB 150 is a database for managing each piece of information related to maintenance performed on the vehicle C1.

[0055] The maintenance information DB 150 stores, in association with each other, a date and time 151, location information 152, facility information 153, maintenance content 154, maintenance portion 155, and appropriateness determination level 156. Each of these pieces of information is stored by the estimation unit 127 and the output control unit 128 every time maintenance is performed by the user U1. Note that these pieces of information are merely examples, and other information may also be stored in the maintenance information DB 150.

[0056] For example, when the estimation unit 127 estimates the type of maintenance of the vehicle C1, each piece of information is stored in the maintenance information DB 150 at the time the maintenance is completed. For example, the date and time when the maintenance was completed is stored in date and time 151, location information regarding the location where the maintenance was performed is stored in location information 152, and facility information regarding the facility where the maintenance was performed is stored in facility information 153. Furthermore, maintenance information for identifying the performed maintenance is stored in maintenance content 154, maintenance part information for identifying the part where the maintenance was performed is stored in maintenance part 155, and judgment level information regarding the appropriateness of the actions of user U1 in the performed maintenance is stored in appropriateness judgment level 156.

[0057] The location information 152 stores location information acquired by the sensors 104 (location information acquisition unit). The facility information 153 stores facility information that can be identified based on the location information acquired by the sensors 104 (location information acquisition unit) and map information stored in the map information DB 180 (see FIG. 3). The judgment level information stored in the appropriateness judgment level 156 will be described in detail with reference to FIGS. 7 to 13.

[0058] [Example of the contents of the appropriate operation information DB] 5 is a diagram schematically illustrating each piece of information stored in the appropriate operation information DB 160. The appropriate operation information DB 160 is a database for managing appropriate operation determination information for determining the operation of user U1 regarding maintenance performed on vehicle C1.

[0059] The appropriate operation information DB 160 stores maintenance details 161 and appropriate operation determination information 162 in association with each other. Each piece of information is stored for each type of maintenance performed by the user U1. The maintenance details 161 correspond to the maintenance details 154 shown in FIG. 5. Each piece of information is an example, and other information may be stored in the maintenance information DB 150.

[0060] The appropriate action determination information 162 is appropriate action determination information for determining whether the maintenance action of the user U1 corresponding to the ongoing maintenance is appropriate. As this appropriate action determination information, for example, user action speed information for specifying the speed of the user's predetermined action (e.g., wiping action), user action count information for specifying the number of times the user's predetermined action (e.g., wiping action) has been performed (e.g., the number of times wiping in the up-down direction, the number of times wiping in the left-right direction), the duration of the user's predetermined action (e.g., wiping action) (e.g., several minutes to several tens of minutes), the order of the user's predetermined action (e.g., wiping action) (e.g., up-down direction, left-right direction, clockwise, counterclockwise), etc. are stored.

[0061] For example, when a user performs a wiping action as a predetermined action, if the wiping speed is higher than a certain speed, the force applied may be too great and the vehicle C1 may be damaged. Therefore, when a user performs a wiping action as a predetermined action and the wiping speed is higher than a certain speed, it is possible to determine that the maintenance action of the user U1 is inappropriate. Furthermore, when a user performs a wiping action as a predetermined action and the number of times the same location is repeatedly scrubbed is too great and the vehicle C1 may be damaged. Therefore, when a user performs a wiping action as a predetermined action and the number of times the same location is repeatedly scrubbed is too great and the vehicle C1 may be damaged. Furthermore, when a user performs a car wash action for the vehicle C1 as a predetermined action and the order of the car wash action is to scrub the vehicle C1 with a sponge before spraying water on the vehicle C1, the vehicle C1 may be damaged. Therefore, when the user performs a predetermined action related to washing the vehicle C1, if the order of the actions related to washing the vehicle C1 includes scrubbing the body of the vehicle C1 with a sponge before spraying water on it, it can be determined that the maintenance action of the user U1 is inappropriate.

[0062] In this way, the output control unit 128 can determine whether the maintenance action of user U1 is appropriate based on the actions of user U1 related to the ongoing maintenance and each piece of information included in the appropriate action determination information corresponding to the ongoing maintenance. That is, examples of appropriate actions for each type of maintenance are stored in advance, and the actions of user U1 are compared with the examples of appropriate actions to determine whether user U1's maintenance action is appropriate (task appropriateness determination). In this case, for example, a value indicating the level of appropriateness of user U1's maintenance action (e.g., an evaluation value indicating the level of appropriateness) can be calculated and stored in the appropriateness determination level 156 (see FIG. 4). The appropriate action determination information 162 will be described in detail with reference to FIGS. 7 to 13, etc.

[0063] [Example of agent device display] FIG. 6 is a simplified diagram showing an example of the display mode of the agent image 200 displayed on the display unit 310 of the output device 300 or the like.

[0064] 6(A), 6(B), and 6(F) show agent images 200 that express a good mood. 6(C), 6(D), and 6(E) show agent images 200 that express a bad mood. 6(G) shows an agent image 200 that expresses gratitude to user U1. 6(H) shows a modified version of agent image 200 that expresses a good mood. Note that these are examples of the display modes of agent image 200, and display modes that express other emotions may also be used. Also, while FIGS. 6(A) to 6(E) show examples of agent images 200 that resemble a human face, an agent image that resembles the entire human body may also be displayed. Also, while FIGS. 6(F) and 6(G) show examples of agent images 200 that resemble the entire human body, an agent image that resembles a human face may also be displayed. Furthermore, a robot that resembles a human face or the entire human body may also be used as an agent.

[0065] [Example of car interior cleaning estimation and agent output] First, an example of estimating the behavior of a user U1 cleaning the interior of a vehicle will be described. Possible cleaning activities include collecting trash, wiping, sterilizing, etc. Therefore, the following describes an estimation method for these cleaning activities.

[0066] [Example of estimated garbage collection] For example, based on the actions of user U1 included in the vehicle interior image acquired by the vehicle interior image acquisition unit 101 and the cleaning tools included in the vehicle interior image, it can be estimated that user U1 is performing the action of collecting trash inside the vehicle. For example, if cleaning tools such as a vacuum cleaner, broom, dustpan, etc. are included in the vehicle interior image and user U1 is holding them and performing the action of collecting trash inside the vehicle, it can be estimated that user U1 is performing the action of collecting trash inside the vehicle. Note that a known detection method can be used to detect the cleaning tool. Also, a known detection method can be used to detect the action of user U1 using the cleaning tool to collect trash inside the vehicle.

[0067] It is possible to estimate that the user U1 is performing an action of collecting dust inside the vehicle based on the sounds inside the vehicle acquired by the sound acquisition unit 105. For example, if the sound of a vacuum cleaner is continuously detected for a predetermined period of time, it is possible to estimate that the user U1 is performing an action of collecting dust inside the vehicle. It is also possible to estimate that the user U1 is performing an action of collecting dust inside the vehicle (the vacuum cleaner is sucking up) based on the pressure (for example, the pressure applied to the seat) at a predetermined location inside the vehicle acquired by the sensors 104 (for example, a pressure sensor). For example, if it is continuously detected that pressure is being applied intermittently to various parts inside the vehicle, it is possible to estimate that the user U1 is performing an action of collecting dust inside the vehicle. It is also possible to make an estimation by combining at least two of image-based detection, sound-based detection, and sensor-based detection.

[0068] Furthermore, based on the comparison result between the image before cleaning and the image during cleaning, it is possible to estimate that the user U1 is taking action to collect trash inside the vehicle. For example, vehicle interior images acquired by the vehicle interior image acquisition unit 101 are periodically (or irregularly) stored in the storage unit 130. Then, the latest vehicle interior image among the vehicle interior images acquired by the vehicle interior image acquisition unit 101 is compared with past vehicle interior images (e.g., multiple images taken over a predetermined period), and based on the comparison result, it is possible to estimate that the user U1 is taking action to collect trash inside the vehicle. Specifically, by comparing the latest vehicle interior image with the past vehicle interior images, if an object, trash, etc. included in the past vehicle interior image is no longer included in the latest vehicle interior image (or if the object, trash, etc. is less present in the latest vehicle interior image), it is possible to estimate that the user U1 is taking action to collect trash inside the vehicle. Note that instead of using the past vehicle interior image, an image of the vehicle interior when there is no dirt such as trash (initial image) may be used. In this case, the latest vehicle interior image is compared with the initial image, and if the latest vehicle interior image is close to the initial image (i.e., if there are fewer objects, debris, etc. in the latest vehicle interior image), it can be estimated that user U1 is collecting debris from the vehicle. Any known detection method can be used to detect objects and debris. It is also possible to make an estimation by combining the above-described detection based on images, sounds, sensors, etc. with detection based on the results of comparing images before cleaning with images during cleaning.

[0069] 7 is a simplified diagram illustrating a case in which a user U1 uses a vacuum cleaner VC1 to collect dust inside a vehicle. Specifically, the example illustrates the operation of collecting dust from the passenger seat. In this case, the dust collection operation can be estimated based on detection based on a pressure sensor installed on the passenger seat, detection based on at least one of an image and sound, detection based on a comparison result between an image before cleaning and an image during cleaning, etc.

[0070] [Example of determining whether maintenance operations during garbage collection are appropriate or inappropriate] Next, a method for determining whether or not the maintenance operation performed by the user U1 to collect dust inside the vehicle C1 is appropriate will be described.

[0071] Here, an inappropriate maintenance operation (i.e., inappropriate work) means, for example, an action that may damage the vehicle C1. In such a case, it is possible to convey to the user U1 that the vehicle C1 may be damaged by performing an effect such as making the agent image 200 look pained.

[0072] For example, based on the appropriate action determination information stored in the appropriate action information DB 350 and the action of the user U1 included in the vehicle interior image acquired by the vehicle interior image acquisition unit 101, it is possible to determine whether the action of the user U1 collecting trash inside the vehicle is appropriate. Specifically, the output control unit 128 acquires appropriate action determination information related to trash collection from the appropriate action information DB 350. Then, based on the degree of match between each user's action included in the appropriate action determination information related to trash collection and the action of the user U1 included in the vehicle interior image, the output control unit 128 can determine whether the action of the user U1 collecting trash inside the vehicle is appropriate. For example, as the degree of match increases, it is possible to increase the evaluation value (appropriateness determination level 156 (see FIG. 4)) for determining that the action of the user U1 collecting trash inside the vehicle is appropriate. Then, based on the evaluation value, the output control unit 128 determines the display mode of the agent image 200 and the output content of the audio information. For example, since it is estimated that the appropriateness of the garbage collection action increases as the evaluation value increases, it is possible to make the agent image 200 smile more brightly or to output audio information praising the appropriateness of the garbage collection action. On the other hand, since it is estimated that the appropriateness of the garbage collection action decreases as the evaluation value decreases, it is possible to display the agent image 200 with a dissatisfied face or to output audio information instructing the appropriate garbage collection action.

[0073] 7 shows an example of the display of the agent image 200 and an example of the output of the audio information S2 when the maintenance action of collecting trash inside the vehicle by the user U1 is determined to be appropriate. In this way, the agent image 200 reacts to the attachment action of the user U1 to the vehicle C1 while the maintenance is being performed, which can further increase the sense of closeness and attachment that the user U1 has to the vehicle C1.

[0074] [Methods for estimating interior cleaning such as wiping and disinfecting] For example, based on the actions of user U1 included in the vehicle interior image acquired by the vehicle interior image acquisition unit 101 and the cleaning tools included in the vehicle interior image, it can be estimated that user U1 is performing actions such as wiping and disinfecting the interior of the vehicle. For example, if disinfectant spray, cloths, laundry detergent, etc. are included as cleaning tools in the vehicle interior image and user U1 is using them to wipe the interior of the vehicle, it can be estimated that user U1 is performing actions such as wiping and disinfecting the interior of the vehicle. As described above, known detection methods can be used to detect cleaning tools. Furthermore, known detection methods can be used to detect the actions of user U1 using cleaning tools to perform cleaning activities such as wiping and disinfecting the interior of the vehicle.

[0075] Based on the sounds inside the vehicle acquired by the sound acquisition unit 105, it can be estimated that the user U1 is performing actions such as wiping or disinfecting the interior of the vehicle. For example, if the sound of spraying disinfectant spray and the sound of wiping are continuously detected alternating, it can be estimated that the user U1 is performing actions such as wiping or disinfecting the interior of the vehicle. Furthermore, the sensors 104 (e.g., humidity sensor, alcohol sensor) can determine whether water, alcohol, or the like has been splashed on a predetermined location inside the vehicle. Therefore, if it is determined that water has been splashed on a predetermined location inside the vehicle, it can be estimated based on the determination result that the user U1 is wiping the interior of the vehicle with water. Furthermore, if it is determined that alcohol has been splashed on a predetermined location inside the vehicle, it can be estimated based on the determination result that the user U1 is disinfecting the interior of the vehicle. Note that estimation may be performed by combining at least two of image-based detection, sound-based detection, and sensor-based detection.

[0076] Furthermore, based on the comparison result between the image before cleaning and the image during cleaning, it is possible to estimate that the user U1 is cleaning the interior of the vehicle, such as wiping or disinfecting. For example, vehicle interior images acquired by the vehicle interior image acquisition unit 101 are stored in the storage unit 130 periodically (or irregularly). Then, the latest vehicle interior image among the vehicle interior images acquired by the vehicle interior image acquisition unit 101 is compared with a previous (e.g., immediately preceding) vehicle interior image, and based on the comparison result, it is possible to estimate that the user U1 is cleaning the interior of the vehicle, such as wiping or disinfecting. Specifically, by comparing the latest vehicle interior image with a previous (e.g., immediately preceding) vehicle interior image, if the interior surface included in the latest vehicle interior image appears glossier than the interior surface included in the previous vehicle interior image, it is possible to estimate that the user U1 is cleaning the interior of the vehicle, such as wiping or disinfecting. Note that known image recognition technology can be used to detect gloss. Also, gloss determination may be performed using various sensors, such as a photoelectric sensor. Note that color determination, etc., can be performed using various sensors, such as a photoelectric sensor. Therefore, it may be possible to determine whether the interior surface included in the vehicle interior image has changed by using a change in color, etc. Note that estimation may be performed by combining the above-described detection based on images, sounds, sensors, etc. with detection based on a comparison result between an image before cleaning and an image during cleaning.

[0077] 8A and 8B are simplified diagrams illustrating a case in which a user U1 is performing a cleaning operation to disinfect the interior of a vehicle using a disinfectant spray DS1 and a cloth Z1, along with examples of displays during the operation. Fig. 8A shows an example of a cleaning operation to disinfect the interior of a vehicle on the passenger seat side. In this case, the cleaning operation, such as wiping and disinfecting, can be estimated based on detection using an alcohol sensor installed on the passenger seat, detection using at least one of images and sounds, and detection based on a comparison of images before and during cleaning.

[0078] Fig. 8(B) shows an example of agent information output when the maintenance operation of user U1 shown in Fig. 8(A) is appropriate. Fig. 8(C) shows an example of agent information output when the maintenance operation of user U1 shown in Fig. 8(A) is inappropriate. The method for determining whether the maintenance operation of user U1 is appropriate will be described below.

[0079] [Examples of determining whether maintenance actions such as wiping and disinfecting are appropriate] Next, a method for determining whether or not the maintenance action performed by the user U1, such as wiping or disinfecting the interior of the vehicle C1, is appropriate will be described.

[0080] Here, an example of a user's actions, such as wiping and disinfecting, is shown. As described above, it is possible to determine whether the actions of user U1, such as wiping and disinfecting the interior of the vehicle, are appropriate based on the appropriate action determination information stored in the appropriate action information DB 350 and the actions of user U1 included in the vehicle interior image acquired by the vehicle interior image acquisition unit 101. Specifically, the output control unit 128 acquires appropriate action determination information related to wiping and disinfecting the interior of the vehicle from the appropriate action information DB 350. Then, the output control unit 128 can determine whether the actions of user U1, such as wiping and disinfecting the interior of the vehicle, are appropriate based on the degree of agreement between each user's action included in the appropriate action determination information related to wiping and disinfecting the interior of the vehicle and the action of user U1 included in the vehicle interior image. For example, it is possible to increase the evaluation value for determining that user U1's actions, such as wiping and disinfecting the interior of the vehicle, are appropriate as the degree of agreement increases. Then, the output control unit 128 determines the display mode of the agent image 200 and the output content of the audio information based on the evaluation value.

[0081] Furthermore, for example, the following cases (1) to (5) can be determined to be inappropriate. (1) The display unit 310 (for example, a liquid crystal monitor) is pressed hard. This can be detected by, for example, a pressure sensor. (2) The display unit 310 (e.g., a liquid crystal monitor) is being wiped with a wet towel (e.g., a towel containing water or alcohol). This can be detected by, for example, a photoelectric sensor, which can detect whether a hand or finger (fingerprint) is present on the monitor surface, making it possible to detect that the user is wiping the monitor. The photoelectric sensor can also detect whether liquid is present on the monitor surface. (3) The user U1 is wiping quickly. In other words, if the user U1's hand is moving quickly, it is estimated that the user U1 is pressing hard on the area to be wiped, which is likely to damage the area. (4) The wiping action is performed randomly (i.e., roughly). In other words, if the wiping action is not performed in a predetermined order (e.g., left / right / up / down), it can be determined to be inappropriate. (5) When the number of times the same target part is wiped is equal to or greater than a predetermined value, that is, when the number of times the part is wiped is too many (that is, when the part is polished too much), it can be determined that there is a high possibility that the vehicle C1 will be damaged.

[0082] Furthermore, for example, when user U1 is performing a wiping action, it can be determined that the action is appropriate based on the user U1's hand movements having a certain directionality. For example, it can be determined that the action is appropriate when the hand movements are in a predetermined order such as left-right or up-down, or when the hand movements are in a predetermined direction such as clockwise or counterclockwise.

[0083] [Example of vehicle body washing estimation and agent output] Next, an example of estimating the actions of a user U1 washing the body of a vehicle C1 will be described. Washing the body of the vehicle C1 can include actions such as pouring water on the body of the vehicle C1, applying shampoo to the body of the vehicle C1, scrubbing with a sponge, and rinsing off the foam with water. Therefore, a method for estimating the actions of the user U1 when performing these cleaning tasks will be described below.

[0084] [Method for estimating vehicle body wash] For example, it is possible to estimate that user U1 is washing the vehicle body based on the actions of user U1 included in the vehicle exterior image acquired by the vehicle exterior image acquisition unit 102 and the cleaning tools, water, shampoo, etc. included in the vehicle exterior image. For example, if a washer (or high-pressure washer), sponge, etc. are included as cleaning tools in the vehicle exterior image and user U1 is using them to wash the body of vehicle C1, it is possible to estimate that user U1 is washing vehicle C1. Furthermore, for example, if white liquid-like parts on the body of vehicle C1 are included in the vehicle exterior image and user U1 is wiping them, it is possible to estimate that user U1 is using shampoo to wash vehicle C1.

[0085] As described above, a known detection method can be used to detect the cleaning tool. Furthermore, a known detection method can be used to detect the action of the user U1 using the cleaning tool to wash the vehicle C1. Based on the sound outside the vehicle acquired by the sound acquisition unit 105, it can be estimated that the user U1 is washing the vehicle C1. For example, if the sound of a washer is continuously detected for a predetermined period of time, it can be estimated that the user U1 is washing the vehicle C1.

[0086] Furthermore, it is possible to estimate that the user U1 is washing the vehicle C1 based on water on the body of the vehicle C1 acquired by the sensors 104 (for example, a raindrop sensor). For example, if the user U1 is carrying tools related to car washing and a state in which water is being intermittently added to the body of the vehicle C1 is continuously detected, it is possible to estimate that the user U1 is washing the vehicle C1. Note that estimation may be performed by combining at least two of image-based detection, sound-based detection, and sensor-based detection.

[0087] Furthermore, as described above, it is possible to estimate that the user U1 is washing the vehicle C1 based on the comparison result between the image before cleaning and the image during cleaning. For example, vehicle exterior images acquired by the vehicle exterior image acquisition unit 102 are periodically (or irregularly) stored in the storage unit 130. Then, the latest vehicle exterior image among the vehicle exterior images acquired by the vehicle exterior image acquisition unit 102 is compared with past vehicle exterior images (e.g., multiple images taken over a predetermined period), and based on the comparison result, it is possible to estimate that the user U1 is washing the vehicle C1. Specifically, by comparing the latest vehicle exterior image with the past vehicle exterior images, if dirt or the like on the body of the vehicle C1 contained in the past vehicle exterior image is no longer contained in the latest vehicle interior image and if water, foam, or the like is contained in the latest vehicle interior image, it is possible to estimate that the user U1 is washing the vehicle C1. Note that, instead of using past vehicle interior images as described above, a vehicle interior image without dirt (initial image) may be used. Furthermore, a known detection method can be used to detect dirt or the like on the body of the vehicle C1. It should be noted that estimation may be performed by combining the above-described detection based on images, sounds, sensors, etc. with detection based on the results of comparing images before cleaning with images during cleaning.

[0088] FIG. 9 is a simplified diagram showing a case where a user U1 is using a washer IJ1 to wash the body of a vehicle C1. FIG. 9(A) shows the state where the user U1 is using the washer IJ1 to spray water onto the body of the vehicle C1. FIG. 9(B) shows the state where the user U1 is using the washer IJ1 to spray shampoo onto the body of the vehicle C1. FIG. 9(C) shows the state after the body of the vehicle C1 has been washed.

[0089] 10 is a simplified diagram showing a case where a user U1 is washing the body of a vehicle C1 using a cloth Z2. For example, FIGS. 10(A) and 10(B) show a state where the user U1 is wiping the body of the vehicle C1 using the cloth Z2 after washing the body of the vehicle C1 using the washer IJ1 shown in FIG.

[0090] As shown in FIGS. 9(A) to 9(C) and 10(A) and 10(B), when the user U1 is working outside the vehicle, instead of displaying the agent image 200 on the display unit 310, the agent image 200 is displayed at a position visible to the user U1. Specifically, as shown in FIGS. 9(A) and 9(B), when the user U1 is working near the passenger-side window W2, the agent image 200 can be displayed on the window W2. Furthermore, as shown in FIGS. 9(C), 10(A), and 10(B), when the user U1 is working near the windshield 4, the agent image 200 can be displayed on the windshield 4. The position of the user U1 can be determined based on the vehicle exterior image acquired by the vehicle exterior image acquisition unit 102.

[0091] Here, it is possible to grasp the status of past maintenance performed by user U1 based on the maintenance information stored in the maintenance information DB 150 (see FIG. 4). Therefore, the output control unit 128 can acquire the date and time when the current maintenance was performed in the past based on the maintenance information stored in the maintenance information DB 150. If there is a predetermined period of time (for example, several months) between the current maintenance and the date and time when that maintenance was performed in the past, it is possible to output audio information indicating this. For example, as shown in FIG. 9(B), it is possible to output audio information S6 such as "It's been a while since I last had a shampoo, it was fun, thank you."

[0092] [Example of determining whether maintenance actions performed during a vehicle wash were appropriate] Next, a method for determining whether or not the maintenance operation performed by the user U1 to wash the body of the vehicle C1 is appropriate when the user U1 is performing the maintenance will be described.

[0093] For example, based on the appropriate action determination information stored in the appropriate action information DB 350 and the actions of the user U1 included in the vehicle exterior image acquired by the vehicle exterior image acquisition unit 102, it is possible to determine whether the maintenance action of the user U1 washing the body of the vehicle C1 is appropriate. Specifically, the output control unit 128 acquires appropriate action determination information related to the washing of the vehicle C1 from the appropriate action information DB 350. Then, based on the degree of match between each user's action included in the appropriate action determination information related to the washing of the vehicle C1 and the action of the user U1 included in the vehicle exterior image, the output control unit 128 can determine whether the maintenance action of the user U1 washing the body of the vehicle C1 is appropriate. For example, as the degree of match increases, it is possible to increase the evaluation value for determining that the maintenance action of the user U1 washing the body of the vehicle C1 is appropriate. Then, based on the evaluation value, the output control unit 128 determines the display mode of the agent image 200 and the output content of the audio information.

[0094] Furthermore, for example, the output control unit 128 can determine whether the procedure by which the user U1 washes the body of the vehicle C1 is appropriate based on the appropriate action determination information stored in the appropriate action information DB 350 and the actions of the user U1 included in the vehicle exterior image acquired by the vehicle exterior image acquisition unit 102. Specifically, the output control unit 128 acquires appropriate action determination information related to washing the vehicle C1 from the appropriate action information DB 350. Then, the output control unit 128 can determine whether the maintenance action of the user U1 washing the body of the vehicle C1 is appropriate based on the degree of match between each user's action procedure included in the appropriate action determination information related to washing the vehicle C1 and the action procedure of the user U1 included in the vehicle exterior image. For example, the output control unit 128 can increase the evaluation value for determining that the procedure by the user U1 to wash the body of the vehicle C1 is appropriate as the degree of match increases. Then, the output control unit 128 determines the display mode of the agent image 200 and the output content of the audio information based on the evaluation value.

[0095] Furthermore, for example, the following cases (1) to (4) can be determined to be inappropriate. (1) The body of the vehicle C1 is suddenly wiped with a sponge (or cloth) without any water being poured on it. This can be determined based on the vehicle exterior image acquired by the vehicle exterior image acquisition unit 102. In such a case, it is also possible to have the agent image 200 output voice information such as "Please be careful when wiping as water has not been poured on it yet." (2) The user U1 is wiping quickly. In other words, if the user U1's hand is moving quickly, it is estimated that the user U1 is pressing hard on the area to be wiped, which is likely to damage the area. (3) The wiping action is performed randomly (i.e., roughly). In other words, if the wiping action is not performed in a predetermined order (e.g., left / right / up / down), it can be determined to be inappropriate. (4) When the number of times the same target part is wiped is equal to or greater than a predetermined value. In other words, when the number of times the part is wiped is too many (i.e., when the part is polished too much), it can be determined that there is a high possibility that the body of the vehicle C1 will be damaged.

[0096] As shown in Fig. 10(A), if the maintenance action of user U1 is determined to be inappropriate, it is possible to execute an effect in which the agent image 200 appears in a bad mood and outputs voice information S8 such as "It hurts a little, please be gentle." On the other hand, as shown in Fig. 10(B), if the maintenance action of user U1 is determined to be appropriate, it is possible to execute an effect in which the agent image 200 appears in a good mood (an expression suggesting that the maintenance action is appropriate) (for example, dancing) and outputs voice information S9 such as "Thank you for wiping it nicely."

[0097] [Example of estimation and agent output for washing a vehicle body using a car wash machine] Next, an example of estimating the behavior of a user U1 who is washing the body of a vehicle C1 using a car wash machine CW1 will be described. The car wash machine CW1 is, for example, a device capable of drive-through car washing. When washing the body of the vehicle C1 using the car wash machine CW1, the user U1's work may include operating the car wash machine CW1. Therefore, below, a method for estimating the behavior of a user U1 who is washing the body of the vehicle C1 using the car wash machine CW1 will be described. Here, the description will be made assuming that the car wash machine CW1 is an external device 400 (see FIG. 3).

[0098] [Method for estimating cleaning behavior of a vehicle body using a car wash machine] When using the car wash machine CW1 to wash the body of the vehicle C1, the user U1 operates the operation unit 440 to set the desired settings. In this case, the control unit 420 of the car wash machine CW1 can transmit operation information regarding the operation content of the user U1 received by the operation unit 440 to the information processing device 110 via the communication unit 410 and the network 20. For example, various information can be directly exchanged between the car wash machine CW1 and the information processing device 110 using short-range wireless communication (e.g., Wi-Fi, Bluetooth (registered trademark)). In addition, the car wash machine CW1 (or a management server) can manage identification information (e.g., membership information) that can identify the vehicle C1. In this case, the control unit 420 of the car wash machine CW1 can acquire location information of the vehicle C1 and start exchanging various information between the car wash machine CW1 and the information processing device 110 on the condition that the location information of the vehicle C1 matches the location of the car wash machine CW1.

[0099] In this way, when various information is exchanged between the car wash machine CW1 and the information processing device 110, the estimation unit 127 of the information processing device 110 can estimate that the user U1 will use the car wash machine CW1 to wash the body of the car, based on the operation information acquired from the car wash machine CW1. Note that, although this example shows an example in which the operation information is transmitted from the car wash machine CW1 to the information processing device 110, the operation information may also be transmitted from another device to the information processing device 110. For example, if there is a server that manages one or more car washes, each car wash can transmit the received operation information to the server. In this case, the server can transmit the operation information received from each car wash to the vehicle C1.

[0100] Furthermore, for example, based on the car wash CW1 included in the vehicle exterior image acquired by the vehicle exterior image acquisition unit 102 and the position information of the vehicle C1 acquired by the sensors 104 (e.g., the position information acquisition unit), it is possible to estimate that the user U1 will wash the body of the car using the car wash CW1. For example, if the position information of the vehicle C1 acquired by the position information acquisition unit matches the position of the car wash CW1, the vehicle C1 has been parked at that position for a predetermined period of time or more, and the vehicle exterior image acquired by the vehicle exterior image acquisition unit 102 includes the exterior of the car wash CW1, it is possible to estimate that the user U1 will wash the body of the car using the car wash CW1. Note that, as described above, it is possible to estimate that the user U1 will wash the body of the car using the car wash CW1 based on detection based on sound, detection based on a comparison result between an image before cleaning and an image during cleaning, etc.

[0101] FIG. 11 is a simplified diagram illustrating a case in which a user U1 uses a car wash machine CW1 to wash the body of a vehicle C1. FIG. 11 illustrates an example in which the user U1 waits outside the vehicle C1 while the vehicle C1 is being washed. In this case, the information processing device 110 receives operation information from the car wash machine CW1 and transmits, to the car wash machine CW1, agent output information that causes the car wash machine CW1 to output agent information in accordance with the operation information. Upon receiving the agent output information, the control unit 420 of the car wash machine CW1 displays an agent image 200 on the display unit 431 based on the agent output information, and also outputs audio information S10 related to the agent from the sound output unit 432. The agent image 200 displayed on the display unit 431 may be based on the agent output information transmitted from the information processing device 110, or may be an agent image preset by the car wash machine CW1.

[0102] 12 is a simplified diagram showing a case where a user U1 uses a car wash machine CW1 to wash the body of a vehicle C1. FIG. 12 shows an example where the user U1 waits inside the vehicle C1 while the vehicle C1 is being washed. In this case, the user U1 cannot see the display unit 431 of the car wash machine CW1 while the vehicle C1 is being washed. Therefore, while the vehicle C1 is being washed, the information processing device 110 receives operation information from the car wash machine CW1 and, in accordance with the operation information, displays an agent image 200 on the display unit 310 and outputs audio information S11 related to the agent from the sound output unit 320.

[0103] Here, an example of communication between the information processing device 110 and the car wash CW1 when the user U1 waits inside the vehicle C1 while the vehicle C1 is being washed, and an example of the relationship between the actions of the user U1 and the responses of the agent will be described.

[0104] First, after parking the vehicle C1 at a predetermined location, the user U1 performs a predetermined operation at the car wash machine CW1. Examples of the predetermined operation include paying the car wash fee and setting the car wash. When this predetermined operation is performed, predetermined information is exchanged between the information processing device 110 and the car wash machine CW1. After performing this predetermined operation, the user U1 returns to the seat of the vehicle C1. While the car wash machine CW1 is performing the car wash operation, the car wash machine CW1 may sequentially transmit various pieces of information related to the car wash, or may transmit preset scheduled time information (e.g., the car wash process) to the information processing device 110 and display the agent image 200 or the like based on the scheduled time information.

[0105] When predetermined information is exchanged between the information processing device 110 and the car wash machine CW1, the output control unit 128 displays an agent image 200 expressing gratitude on the display unit 310 and outputs audio information such as "Thank you for washing my car" from the sound output unit 320.

[0106] Next, when the car wash machine CW1 moves to the designated start position for the car wash, the output control unit 128 displays an agent image 200 expressing excitement on the display unit 310, and outputs audio information such as "I'm excited" from the sound output unit 320.

[0107] Similarly, each time a car wash operation such as rinsing, shampooing, or drying is performed, the output control unit 128 displays the agent image 200 expressing the content corresponding to the car wash operation on the display unit 310, and outputs audio information expressing the content from the sound output unit 320. In this way, each time a car wash operation is performed by the car wash machine CW1, a different agent's reaction is expressed.

[0108] Next, when the car wash operation of the vehicle C1 by the car wash machine CW1 is completed, the output control unit 128 displays an agent image 200 expressing gratitude, thanks, etc. on the display unit 310, and outputs audio information such as "Thank you for your hard work" and "Thank you very much" from the sound output unit 320.

[0109] In the above, an example has been shown in which the car wash machine CW1 acts as a master and transmits various command information to the information processing device 110, and the information processing device 110 executes display of an agent image based on the various command information (e.g., car wash process), but the present invention is not limited to this. For example, the information processing device 110 may act as a master and transmit various command information to the car wash machine CW1, and the car wash machine CW1 may execute display of an agent image based on the various command information.

[0110] [Example of determining whether maintenance operations performed when washing a vehicle using a car wash machine are appropriate or inappropriate] When the car wash machine CW1 is used to wash the body of the vehicle C1, the car wash machine CW1 performs the car wash operation in response to the operation of the user U1. During the car wash, the user U1 waits around the car wash machine CW1 or inside the car. Therefore, since the user U1 does not directly perform maintenance operations during the car wash, it is difficult to determine whether the maintenance operations are appropriate or inappropriate. Therefore, for example, it is possible to determine the appropriateness of the car wash operation based on various actions of the user U1 while the car is being washed by the car wash machine CW1. For example, if the user U1 waiting inside the car is detected to open a door or a window, the user U1's actions during the car wash can be determined to be inappropriate. Furthermore, for example, if the user U1 waiting around the car wash machine CW1 is detected to get too close to the car wash machine CW1 or to perform unnecessary operations, the user U1's actions during the car wash can be determined to be inappropriate. Furthermore, for example, the level of appropriateness of the user U1's actions can be determined based on the car wash course selected by the car wash machine CW1 in response to the user U1's operation. For example, it is assumed that a water wash course, a shampoo wash course, a wax course, etc. can be set. Note that the level of the car wash course increases in the order of water wash course, shampoo wash course, and wax course, and the car wash cost also increases. For example, if the water wash course is selected, the level of appropriateness of the user U1's actions can be determined to be a relatively low level; if the shampoo wash course is selected, the level of appropriateness of the user U1's actions can be determined to be a medium level; and if the wax course is selected, the level of appropriateness of the user U1's actions can be determined to be a relatively high level. Furthermore, the display mode of the agent image 200 and the output mode of the voice of the agent image 200 can be changed depending on the level of appropriateness of the user U1's actions.

[0111] [Examples of estimations and agent output for replacing deteriorated items, replacing or replenishing items that are in short supply] Next, a detection example of detecting the actions of user U1 when replacing a deteriorated item in vehicle C1 or replacing or replenishing a missing item will be described. Examples of objects to be replaced when a deteriorated item is required in vehicle C1 include wipers, tires, and headlights. Examples of objects to be replaced when a shortage occurs in vehicle C1 include windshield washers, coolant, and tire air. For example, the actions of user U1 replacing wipers, tires, and headlights can be detected based on the actions of user U1 included in the acquired vehicle exterior image and the objects to be replaced (e.g., wipers, tires, and headlights) included in the vehicle exterior image. For example, if a wiper is included in the vehicle exterior image as the object to be replaced and user U1 is holding the wiper near the bottom of the windshield 4, it can be estimated that user U1 is replacing the wiper.

[0112] Also, for example, if a headlight goes out, an error signal is output to the CAN signal. Therefore, if this error signal is output to the CAN signal, a headlight is included as an object to be replaced in the vehicle exterior image, and the user U1 is holding the headlight near the front of the vehicle C1, it can be assumed that the user U1 is replacing the headlight. It is assumed that when the hood 6 is closed, the engine compartment 7 is dark, making it difficult for the engine compartment image acquisition unit 103 to acquire an appropriate image. Therefore, the engine compartment image acquisition unit 103 can be activated on the condition that the engine compartment 7 is opened. It is also possible to activate the engine compartment image acquisition unit 103 and acquire and use an image of the engine compartment 7 even when the hood 6 is closed.

[0113] Furthermore, for example, the action of user U1 replacing or refilling a windshield washer or coolant can be detected based on the action of user U1 included in the engine room image acquired by the engine room image acquisition unit 103 (see FIG. 2) and the object to be replaced or refilled (for example, windshield washer, coolant) included in the engine room image. For example, if a windshield washer container is included in the engine room image as the object to be refilled and user U1 is holding the windshield washer container near the engine room 7 (see FIG. 2), it can be estimated that user U1 is refilling the windshield washer.

[0114] Here, the remaining amounts of windshield washer, coolant, etc. are displayed on meters and are output to the CAN signal. Therefore, the status of the windshield washer, coolant, etc. can be obtained by acquiring the CAN signal. Furthermore, when the hood 6 is opened, the fact that the hood 6 has been opened is output to the CAN signal. Therefore, for example, if the CAN signal detects that the hood 6 has been opened and the remaining amount of windshield washer has increased, it can be assumed that the windshield washer has been refilled. Furthermore, for example, it can be assumed that user U1 is refilling the windshield washer on the condition that the remaining amount of windshield washer was equal to or less than a threshold and that the above-mentioned determination processes determine that user U1 is refilling the windshield washer. The refill status of the windshield washer can also be obtained based on the remaining amount of windshield washer. The same can be estimated for coolant.

[0115] Furthermore, for example, the action of user U1 refilling air into a tire can be detected based on the action of user U1 included in the acquired vehicle exterior image and the tire included in the vehicle exterior image. For example, if a device such as an air pump is included in the vehicle exterior image and user U1 is holding the device near one of the tires, it can be assumed that user U1 is refilling air into a tire. Here, tire pressure can be constantly acquired by a tire pressure sensor. Therefore, it may be officially assumed that user U1 is refilling air into a tire if the tire pressure acquired by the tire pressure sensor is equal to or less than a threshold and the above-mentioned determination process determines that user U1 is refilling air into a tire.

[0116] As described above, known detection methods can be used to detect each object. Known detection methods can be used to detect the actions of the user U1 using each object to perform maintenance on the vehicle C1. It is possible to estimate that the user U1 is performing maintenance on the vehicle C1 based on sounds outside the vehicle acquired by the sound acquisition unit 105. For example, if the sound of tire inflation is continuously detected for a predetermined period of time, it is possible to estimate that the user U1 is refilling the tire pressure of the vehicle C1. It is also possible to perform estimation by combining at least two of image-based detection, sound-based detection, and sensor-based detection (for example, detection using sensor values ​​output to a CAN signal from an existing sensor of the vehicle C1).

[0117] [Example of determining whether maintenance actions performed during a vehicle wash were appropriate] Next, a method for determining whether the maintenance operation of the user U1 is appropriate when the user U1 replaces deteriorated items in the vehicle C1 or replaces or replenishes items that are in short supply will be described.

[0118] For example, based on the appropriate action determination information stored in the appropriate action information DB 350 and the actions of the user U1 included in the vehicle exterior image (or engine room image) acquired by the vehicle exterior image acquisition unit 102 (or the engine room image acquisition unit 103), it is possible to determine whether the maintenance action performed by the user U1 to replace or replenish the vehicle C1 is appropriate. Specifically, the output control unit 128 acquires appropriate action determination information related to the replacement or replenishment of the vehicle C1 from the appropriate action information DB 350. Then, based on the degree of match between each user's action included in the appropriate action determination information related to the replacement or replenishment of the vehicle C1 and the action of the user U1 included in the vehicle exterior image (or engine room image), the output control unit 128 can determine whether the maintenance action performed by the user U1 to replace or replenish the vehicle C1 is appropriate. For example, as the degree of match increases, it is possible to increase the evaluation value for determining that the maintenance action performed by the user U1 to replace or replenish the vehicle C1 is appropriate. Then, based on the evaluation value, the output control unit 128 determines the display mode of the agent image 200 and the output content of the audio information.

[0119] Furthermore, for example, based on the appropriate action determination information stored in the appropriate action information DB 350 and the action of the user U1 included in the vehicle exterior image (or engine room image) acquired by the vehicle exterior image acquisition unit 102 (or the engine room image acquisition unit 103), it is possible to determine whether the procedure by which the user U1 replaces or replenishes the vehicle C1 is appropriate. Specifically, the output control unit 128 acquires appropriate action determination information related to the replacement or replenishment of the vehicle C1 from the appropriate action information DB 350. Then, based on the degree of agreement between each user's action procedure included in the appropriate action determination information related to the replacement or replenishment of the vehicle C1 and the action procedure of the user U1 included in the vehicle exterior image (or engine room image), the output control unit 128 can determine whether the procedure by which the user U1 replaces or replenishes the vehicle C1 is appropriate. For example, as the degree of agreement increases, it is possible to increase the evaluation value for determining that the procedure by which the user U1 replaces or replenishes the vehicle C1 is appropriate. Then, based on the evaluation value, the output control unit 128 determines the display mode of the agent image 200 and the output content of the audio information.

[0120] For example, if it is estimated that user U1 is replacing the windshield washer, it is possible to display the agent image 200 in a position where user U1 can see (for example, the front window 4) and output audio information such as "It smells nice." Also, if it is detected that user U1 has completed replacing the windshield washer, it is possible to display the agent image 200 in a position where user U1 can see (for example, the front window 4) and output audio information such as "Thank you for replacing the washer fluid" or "Be careful not to add too much." Also, for example, a display unit may be provided on the underside of the hood 6, and the agent image 200 may be displayed on this display unit to realize a specific effect, or the agent image 200 may be projected onto the underside of the hood 6 to realize a specific effect of the agent image 200.

[0121] [Example of estimation and agent output when refueling a vehicle] Next, an example of estimating the behavior of a user U1 who uses a fuel dispenser TA1 to refuel a vehicle C1 will be described. Note that, although the example will be described using the fuel dispenser TA1, if the vehicle C1 is an electric vehicle, the example can also be applied to a case where a charging facility is used instead of the fuel dispenser TA1.

[0122] When refueling a vehicle C1 using a refueling machine TA1, the user U1's task may include operating the refueling machine TA1. Hereinafter, a method for estimating the refueling task when refueling a vehicle C1 using the refueling machine TA1 will be described. Here, the description will be made assuming that the refueling machine TA1 is an external device 400 (see FIG. 3).

[0123] [Method for estimating vehicle refueling behavior using a fuel pump] When refueling the vehicle C1 using the fuel dispenser TA1, the user U1 operates the operation unit 440 to make desired settings. In this case, the control unit 420 of the fuel dispenser TA1 can transmit operation information regarding the operation content of the user U1 received by the operation unit 440 via the communication unit 410 and the network 20 to the information processing device 110. For example, as in the case of the car wash machine CW1, various pieces of information can be exchanged between the fuel dispenser TA1 and the information processing device 110 by direct communication using short-range wireless communication, communication using position information, etc.

[0124] In this case, the estimation unit 127 of the information processing device 110 can estimate, based on the operation information, that the user U1 will refuel using the refueling aircraft TA1. Note that, although this example shows an example in which operation information is transmitted from the refueling aircraft TA1 to the information processing device 110, operation information may also be transmitted from another device to the information processing device 110. For example, if there is a server that manages one or more refueling aircraft, each refueling aircraft can transmit the received operation information to that server. In this case, the server can transmit the operation information received from each refueling aircraft to the vehicle C1.

[0125] Furthermore, for example, based on the refueling machine TA1 included in the vehicle exterior image acquired by the vehicle exterior image acquisition unit 102 and the position information of the vehicle C1 acquired by the sensors 104 (e.g., the position information acquisition unit), it is possible to estimate that the user U1 will refuel using the refueling machine TA1. For example, if the position information of the vehicle C1 acquired by the position information acquisition unit matches the position of the refueling machine TA1, and the vehicle C1 has been stopped at that position for a predetermined period of time or more, and the vehicle exterior image acquired by the vehicle exterior image acquisition unit 102 is detected to include the appearance of the refueling machine TA1, it is possible to estimate that the user U1 will refuel using the refueling machine TA1. Furthermore, if a sensor that detects the opening and closing of the refueling cap detects that the refueling cap has been opened and a fuel level sensor detects that the fuel has increased, it is possible to estimate that the user U1 will refuel using the refueling machine TA1. It is also possible to estimate that the user U1 will refuel using the refueling machine TA1 based on detection based on the refueling sound, detection based on a comparison of the remaining fuel amount, etc.

[0126] FIG. 13 is a simplified diagram illustrating a case in which a user U1 uses a refueling machine TA1 to refuel a vehicle C1. FIG. 13 illustrates an example in which the user U1 refuels the vehicle C1 himself. In this case, the information processing device 110 receives operation information from the refueling machine TA1 and transmits agent output information to the refueling machine TA1 in accordance with the operation information, causing the refueling machine TA1 to output agent information. Upon receiving the agent output information, the control unit 420 of the refueling machine TA1 displays an agent image 200 on the display unit 431 based on the agent output information, and also outputs audio information S12 related to the agent from the sound output unit 432. The agent image 200 displayed on the display unit 431 may be based on the agent output information transmitted from the information processing device 110, or may be an agent image preset on the refueling machine TA1 side.

[0127] For example, after the user U1 starts operating the refueling machine TA1, communication between the information processing device 110 and the refueling machine TA1 can be continuously performed. For example, before refueling begins, an agent image 200 is displayed on some part of the vehicle C1 (for example, the window glass behind the driver's seat of the vehicle C1). Then, when refueling begins, an effect is executed in which the agent image 200 moves from the vehicle C1 to the refueling machine TA1. That is, an effect is executed in which the agent image 200 displayed on the vehicle C1 moves toward the refueling machine TA1, and after the effect of the movement, the agent image 200 is displayed on the display unit 431 of the refueling machine TA1. In this case, an agent image may be displayed in the same display mode as the agent image 200 displayed on the vehicle C1, or a different agent image may be displayed depending on the refueling machine TA1. FIG. 13 shows an example in which an agent image 200 resembling a panda is displayed. FIG. 13 also shows an example in which the agent image 200 resembling a panda acts as if it is drinking a drink during refueling. Then, when refueling is completed, voice information S12 such as "Ah, I'm full, it was cheap and delicious. Thank you for treating me" is output. Note that the agent image 200 may be displayed on the front window 4 or a window glass near the fuel filler opening.

[0128] [Example of determining whether maintenance operations performed when refueling a vehicle using a fuel pump are appropriate or inappropriate] When a user U1 uses a refueling machine TA1 to refuel a vehicle C1, the refueling machine TA1 performs a refueling operation in response to the user U1's operation. During this refueling, the user U1 performs the refueling operation around the refueling machine TA1. Here, because the refueling operation is determined according to a predetermined procedure, it is difficult to determine whether the maintenance operation performed by the user U1 during refueling is appropriate or inappropriate. Therefore, for example, it is possible to determine whether the operation is appropriate or inappropriate based on various actions performed by the user U1 during refueling using the refueling machine TA1. For example, if it is detected that the user U1 tries to move away from the refueling machine TA1 during refueling, that the user U1 fills the tank to the brim, or that the user U1 performs an unnecessary operation, it is possible to determine the user U1's actions during refueling as inappropriate. Furthermore, for example, the level of appropriateness of the user U1's actions may be determined based on the amount of fuel selected by the refueling machine TA1 in response to the user U1's operation. For example, if a relatively large amount of fuel is selected, the level of appropriateness of the user U1's actions can be determined to be relatively high, and if a relatively small amount of fuel is selected, the level of appropriateness of the user U1's actions can be determined to be relatively high. Also, the display mode of the agent image 200 and the output mode of the voice of the agent image 200 can be changed depending on the level of appropriateness of the user U1's actions.

[0129] [Maintenance information display example] Fig. 14 is a diagram showing an example of outputting information (maintenance information) relating to maintenance being performed by user U1. Fig. 14 shows an example in which a vehicle image C2 of a vehicle C1 viewed from above is displayed on the display unit 310 together with an agent image 200.

[0130] As described above, during maintenance, the type of maintenance and the maintenance actions (actions performed by user U1) related to that maintenance can be grasped. Therefore, in this embodiment, an example is shown in which an image corresponding to the type of maintenance being performed and the progress status of the maintenance being performed are provided to user U1. FIG. 12 shows an example in which, when a vehicle C1 is washed as maintenance, a vehicle image C2 corresponding to the type of maintenance being performed (car wash) and the progress status of the maintenance being performed (maintenance status displays MP1 to MP7) are provided to user U1.

[0131] The maintenance status displays MP1 to MP7 are images that present the user U1 with a visual distinction between areas that have been maintained and areas that have not yet been maintained. For example, areas that have been washed can be displayed in a first color (e.g., green or white), and areas that have not been washed can be displayed in a second color (e.g., orange or gray) different from the first color. Whether each area has been washed can be determined based on a comparison between a past image and a current image. The display mode of the agent image 200 and the output of audio information can be sequentially changed depending on the progress of the ongoing maintenance. For example, when more than half of the car has been washed, a smiling agent image 200 can be displayed, and audio information S12 such as "You're almost done washing, keep up the good work!" can be output from the audio output unit 320. As described above, the display mode of the agent image 200 and the output of audio information can be appropriately changed based on the appropriateness of the maintenance.

[0132] As described above, when determining whether the maintenance action of user U1 is appropriate or inappropriate, the appropriateness may be determined directly, or determination level information indicating the level of appropriateness may be calculated and the appropriateness may be determined based on the result of comparing this determination level information with a threshold. This determination level information is, for example, an evaluation value indicating the level of appropriateness and is information stored in the appropriateness determination level 156 (see FIG. 4). For example, if the maintenance action of user U1 is determined to be inappropriate, a low value is calculated as the determination level information. Furthermore, if it is not possible to determine whether the maintenance action of user U1 is appropriate or inappropriate, calculation of the determination level information may not be performed (for example, information indicating that determination is impossible may be output), or medium determination level information (for example, a value of approximately 50%) may be output. For example, if a value corresponding to the determination level information is equal to or greater than a threshold (for example, a value of approximately 50%), the maintenance action may be determined to be appropriate, and if the value corresponding to the determination level information is less than the threshold, the maintenance action may be determined to be inappropriate.

[0133] Furthermore, when determining whether a maintenance action of user U1 is appropriate or inappropriate, one determination of appropriateness may be made for one maintenance action, or one determination of appropriateness may be made based on multiple actions that make up one maintenance action. Similarly, when calculating determination level information that indicates the level of appropriateness of a maintenance action of user U1, one determination level information may be calculated for one maintenance action, or one determination level information may be calculated based on multiple actions that make up one maintenance action.

[0134] For example, when making a judgment on whether a single maintenance operation is appropriate or inappropriate, or calculating a single piece of judgment level information, it is possible to calculate a value indicating the appropriateness of each operation (e.g., an evaluation value indicating the level of appropriateness), perform a predetermined calculation process on each of these values, and make a judgment on whether a single operation is appropriate or inappropriate and calculate a single piece of judgment level information based on the calculation results. For example, it is possible to perform an addition process that adds up the values ​​indicating the appropriateness of each operation (e.g., evaluation values ​​indicating the level of appropriateness), and make a judgment on whether a single operation is appropriate or inappropriate and calculate a single piece of judgment level information based on the total value that is the result of the addition.

[0135] Furthermore, when multiple operations are performed during the execution of one maintenance operation, it is possible that a predetermined number (e.g., one or two) of the multiple operations may be determined to be inappropriate, while the remaining operations may be determined to be appropriate. In this case, if the number of operations determined to be appropriate is large, the operations determined to be inappropriate may be offset, resulting in a high final judgment level information. Therefore, in such a case, the weight of the operations determined to be inappropriate may be increased and calculation processing (e.g., weighted averaging processing) may be performed, and the judgment level information for one maintenance operation may be calculated based on the calculation results.

[0136] Although the above describes an example in which the appropriateness of the maintenance operation of user U1 is determined within the information processing device 110, the appropriateness of the maintenance operation of user U1 may be determined using a device other than the information processing device 110. For example, the information processing device 110 may acquire information about maintenance available via a predetermined network and use the information for the determination. Furthermore, for example, information about maintenance (e.g., the type of maintenance and operation information of user U1 (e.g., car washing procedure)) may be transmitted to a device available via the predetermined network, and the device may determine whether the maintenance operation of user U1 is appropriate or inappropriate, and the information processing device 110 may acquire the determination result from the device.

[0137] Furthermore, artificial intelligence (AI) may be used to determine whether user U1's maintenance actions are appropriate. For example, it is possible to learn the actions of many users during maintenance of various vehicles and the evaluation of each action (e.g., the level of appropriateness), and use the learning results to determine whether user U1's maintenance actions during maintenance of vehicle C1 are appropriate. For example, information about the maintenance (e.g., the type of maintenance and user U1's action information) may be sent to an available AI device via a predetermined network, and the AI ​​device may determine whether user U1's maintenance actions are appropriate or inappropriate. The information processing device 110 may then obtain the determination result from the AI ​​device.

[0138] [Example of information processing system operation] Figure 15 is a flowchart showing an example of specific performance control processing in the information processing system 100. This specific performance control processing is executed by the control unit 120 (see Figure 3) based on a program stored in the storage unit 130 (see Figure 3). This specific performance control processing is executed for each control cycle. This specific performance control processing will be explained with appropriate reference to Figures 1 to 14.

[0139] In step S501, the detection unit 126 detects the actions of the user U1 of the vehicle C1 based on the information output from each unit. That is, the detection unit 126 acquires information output from the vehicle interior image acquisition unit 101, the vehicle exterior image acquisition unit 102, the engine room image acquisition unit 103, the sensors 104, the sound acquisition unit 105, the vehicle state acquisition unit 125, and the communication unit 140 as action information related to the actions of the user U1. Then, the detection unit 126 detects the actions of the user U1 of the vehicle C1 based on the action information. For example, the detection unit 126 detects actions such as washing the body of the vehicle C1, vacuuming up dirt inside the vehicle C1, and wiping the inside of the vehicle C1 with a cloth. The method for detecting the actions of the user U1 is the same as the above-mentioned detection methods.

[0140] In step S502, the estimation unit 127 determines whether the action of user U1 detected in step S501 is an action related to maintenance of vehicle C1. If the action of user U1 is an action related to maintenance of vehicle C1, the estimation unit 127 estimates the type of maintenance of vehicle C1 performed by the action of user U1 detected in step S501. Note that these determination and estimation methods are the same as the determination and estimation methods described above. If the action of user U1 detected in step S501 is an action related to maintenance of vehicle C1, the process proceeds to step S503. On the other hand, if the action of user U1 detected in step S501 is not an action related to maintenance of vehicle C1, the operation of the specific performance control process is terminated.

[0141] In step S503, the output control unit 128 performs control to execute a specific effect during the maintenance operation of the user U1 based on the operation information acquired during the maintenance operation estimated in step S502 and the estimated type of maintenance. As this specific effect, for example, as shown in Figures 7 to 13, it is possible to execute an effect using the agent image 200 displayed on the display unit 310, an effect using sound output from the sound output unit 320, an effect using images and sound output from other devices, etc.

[0142] In this way, in this embodiment, by having the agent respond according to the maintenance actions of user U1 while maintenance is being performed on vehicle C1, it is possible to increase user U1's attachment to vehicle C1 and increase the motivation of user U1 to actively perform maintenance.

[0143] [Example of information processing system operation] Fig. 16 is a flowchart showing an example of the specific performance control process in the information processing system 100. Note that this specific performance control process is a modified example of Fig. 15 and corresponds to Figs. 7 and 8. That is, Fig. 16 shows an example of the specific performance control process when user U1 cleans the interior of vehicle C1. Note that this specific performance control process will be explained with appropriate reference to Figs. 1 to 15.

[0144] In step S511, the detection unit 126 detects the movement of the user U1 of the vehicle C1 based on the information output from each unit. This process corresponds to step S501 shown in FIG.

[0145] In step S512, the estimation unit 127 determines whether the movement of the user U1 detected in step S511 is related to cleaning the interior of the vehicle C1. That is, the estimation unit 127 estimates the type of maintenance of the vehicle C1 performed by the movement of the user U1 detected in step S511. This process corresponds to step S502 shown in FIG. 15. If the movement of the user U1 detected in step S511 is related to cleaning the interior of the vehicle C1, the process proceeds to step S513. On the other hand, if the movement of the user U1 detected in step S511 is not related to cleaning the interior of the vehicle C1, the operation of the specific performance control process is terminated. Note that FIG. 16 illustrates the specific performance control process when the user U1 cleans the interior of the vehicle C1. If the user U1 performs other maintenance on the vehicle C1, the processes of FIGS. 15, 17, etc. are executed.

[0146] Steps S513 to S522 correspond to step S503 shown in Fig. 15. That is, Fig. 16 shows more detailed processing in the case where the user U1 cleans the interior of the vehicle C1 in the processing shown in Fig. 15.

[0147] In step S513, the output control unit 128 determines whether the behavior (maintenance behavior) of the user U1 detected in step S511 is appropriate for cleaning the interior of the vehicle C1. The above-described methods can be used to determine the appropriateness of the ongoing maintenance behavior. If the maintenance behavior of the user U1 for cleaning the interior of the vehicle C1 is appropriate, the process proceeds to step S514. On the other hand, if the maintenance behavior of the user U1 for cleaning the interior of the vehicle C1 is inappropriate, the process proceeds to step S522. Note that FIG. 16 illustrates an example in which, after the interior cleaning of the vehicle C1 is estimated, the process determines whether the maintenance behavior of the user U1 currently performing for the interior cleaning is appropriate in step S513. However, this is not limiting. For example, after the determinations in steps S514, S516, and S518, the process may determine whether the determined maintenance behavior of the user U1 for the specific interior cleaning is appropriate. The same applies to the example shown in FIG. 17.

[0148] In step S514, the output control unit 128 determines whether the user U1 is cleaning the interior of the vehicle using a vacuum cleaner. This determination can be made using any of the determination methods described above. For example, it is determined whether the user U1 is using a high-pressure vacuum cleaner to suck up dirt inside the vehicle C1. If the user U1 is cleaning the interior of the vehicle using a vacuum cleaner, the process proceeds to step S515. On the other hand, if the user U1 is not cleaning the interior of the vehicle using a vacuum cleaner, the process proceeds to step S516.

[0149] In step S515, since it has been determined in step S513 that the maintenance operation of user U1 regarding the interior cleaning of vehicle C1 is appropriate, the output control unit 128 causes the agent to perform an effect showing that the agent is in a good mood (an effect of feeling good), an effect expressing that the maintenance operation of user U1 is appropriate, etc. For example, as shown in Fig. 7, it is possible to make the agent image 200 displayed on the display unit 310 have an expression of a good mood, and to output audio information conveying that the maintenance operation of user U1 is appropriate.

[0150] In step S516, the output control unit 128 determines whether the user U1 is cleaning the interior of the vehicle using a predetermined liquid. This determination can be made using any of the determination methods described above. For example, it is determined whether the user U1 is spraying disinfectant alcohol on the inside of the vehicle C1. If the user U1 is cleaning the interior of the vehicle using a predetermined liquid, the process proceeds to step S517. On the other hand, if the user U1 is not cleaning the interior of the vehicle using a predetermined liquid, the process proceeds to step S518.

[0151] In step S517, since it has been determined in step S513 that the maintenance operation of user U1 regarding the interior cleaning of vehicle C1 is appropriate, the output control unit 128 causes the agent to perform an effect that indicates that the agent is in a good mood (an effect of feeling good, an effect of expressing joy), an effect that expresses that the maintenance operation of user U1 is appropriate, etc. For example, as shown in Fig. 8(B), it is possible to display the agent image 200 displayed on the display unit 310 with a good-humored expression, and to output audio information that conveys that the maintenance operation of user U1 is appropriate.

[0152] In step S518, the output control unit 128 determines whether the user U1 is cleaning the interior of the vehicle by wiping it with their hands. This determination can be made using any of the determination methods described above. For example, it is determined whether the user U1 is wiping the interior of the vehicle C1 with a cloth. If the user U1 is cleaning the interior of the vehicle by wiping it with their hands, the process proceeds to step S519. On the other hand, if the user U1 is not cleaning the interior of the vehicle by wiping it with their hands, the process proceeds to step S520.

[0153] In step S519, since it was determined in step S513 that the maintenance operation of user U1 regarding the interior cleaning of vehicle C1 was appropriate, the output control unit 128 causes the agent to execute a specific effect, such as an effect showing that the agent is in a good mood (an effect showing a good mood, an effect showing fun), an effect showing that user U1's maintenance operation is appropriate, etc. For example, as shown in FIG. 8(B), it is possible to display the agent image 200 displayed on the display unit 310 with a good-humored expression and output audio information conveying that user U1's maintenance operation is appropriate. In addition, an effect may be executed in which the agent image 200 dances or shakes its head.

[0154] In the determination process in steps S514, S516, and S518, an example of determining a typical interior cleaning is shown, but the present invention is not limited to this. For example, it may be possible to determine whether other interior cleanings have been performed, and then execute a specific performance by the agent based on the determination result.

[0155] Furthermore, the processing in steps S515, S517, and S519 shows an example of a specific effect, but is not limited to this. For example, other specific effects may be executed by the agent as appropriate based on the determination processing in steps S514, S516, and S518. Furthermore, as described above, if it is possible to determine an evaluation value for the appropriateness of user U1's maintenance operation, the specific effect may be changed as appropriate based on the magnitude of that evaluation value.

[0156] In step S520, the estimation unit 127 determines whether the user U1 has finished cleaning the interior of the vehicle C1 based on the movement of the user U1 detected in step S511. For example, after the movement of the user U1 is estimated in step S512 to be an operation related to cleaning the interior of the vehicle C1, it can be determined that the user U1 has finished cleaning the interior of the vehicle C1 on the condition that the door of the vehicle C1 is closed and the user U1 has left the vehicle C1. In this case, it can be determined that the user U1 has finished cleaning the interior of the vehicle C1 on the condition that a certain amount of time has passed since the user U1 left the vehicle C1 and there has been no movement of people or objects inside or outside the vehicle C1 after the user U1 left the vehicle C1. Also, for example, after the movement of the user U1 is estimated in step S512 to be an operation related to cleaning the interior of the vehicle C1, it can be determined that the user U1 has finished cleaning the interior of the vehicle C1 on the condition that all preset operations related to cleaning the interior of the vehicle C1 (e.g., collecting trash and wiping) have been completed. If the user U1 has finished cleaning the interior of the vehicle C1, the process proceeds to step S521. On the other hand, if the user U1 has not finished cleaning the interior of the vehicle, the operation of the specific performance control process is terminated.

[0157] In step S521, since it was determined in step S520 that the user U1 has finished cleaning the interior of the vehicle, the output control unit 128 causes the agent to execute a performance or the like to express gratitude for the interior cleaning. For example, it is possible to display an agent image 200 as shown in FIG. 8(B) on the display unit 310 and output audio information conveying gratitude for the interior cleaning. In this way, when the user U1 has finished cleaning the interior of the vehicle, it is possible to convey gratitude to the user U1 by causing the agent to execute a performance or the like to express gratitude for the interior cleaning, regardless of whether the maintenance operation being performed was appropriate or inappropriate.

[0158] In step S522, since it was determined in step S513 that the maintenance operation by user U1 regarding the interior cleaning of vehicle C1 was inappropriate, the output control unit 128 causes the agent to execute an effect that indicates the agent is in a bad mood (an effect of feeling unwell), an effect that expresses that user U1's maintenance operation is inappropriate, or the like. For example, as shown in FIG. 8(C), it is possible to display the agent image 200 displayed on the display unit 310 with a bad-tempered expression and output audio information instructing the user U1 on the correct operation. In this way, it is possible to notify user U1 of the correct operation regarding the maintenance being performed.

[0159] [Example of information processing system operation] Fig. 17 is a flowchart showing an example of the specific performance control process in the information processing system 100. Note that this specific performance control process is a modified example of Fig. 16 and corresponds to Figs. 9 and 10. That is, Fig. 17 shows an example of the specific performance control process when user U1 washes vehicle C1. Note that this specific performance control process will be explained with appropriate reference to Figs. 1 to 16.

[0160] In step S531, the detection unit 126 detects the movement of the user U1 of the vehicle C1 based on the information output from each unit. This process corresponds to step S501 shown in FIG.

[0161] In step S532, the estimation unit 127 determines whether the action of user U1 detected in step S531 is an action related to washing the vehicle C1. That is, the estimation unit 127 estimates the type of maintenance of the vehicle C1 performed by the action of user U1 detected in step S531. This process corresponds to step S512 shown in FIG. 16. If the action of user U1 detected in step S531 is an action related to washing the vehicle C1, the process proceeds to step S533. On the other hand, if the action of user U1 detected in step S531 is not an action related to washing the vehicle C1, the operation of the specific performance control process is terminated. Note that FIG. 17 illustrates the specific performance control process when user U1 washes the vehicle C1. If user U1 performs other maintenance on the vehicle C1, the processes of FIGS. 15, 16, etc. are executed.

[0162] Note that steps S533 to S542 correspond to steps S513 to S522 shown in FIG. 16, and therefore, in the following, a description of the parts common to FIG. 16 will be omitted.

[0163] In step S533, the output control unit 128 determines whether the behavior (maintenance behavior) of the user U1 detected in step S531 regarding washing the vehicle C1 is appropriate. If the maintenance behavior of the user U1 regarding washing the vehicle C1 is appropriate, the process proceeds to step S534. On the other hand, if the maintenance behavior of the user U1 regarding washing the vehicle C1 is inappropriate, the process proceeds to step S542.

[0164] In step S534, the output control unit 128 determines whether the user U1 is washing the vehicle C1 using a high-pressure gun. This determination can be made using any of the determination methods described above. For example, it is determined whether the user U1 is washing the vehicle C1 by spraying water on the vehicle C1 with the high-pressure gun. If the user U1 is washing the vehicle C1 using a high-pressure gun, the process proceeds to step S535. On the other hand, if the user U1 is not washing the vehicle C1 using a high-pressure gun, the process proceeds to step S536.

[0165] In step S535, since it has been determined in step S533 that the maintenance operation of user U1 regarding washing vehicle C1 is appropriate, the output control unit 128 causes the agent to perform an effect showing that the agent is in a good mood (an effect of feeling good), an effect expressing that the maintenance operation of user U1 is appropriate, etc. For example, as shown in Fig. 9(A), it is possible to display the agent image 200 displayed on the display unit 310 with a good-humored expression, and output audio information conveying that the maintenance operation of user U1 is appropriate.

[0166] In step S536, the output control unit 128 determines whether the user U1 has washed the car using shampoo. This determination can be made using any of the determination methods described above. If the user U1 has washed the car using shampoo, the process proceeds to step S537. On the other hand, if the user U1 has not washed the car using shampoo, the process proceeds to step S538.

[0167] In step S537, since it was determined in step S533 that the maintenance operation of user U1 regarding the washing of vehicle C1 was appropriate, the output control unit 128 causes the agent to perform an effect that indicates that the agent is in a good mood (an effect of feeling good, an effect of expressing joy), an effect that expresses that the maintenance operation of user U1 is appropriate, or the like. For example, as shown in FIG. 9(B), it is possible to display the agent image 200 displayed on the display unit 310 with a good-humored expression and output audio information that conveys that the maintenance operation of user U1 is appropriate. Alternatively, an effect may be performed that expresses that user U1 is unable to see due to white foam from washing the car with shampoo.

[0168] In step S538, the output control unit 128 determines whether the user U1 is performing a hand-wiping car wash. This determination can be made using any of the determination methods described above. For example, it is determined whether the user U1 is wiping the body of the vehicle C1 with a cloth. If the user U1 is performing a hand-wiping car wash, the process proceeds to step S539. On the other hand, if the user U1 is not performing a hand-wiping car wash, the process proceeds to step S540.

[0169] In step S539, since it has been determined in step S533 that the maintenance operation of user U1 regarding washing vehicle C1 is appropriate, the output control unit 128 causes the agent to execute a specific effect, such as an effect showing that the agent is in a good mood (an effect showing a good mood, an effect showing fun), an effect showing that the maintenance operation of user U1 is appropriate, etc. For example, as shown in FIG. 10(B), it is possible to execute an effect in which the agent image 200 displayed on the display unit 310 dances, or to output audio information conveying that the maintenance operation of user U1 is appropriate, etc. Furthermore, it is also possible to execute an effect in which the agent image 200 has a good-humored expression or the agent image 200 shakes its head.

[0170] Although the determination process in steps S534, S536, and S538 shows an example of determining a typical car wash, the present invention is not limited to this. For example, the presence or absence of other car washes may be determined, and a specific performance by the agent may be performed based on the determination result.

[0171] Furthermore, the processing in steps S535, S537, and S539 shows an example of a specific effect, but is not limited to this. For example, other specific effects may be executed by the agent as appropriate based on the determination processing in steps S534, S536, and S538. Furthermore, as described above, if it is possible to obtain an evaluation value for the appropriateness of the maintenance operation of user U1, the specific effect may be changed as appropriate based on the magnitude of that evaluation value.

[0172] In step S540, the estimation unit 127 determines whether the user U1 has finished washing the car, based on the movement of the user U1 detected in step S531. For example, after the movement of the user U1 is estimated in step S532 to be a movement related to washing the vehicle C1, it can be determined that the user U1 has finished washing the car on the condition that the door of the vehicle C1 is opened and closed and the user U1 has left the vehicle C1. In this case, it can be determined that the user U1 has finished washing the car, with the additional condition that a certain amount of time has passed since there was no movement of people or objects inside or outside the vehicle C1 after the user U1 left the vehicle C1. Also, for example, after the movement of the user U1 is estimated in step S532 to be a movement related to washing the vehicle C1, it can be determined that the user U1 has finished washing the car on the condition that all preset movements related to washing the vehicle C1 (e.g., watering, shampooing, wiping) have been completed. If the user U1 has finished washing the car, the process proceeds to step S541. On the other hand, if the user U1 has not finished washing the car, the operation of the specific performance control process is terminated.

[0173] In step S541, since it was determined in step S540 that user U1 had finished washing the car, the output control unit 128 causes the agent to execute a performance or the like expressing gratitude for the car wash. For example, it is possible to display an agent image 200 as shown in FIG. 9C on the display unit 310 and output audio information conveying gratitude for the car wash. In this way, when user U1 has finished washing the car, it is possible to cause the agent to execute a performance or the like expressing gratitude for the car wash, thereby conveying gratitude to user U1, regardless of whether the maintenance operation being performed was appropriate.

[0174] In step S542, since it was determined in step S533 that the maintenance operation by user U1 regarding washing vehicle C1 was inappropriate, the output control unit 128 causes the agent to execute an effect indicating that the agent is in a bad mood (an effect of feeling unwell), an effect expressing that the maintenance operation by user U1 is inappropriate, or the like. For example, as shown in FIG. 10(A), it is possible to display an agent image 200 displayed on the display unit 310 with a bad mood expression and output audio information instructing the user U1 on the correct operation. In this way, it is possible to notify user U1 of the correct operation regarding the maintenance being performed.

[0175] As described above, in this embodiment, the agent can monitor the state of the vehicle C1 due to the maintenance performed and the actions of the user U1, and can execute an appropriate response according to the actions. That is, during the maintenance, a specific agent effect can be generated in accordance with the maintenance action of the user U1 based on the estimated type of maintenance and the maintenance action performed by the user U1 related to the maintenance, and the specific effect can be displayed to the user U1. In other words, during the maintenance, information (agent information) related to the maintenance action of the user U1 according to the type of maintenance can be output. Furthermore, in other words, it is possible to easily realize the agent understanding the various tasks performed by the user U1 and responding according to the tasks. For example, the agent can continuously respond along with the maintenance action of the user U1 during the maintenance. For example, if the user U1 is washing the vehicle C1, it is possible to realize a specific agent effect that continues to cheer on the user U1 as he washes the vehicle. As a result, by the agent responding to the user U1's behavior of attachment (i.e., performing maintenance on the vehicle C1) during the maintenance, the user U1's sense of affinity and attachment to the vehicle C1 can be further enhanced. In other words, by having the agent respond according to the maintenance actions of user U1 while maintenance is being performed on vehicle C1, it is possible to increase the user's attachment to vehicle C1 and increase the user's motivation to actively perform maintenance.

[0176] In this way, when user U1 is performing maintenance on vehicle C1, which is an act of affection, user U1 can easily check the agent's reaction to the act of affection, which makes it possible for user U1 to increase his or her affection for the agent. In other words, while performing maintenance, the agent's empathetic behavior toward user U1's maintenance action can increase attachment to the agent and increase intimacy.

[0177] Furthermore, in this embodiment, the output position of the agent can be switched as needed between the inside and outside of the vehicle C1, the car wash CW1, and the fuel supply machine TA1. That is, the output position of the agent can be switched depending on the location of the user U1 performing maintenance. For example, the display position of the agent image can be set to a location that is directly visible to the user U1. This allows the agent information to be appropriately output at a position appropriate for the work, regardless of where the user U1 performing maintenance is working. This allows the user U1 performing maintenance to easily grasp the presence of the agent visually or audibly, thereby increasing their attachment to the vehicle C1. Furthermore, because the agent's behavior changes depending on the content of the maintenance, it is possible to increase the motivation of the user U1 to actively perform maintenance.

[0178] For example, when the user U1 is having a tire changed at a dealership of the vehicle C1, the user U1 is often resting inside the dealership. In this case, an agent image may be displayed on an electronic device (e.g., a smartphone or tablet terminal) owned by the user U1 or an electronic device (e.g., a television or tablet terminal) installed in the dealership, and the agent image may be made to perform a specific performance.

[0179] [Example of agent image display] The above describes an example in which the agent image 200 and the like are displayed based on information stored in the agent information DB 170. However, the agent image may be, for example, an image generated by CG (Computer Graphics), or an image acquired from another device (for example, a server). Also, a HUD or the like may be used as a display medium for constantly displaying the agent image 200 and the like.

[0180] Furthermore, for example, the agent image may be changed depending on the user U1's behavior, etc. For example, using a technology (e.g., an image generation model, an AI model, or a generation AI) that generates new data using artificial intelligence (AI), a personified character image may be generated based on the user U1's behavior, etc., and this personified character image may be displayed instead of the agent image. For example, a generative adversarial network (GAN) may be used as this technology. For example, based on an image of the surroundings of the vehicle C1 acquired by the vehicle exterior image acquisition unit 102, at least one of the appearance and characteristics of the user U1 and the vehicle C1 contained in the image (e.g., the user U1's face, hairstyle, clothing, etc., the vehicle's color, model, vehicle speed, etc.) may be acquired. Therefore, the output control unit 128 may extract feature amounts based on the appearance of the user U1 and the vehicle C1 and generate personified character images based on these feature amounts. Furthermore, the output control unit 128 may generate personified character images based on the characteristics of the user U1 and the vehicle C1. Then, the output control unit 128 can display the personified character image on the output device 300 and execute a specific effect using the personified character image.

[0181] [Example of agent device being a robot] In the above, the output device 300 that displays the agent image 200 etc. has been described as an example of an agent device. However, a robot may also be used as the agent device. This robot agent device may be a robot that imitates a human, a robot that imitates an animal such as a rabbit or a pig, or a robot that imitates a virtual living thing (for example, the face of an anime character). Furthermore, this robot agent device may be a robot that expresses the likeness of a living thing with its entire body, or a robot that expresses only a part of its entire body (for example, its face). Furthermore, it is preferable to use a device that can perform various actions (for example, changing facial expressions, moving the head, face, limbs, etc.) based on instructions from the information processing device 110 as the robot agent device.

[0182] When using a robot agent device, it is assumed that the robot agent device cannot move outside the vehicle C1. For example, when performing maintenance on the vehicle C1 using a car wash CW1 or a fuel pump TA1, the user U1 will be outside the vehicle C1, and therefore it is assumed that the user U1 will not be able to see the robot agent device inside the vehicle. Therefore, when washing the car, filling up with fuel, etc., it is preferable to switch the display from the robot agent device to another output device (the car wash CW1 or the fuel pump TA1) and execute a specific agent performance.

[0183] [Example of executing processing on other devices or systems] In the above, examples have been shown in which the detection process, estimation process, control process, etc. are executed in the output device 300 and the information processing device 110 (or the information processing system 100), but all or part of each process may be executed in other devices. In this case, the information processing system is configured by the devices that execute part of each process. For example, at least part of each process can be executed using various information processing devices and various electronic devices, such as in-vehicle devices, devices that can be used by the user (e.g., smartphones, tablet terminals, personal computers, car navigation devices, IVI), and servers that can be connected via a predetermined network such as the Internet.

[0184] Furthermore, a part (or all) of the information processing system capable of executing the functions of the information processing device 110 (or the information processing system 100) may be provided by an application that can be provided via a predetermined network such as the Internet. This application is, for example, SaaS (Software as a Service).

[0185] [Configuration example and effects of this embodiment] The information processing method according to this embodiment is an information processing method for controlling effects (e.g., effects using the agent image 200) related to an agent capable of communicating with a user U1 riding in a vehicle C1. This information processing method includes a motion information acquisition process (steps S501, S511, and S531) for acquiring motion information related to the motion of the user U1, an estimation process (steps S502, S512, and S532) for estimating the type of maintenance of the vehicle C1 to be performed by the motion of the user U1 based on the motion information, and a control process (steps S503, S513 to S522, and S532 to S542) for executing specific effects (e.g., specific effects based on the display mode of the agent image 200, specific effects based on audio output) corresponding to the maintenance motion of the user U1 during the maintenance based on the motion information acquired during the maintenance after estimation in the estimation process and the estimated type of maintenance. Furthermore, the program according to this embodiment is a program for causing a computer to execute each of these processes. In other words, the program according to this embodiment is a program that causes a computer to realize each function that can be executed by the information processing device 110.

[0186] With this configuration, it is possible to have the agent respond according to the maintenance operation of the user U1 while maintenance is being performed on the vehicle C1. This makes it possible to increase the attachment of the user U1 to the vehicle C1 and increase the motivation of the user U1 to actively perform maintenance. Furthermore, proactive maintenance can prevent breakdowns, deterioration, etc. of the vehicle C1.

[0187] In the information processing method according to this embodiment, the control process (steps S503, S513, and S533) determines whether the maintenance operation corresponding to the estimated type of maintenance is appropriate based on operation information acquired during the execution of the estimated maintenance and the type of the estimated maintenance. Furthermore, in the control process (steps S503, S514 to S519, and S534 to S539), if it is determined that the maintenance operation is appropriate, a specific effect indicating that the maintenance operation is appropriate is executed. For example, the agent image 200 shown in FIGS. 6A and 6B can be displayed, and a specific effect can be executed. Furthermore, in the control process (steps S503, S522, and S542), if it is determined that the maintenance operation is inappropriate, a specific effect indicating that the maintenance operation is inappropriate is executed. For example, the agent image 200 shown in FIGS. 6C to 6E can be displayed, and a specific effect can be executed.

[0188] With this configuration, it is possible to change the agent's response depending on whether the maintenance actions of user U1 are appropriate during maintenance, which allows user U1 to easily understand whether the maintenance actions of user U1 are appropriate during maintenance, thereby increasing user U1's motivation to perform the maintenance appropriately.

[0189] In the information processing method according to this embodiment, the control process (steps S503, S513, S533) may acquire appropriate operation determination information corresponding to the type of maintenance estimated in the estimation process (steps S502, S512, S532) from appropriate operation information DB 160 (an example of a storage unit) in storage unit 130, which stores appropriate operation determination information for each type of maintenance to determine an appropriate maintenance operation according to the type of maintenance, and may determine whether the maintenance operation corresponding to the estimated type of maintenance is appropriate based on the operation information acquired during the implementation of the maintenance after the estimation and the acquired appropriate operation determination information.

[0190] According to this configuration, it is possible to determine whether the maintenance operation of user U1 is appropriate for performing maintenance using appropriate operation determination information set for each type of maintenance, thereby improving the accuracy of the determination.

[0191] In the information processing method according to this embodiment, the control process (steps S503, S513, S533) may determine whether the maintenance operation is appropriate based on at least one of the speed of the specified action of user U1, the number of specified actions of user U1, the duration of the specified action of user U1, and the order of the specified actions of user U1.

[0192] According to this configuration, it is possible to determine whether the maintenance action of the user U1 is appropriate for the implementation of maintenance using the speed of the predetermined action of the user U1, etc., and therefore it is possible to improve the accuracy of the determination.

[0193] In the information processing method according to this embodiment, the specific effect indicating that the maintenance operation is appropriate may be an effect indicating that the agent is in a good mood. For example, it is possible to display the agent image 200 shown in FIGS. 6(A) and 6(B) and execute the specific effect. Also, the specific effect indicating that the maintenance operation is inappropriate may be an effect indicating that the agent is in a bad mood. For example, it is possible to display the agent image 200 shown in FIGS. 6(C) to 6(E) and execute the specific effect.

[0194] With this configuration, it is possible to change the specific presentation of the agent depending on whether the maintenance actions of user U1 are appropriate during maintenance, which allows user U1 to easily understand visually or audibly whether the maintenance actions of user U1 are appropriate during maintenance, thereby increasing the motivation of user U1 to perform the maintenance appropriately.

[0195] The information processing method according to this embodiment further includes an image acquisition process (steps S501, S511, S531) for acquiring at least one image of the interior and exterior of the vehicle C1. In the operation information acquisition process (steps S501, S511, S531), a user image including the user U1 is acquired as operation information, and in the estimation process (steps S502, S512, S532), the type of maintenance of the vehicle C1 may be estimated based on the user image.

[0196] According to this configuration, it is possible to estimate the type of maintenance of the vehicle C1 using images acquired by the image acquisition units (vehicle interior image acquisition unit 101, vehicle exterior image acquisition unit 102, and engine room image acquisition unit 103). This makes it possible to eliminate the need for other sensors, and to reduce the manufacturing cost of the vehicle C1.

[0197] The information processing method according to this embodiment further includes a sensor value acquisition process (steps S501, S511, S531) for acquiring sensor values ​​output from sensors installed at least inside and outside the vehicle C1. In the operation information acquisition process (steps S501, S511, S531), the sensor values ​​are acquired as operation information, and in the estimation process (steps S502, S512, S532), the type of maintenance of the vehicle C1 may be estimated based on the sensor values.

[0198] According to this configuration, it is possible to estimate the type of maintenance of the vehicle C1 using sensor values ​​acquired using the sensors 104 installed in the vehicle C1. Therefore, it is not necessary to add a sensor for the estimation process, and it is possible to reduce the manufacturing cost of the vehicle C1.

[0199] The information processing method according to the present embodiment may further include an external information acquisition process (steps S501, S511, S531) for acquiring external information related to the external device 400 or another user (e.g., a gas station employee or a mechanic at a dealership of the vehicle C1) in a case where maintenance is performed by the external device 400 or another user (e.g., a gas station employee or a mechanic at a dealership of the vehicle C1) in response to an operation or action of the user U1 as an action of the user U1. For example, the external information related to the external device 400 (e.g., operation information, car wash process) can be acquired using an image acquired by an image acquisition unit (vehicle exterior image acquisition unit 102) or information acquired from the external device 400 via the communication unit 140. Furthermore, the external information related to another user can be acquired using an image acquired by an image acquisition unit (vehicle interior image acquisition unit 101, vehicle exterior image acquisition unit 102, engine compartment image acquisition unit 103). Furthermore, the estimation process (steps S502, S512, S532) may estimate the type of maintenance of the vehicle C1 based on the operation information and the external information. For example, if a user U1 in an image acquired by an image acquisition unit (interior image acquisition unit 101, exterior image acquisition unit 102, engine room image acquisition unit 103) is detected as sitting and resting, and another user (e.g., a mechanic) in the image is detected as performing maintenance on a vehicle C1, the type of maintenance on the vehicle C1 can be estimated based on the actions of the other user (e.g., the mechanic).

[0200] With this configuration, even when maintenance is performed by the external device 400 or another user (for example, a gas station employee or a mechanic at the dealership of the vehicle C1), it is possible to make the agent respond according to the actions of each device or each user related to the performance of the maintenance. This makes it possible to increase the attachment of the user U1 to the vehicle C1 and increase the motivation of the user U1 to actively perform maintenance.

[0201] In the information processing method according to this embodiment, the control process (steps S503, S513 to S522, S532 to S542) may identify external device 400 based on external information (e.g., operation information, car wash process), and cause external device 400 or a device in its vicinity to execute a specific effect related to the agent. For example, as shown in Figs. 11 and 13, it is possible to cause a car wash machine CW1 and a fueling machine TA1 to display an agent image 200 and execute a specific effect.

[0202] According to this configuration, when maintenance is performed by the external device 400, it is possible to make the agent respond according to the operation of the external device 400. This makes it possible to increase the attachment of the user U1 to the vehicle C1 and to increase the motivation of the user U1 to actively perform maintenance using the external device 400.

[0203] In the information processing method according to this embodiment, the external device 400 includes a display unit 431, and in the control process (steps S503, S513 to S522, S532 to S542), an agent image 200 relating to an agent may be displayed on the display unit 431, and a specific effect may be executed using the agent image 200. For example, as shown in Figs. 11 and 13, the agent image 200 may be displayed on the display unit 431 of a car wash machine CW1 and a fueling machine TA1, and a specific effect may be executed.

[0204] According to this configuration, when maintenance is performed by the external device 400, it is possible to display the agent image 200 and execute a specific effect in accordance with the operation of the external device 400. This makes it easy to visually grasp the agent image 200, thereby increasing the motivation of the user U1 to actively perform maintenance using the external device 400.

[0205] In the information processing method according to this embodiment, the control processing (steps S503, S513 to S522, S532 to S542) may execute at least one of a sound output processing that outputs audio information about the agent from the sound output unit 320 provided in the vehicle C1 to perform a specific effect, and a display processing that displays an agent image 200 about the agent on the display unit 310 provided in the vehicle C1 to perform a specific effect.

[0206] According to this configuration, the specific effects by the agent can be easily understood visually or audibly, which can increase the motivation of user U1 to proactively carry out maintenance.

[0207] In the information processing method according to this embodiment, in the control process (steps S503, S513 to S522, S532 to S542), the maintenance parts of the vehicle C1 that correspond to the type of maintenance estimated in the estimation process (steps S502, S512, S532) may be displayed on the display unit 310, and the maintenance parts that have been completed and the maintenance parts that have not yet been completed may be distinguishable on the display unit 310. For example, as shown in Fig. 14, the maintenance parts that have been completed and the maintenance parts that have not yet been completed may be distinguishable by the maintenance status displays MP1 to MP7.

[0208] According to this configuration, it is possible to visually and easily grasp which parts of the maintenance have been performed and which parts have not yet been performed, so that the user U1 performing the maintenance can easily grasp the progress status.

[0209] In the information processing method of this embodiment, the estimation process (steps S502, S512, S532) may estimate at least one of cleaning the interior and / or exterior of the vehicle C1 (including removing dirt from the interior and / or exterior of the vehicle C1), replenishing objects or liquids in the vehicle C1, and replacing deteriorated items in the vehicle C1 as the type of maintenance.

[0210] According to this configuration, it is possible to estimate an appropriate type of maintenance.

[0211] In the information processing method according to this embodiment, in the control process (steps S503, S513 to S522, S532 to S542), the position of user U1 may be estimated based on the operation information, and a special effect related to the agent may be executed on an output unit (e.g., display unit 310, windshield 4, window glasses W1, W2) corresponding to the position of user U1 in vehicle C1. Note that if the estimated position of user U1 is closer to external device 400 than vehicle C1, a special effect related to the agent may be executed on external device 400.

[0212] According to this configuration, the specific effect by the agent is output at a position that is easily recognized by the user U1, which can increase the motivation of the user U1 to proactively carry out maintenance.

[0213] The information processing device 110 is an information processing device that controls effects (e.g., effects using an agent image 200) related to an agent that can communicate with a user U1 riding in a vehicle C1. The information processing device 110 includes a detection unit 126 (an example of a motion information acquisition unit) that acquires motion information related to the motion of the user U1, an estimation unit 127 that estimates the type of maintenance of the vehicle C1 that will be performed by the motion of the user U1 based on the motion information, and an output control unit 128 (an example of a control unit) that executes specific effects corresponding to the maintenance motion of the user U1 during the maintenance based on the motion information acquired during the maintenance after estimation by the estimation unit 127 and the estimated type of maintenance. The information processing device 110 may be a device built into an agent device (output device 300, external device 400) or may be a device separate from the agent device (output device 300, external device 400). Instead of the information processing device 110, an information processing system configured with multiple devices capable of executing the processes realized by the information processing device 110 may be used.

[0214] With this configuration, it is possible to have the agent respond according to the maintenance operation of the user U1 while maintenance is being performed on the vehicle C1. This makes it possible to increase the attachment of the user U1 to the vehicle C1 and increase the motivation of the user U1 to actively perform maintenance. Furthermore, proactive maintenance can prevent breakdowns, deterioration, etc. of the vehicle C1.

[0215] Note that each processing procedure shown in this embodiment is an example for realizing this embodiment, and the order of some of the processing procedures may be changed within the scope that makes it possible to realize this embodiment, and some of the processing procedures may be omitted or other processing procedures may be added.

[0216] Each process in this embodiment is executed based on a program that causes a computer to execute various processing procedures. This embodiment can also be understood as an embodiment of a program that realizes the function of executing each process and a recording medium that stores the program. For example, an update process for adding a new function to an information processing device can store the program in the storage device of the information processing device. This makes it possible to cause the updated information processing device to execute each process described in this embodiment.

[0217] Although the embodiments of the present invention have been described above, the above embodiments merely show application examples of the present invention, and are not intended to limit the technical scope of the present invention to the specific configurations of the above embodiments. [Explanation of symbols]

[0218] 20 network, 100 information processing system, 101 vehicle interior image acquisition unit, 102 vehicle exterior image acquisition unit, 103 engine room image acquisition unit, 104 sensors, 105 sound acquisition unit, 110 information processing device, 120, 420 control unit, 125 vehicle state acquisition unit, 126 detection unit, 127 maintenance determination unit, 128 output control unit, 130 memory unit, 140, 410 communication unit, 150 maintenance information DB, 160 proper operation information DB, 170 agent information DB, 180 map information DB, 300 output device, 310, 431 display unit, 320, 432 sound output unit, 400 external device, 430 output unit, 440 operation unit

Claims

1. An information processing method for controlling a performance related to an agent capable of communicating with a user riding in a vehicle, comprising: a motion information acquisition process for acquiring motion information related to the user's motion; an estimation process for estimating a type of maintenance of the vehicle performed by the user's action based on the action information; a control process for executing a specific effect during the maintenance operation according to the maintenance operation of the user based on the operation information acquired during the maintenance operation after the estimation in the estimation process and the estimated type of the maintenance; An information processing method including:

2. 2. The information processing method according to claim 1, In the control process, determining whether the maintenance operation corresponding to the estimated type of maintenance is appropriate based on the operation information acquired during the execution of the maintenance after the estimation and the estimated type of maintenance; When it is determined that the maintenance operation is appropriate, the specific performance indicating that the maintenance operation is appropriate is executed, and when it is determined that the maintenance operation is inappropriate, the specific performance indicating that the maintenance operation is inappropriate is executed. Information processing methods.

3. 3. The information processing method according to claim 2, In the control process, acquiring appropriate operation determination information corresponding to the type of maintenance estimated in the estimation process from a storage unit that stores appropriate operation determination information for each type of maintenance, the appropriate operation determination information being used to determine the maintenance operation that is appropriate according to the type of maintenance; determining whether the maintenance operation corresponding to the estimated type of maintenance is appropriate based on the operation information acquired during the execution of the maintenance after the estimation and the acquired appropriate operation determination information; Information processing methods.

4. 4. The information processing method according to claim 2 or 3, In the control process, it is determined whether the maintenance operation is appropriate based on at least one of a speed of the predetermined operation of the user, a number of times the predetermined operation of the user, a duration of the predetermined operation of the user, and an order of the predetermined operation of the user. Information processing methods.

5. 3. The information processing method according to claim 2, the specific effect indicating that the maintenance operation is appropriate is an effect indicating that the agent is in a good mood, The specific effect indicating that the maintenance operation is inappropriate is an effect indicating that the agent is in a bad mood. Information processing methods.

6. 2. The information processing method according to claim 1, and an image capture process for capturing an image of at least one of the interior and exterior of the vehicle. In the motion information acquisition process, a user image including the user is acquired as the motion information; In the estimation process, the type of maintenance is estimated based on the user image. Information processing methods.

7. 10. The information processing method according to claim 1, further comprising: a sensor value acquisition process for acquiring sensor values ​​output from sensors installed at least inside and outside the vehicle; In the operation information acquisition process, the sensor value is acquired as the operation information; In the estimation process, the type of maintenance is estimated based on the sensor value. Information processing methods.

8. 2. The information processing method according to claim 1, an external information acquisition process for acquiring external information relating to the external device or the other user when the maintenance is performed by an external device or the other user in response to an operation or action of the user as the user's behavior, In the estimation process, the type of maintenance is estimated based on the operation information and the external information. Information processing methods.

9. 9. The information processing method according to claim 8, the control process identifies the external device based on the external information, and causes the external device or a device nearby to execute the specific performance related to the agent. Information processing methods.

10. 10. The information processing method according to claim 9, the external device includes a display unit, In the control process, an agent image relating to the agent is displayed on the display unit, and the specific performance is executed using the agent image. Information processing methods.

11. 2. The information processing method according to claim 1, In the control process, at least one of a sound output process for outputting audio information related to the agent from a sound output unit provided in the vehicle to execute the specific performance, and a display process for displaying an agent image related to the agent on a display unit provided in the vehicle to execute the specific performance is executed. Information processing methods.

12. 2. The information processing method according to claim 1, In the control process, a maintenance part of the vehicle corresponding to the type of maintenance estimated in the estimation process is displayed on a display unit, and a maintenance-treated part and a maintenance-untreated part among the maintenance parts can be distinguished on the display unit. Information processing methods.

13. 2. The information processing method according to claim 1, In the estimation process, at least one of cleaning the interior and / or exterior of the vehicle, replenishing objects or liquids in the vehicle, and replacing deteriorated items in the vehicle is estimated as the type of maintenance. Information processing methods.

14. 2. The information processing method according to claim 1, the control processing estimates a position of the user based on the operation information, and causes an output unit corresponding to the position of the user in the vehicle to execute a specific performance related to the agent. Information processing methods.

15. An information processing device that controls a performance related to an agent that can communicate with a user riding in a vehicle, a motion information acquisition unit that acquires motion information related to the user's motion; an estimation unit that estimates a type of maintenance of the vehicle performed by the user's action based on the action information; a control unit that executes a specific performance according to the maintenance operation of the user during the maintenance based on the operation information acquired during the maintenance operation after the estimation by the estimation unit and the estimated type of the maintenance; An information processing device comprising:

Images