Information processing device

Abstract

To improve the technology related to vehicle agent systems. [Solution] An information processing device 200 comprising a control unit 203, wherein the control unit 203 estimates the user's state based on user information including at least one of voice data, image data, or biometric data of the user of the vehicle 100, and if it is determined that the estimated state satisfies predetermined conditions, the vehicle 100 executes a predetermined process selected according to the estimated state.

Description

Technical Field

[0001] The present disclosure relates to an information processing apparatus.

Background Art

[0002] Conventionally, technologies related to in-vehicle agent systems are known. For example, Patent Document 1 discloses an agent apparatus that presents response information corresponding to voice information of an occupant boarding a vehicle with respect to the voice information of the occupant.

Prior Art Literature

Patent Literature

[0003]

Patent Document 1

Summary of Invention

Problem to be Solved by the Invention

[0004] Conventional in-vehicle agent systems have room for improvement from the perspective of user experience.

[0005] In view of such circumstances, an object of the present disclosure is to improve the technology related to in-vehicle agent systems.

Means for Solving the Problem

[0006] An information processing apparatus according to an embodiment of the present disclosure is an information processing apparatus comprising a control unit, wherein the control unit estimates the state of the user based on user information including at least one of voice data, image data, or biometric data of a user of a vehicle, and when it is determined that the estimated state satisfies a predetermined condition, executes a predetermined process selected in accordance with the estimated state in the vehicle.

Effect of the Invention

[0007] According to one embodiment of this disclosure, the technology relating to vehicle agent systems is improved. [Brief explanation of the drawing]

[0008] [Figure 1] This block diagram shows a schematic configuration of a system according to one embodiment of the present disclosure. [Figure 2] This is a block diagram showing the general configuration of the vehicle. [Figure 3] This is a block diagram illustrating the schematic configuration of an information processing device. [Figure 4] This is a flowchart showing the operation of an information processing device. [Modes for carrying out the invention]

[0009] The embodiments of this disclosure will be described below.

[0010] <Overview of Embodiments> Referring to Figure 1, an overview of the system 10 according to the embodiment of this disclosure will be described. The system 10 comprises a vehicle 100 and an information processing device 200. The vehicle 100 and the information processing device 200 are communicably connected to a network 300, which includes, for example, the Internet and a mobile communication network.

[0011] Vehicle 100 is, for example, an automobile, but is not limited to this and may be any vehicle. An automobile is any type of vehicle, such as a gasoline car, diesel car, HEV (hybrid electric vehicle), PHEV (plug-in hybrid electric vehicle), BEV (battery electric vehicle), or FCEV (fuel cell electric vehicle). In this embodiment, Vehicle 100 is driven by a driver, but the driving may be automated to any level. The level of automation may be, for example, any of levels 1 to 5 in the SAE (Society of Automotive Engineers) classification. Vehicle 100 may also be a vehicle dedicated to MaaS (Mobility as a Service). The number of vehicles 100 provided by System 10 can be arbitrarily determined.

[0012] In this embodiment, the vehicle 100 has a function to acquire voice data of a user riding in the vehicle 100 (hereinafter also referred to as "voice data"). The vehicle 100 has a function to acquire images or videos (hereinafter also referred to as "image data") of the interior of the vehicle, including the user's facial expressions and body movements other than facial expressions. The vehicle 100 has a function to acquire data (hereinafter also referred to as "biometric data") indicating the user's biological state, such as heart rate variability, heat dissipation, or respiration. Biometric data may be acquired from biosensors such as vein sensors attached to any position inside the vehicle 100, or from wearable devices such as wristwatch-type devices or clothing-type devices that can be attached to any position on the user's body or clothing. The vehicle 100 has a function to acquire data (hereinafter also referred to as "driving data") that changes as the vehicle 100 is driving, such as the vehicle's position, speed, acceleration, shift position, ignition ON, ignition OFF, or mileage. Furthermore, vehicle 100 has the function of storing and / or transmitting the acquired data.

[0013] The information processing device 200 is installed in a facility such as a data center. The information processing device 200 is a computer, such as a server device belonging to a cloud computing system or other computing system. The information processing device 200 can communicate with the vehicle 100 via the network 300. The information processing device 200 can acquire any data from each vehicle, including voice data, image data, biometric data, and / or driving data, relating to the user riding in the vehicle and / or the vehicle itself.

[0014] In this embodiment, system 10 is used as a vehicle agent system.

[0015] First, an overview of this embodiment will be described, and details will be described later. The information processing device 200 estimates the user's state based on user information, which includes at least one of the following: voice data, image data, or biometric data of the user of the vehicle 100. If the information processing device 200 determines that the estimated state satisfies predetermined conditions, it executes a predetermined process in the vehicle 100 that is selected according to the estimated state.

[0016] Thus, according to this embodiment, by estimating the user's state in the vehicle 100, predetermined processes can be proactively executed without waiting for instructions from the user. For example, if it is estimated that the user is under stress or drowsy, the system can proactively perform processes to mitigate adverse effects on driving by changing the in-vehicle environment of the vehicle 100 to counteract that state. Therefore, it is possible to provide a highly satisfying travel experience even to users who find conventional passive vehicle agent services unsatisfactory. Consequently, the technology related to vehicle agent systems is improved in that it is easier to enhance the user experience.

[0017] Next, we will describe each component of System 10 in detail.

[0018] <Vehicle Configuration> As shown in FIG. 2, the vehicle 100 includes a communication unit 101, an acquisition unit 102, an imaging unit 103, an agent 104, a spray mechanism 105, an audio device 106, an air conditioner 107, a window opening / closing device 108, a seat vibration device 109, an output unit 110, an input unit 111, a storage unit 112, and a control unit 113.

[0019] The communication unit 101 includes one or more communication interfaces connected to the network 300. The communication interface corresponds to, for example, but is not limited to, mobile communication standards such as 4G (4th Generation) or 5G (5th Generation). In the present embodiment, the vehicle 100 communicates with the information processing apparatus 200 via the communication unit 101 and the network 300.

[0020] The acquisition unit 102 includes a first sensor module capable of acquiring travel data of the vehicle 100. For example, the first sensor module includes one or more devices that acquire position information of the vehicle 100. Specifically, the first sensor module includes, for example, a receiver compatible with GPS (Global Positioning System), but is not limited thereto, and may include a receiver compatible with any satellite positioning system. Further, the first sensor module includes a speed sensor, an acceleration sensor, an angular velocity sensor, a geomagnetic sensor, a distance sensor such as LiDAR (light detection and ranging), a temperature and humidity sensor, or a combination thereof. The vehicle 100 can acquire travel data of the vehicle 100 using the first sensor module of the acquisition unit 102.

[0021] Further, the acquisition unit 102 includes a second sensor module capable of acquiring biometric data of a user riding in the vehicle 100. For example, the second sensor module includes a vein sensor, a fingerprint sensor, a voiceprint sensor, an infrared sensor, a respiration sensor, or a combination thereof. The vein sensor may be any sensor capable of detecting a user's vein pattern. The fingerprint sensor may be any sensor capable of reading a user's fingerprint pattern. The voiceprint sensor may be any sensor capable of reading a user's voiceprint pattern. The infrared sensor may be any infrared sensor capable of measuring the amount of heat dissipated by the user over time. The respiration sensor may be any sensor capable of measuring the user's respiration cycle over time. The vehicle 100 can acquire biometric data of each user using the second sensor module of the acquisition unit 102. The acquired biometric data can be used for user authentication by a well-known method such as pattern matching.

[0022] The imaging unit 103 includes one or more cameras. Each camera included in the imaging unit 103 may be provided in the vehicle 100 so as to be capable of imaging a subject inside or outside the vehicle, for example. In the present embodiment, the imaging unit 103 includes an in-vehicle camera capable of imaging a subject inside the vehicle 100. The imaging unit 103 is not limited thereto, and may include, for example, a front camera and a rear camera capable of imaging subjects in front of and behind the vehicle 100, or an omnidirectional camera capable of imaging subjects around the vehicle 100. Further, the imaging unit 103 includes an input interface capable of communicating with a camera such as a drive recorder installed in the vehicle 100 so as to be capable of imaging a subject inside or outside the vehicle, and may be capable of acquiring images of each user riding in the vehicle 100 from the camera. In the present embodiment, the vehicle 100 can acquire image data of each user using the imaging unit 103. The acquired image data can be used for user authentication by a well-known method such as pattern matching, or for state analysis of each riding user.

[0023] Agent 104 is an ECU (Electronic Control Unit) equipped with a computer that incorporates voice dialogue software that performs speech recognition and analysis of the user of the vehicle 100 and generates questions or responses to the user. The voice dialogue software includes, for example, voice dialogue AI (artificial intelligence). Agent 104 can use the dialogue history to interpret the content of the utterance or to generate responses or questions. In this embodiment, Agent 104 is configured to perform user interaction as described later. The vehicle 100 can use Agent 104 to perform a process of speaking to the user according to the speech modes that Agent 104 can perform (hereinafter also referred to as "user interaction").

[0024] The spraying mechanism 105 is an arbitrary mechanism that can be installed at any location within the vehicle 100 and spray fragrance inside the vehicle 100. The spraying mechanism 105 includes one or more discharge ports. The vehicle 100 can use the spraying mechanism 105 to perform a process (hereinafter also referred to as "fragrance spraying") in which any fragrance is sprayed inside the vehicle 100 in any amount and concentration. In this embodiment, the fragrance is a so-called functional fragrance that is expected to have effects such as "relaxing" or "enhancing concentration". For example, the fragrance includes any type of fragrance, such as a floral fragrance that is expected to have a relaxing effect on people, or a citrus fragrance that is expected to enhance concentration or awaken people.

[0025] The audio device 106 is any device such as a music player that generates audio signals based on acoustic data. The acoustic data is provided to the audio device 106, for example, by a storage medium, broadcast, or via the network 300. The audio device 106 may include one or more of the following: a CD player, DVD player, Blu-ray player, hard disk drive, AM tuner, FM tuner, TV tuner, or audio decoder. The vehicle 100 may use the audio device 106 to perform a process (hereinafter also referred to as "music playback") that allows the user to listen to any music content or broadcast output from a speaker, which will be described later as an output unit 110.

[0026] The air conditioning unit 107 is an optional device that has the function of sensing and adjusting the temperature and humidity inside the vehicle 100. The air conditioning unit 107 includes one or more air vents. Each air vent may be provided inside the vehicle 100 so as to direct airflow to any user inside the vehicle. The vehicle 100 can use the air conditioning unit 107 to adjust the temperature and humidity inside the vehicle 100 to any desired temperature and humidity, and to perform a process (hereinafter also referred to as "air conditioning adjustment") to adjust the airflow direction and airflow intensity of each air vent.

[0027] The window opening / closing device 108 is an arbitrary device that has the function of controlling the opening and closing of each window of the vehicle 100. The vehicle 100 can use the window opening / closing device 108 to perform the process of opening and closing any window of the vehicle 100 by any width (hereinafter also referred to as "window opening / closing").

[0028] The seat vibration device 109 is an optional device that has the function of controlling the vibration of each seat of the vehicle 100. The seat vibration device 109 can vibrate any seat of the vehicle 100 with any strength. For example, a vibrating plate may be embedded in the backrest, seat surface, or both of each seat. The vehicle 100 can perform a process (hereinafter also referred to as "seat vibration") that vibrates any seat with any strength by vibrating any vibrating plate of each seat using the seat vibration device 109. The seat vibration device 109 may also have a massage function.

[0029] The output unit 110 may include one or more output devices that output information to notify the user. These output devices are, for example, speakers that output information as sound, or displays that output information as video, but are not limited to these. The output unit 110 may also include an interface for connecting external output devices. The output unit 110 can be used for music playback using the audio device 106 of this embodiment.

[0030] The input unit 111 includes one or more input devices for detecting user input. In this embodiment, the input unit 111 includes a microphone that receives input of user voice and sounds from inside the vehicle 100. However, the input devices are not limited to these examples and may include, for example, physical keys, capacitive keys, or touchscreens integrated with a display. The input unit 111 may also include an interface for connecting external input devices. In this embodiment, the vehicle 100 can acquire voice data from each user using the input unit 111. The acquired voice data can be used for user authentication using well-known methods such as pattern matching, or for analyzing the state of each user riding in the vehicle.

[0031] The storage unit 112 includes one or more memories. The memories are, for example, semiconductor memories, magnetic memories, or optical memories, but are not limited to these. Each memory included in the storage unit 112 may function as, for example, a main memory, an auxiliary memory, or a cache memory. The storage unit 112 stores any information used for the operation of the vehicle 100. For example, the storage unit 112 may store system programs, application programs, embedded software, and map information. The information stored in the storage unit 112 may be updateable with information obtained from the network 300 via, for example, the communication unit 101.

[0032] In this embodiment, the storage unit 112 can store data acquired in the vehicle 100, namely voice data, image data, biometric data, and driving data. The storage unit 112 may also store usage history of elements such as the spraying mechanism 105 and the audio device 106 (such as identification information of the executed function and a timestamp of the execution time) in association with the user.

[0033] The control unit 113 includes one or more processors, one or more programmable circuits, one or more dedicated circuits, or a combination thereof. The processor is a general-purpose processor such as a CPU (Central Processing Unit) or GPU (Graphics Processing Unit), or a dedicated processor specialized for a specific process, but is not limited to these. The programmable circuit is an FPGA (Field-Programmable Gate Array), but is not limited to this. The dedicated circuit is an ASIC (Application Specific Integrated Circuit), but is not limited to this. The control unit 113 controls the operation of the entire vehicle 100.

[0034] In this embodiment, the control unit 113 transmits voice data, image data, biometric data, and / or driving data to the information processing device 200 via the communication unit 101 and the network 300. The control unit 113 also receives data from the information processing device 200 that includes instructions for executing a predetermined process in the vehicle 100 and specifications for detailed functions (hereinafter also referred to as "instruction data"). The control unit 113 may execute the predetermined process in the vehicle 100 using at least one of the following elements: agent 104, audio device 106, spray mechanism 105, air conditioning device 107, window opening / closing device 108, or seat vibration device 109, according to the instruction data.

[0035] <Configuration of the information processing device> As shown in Figure 3, the information processing device 200 comprises a communication unit 201, a storage unit 202, and a control unit 203.

[0036] The communication unit 201 includes one or more communication interfaces connected to the network 300. These communication interfaces may support, for example, mobile communication standards, wired LAN (Local Area Network) standards, or wireless LAN standards, but are not limited to these, and may support any communication standard. In this embodiment, the information processing device 200 communicates with the vehicle 100 via the communication unit 201 and the network 300.

[0037] The storage unit 202 includes one or more memories. Each memory included in the storage unit 202 may function as, for example, a main memory, an auxiliary memory, or a cache memory. The storage unit 202 stores any information used in the operation of the information processing device 200. For example, the storage unit 202 may store system programs, application programs, databases, and map information. The information stored in the storage unit 202 may be updatable with information obtained from the network 300 via, for example, the communication unit 201.

[0038] The control unit 203 includes one or more processors, one or more programmable circuits, one or more dedicated circuits, or a combination thereof. The control unit 203 controls the operation of the entire information processing device 200.

[0039] In this embodiment, the control unit 203 receives voice data, image data, biometric data, and / or driving data from the communication unit 101 of each vehicle 100 via the communication unit 201 and the network 300. Based on the received data, the control unit 203 estimates the user's state and generates instruction data. The control unit 203 can transmit the instruction data to the communication unit 101 of the vehicle 100 via the communication unit 201 and the network 300 to cause the vehicle 100 to execute a predetermined process. By causing the vehicle 100 to execute a predetermined process according to the instruction data, the control unit 203 realizes the vehicle agent service of this embodiment.

[0040] <Operation Flow of Information Processing Device> Referring to Figure 4, the operation of the information processing device 200 according to this embodiment will be described. The operation in Figure 4 corresponds to the method according to this embodiment. In this embodiment, the operation in Figure 4 is started when the ignition of the vehicle 100 is detected to be ON, and is repeatedly executed at a predetermined period that can be set arbitrarily. However, the conditions or timing for starting this operation are not limited to this example. The number of users of the vehicle 100 may be one or multiple, but unless otherwise specified below, the user of the vehicle 100 will be assumed to be only the driver seated in the driver's seat.

[0041] Step S100: The control unit 203 of the information processing device 200 acquires user information, which includes at least one of the following: voice data, image data, or biometric data of the user of the vehicle 100.

[0042] Specifically, the control unit 203 acquires user information by receiving at least one of the following from the vehicle 100's communication unit 101 via the communication unit 201 and the network 300: voice data, image data, or biometric data acquired by the vehicle 100 during a predetermined aggregation period. The predetermined aggregation period can be arbitrarily determined, but for example, it may be one minute.

[0043] Step S101: The control unit 203 estimates the user's state based on user information. The user's state may include any state of the user that could adversely affect the in-vehicle environment or operation of the vehicle 100, but in this embodiment it refers to the user's stress level or drowsiness level.

[0044] Specifically, the control unit 203 analyzes the voice data, image data, or biometric data included in the user information to determine the intensity of the user's stress or drowsiness, thereby estimating the user's state during the aggregation period. The intensity of the user's stress or drowsiness can be determined by any method, but in this embodiment, it is determined based on a stress score, which is an evaluation index for the intensity of stress, or a drowsiness score, which is an evaluation index for the intensity of drowsiness. In this embodiment, both the stress score and the drowsiness score are represented by numerical values ​​(for example, integers greater than or equal to 0), but are not limited to this. A higher stress score indicates that the user is under more stress. A higher drowsiness score indicates that the user is more drowsy. For each of the acquired voice data, image data, or biometric data, the control unit 203 calculates the cumulative value of the stress score (hereinafter also referred to as the "cumulative stress score") and the cumulative value of the drowsiness score (hereinafter also referred to as the "cumulative drowsiness score") during the aggregation period. Specific examples for each type of data are shown below. Furthermore, any method can be used to calculate the cumulative stress score and cumulative sleepiness score, and known stress and sleepiness estimation methods can be used in addition to or instead of the following specific examples.

[0045] In the case of voice data, the control unit 203 can calculate the cumulative stress score and cumulative drowsiness score of the voice data for the aggregation period by extracting and analyzing the user's voice information from the acquired voice data. Specifically, the control unit 203 can calculate the cumulative stress score or cumulative drowsiness score by extracting voice information such as the user's utterances, sound pressure, or tone from the voice data, performing voice recognition processing, and executing pattern matching. In this case, for example, typical keywords or phrases that a person utters when they feel stressed or drowsy may be registered in the memory unit 202, and the score may be calculated so that the higher the detection frequency of these keywords or phrases by pattern matching, the higher the score value.

[0046] In the case of image data, the control unit 203 extracts and analyzes user image information from the acquired image data to calculate the cumulative stress score and cumulative drowsiness score for the image data during the aggregation period. For example, the control unit 203 may extract image information such as the user's facial expressions or body movements from the image data, perform image recognition processing, and perform pattern matching to calculate the cumulative stress score or cumulative drowsiness score. In this case, for example, typical facial expressions or body movements that a person displays when they feel stressed or drowsy may be registered in the memory unit 202, and the score may be calculated so that the higher the detection frequency of these by pattern matching, the higher the numerical value. Examples of facial expressions or body movements include furrowing the brow, tapping the steering wheel with a finger, yawning, or a decrease in the opening and closing range of the eyelids. In this case, the cumulative value may be calculated so that the longer the duration of such facial expressions or body movements, the higher the cumulative value.

[0047] In the case of biometric data, the control unit 203 can extract and analyze biometric information such as the user's heart rate variability, heat dissipation, or respiration from the acquired biometric data to calculate the cumulative stress score and cumulative sleepiness score for the biometric data during the aggregation period. For example, the control unit 203 can calculate a numerical value for the stress score based on the LF (low frequency) / HF (high frequency) ratio obtained from the time-series data of the user's heart rate variability. In this case, the calculation may be such that the higher the LF / HF ratio, the higher the numerical value for the stress score. Alternatively, for example, the control unit 203 can calculate a numerical value for the sleepiness score based on the user's heat dissipation. In this case, the calculation may be such that the higher the user's heat dissipation, the higher the numerical value for the sleepiness score.

[0048] These voice data, image data, and biometric data may be analyzed individually, but from the viewpoint of improving the accuracy of estimating the user's state, it is desirable to analyze two or more arbitrary combinations. Specifically, the control unit 203 calculates a score by summing the cumulative stress scores of the various data (hereinafter also referred to as the "total stress score") and a score by summing the cumulative drowsiness scores of the various data (hereinafter also referred to as the "total drowsiness score") for each aggregation period. In other words, one total stress score and one total drowsiness score are calculated for each aggregation period.

[0049] Based on the total stress score and total sleepiness score calculated in this manner, the control unit 203 determines the user's stress level and sleepiness level during the aggregation period. In this embodiment, the stress level and sleepiness level are evaluation indices of the intensity of the user's stress and sleepiness, respectively, and are indicated by a five-level scale from level 1 to level 5. A higher level indicates a stronger user stress or sleepiness. However, the evaluation indices are not limited to this example. The correspondence between scores and levels can be set arbitrarily. Based on the total stress score and total sleepiness score, the control unit 203 determines the user's stress level and sleepiness level during the aggregation period.

[0050] Based on the stress level and sleepiness level determined in this way, the control unit 203 can estimate the user's state during each aggregation period (in this case, whether the user is experiencing high levels of stress or sleepiness). For example, if the stress level or sleepiness level is level 4 or higher, the control unit 203 may estimate that the user is experiencing high levels of stress or sleepiness during that aggregation period. If the stress level or sleepiness level is level 3 or lower, the control unit 203 may estimate that the user is not experiencing high levels of stress or sleepiness during that aggregation period.

[0051] Step S102: The control unit 203 determines whether the estimated state satisfies predetermined conditions. If the control unit 203 determines that the predetermined conditions are met (Step S102-Yes), the process proceeds to Step S103. On the other hand, if the control unit 203 determines that the predetermined conditions are not met (Step S102-No), the process terminates.

[0052] The predetermined conditions can be set arbitrarily, but in this embodiment, the user is in a state of high stress or drowsiness. Specifically, the user's stress level or drowsiness level during the aggregation period is at or above a predetermined level (level 4 in this example). If the stress level or drowsiness level determined in step S101 is level 4 or higher, the control unit 203 estimates that the user was in a state of high stress or drowsiness during the aggregation period. In this case, the control unit 203 can determine that the user's state satisfies the predetermined conditions.

[0053] Step S103: If the control unit 203 determines that the estimated state satisfies a predetermined condition (Step S102-Yes), the control unit 203 executes a predetermined process in the vehicle 100 that is selected according to the estimated state.

[0054] Specifically, the control unit 203 selects a process from among the candidates for processes that can be executed in the vehicle 100 ("process candidates") that changes the in-vehicle environment of the vehicle 100 in a way that cancels out the user's state ("user state") which is determined to meet predetermined conditions, thereby mitigating adverse effects on driving. Any process can be set as a process candidate, but in this embodiment, fragrance spraying, music playback, air conditioning adjustment, window opening and closing, seat vibration, and speaking to the user are set in the vehicle 100. The control unit 203 selects at least one of these process candidates as the predetermined process in this embodiment.

[0055] For example, if step S102 detects that the stress level has risen above a predetermined level, it is estimated that the user is in a highly stressed state during the aggregation period. In this case, the control unit 203 may select at least one process, such as fragrance spraying, music playback, and talking to the user in the vehicle 100, in order to counteract the user's state (in this case, high stress).

[0056] For example, if step S102 detects that the drowsiness level has reached a predetermined level or higher, it is estimated that the user is in a state of severe drowsiness during the aggregation period. In this case, the control unit 203 may select at least one action to counteract the user's state (in this case, severe drowsiness), such as fragrance spraying, music playback, air conditioning adjustment, window opening / closing, seat vibration, and speaking to the user.

[0057] The control unit 203 then specifies the detailed functions of at least one selected process. In the case of music playback, the control unit 203 specifies, for example, the music content to be played or the volume. In the case of air conditioning adjustment, the control unit 203 specifies, for example, the temperature and humidity, the air outlet, the airflow direction, or the airflow strength. In the case of window opening and closing, the control unit 203 specifies, for example, which windows of the vehicle 100 (driver's seat window, other windows, or both, etc.) to open and close and to what extent (opening and closing width, etc.). In the case of seat vibration, the control unit 203 specifies, for example, which seats (driver's seat, other seats, or both, etc.) to vibrate and to what extent (vibration intensity, etc.). In the case of speaking to the user, the control unit 203 specifies, for example, the speech mode for speaking (described later).

[0058] Furthermore, when selecting at least one process, information indicating the user's preferences (hereinafter also referred to as "preference information") may be taken into consideration. Preference information can be obtained by any method. For example, preference information may be generated based on the usage history of elements such as the spraying mechanism 105 and audio device 106 of the vehicle 100, feedback information indicating the user's evaluation results obtained at any time after the execution of each process, or both. Preference information may include, for example, information indicating the user's preference for fragrance or music content. The control unit 203 may obtain preference information by receiving preference information stored in the storage unit 112 of the vehicle 100 from the vehicle 100 via the network 300. Preference information may be stored in the storage unit 202. The control unit 203 may determine the user's preferences by reading the user's preference information from the storage unit 202. For example, if the control unit 203 determines that the user does not like a certain type of fragrance, the control unit 203 may exclude the fragrance spraying of that fragrance from the processing candidates. In other words, the control unit 203 can disable one or more processing candidates in relation to the user based on the user's preference information. This prevents the spraying of fragrances that the user dislikes in the vehicle 100, even if, for example, the drowsiness level is detected to be above a predetermined level in step S102. As a result, it becomes easier to provide the user of the vehicle 100 with a more satisfying travel experience. The user of the vehicle 100 can be identified by user authentication using voice data, image data, or biometric data.

[0059] Additionally or alternatively, machine learning, such as deep learning, may be used in the selection of at least one processing method. For example, suppose an analysis of the changes in a user U1's biometric data (e.g., LF / HF ratio) suggests that the user U1 was under high stress during a certain aggregation period P1, but that when fragrance X was sprayed inside the car, user U1 gradually relaxed. In such a case, a trained model can be generated by using machine learning with training data that associates the user, the changes in biometric data, and the processing actually performed. By inputting processing candidates into the generated trained model, it is possible to determine whether the processing candidate is effective in mitigating adverse effects on operation in relation to the user and user state.

[0060] The control unit 203 then transmits an instruction to the communication unit 101 of the vehicle 100 via the communication unit 201 and the network 300, including an instruction to perform at least one of the selected processes, along with the specification of the corresponding detailed function, in the instruction data. The control unit 113 of the vehicle 100 then performs at least one process and the corresponding detailed function indicated in the received instruction data in the vehicle 100. In this way, the control unit 203 can perform a predetermined process selected according to the estimated state in the vehicle 100.

[0061] Up to this point, for convenience, we have described the case where there is only one user for vehicle 100, but there may be multiple users for vehicle 100. In this case, if a predetermined process is selected based only on the user state of the driver user, there is a possibility that an undesirable process may be executed depending on the user state of the passenger users. Therefore, in order to make it easier to prevent such a situation, when there are multiple users for vehicle 100, the control unit 203 may select at least one process for each user according to the estimated state for each user.

[0062] Specifically, the control unit 203 determines whether each user is the driver or a passenger based on the position of the seat in the vehicle 100 where the user is seated. After identifying the driver and passenger, the control unit 203 determines whether each processing candidate is effective for the user and user state. This allows the control unit 203 to select different processing depending on whether a user who is presumed to be very drowsy is the driver or a passenger. If a user who is presumed to be very drowsy is a passenger (e.g., a sleeping child), executing certain processing (such as opening / closing a window or vibrating the seat) may lead to results contrary to the wishes of that user or other users (such as a guardian). As a result, it becomes difficult to provide the user with a highly satisfying travel experience. On the other hand, by not selecting certain processing when a user who is presumed to be very drowsy is a passenger, the control unit 203 can reduce the possibility of results contrary to the user's wishes. In other words, the control unit 203 can selectively disable certain processing depending on each user's role in the vehicle 100 (driver or passenger) and the user state of that user. This makes it easier to prevent undesirable processes from being executed in vehicle 100. As a result, it becomes easier to provide users of vehicle 100 with a more satisfying travel experience.

[0063] Furthermore, if at least one process involves speaking to a user, the control unit 203 may change the speech mode when speaking to the user in the vehicle 100 depending on whether there is one user or multiple users in the vehicle 100. Specifically, as a detailed function for speaking to the user, the control unit 203 specifies a different speech mode from among multiple speech modes so that the content spoken by agent 104 to the user differs depending on whether there is one user or multiple users. Multiple speech modes can be arbitrarily set, but in this embodiment, they include a mode in which speech is made from a position of empathy only for a single user (hereinafter also referred to as "empathy mode") and a mode in which speech is made from a neutral position toward multiple users (hereinafter also referred to as "neutral mode"). The empathy mode may be set by machine learning of conversations between one person and another person. The neutral mode may be set by machine learning of conversations between one person and many people. In empathy mode, speech is made with consideration only for a single user (in this case, the driver). For example, depending on the analysis of information such as the driving conditions of other vehicles around vehicle 100 or the scenery, a single user may be given a message that encourages, comforts, or calms them down. On the other hand, in neutral mode, the message is delivered with consideration not only for the driver but also for the passengers. For example, if a fight is detected, a message may be given to calm them down. Also, if silence is detected for a predetermined number of consecutive aggregation periods, a common topic of conversation may be provided to each user.

[0064] Thus, the control unit 203 can specify a speech mode as a detailed function for speaking to users based on the number of users in vehicle 100. If there is one user in vehicle 100, the control unit 203 sends an instruction to vehicle 100 to perform a speech to the user, along with the specification of the empathy mode, in the instruction data. If there are multiple users in vehicle 100, the control unit 203 sends an instruction to vehicle 100 to perform a speech to the users, along with the specification of the neutral mode, in the instruction data. The control unit 113 of vehicle 100 causes agent 104 to perform a speech in the speech mode indicated in the received instruction data. This makes it easier to provide users of vehicle 100 with a more satisfying travel experience.

[0065] As described above, the information processing device 200 according to this embodiment estimates the user's state based on user information including at least one of the following: voice data, image data, or biometric data of the user of the vehicle 100. If the information processing device 200 determines that the estimated state satisfies predetermined conditions, it executes a predetermined process in the vehicle 100 that is selected according to the estimated state.

[0066] With this configuration, by estimating the user's state in vehicle 100, predetermined processes can be proactively executed without waiting for instructions from the user. For example, if it is estimated that the user is under stress or drowsy, the system can proactively perform processes to mitigate adverse effects on driving by changing the in-vehicle environment of vehicle 100 to counteract these conditions. Therefore, it can provide a highly satisfying travel experience even to users who find conventional passive vehicle agent services unsatisfactory. Thus, the technology related to vehicle agent systems is improved in that it is easier to enhance the user experience.

[0067] While this disclosure has been described based on the drawings and embodiments, it should be noted that those skilled in the art may make various modifications and alterations based on this disclosure. Therefore, it should be noted that these modifications and alterations are within the scope of this disclosure. For example, the functions, etc., included in each component or step can be rearranged in a logically consistent manner, and multiple components or steps can be combined into one or divided into two.

[0068] For example, in one modification of this embodiment, the control unit 203 may calculate the estimated time of arrival at the destination by using driving data in addition to the user information acquired in step S100. In this case, the control unit 203 may determine the timing of execution of a predetermined process based on the calculated estimated time of arrival. The execution timing can be any timing before the estimated time of arrival. For example, the control unit 203 may add information to the instruction data indicating the timing of seat vibration for a passenger user who is presumed to be in a state of severe drowsiness. This can wake the user before arriving at the destination and facilitate disembarking after arriving at the destination.

[0069] Furthermore, for example, in the embodiments described above, it is also possible to distribute the configuration and operation of the information processing device 200 to multiple computers that can communicate with each other. Also, for example, it is possible to provide some or all of the components of the information processing device 200 in the vehicle 100. For example, a navigation device mounted in the vehicle 100 may comprise some or all of the components of the information processing device 200.

[0070] Furthermore, it is also possible to implement an embodiment in which a general-purpose computer functions as the information processing device 200 according to the above-described embodiment. Specifically, a program describing the processing content that realizes each function of the information processing device 200 according to the above-described embodiment is stored in the memory of the general-purpose computer, and the processor reads and executes the program. Therefore, this disclosure can also be implemented as a program that can be executed by a processor, or as a non-temporary computer-readable medium that stores said program. [Explanation of Symbols]

[0071] 10 Systems 100 vehicles 101 Communications Department 102 Acquisition Department 103 Imaging Unit 104 Agent 105 Spray mechanism 106 Audio equipment 107 Air conditioner 108 Window opening and closing device 109 Seat Vibration Device 110 Output section 111 Input Section 112 Storage section 113 Control Unit 200 Information Processing Devices 201 Communications Department 202 Storage section 203 Control Unit 300 Networks

Claims

1. An information processing device comprising a control unit, The control unit, Based on user information including at least one of the following: voice data, image data, or biometric data of the vehicle user, the user's state is estimated. An information processing device that, when it is determined that the estimated state satisfies predetermined conditions, executes a predetermined process selected according to the estimated state in the vehicle.

2. An information processing apparatus according to claim 1, The aforementioned predetermined condition is that the user is in a state of high stress or drowsiness, in the information processing device.

3. An information processing apparatus according to claim 1 or 2, The control unit is an information processing device that selects at least one of the following processes in the vehicle as the predetermined process: fragrance spraying, music playback, air conditioning adjustment, window opening / closing, seat vibration, or speaking to the user.

4. An information processing apparatus according to claim 3, The user of the aforementioned vehicle is multiple users, The control unit is an information processing device that selects at least one of the processes for each user according to the estimated state for each user.

5. An information processing apparatus according to claim 3, The at least one of the processes includes speaking to the user, The control unit is an information processing device that changes the speech mode depending on whether there is one or multiple users in the vehicle when speaking to the user in the vehicle.

Images