Information processing device, information processing method, and program

Abstract

One example of how to support a confused user is to encourage them to contact you for assistance. [Solution] The information processing device 10 comprises a determination unit 110 and a control unit 130. The determination unit 110 determines whether or not the user is in a confused state using information on at least one of the user's situation and the user's actions. In this embodiment, if the determination unit 110 determines that the user is in a confused state, the control unit 130 causes the voice output unit 290 to output an audio message for inquiry.

Description

Technical Field

[0001] The present invention relates to an information processing apparatus, an information processing method, and a program.

Background Art

[0002] Devices that can respond to user inquiries and the like have been developed.

[0003] Patent Document 1 discloses a device that generates and outputs information for causing a character to speak or the like based on the result of estimating the user's state.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, in the technology of Patent Document 1, when the user is confused for some reason, an inquiry cannot be made.

[0006] An example of the problem to be solved by the present invention is to assist the user by generating an inquiry to the user in a confused state.

Means for Solving the Problems

[0007] The invention according to claim 1 a determination unit that determines whether or not the user is in a confused state using information related to at least one of the user's situation and the user's operation; and a control unit that causes the voice output unit to execute a voice output for an inquiry when the determination unit determines that the user is in a confused state. is an information processing apparatus.

[0008] The invention described in claim 12 is, An information processing method performed by one or more computers, A determination step of determining whether the user is in a confused state using information on the user's situation and at least one of the user's actions, The control step includes, if the determination step determines that the user is in a confused state, causing the voice output unit to output an audio message for inquiry. It is an information processing method.

[0009] The invention described in claim 13 is, Computers, A determination means for determining whether or not the user is in a confused state, using information on the user's situation and at least one of the user's actions, If the determination means determines that the user is in a state of confusion, the control means causes the voice output unit to output an audio message for inquiry. To make it function as It is a program. [Brief explanation of the drawing]

[0010] [Figure 1] This figure illustrates an overview of an information processing device according to the first embodiment. [Figure 2] This figure illustrates an overview of the information processing method according to the first embodiment. [Figure 3] This figure illustrates the functional configuration of the information processing device and system according to the first embodiment. [Figure 4] This diagram illustrates a computer used to implement an information processing device. [Figure 5] This is a flowchart illustrating the processing flow performed by the information processing device according to the first embodiment. [Figure 6] This figure illustrates the functional configuration of the information processing device and system according to the second embodiment. [Figure 7]It is a figure illustrating the functional configuration of an information processing apparatus and a system according to the third embodiment. [Figure 8] It is a flowchart showing a first example of processing executed by an information processing apparatus according to the third embodiment. [Figure 9] It is a flowchart showing a second example of processing executed by an information processing apparatus according to the third embodiment.

Mode for Carrying Out the Invention

[0011] Hereinafter, embodiments of the present invention will be described with reference to the drawings. In all the drawings, the same components are denoted by the same reference numerals, and the description will be omitted as appropriate.

[0012] (First Embodiment) FIG. 1 is a diagram illustrating an overview of an information processing apparatus 10 according to the first embodiment. The information processing apparatus 10 according to the present embodiment includes a determination unit 110 and a control unit 130. The determination unit 110 determines whether to perform voice output for an inquiry to the user. When the determination unit 110 determines that voice output for an inquiry is to be performed, the control unit 130 causes the voice output unit 290 to perform the voice output for the inquiry. Details will be described below.

[0013] In the present embodiment, the determination unit 110 determines whether the user is in a confused state by using information related to at least one of the user's situation and the user's operation. In the present embodiment, when the determination unit 110 determines that the user is in a confused state, the control unit 130 causes the voice output unit 290 to perform the voice output for the inquiry.

[0014] That is, in the present embodiment, the determination unit 110 determines whether to perform voice output for an inquiry to the user by determining whether the user is in a confused state.

[0015] FIG. 2 is a diagram illustrating an overview of the information processing method according to the present embodiment. The information processing method according to the present embodiment is executed by one or more computers. The information processing method according to the present embodiment includes a determination step S10 and a control step S20. In the determination step S10, one or more computers determine whether the user is in a confused state by using information related to at least one of the user's situation and the user's operation. In the control step S20, when it is determined in the determination step S10 that the user is in a confused state, one or more computers cause the voice output unit 290 to execute a voice output for inquiry.

[0016] The information processing method according to the present embodiment can be executed by the information processing apparatus 10 according to the present embodiment.

[0017] FIG. 3 is a diagram illustrating a functional configuration of the information processing apparatus 10 and the system 20 according to the present embodiment. The system 20 according to the present embodiment includes the information processing apparatus 10. In the example of FIG. 3, the system 20 further includes one or more sensors 210, a microphone 240, a voice output unit 290, a light emitting unit 280, and a fragrance output unit 270. However, the system 20 may include at least the information processing apparatus 10 and the voice output unit 290.

[0018] The voice output unit 290 is, for example, a speaker or earphone. The voice output from the voice output unit 290 is recognizable by the user. The voice output for inquiry is, for example, an output of a message that is a question to the user. Specific examples of the voice output for inquiry include outputs of messages such as "What's wrong?", "Do you have any problems?", and "Do you need help?". The voice output for inquiry is hereinafter also referred to as "inquiry output".

[0019] Information regarding at least one of the user's situation and / or user's actions is, in one example, sensor information obtained from one or more sensors that generate information regarding at least one of the user's situation and / or user's actions. The determination unit 110 can acquire sensor information from one or more sensors 210 and use it for determination. The determination unit 110 can determine whether or not the user is in a confused state based on whether or not the sensor information satisfies predetermined conditions. For example, it can determine whether or not the user is in a confused state by comparing the sensor information with predetermined criteria. The determination unit 110 may also use a combination of sensor information from multiple sensors 210 for determination.

[0020] As another example, the determination unit 110 can input sensor information into a machine learning model and use the obtained output data to determine whether or not the user is in a confused state. The output data of the machine learning model may include, for example, information indicating the likelihood that the user is in a confused state. If the likelihood that the user is in a confused state is above a predetermined threshold, the determination unit 110 can determine that the user is in a confused state. On the other hand, if the likelihood that the user is in a confused state is below a predetermined threshold, the determination unit 110 can determine that the user is not in a confused state.

[0021] Alternatively, the output data of the machine learning model may include, for example, information indicating the user's level of confusion. If the user's level of confusion is above a predetermined threshold, the determination unit 110 can determine that the user is confused. On the other hand, if the user's level of confusion is below a predetermined threshold, the determination unit 110 can determine that the user is not confused.

[0022] The criteria or thresholds described above may be set for each user. The determination unit 110 identifies the user, for example, by facial recognition or by communicating with a device (such as a smartphone) that the user possesses. Then, it may determine whether or not the user is in a state of confusion using the criteria or thresholds corresponding to the identified user.

[0023] The information processing device 10 may be a portable device or a stationary device. The information processing device 10 may be, for example, part of a smart speaker. However, the information processing device 10 may be implemented as a server machine or the like located at a location separate from the user.

[0024] The control unit 130 may be capable of processing for user interaction. This user interaction processing can be performed using a machine learning model. In addition to outputting audio to the audio output unit 290, the control unit 130 may also display information on a display. The information processing device 10 may be connected to a microphone 240 and capable of acquiring user speech. After outputting an inquiry, the control unit 130 can proceed with the interaction with the user, present various information to resolve user confusion, or control devices around the user.

[0025] Sensor 210 detects at least one of the user's situation and / or user's actions. Sensor information is information indicating the detection results of sensor 210. Examples of the user's situation include facial expressions and vital signs. Examples of vital signs include body temperature, surface temperature, pulse rate, sweating status, pupil condition, and blinking. Examples of user actions include changes in posture, hand movements, foot movements, and vocalizations. By using vital sign information, it is possible to estimate the user's emotions such as anxiety or unease, and to accurately determine whether the user is in a state of confusion.

[0026] Examples of sensors 210 include imaging devices, temperature sensors, pulse sensors, sweat sensors, and acceleration sensors. Examples of imaging devices include visible light cameras, infrared cameras, and thermal cameras. Sensor information may include vital information indicating the user's vitals as described above. That is, the determination unit 110 may use the user's vital information to determine whether or not the user is in a state of distress. The determination unit 110 may acquire vital information from wearable devices worn by the user, such as smartwatches, smart wristbands, smart rings, and smart glasses.

[0027] If one or more sensors 210 include an imaging device for detecting the direction of the user's gaze, the determination unit 110 uses the information generated by the imaging device for detecting the direction of the user's gaze to determine whether the user's gaze meets predetermined gaze conditions. That is, the determination unit 110 can determine that the user is in a confused state when the user's gaze meets predetermined gaze conditions. Specifically, the determination unit 110 can determine that the user is in a confused state when the user's gaze moves beyond a predetermined threshold. The imaging device outputs at least an image or video of the user's eyes as sensor information. The movement of the user's gaze can be identified by analyzing the image or video obtained by the imaging device. Alternatively, the determination unit 110 may determine whether the user is in a confused state by inputting the image or video into a trained model.

[0028] One or more sensors 210 may include an imaging device for detecting user movements. The determination unit 110 may use information generated by the imaging device for detecting user movements to determine whether or not the user is in a confused state. The determination unit 110 can determine that the user is in a confused state when the user's movements meet predetermined operating conditions. Specifically, the determination unit 110 can determine that the user is in a confused state when the direction of the user's head moves beyond a predetermined standard. The movement of the user's head can be identified by analyzing images or videos obtained from the imaging device. Preferably, the images or videos show at least a part of the user's body. The determination unit 110 may also determine whether or not the user is in a confused state by inputting images or videos into a trained model.

[0029] When the user's gaze moves beyond a predetermined threshold, or when the user's head direction moves beyond a predetermined threshold, the control unit 130 may output a message to the voice output unit 290 as an inquiry output, such as "Are you looking for something?", asking if the user is looking for something. For example, in a conversation with the user that takes place after the inquiry output, the information processing device 10 may, upon receiving instructions from the user, identify the location of the item the user is looking for based on the room conditions captured by the imaging device and inform the user. If the user is looking for a remote control for an electrical appliance, the information processing device 10 may send a control signal to the appliance to operate it on their behalf.

[0030] The control unit 130 may further cause at least one of the light-emitting unit 280 and the fragrance output unit 270 to perform a predetermined operation based on the determination result of the determination unit 110. That is, if the determination unit 110 determines that the user is in a state of confusion, the control unit 130 may cause the light-emitting unit 280 to output light that calms the user. Also, if the determination unit 110 determines that the user is in a state of confusion, the control unit 130 may cause the fragrance output unit 270 to emit a fragrance that calms the user. Examples of the light-emitting unit 280 include lighting devices, decorative devices, and light-emitting devices. Examples of the fragrance output unit 270 include aroma diffusers.

[0031] The hardware configuration of the information processing device 10 is described below. Each functional component of the information processing device 10 (determination unit 110 and control unit 130) may be implemented by hardware that realizes each functional component (e.g., hardwired electronic circuits), or by a combination of hardware and software (e.g., a combination of electronic circuits and a program that controls them). The case in which each functional component of the information processing device 10 is implemented by a combination of hardware and software will be further explained below.

[0032] Figure 4 illustrates a computer 1000 for implementing the information processing device 10. Computer 1000 is any computer. For example, computer 1000 could be an SoC (System on Chip), a Personal Computer (PC), a server machine, a tablet terminal, or a smartphone. Computer 1000 may be a dedicated computer designed to implement the information processing device 10, or it may be a general-purpose computer. Furthermore, the information processing device 10 may be implemented by a single computer 1000, or by a combination of multiple computers 1000.

[0033] Computer 1000 includes a bus 1020, a processor 1040, memory 1060, a storage device 1080, an input / output interface 1100, and a network interface 1120. Bus 1020 is a data transmission path for the processor 1040, memory 1060, storage device 1080, input / output interface 1100, and network interface 1120 to send and receive data to and from each other. However, the method of connecting the processor 1040 and the other components is not limited to bus connection. The processor 1040 is a variety of processor such as a CPU (Central Processing Unit), GPU (Graphics Processing Unit), or FPGA (Field-Programmable Gate Array). Memory 1060 is a main memory device implemented using RAM (Random Access Memory), etc. Storage device 1080 is an auxiliary storage device implemented using a hard disk, SSD (Solid State Drive), memory card, or ROM (Read Only Memory), etc.

[0034] The input / output interface 1100 is an interface for connecting the computer 1000 with input / output devices. For example, input devices and output devices are connected to the input / output interface 1100. The method by which the input / output interface 1100 connects to the input devices and output devices may be wireless or wired.

[0035] The network interface 1120 is an interface for connecting the computer 1000 to a network. Examples of such networks include LANs (Local Area Networks) and WANs (Wide Area Networks). The network interface 1120 may connect to the network via a wireless connection or a wired connection.

[0036] In this embodiment, one or more sensors 210, a microphone 240, an audio output unit 290, a light-emitting unit 280, and a scent output unit 270 are connected to the information processing device 10 via an input / output interface 1100 or a network interface 1120.

[0037] The storage device 1080 stores program modules that realize each functional component of the information processing device 10. The processor 1040 reads these program modules into the memory 1060 and executes them to realize the functions corresponding to each program module.

[0038] Figure 5 is a flowchart illustrating the processing flow executed by the information processing device 10 according to this embodiment. In step S101, the determination unit 110 acquires sensor information from one or more sensors 210. Step S101 is executed at predetermined timings. For example, the determination unit 110 executes step S101 at predetermined intervals. That is, the flow shown in Figure 5 is executed repeatedly. It is preferable that the determination unit 110 acquires the information generated by the sensors 210 in real time.

[0039] In step S102, following step S101, the determination unit 110 determines whether or not to output an inquiry. In this embodiment, the determination unit 110 determines whether or not to output an audio inquiry to the user by determining whether or not the user is in a confused state. If the determination unit 110 determines that the user is in a confused state, that is, determines to output an inquiry (Yes in step S102), the control unit 130 causes the audio output unit 290 to execute the inquiry (step S103). On the other hand, if the determination unit 110 determines that the user is not in a confused state, that is, determines not to output an inquiry (No in step S102), the control unit 130 does not cause the audio output unit 290 to execute the inquiry and terminates the processing based on the sensor information.

[0040] Furthermore, even if the information processing device 10 is a smart speaker or the like, the user does not need to utter a so-called wake word. In addition, the information processing device 10 can proactively execute query outputs based on sensor information, even when not interacting with the user.

[0041] In this embodiment, several examples of determination methods executed by the determination unit 110 have been described. These determination methods may be used individually or in combination of two or more. When two or more determination methods are used in combination, if the determination unit 110 determines that an inquiry output should be made using at least one of the determination methods, the control unit 130 causes the audio output unit 290 to execute the inquiry output.

[0042] As described above, according to this embodiment, when the determination unit 110 determines that it is necessary to output an audio message for an inquiry, the control unit 130 causes the audio output unit 290 to execute the audio message for the inquiry. Therefore, by generating an inquiry for a confused user, it is possible to support the user.

[0043] (Second embodiment) The information processing device 10 and system 20 according to the second embodiment are the same as the information processing device 10 and system 20 according to the first embodiment, except for the points described below.

[0044] An overview of the information processing device 10 according to the second embodiment is shown in Figure 1, similar to the first embodiment. The information processing device 10 according to this embodiment includes a determination unit 110 and a control unit 130. In this embodiment, the user is riding on a mobile vehicle. The determination unit 110 according to this embodiment can further determine whether the user is in a confused state using information about at least one of the conditions of the mobile vehicle on which the user is riding and the conditions around the mobile vehicle. This will be explained in detail below.

[0045] The determination unit 110 according to this embodiment may determine whether or not to output voice for inquiry to the user using at least one of sensor information and control information in the mobile body. In this embodiment, the sensor information is information obtained from one or more sensors that generate information about at least one of the following: the status of the mobile body on which the user is riding, the status around the mobile body, the user's status, and the user's actions.

[0046] Figure 6 is a diagram illustrating the functional configuration of the information processing device 10 and system 20 according to this embodiment. The system 20 according to this embodiment includes the information processing device 10. In the example in Figure 6, the information processing device 10 further includes a display control unit 150. In the example in Figure 6, the system 20 includes the information processing device 10, one or more sensors 210, in-vehicle equipment 220, mobile unit control unit 230, microphone 240, voice output unit 290, light-emitting unit 280, scent output unit 270, and display 260. However, the system 20 only needs to include the information processing device 10 and the voice output unit 290. Examples of the voice output unit 290 and examples of query output are as described in the first embodiment.

[0047] As described above, in this embodiment, the user is a person riding in a mobile vehicle. The mobile vehicle is not particularly limited, but could be a bicycle, motorcycle, automobile, or other vehicle. The user may or may not be the driver of the mobile vehicle.

[0048] In this embodiment, the determination unit 110 may acquire at least one of sensor information and control information and use it for determination. The sensor 210 according to this embodiment generates information about at least one of the following: the status of the vehicle the user is riding in, the surroundings of the vehicle, the user's status, and the user's actions. Examples of the user's status and user actions are as described in the first embodiment. Examples of the vehicle's status include the vehicle's steering status, brake operation status, accelerator operation status, and the operation status of various driving assistance functions. The determination unit 110 can also acquire information about the vehicle's actions as control information, as will be described later. Examples of the surroundings of the vehicle include the distance between the vehicle the user is riding in and surrounding vehicles (such as the distance between vehicles), the brightness around the vehicle, and the presence of pedestrians and the movement of surrounding vehicles.

[0049] Examples of control information in a moving vehicle include information related to steering control, brake control, accelerator control, driver assistance functions, and information indicating the operation of in-vehicle equipment.

[0050] The determination unit 110 can acquire control information from the mobile control unit 230 of the mobile vehicle in which the user is riding. The determination unit 110 may also acquire information indicating the operation of in-vehicle equipment from the in-vehicle equipment 220. In addition, the determination unit 110 may acquire sensor information indicating the status of the mobile vehicle or the status of the surrounding area via the mobile control unit 230. The mobile control unit 230 is a device that controls the mobile vehicle, and is, for example, an Electronic Control Unit (ECU).

[0051] The determination unit 110 can determine whether or not to output an audio message to the user based on whether at least one of the sensor information and the control information satisfies a predetermined condition. For example, it can determine whether or not to output an audio message to the user by comparing at least one of the sensor information and the control information with a predetermined criterion. In this embodiment, the determination unit 110 may determine whether or not to output an audio message to the user by determining whether or not the user is in a confused state, similar to the first embodiment. The determination unit 110 may use a combination of sensor information from multiple sensors 210 for determination, or it may use a combination of sensor information and control information for determination.

[0052] The determination unit 110 may input at least one of the sensor information and control information into a machine learning model and use the obtained output data to determine whether or not to output voice for a user inquiry. The output data of the machine learning model may include, for example, information indicating the likelihood that the user is in a confused state. If the likelihood that the user is in a confused state is above a predetermined threshold, the determination unit 110 can determine that the user is in a confused state. On the other hand, if the likelihood that the user is in a confused state is below a predetermined threshold, the determination unit 110 can determine that the user is not in a confused state.

[0053] Alternatively, the output data of the machine learning model may include, for example, information indicating the user's level of confusion. If the user's level of confusion is above a predetermined threshold, the determination unit 110 can determine that the user is confused. On the other hand, if the user's level of confusion is below a predetermined threshold, the determination unit 110 can determine that the user is not confused.

[0054] Users riding in a moving vehicle are prone to situations that can cause confusion, such as getting lost, encountering unclear navigation, or experiencing unusual circumstances in their surroundings. According to the information processing device 10 of this embodiment, it is possible to assist users in a confused state while riding in a moving vehicle by generating inquiries.

[0055] Similar to the first embodiment, the criteria or thresholds may be set for each user. This increases the opportunities for inquiries for users who are unfamiliar with driving. On the other hand, it reduces the burden caused by frequent inquiries for users who are familiar with driving. In other words, the frequency of inquiries can be optimized for each user.

[0056] For example, if, based on sensor information and control information, at least one of the following is detected—that the moving object has swayed from side to side, that the distance between vehicles has shortened, that the brakes have been applied suddenly, that the lane keeping function has been activated (i.e., that the moving object is about to deviate from its lane), that the driver has applied the brakes while a driving assistance function or an autonomous driving function is in operation, that the brightness around the moving object has increased by a predetermined rate of change or more (i.e., that it has suddenly become dazzling), or that pedestrians or other moving objects around the moving object have suddenly changed their movement—the determination unit 110 will determine to output an audio message for inquiry. This is because these situations may be caused by the user's confusion or may be events that could cause the user confusion.

[0057] The control unit 130 may output a message to the voice output unit 290 as an inquiry output, which may include specific suggestions such as, "Shall we pull over to the side of the road?". In addition, the information processing device 10 may, in the interaction with the user that takes place after the inquiry output, send information to the mobile unit control unit 230 to stop the mobile unit, or activate the mobile unit's driving assistance function or automatic driving function, if it receives instructions from the user.

[0058] As described above, the control information may include information indicating the content of operations performed on the in-vehicle equipment 220 installed on the mobile vehicle. That is, the determination unit 110 may use the information indicating the content of operations performed on the in-vehicle equipment installed on the mobile vehicle to determine whether or not the user is in a confused state. Examples of in-vehicle equipment 220 include navigation systems, audio systems, air conditioners, etc. For example, a user may operate the in-vehicle equipment 220 while stopped at a red light, but the light turns green midway through the operation, requiring the user to start moving. In such cases, the determination unit 110 may determine to output an inquiry if it detects that operations on the in-vehicle equipment 220 have been performed intermittently.

[0059] In addition, users may not fully understand how to operate the in-vehicle equipment 220, and may repeatedly switch between similar operation screens on the navigation system and audio system. Therefore, if the determination unit 110 detects that operations on the in-vehicle equipment 220 have been performed consecutively for a predetermined number of times or more, it may decide to output an inquiry.

[0060] Similar to the first embodiment, one or more sensors 210 may include an imaging device for detecting user actions. The determination unit 110 may use the information generated by the imaging device for detecting user actions to determine whether or not the user is in a confused state. For example, the determination unit 110 may use the output of the imaging device to detect actions of operating an in-vehicle device 220 installed on a moving object. Similar to the example described above, if the determination unit 110 detects that the user has intermittently performed actions of operating the in-vehicle device 220, or if it detects that the user has performed operations on the in-vehicle device 220 for a predetermined number of consecutive times or more, the determination unit 110 may decide to issue an inquiry output.

[0061] Furthermore, in these examples related to the operation of the in-vehicle equipment 220, the control unit 130 may output a message regarding voice operation, such as "Do you want to enable voice operation?" or "What operation would you like to perform?".

[0062] Similar to the first embodiment, one or more sensors 210 may include an imaging device for detecting the direction of the user's gaze. The determination unit 110 may use the information generated by the imaging device for detecting the direction of the user's gaze to determine whether the user's gaze meets predetermined gaze conditions. That is, the determination unit 110 may determine that the user is in a state of confusion if the user's gaze meets predetermined gaze conditions. Situations that cause or cause confusion in the user include, for example, a situation where an unfamiliar display appears on the meter panel of a moving vehicle, but the user does not understand the meaning of the display; a situation where pedestrians around the moving vehicle or the vehicle itself are moving unnaturally; a situation where the user is concerned about a child in the back seat; or a situation where the user is trying to fasten a seat belt but cannot find the buckle. Therefore, the determination unit 110 may determine that the user is in a state of confusion if the user's gaze looks at a predetermined part a predetermined number of times or more within a predetermined time.

[0063] The information processing device 10 may, depending on the content of the interaction with the user following the query output, present necessary information to the user or perform control such as stopping the moving object.

[0064] The information processing device 10 may further include a display control unit 150 that displays route guidance information on the display 260. The display 260 may have a touch panel function. The determination unit 110 may determine that the user is in a state of confusion when it detects that the user's gaze has been directed at the display 260 a predetermined number of times or more, using information generated by an imaging device for detecting the direction of the user's gaze. For example, the user may not be able to fully grasp the content of the route guidance displayed on the display 260 or the timing of turns. Also, the user may miss voice guidance for navigation, have difficulty reading road signs, or find the display 260 difficult to see due to light reflection. In these cases, it is expected that the user will look at the display 260 many times. Therefore, the user's state of confusion can be determined based on the number of times the user's gaze has been directed at the display 260. For example, the determination unit 110 may treat the number of times the user has looked at the display 260 within a predetermined time as the degree of confusion.

[0065] If the control unit 130 is capable of processing for interaction with the user, after executing an inquiry output, the control unit 130 may control the mobile vehicle to assist the user's driving based on the interaction with the user driving the mobile vehicle. For example, a user may have difficulty parking. Therefore, if the determination unit 110 detects that the mobile vehicle has moved forward and backward more than a predetermined number of times based on the control information, it may decide to output an inquiry. At this time, the control unit 130 may output a message that includes a control suggestion such as, "Do you want to activate the parking assist function?" The user may respond, for example, "Yes." When the control unit 130 receives such an affirmative response from the user, it may send information to the mobile vehicle control unit 230 for performing control based on the suggestion, that is, information for activating the parking assist function.

[0066] As another example, if the determination unit 110 detects a lateral sway of the moving object, the control unit 130 may suggest activating the lane-keeping function. If the determination unit 110 detects that pedestrians or other moving objects around the moving object are moving unnaturally, the control unit 130 may suggest setting a longer following distance in the driving assistance function or the automated driving function.

[0067] The hardware configuration of the computer implementing the information processing device 10 according to this embodiment is shown, for example, in Figure 4, similar to the first embodiment. However, the storage device 1080 of the computer 1000 implementing the information processing device 10 according to this embodiment may further store program modules that implement the functions of the display control unit 150 according to this embodiment.

[0068] The information processing device 10 may be installed on a mobile device, or it may be implemented on a server machine or the like located away from the mobile device. The information processing device 10 may also be a navigation device. In that case, the information processing device 10 and the in-vehicle equipment 220 may be integrated. As another example, the information processing device 10 may be a smartphone, or it may be implemented as a combination of a smartphone and a server machine.

[0069] In this embodiment, one or more sensors 210, in-vehicle equipment 220, mobile control unit 230, microphone 240, display 260, audio output unit 290, light-emitting unit 280, and scent output unit 270 are connected to the information processing device 10 via an input / output interface 1100 or a network interface 1120.

[0070] The processing flow executed by the information processing device 10 according to this embodiment is illustrated by Figure 5, similar to the first embodiment. However, in this embodiment, in step S101, the determination unit 110 acquires at least one of the sensor information and the control information. It is preferable that the determination unit 110 acquires at least one of the sensor information generated by the sensor 210 and the control information generated by the mobile body control unit 230 in real time.

[0071] In this embodiment, in step S102, the determination unit 110 determines whether or not to output an inquiry using at least one of the acquired sensor information and control information.

[0072] In this embodiment, several examples of determination methods executed by the determination unit 110 have been described. These determination methods may be used individually or in combination of two or more. When two or more determination methods are used in combination, if the determination unit 110 determines that an inquiry output should be made using at least one of the determination methods, the control unit 130 causes the audio output unit 290 to execute the inquiry output.

[0073] According to this embodiment, the same actions and effects as in the first embodiment can be obtained.

[0074] (Third embodiment) Figure 7 illustrates the functional configuration of the information processing device 10 and system 20 according to the third embodiment. The information processing device 10 and system 20 according to this embodiment are the same as the information processing device 10 and system 20 according to the second embodiment, except for the points described below.

[0075] The information processing device 10 according to this embodiment further comprises a position determination unit 170. The position determination unit 170 determines whether the position of the mobile body on which the user is riding satisfies predetermined position conditions. Preferably, the determination unit 110 according to this embodiment determines whether the user is in a confused state using information regarding at least one of the following: the state of the mobile body, the state of the mobile body, the state of the user, and the user's actions, at the time when the position of the mobile body satisfies the predetermined position conditions.

[0076] Users are particularly likely to become confused when traveling on unfamiliar or new roads in their vehicle. On the other hand, users are less likely to become confused on roads they travel daily for commuting or within their usual living area. Therefore, a target location may be defined, and the control unit 130 may only cause the voice output unit 290 to issue an inquiry output if the vehicle the user is riding in is at that location (the target location may be an area of ​​a certain size). In other words, if the vehicle the user is riding in is not at the target location, the determination unit 110 does not need to determine whether or not to issue an inquiry output, and the control unit 130 does not need to cause the voice output unit 290 to issue an inquiry output regardless of the result of the determination made by the determination unit 110.

[0077] The location information of a moving object can be determined by the Global Positioning System (GPS). The location determination unit 170 can acquire the location information of a moving object from a GPS module installed on the moving object.

[0078] Target locations are pre-registered by the user, for example. The user can perform an input operation to the information processing device 10 to specify an area or an area or route. The user may specify an area or route to exclude, or an area or route to include. The location determination unit 170 obtains the user's input result and identifies the target locations based on the input result. For example, if the user inputs an area or route to exclude, the location determination unit 170 will consider all locations outside that area or route as target locations. If the user specifies an area or route to include, the location determination unit 170 will consider all locations within that area or along that route as target locations. In other words, typically multiple locations are set as target locations.

[0079] In addition, the location determination unit 170 may identify target locations based on the movement history of the moving object. For example, the location determination unit 170 may identify a location as a target location that the moving object has never passed through in its movement history. The location determination unit 170 may identify a location as a target location that the moving object has passed through a predetermined number of times or less in its movement history. The location determination unit 170 may identify a location as a target location that the moving object has not passed through even once in the most recent predetermined number of days in its movement history. In other words, the target locations are locations that the user is less familiar with than other locations.

[0080] The location determination unit 170 can identify a target location at a predetermined timing. For example, the location determination unit 170 may identify a target location at a predetermined time each day based on the movement history at that time. Alternatively, the location determination unit 170 may identify a target location at the time the information processing device 10 is first started up during the day, based on the movement history at that time.

[0081] The position determination unit 170 acquires the position information of the moving object in real time. The position determination unit 170 also determines whether the position of the moving object, identified by the position information, is one of a plurality of target points. If the position of the moving object is a target point and the determination unit 110 determines that an inquiry output should be made, the control unit 130 causes the voice output unit 290 to make the inquiry output. Here, it is preferable that the determination unit 110 determines whether or not to make a voice output for the inquiry using at least one of the sensor information and control information generated at the timing when the position of the moving object satisfies predetermined position conditions, that is, when the moving object is located at a target point. However, the determination unit 110 may also determine whether or not to make a voice output for the inquiry using at least one of the sensor information and control information generated immediately after the position determination unit 170 determines that the moving object is located at a target point (for example, within 1 second).

[0082] Figure 8 is a flowchart showing a first example of the processing performed by the information processing device 10 according to this embodiment. The position determination unit 170 executes step S201 at predetermined intervals. That is, the flow shown in Figure 8 is executed repeatedly. Multiple target points are predetermined. In step S201, the position determination unit 170 acquires the position information of the moving object and determines whether the moving object is located at a target point. If the moving object is located at a target point (Yes in step S201), the determination unit 110 executes step S202. Steps S202 to S204 are the same as steps S101 to S103 in Figure 5. If the moving object is not located at a target point (No in step S201), processing for that position information is terminated. However, step S201 may be executed after step S202 or after step S203.

[0083] Figure 9 is a flowchart showing a second example of processing performed by the information processing device 10 according to this embodiment. In the second example, when the moving object is located at the target location, the determination unit 110 switches the determination criteria to increase the frequency of query output compared to when the moving object is not located at the target location.

[0084] For example, the position determination unit 170 determines whether or not the moving object is located at the target location. Regardless of the determination result of the position determination unit 170, the determination unit 110 uses information on at least one of the following: the status of the moving object, the surrounding environment of the moving object, the user's status, and the user's actions to determine the user's level of confusion. If the moving object is located at the target location, the determination unit 110 determines that the user is confused if the level of confusion is equal to or greater than the first criterion. On the other hand, if the moving object is not located at the target location, the determination unit 110 determines that the user is confused if the level of confusion is equal to or greater than the second criterion. Here, the first criterion is lower than the second criterion. This will be explained in detail below.

[0085] Step S301 is the same as step S201 described above. Similar to Figure 8, the flow shown in Figure 9 is executed repeatedly. If it is determined that the moving object is located at the target point (Yes in step S301), the determination unit 110 acquires at least one of the sensor information and the control information (step S302). Then, in step S303, the determination unit 110 uses the acquired information to determine the degree of confusion. The method for determining the degree of confusion is as described in the first or second embodiment. Then, the determination unit 110 compares the determined degree of confusion with a predetermined first criterion. If the degree of confusion is equal to or greater than the first criterion, the determination unit 110 determines to issue an inquiry output (Yes in step S303). Then, the control unit 130 causes the voice output unit 290 to execute the inquiry output (step S304).

[0086] On the other hand, the determination unit 110 determines that if the level of confusion is less than the first criterion, it will not issue an inquiry output (No. in step S303). Then, the control unit 130 does not cause the voice output unit 290 to execute the inquiry output and terminates the processing based on the acquired information.

[0087] In step S301, if it is determined that the moving object is not located at the target point (No in step S301), the determination unit 110 acquires at least one of the sensor information and the control information (step S305). Then, in step S306, the determination unit 110 uses the acquired information to determine the degree of confusion. Furthermore, the determination unit 110 compares the determined degree of confusion with a predetermined second criterion. If the degree of confusion is equal to or greater than the second criterion, the determination unit 110 determines to issue an inquiry output (Yes in step S306). Then, the control unit 130 causes the voice output unit 290 to execute the inquiry output (step S304).

[0088] On the other hand, the determination unit 110 determines that if the level of confusion is less than the second criterion, it will not output an inquiry (No. in step S306). Then, the control unit 130 does not have the voice output unit 290 execute the inquiry output and terminates the processing based on the acquired information.

[0089] The information processing device 10 may be configured to allow setting a target location for each user. That is, a storage device accessible from the location determination unit 170 (for example, the storage device 1080 of the computer 1000 that implements the information processing device 10) may store the movement history of each user. The location determination unit 170 identifies the user, for example, by facial recognition or by communicating with a terminal (smartphone, etc.) that the user possesses. The location determination unit 170 then uses the movement history of the identified user to determine the target location.

[0090] Furthermore, the criteria such as the first and second criteria mentioned above may be set for each user. The determination unit 110 may identify the user in the same manner as described above, or obtain the identification result performed by the location determination unit 170, and use the criteria corresponding to the identified user to determine whether the user is in a confused state. In this way, it is possible to increase the opportunities for inquiries to users who are unfamiliar with driving. On the other hand, it is possible to reduce the burden caused by frequent inquiries to users who are familiar with driving. In other words, the frequency of inquiries can be optimized for each user.

[0091] The hardware configuration of the computer implementing the information processing device 10 according to this embodiment is shown, for example, in Figure 4, similar to the first embodiment. However, the storage device 1080 of the computer 1000 implementing the information processing device 10 according to this embodiment may further store program modules that implement the functions of the position determination unit 170 according to this embodiment.

[0092] As described above, this embodiment provides the same functions and effects as the first embodiment. In addition, according to this embodiment, the position determination unit 170 determines whether the position of the mobile body on which the user is riding satisfies predetermined position conditions. Therefore, it is possible to suppress query output on familiar roads, etc.

[0093] The embodiments of the present invention have been described above with reference to the drawings, but these are merely examples of the present invention, and various other configurations can also be adopted.

[0094] Furthermore, while the flowcharts used in the above description show multiple steps (processes) in sequence, the execution order of the steps performed in each embodiment is not limited to the order in which they are described. In each embodiment, the order of the illustrated steps can be changed to the extent that it does not impede the content. Also, the above embodiments can be combined to the extent that their contents do not conflict.

[0095] Examples of reference formats are provided below. 1. A determination unit that determines whether or not the user is in a confused state using information on the user's situation and at least one of the user's actions, The system includes a control unit that, when the determination unit determines that the user is in a confused state, causes the voice output unit to execute an audio output for inquiry. Information processing device. 2. In the information processing device described in 1., The determination unit further determines whether the user is in a state of confusion, using information regarding at least one of the following: the condition of the vehicle the user is riding in, and the condition of the area surrounding the vehicle. Information processing device. 3. In the information processing device described in 2. The system further includes a position determination unit that determines whether the position of the moving body satisfies predetermined positional conditions. The determination unit determines whether the user is in a confused state using information regarding at least one of the following: the state of the moving body, the state of the moving body's surroundings, the state of the user, and the user's actions, at the time when the position of the moving body satisfies the predetermined positional conditions. Information processing device. 4. In the information processing device described in 3. The position determination unit determines whether the moving object is located at the target point, The determination unit, Using information relating to at least one of the following: the status of the moving object, the surrounding environment of the moving object, the user's status, and the user's actions, the degree of the user's confusion is identified. When the moving object is located at the target location, if the degree of confusion is equal to or greater than the first criterion, it is determined that the user is in a state of confusion. If the moving object is not located at the target point, and the degree of confusion is equal to or greater than the second criterion, the user is determined to be in a state of confusion. The first criterion is lower than the second criterion. Information processing device. 5. In the information processing device described in any one of 2. to 4., The determination unit further determines whether the user is in a state of confusion using information indicating the content of operations performed on the in-vehicle equipment provided on the mobile vehicle. Information processing device. 6. In the information processing device described in any one of items 2 to 5, The control unit, Processing for interaction with the aforementioned user is possible, After executing the voice output for the aforementioned inquiry, the system controls the mobile vehicle to assist the user's driving based on the interaction with the user who is operating the mobile vehicle. Information processing device. 7. In the information processing device described in any one of items 2 to 6, It further includes a display control unit that displays route guidance information on a display, The determination unit determines that the user is in a state of confusion when it detects that the user's gaze has been directed at the display a predetermined number of times or more, using information generated by an imaging device for detecting the direction of the user's gaze. Information processing device. 8. In the information processing device described in any one of items 1 to 6, The determination unit uses information generated by the imaging device for detecting the direction of the user's gaze to determine whether the user's gaze meets predetermined gaze conditions. Information processing device. 9. In the information processing device described in any one of items 1 to 8, The determination unit uses information generated by the imaging device for detecting the user's actions to determine whether or not the user is in a confused state. Information processing device. 10. In the information processing device described in any one of items 1 to 9, The determination unit uses the user's vital information to determine whether or not the user is in a state of confusion. Information processing device. 11. In the information processing device described in any one of items 1 to 10, Based on the determination result of the determination unit, the control unit causes at least one of the light-emitting unit and the fragrance output unit to perform a predetermined operation. Information processing device. 12. An information processing method performed by one or more computers, A determination step of determining whether the user is in a confused state using information on the user's situation and at least one of the user's actions, The control step includes, if the determination step determines that the user is in a confused state, causing the voice output unit to output an audio message for inquiry. Information processing methods. 13. Computers, A determination means for determining whether or not the user is in a confused state, using information on the user's situation and at least one of the user's actions, If the determination means determines that the user is in a state of confusion, the control means causes the voice output unit to output an audio message for inquiry. To make it function as program. [Explanation of Symbols]

[0096] 10 Information Processing Devices 20 Systems 110 Judgment section 130 Control Unit 150 Display Control Unit 170 Position determination section 210 sensors 220 Automotive equipment 230 Mobile Unit Control Unit 240 microphones 260 displays 270 Fragrance output section 280 Light-emitting part 290 Audio output section 1000 calculator 1020 Bus 1040 processor 1060 memory 1080 Storage Devices 1100 Input / Output Interface 1120 Network Interface

Claims

1. A determination unit that determines whether or not the user is in a confused state using information on the user's situation and at least one of the user's actions, The system includes a control unit that, when the determination unit determines that the user is in a confused state, causes the voice output unit to execute an audio output for inquiry. Information processing device.

2. In the information processing apparatus according to claim 1, The determination unit further determines whether the user is in a state of confusion, using information regarding at least one of the following: the condition of the vehicle the user is riding in, and the condition of the area surrounding the vehicle. Information processing device.

3. In the information processing apparatus according to claim 2, The system further includes a position determination unit that determines whether the position of the moving body satisfies predetermined positional conditions. The determination unit determines whether the user is in a confused state using information regarding at least one of the following: the state of the moving body, the state of the moving body's surroundings, the state of the user, and the user's actions, at the time when the position of the moving body satisfies the predetermined positional conditions. Information processing device.

4. In the information processing apparatus described in claim 3, The position determination unit determines whether the moving object is located at the target point, The determination unit, Using information relating to at least one of the following: the status of the moving object, the surrounding environment of the moving object, the user's status, and the user's actions, the degree of the user's confusion is identified. When the moving object is located at the target point, if the degree of confusion is equal to or greater than the first criterion, it is determined that the user is in a state of confusion. If the moving object is not located at the target point, and the degree of confusion is equal to or greater than the second criterion, the user is determined to be in a state of confusion. The first criterion is lower than the second criterion. Information processing device.

5. In the information processing apparatus according to any one of claims 2 to 4, The determination unit further determines whether the user is in a state of confusion using information indicating the content of operations performed on the in-vehicle equipment provided on the mobile vehicle. Information processing device.

6. In the information processing apparatus according to any one of claims 2 to 4, The control unit, Processing for interaction with the aforementioned user is possible, After executing the voice output for the aforementioned inquiry, the system controls the mobile vehicle to assist the user's driving based on the interaction with the user who is operating the mobile vehicle. Information processing device.

7. In the information processing apparatus according to any one of claims 2 to 4, It further includes a display control unit that displays route guidance information on a display, The determination unit determines that the user is in a state of confusion when it detects that the user's gaze has been directed at the display a predetermined number of times or more, using information generated by an imaging device for detecting the direction of the user's gaze. Information processing device.

8. In the information processing apparatus according to any one of claims 1 to 4, The determination unit uses information generated by the imaging device for detecting the direction of the user's gaze to determine whether the user's gaze meets predetermined gaze conditions. Information processing device.

9. In the information processing apparatus according to any one of claims 1 to 4, The determination unit uses information generated by the imaging device for detecting the user's actions to determine whether or not the user is in a confused state. Information processing device.

10. In the information processing apparatus according to any one of claims 1 to 4, The determination unit uses the user's vital information to determine whether or not the user is in a state of confusion. Information processing device.

11. In the information processing apparatus according to any one of claims 1 to 4, Based on the determination result of the determination unit, the control unit causes at least one of the light-emitting unit and the fragrance output unit to perform a predetermined operation. Information processing device.

12. An information processing method performed by one or more computers, A determination step of determining whether the user is in a confused state using information on the user's situation and at least one of the user's actions, The control step includes, if the determination step determines that the user is in a confused state, causing the voice output unit to output an audio message for inquiry. Information processing methods.

13. Computers, A determination means for determining whether or not the user is in a confused state, using information on the user's situation and at least one of the user's actions, If the determination means determines that the user is in a state of confusion, the control means causes the voice output unit to output an audio message for inquiry. To make it function as program.

Images