Control system and program

The control system enhances user engagement and comprehension by dynamically adjusting the robot's output mode based on emotional and understanding assessments, addressing the limitations of existing systems in adapting to user states during interactions.

JP2024158721APending Publication Date: 2024-11-08SOFTBANK GROUP CORP
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
JP2023074143
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Filing Date
2023-04-28
Publication Date
2024-11-08

AI Technical Summary

Technical Problem

Existing robot behavior control systems fail to effectively adjust their output mode based on the user's emotional state and understanding level during interactions, particularly when explaining important matters, leading to suboptimal engagement and comprehension.

Method used

A control system that includes an emotion determining unit to assess user emotions and understanding levels, adjusting voice tone, speaking speed, and body language to enhance user engagement and comprehension, using a server to process sensor data and control a robot's behavior.

Benefits of technology

The system improves user understanding and attention by dynamically adapting the robot's output mode, ensuring clearer communication and increased user engagement, especially when explaining complex or important information.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024158721000001_ABST
    Figure 2024158721000001_ABST
Patent Text Reader

Abstract

To provide a control system and a program which determine an appropriate action of an electronic apparatus (robot) relative to a state of a user conversing with the robot.SOLUTION: In a control system including a server and a robot as a humanoid robot, a processing section 110 of a server 100 consists of: an emotion determination section for determining a user emotion value indicating an emotion of a user conversing with a robot 200; a determination section for determining at least one of an understanding level and an attention level of the user relative to an object on the basis of a transition of the user emotion value of the user when the robot explains the object to the user; and a control section for adjusting an output mode of the explanation of the object by the robot in a manner to increase the attention level of the user on the basis of a determination result by the determination section.SELECTED DRAWING: Figure 2
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to a control system and a program. [Background technology]

[0002] Patent Document 1 describes a behavior control system that determines an appropriate robot behavior in response to the state of a user who is a conversation partner of the robot. [Prior art document] [Patent documents] [Patent Document 1] JP 2016-012341 A Summary of the Invention [Means for solving the problem]

[0003] According to one embodiment of the present invention, a control system is provided. The control system may include an emotion determination unit that determines a user emotion value indicating the emotion of a user conversing with an electronic device. The control system may include a determination unit that determines at least one of the user's level of understanding and level of attention to the object based on a transition of the user's user emotion value when the electronic device is explaining the object to the user. The control system may include a control unit that adjusts an output mode of the explanation of the object by the electronic device based on a result of the determination by the determination unit.

[0004] In the control system, when the user's emotions are negative at the beginning of the electronic device's explanation of the object, the judgment unit may judge that the lower the degree of the user's negative emotions compared to the degree of the negative emotions at the beginning of the explanation, the higher at least one of the user's level of understanding and level of attention is.

[0005] In any of the control systems, the control unit may cause the electronic device to ask the user a question in the middle of explaining the object, and the judgment unit may further judge at least one of the user's level of understanding and level of attention to the object based on the user's response to the question.

[0006] In any of the control systems, the control unit may adjust at least one of the tone of voice of the electronic device, the speaking speed of the electronic device, the amount of gestures of the electronic device, the amount of nodding of the electronic device in response to the user, and the amount of spacing of the electronic device based on the determination result by the determination unit. The control system may include a storage unit that stores history data that associates the level of understanding or the level of attention of the user determined by the determination unit based on a transition of the user emotion value of the user when the control unit makes the adjustment with content of the adjustment, and the control unit may determine the content of the adjustment based on the multiple pieces of history data stored in the storage unit.

[0007] In any of the control systems, the control unit may perform at least one of the following adjustments: adjusting the tone of voice of the electronic device to be stronger the lower the user's level of understanding; adjusting the speaking speed of the electronic device to be slower the lower the user's level of understanding; adjusting the amount of gestures and movements of the electronic device to be greater the lower the user's level of understanding; adjusting the number of times the electronic device nods in accordance with the user the lower the user's level of understanding; and adjusting the number of times the electronic device pauses the lower the user's level of understanding.

[0008] In any of the control systems, the control unit may perform at least one of the following adjustments: an adjustment to make the tone of voice of the electronic device stronger the lower the level of attention of the user; an adjustment to make the speaking speed of the electronic device slower the lower the level of attention of the user; an adjustment to make the electronic device increase the amount of gestures and movements the lower the level of attention of the user; an adjustment to make the electronic device nod more frequently to match the user the lower the level of attention of the user; and an adjustment to make the electronic device pause more frequently the lower the level of attention of the user.

[0009] Any of the control systems may further comprise the electronics.

[0010] According to one embodiment of the present invention, there is provided a program for causing a computer to function as the control system.

[0011] The above summary of the invention does not list all of the necessary features of the present invention. Also, subcombinations of these features may also be inventions. [Brief description of the drawings]

[0012] [Figure 1] 1 shows an example of the overall configuration of a system 10 according to the present embodiment. [Diagram 2] 1 shows a schematic functional block configuration of a robot 200 and a server 100. [Diagram 3] An example of the hardware configuration of a computer 1200 that functions as a part of the server 100 or the robot 200 is shown in schematic form. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] The present invention will be described below through embodiments of the invention, but the following embodiments do not limit the scope of the invention according to the claims. Furthermore, not all of the combinations of features described in the embodiments are necessarily essential to the solution of the invention.

[0014] FIG. 1 illustrates an example of an overall configuration of a system 10 according to the present embodiment. The system 10 includes a server 100 and a robot 200. The system 10 may include a plurality of robots 200. The robot 200 may be a humanoid robot. The robot 200 may be a non-humanoid robot. The robot 200 may be an example of an electronic device. The server 100 may function as a control system. The robot 200 may be an example of a control target of the control system. The system 10 may be an example of a control system.

[0015] The robot 200 is placed in an arbitrary location. The user 20 is a person who uses the robot 200.

[0016] The robot 200 is placed, for example, in a hospital or a clinic. The user 20 may be a patient visiting a hospital or a clinic. The robot 200 is placed, for example, in a home. The user 20 may be a family member of the home in which the robot 200 is placed, or a stranger visiting the home. The robot 200 is placed, for example, at the reception desk of a store or an office. The user 20 may be a customer visiting the store or office. The manner of use of the robot 200 is not limited to these manners of use.

[0017] The server 100 is provided remotely from the robot 200. The server 100 can control the robot 200 through a communication network 50. For example, the server 100 acquires sensor information detected by the robot 200 through the communication network 50, determines an action to be performed by the robot 200 based on the acquired sensor information, and transmits control information through the communication network 50 to instruct the robot 200.

[0018] For example, the robot 200 acquires various types of information detected by sensors, such as the voice and image of the user 20 and information on an external force received by the robot 200, and transmits the information to the server 100. The server 100 generates control information for controlling the robot 200 based on the information acquired from the robot 200. The server 100 generates control information for controlling the robot 200 based on the information acquired from the robot 200, for example, by using a neural network (NN).

[0019] In the system 10, the server 100 may execute processes for processing various information to determine the behavior of the robot 200, and processes for determining the behavior of the robot 200 based on the results of processing the various information. Note that part or all of these processes may be executed by the robot 200. In this case, part of the robot 200 functions as a control system.

[0020] The system 10 according to this embodiment executes a process of explaining an arbitrary object to the user 20. The object may be any object that can be the subject of an explanation, including any thing. The system 10 may explain important matters to the user 20. For example, the system 10 obtains informed consent from the user 20 by a robot 200 arranged in a hospital or clinic.

[0021] The server 100 has a function of recognizing the emotion of the user 20. The server 100 determines a user emotion value indicating the emotion of the user 20 from the facial expression of the user 20 and a conversation with the user 20. The server 100 determines the degree of understanding and attention of the user 20 with respect to an object being explained to the user 20. The server 100 may adjust the output mode of the explanation of the object by the robot 200 according to the determined degree of understanding and degree of attention of the user 20. Note that, as described above, the robot 200 may execute some or all of the process of recognizing the emotion of the user 20, the process of determining the degree of understanding and degree of attention of the user 20, and the process of adjusting the output mode of the explanation of the object.

[0022] For example, when the user 20 has a high level of understanding 22, the server 100 simply reads out and speaks the contents, and when the user 20 has a low level of understanding 22, the server 100 not only reads out and speaks the contents but also adds movements such as tone of voice, speaking speed, and gestures according to changes in the other person's level of understanding 22, thereby improving the user 20's level of understanding of important matters.

[0023] As a specific example, the server 100 adjusts the voice tone of the robot 200 to be stronger as the level of understanding 22 of the user 20 is lower. Also, for example, the server 100 adjusts the speech speed of the robot 200 to be slower as the level of understanding 22 of the user 20 is lower. Also, for example, the server 100 adjusts the amount of gestures of the robot 200 to be greater as the level of understanding 22 of the user 20 is lower. Also, for example, the server 100 adjusts the number of times the robot 200 nods in response to the user 20 to be greater as the level of understanding 22 of the user 20 is lower. Also, for example, the server 100 adjusts the number of times the robot 200 pauses to be greater as the level of understanding 22 of the user 20 is lower.

[0024] In normal conversations and customer service, the level of understanding of the other person is not considered very important, and it is considered that the other person is often led to a more positive feeling. In contrast, when explaining important matters, it is necessary to place more importance on the level of understanding than on positivity, and the system 10 according to the present embodiment executes processing that places emphasis on the level of understanding.

[0025] The server 100 applies the transition of emotions of the user 20 to measure the level of understanding 22 of the user 20. For example, in many cases, when an explanation of an important matter is given, the user feels anxiety, fear, or sadness at first, but the negative emotions gradually fade away as more explanation is given and questions are answered. Therefore, for example, when the emotion of the user 20 is negative at the beginning of the explanation, the server 100 determines that the level of understanding 22 of the user 20 is higher as the degree of the user's negative emotion decreases compared to the degree of the negative emotion at the beginning of the explanation.

[0026] The server 100 may measure the level of understanding 22 of the user 20 not only based on the transition of the user 20's emotions, but also based on conversations with the user 20 and questions to the user 20. For example, the server 100 may, during an explanation, have conversations or questions with the user 20 through the robot 200 to confirm the contents of the explanation so far, and determine whether or not the reply to the conversation or the answer to the question is consistent with the explanation, thereby measuring the level of understanding 22 of the user 20. The level of understanding 22 of the user 20 may be measured further based on the emotions of the user 20 when replying to the conversation or when answering the question.

[0027] 2 shows an example of a functional block configuration of the robot 200 and the server 100. First, the functional block configuration of the robot 200 will be described. The robot 200 includes a sensor unit 210, an information processing unit 220, a control target 230, and a communication unit 240. The information processing unit 220 may be a processor such as an MPU. The communication unit 240 is responsible for communication with the server 100. The communication unit 240 may be a communication device such as a network IF.

[0028] The control target 230 includes a speaker. The control target 230 includes a display unit. The display unit may be disposed at any position of the robot 200, such as the chest of the robot 200. The control target 230 includes LEDs installed in the eyes or the like. The control target 230 includes motors and the like that drive movable parts of the robot 200, such as the limbs and head.

[0029] The sensor unit 210 has various sensors such as a microphone 211, a gyro sensor 212, a motor sensor 213, a camera 214, an infrared sensor 215, and a battery remaining amount sensor 216. It is not essential that the sensor unit 210 has all of these sensors, and the sensor unit 210 may have sensors other than these.

[0030] The microphone 211 acquires surrounding sounds. For example, the microphone 211 acquires the voice of the user 20. The gyro sensor 212 detects the angular velocity of the entire robot 200 and each part of the robot 200. The motor sensor 213 detects the rotation angle of the drive shaft of the motor that drives the movable part of the robot 200. The camera 214 captures images using visible light and generates image information of moving images and still images. The infrared sensor 215 detects surrounding objects using infrared rays. The battery remaining capacity sensor 216 detects the remaining capacity of the battery equipped in the robot 200.

[0031] The sensor unit 210 outputs various sensor data, such as voice data acquired by the microphone 211, an image captured by the camera 214, an angular velocity detected by the gyro sensor 212, a rotation angle detected by the motor sensor 213, a remaining capacity detected by the battery remaining amount sensor 216, and object information detected by the infrared sensor 215, to the information processing unit 220. The information processing unit 220 supplies the acquired sensor signals to the communication unit 240, and transmits them to the server 100. Furthermore, based on the control information acquired from the server 100, the information processing unit 220 causes the speaker of the robot 200 to speak, causes an LED of the robot 200 to emit light, and causes limbs and the like of the robot 200 to operate.

[0032] Next, a functional block configuration of the server 100 will be described. The server 100 has a processing unit 110, a communication unit 120, and a storage unit 130. The processing unit 110 has a control unit 111, a user state recognition unit 112, an emotion determination unit 113, and a judgment unit 114. The communication unit 120 has an information acquisition unit 122.

[0033] The communication unit 120 is responsible for communication with the robot 200. The communication unit 120 may be a communication device such as a network IF. The information acquisition unit 122 acquires various information. For example, the information acquisition unit 122 acquires information detected by the sensor unit 210 of the robot 200 via the communication network 50.

[0034] The storage unit 130 has a storage medium such as a hard disk device, a flash memory, etc. The storage unit 130 also has a volatile storage device such as a RAM. The storage unit 130 stores program codes and various temporary data that the processing unit 110 reads out at execution time, as well as information required for the processing of the processing unit 110. The storage unit 130 may store so-called Large Language Models (LLM). The storage unit 130 stores information acquired by the communication unit 120 from the robot 200.

[0035] The control unit 111 uses the information stored in the memory unit 130 and the information acquired by the information acquisition unit 122 to determine the behavior of the robot 200, and transmits control information for executing the determined behavior to the robot 200 via the communication unit 120.

[0036] The storage unit 130 stores, for example, a description of an object and an output format in which the robot 200 outputs the description.

[0037] The output manner may include the contents of the display output. The output manner may include the contents of the audio output. The output manner may include the tone of the voice of the robot 200. The output manner may include the speaking speed of the robot 200. The output manner may include information on the amount of pauses of the robot 200. The output manner may include the contents of the body movements of the robot 200. The output manner may include information on gestures of the robot 200. The output manner may include information on the amount of nodding of the robot 200 in accordance with the user 20.

[0038] For example, in the case of obtaining informed consent, the object is the condition of the user 20 and a medical procedure to be performed, and a specific explanation thereof is stored in the storage unit 130. The explanation may be composed of a plurality of stages, and the output mode may be registered for each stage. Within each stage, the output mode may be changed in detail.

[0039] The explanation in the informed consent needs to be accurately understood by the user 20, and needs to attract the user's attention. It is considered that the user's 20 understanding and attention to an object largely depend on the manner of explanation. In particular, when an explanation is given by the robot 200, the output mode of the robot 200 has a large effect on the user's 20 understanding and attention. Therefore, in the control system 10 according to this embodiment, for example, a basic output mode for the explanation of an object is registered, and it is appropriately adjusted so as to increase the user's 20 understanding and attention.

[0040] The user state recognition unit 112 recognizes the state of the user 20 based on the information acquired by the information acquisition unit 122. The user state recognition unit 112 may analyze a captured image captured by the camera 214 to recognize the face of the user 20. For example, the storage unit 130 stores face images of a plurality of people, and the user state recognition unit 112 recognizes the face of the user 20 by matching the captured image with the stored face images. The user state recognition unit 112 may analyze a captured image captured by the camera 214 to recognize the facial expression of the user 20. The user state recognition unit 112 may analyze the voice of the user 20 detected by the microphone 211 to recognize the contents of the utterance of the user 20. The user state recognition unit 112 may perform voice emotion recognition by analyzing the voice of the user 20 detected by the microphone 211. For example, the user state recognition unit 112 extracts feature amounts such as frequency components of the voice, and recognizes the emotion of the user 20 based on the extracted feature amounts.

[0041] The user state recognition unit 112 may recognize the behavior of the user 20 based on the information acquired by the information acquisition unit 122. For example, the user state recognition unit 112 stores in advance a neural network that outputs the behavior of the user 20 from the information acquired by the information acquisition unit 122, which is learned using the information acquired by the information acquisition unit 122 and the actual behavior of the user 20 at the time when the information acquisition unit 122 acquired the information as training data, and specifies the behavior of the user 20 by inputting the information acquired by the information acquisition unit 122 into the neural network. The neural network may be configured to receive the facial expression of the user 20 and the contents of the voice of the user 20 recognized by the user state recognition unit 112. The method of recognizing the user's behavior is not limited to this, and other known methods may be used.

[0042] The emotion determination unit 113 determines a user emotion value indicating the emotion of the user 20 conversing with the robot 200. The emotion determination unit 113 may determine the user emotion value of the user 20 using information acquired by the information acquisition unit 122. The emotion determination unit 113 may determine the user emotion value of the user 20 using a user state recognized by the user state recognition unit 112. When voice emotion recognition is executed by the user state recognition unit 112, the emotion determination unit 113 may determine the user emotion value of the user 20 using information acquired by the information acquisition unit 122 and a voice emotion recognition result.

[0043] The user emotion value may include a type of emotion of the user 20, and a value indicating the positive or negative emotion and the degree of the emotion of the user 20. Examples of the types of emotion of the user 20 include "joy", "pleasure", "comfort", "relief", "excitement", "relief", "fulfillment", "anger", "sorrow", "discomfort", "anxiety", "sadness", "concern", and "emptiness". Note that these are merely examples and are not limited to these. The value indicating the positive or negative emotion of the user 20 indicates a positive value if the emotion of the user 20 is a cheerful emotion accompanied by a pleasant feeling or comfort, such as "joy", "pleasure", "comfort", "relief", "excitement", "relief", and "fulfillment", and the brighter the emotion, the larger the value. If the emotion of the user is an unpleasant emotion, such as "anger", "sorrow", "discomfort", "anxiety", "sadness", "concern", and "emptiness", the value indicates a negative value, and the more unpleasant the emotion, the larger the absolute value of the negative value. If the emotion of the user is not any of the above ("normal"), the value indicating the positive or negative indicates a value of 0.

[0044] For example, the emotion determination unit 113 pre-stores a neural network that outputs a user emotion value of the user 20 from the information acquired by the information acquisition unit 122, the neural network having been trained using the information acquired by the information acquisition unit 122 and a user emotion value indicating the actual emotion of the user 20 at the time the information acquisition unit 122 acquired the information as training data, and acquires the user emotion value by inputting the information acquired by the information acquisition unit 122 into the neural network. The neural network may be configured to accept the facial expression of the user 20 and the vocal content of the user 20 recognized by the user state recognition unit 112. The method of determining the user emotion value is not limited to this, and other known methods may be used.

[0045] The determination unit 114 determines the user's level of understanding of an object based on the transition of the user emotion value of the user when the robot 200 is explaining the object to the user.

[0046] For example, when the emotion of the user 20 at the beginning of the robot 200's explanation of an object is negative, the determination unit 114 determines that the user 20 has a higher level of understanding as the degree of the negative emotion of the user 20 decreases with respect to the degree of the negative emotion at the beginning of the explanation. In addition, when the emotion of the user 20 at the beginning of the robot 200's explanation of an object is negative, the determination unit 114 may determine that the degree of understanding of the user 20 is lower as the degree of the negative emotion of the user 20 increases with respect to the degree of the negative emotion at the beginning of the explanation. Many important matters are initially explained with negative emotions, and the negative emotions fade as the understanding progresses, but when the user does not understand, the negative emotions tend not to fade or to become stronger. Therefore, by making the above-mentioned determination based on the transition of the user emotion value, the user 20's level of understanding can be determined with high accuracy.

[0047] The control unit 111 may cause the robot 200 to ask a question to the user 20 during the explanation of the object. The determination unit 114 may determine the user 20's level of understanding of the object based on the user 20's response to the question. This can further improve the accuracy of the user 20's level of understanding. The timing and type of question to be asked during the explanation of the object may be registered in advance and stored in the storage unit 130. For example, when the explanation of the object includes a plurality of stages, a question asking a question about the content of each stage is registered for each stage. The determination unit 114 may determine that the user 20's level of understanding is high when the answer of the user 20 to the question is correct, compared with when the answer is incorrect. The determination unit 114 may determine the user 20's level of understanding based on the user emotion value of the user 20 when the answer is correct. For example, when the answer of the user 20 is correct, the determination unit 114 determines that the user 20's level of understanding is high when the user emotion value is high. In this way, if the user 20 is confident in the answer and has a strong positive emotion, it can be determined that the user 20 has a high level of understanding, and if the user is not so confident in the answer and has a strong negative emotion, it can be determined that the user 20 has a low level of understanding.

[0048] When the emotion determination unit 113 determines the user emotion value, it may output the accuracy of the user emotion value. When the user emotion value of the user 20 when the robot 200 is explaining the object to the user 20 meets a predetermined standard, the determination unit 114 may determine the level of understanding of the user 20 based only on the transition of the user emotion value, and when the standard is not met, the control unit 111 may cause the robot 200 to ask the user 20 a question, and determine the level of understanding of the user 20 based on the reaction to the question and the transition of the user emotion value. In this way, when the accuracy of the user emotion value is relatively low and the degree of understanding of the user 20 based on the user emotion value is unlikely to be accurate, the question is asked to determine the level of understanding more accurately, and when the accuracy of the user emotion value is relatively high and the degree of understanding of the user 20 based on the user emotion value is likely to be accurate, the question is not asked, thereby preventing the user 20 from feeling that the question is bothersome. The judgment unit 114 judges whether or not a predetermined criterion is satisfied based on, for example, whether or not the average user emotion value of the user 20 when the robot 200 is explaining an object to the user 20 is higher than a predetermined threshold value.

[0049] Based on the result of the judgment by the judgment unit 114 of the degree of understanding of the user 20, the control unit 111 may adjust at least any of the tone of voice of the robot 200, the speaking speed of the robot 200, the amount of gestures and movements of the robot 200, the amount of nodding by the robot 200 in response to the user 200, and the amount of pauses by the robot 200.

[0050] For example, the control unit 111 determines the content of the adjustment by learning. For example, the storage unit 130 stores history data in which the degree of understanding of the user 20 determined by the determination unit 114 based on the transition of the user emotion value of the user 20 when the control unit 111 makes the adjustment is associated with the content of the adjustment. The storage unit 130 stores history data for various adjustment contents. The control unit 111 determines the content of the adjustment based on a plurality of history data stored in the storage unit 130. For example, when the robot 200 is explaining an object to the user 20, the control unit 111 analyzes how the degree of understanding of the user 20 changes when the tone of the voice of the robot 200 is made stronger and weaker, how the degree of understanding of the user 20 changes according to the level of strength when the tone of the voice of the robot 200 is made stronger, and how the degree of understanding of the user 20 changes according to the level of weakness when the tone of the voice of the robot 200 is made weaker, and specifies the adjustment content of the tone of the voice that increases the degree of understanding of the user. The control unit 111 also executes similar processing for the speech speed, the amount of gestures, the amount of nodding in response to the user 20, the amount of pauses, etc. By repeatedly executing such processing, it is possible to adjust the output mode of the explanation of the object by the robot 200 in accordance with the flow of the explanation so that the user 20 can further increase his / her understanding.

[0051] Also, for example, the control unit 111 determines the content of the adjustment according to a predetermined rule. The control unit 111 may execute an adjustment to make the tone of the voice of the robot 200 stronger as the user 20 has a lower level of understanding. The control unit 111 may execute an adjustment to make the speech speed of the robot 200 slower as the user 20 has a lower level of understanding. The control unit 111 may execute an adjustment to make the amount of gestures of the robot greater as the user has a lower level of understanding. The control unit 111 may execute an adjustment to make the robot 200 nod more frequently in accordance with the user 20 as the user has a lower level of understanding. The control unit 111 may execute an adjustment to make the robot 200 pause more frequently as the user has a lower level of understanding. The control unit 111 may execute only one of these, may execute a plurality of these, or may execute all of these. These can contribute to improving the user 20's level of understanding of the object.

[0052] The determination unit 114 determines the degree of attention of the user 20 to the object based on the transition of the user emotion value of the user 20 when the robot 200 is explaining the object to the user 20.

[0053] For example, the determination unit 114 determines that the lower the degree of transition of the user emotion value of the user 20 while the robot 200 is explaining an object, the lower the level of attention of the user 20. When important matters are being explained, the emotion of the user 20 tends to change significantly. Nevertheless, if the emotion of the user 20 changes little, it is highly likely that the user 20 is not paying much attention to the explanation. Therefore, by making the above-mentioned determination based on the transition of the user emotion value, the attention level of the user 20 can be determined with high accuracy.

[0054] The control unit 111 may cause the robot 200 to ask a question to the user 20 during the explanation of the object. The determination unit 114 may determine the attention level of the user 20 to the object based on the reaction of the user 20 to the question. This can further improve the accuracy of the attention level of the user 20. The timing and type of question to be asked during the explanation of the object may be registered in advance and stored in the storage unit 130. For example, when the explanation of the object includes multiple stages, a question asking a question about the content of each stage is registered for each stage. The determination unit 114 may determine that the attention level of the user 20 is high when the answer of the user 20 to the question is correct compared to when the answer is incorrect.

[0055] If the user emotion value of the user 20 when the robot 200 is explaining an object to the user 20 satisfies a predetermined criterion, the judgment unit 114 may judge the attention level of the user 20 based only on the transition of the user emotion value, and if the criterion is not satisfied, the judgment unit 114 may cause the control unit 111 to make the robot 200 ask a question to the user 20 and judge the attention level of the user 20 based on the reaction to the question and the transition of the user emotion value.

[0056] Based on the result of the determination of the level of attention of the user 20 by the determination unit 114, the control unit 111 may adjust at least any of the tone of voice of the robot 200, the speaking speed of the robot 200, the amount of gestures and movements of the robot 200, the amount of nodding by the robot 200 in response to the user 200, and the amount of pauses the robot 200 makes.

[0057] For example, the control unit 111 determines the content of the adjustment by learning. For example, the storage unit 130 stores history data in which the attention level of the user 20 determined by the determination unit 114 based on the transition of the user emotion value of the user 20 when the control unit 111 makes the adjustment is associated with the content of the adjustment. The storage unit 130 stores history data for various adjustment contents. The control unit 111 determines the content of the adjustment based on a plurality of history data stored in the storage unit 130. For example, when the robot 200 is explaining an object to the user 20, the control unit 111 analyzes how the attention level of the user 20 transitions when the tone of the voice of the robot 200 is made stronger and weaker, how the attention level of the user 20 transitions according to the level of strength when the tone of the voice of the robot 200 is made stronger, and how the attention level of the user 20 transitions according to the level of weakness when the tone of the voice of the robot 200 is made weaker, and specifies the adjustment content of the tone of the voice that increases the attention level of the user. The control unit 111 also executes similar processing for the speech speed, the amount of gestures, the amount of nodding in response to the user 20, the amount of pauses, etc. By repeatedly executing such processing, it is possible to adjust the output mode of the explanation of the object by the robot 200 in accordance with the flow of the explanation so as to further increase the attention of the user 20.

[0058] Also, for example, the control unit 111 determines the content of the adjustment according to a predetermined rule. The control unit 111 may execute an adjustment to make the tone of the voice of the robot 200 stronger as the level of attention of the user 20 is lower. The control unit 111 may execute an adjustment to make the speech speed of the robot 200 slower as the level of attention of the user 20 is lower. The control unit 111 may execute an adjustment to make the amount of gestures of the robot greater as the level of attention of the user is lower. The control unit 111 may execute an adjustment to make the robot 200 nod more frequently in response to the user 20 as the level of attention of the user 20 is lower. The control unit 111 may execute an adjustment to make the robot 200 pause more frequently as the level of attention of the user 20 is lower. The control unit 111 may execute only one of these, may execute a plurality of these, or may execute all of these. These can contribute to improving the level of attention of the user 20 to the object.

[0059] The functions of the server 100 described above may be implemented by one or more computers. At least some of the functions of the server 100 may be implemented by a virtual machine. At least some of the functions of the server 100 may be implemented in the cloud. In the above embodiment, the robot 200 has been mainly given as an example of an electronic device, but this is not limited to this. Examples of electronic devices include smartphones, tablet terminals, PCs (Personal Computers), smart speakers, stuffed toys, vehicles such as automobiles and motorcycles, home appliances, etc. Note that these are merely examples, and electronic devices are not limited to these.

[0060] 3 is a schematic diagram showing an example of a hardware configuration of a computer 1200 functioning as a part of the server 100 or the robot 200. The computer 1200 may function as the server 100, for example. A program installed on the computer 1200 may cause the computer 1200 to function as one or more "parts" of the apparatus according to the present embodiment, or cause the computer 1200 to execute operations or one or more "parts" associated with the apparatus according to the present embodiment, and / or cause the computer 1200 to execute a process or steps of the process according to the present embodiment. Such a program may be executed by the CPU 1212 to cause the computer 1200 to execute specific operations associated with some or all of the blocks of the flowcharts and block diagrams described herein.

[0061] The computer 1200 according to this embodiment includes a CPU 1212, a RAM 1214, and a graphics controller 1216, which are connected to each other by a host controller 1210. The computer 1200 also includes input / output units such as a communication interface 1222, a storage device 1224, a DVD drive 1226, and an IC card drive, which are connected to the host controller 1210 via an input / output controller 1220. The DVD drive 1226 may be a DVD-ROM drive, a DVD-RAM drive, or the like. The storage device 1224 may be a hard disk drive, a solid state drive, or the like. The computer 1200 also includes a legacy input / output unit such as a ROM 1230 and a keyboard, which are connected to the input / output controller 1220 via an input / output chip 1240.

[0062] The CPU 1212 operates according to a program stored in the ROM 1230 and the RAM 1214, thereby controlling each unit. The graphic controller 1216 acquires image data generated by the CPU 1212 into a frame buffer or the like provided in the RAM 1214 or into itself, and causes the image data to be displayed on the display device 1218.

[0063] The communication interface 1222 communicates with other electronic devices via a network. The storage device 1224 stores programs and data used by the CPU 1212 in the computer 1200. The DVD drive 1226 reads programs or data from a DVD-ROM 1227 or the like and provides the programs or data to the storage device 1224. The IC card drive reads programs and data from an IC card and / or writes programs and data to an IC card.

[0064] The ROM 1230 stores therein a boot program or the like that is executed by the computer 1200 upon activation, and / or a program that depends on the hardware of the computer 1200. The input / output chip 1240 may also connect various input / output units to the input / output controller 1220 via a USB port, a parallel port, a serial port, a keyboard port, a mouse port, and the like.

[0065] The programs are provided by a computer-readable storage medium such as a DVD-ROM 1227 or an IC card. The programs are read from the computer-readable storage medium, installed in the storage device 1224, the RAM 1214, or the ROM 1230, which are also examples of computer-readable storage media, and executed by the CPU 1212. Information processing described in these programs is read by the computer 1200, and brings about cooperation between the programs and the various types of hardware resources described above. An apparatus or method may be constructed by implementing operations or processing of information according to the use of the computer 1200.

[0066] For example, when communication is performed between the computer 1200 and an external device, the CPU 1212 may execute a communication program loaded in the RAM 1214 and instruct the communication interface 1222 to perform communication processing based on the processing described in the communication program. Under the control of the CPU 1212, the communication interface 1222 reads transmission data stored in a transmission buffer area provided in the RAM 1214, the storage device 1224, the DVD-ROM 1227, or a recording medium such as an IC card, and transmits the read transmission data to a network, or writes reception data received from the network to a reception buffer area or the like provided on the recording medium.

[0067] Furthermore, the CPU 1212 may cause all or a necessary portion of a file or database stored in an external recording medium such as the storage device 1224, the DVD drive 1226 (DVD-ROM 1227), an IC card, etc. to be read into the RAM 1214, and perform various types of processing on the data on the RAM 1214. The CPU 1212 may then write back the processed data to the external recording medium.

[0068] Various types of information, such as various types of programs, data, tables, and databases, may be stored in the recording medium and undergo information processing. The CPU 1212 may perform various types of processing on the data read from the RAM 1214, including various types of operations, information processing, conditional judgment, conditional branching, unconditional branching, information search / replacement, etc., as described throughout this disclosure and specified by the instruction sequence of the program, and write back the results to the RAM 1214. The CPU 1212 may also search for information in a file, database, etc. in the recording medium. For example, when a plurality of entries each having an attribute value of a first attribute associated with an attribute value of a second attribute are stored in the recording medium, the CPU 1212 may search for an entry whose attribute value of the first attribute matches a specified condition from among the plurality of entries, read the attribute value of the second attribute stored in the entry, and thereby obtain the attribute value of the second attribute associated with the first attribute that satisfies a predetermined condition.

[0069] The above-described programs or software modules may be stored in a computer-readable storage medium on the computer 1200 or in the vicinity of the computer 1200. In addition, a recording medium such as a hard disk or a RAM provided in a server system connected to a dedicated communication network or the Internet can be used as a computer-readable storage medium, thereby providing the programs to the computer 1200 via the network.

[0070] The blocks in the flowcharts and block diagrams in the present embodiment may represent stages of a process in which an operation is performed or "parts" of an apparatus responsible for performing the operation. Particular stages and "parts" may be implemented by dedicated circuitry, programmable circuitry provided with computer-readable instructions stored on a computer-readable storage medium, and / or a processor provided with computer-readable instructions stored on a computer-readable storage medium. The dedicated circuitry may include digital and / or analog hardware circuits, and may include integrated circuits (ICs) and / or discrete circuits. The programmable circuitry may include reconfigurable hardware circuits, such as, for example, field programmable gate arrays (FPGAs), programmable logic arrays (PLAs), and the like, including AND, OR, XOR, NAND, NOR, and other logical operations, flip-flops, registers, and memory elements.

[0071] A computer-readable storage medium may include any tangible device capable of storing instructions that are executed by a suitable device, such that a computer-readable storage medium having instructions stored thereon comprises an article of manufacture that includes instructions that can be executed to create means for performing the operations specified in the flowcharts or block diagrams. Examples of computer-readable storage media may include electronic storage media, magnetic storage media, optical storage media, electromagnetic storage media, semiconductor storage media, and the like. More specific examples of computer-readable storage media may include floppy disks, diskettes, hard disks, random access memories (RAMs), read-only memories (ROMs), erasable programmable read-only memories (EPROMs or flash memories), electrically erasable programmable read-only memories (EEPROMs), static random access memories (SRAMs), compact disk read-only memories (CD-ROMs), digital versatile disks (DVDs), Blu-ray disks, memory sticks, integrated circuit cards, and the like.

[0072] The computer readable instructions may include either assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk®, JAVA®, C++, etc., and conventional procedural programming languages ​​such as the “C” programming language or similar programming languages.

[0073] Computer readable instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus, or to a programmable circuit, either locally or over a local area network (LAN), a wide area network (WAN), such as the Internet, etc., to cause the processor of the general purpose computer, special purpose computer, or other programmable data processing apparatus, or to a programmable circuit, to execute the computer readable instructions to generate means for performing the operations specified in the flowcharts or block diagrams. Examples of processors include computer processors, processing units, microprocessors, digital signal processors, controllers, microcontrollers, etc.

[0074] Although the present invention has been described above using the embodiments, the technical scope of the present invention is not limited to the scope described in the above embodiments. It is clear to those skilled in the art that various modifications and improvements can be made to the above embodiments. It is clear from the description of the claims that such modifications and improvements can also be included in the technical scope of the present invention.

[0075] It should be noted that the order of execution of each process, such as operations, procedures, steps, and stages, in the devices, systems, programs, and methods shown in the claims, specifications, and drawings is not specifically stated as "before" or "prior to," and that the process may be performed in any order unless the output of a previous process is used in a later process. Even if the operational flow in the claims, specifications, and drawings is described using "first," "next," and the like for convenience, this does not mean that the process must be performed in this order. [Explanation of symbols]

[0076] 10 system, 20 user, 22 understanding, 50 communication network, 100 server, 110 processing unit, 111 control unit, 112 user state recognition unit, 113 emotion determination unit, 114 judgment unit, 120 communication unit, 122 information acquisition unit, 130 memory unit, 200 robot, 210 sensor unit, 211 microphone, 212 gyro sensor, 213 motor sensor, 214 camera, 215 infrared sensor, 216 battery remaining sensor, 220 information processing unit, 230 control target, 240 communication unit, 1200 computer, 1210 host controller, 1212 CPU, 1214 RAM, 1216 graphic controller, 1218 display device, 1220 input / output controller, 1222 communication interface, 1224 storage device, 1226 DVD drive, 1227 DVD-ROM, 1230 ROM, 1240 I / O chips

Claims

1. an emotion determining unit for determining a user emotion value indicating an emotion of a user who is talking to the electronic device; a determination unit that determines at least one of a level of understanding and a level of attention of the user with respect to the object based on a transition of the user emotion value of the user when the electronic device is explaining the object to the user; a control unit that adjusts an output mode of the description of the object by the electronic device based on a result of the determination by the determination unit; A control system comprising:

2. The control system of claim 1, wherein the determination unit determines that the user's level of understanding is higher when the user's emotions at the beginning of the electronic device's explanation of the object are negative, and the degree of the user's negative emotions decreases relative to the degree of the negative emotions at the beginning of the explanation.

3. The control unit causes the electronic device to ask a question to the user during the explanation of the object, The control system according to claim 1 , wherein the determination unit determines at least one of the level of understanding and the level of attention of the user to the object further based on a response of the user to the question.

4. The control system according to any one of claims 1 to 3, wherein the control unit adjusts at least one of the tone of voice of the electronic device, the speaking speed of the electronic device, the amount of gestures of the electronic device, the amount of nodding of the electronic device in response to the user, and the amount of spacing of the electronic device based on the judgment result by the judgment unit.

5. a storage unit that stores history data in which the degree of understanding or the degree of attention of the user determined by the determination unit based on a transition of the user emotion value of the user when the control unit makes the adjustment is associated with the content of the adjustment. Equipped with The control system according to claim 4 , wherein the control unit determines the content of the adjustment based on a plurality of pieces of the history data stored in the storage unit.

6. The control system of claim 4, wherein the control unit performs at least one of the following adjustments: an adjustment to make the tone of voice of the electronic device stronger as the user's level of understanding becomes lower; an adjustment to make the speaking speed of the electronic device slower as the user's level of understanding becomes lower; an adjustment to make the electronic device increase the amount of gestures and movements as the user's level of understanding becomes lower; an adjustment to make the electronic device nod more frequently in accordance with the user as the user's level of understanding becomes lower; and an adjustment to make the electronic device pause more frequently as the user's level of understanding becomes lower.

7. The control system of claim 4, wherein the control unit performs at least one of the following adjustments: an adjustment to make the voice tone of the electronic device stronger as the level of attention of the user is lower; an adjustment to make the speaking speed of the electronic device slower as the level of attention of the user is lower; an adjustment to make the amount of gestures of the electronic device greater as the level of attention of the user is lower; an adjustment to make the electronic device nod more frequently to match the user as the level of attention of the user is lower; and an adjustment to make the electronic device pause more frequently as the level of attention of the user is lower.

8. The control system of claim 1 , further comprising the electronics.

9. A program for causing a computer to function as the control system according to any one of claims 1 to 3.