Action control system and program

Abstract

To make a robot take an appropriate action in response to an action of a user.SOLUTION: An action control system disclosed herein comprises a user condition recognition unit for recognizing the user condition including an action of a user, an emotion determination unit configured to determine an emotion value indicative of the emotions of the user based on the user condition, and an action determination unit configured to determine an action corresponding to the action of the user as an action to be taken when the emotion value is positive, determine an action defined based on reaction rules according to a category of the action of the user as an action to be taken when the emotion value is not positive and the absolute value thereof is less than a threshold, and determine an action of notifying a supervisor of the user of the state of the user as an action to be taken when the emotion value is negative and the absolute value thereof is equal to or greater than the threshold.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to a behavior control system and a program. [Background technology]

[0002] A behavior control system is known that includes a behavior information storage unit that stores decision information for deciding an action based on a user's state; a user state recognition unit that recognizes the user's state; a behavior decision unit that decides an action to be performed based on the user's state recognized by the user state recognition unit and the decision information; a reaction recognition unit that recognizes the user's reaction to performing the action decided by the behavior decision unit; a transmission unit that transmits the user's state recognized by the user state recognition unit, the action decided by the behavior decision unit, and the user's reaction recognized by the reaction recognition unit to an external server; a receiving unit that receives from the server an action appropriate to the user's state recognized by the user state recognition unit when the behavior decision unit is unable to decide an action to be performed based on the user's state and the decision information; and a behavior information update unit that updates the decision information based on the action received by the receiving unit from the server (see, for example, Patent Document 1).

[0003] [Prior art document]

[0004] [Patent Documents]

[0005] [Patent Document 1] Patent No. 6053847 Summary of the Invention [Problem to be solved by the invention]

[0006] It is desirable to make the robot perform an appropriate action in response to the user's action. [Means for solving the problem]

[0007] According to a first aspect of the present invention, there is provided a behavior control system including: an emotion determination unit that determines an emotion value indicating the emotion of the user based on the user state; and an action determination unit that determines, as an action to be executed, an action corresponding to the user's behavior if the emotion value is a positive value, determines, as an action to be executed, an action determined based on a reaction rule in accordance with the user's behavior classification if the emotion value is not a positive value and its absolute value is less than a threshold, and determines, as an action to be executed, an action to notify the user's supervisor of the user's state if the emotion value is a negative value and its absolute value is equal to or greater than the threshold.

[0008] If the emotion value is a negative value and the absolute value is equal to or greater than the threshold value, the behavior determining unit may further determine an behavior of the robot to rush to a location where the user is.

[0009] If the emotion value is a negative value and the absolute value is equal to or greater than the threshold, the user state recognition unit may record the state of the user using at least one of a microphone and a camera.

[0010] The user state recognition unit may record at least one of the user's speech content collected by the microphone and the image of the user taken by the camera as the user's state.

[0011] The user state recognition unit may record the result of converting the user's speech collected by the microphone into text information.

[0012] In a second aspect, there is provided a program for causing a computer to: It functions as a behavior control system.

[0013] The above summary of the invention does not list all of the necessary features of the present invention, and subcombinations of these features may also constitute inventions. [Brief explanation of the drawings]

[0014] [Figure 1] An example of a system 5 is shown schematically. [Figure 2] 1 shows a schematic functional configuration of a robot 100. [Figure 3] 10A and 10B show an example of an operation flow of the robot 100 according to the first embodiment. [Figure 4] 10A and 10B schematically show an example of an operation flow of the robot 100 according to the second embodiment. [Figure 5] 1 shows an example of a hardware configuration of a computer 1200. DETAILED DESCRIPTION OF THE INVENTION

[0015] The present invention will be described below through embodiments of the invention, but the following embodiments do not limit the scope of the invention according to the claims. Furthermore, not all of the combinations of features described in the embodiments are necessarily essential to the solution of the invention.

[0016] [First embodiment] FIG. 1 schematically illustrates an example of a system 5 according to this embodiment. The system 5 includes a robot 100, a robot 101, a robot 102, and a server 300. Users 10a, 10b, 10c, and 10d are users of the robot 100. Users 11a, 11b, and 11c are users of the robot 101. Users 12a and 12b are users of the robot 102. In the description of this embodiment, users 10a, 10b, 10c, and 10d may be collectively referred to as users 10. Users 11a, 11b, and 11c may be collectively referred to as users 11. Users 12a and 12b may be collectively referred to as users 12. The robots 101 and 102 have substantially the same functions as the robot 100. Therefore, the system 5 will be described mainly focusing on the functions of the robot 100.

[0017] The robot 100 converses with the user 10 and provides the user 10 with video. At this time, the robot 100 cooperates with a server 300 or the like with which it can communicate via a communication network 20 to converse with the user 10 and provide the video, etc. to the user 10. For example, the robot 100 not only learns appropriate conversation by itself, but also cooperates with the server 300 to learn how to have a more appropriate conversation with the user 10. The robot 100 also records captured video data of the user 10 in the server 300, requests the video data, etc. from the server 300 as needed, and provides the video data, etc. to the user 10.

[0018] The robot 100 also has an emotional value that indicates the type of its own emotion. For example, the robot 100 has emotional values ​​that indicate the intensity of each of the following emotions: "joy," "anger," "sorrow," "pleasure," "discomfort," "relief," "anxiety," "sadness," "excitement," "worry," "relief," "fulfillment," "emptiness," and "neutral." For example, when the robot 100 is in a state where the emotional value of excitement is high, the robot 100 speaks at a fast speed. In this way, the robot 100 can express its own emotions through its actions.

[0019] The robot 100 also has a function of recognizing the behavior of the user 10. The robot 100 recognizes the behavior of the user 10 by analyzing a facial image of the user 10 acquired by a camera function and a voice of the user 10 acquired by a microphone function. The robot 100 determines the behavior to be performed by the robot 100 based on the recognized behavior of the user 10, etc.

[0020] The robot 100 stores rules that define the actions that the robot 100 will take based on the actions of the user 10, and performs various actions in accordance with the rules.

[0021] Specifically, the robot 100 has reaction rules for determining the behavior of the robot 100 based on the behavior of the user 10. For example, the reaction rules define the behavior of the robot 100 as "laughing" when the behavior of the user 10 is "laughing." Furthermore, the reaction rules define the behavior of the robot 100 as "apologizing" when the behavior of the user 10 is "angry." Furthermore, the reaction rules define the behavior of the robot 100 as "answering" when the behavior of the user 10 is "asking a question." Furthermore, the reaction rules define the behavior of the robot 100 as "calling out" when the behavior of the user 10 is "sad."

[0022] When the robot 100 recognizes that the behavior of the user 10 is "angry" based on the reaction rules, the robot 100 selects the behavior of "apologizing" defined in the reaction rules as the behavior to be performed by the robot 100. For example, when the robot 100 selects the behavior of "apologizing," the robot 100 performs the motion of "apologizing" and outputs a voice representing the word "apologize."

[0023] The robot 100 also transmits to the server 300 user reaction information indicating that this behavior has elicited a positive reaction from the user 10. The user reaction information includes, for example, the user's behavior of "getting angry," the robot's 100 behavior of "apologizing," the fact that the user's 10 reaction was positive, and the attributes of the user 10.

[0024] The server 300 stores the user reaction information received from the robot 100. The server 300 receives and stores user reaction information not only from the robot 100 but also from each of the robots 101 and 102. The server 300 then analyzes the user reaction information from the robots 100, 101, and 102 and updates the reaction rules.

[0025] The robot 100 receives the updated reaction rules from the server 300 by inquiring about the updated reaction rules from the server 300. The robot 100 incorporates the updated reaction rules into the reaction rules stored in the robot 100. This allows the robot 100 to incorporate the reaction rules acquired by the robot 101, the robot 102, etc. into its own reaction rules.

[0026] 2 shows a schematic functional configuration of the robot 100. The robot 100 has a sensor unit 200, a sensor module unit 210, a storage unit 220, a user state recognition unit 230, an emotion determination unit 232, a behavior recognition unit 234, a behavior determination unit 236, a behavior control unit 250, a control target 252, and a communication processing unit 280.

[0027] The control object 252 includes a display device, a speaker, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 100 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 100 can be expressed by controlling these motors. In addition, the facial expressions of the robot 100 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 100. The posture, gestures, and facial expressions of the robot 100 are examples of the attitude of the robot 100.

[0028] The sensor unit 200 includes a microphone 201, a 3D depth sensor 202, a 2D camera 203, and a distance sensor 204. The microphone 201 continuously detects sound and outputs sound data. The microphone 201 may be provided on the head of the robot 100 and may have a function for binaural recording. The 3D depth sensor 202 continuously emits an infrared pattern and analyzes the infrared pattern from infrared images continuously captured by the infrared camera. , and detects the contour of an object. The 2D camera 203 is an example of an image sensor. The 2D camera 203 captures an image using visible light and generates visible light video information. The distance sensor detects the distance to an object by emitting, for example, a laser or ultrasonic wave. Note that the sensor unit 200 may also include a clock, a gyro sensor, a touch sensor, a sensor for motor feedback, etc.

[0029] 2, the components of the robot 100 excluding the control target 252 and the sensor unit 200 are examples of components included in the behavior control system of the robot 100. The behavior control system of the robot 100 controls the control target 252.

[0030] The storage unit 220 includes a reaction rule 221. At least a part of the storage unit 220 is implemented by a storage medium such as a memory. The storage unit 220 may include a person DB that stores facial images of the user 10, attribute information of the user 10, and the like. Of the components of the robot 100 shown in FIG. 2, the functions of the components excluding the control target 252, the sensor unit 200, and the storage unit 220 can be realized by a CPU operating based on a program. For example, the functions of these components can be implemented as CPU operations by an operating system (OS) and a program running on the OS.

[0031] The sensor module unit 210 includes a voice emotion recognition unit 211, a speech understanding unit 212, a facial expression recognition unit 213, and a face recognition unit 214. Information detected by the sensor unit 200 is input to the sensor module unit 210. The sensor module unit 210 analyzes the information detected by the sensor unit 200 and outputs the analysis result to the user state recognition unit 230.

[0032] The voice emotion recognition unit 211 of the sensor module unit 210 analyzes the voice of the user 10 detected by the microphone 201 and recognizes the emotion of the user 10. For example, the voice emotion recognition unit 211 extracts feature quantities such as frequency components of the voice and recognizes the emotion of the user 10 based on the extracted feature quantities. The speech understanding unit 212 analyzes the voice of the user 10 detected by the microphone 201 and outputs text information representing the content of the utterance of the user 10.

[0033] The facial expression recognition unit 213 recognizes the facial expression and emotion of the user 10 from the image of the user 10 captured by the 2D camera 203. For example, the facial expression recognition unit 213 recognizes the facial expression and emotion of the user 10 based on the shapes, positional relationships, etc. of the eyes and mouth.

[0034] The face recognition unit 214 recognizes the face of the user 10. The face recognition unit 214 recognizes the user 10 by matching a face image stored in a person DB (not shown) with a face image of the user 10 captured by the 2D camera 203.

[0035] The user state recognition unit 230 recognizes the state of the user 10 based on the information analyzed by the sensor module unit 210. For example, it mainly performs processing related to perception using the analysis results of the sensor module unit 210. For example, it generates perceptual information such as "Dad is alone" or "There is a 90% chance that Dad is not smiling." It then performs processing to understand the meaning of the generated perceptual information. For example, it generates semantic information such as "Dad is alone and looks lonely." The user state recognition unit 230 also stores the information detected by the sensor unit 200 in the storage unit 200 and analyzes the information detected by the sensor unit 200.

[0036] The emotion determination unit 232 determines an emotion value indicating the emotion of the user 10 based on the information analyzed by the sensor module unit 210 and the state of the user 10 recognized by the user state recognition unit 230. For example, the information analyzed by the sensor module unit 210 and the recognized state of the user 10 are input into a pre-trained neural network to obtain an emotion value indicating the emotion of the user 10.

[0037] Here, the emotion value indicating the emotion of the user 10 is a value indicating the positive or negative emotion of the user. For example, if the user's emotion is a cheerful emotion accompanied by a sense of pleasure or comfort, such as "joy," "pleasure," "comfort," "relief," "excitement," "relief," and "fulfillment," it will show a positive value, and the cheerfulr the emotion, the larger the value. If the user's emotion is a negative emotion, such as "anger," "sorrow," "discomfort," "anxiety," "sorrow," "worry," and "emptiness," it will show a negative value, and the more unpleasant the emotion, the larger the absolute value of the negative value. If the user's emotion is none of the above ("neutral"), it will show a value of 0.

[0038] The behavior recognition unit 234 recognizes the behavior of the user 10 based on the information analyzed by the sensor module unit 210 and the state of the user 10 recognized by the user state recognition unit 230. For example, the information analyzed by the sensor module unit 210 and the recognized state of the user 10 are input into a pre-trained neural network, the probability of each of a plurality of predetermined behavior classifications (for example, "laughing," "angry," "asking a question," and "sad") is obtained, and the behavior classification with the highest probability is recognized as the behavior of the user 10.

[0039] If the emotion value of the user 10 determined by the emotion determination unit 232 is a positive value, the behavior determination unit 236 determines the behavior to be performed as an action that imitates the behavior of the user 10.

[0040] If the emotion value of the user 10 determined by the emotion determination unit 232 is not a positive value, i.e., if the emotion value of the user 10 is a negative value or 0, the behavior determination unit 236 determines the behavior of the robot 100 based on the user behavior recognized by the behavior recognition unit 234 and the reaction rules 221. At this time, the reaction rules 221 prescribe a behavior for changing the emotion value of the user 19 to a positive value in response to a user behavior corresponding to a negative emotion. Specifically, the reaction rules 221 prescribe the behavior of "apologizing" as the behavior of the robot 100 when the user 10 behavior is "anger." Furthermore, the reaction rules 221 prescribe the behavior of "calling out" as the behavior of the robot 100 when the user 10 behavior is "sad." Furthermore, the reaction rules 221 prescribe the behavior of "answering" as the behavior of the robot 100 in response to a user behavior corresponding to an emotion value of 0, such as "asking a question."

[0041] The behavior control unit 250 controls the control target 252 based on the behavior determined by the behavior determination unit 236. For example, when the behavior determination unit 236 determines an behavior including speaking, the behavior control unit 250 outputs a sound from a speaker included in the control target 252. At this time, the behavior control unit 250 may determine the speaking speed of the sound based on the emotion value of the robot 100.

[0042] Furthermore, when the behavior determination unit 236 determines an action to imitate the action of the user 10, the behavior control unit 250 outputs a sound corresponding to the voice of the user 10 from a speaker, expresses an expression corresponding to the expression of the user 10 by the light emission state of LEDs in the eyes of the robot 100, and expresses a posture and gesture of the robot 100 corresponding to the movement of the user 10 by driving motors of the arms, hands, feet, etc. In this way, when the action of the user 10 is "laughing", the robot 100 can "laugh" together with the user 10 by imitating the action of the user 10.

[0043] The behavior control unit 250 may recognize a change in the user 10's emotion in response to the execution of the behavior determined by the behavior determination unit 236. For example, the change in emotion may be recognized based on the voice or facial expression of the user 10. Alternatively, the change in emotion of the user 10 may be recognized based on the detection of an impact by a touch sensor included in the sensor unit 200. If an impact is detected by the touch sensor included in the sensor unit 200, the behavior control unit 250 may recognize that the user 10's emotion has worsened. If the detection result of the touch sensor included in the sensor unit 200 indicates that the user 10 is smiling, happy, or the like, the behavior control unit 250 may recognize that the user 10's emotion has improved. Information indicating the user 10's reaction is output to the communication processing unit 280.

[0044] The communication processing unit 280 is responsible for communication with the server 300. As described above, the communication processing unit 280 transmits user reaction information to the server 300. The communication processing unit 280 also receives updated reaction rules from the server 300. When the communication processing unit 280 receives the updated reaction rules from the server 300, it updates the reaction rules 221.

[0045] The server 300 communicates between the robot 100, the robot 101, and the robot 102 and the server 300, receives user reaction information sent from the robot 100, and updates the reaction rules based on reaction rules that include actions that have received positive reactions.

[0046] Fig. 3 shows an example of an operation flow relating to an operation for determining an action in the robot 100. The operation flow shown in Fig. 3 is repeatedly executed. At this time, it is assumed that information analyzed by the sensor module unit 210 is input. Note that "S" in the operation flow indicates the step that is executed.

[0047] First, in step S100 , the user state recognition unit 230 recognizes the state of the user 10 based on the information analyzed by the sensor module unit 210 .

[0048] In step S102, the emotion determination unit 232 determines an emotion value indicating the emotion of the user 10 based on the information analyzed by the sensor module unit 210 and the state of the user 10 recognized by the user state recognition unit 230.

[0049] In step S104, it is determined whether the emotion value of user 10 determined in step S102 above is a positive value. If the emotion value of user 10 determined in step S102 above is a positive value, the process proceeds to step S106. On the other hand, if the emotion value of user 10 determined in step S102 above is a negative value or 0, the process proceeds to step S108.

[0050] In step S106, the behavior determining unit 236 determines a behavior that imitates the behavior of the user 10 as the behavior to be executed.

[0051] In step S108, the behavior recognition unit 234 recognizes the behavior classification of the user 10 based on the information analyzed by the sensor module unit 210 and the state of the user 10 recognized by the user state recognition unit 230.

[0052] In step S110, the behavior determining unit 236 determines the behavior of the robot 100 based on the user's behavior recognized by the behavior recognizing unit 234 and the reaction rule 221.

[0053] In step S112, the behavior control unit 250 controls the control target 252 based on the behavior determined by the behavior determination unit 236.

[0054] As described above, the robot 100 according to the first embodiment can cause the robot 100 to perform an appropriate action in response to the user's 10 behavior. Conventionally, the user's behavior is classified and the robot's behavior, including facial expressions and posture, is determined. In contrast, the robot 100 determines the user's 10 emotional value and, if the user's emotional value is positive, imitates the user's 10 behavior. Imitating the user's 10 behavior, which represents a positive emotional state, does not cause discomfort to the user 10 and can avoid problems caused by the robot 10 misrecognizing the user's 10 behavior. Furthermore, if the user's emotional state is negative, the robot 100 recognizes the user's 10 behavior classification using a neural network and performs an action on the user 10 based on a reaction rule. This allows the robot 100 to perform an action such as speaking to the user 10 to cheer them up and bring them to a positive emotional state, for example, if the user 10 appears sad. Furthermore, if the user's emotional value is 0, the robot 100 recognizes the user's behavior classification using a neural network and performs an appropriate action on the user's 10 behavior based on the reaction rule.

[0055] Note that the behavior determining unit 236 may determine, as the behavior to be performed, an behavior that imitates the behavior of the user when the emotion value is a positive value and equal to or greater than the threshold value. Furthermore, the behavior determining unit 236 may determine, as the behavior to be performed, an behavior that is determined based on a reaction rule according to the behavior classification of the user when the emotion value is a positive value and less than the threshold value, or when the emotion value is a negative value. This allows the robot 100 to perform an action that imitates the behavior of the user 10 when the emotion is strong positive, and to determine and perform the behavior of the robot 100 based on the reaction rule 221 in other cases.

[0056] The robot 100 is an example of an electronic device equipped with a behavior control system. The application of the behavior control system is not limited to the robot 100, and the behavior control system can be applied to various electronic devices. The functions of the server 300 may be implemented by one or more computers. At least some of the functions of the server 300 may be implemented by a virtual machine. At least some of the functions of the server 300 may be implemented in the cloud.

[0057] [Second embodiment] In the first embodiment, when the emotion value of the user 10 is negative, the robot 100 executes an action determined based on the reaction rule against the user 10, for example, by changing the emotion of the user 10 to a positive emotion. However, when the emotion value of the user 10 is negative and the magnitude of the negative value, i.e., the absolute value of the negative value, is too large, it is conceivable that the emotion of the user 10 becomes unstable and the user 10 may behave unexpectedly. In this case, it is preferable that the robot 10 notify a supervisor of the user 10, such as a parent, a teacher at school, or a boss at work, that the user 10 is feeling a strong negative emotion.

[0058] In the second embodiment, an operation of the robot 100 that notifies the supervisor of the user 10 of the state of the user 10 when the absolute value of the negative emotion value of the user 10 is equal to or greater than a predetermined threshold will be described.

[0059] The configuration of the system 5 and the functional configuration of the robot 100 in the second embodiment are the same as those shown in FIGS. 1 and 2, respectively.

[0060] Fig. 4 shows an example of an operation flow relating to an operation for determining an action in the robot 100 of the second embodiment. The operation flow shown in Fig. 4 is repeatedly executed. At this time, it is assumed that information analyzed by the sensor module unit 210 has been input.

[0061] The operation flow shown in Fig. 4 differs from the operation flow according to the first embodiment shown in Fig. 3 in that steps S105, S107, S109, and S111 have been added, and the other processes are the same as those in the operation flow shown in Fig. 3. Therefore, hereinafter, the operation flow shown in Fig. 4 will be described, focusing on the differences from the operation flow shown in Fig. 3.

[0062] In step S104 of the operational flow in FIG. 4, if the emotion value of the user 10 determined in step S102 is a negative value or 0, the process proceeds to step S105.

[0063] In step S105, it is determined whether the absolute value of the emotion value of user 10 is equal to or greater than a predetermined threshold H. Threshold H is a predetermined value that indicates that if the emotion value of user 10 is equal to or greater than this value, it is necessary to be concerned about the state of user 10. Threshold H is changeable and is stored in advance in storage unit 220, for example. If the absolute value of the emotion value of user 10 is less than threshold H, the process proceeds to step S108, and the processes from step S108 onwards that have already been described in the first embodiment are executed.

[0064] That is, if the absolute value of the emotion value of the user 10 is less than the threshold value H, the control object 252 is controlled based on the behavior of the user 10 recognized by the behavior recognition unit 234 and the behavior of the robot 100 determined based on the reaction rule 221.

[0065] On the other hand, if the absolute value of the emotion value of user 10 is equal to or greater than threshold H, the process moves to step S107.

[0066] In step S107, the behavior determining unit 236 determines an behavior to notify the supervisor of the user 10 that the emotion value of the user 10 is a negative value and that an extremely negative emotion value with an absolute value equal to or greater than the threshold value H has been recognized.

[0067] In step S109, the user state recognition unit 230 starts acquiring at least one of an image of the user 10 via the 2D camera 203 and the content of the user 10's speech via the microphone 201, and records the state of the user 10. "Recording" refers to storing the acquired information so that the acquired information can be obtained later. Specifically, the user state recognition unit 230 stores the acquired state of the user 10 in the storage unit 220.

[0068] Note that, since negative emotions can cause a drop in body temperature, which can lead to symptoms such as dizziness in the user 10, the user state recognition unit 230 may obtain and record the body temperature of the user 10 from the 3D depth sensor 202.

[0069] Furthermore, the user state recognition unit 230 may record the results of converting the contents of the user's utterance collected through the microphone 201 into text information.

[0070] While continuing to record the state of the user 10, in step S111, the behavior decision unit 236 decides that the behavior to be executed is to rush to the location of the user 10. The location of the user 10 refers to, for example, an area within which the robot 100 can come into contact with the user.

[0071] Therefore, in step S112, the behavior control unit 250 refers to the email address of the supervisor for each user 10 stored in advance in the storage unit 220, and transmits information to the email address of the supervisor of the user 10 for whom an extremely negative emotional value is recognized. That is, the robot 100 notifies an information device such as a smartphone carried by the supervisor of the user 10 of the state of the user 10. The behavior control unit 250 also controls the control target 252 so that the robot 100 moves to the location of the user 10.

[0072] As described above, according to the robot 100 of the second embodiment, when the emotion value of the user 10 is a negative emotion value with an absolute value equal to or greater than the threshold H, the supervisor of the user 10 is notified that an extremely negative emotion value is recognized in the user 10. This enables the supervisor of the user 10 to take care of the user 10 whose emotions are unstable.

[0073] In addition, the robot 100 rushes to the location of the user 10 and records the user 10's behavior, making it possible to check the user 10's behavior in detail later, compared to when the robot 100 is located far away from the user 10.

[0074] By checking the recorded behavior of the user 10, the robot 100 can take action according to the behavior of the user 10. For example, if the user 10 is arguing, the robot 100 may check the situation of the argument and the surrounding circumstances and later identify the cause of the argument. In some cases, the robot 100 may mediate the argument based on the recorded behavior of the user 10.

[0075] Furthermore, the robot 100 notifies the supervisor of the user 10 of the state of the user 10 when an extremely negative emotional value is detected in step S107 of FIG. 4, but the robot 100 may also notify the supervisor of the user 10 of the state of the user 10 when the user 10 is a child and comes home crying. In this way, the robot 100 may determine its own behavior based on the behavior of the user 10 confirmed from the records.

[0076] 5 schematically shows an example of the hardware configuration of a computer 1200 that functions as the robot 100 and the server 300 according to the first and second embodiments. A program installed on the computer 1200 can cause the computer 1200 to function as one or more "parts" of an apparatus according to the embodiments, or can cause the computer 1200 to perform operations associated with the apparatus according to the embodiments or one or more "parts," and / or can cause the computer 1200 to perform a process or steps of the process according to the embodiments. Such a program can be executed by the CPU 1212 to cause the computer 1200 to perform specific operations associated with some or all of the blocks in the flowcharts and block diagrams described herein.

[0077] The computer 1200 according to this embodiment includes a CPU 1212, a RAM 1214, and a graphics controller 1216, which are interconnected by a host controller 1210. The computer 1200 also includes input / output units such as a communications interface 1222, a storage device 1224, a DVD drive 1226, and an IC card drive, which are connected to the host controller 1210 via an input / output controller 1220. The DVD drive 1226 may be a DVD-ROM drive, a DVD-RAM drive, or the like. The storage device 1224 may be a hard disk drive, a solid-state drive, or the like. The computer 1200 also includes a ROM 1230 and legacy input / output units such as a keyboard, which are connected to the input / output controller 1220 via an input / output chip 1240.

[0078] The CPU 1212 operates according to programs stored in the ROM 1230 and the RAM 1214, thereby controlling each unit. The graphics controller 1216 acquires image data generated by the CPU 1212 into a frame buffer or the like provided in the RAM 1214 or into the graphics controller itself, and causes the image data to be displayed on the display device 1218.

[0079] The communication interface 1222 communicates with other electronic devices via a network. The storage device 1224 stores programs and data used by the CPU 1212 in the computer 1200. The DVD drive 1226 reads programs or data from a DVD-ROM 1227 or the like and provides the programs or data to the storage device 1224. The IC card drive reads programs and data from an IC card and / or stores the programs and data in a DVD-ROM drive. Write it to the IC card.

[0080] The ROM 1230 stores therein a boot program or the like that is executed by the computer 1200 upon activation, and / or programs that depend on the hardware of the computer 1200. The input / output chip 1240 may also connect various input / output units to the input / output controller 1220 via a USB port, a parallel port, a serial port, a keyboard port, a mouse port, etc.

[0081] The programs are provided by a computer-readable storage medium such as a DVD-ROM 1227 or an IC card. The programs are read from the computer-readable storage medium, installed in the storage device 1224, RAM 1214, or ROM 1230, which are also examples of computer-readable storage media, and executed by the CPU 1212. Information processing described in these programs is read by the computer 1200, and causes cooperation between the programs and the various types of hardware resources described above. An apparatus or method may be configured by implementing operations or processing of information in accordance with the use of the computer 1200.

[0082] For example, when communication is performed between the computer 1200 and an external device, the CPU 1212 may execute a communication program loaded into the RAM 1214 and instruct the communication interface 1222 to perform communication processing based on the processing described in the communication program. Under the control of the CPU 1212, the communication interface 1222 reads transmission data stored in a transmission buffer area provided in the RAM 1214, the storage device 1224, the DVD-ROM 1227, or a recording medium such as an IC card, and transmits the read transmission data to the network, or writes reception data received from the network to a reception buffer area or the like provided on the recording medium.

[0083] Furthermore, the CPU 1212 may cause all or a necessary portion of a file or database stored in an external recording medium such as the storage device 1224, the DVD drive 1226 (DVD-ROM 1227), an IC card, etc. to be read into the RAM 1214, and may perform various types of processing on the data on the RAM 1214. The CPU 1212 may then write back the processed data to the external recording medium.

[0084] Various types of information, such as various types of programs, data, tables, and databases, may be stored on the recording medium and may undergo information processing. The CPU 1212 may perform various types of processing on data read from the RAM 1214, including various types of operations, information processing, conditional judgment, conditional branching, unconditional branching, information search / replacement, etc., as described throughout this disclosure and specified by the instruction sequences of the programs, and write the results back to the RAM 1214. The CPU 1212 may also search for information in a file, database, etc. on the recording medium. For example, if multiple entries, each having an attribute value of a first attribute associated with an attribute value of a second attribute, are stored on the recording medium, the CPU 1212 may search for an entry whose attribute value of the first attribute matches a specified condition from among the multiple entries, read the attribute value of the second attribute stored in the entry, and thereby obtain the attribute value of the second attribute associated with the first attribute that satisfies a predetermined condition.

[0085] The above-described programs or software modules may be stored in a computer-readable storage medium on or near the computer 1200. A recording medium such as a hard disk or RAM provided in a server system connected to a dedicated communication network or the Internet can also be used as a computer-readable storage medium, thereby providing the programs to the computer 1200 via the network.

[0086] The blocks in the flowcharts and block diagrams in the present embodiments may represent stages of a process in which an operation is performed or "parts" of an apparatus responsible for performing the operation. Particular stages and "parts" may be implemented by dedicated circuitry, programmable circuitry provided with computer-readable instructions stored on a computer-readable storage medium, and / or a processor provided with computer-readable instructions stored on a computer-readable storage medium. The dedicated circuitry may include digital and / or analog hardware circuits, including integrated circuits (ICs) and / or discrete circuits. The programmable circuitry may include reconfigurable hardware circuits, such as field programmable gate arrays (FPGAs) and programmable logic arrays (PLAs), including AND, OR, XOR, NAND, NOR, and other logical operations, flip-flops, registers, and memory elements.

[0087] A computer-readable storage medium may include any tangible device capable of storing instructions that are executed by an appropriate device, such that a computer-readable storage medium having instructions stored thereon comprises an article of manufacture, including instructions that can be executed to create means for performing the operations specified in the flowcharts or block diagrams. Examples of computer-readable storage media may include electronic storage media, magnetic storage media, optical storage media, electromagnetic storage media, semiconductor storage media, etc. More specific examples of computer-readable storage media may include floppy disks, diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), electrically erasable programmable read-only memory (EEPROM), static random access memory (SRAM), compact disc read-only memory (CD-ROM), digital versatile disc (DVD), Blu-ray disc, memory stick, integrated circuit card, etc.

[0088] The computer readable instructions may include either assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state-setting data, or source or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, JAVA, C++, etc., and conventional procedural programming languages ​​such as the "C" programming language or similar programming languages.

[0089] Computer-readable instructions may be provided locally or over a wide area network (WAN) such as a local area network (LAN), the Internet, etc. to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus, or programmable circuitry, such that the processor or programmable circuitry executes the computer-readable instructions to generate means for performing the operations specified in the flowcharts or block diagrams. Examples of processors include computer processors, processing units, microprocessors, digital signal processors, controllers, microcontrollers, etc.

[0090] Although the present invention has been described above using the embodiments, the technical scope of the present invention is not limited to the scope described in the above embodiments. It will be apparent to those skilled in the art that various modifications and improvements can be made to the above embodiments. It is clear from the claims that such modifications and improvements can also be included within the technical scope of the present invention.

[0091] The device, system, program, and It should be noted that the order of execution of each process, such as an operation, procedure, step, or stage in a method, is not specifically stated as "before" or "prior to," and that the process may be performed in any order unless the output of a previous process is used in a subsequent process. Even if the operational flow in the claims, specification, and drawings is described using "first," "next," etc. for convenience, this does not mean that the process must be performed in this order. [Explanation of symbols]

[0092] 5 System, 10, 11, 12 User, 20 Communication network, 100, 101, 102 Robot, 200 Sensor unit, 201 Microphone, 202 Depth sensor, 203 Camera, 204 Distance sensor, 210 Sensor module unit, 211 Voice emotion recognition unit, 212 Speech understanding unit, 213 Facial expression recognition unit, 214 Face recognition unit, 220 Storage unit, 221 Response rule, 230 User state recognition unit, 232 Emotion determination unit, 234 Action recognition unit, 236 Action determination unit, 250 Action control unit, 252 Control target, 280 Communication processing unit, 300 Server, 1200 Computer, 1210 Host controller, 1212 CPU, 1214 RAM, 1216 Graphics controller, 1218 Display device, 1220 Input / output controller, 1222 Communication interface, 1224 Storage device, 1226 DVD drive, 1227 DVD-ROM, 1230 ROM, 1240 Input / Output Chip

Claims

1. A user state recognition unit that recognizes a user state including a user's behavior; an emotion determining unit that determines an emotion value indicating an emotion of the user based on the user state; If the emotion value is a positive value, determining an action to be performed that corresponds to an action of the user; When the emotion value is not a positive value and the absolute value is less than a threshold, a behavior to be executed is determined based on a reaction rule in accordance with a behavior classification of the user; an action decision unit that decides, when the emotion value is a negative value and the absolute value is equal to or greater than the threshold value, to notify a supervisor of the user of the user's state as an action to be executed; A behavior control system including:

2. The action determination unit is If the emotion value is a negative value and the absolute value is equal to or greater than the threshold value, the robot determines an action of rushing to the location of the user. The behavior control system according to claim 1.

3. The user state recognition unit If the emotion value is a negative value and the absolute value is equal to or greater than the threshold value, the state of the user is recorded using at least one of a microphone and a camera. The behavior control system according to claim 1.

4. The user state recognition unit records at least one of the speech content of the user collected by the microphone and the image of the user captured by the camera as the state of the user. The behavior control system according to claim 3.

5. The user state recognition unit converts the user's speech content collected by the microphone into text information and records the result. The behavior control system according to claim 4.

6. A program for causing a computer to function as the behavior control system according to any one of claims 1 to 5.

Images