Action control system and program
Patent Information
- Application Number
- JP2024037892
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-06-23
- Filing Date
- 2024-03-12
- Publication Date
- 2025-01-09
Smart Images
Figure 2025003299000001_ABST
Abstract
Description
[Technical field]
[0001] The present invention relates to a behavior control system and a program. [Background technology]
[0002] A behavior control system is known that includes a behavior information storage unit that stores decision information for deciding an action based on a user's state, a user state recognition unit that recognizes the user's state, a behavior decision unit that decides an action to be performed based on the user's state recognized by the user state recognition unit and the decision information, a reaction recognition unit that recognizes the user's reaction to performing the action decided by the behavior decision unit, a transmission unit that transmits the user's state recognized by the user state recognition unit, the action decided by the behavior decision unit, and the user's reaction recognized by the reaction recognition unit to an external server, a receiving unit that receives from the server an action appropriate to the user's state recognized by the user state recognition unit when the behavior decision unit is unable to decide an action to be performed based on the user's state and the decision information, and a behavior information update unit that updates the decision information based on the action received by the receiving unit from the server (see, for example, Patent Document 1).
[0003] [Prior art document]
[0004] [Patent documents]
[0005] [Patent Document 1] Patent No. 6053847 Summary of the Invention [Problem to be solved by the invention]
[0006] It is desirable to have a robot perform appropriate actions in response to questions from a user. [Means for solving the problem]
[0007] According to a first aspect of the present invention, there is provided a behavior control system including: a behavior recognition unit that recognizes a user's behavior; a difficulty estimation unit that estimates the difficulty of the question when the user's behavior is recognized as a question; and a behavior determination unit that determines, as behaviors to be executed, a behavior of being happy to be asked a question by the user and a behavior of answering the question when a score according to the difficulty is equal to or greater than a threshold.
[0008] The behavior decision unit may determine, as the behavior to be executed when the cumulative value of the score is equal to or greater than a threshold value, or when the number of times the score according to the difficulty level has been equal to or greater than the threshold value is equal to or greater than a predetermined number of times, a behavior of enjoying being asked a question by the user and a behavior of answering the question.
[0009] In a second aspect, there is provided a program that causes a computer to function as the above-mentioned behavior control system.
[0010] The above summary of the invention does not list all of the necessary features of the present invention. Also, subcombinations of these features may also be inventions. [Brief description of the drawings]
[0011] [Figure 1] 1 illustrates an example of a system 5 according to a first embodiment. [Diagram 2] 1 illustrates a schematic functional configuration of a robot 100 according to a first embodiment. [Diagram 3] 2 shows an example of an operation flow of the robot 100 according to the first embodiment. [Figure 4] 13 shows a schematic functional configuration of a robot 100 according to a second embodiment. [Diagram 5] 13 shows an example of an utterance for a response action of the robot 100 according to the second embodiment. [Figure 6] 11 shows an example of the movement of a response behavior of the robot 100 according to the second embodiment. [Figure 7]13A and 13B are schematic diagrams illustrating an example of an operation flow of the robot 100 according to the second embodiment. [Figure 8] 1 illustrates an example of a hardware configuration of a computer 1200. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0012] The present invention will be described below through embodiments of the invention, but the following embodiments do not limit the scope of the invention according to the claims. Furthermore, not all of the combinations of features described in the embodiments are necessarily essential to the solution of the invention.
[0013] (First embodiment) FIG. 1 is a schematic diagram of an example of a system 5 according to the present embodiment. The system 5 includes a robot 100, a robot 101, a robot 102, and a server 300. A user 10a, a user 10b, a user 10c, and a user 10d are users of the robot 100. A user 11a, a user 11b, and a user 11c are users of the robot 101. A user 12a and a user 12b are users of the robot 102. In the description of the present embodiment, the user 10a, the user 10b, the user 10c, and the user 10d may be collectively referred to as the user 10. The user 11a, the user 11b, and the user 11c may be collectively referred to as the user 11. The user 12a and the user 12b may be collectively referred to as the user 12. The robot 101 and the robot 102 have substantially the same functions as the robot 100. Therefore, the system 5 will be described by mainly focusing on the functions of the robot 100.
[0014] The robot 100 converses with the user 10 and provides images to the user 10. At this time, the robot 100 cooperates with a server 300 or the like with which it can communicate via a communication network 20 to converse with the user 10 and provide images, etc. to the user 10. For example, the robot 100 not only learns appropriate conversation by itself, but also cooperates with the server 300 to learn how to have a more appropriate conversation with the user 10. The robot 100 also causes the server 300 to record captured image data, etc. of the user 10, and requests the server 300 for the image data, etc. as necessary, and provides the data, etc. to the user 10.
[0015] The robot 100 also has an emotion value that indicates the type of its own emotion. For example, the robot 100 has emotion values that indicate the intensity of each of the emotions of "joy", "anger", "sorrow", "pleasure", "discomfort", "relief", "anxiety", "sadness", "excitement", "concern", "relief", "fulfillment", "emptiness", and "neutral". When the robot 100 converses with the user 10 in a state where the emotion value of excitement is high, for example, the robot 100 utters sounds at a fast speed. In this way, the robot 100 can express its own emotion through actions.
[0016] The robot 100 also has a function of recognizing the behavior of the user 10. The robot 100 recognizes the behavior of the user 10 by analyzing a facial image of the user 10 acquired by a camera function and a voice of the user 10 acquired by a microphone function. The robot 100 determines the behavior to be performed by the robot 100 based on the recognized behavior of the user 10, etc.
[0017] The robot 100 stores rules that define actions that the robot 100 will execute based on the actions of the user 10, and performs various actions according to the rules.
[0018] Specifically, the robot 100 has reaction rules for determining the behavior of the robot 100 based on the behavior of the user 10. For example, the reaction rules define the behavior of the robot 100 as "laughing" when the behavior of the user 10 is "laughing". The reaction rules also define the behavior of the robot 100 as "apologizing" when the behavior of the user 10 is "getting angry". The reaction rules also define the behavior of the robot 100 as "answering" when the behavior of the user 10 is "asking a question". The reaction rules also define the behavior of the robot 100 as "calling out" when the behavior of the user 10 is "getting sad".
[0019] When the robot 100 recognizes that the behavior of the user 10 is "laughing" based on the reaction rules, the robot 100 selects the behavior of "laughing" defined by the reaction rules as the behavior to be executed by the robot 100. For example, when the robot 100 selects the behavior of "laughing", the robot 100 performs a "laughing" motion and outputs a sound representing laughter. This allows the robot 100 to laugh together with the user 10.
[0020] The robot 100 also transmits to the server 300 user reaction information indicating that this action has elicited a positive reaction from the user 10. The user reaction information includes, for example, the user action of "laughing", the robot 100 action of "laughing", the fact that the reaction of the user 10 was positive, and attributes of the user 10.
[0021] The server 300 stores the user reaction information received from the robot 100. The server 300 receives and stores user reaction information not only from the robot 100, but also from each of the robots 101 and 102. The server 300 analyzes the user reaction information from the robots 100, 101, and 102, and updates the reaction rules.
[0022] The robot 100 receives the updated reaction rules from the server 300 by inquiring about the updated reaction rules from the server 300. The robot 100 incorporates the updated reaction rules into the reaction rules stored in the robot 100. This allows the robot 100 to incorporate the reaction rules acquired by the robot 101, the robot 102, etc. into its own reaction rules.
[0023] 2 shows a schematic functional configuration of the robot 100. The robot 100 includes a sensor unit 200, a sensor module unit 210, a storage unit 220, a user state recognition unit 230, a behavior recognition unit 232, a difficulty estimation unit 234, a behavior determination unit 236, a behavior control unit 250, a control target 252, and a communication processing unit 280.
[0024] The control target 252 includes a display device, a speaker, LEDs in the eyes, and motors for driving the arms, hands, legs, etc. The posture and gestures of the robot 100 are controlled by controlling the motors of the arms, hands, legs, etc. Some of the emotions of the robot 100 can be expressed by controlling these motors. Some of the emotions of the robot 100 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 100. The posture, gestures, and facial expressions of the robot 100 are examples of the attitude of the robot 100.
[0025] The sensor unit 200 includes a microphone 201, a 3D depth sensor 202, a 2D camera 203, and a distance sensor 204. The microphone 201 continuously detects sound and outputs sound data. The microphone 201 may be provided on the head of the robot 100 and may have a function of performing binaural recording. The 3D depth sensor 202 continuously emits an infrared pattern and analyzes the infrared pattern from infrared images continuously captured by the infrared camera to detect the location of the object. , and detects the contour of an object. The 2D camera 203 is an example of an image sensor. The 2D camera 203 captures an image using visible light and generates visible light video information. The distance sensor detects the distance to an object by emitting, for example, a laser or ultrasonic waves. In addition, the sensor unit 200 may also include a clock, a gyro sensor, a touch sensor, a sensor for motor feedback, and the like.
[0026] 2, the components other than the control target 252 and the sensor unit 200 are examples of components included in a behavior control system included in the robot 100. The behavior control system of the robot 100 controls the control target 252.
[0027] The storage unit 220 includes reaction rules 221. At least a part of the storage unit 220 is implemented by a storage medium such as a memory. It may include a person DB that stores the face image of the user 10, attribute information of the user 10, and the like. Among the components of the robot 100 shown in FIG. 2, the functions of the components excluding the control target 252, the sensor unit 200, and the storage unit 220 can be realized by a CPU operating based on a program. For example, the functions of these components can be implemented as the operation of a CPU by an operating system (OS) and a program operating on the OS.
[0028] The sensor module unit 210 includes a voice emotion recognition unit 211, a speech understanding unit 212, a facial expression recognition unit 213, and a face recognition unit 214. Information detected by the sensor unit 200 is input to the sensor module unit 210. The sensor module unit 210 analyzes the information detected by the sensor unit 200, and outputs the analysis result to the user state recognition unit 230.
[0029] The voice emotion recognition unit 211 of the sensor module unit 210 analyzes the voice of the user 10 detected by the microphone 201 and recognizes the emotion of the user 10. For example, the voice emotion recognition unit 211 extracts feature amounts such as frequency components of the voice and recognizes the emotion of the user 10 based on the extracted feature amounts. The speech understanding unit 212 analyzes the voice of the user 10 detected by the microphone 201 and outputs text information representing the content of the utterance of the user 10.
[0030] The facial expression recognition unit 213 recognizes the facial expression and emotion of the user 10 from the image of the user 10 captured by the 2D camera 203. For example, the facial expression recognition unit 213 recognizes the facial expression and emotion of the user 10 based on the shapes, positional relationship, etc. of the eyes and mouth.
[0031] The face recognition unit 214 recognizes the face of the user 10. The face recognition unit 214 recognizes the user 10 by matching a face image stored in a person DB (not shown) with a face image of the user 10 captured by the 2D camera 203.
[0032] The user state recognition unit 230 recognizes the state of the user 10 based on the information analyzed by the sensor module unit 210. For example, the user state recognition unit 230 mainly performs processing related to perception using the analysis results of the sensor module unit 210. For example, it generates perceptual information such as "Daddy is alone" or "There is a 90% chance that Daddy is not smiling." It performs processing to understand the meaning of the generated perceptual information. For example, it generates semantic information such as "Daddy is alone and looks lonely."
[0033] The behavior recognition unit 232 recognizes the behavior of the user 10 based on the information analyzed by the sensor module unit 210 and the state of the user 10 recognized by the user state recognition unit 230. For example, the information analyzed by the sensor module unit 210 and the recognized state of the user 10 are input to a pre-trained neural network, the probability of each of a plurality of predetermined behavior classifications (for example, "laughing," "getting angry," "asking a question," and "getting sad") is obtained, and the behavior classification with the highest probability is recognized as the behavior of the user 10.
[0034] When the user's behavior recognized by the behavior recognition unit 232 is recognized as "asking a question," the difficulty estimation unit 234 estimates a score according to the difficulty of the question based on the text information representing the content of the user's utterance from among the information analyzed by the sensor module unit 210. For example, the text information representing the content of the user's utterance is input to a pre-trained neural network to obtain a score according to the difficulty of the question. Note that the score is assumed to be higher as the difficulty of the question increases.
[0035] The behavior determination unit 236 determines the behavior of the robot 100 based on the behavior of the user recognized by the behavior recognition unit 232 and the reaction rules 221. At this time, if the score according to the difficulty level estimated by the difficulty estimation unit 234 is equal to or greater than a threshold, the behavior determination unit 236 determines, as the behavior to be executed, a behavior of being happy to be asked a question by the user 10 and a behavior of answering the question. Here, "a behavior of being happy to be asked a question by the user" is, for example, a proud behavior (a boastful behavior) when asked a question by the user.
[0036] The behavior control unit 250 controls the control target 252 based on the behavior determined by the behavior determination unit 236. For example, when the behavior determination unit 236 determines an behavior including speaking, the behavior control unit 250 outputs a sound from a speaker included in the control target 252. At this time, the behavior control unit 250 may determine the speaking speed of the sound based on the emotion value of the robot 100.
[0037] In addition, when the behavior decision unit 236 decides on "behavior that is pleasing to be asked a question by the user," the behavior control unit 250 outputs a voice that expresses pride (for example, "I'm good at answering questions like this") from the speaker, and expresses a posture and gesture of the robot 100 that expresses pride (for example, the movement of slowly raising one hand) by driving the motors of the arms, hands, legs, etc.
[0038] The behavior control unit 250 may recognize a change in the user 10's emotion in response to the execution of the behavior determined by the behavior determination unit 236. For example, the change in emotion may be recognized based on the voice or facial expression of the user 10. In addition, the change in emotion of the user 10 may be recognized based on the detection of an impact by a touch sensor included in the sensor unit 200. When an impact is detected by the touch sensor included in the sensor unit 200, the emotion of the user 10 may be recognized as having worsened, and when the reaction of the user 10 is determined to be laughing or happy, etc., based on the detection result of the touch sensor included in the sensor unit 200, the emotion of the user 10 may be recognized as having improved. Information indicating the reaction of the user 10 is output to the communication processing unit 280.
[0039] The communication processing unit 280 is responsible for communication with the server 300. As described above, the communication processing unit 280 transmits user reaction information to the server 300. In addition, the communication processing unit 280 receives updated reaction rules from the server 300. When the communication processing unit 280 receives the updated reaction rules from the server 300, it updates the reaction rules 221.
[0040] The server 300 communicates between the robots 100, 101, and 102 and the server 300, receives user reaction information transmitted from the robot 100, and updates the reaction rules based on reaction rules that include actions that have received positive reactions.
[0041] Fig. 3 shows an example of an operation flow for determining an action of the robot 100. The operation flow shown in Fig. 3 is repeatedly executed. At this time, it is assumed that information analyzed by the sensor module unit 210 is input. Note that "S" in the operation flow indicates a step to be executed.
[0042] First, in step S100, the user state recognition unit 230 Based on the information analyzed in 10, the state of the user 10 is recognized.
[0043] In step S102 , the behavior recognition unit 232 recognizes a behavior classification of the user 10 based on the information analyzed by the sensor module unit 210 and the state of the user 10 recognized by the user state recognition unit 230 .
[0044] In step S104, it is determined whether the behavior classification of the user 10 recognized in step S102 is "asking a question". If the behavior classification of the user 10 recognized in step S102 is "asking a question", the process proceeds to step S108. On the other hand, if the behavior classification of the user 10 recognized in step S102 is not "asking a question", the process proceeds to step S106.
[0045] In step S106, the behavior determining unit 236 determines the behavior of the robot 100 based on the user's behavior recognized by the behavior recognizing unit 232 and the reaction rule 221.
[0046] In step S108, the difficulty level estimation unit 234 estimates a score according to the difficulty level of the question based on the text information expressing the content of the utterance of the user 10, out of the information analyzed by the sensor module unit 210.
[0047] In step S110, it is determined whether the score according to the difficulty of the question estimated in step S108 is equal to or greater than a threshold. If the score according to the difficulty of the question estimated in step S108 is equal to or greater than the threshold, the process proceeds to step S112. On the other hand, if the score according to the difficulty of the question estimated in step S108 is less than the threshold, the process proceeds to step S114.
[0048] In step S112, the behavior determining unit 236 determines, as the behavior to be executed, a behavior of being happy to be asked a question by the user 10 and a behavior of answering the question.
[0049] In step S114, the behavior determining section 236 determines, based on the reaction rules 221, an action to be executed, which is to answer the question.
[0050] In step S 116 , the behavior control unit 250 controls the control target 252 based on the behavior determined by the behavior determining unit 236 .
[0051] As described above, the robot 100 can act in a boastful manner at an appropriate time in response to a question from a user. That is, when the difficulty of a question is higher than a threshold, the robot can adopt a boastful expression and appearance. For example, when asked the answer to a difficult calculation or when asked about a good restaurant, the robot will adopt a boastful expression and appearance while answering the question. On the other hand, when the difficulty of a question is lower than the threshold, for example, when asked a low-difficulty question such as "What time is it now?" or "What's the weather today?", the robot will answer the question without adopting a boastful expression or appearance.
[0052] In addition, when the cumulative value of the score according to the difficulty level is equal to or greater than a threshold value, or when the number of times that the score according to the difficulty level is equal to or greater than the threshold value exceeds a predetermined number of times, the behavior decision unit 236 may decide that the behavior to be executed is a behavior of being pleased to be asked a question by the user, and a behavior of answering the question.
[0053] Second embodiment In the second embodiment, when the robot 100 is asked a question by a user, if any of the users answers before the robot 100 answers, the robot 100 will respond. The second embodiment is characterized by its response behavior. Differences from the first embodiment will be described below. Note that the same components are given the same reference numerals, and detailed descriptions will be omitted.
[0054] The robot 100 not only converses with the user 10, provides images to the user 10, and answers questions from the user, but also performs responsive actions such as indicating whether the user's answers are correct or incorrect and reacting to them. For example, when a user 10a asks a question to the robot 100, if the user 10b answers before the robot 100, the robot 100 not only indicates whether the answer is correct or incorrect, but also reacts according to the attitude of the user 10a or the user 10b.
[0055] The robot 100 cooperates with a server 300 and the like with which it can communicate via the communication network 20 to converse with the user 10 and provide images and the like to the user 10. For example, the robot 100 not only learns appropriate conversation by itself, but also cooperates with the server 300 to learn how to have a more appropriate conversation with the user 10. The robot 100 also causes the server 300 to record captured image data and the like of the user 10, and requests the server 300 for the image data and the like as necessary to provide it to the user 10.
[0056] The robot 100 stores, as an example of a behavior decision model, rules that define the behaviors that the robot 100 will execute based on the emotions of the user 10, the emotions of the robot 100, and the behaviors of the user 10, and performs various behaviors according to the rules.
[0057] Specifically, the robot 100 has reaction rules for determining the behavior of the robot 100 based on the emotion of the user 10, the emotion of the robot 100, and the behavior of the user 10, as an example of the behavior decision model 223. For example, the reaction rules define the behavior of the robot 100 as "laughing" when the behavior of the user 10 is "laughing". Also, the reaction rules define the behavior of the robot 100 as "apologizing" when the behavior of the user 10 is "angry". Also, the reaction rules define the behavior of the robot 100 as "answering" when the behavior of the user 10 is "asking a question". Also, the reaction rules define the behavior of the robot 100 as "indicating the correct answer" when the behavior of the user 10 is "answering". Also, the reaction rules define the behavior of the robot 100 as "calling out" when the behavior of the user 10 is "sad".
[0058] When the robot 100 recognizes the behavior of the user 10 as "anger" based on the reaction rules, the robot 100 selects the behavior of "apologizing" defined in the reaction rules as the behavior to be executed by the robot 100. For example, when the robot 100 selects the behavior of "apologizing", the robot 100 performs the motion of "apologizing" and outputs a voice expressing the word "apologize".
[0059] In addition, it is defined that when the emotion of the robot 100 is "normal" (i.e., "happy" = 0, "anger" = 0, "sad" = 0, and "happy" = 0) and the condition that the state of the user 10 is "alone and looks lonely" is satisfied, the emotion of the robot 100 will change to "worried" and the action of "calling out" will be executed.
[0060] When the robot 100 recognizes based on the reaction rule that the current emotion of the robot 100 is "normal" and that the user 10 is alone and seems lonely, the robot 100 increases the emotion value of "sadness" of the robot 100. In addition, the robot 100 selects the action of "calling out" defined in the reaction rule as the action to be performed toward the user 10. For example, when the robot 100 selects the action of "calling out", the robot 100 converts the words "What's wrong?", which indicate concern, into a worried voice and outputs it.
[0061] The robot 100 also transmits to the server 300 user reaction information indicating that this action has elicited a positive reaction from the user 10. The user reaction information includes, for example, the user action of "getting angry," the robot 100 action of "apologizing," the fact that the reaction of the user 10 was positive, and attributes of the user 10.
[0062] The server 300 stores the user reaction information received from the robot 100. The server 300 receives and stores user reaction information not only from the robot 100, but also from each of the robots 101 and 102. The server 300 analyzes the user reaction information from the robots 100, 101, and 102, and updates the reaction rules.
[0063] The robot 100 receives the updated reaction rules from the server 300 by inquiring about the updated reaction rules from the server 300. The robot 100 incorporates the updated reaction rules into the reaction rules stored in the robot 100. This allows the robot 100 to incorporate the reaction rules acquired by the robot 101, the robot 102, etc. into its own reaction rules.
[0064] Furthermore, when the robot 100 recognizes that the behavior of the user 10 is "asking a question", the robot 100 uses the sentence generation model. Furthermore, when one of the users 10 answers a question from the user 10 before the robot 100, the robot 100 can indicate whether the answer is correct or incorrect using the sentence generation model. Note that the sentence generation model may not only execute processing within the CPU 1212 of the robot 100, but may also acquire the result of executing processing outside the robot 100, such as on the server 300.
[0065] Here, the sentence generation model may be interpreted as an algorithm and calculation for automatic dialogue processing by text. The sentence generation model is, for example, disclosed in JP2018-081444A and ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ) was disclosed As described above, this is well known, and therefore detailed description thereof will be omitted. Such a sentence generation model is configured using a large language model (LLM).
[0066] As described above, the robot 100 has a function of recognizing the behavior and state of the user 10. Specifically, the robot 100 recognizes whether the user 10 is multiple (users 10a, 10b, 10c, and 10d in the example of FIG. 1) from an image acquired by the camera function, and recognizes whether the user 10 is joking or serious by analyzing the facial expression of the user 10. The robot 100 can distinguish not only the contents of the user 10's speech but also which user is speaking by analyzing the voice of the user 10 acquired by the microphone function. The speaker may be identified by using the camera function. The speaker may also be identified by analyzing the speech direction using multiple microphones. Furthermore, the robot 100 can recognize whether the user 10 is joking or serious by analyzing the voice of the user 10. The robot 100 determines the behavior to be performed by the robot 100 based on the recognized behavior and state of the user 10 in accordance with the reaction rules of the behavior decision model 223.
[0067] The robot 100 uses the behavior decision model 223 to determine the behavior to be performed by the robot 100 based on the behavior and state of the user 10.
[0068] As described above, the behavior decision model 223 has reaction rules for determining the behavior of the robot 100 based on the emotions of the user 10, the emotions of the robot 100, and the behavior of the user 10. The robot 100 transmits to the server 300 reaction information of the user 10 and an emotion value of the robot 100 obtained as a result of repeated dialogue between the user 10 and the robot 100. The server 300 stores the user reaction information and the robot emotion value received from the robot 100. The server 300 analyzes the user reaction information not only from the robot 100 but also from the robots 101 and 102, etc., and updates the reaction rules. In other words, the behavior decision model 223 is updated by the event processing of the robot 100, etc.
[0069] 4 shows a schematic functional configuration of the second embodiment of the robot 100. The robot 100 has a sensor unit 200, a sensor module unit 210, a storage unit 220, a control unit 228, and a control target 252. The control unit 228 includes a state recognition unit 230, an emotion determination unit 231, a behavior recognition unit 232, an acquisition unit 235, a behavior determination unit 236, a memory control unit 238, a behavior control unit 250, and a communication processing unit 280.
[0070] The sensor unit 200 includes a microphone 201, a 3D depth sensor 202, a 2D camera 203, a distance sensor 204, a touch sensor 205, and an acceleration sensor 206. The microphone 201 continuously detects sound and outputs sound data. The microphone 201 may be provided on the head of the robot 100 and may have a function of performing binaural recording. The 3D depth sensor 202 detects the contour of an object by continuously irradiating an infrared pattern and analyzing the infrared pattern from infrared images continuously captured by the infrared camera. The 2D camera 203 is an example of an image sensor. The 2D camera 203 captures an image using visible light and generates visible light video information. The distance sensor detects the distance to an object by irradiating, for example, a laser or ultrasonic waves. The sensor unit 200 may also include a clock, a gyro sensor, a touch sensor, a sensor for motor feedback, and the like.
[0071] 4, the components of the robot 100 excluding the control target 252 and the sensor unit 200 are examples of components of a behavior control system of the robot 100. The behavior control system of the robot 100 controls the control target 252.
[0072] The storage unit 220 includes a behavior decision model 223 and history data 222. The history data 222 includes the behavior of the user 10, the past emotional value of the user 10, the user's reaction information, the past emotional value of the robot 100, and the history of the behavior of the robot 100. Specifically, the history data 222 includes a plurality of event data including the emotional value of the user 10, the emotional value of the robot 100, and the behavior of the user 10. The data including the behavior of the user 10 includes a camera image representing the behavior of the user 10. The emotional value and the history of the behavior are recorded for each user 10, for example, by being associated with the identification information of the user 10. At least a part of the storage unit 220 is implemented by a storage medium such as a memory. It may include a person DB that stores the face image of the user 10, the attribute information of the user 10, and the like. Note that the functions of the components of the robot 100 shown in FIG. 2, except for the control target 252, the sensor unit 200, and the storage unit 220, can be realized by a CPU operating based on a program. For example, the functions of these components can be implemented as CPU operations using an operating system (OS) and a program that runs on the OS.
[0073] The sensor module unit 210 includes a voice emotion recognition unit 211, a speech understanding unit 212, a facial expression recognition unit 213, and a face recognition unit 214. Information detected by the sensor unit 200 is input to the sensor module unit 210. The sensor module unit 210 analyzes the information detected by the sensor unit 200, and outputs the analysis result to the state recognition unit 230.
[0074] The voice emotion recognition unit 211 of the sensor module unit 210 analyzes the voice of the user 10 detected by the microphone 201 to recognize the state of the user 10. For example, the voice emotion recognition unit 211 extracts feature amounts such as frequency components of the voice, and recognizes the state of the user 10 based on the extracted feature amounts. The speech understanding unit 212 analyzes the voice of the user 10 detected by the microphone 201, and outputs text information representing the contents of the utterance of the user 10.
[0075] The facial expression recognition unit 213 recognizes the facial expression and emotion of the user 10 from the image of the user 10 captured by the 2D camera 203. For example, the facial expression recognition unit 213 recognizes the facial expression and emotion of the user 10 based on the shapes, positional relationship, etc. of the eyes and mouth.
[0076] The face recognition unit 214 recognizes the face of the user 10. The face recognition unit 214 recognizes the user 10 by matching a face image stored in a person DB (not shown) with a face image of the user 10 captured by the 2D camera 203.
[0077] The state recognition unit 230 recognizes the state of the user 10 based on the information analyzed by the sensor module unit 210. For example, the state recognition unit 230 mainly performs processing related to perception using the analysis results of the sensor module unit 210. For example, it generates sensory information such as "Daddy is alone" or "There is a 90% chance that Daddy is not smiling." It performs processing to understand the meaning of the generated sensory information. For example, it generates semantic information such as "Daddy is alone and looks lonely."
[0078] The state recognition unit 230 also recognizes whether there are multiple users 10, based on the information analyzed by the sensor module unit 210. As shown in Fig. 1, when multiple users 10a, 10b, 10c, and 10d are present around the robot 100, the state recognition unit 230 can recognize the state of each user 10 and generate perception information for each user 10.
[0079] The emotion determination unit 231 determines an emotion value indicating the emotion of the user 10 based on the information analyzed by the sensor module unit 210 and the state of the user 10 recognized by the state recognition unit 230. For example, the information analyzed by the sensor module unit 210 and the recognized state of the user 10 are input to a pre-trained neural network to obtain an emotion value indicating the emotion of the user 10.
[0080] Here, the emotion value indicating the emotion of the user 10 is a value indicating the positive or negative emotion of the user. For example, if the emotion of the user is a cheerful emotion accompanied by a sense of pleasure or comfort, such as "joy", "pleasure", "comfort", "relief", "excitement", "relief", and "fulfillment", the value indicates a positive value, and the more cheerful the emotion, the larger the value. If the emotion of the user is an unpleasant emotion, such as "anger", "sorrow", "discomfort", "anxiety", "sorrow", "worry", and "emptiness", the value indicates a negative value, and the more unpleasant the emotion, the larger the absolute value of the negative value. If the emotion of the user is not any of the above ("neutral"), the value indicates 0.
[0081] In addition, the emotion determination unit 231 determines an emotion value indicating the emotion of the robot 100 based on the information analyzed by the sensor module unit 210, the information detected by the sensor unit 200, and the state of the user 10 recognized by the state recognition unit 230.
[0082] The emotion value of the robot 100 includes emotion values for each of a plurality of emotion categories, and is, for example, a value (0 to 5) indicating the strength of each of "happiness," "anger," "sorrow," and "pleasure."
[0083] Specifically, the emotion determination unit 231 determines an emotion value indicating the emotion of the robot 100 according to a rule for updating the emotion value of the robot 100, which rule is determined in association with the information analyzed by the sensor module unit 210 and the state of the user 10 recognized by the state recognition unit 230.
[0084] For example, when the state recognition unit 230 recognizes that the user 10 looks lonely, the emotion determination unit 231 increases the emotion value of "sadness" of the robot 100. Also, when the state recognition unit 230 recognizes that the user 10 is smiling, the emotion determination unit 231 increases the emotion value of "joy" of the robot 100.
[0085] The emotion determining unit 231 may determine the emotion value indicating the emotion of the robot 100 by further considering the state of the robot 100. For example, when the battery level of the robot 100 is low or when the surrounding environment of the robot 100 is pitch black, the emotion value of "sadness" of the robot 100 may be increased. Furthermore, when the user 10 continues to talk to the robot 100 despite the battery level being low, the emotion value of "anger" may be increased.
[0086] The behavior recognition unit 232 recognizes the behavior of the user 10 based on the information analyzed by the sensor module unit 210 and the state of the user 10 recognized by the state recognition unit 230. For example, the information analyzed by the sensor module unit 210 and the recognized state of the user 10 are input to a pre-trained neural network, the probability of each of a plurality of predetermined behavior classifications (for example, "laughing", "getting angry", "asking a question", "getting sad") is obtained, and the behavior classification with the highest probability is recognized as the behavior of the user 10.
[0087] When the behavior of the user 10a is recognized as a question, the acquisition unit 235 acquires a behavior of a correct answer that is a correct answer to the question. Specifically, the question is input to the sentence generation model, and the answer content is acquired as the output. Furthermore, the acquisition unit 235 acquires a behavior indicating a correct or incorrect answer to an answer provided by any of the users 10 from the sentence generation model before the correct answer is notified to the user 10a. Specifically, an answer provided by any of the users 10 is input to the sentence generation model, and a correct or incorrect answer to the answer is acquired.
[0088] The behavior determining unit 236 determines a behavior corresponding to the behavior of the user 10 recognized by the behavior recognizing unit 232 based on the current emotion value of the user 10 determined by the emotion determining unit 231, the history data 222 of past emotion values determined by the emotion determining unit 231 before the current emotion value of the user 10 was determined, and the emotion value of the robot 100. In this embodiment, the behavior determining unit 236 uses one most recent emotion value included in the history data 222 as the past emotion value of the user 10, but the disclosed technology is not limited to this aspect. For example, the behavior determining unit 236 may use a plurality of most recent emotion values as the past emotion value of the user 10, or may use an emotion value from a unit period ago, such as one day ago. In addition, the behavior determining unit 236 may determine a behavior corresponding to the behavior of the user 10 by further considering not only the current emotion value of the robot 100 but also the history of the past emotion values of the robot 100. In addition, the behavior determining unit 236 may determine the behavior of the robot 100 by further considering a history of positive reaction information of the user 10 to the behavior of the robot 100. The behavior determined by the behavior determining unit 236 includes a gesture performed by the robot 100 or a speech content of the robot 100.
[0089] When the robot 100 recognizes that the behavior of the user 10a is "asking a question," if any of the users 10 answers before the robot 100, the behavior determining unit 236 determines a behavior indicating whether the answer is correct or incorrect. In addition, the behavior determining unit determines the behavior of the robot 100 so that the robot 100 responds not only by simply indicating the result, but also by making a large gesture to express the result, or by responding in accordance with the attitude or mood of the user 10. At that time, the behavior of the robot 100 may be determined using past reaction information of the user 10.
[0090] Specifically, the behavior determining unit 236 uses the answer content acquired by the acquiring unit 235 to determine a response behavior such as a reaction.
[0091] FIG. 5 is a diagram showing variations of examples of utterances for response actions of the robot 100. 5, the utterance contents of the response behavior of the robot 100 differ depending on whether the answer result of the user 10 matches the answer prepared by the robot 100, does not match, or does not match either. In addition, the utterance contents of the response behavior of the robot 100 differ not only depending on the answer result of the user 10 but also on the user's behavior (whether it is a playful behavior or a serious behavior).
[0092] Here, the answer of the user 10 refers to a case where a speaker (e.g., user 10a) asks a question to the robot 100 and another person (e.g., user 10b) answers before the robot 100, or a case where a speaker (e.g., user 10a) asks a question to the robot 100 and the person who asked the question (user 10a) answers before the robot 100.
[0093] The example of the speech content shown in FIG. 5 will be explained below. When the answer of the user 10 matches the answer prepared by the robot 100 ("◯" in the figure) and the behavior of the user 10 is playful ("A" in the figure), the robot 100 makes utterances that mock the user 10, such as "Maybe it was just a coincidence!", "Lucky!", or "You're right on the money."
[0094] Furthermore, when the answer of the user 10 matches the answer prepared by the robot 100 and the user 10 behaves seriously ("B" in the figure), the robot 100 makes speech praising the user 10, such as "As expected, that's the right answer!", "You know a lot. To be more precise...", or "You're a genius! If you can say that, that's perfect."
[0095] Furthermore, if the answer of the user 10 does not match the answer prepared by the robot 100 ("X" in the figure) and the behavior of the user 10 is playful, the robot 100 will make utterances that are negative towards the user 10, such as "Why?", "No!", or "Well, goodbye!".
[0096] Furthermore, if the answer of the user 10 does not match the answer prepared by the robot 100 and the user 10 is acting seriously, the robot 100 will give advice to the user 10 such as "That's too bad, try thinking about it from a different angle," "Let's get rid of your preconceptions!" or "That's true, but a different answer would be better this time."
[0097] In addition, when the answer of the user 10 is not applicable ("?" in the figure) because the robot 100 cannot determine whether the answer matches or does not match the answer prepared by the robot 100, and the user 10 behaves in a playful manner, the robot 100 will make utterances expressing confusion to the user 10, such as "That's a bit strange," "Come back later," or "I don't understand at all."
[0098] In addition, if the answer of the user 10 is not applicable because the robot 100 cannot determine whether it matches or does not match the answer prepared by the robot 100, and the behavior of the user 10 is serious, the robot 100 will make an utterance asking the user 10 to reconsider, such as "Let's think about it again!", "Hmm, that's difficult. To put it simply?", or "I really can't decide."
[0099] The robot 100 can change its attitude and tone of speech depending on whether the respondent (user 10a or user 10b in this example) is a familiar user with whom the robot 100 has had multiple conversations or a first-time user.
[0100] Moreover, the contents of the utterances of the response behavior of the robot 100 shown in FIG. 5 are merely examples, and the present invention is not limited to these contents.
[0101] Here, the robot 100 can objectively recognize whether the respondent's behavior is playful or serious. Specifically, the respondent's state can be identified based on the voice features detected by the microphone 201 of the sensor unit 200 and recognized by the voice emotion recognition unit 211, and the facial expression pattern captured by the 2D camera 203 of the sensor unit 200 and recognized by the facial expression recognition unit 213.
[0102] FIG. 6 is a diagram showing variations of examples of response behavior of the robot 100. 6, the utterance contents of the response behavior of the robot 100 differ depending on whether the answer result of the user 10 matches the answer prepared by the robot 100, does not match, or does not match either. In addition, the movement of the response behavior of the robot 100 differs not only depending on the answer result of the user 10 but also depending on the user's behavior (whether it is a playful behavior or a serious behavior).
[0103] Here, the answer of the user 10 refers to a case where a speaker (e.g., user 10a) asks a question to the robot 100 and another person (e.g., user 10b) answers before the robot 100, or a case where a speaker (e.g., user 10a) asks a question to the robot 100 and the person who asked the question (user 10a) answers before the robot 100.
[0104] An example of the movement shown in FIG. 6 will now be described. When the answer of the user 10 matches the answer prepared by the robot 100 ("◯" in the figure) and the behavior of the user 10 is playful ("A" in the figure), the robot 100 performs actions to entertain the user 10, such as "doing a cheerful dance," "humming," or "playing background music."
[0105] Furthermore, when the answer of the user 10 matches the answer prepared by the robot 100 and the behavior of the user 10 is serious ("B" in the figure), the robot 100 performs actions that praise the user 10, such as "making a circle pose," "patting the head," or "clap[ing]."
[0106] Furthermore, if the answer of the user 10 does not match the answer prepared by the robot 100 ("X" in the figure) and the behavior of the user 10 is playful, the robot 100 will make a motion that denies the user 10, such as "making a gesture of retort," "shaking the head from side to side in denial," or "waving the hand from side to side in goodbye."
[0107] In addition, when the answer of the user 10 does not match the answer prepared by the robot 100, and the behavior of the user 10 is serious, the robot 100 behaves in a manner that is sympathetic to the user 10, such as making an X pose, making a disappointing hand gesture, or giving a serious hint while making a gesture.
[0108] In addition, when the answer of the user 10 is not applicable ("?" in the figure) because it is impossible to determine whether the answer corresponds or does not correspond to the answer prepared by the robot 100, and the behavior of the user 10 is playful, the robot 100 will make a movement that expresses confusion toward the user 10, such as "making a strange pose," "making a troubled pose," or "making another hand gesture."
[0109] In addition, if the answer of the user 10 is not applicable because the robot 100 cannot determine whether it matches or does not match the answer prepared by the robot 100, and the user 10 is acting seriously, the robot 100 will make a movement to ask the user 10 to reconsider, such as "making a hand gesture to try again," "making a suggestion hand gesture," or "making an unsure pose."
[0110] The robot 100 can behave differently in terms of attitude and style depending on whether the respondent (user 10a or user 10b in this example) is a familiar user with whom the robot 100 has had multiple conversations or a first-time user.
[0111] Moreover, the content of the response behavior of the robot 100 shown in FIG. 6 is merely an example, and the robot 100 is not limited to this content.
[0112] Furthermore, the robot 100 can not only indicate whether the answer of the respondent is correct or incorrect as described above, but can also pretend to have forgotten the answer and act silly. In this case, when the robot 100 forgets the answer or pretends not to understand and the user 10 points out the problem, the robot 100 can respond by acting embarrassed.
[0113] In this way, the robot 100 can respond in a human-like manner when engaging in a dialogue with the user 10.
[0114] The memory control unit 238 determines whether or not to store data including the behavior of the user 10 in the history data 222 based on the predetermined behavior strength for the behavior determined by the behavior determination unit 236 and the emotional value of the robot 100.
[0115] Specifically, when a total intensity value, which is the sum of the sum of the emotion values for each of the multiple emotion classifications of the robot 100, the predetermined intensity for the gesture including the behavior determined by the behavior determination unit 236, and the predetermined intensity for the speech content included in the behavior determined by the behavior determination unit 236, is equal to or greater than a threshold value, it is determined that data including the behavior of the user 10 is to be stored in the history data 222.
[0116] When the memory control unit 238 decides to store data including the behavior of the user 10 in the history data 222, it stores in the history data 222 the behavior determined by the behavior determination unit 236, information analyzed by the sensor module unit 210 from the present time up to a certain period of time ago (e.g., all peripheral information such as data on the sound, images, smells, etc. of the scene), and the state of the user 10 recognized by the state recognition unit 230 (e.g., the facial expression, emotions, etc. of the user 10).
[0117] Furthermore, since the emotion value of the robot 100 is updated depending on whether the answer of the user 10 is correct or incorrect and the attitude of the user 10, the memory control unit 238 is also executed after the robot 100 executes the event processing.
[0118] For example, if the answer of the user 10 is correct, the emotion determination unit 231 determines to increase the emotion value of "happiness" of the robot 100, and based on the updated value, the storage control unit 238 determines whether or not to store data including the behavior of the user 10 in the history data 222. Also, for example, if the answer of the user 10 is incorrect, the emotion determination unit 231 determines to decrease the emotion value of "happiness" of the robot 100 and increase the emotion value of "sadness", and based on the updated value, the storage control unit 238 determines whether or not to store data including the behavior of the user 10 in the history data 222.
[0119] The behavior control unit 250 controls the control target 252 based on the behavior determined by the behavior determination unit 236. For example, when the behavior determination unit 236 determines an behavior including speaking, the behavior control unit 250 outputs a sound from a speaker included in the control target 252. At this time, the behavior control unit 250 may determine the speaking speed of the sound based on the emotion value of the robot 100. For example, the behavior control unit 250 determines a faster speaking speed as the emotion value of the robot 100 increases. In this way, the behavior control unit 250 determines the execution form of the behavior determined by the behavior determination unit 236 based on the emotion value determined by the emotion determination unit 231.
[0120] Furthermore, the behavior control unit 250 may control the magnitude of the gesture movement based on the behavior determined by the behavior determination unit 236 and in accordance with the magnitude of the emotion value of the robot 100. For example, when the emotion value of "happiness" of the robot 100 becomes larger than a preset reference value, the robot 100 is controlled to move its arms more vigorously as an expression of dancing.
[0121] The behavior control unit 250 may recognize a change in the user's 10 emotion in response to the execution of the behavior determined by the behavior determination unit 236. For example, the change in emotion may be recognized based on the voice or facial expression of the user 10. Alternatively, the change in emotion may be recognized based on the detection of an impact by the touch sensor 205 included in the sensor unit 200. When an impact is detected by the touch sensor 205 included in the sensor unit 200, the emotion of the user 10 may be recognized as having worsened, or when the reaction of the user 10 is determined to be laughing or happy based on the detection result of the touch sensor 205 included in the sensor unit 200, the emotion of the user 10 may be recognized as having improved. Information indicating the reaction of the user 10 is output to the communication processing unit 280.
[0122] Furthermore, after the behavior control unit 250 executes the behavior determined by the behavior determining unit 236 in the execution mode determined according to the emotion of the robot 100, the emotion determining unit 231 further changes the emotion value of the robot 100 based on the user's reaction to the execution of the behavior. Specifically, the emotion determining unit 231 increases the emotion value of "happiness" of the robot 100 when the user's reaction to the behavior determined by the behavior determining unit 236 being performed on the user in the execution mode determined by the behavior control unit 250 is not bad. Furthermore, the emotion determining unit 231 increases the emotion value of "sadness" of the robot 100 when the user's reaction to the behavior determined by the behavior determining unit 236 being performed on the user in the execution mode determined by the behavior control unit 250 is bad.
[0123] Furthermore, the behavior control unit 250 expresses the emotion of the robot 100 based on the determined emotion value of the robot 100. For example, when the behavior control unit 250 increases the emotion value of "happiness" of the robot 100, the behavior control unit 250 controls the control object 252 to make the robot 100 perform a happy gesture. When the behavior control unit 250 increases the emotion value of "sadness" of the robot 100, the behavior control unit 250 controls the control object 252 to make the robot 100 assume a droopy posture.
[0124] The communication processing unit 280 is responsible for communication with the server 300. As described above, the communication processing unit 280 transmits user reaction information to the server 300. In addition, the communication processing unit 280 receives updated reaction rules from the server 300. When the communication processing unit 280 receives the updated reaction rules from the server 300, it updates the reaction rules as the behavioral determination model 223.
[0125] The server 300 communicates between the robots 100, 101, and 102 and the server 300, receives user reaction information transmitted from the robot 100, and updates the reaction rules based on reaction rules that include actions that have received positive reactions.
[0126] The operation of the robot 100 according to this embodiment will be described below. In the robot 100, a behavior decision process shown in Fig. 7 is executed. The process in the robot 100 is executed by the CPU 1212 functioning as a state recognition unit 230, an emotion decision unit 231, a behavior recognition unit 232, an acquisition unit 235, a behavior decision unit 236, a memory control unit 238, a behavior control unit 250, and a communication processing unit 280.
[0127] FIG. 7 is a flowchart showing the behavior decision process of the robot 100. The CPU 1212 repeatedly executes the flowchart shown in FIG.
[0128] First, in step S200, the CPU 1212 recognizes the state of the user 10 and the state of the robot 100 based on the information analyzed by the sensor module unit 210.
[0129] In step S202, the CPU 1212 recognizes the behavior classification of the user 10 based on the information analyzed by the sensor module unit 210 and the state of the user 10 recognized by the state recognition unit 230.
[0130] In step S204, the CPU 1212 judges whether the behavior of the user 10 is "asking a question". If it is recognized that the behavior of the user 10 is "asking a question" (step S204: YES), the process proceeds to step S206. On the other hand, if it is recognized that the behavior of the user 10 is not "asking a question" (step S204: NO), the process proceeds to step S212.
[0131] In step S206, the CPU 1212 acquires the action of the user 10 in response to the question from the sentence generation model.
[0132] In step S208, the CPU 1212 determines the behavior of the robot 100 based on the result (the action content of the answer) acquired from the sentence generation model and the action determination model. Note that when step S208 is passed twice or more, the already determined action of the robot 100 is updated based on the latest answer acquired in step S206.
[0133] In step S210, the CPU 1212 determines whether or not the user 10 has responded before the robot 100. If the user 10 has responded first (step S210: YES), step S206 is executed again. On the other hand, if the user 10 has not responded first (step S210: NO), the process proceeds to step S214.
[0134] In step S212, the CPU 1212 determines the behavior of the robot 100 based on the behavior determination model.
[0135] In step S214, the CPU 1212 controls the determined behavior to execute the behavior of the robot 101. Then, the CPU 1212 ends the processing of this flowchart.
[0136] The robot 100 is an example of an electronic device equipped with a behavior control system. The application of the behavior control system is not limited to the robot 100, and the behavior control system can be applied to various electronic devices. The functions of the server 300 may be implemented by one or more computers. At least a part of the functions of the server 300 may be implemented by a virtual machine. At least a part of the functions of the server 300 may be implemented in the cloud.
[0137] 8 shows a schematic example of a hardware configuration of a computer 1200 functioning as the robot 100 and the server 300. A program installed on the computer 1200 can cause the computer 1200 to function as one or more "parts" of an apparatus according to the present embodiment, or cause the computer 1200 to execute operations or one or more "parts" associated with the apparatus according to the present embodiment, and / or cause the computer 1200 to execute a process or steps of the process according to the present embodiment. Such a program can be executed by the CPU 1212 to cause the computer 1200 to execute specific operations associated with some or all of the blocks of the flowcharts and block diagrams described herein.
[0138] The computer 1200 according to this embodiment includes a CPU 1212, a RAM 1214, and a graphics controller 1216, which are interconnected by a host controller 1210. The computer 1200 also includes input / output units such as a communication interface 1222, a storage device 1224, a DVD drive 1226, and an IC card drive, which are connected to the host controller 1210 via the input / output controller 1220. The DVD drive 1226 may be a DVD-ROM drive, a DVD-RAM drive, or the like. The storage device 1224 may be a hard disk drive, a solid-state drive, or the like. The computer 1200 also includes a ROM 1230 and a legacy input / output unit such as a keyboard, which are connected to the input / output controller 1220 via an input / output chip 1240.
[0139] The CPU 1212 operates according to a program stored in the ROM 1230 and the RAM 1214, thereby controlling each unit. The graphic controller 1216 acquires image data generated by the CPU 1212 into a frame buffer or the like provided in the RAM 1214 or into itself, and causes the image data to be displayed on the display device 1218.
[0140] The communication interface 1222 communicates with other electronic devices via a network. The storage device 1224 stores programs and data used by the CPU 1212 in the computer 1200. The DVD drive 1226 reads programs or data from a DVD-ROM 1227 or the like and provides the programs or data to the storage device 1224. The IC card drive reads programs and data from an IC card and / or writes programs and data to an IC card.
[0141] The ROM 1230 stores therein a boot program or the like that is executed by the computer 1200 upon activation, and / or a program that depends on the hardware of the computer 1200. The input / output chip 1240 may also connect various input / output units to the input / output controller 1220 via a USB port, a parallel port, a serial port, a keyboard port, a mouse port, and the like.
[0142] The programs are provided by a computer-readable storage medium such as a DVD-ROM 1227 or an IC card. The programs are read from the computer-readable storage medium, installed in the storage device 1224, the RAM 1214, or the ROM 1230, which are also examples of computer-readable storage media, and executed by the CPU 1212. Information processing described in these programs is read by the computer 1200, and brings about cooperation between the programs and the various types of hardware resources described above. An apparatus or method may be constructed by implementing operations or processing of information according to the use of the computer 1200.
[0143] For example, when communication is performed between the computer 1200 and an external device, the CPU 1212 may execute a communication program loaded in the RAM 1214 and instruct the communication interface 1222 to perform communication processing based on the processing described in the communication program. Under the control of the CPU 1212, the communication interface 1222 reads transmission data stored in a transmission buffer area provided in the RAM 1214, the storage device 1224, the DVD-ROM 1227, or a recording medium such as an IC card, and transmits the read transmission data to a network, or writes reception data received from the network to a reception buffer area or the like provided on the recording medium.
[0144] Furthermore, the CPU 1212 may cause all or a necessary portion of a file or database stored in an external recording medium such as the storage device 1224, the DVD drive 1226 (DVD-ROM 1227), an IC card, etc. to be read into the RAM 1214, and perform various types of processing on the data on the RAM 1214. The CPU 1212 may then write back the processed data to the external recording medium.
[0145] Various types of information, such as various types of programs, data, tables, and databases, may be stored in the recording medium and processed. The CPU 1212 reads data from the RAM 1214 and executes the instruction sequence of the program as described elsewhere in this disclosure. The CPU 1212 may execute various types of processing including various types of operations, information processing, conditional judgment, conditional branching, unconditional branching, information search / replacement, etc., specified by the conditions, and write back the results to the RAM 1214. The CPU 1212 may also search for information in a file, database, etc. in the recording medium. For example, when a plurality of entries each having an attribute value of a first attribute associated with an attribute value of a second attribute are stored in the recording medium, the CPU 1212 may search for an entry whose attribute value of the first attribute matches a specified condition from among the plurality of entries, read the attribute value of the second attribute stored in the entry, and thereby obtain the attribute value of the second attribute associated with the first attribute that satisfies a predetermined condition.
[0146] The above-described programs or software modules may be stored in a computer-readable storage medium on the computer 1200 or in the vicinity of the computer 1200. In addition, a recording medium such as a hard disk or a RAM provided in a server system connected to a dedicated communication network or the Internet can be used as a computer-readable storage medium, thereby providing the programs to the computer 1200 via the network.
[0147] The blocks in the flowcharts and block diagrams in the present embodiment may represent stages of a process in which an operation is performed or "parts" of an apparatus responsible for performing the operation. Particular stages and "parts" may be implemented by dedicated circuitry, programmable circuitry provided with computer-readable instructions stored on a computer-readable storage medium, and / or a processor provided with computer-readable instructions stored on a computer-readable storage medium. The dedicated circuitry may include digital and / or analog hardware circuits, and may include integrated circuits (ICs) and / or discrete circuits. The programmable circuitry may include reconfigurable hardware circuits, such as, for example, field programmable gate arrays (FPGAs), programmable logic arrays (PLAs), and the like, including AND, OR, XOR, NAND, NOR, and other logical operations, flip-flops, registers, and memory elements.
[0148] A computer-readable storage medium may include any tangible device capable of storing instructions that are executed by a suitable device, such that a computer-readable storage medium having instructions stored thereon comprises an article of manufacture that includes instructions that can be executed to create means for performing the operations specified in the flowcharts or block diagrams. Examples of computer-readable storage media may include electronic storage media, magnetic storage media, optical storage media, electromagnetic storage media, semiconductor storage media, and the like. More specific examples of computer-readable storage media may include floppy disks, diskettes, hard disks, random access memories (RAMs), read-only memories (ROMs), erasable programmable read-only memories (EPROMs or flash memories), electrically erasable programmable read-only memories (EEPROMs), static random access memories (SRAMs), compact disk read-only memories (CD-ROMs), digital versatile disks (DVDs), Blu-ray disks, memory sticks, integrated circuit cards, and the like.
[0149] The computer readable instructions may include either assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state-setting data, or source or object code written in any combination of one or more programming languages, including object-oriented programming languages such as Smalltalk, JAVA, C++, etc., and conventional procedural programming languages such as the "C" programming language or similar programming languages.
[0150] The computer readable instructions may be implemented by a general purpose computer, special purpose computer, or other The processor of a programmable data processing apparatus, or programmable circuitry, may be provided in a general purpose computer, special purpose computer, or other programmable data processing apparatus, processor, or programmable circuitry, locally or over a wide area network (WAN) such as a local area network (LAN), the Internet, etc., to execute the computer readable instructions to generate means for performing the operations specified in the flowcharts or block diagrams. Examples of processors include computer processors, processing units, microprocessors, digital signal processors, controllers, microcontrollers, etc.
[0151] Although the present invention has been described above using the embodiments, the technical scope of the present invention is not limited to the scope described in the above embodiments. It is clear to those skilled in the art that various modifications and improvements can be made to the above embodiments. It is clear from the description of the claims that such modifications and improvements can also be included in the technical scope of the present invention.
[0152] It should be noted that the order of execution of each process, such as operations, procedures, steps, and stages, in the devices, systems, programs, and methods shown in the claims, specifications, and drawings is not specifically stated as "before" or "prior to," and that the process may be performed in any order unless the output of a previous process is used in a later process. Even if the operational flow in the claims, specifications, and drawings is described using "first," "next," etc. for convenience, it does not mean that the process must be performed in this order. [Explanation of symbols]
[0153] 5 System, 10, 11, 12 User, 20 Communication network, 100, 101, 102 Robot, 200 Sensor unit, 201 Microphone, 202 Depth sensor, 203 Camera, 204 Distance sensor, 205 Touch sensor, 206 Acceleration sensor, 210 Sensor module unit, 211 Voice emotion recognition unit, 212 Speech understanding unit, 213 Facial expression recognition unit, 214 Face recognition unit, 220 storage unit, 221 reaction rule, 223 behavior decision model, 222 history data, 228 control unit, 230 user state recognition unit (state recognition unit), 231 emotion determination unit, 232 behavior recognition unit, 234 difficulty estimation unit, 235 acquisition unit, 236 behavior decision unit, 238 memory control unit, 250 behavior control unit, 252 control target, 280 communication processing unit, 300 server, 1200 computer, 1210 host controller, 1212 CPU, 1214 RAM, 1216 graphic controller, 1218 display device, 1220 input / output controller, 1222 communication interface, 1224 storage device, 1226 DVD drive, 1227 DVD-ROM, 1230 ROM, 1240 input / output chip
Claims
1. A behavior recognition unit that recognizes one or more user behaviors; an acquisition unit that acquires a correct answer action that is a correct answer to the question when a behavior of a user is recognized as a question; an action determination unit that, when the acquisition unit acquires an alternative answer that is an answer provided by any of the users before the correct answer is notified to the one user, further determines an action to be taken with respect to the alternative answer; A behavior control system including:
2. The behavior control system of claim 1, wherein the behavior decision unit, when determining an action by comparing the different answer with the correct answer, changes the manner of the behavior depending on whether the different answer matches the correct answer, whether they do not match, or whether there is no match.
3. The behavior control system according to claim 2 , wherein the behavior determination unit changes the manner of behavior depending on whether the behavior recognition unit recognizes the behavior of the one or more users as playful behavior or as serious behavior.
4. A program for causing a computer to function as the behavior control system according to any one of claims 1 to 3.