electronic equipment

Abstract

To provide an electronic device that executes appropriate action for an action of a user.SOLUTION: A robot 100 serving as an electronic device is configured to: recognize an action of a user who is giving a presentation; determine an action of the robot corresponding to the recognized action of the user; control a control target on the basis of the determined action of the robot; performs, when the user practices the presentation, an action to improve the contents of the presentation of the user to attain a predetermined perfection level; determine, when recognizing practice as a presentation of the user, an action to heighten the emotion of the user giving the presentation and identify the contents of the presentation; and determine an action related to a predetermined feedback in accordance with the identified contents of the presentation.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The disclosed embodiments relate to electronic devices. [Background technology]

[0002] A technique for determining an appropriate robot behavior for a user's state has been disclosed (see, for example, Patent Document 1). Patent Document 1 discloses that a robot recognizes a user's reaction when the robot performs a specific action, and if the robot is unable to determine an action for the recognized user's reaction, the robot updates its behavior by receiving information about an action appropriate for the recognized user's state from a server. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 6053847 Summary of the Invention [Problem to be solved by the invention]

[0004] However, conventional techniques have room for improvement in terms of executing appropriate actions in response to user actions.

[0005] The present invention has been made in view of the above, and has an object to provide an electronic device that can execute appropriate actions. [Means for solving the problem]

[0006] An electronic device according to one aspect of the embodiment recognizes the behavior of a user giving a presentation, determines its own behavior corresponding to the recognized user behavior, and controls a control target based on the determined own behavior. [Effects of the Invention]

[0007] According to one aspect of the embodiment, appropriate actions can be taken. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram schematically illustrating an example of a control system according to this embodiment. [Figure 2] FIG. 2 is a diagram illustrating a schematic functional configuration of the robot. [Figure 3] FIG. 3 is a diagram illustrating an example of an operational flow relating to an operation for determining an action in a robot. [Figure 4] FIG. 4 is a diagram illustrating an example of a hardware configuration of a robot and a computer that functions as a server. [Figure 5] FIG. 5 is a diagram showing an emotion map onto which a plurality of emotions are mapped. [Figure 6] FIG. 6 is a diagram showing another example of an emotion map. [Figure 7] FIG. 7 is a diagram illustrating an example of an emotion table. [Figure 8] FIG. 8 is a diagram illustrating an example of an emotion table. DETAILED DESCRIPTION OF THE INVENTION

[0009] The present invention will be described below through embodiments, but the following embodiments do not limit the invention according to the claims. Furthermore, not all of the combinations of features described in the embodiments are necessarily essential to the solution of the invention. Furthermore, in the present embodiments, a "robot" will be used as an example of an electronic device. The electronic device may be a stuffed toy, a portable terminal device such as a smartphone, an input device such as a smart speaker, or the like, in addition to a robot.

[0010] Fig. 1 is a diagram schematically illustrating an example of a control system 1 according to this embodiment. As shown in Fig. 1, the control system 1 includes a plurality of robots 100, a linked device 400, and a server 300. Each of the plurality of robots 100 is managed by a user.

[0011] The robot 100 converses with the user and provides the user with video. At this time, the robot 100 converses with the user and provides the user with video, etc., in cooperation with a server 300 or the like with which it can communicate via a communication network 20. For example, the robot 100 not only learns appropriate conversation by itself, but also learns to have a more appropriate conversation with the user in cooperation with the server 300. The robot 100 also records captured video data of the user, etc., in the server 300, and requests the video data, etc. from the server 300 as needed, and provides the video data, etc. to the user.

[0012] The robot 100 also has an emotional value that indicates the type of its own emotion. For example, the robot 100 has emotional values ​​that indicate the intensity of each of the following emotions: "joy," "anger," "sorrow," "pleasure," "discomfort," "relief," "anxiety," "sadness," "excitement," "worry," "relief," "fulfillment," "emptiness," and "neutral." For example, when the robot 100 is in a state where the emotional value of excitement is high and it is conversing with a user, it speaks at a fast speed. In this way, the robot 100 can express its own emotions through its actions.

[0013] Furthermore, the robot 100 may be configured to determine the behavior of the robot 100 corresponding to the emotion of the user 10 by matching a sentence generation model (so-called AI (Artificial Intelligence) chat engine) with an emotion engine. Specifically, the robot 100 may be configured to recognize the behavior of the user 10, determine the emotion of the user 10 regarding the user's behavior, and determine the behavior of the robot 100 corresponding to the determined emotion.

[0014] More specifically, when the robot 100 recognizes the behavior of the user 10, it automatically generates the behavioral content that the robot 100 should take in response to the behavior of the user 10, using a preset sentence generation model. The sentence generation model may be interpreted as an algorithm and calculation for automatic dialogue processing using text. The sentence generation model is described, for example, in JP 2018-081444 A and chatGPT (Internet Search<URL:https: / / openai.com / blog / chatgpt> ), and therefore detailed description thereof will be omitted. Such a sentence generation model is configured using a large language model (LLM). As described above, in this embodiment, by combining a large language model with an emotion engine, it is possible to reflect the emotions of the user 10 and the robot 100, as well as various linguistic information, in the behavior of the robot 100. In other words, according to this embodiment, a synergistic effect can be obtained by combining a sentence generation model with an emotion engine.

[0015] The robot 100 also has a function of recognizing the user's actions. The robot 100 recognizes the user's actions by analyzing the user's facial image acquired by the camera function and the user's voice acquired by the microphone function. The robot 100 determines the action to be performed by the robot 100 based on the recognized user's actions, etc.

[0016] The robot 100 stores rules that define actions that the robot 100 will take based on the user's emotions, the robot's own emotions, and the user's actions, and performs various actions in accordance with the rules.

[0017] Specifically, the robot 100 has reaction rules for determining the behavior of the robot 100 based on the user's emotions, the robot's emotions, and the user's behavior. For example, the reaction rules define the behavior of the robot 100 as "laughing" when the user's behavior is "laughing." Furthermore, the reaction rules define the behavior of the robot 100 as "apologizing" when the user's behavior is "angry." Furthermore, the reaction rules define the behavior of the robot 100 as "answering" when the user's behavior is "asking a question." Furthermore, the reaction rules define the behavior of the robot 100 as "calling out" when the user's behavior is "sad."

[0018] When the robot 100 recognizes that the user's behavior is "anger" based on the reaction rules, it selects the behavior of "apologizing" defined in the reaction rules as the behavior to be performed by the robot 100. For example, when the robot 100 selects the behavior of "apologizing," it performs the motion of "apologizing" and outputs a voice representing the word "apologize."

[0019] In addition, it is defined that when the emotion of the robot 100 is "normal" (i.e., "joy" = 0, "anger" = 0, "sadness" = 0, "happiness" = 0) and the condition that the user's state is "alone and lonely" is met, the emotion of the robot 100 changes to "worried" and the action of "calling out" can be performed.

[0020] When the robot 100 recognizes based on the reaction rule that the current emotion of the robot 100 is "normal" and that the user is alone and seems lonely, the robot 100 increases the emotion value of "sadness" of the robot 100. Furthermore, the robot 100 selects the action of "calling out" defined in the reaction rule as the action to be performed toward the user. For example, when the robot 100 selects the action of "calling out," the robot 100 converts the phrase "What's wrong?", which indicates concern, into a worried voice and outputs it.

[0021] The robot 100 also transmits user reaction information indicating that this behavior has elicited a positive reaction from the user to the server 300. The user reaction information includes, for example, the user's behavior of "getting angry," the robot's behavior of "apologizing," the fact that the user's reaction was positive, and the user's attributes.

[0022] The server 300 stores the user reaction information received from each robot 100. Then, the server 300 analyzes the user reaction information from each robot 100 and updates the reaction rules.

[0023] The robot 100 receives the updated reaction rules from the server 300 by inquiring about the updated reaction rules from the server 300. The robot 100 incorporates the updated reaction rules into the reaction rules stored in the robot 100. This allows the robot 100 to incorporate reaction rules acquired by other robots 100 into its own reaction rules. When the reaction rules are updated, they may be automatically transmitted from the server 300 to the robot 100.

[0024] The robot 100 can also perform actions in cooperation with linked devices 400. Examples of the linked devices 400 include karaoke machines, wine cellars, refrigerators, terminal devices (such as personal computers (PCs), smartphones, and tablets), washing machines, automobiles, cameras, toilet equipment, electric toothbrushes, televisions, displays, furniture (such as closets), medicine boxes, musical instruments, lighting equipment, and exercise toys (such as unicycles). These linked devices 400 are communicably connected to the robot 100 via the communication network 20, and transmit and receive information to and from the robot 100. With this configuration, the linked devices 400 control themselves and converse with the user according to instructions from the robot 100.

[0025] In the present disclosure, an example will be described in which the robot 100 cooperates with a terminal device (such as a personal computer (PC) 400a, a smartphone 400b, or a tablet 400c) that is a cooperative device 400 to perform various actions on a user.

[0026] The robot 100 recognizes the behavior of the user who is giving a presentation, determines its own behavior corresponding to the recognized user behavior, and controls the control target based on its own determined behavior. Specifically, when the user practices a presentation, the robot 100 performs an action that improves a predetermined level of completion of the content of the user's presentation.

[0027] As described above, when a user practices a presentation, the robot 100 performs actions to improve the quality of the presentation, such as listening to the content of the presentation, responding to the content, pointing out the content, and suggesting improvements. For example, the robot 100 analyzes the content of the presentation in the practice presentation, extracts sections where a predetermined indicator of the quality of the presentation is below a threshold, and determines an action related to predetermined feedback for the extracted sections. Specifically, the robot 100 uses one or a combination of typos, omissions, content errors, the quality of the content, the user's voice volume, the speed of the presentation, eye contact, and changes in the user's emotions as the predetermined indicator of the quality of the presentation, and extracts sections where the indicator of the quality of the presentation is below a threshold. For example, as an action to improve the quality of the presentation, the robot 100 analyzes the content of the practice presentation and the presentation, and provides feedback for the relevant sections when a predetermined indicator is below a predetermined threshold. The feedback referred to here refers to comments, guidance, and suggestions to improve the quality of the user's presentation, such as correcting typos, changing expressions, deleting or adding content, changing the structure, and improving the volume, posture, and attitude of the speaker during the presentation.

[0028] Furthermore, when the robot 100 recognizes that the user is practicing a presentation as a presentation, it determines an action to enhance the emotions of the user who is giving the presentation. Specifically, when the robot 100 recognizes that the user is practicing a presentation as a presentation, it can utter utterances to the user such as "That was a great presentation!" or "Your voice is loud and easy to understand!" as an action to enhance the emotions of the user.

[0029] Furthermore, the robot 100 can make utterances related to the content of the presentation when the user has finished practicing the presentation. Specifically, when the user has finished practicing the presentation, the robot 100 can make utterances to the user such as "That was a great presentation!" or "Your voice is loud and easy to understand!" to enhance the user's emotions. Furthermore, when the user has finished practicing the presentation, the robot 100 can make utterances to the user such as "It would be better if you explained it a little more slowly" or "There was a mistake in part XX, so let's fix that" to improve the content of the presentation.

[0030] Furthermore, when the robot 100 receives a voice from the user requesting that the content of the presentation be improved, the robot 100 determines an action to improve the quality of the content of the presentation. Specifically, the robot 100 can determine an action to improve the quality of the content of the presentation based on the user's utterance during or after the user has finished practicing the presentation. For example, when the user utters, "Is there anything I can improve about the content of the presentation?", the robot 100 can make suggestions such as, "It would be better if you explained it a little more slowly," or "There was a mistake in part XX, so let's fix it."

[0031] Furthermore, the robot 100 can identify the content of the presentation and determine an action related to predetermined feedback that is tailored to the identified content of the presentation. Specifically, the robot 100 can analyze the content of the user's presentation and determine an action that will improve the completeness of the presentation content. For example, as an action to improve the completeness, the robot 100 acquires and analyzes the content of the presentation, and when a preset index is less than a predetermined threshold, provides the above-described feedback on the relevant part. Note that the "acquire" in this section does not necessarily mean a speech format such as a presentation practice by the user, but may also mean that data related to the content of the presentation is read and analyzed by an information processing device or the like.

[0032] Furthermore, when the user practices a presentation, the robot 100 performs an action based on the user's state and behavior to improve the level of completion of the content of the user's presentation. For example, when the user holds a terminal device and the robot 100 recognizes that "the speed of the user's practice presentation is too fast," the robot 100 can talk to the user in a conversational manner and make suggestions to improve the content of the presentation.

[0033] In this manner, in the present disclosure, the robot 100 can perform actions that improve the quality of a presentation for a user, i.e., practice the presentation together with the user, by performing actions in cooperation with a terminal device (such as a PC, a smartphone, or a tablet). In other words, the robot 100 according to the present disclosure can perform appropriate actions for a user.

[0034] 2 is a diagram illustrating a schematic functional configuration of the robot 100. The robot 100 is configured by a control unit having a sensor unit 200, a sensor module unit 210, a storage unit 220, a user state recognition unit 230, an emotion determination unit 232, a behavior recognition unit 234, a behavior determination unit 236, a memory control unit 238, a behavior control unit 250, a control target 252, and a communication processing unit 280.

[0035] The control object 252 includes a display device, a speaker, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 100 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 100 can be expressed by controlling these motors. The facial expressions of the robot 100 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 100. For example, the display device is provided on the chest of the robot 100. The facial expressions of the robot 100 can also be expressed by controlling the display on the display device. The display device may display the content of a conversation with a user as text. The posture, gestures, and facial expressions of the robot 100 are examples of the attitude of the robot 100.

[0036] The sensor unit 200 includes a microphone 201, a 3D depth sensor 202, a 2D camera 203, a distance sensor 204, an acceleration sensor 205, a thermosensor 206, and a touch sensor 207. The microphone 201 continuously detects sound and outputs audio data. The microphone 201 may be provided on the head of the robot 100 and may have a binaural recording function. The 3D depth sensor 202 continuously emits an infrared pattern and detects the contour of an object by analyzing the infrared pattern from infrared images continuously captured by the infrared camera. The 2D camera 203 is an example of an image sensor. The 2D camera 203 captures images using visible light and generates visible light video information. The contour of an object may be detected from the video information generated by the 2D camera 203. The distance sensor 204 detects the distance to an object by emitting, for example, a laser or ultrasonic wave. The acceleration sensor 205 is, for example, a gyro sensor and detects the acceleration of the robot 100. The thermosensor 206 detects the temperature around the robot 100. The touch sensor 207 is a sensor that detects a touch operation by the user, and is disposed, for example, on the head and hands of the robot 100. The sensor unit 200 may also include a clock, a sensor for motor feedback, and the like.

[0037] 2, the components of the robot 100 excluding the control target 252 and the sensor unit 200 are examples of components included in the behavior control system of the robot 100. The behavior control system of the robot 100 controls the control target 252.

[0038] The storage unit 220 includes reaction rules 221 and history data 222. The history data 222 includes the user's past emotional values ​​and behavioral history. This emotional value and behavioral history is recorded for each user, for example, by being associated with the user's identification information. At least a portion of the storage unit 220 is implemented as a storage medium such as a memory. It may also include a person DB that stores the user's facial image, user attribute information, and the like. Note that the functions of the components of the robot 100 shown in FIG. 2 , excluding the control target 252, the sensor unit 200, and the storage unit 220, can be realized by a CPU operating based on a program. For example, the functions of these components can be implemented as CPU operations using operating system (OS) and a program running on the OS.

[0039] The sensor module unit 210 includes a voice emotion recognition unit 211, a speech understanding unit 212, a facial expression recognition unit 213, and a face recognition unit 214. Information detected by the sensor unit 200 is input to the sensor module unit 210. The sensor module unit 210 analyzes the information detected by the sensor unit 200 and outputs the analysis result to the user state recognition unit 230.

[0040] The voice emotion recognition unit 211 of the sensor module unit 210 analyzes the user's voice detected by the microphone 201 and recognizes the user's emotion. For example, the voice emotion recognition unit 211 extracts feature quantities such as frequency components of the voice and recognizes the user's emotion based on the extracted feature quantities. The utterance understanding unit 212 analyzes the user's voice detected by the microphone 201 and outputs text information representing the content of the user's utterance. For example, the utterance understanding unit 212 can analyze the content of a question to the robot 100, such as "Is there anything we can improve on in the content of our presentation?" and output text information representing the content of the user's utterance.

[0041] The facial expression recognition unit 213 recognizes the facial expression and emotion of the user from the image of the user captured by the 2D camera 203. For example, the facial expression recognition unit 213 recognizes the facial expression and emotion of the user based on the shape, positional relationship, etc. of the eyes and mouth. For example, the facial expression recognition unit 213 can recognize the facial expression and emotion when giving a presentation or when asking the robot 100 a question.

[0042] The face recognition unit 214 recognizes the face of the user. The face recognition unit 214 recognizes the user by matching a face image stored in a person DB (not shown) with the face image of the user captured by the 2D camera 203.

[0043] The user state recognition unit 230 recognizes the state of the user based on the information analyzed by the sensor module unit 210. For example, the user state recognition unit 230 mainly performs processing related to perception using the analysis results of the sensor module unit 210. For example, the user state recognition unit 230 generates perceptual information such as "The user is practicing for a presentation" or "The user's speaking speed exceeds a predetermined threshold" and performs processing to understand the meaning of the generated perceptual information. For example, the user state recognition unit 230 generates semantic information such as "The user is speaking too fast during the presentation practice."

[0044] The emotion determination unit 232 determines an emotion value indicating the user's emotion based on the information analyzed by the sensor module unit 210 and the user's state recognized by the user state recognition unit 230. For example, the information analyzed by the sensor module unit 210 and the recognized user's state are input into a pre-trained neural network to obtain an emotion value indicating the user's emotion.

[0045] Here, the emotion value indicating the user's emotion is a value indicating whether the user's emotion is positive or negative. For example, if the user's emotion is a cheerful emotion accompanied by a sense of pleasure or comfort, such as "joy," "pleasure," "comfort," "relief," "excitement," "relief," and "fulfillment," the value is positive, and the cheerfulr the emotion, the larger the value. If the user's emotion is a negative emotion, such as "anger," "sorrow," "discomfort," "anxiety," "sorrow," "worry," and "emptiness," the value is negative, and the more unpleasant the emotion, the larger the absolute value of the negative value. If the user's emotion is none of the above ("neutral"), the value is 0.

[0046] Furthermore, the emotion determination unit 232 determines an emotion value that indicates the emotion of the robot 100 based on the information analyzed by the sensor module unit 210 and the state of the user recognized by the user state recognition unit 230.

[0047] The emotion value of the robot 100 includes emotion values ​​for each of a plurality of emotion categories, and is a value (0 to 5) indicating the strength of each of "joy," "anger," "sorrow," and "happiness," for example.

[0048] Specifically, the emotion determination unit 232 determines an emotion value indicating the emotion of the robot 100 in accordance with a rule for updating the emotion value of the robot 100, which rule is determined in association with the information analyzed by the sensor module unit 210 and the state of the user recognized by the user state recognition unit 230.

[0049] For example, when the user state recognition unit 230 recognizes that the user looks lonely, the emotion determination unit 232 increases the emotion value of "sadness" of the robot 100. When the user state recognition unit 230 recognizes that the user is smiling, the emotion determination unit 232 increases the emotion value of "joy" of the robot 100.

[0050] The emotion determination unit 232 may determine the emotion value indicating the emotion of the robot 100 by further considering the state of the robot 100. For example, when the remaining battery power of the robot 100 is low or when the surrounding environment of the robot 100 is pitch black, the emotion value of "sadness" of the robot 100 may be increased. Furthermore, when a user continues to talk to the robot 100 despite the remaining battery power being low, the emotion value of "anger" may be increased.

[0051] The behavior recognition unit 234 recognizes the user's behavior based on the information analyzed by the sensor module unit 210 and the user's state recognized by the user state recognition unit 230. For example, the information analyzed by the sensor module unit 210 and the recognized user's state are input to a pre-trained neural network, the probability of each of a plurality of predetermined behavior categories (e.g., "laughing," "angry," "asking a question," "sad") is obtained, and the behavior category with the highest probability is recognized as the user's behavior. For example, the behavior recognition unit 234 recognizes the user's behavior such as "operating a terminal device," "speaking the content of a presentation," "thinking about what to say during a presentation," "answering a question," etc.

[0052] As described above, in this embodiment, the robot 100 identifies the user and then acquires the user's speech content. When acquiring and using the speech content, the robot 100 obtains the necessary consent from the user in accordance with laws and regulations, and the behavior control system of the robot 100 according to this embodiment takes into consideration the protection of the user's personal information and privacy.

[0053] The behavior determination unit 236 determines a behavior corresponding to the user's behavior recognized by the behavior recognition unit 234 based on the user's current emotion value determined by the emotion determination unit 232, history data 222 of past emotion values ​​determined by the emotion determination unit 232 before the user's current emotion value was determined, and the emotion value of the robot 100. In this embodiment, the behavior determination unit 236 uses one of the most recent emotion values ​​included in the history data 222 as the user's past emotion value, but the disclosed technology is not limited to this aspect. For example, the behavior determination unit 236 may use multiple most recent emotion values ​​as the user's past emotion value, or may use an emotion value from a unit period ago, such as one day ago. Furthermore, the behavior determination unit 236 may determine a behavior corresponding to the user's behavior by taking into consideration not only the robot 100's current emotion value but also the history of the robot 100's past emotion values. The behavior determined by the behavior determination unit 236 includes gestures performed by the robot 100 or the content of utterances made by the robot 100.

[0054] The behavior determining unit 236 may determine a behavior corresponding to the user's behavior based on the emotion of the robot 100. For example, if the robot 100 is verbally abused by the user or is arrogant to the user (i.e., if the user's reaction is poor), if the user's voice cannot be detected due to loud ambient noise, or if the remaining battery charge of the robot 100 is low, and the emotional value of "anger" or "sadness" of the robot 100 increases, the behavior determining unit 236 may determine a behavior corresponding to the user's behavior according to the increase in the emotional value of "anger" or "sadness." Also, if the user's reaction is good or the remaining battery charge of the robot 100 is high, and the emotional value of "joy" or "happiness" of the robot 100 increases, the behavior determining unit 236 may determine a behavior corresponding to the user's behavior according to the increase in the emotional value of "joy" or "happiness." Furthermore, the behavior determining unit 236 may determine a behavior for a user who has increased the emotional values ​​of "joy" or "happiness" of the robot 100 that is different from the behavior for a user who has increased the emotional values ​​of "anger" or "sadness" of the robot 100. In this way, the behavior determining unit 236 may determine a different behavior depending on the emotion of the robot itself and how the user has changed the emotion of the robot 100 through the user's behavior.

[0055] The behavior determination unit 236 according to this embodiment determines the behavior of the robot 100 as a behavior corresponding to the user's behavior, based on a combination of the user's past and current emotional values, the emotional value of the robot 100, the user's behavior, and the reaction rules 221. For example, if the user's past emotional value is a positive value and the current emotional value is a negative value, the behavior determination unit 236 determines, as a behavior corresponding to the user's behavior, a behavior that will change the user's emotional value to a positive value.

[0056] The reaction rule 221 defines the behavior of the robot 100 according to a combination of the user's past emotional value and current emotional value, the emotional value of the robot 100, and the user's behavior. For example, if the user's past emotional value is a positive value and the current emotional value is a negative value, and the user's behavior is sad, a combination of gestures and speech content when asking a question to encourage the user with gestures is defined as the behavior of the robot 100.

[0057] For example, the reaction rules 221 define behaviors of the robot 100 for all combinations of patterns of the robot 100's emotional values ​​(1296 patterns, which are the fourth power of six values ​​from "0" to "5" for "joy," "anger," "sadness," and "happiness"), patterns of combinations of the user's past emotional values ​​and current emotional values, and the user's behavior patterns. That is, for each pattern of the robot 100's emotional values, behaviors of the robot 100 are defined according to the user's behavior patterns for each of a plurality of combinations of the user's past emotional values ​​and current emotional values, such as negative and negative values, negative and positive values, positive and negative values, positive and positive values, negative and normal values, and normal and normal values. Note that the behavior determination unit 236 may transition to an operation mode in which the behavior of the robot 100 is determined using the history data 222 when the user makes an utterance intending to continue a conversation from a past topic, such as "I want to talk about that topic we talked about last time."

[0058] The reaction rules 221 may prescribe at least one of a gesture and a statement as the behavior of the robot 100 for each of the patterns (1296 patterns) of the emotional value of the robot 100. Alternatively, the reaction rules 221 may prescribe at least one of a gesture and a statement as the behavior of the robot 100 for each group of patterns of the emotional value of the robot 100.

[0059] The strength of each gesture included in the behavior of the robot 100 defined in the reaction rules 221 is predetermined. The strength of each utterance included in the behavior of the robot 100 defined in the reaction rules 221 is predetermined.

[0060] For example, the reaction rules 221 define the behavior of the robot 100 corresponding to behavioral patterns such as when operating a terminal device, when speaking about the content of a presentation, when thinking about what to say during a presentation, when answering a question, and when speaking about a user's request. An example of a user's request is a question to the robot 100 such as, "Is there anything I can improve on in the content of the presentation?"

[0061] The memory control unit 238 determines whether or not to store data including the user's behavior in the history data 222 based on the predetermined behavior intensity for the behavior determined by the behavior determination unit 236 and the emotion value of the robot 100 determined by the emotion determination unit 232.

[0062] Specifically, if the total intensity value, which is the sum of the sum of the emotion values ​​for each of the multiple emotion classifications of the robot 100, the predetermined intensity for the gesture included in the behavior determined by the behavior determination unit 236, and the predetermined intensity for the speech content included in the behavior determined by the behavior determination unit 236, is equal to or greater than a threshold value, it is determined to store data including the user's behavior in the history data 222.

[0063] When the memory control unit 238 decides to store data including the user's behavior in the history data 222, it stores in the history data 222 the behavior determined by the behavior determination unit 236, information analyzed by the sensor module unit 210 from the present time up to a certain period of time ago (for example, all peripheral information such as data on the sound, images, smells, etc. of the scene), and the user's state recognized by the user state recognition unit 230 (for example, the user's facial expression, emotions, etc.).

[0064] The behavior control unit 250 controls the control target 252 based on the behavior determined by the behavior determination unit 236. For example, when the behavior determination unit 236 determines an behavior including speaking, the behavior control unit 250 outputs a sound from a speaker included in the control target 252. At this time, the behavior control unit 250 may determine the speaking rate of the sound based on the emotional value of the robot 100. For example, the behavior control unit 250 determines a faster speaking rate as the emotional value of the robot 100 increases. In this way, the behavior control unit 250 determines the execution form of the behavior determined by the behavior determination unit 236 based on the emotional value determined by the emotion determination unit 232. Specifically, when practicing a presentation to be given by a user, the behavior control unit 250 performs behaviors to improve the quality of the presentation, such as listening to the content of the presentation, reacting to it, pointing out the content, and suggesting improvements.

[0065] For example, the behavior control unit 250 analyzes the content of a presentation during a practice presentation, extracts sections where a predetermined indicator of completeness is below a threshold, and determines a behavior related to predetermined feedback for the extracted sections. The behavior control unit 250 also uses one or a combination of typos, omissions, content errors, content completeness, the user's voice volume, presentation speed, eye contact, and changes in the user's emotions as predetermined indicators of completeness in the presentation content, and extracts sections where the indicator of completeness is below a threshold. The behavior control unit 250 also analyzes the content of the practice presentation and the presentation as an action to improve the completeness of the presentation, and provides feedback on the relevant sections if a predetermined indicator is below a predetermined threshold. The feedback referred to here refers to guidance, instruction, and suggestions to improve the completeness of the user's presentation, such as correcting typos, changing expressions, deleting or adding content, changing the structure, and improving the voice volume, posture, and attitude during the presentation.

[0066] Furthermore, when the behavior control unit 250 recognizes that the user is practicing a presentation, it can speak to the user during or after practice, such as "That was a great presentation!" or "Your voice is loud and easy to understand!" as an action to enhance the emotions of the user giving the presentation. Furthermore, the behavior control unit 250 can speak to the user during or after practice, such as "It would be better if you explained it a little more slowly" or "There was a mistake in part XX, so let's fix that" in order to improve the content of the presentation.

[0067] Furthermore, the behavior control unit 250 analyzes the content of the user's presentation, and provides feedback on the relevant points when preset indicators such as typos, omissions, errors in the content, the quality of the content, the user's voice volume, the speed of the presentation, the direction of the user's eyes, and changes in the user's emotions are below a predetermined threshold. Note that the behavior control unit 250 may read and analyze data related to the content of the presentation, rather than using speech format such as a presentation practice by the user.

[0068] Furthermore, when the behavior control unit 250 receives a voice request from the user requesting that the content of the presentation be made more complete during or after the user has finished practicing the presentation, the behavior control unit 250 can determine an action that will make the content of the presentation more complete. For example, when the user says, "Is there anything that can be improved about the content of the presentation?", the behavior control unit 250 can make suggestions such as, "It would be better if you explained it a little more slowly," or "There was a mistake in part XX, so let's fix it."

[0069] The behavior control unit 250 may recognize a change in the user's emotions in response to the execution of the behavior determined by the behavior determination unit 236. For example, the change in emotions may be recognized based on the user's voice or facial expression. Alternatively, the change in the user's emotions may be recognized based on the detection of an impact by a touch sensor included in the sensor unit 200. If an impact is detected by the touch sensor included in the sensor unit 200, the behavior control unit 250 may recognize that the user's emotions have worsened, or if the detection result of the touch sensor included in the sensor unit 200 indicates that the user's reaction is laughing, happy, or the like, the behavior control unit 250 may recognize that the user's emotions have improved. Information indicating the user's reaction is output to the communication processing unit 280.

[0070] Furthermore, after the behavior control unit 250 executes the behavior determined by the behavior determination unit 236 in the execution mode determined according to the emotion of the robot 100, the emotion determination unit 232 further changes the emotion value of the robot 100 based on the user's reaction to the execution of the behavior. Specifically, the emotion determination unit 232 increases the emotion value of "joy" of the robot 100 when the user's reaction to the behavior determined by the behavior determination unit 236 being performed on the user in the execution mode determined by the behavior control unit 250 is not negative. Furthermore, the emotion determination unit 232 increases the emotion value of "sad" of the robot 100 when the user's reaction to the behavior determined by the behavior determination unit 236 being performed on the user in the execution mode determined by the behavior control unit 250 is negative.

[0071] Furthermore, the behavior control unit 250 expresses the emotion of the robot 100 based on the determined emotion value of the robot 100. For example, when the emotion value of "happiness" of the robot 100 is increased, the behavior control unit 250 controls the control object 252 to make the robot 100 perform a happy gesture. When the emotion value of "sadness" of the robot 100 is increased, the behavior control unit 250 controls the control object 252 to make the robot 100 assume a droopy posture.

[0072] Furthermore, the behavior control unit 250 changes the behavior of the robot 100 based on the above-described change in the emotion of the robot 100. For example, when the robot 100, which has listened to a user's practice presentation, increases its emotion value of "joy," the behavior control unit 250 can take an action of actively praising the user's presentation, such as saying, "That was a great presentation, and I could understand the content very well!" On the other hand, when the behavior control unit 250 increases the emotion value of "sadness," the control target 252 can take an action of encouraging the user, such as saying, "I think your presentation is a little hard to understand, but let's work together to improve it!"

[0073] The communication processing unit 280 is responsible for communication with the server 300. As described above, the communication processing unit 280 transmits user reaction information to the server 300. The communication processing unit 280 also receives updated reaction rules from the server 300. When the communication processing unit 280 receives the updated reaction rules from the server 300, it updates the reaction rules 221. The communication processing unit 280 can transmit and receive information to and from the linked device 400.

[0074] The server 300 communicates between each robot 100 and the server 300, receives user reaction information transmitted from the robot 100, and updates the reaction rules based on reaction rules that include actions that have received positive reactions.

[0075] Fig. 3 is a diagram showing an example of an operation flow relating to an operation for determining an action in the robot 100. The operation flow shown in Fig. 3 is repeatedly executed. At this time, it is assumed that information analyzed by the sensor module unit 210 is input. Note that "S" in the operation flow indicates the step that is executed.

[0076] First, in step S101, the user state recognition unit 230 recognizes the state of the user based on the information analyzed by the sensor module unit 210. For example, the user state recognition unit 230 generates perceptual information such as "The user is practicing for a presentation" or "The user's speaking speed exceeds a predetermined threshold" and performs processing to understand the meaning of the generated perceptual information. For example, the user state recognition unit 230 generates semantic information such as "The user's speaking speed during the presentation practice is too fast."

[0077] In step S102, the emotion determination unit 232 determines an emotion value indicating the emotion of the user based on the information analyzed by the sensor module unit 210 and the state of the user recognized by the user state recognition unit 230.

[0078] In step S103, the emotion determination unit 232 determines an emotion value indicating the emotion of the robot 100 based on the information analyzed by the sensor module unit 210 and the state of the user recognized by the user state recognition unit 230. The emotion determination unit 232 adds the determined emotion value of the user to the history data 222.

[0079] In step S104, the behavior recognition unit 234 recognizes the user's behavior classification based on the information analyzed by the sensor module unit 210 and the user's state recognized by the user state recognition unit 230. For example, the behavior recognition unit 234 recognizes the user's behavior such as "operating the terminal device," "speaking the contents of the presentation," "thinking about what to say during the presentation," "answering a question," etc.

[0080] In step S105, the behavior determination unit 236 determines the behavior of the robot 100 based on a combination of the user's current emotional value determined in step S102 and the past emotional values ​​included in the history data 222, the emotional value of the robot 100, the user's behavior recognized by the behavior recognition unit 234, and the reaction rules 221.

[0081] In step S106, the behavior control unit 250 controls the control target 252 based on the behavior determined by the behavior determination unit 236. For example, when the user practices a presentation, the behavior control unit 250 executes behaviors to improve the quality of the presentation, such as listening to the content of the presentation, reacting, pointing out the content, and suggesting improvements.

[0082] In step S107, the memory control unit 238 calculates a total intensity value based on the predetermined behavior intensities for the behavior determined by the behavior determination unit 236 and the emotion value of the robot 100 determined by the emotion determination unit 232.

[0083] In step S108, the storage control unit 238 determines whether the total intensity value is equal to or greater than a threshold. If the total intensity value is less than the threshold, the process ends without storing data including the user's behavior in the history data 222. On the other hand, if the total intensity value is equal to or greater than the threshold, the process proceeds to step S109.

[0084] In step S109, the behavior determined by the behavior determination unit 236, the information analyzed by the sensor module unit 210 from the present time up to a certain period of time ago, and the user's state recognized by the user state recognition unit 230 are stored in the history data 222.

[0085] As described above, the robot 100 includes a control unit that recognizes the behavior of the user giving a presentation, determines its own behavior corresponding to the recognized user behavior, and controls the control target based on its own determined behavior. In this way, the robot 100 takes appropriate behavior for each practice presentation that the user gives, thereby achieving the effect of improving the quality of the content of the user's presentation.

[0086] Specifically, when a user practices a presentation, the control unit of the robot 100 performs an action to improve the completeness of the content of the user's presentation. For example, the control unit of the robot 100 analyzes the content of the presentation in the practice presentation, extracts sections where a predetermined completeness index is below a threshold, and determines an action related to predetermined feedback for the extracted sections. As a specific example, the robot 100 uses one or a combination of typos, omissions, content errors, content richness, the user's voice volume, presentation speed, eye contact, and changes in the user's emotions as the predetermined completeness index and extracts sections where the completeness index is below a threshold. As a result, when the robot 100 practices a presentation as a user's presentation, it can perform actions to improve the completeness of the presentation, such as listening to the presentation content, responding to it, pointing out the content, and suggesting improvements. Furthermore, the robot 100 enables the user to point out specific parts and content of the presentation rather than just pointing out abstract areas for improvement. In this way, the robot 100 participates in the user's presentation practice and achieves the effect of improving the quality of the content of the user's presentation through two-way communication.

[0087] Furthermore, when the control unit of the robot 100 recognizes that the user is practicing a presentation as a presentation, it determines an action to enhance the emotions of the user who is giving the presentation. As a result, the robot 100 can utter utterances that enhance the emotions of the user practicing the presentation as a presentation, such as "That was a great presentation!" or "Your voice is loud and easy to understand!". As a result, the robot 100 participates in the user's presentation practice together, and enhances the user's emotions and motivation for the presentation through two-way communication, thereby achieving the effect of improving the quality of the presentation.

[0088] Furthermore, the control unit of the robot 100 makes a speech regarding the content of the presentation when the user finishes practicing the presentation. As a result, when the practice of the presentation is finished, the robot 100 can make speeches to the user that enhance the user's emotions, such as "That was a great presentation!" or "Your voice is loud and easy to understand!" On the other hand, when the practice of the presentation is finished, the robot 100 can make speeches to the user to improve the quality of the presentation, such as "It would be better if you explained it a little more slowly" or "There was a mistake in part XX, so let's fix that." As a result, the robot 100 can participate in the user's presentation practice together, communicate with the user two-way, enhance the user's emotions and motivation for the presentation, and make suggestions for improvement to improve the quality of the presentation.

[0089] Furthermore, when the control unit of the robot 100 receives a voice request from the user requesting that the content of the presentation be improved, the control unit of the robot 100 determines an action to improve the content of the presentation. In this way, when the user utters, "Is there anything that can be improved in the content of the presentation?", the robot 100 can make suggestions such as, "It would be better if you explained it a little more slowly," or "There was a mistake in part XX, so let's fix that." In this way, the robot 100 participates in the user's presentation practice together, and achieves the effect of improving the content of the user's presentation through two-way communication.

[0090] Furthermore, the control unit of the robot 100 identifies the content of the presentation and determines a behavior related to predetermined feedback that matches the identified content of the presentation. This allows the robot 100 to acquire and analyze the content of the presentation, and if a preset indicator is below a predetermined threshold, provide feedback on the relevant part. Therefore, even for a user who has difficulty practicing a presentation, the robot 100 can accurately suggest improvements by using data related to the presentation prepared by the user. In this way, the robot 100 achieves the effect of improving the quality of presentations regardless of the user's attributes.

[0091] In the above embodiment, the robot 100 recognizes the user using a facial image of the user, but the disclosed technology is not limited to this. For example, the robot 100 may recognize the user using a voice uttered by the user, the user's email address, the user's SNS ID, or an ID card with a built-in wireless IC tag that the user owns.

[0092] The robot 100 is an example of an electronic device equipped with a behavior control system. The application of the behavior control system is not limited to the robot 100, and the behavior control system can be applied to various electronic devices. The functions of the server 300 may be implemented by one or more computers. At least some of the functions of the server 300 may be implemented by a virtual machine. At least some of the functions of the server 300 may be implemented in the cloud.

[0093] 4 is a diagram schematically illustrating an example of the hardware configuration of a computer 1200 that functions as the robot 100 and the server 300. A program installed on the computer 1200 can cause the computer 1200 to function as one or more "parts" of an apparatus according to the present embodiment, or can cause the computer 1200 to perform operations associated with the apparatus according to the present embodiment or one or more "parts," and / or can cause the computer 1200 to perform a process according to the present embodiment or steps of the process. Such a program can be executed by the CPU 1212 to cause the computer 1200 to perform specific operations associated with some or all of the blocks in the flowcharts and block diagrams described herein.

[0094] The computer 1200 according to this embodiment includes a CPU 1212, a RAM 1214, and a graphics controller 1216, which are interconnected by a host controller 1210. The computer 1200 also includes input / output units such as a communications interface 1222, a storage device 1224, a DVD drive, and an IC card drive, which are connected to the host controller 1210 via an input / output controller 1220. The DVD drive may be a DVD-ROM drive, a DVD-RAM drive, or the like. The storage device 1224 may be a hard disk drive, a solid-state drive, or the like. The computer 1200 also includes input / output units such as a ROM 1230 and a keyboard, which are connected to the input / output controller 1220 via an input / output chip 1240.

[0095] The CPU 1212 operates according to programs stored in the ROM 1230 and the RAM 1214, thereby controlling each unit. The graphics controller 1216 acquires image data generated by the CPU 1212 into a frame buffer or the like provided in the RAM 1214 or into the graphics controller itself, and causes the image data to be displayed on the display device 1218.

[0096] The communication interface 1222 communicates with other electronic devices via a network. The storage device 1224 stores programs and data used by the CPU 1212 in the computer 1200. The DVD drive reads programs or data from a DVD-ROM or the like and provides them to the storage device 1224. The IC card drive reads programs and data from an IC card and / or writes programs and data to an IC card.

[0097] The ROM 1230 stores therein a boot program or the like that is executed by the computer 1200 upon activation, and / or programs that depend on the hardware of the computer 1200. The input / output chip 1240 may also connect various input / output units to the input / output controller 1220 via a USB port, a parallel port, a serial port, a keyboard port, a mouse port, etc.

[0098] The programs are provided by a computer-readable storage medium such as a DVD-ROM or an IC card. The programs are read from the computer-readable storage medium, installed in the storage device 1224, RAM 1214, or ROM 1230, which are also examples of computer-readable storage media, and executed by the CPU 1212. Information processing described in these programs is read by the computer 1200, and causes cooperation between the programs and the various types of hardware resources described above. An apparatus or method may be configured by implementing operations or processing of information in accordance with the use of the computer 1200.

[0099] For example, when communication is performed between computer 1200 and an external device, CPU 1212 may execute a communication program loaded into RAM 1214 and instruct communication interface 1222 to perform communication processing based on the processing described in the communication program. Under the control of CPU 1212, communication interface 1222 reads transmission data stored in a transmission buffer area provided in RAM 1214, storage device 1224, a DVD-ROM, or a recording medium such as an IC card, and transmits the read transmission data to the network, or writes received data received from the network to a reception buffer area or the like provided on the recording medium.

[0100] Furthermore, the CPU 1212 may cause all or a necessary portion of a file or database stored in an external recording medium such as the storage device 1224, a DVD drive (DVD-ROM), an IC card, etc. to be read into the RAM 1214, and may perform various types of processing on the data on the RAM 1214. The CPU 1212 may then write back the processed data to the external recording medium.

[0101] Various types of information, such as various types of programs, data, tables, and databases, may be stored on the recording medium and may undergo information processing. CPU 1212 may perform various types of processing on data read from RAM 1214, including various types of operations, information processing, conditional judgment, conditional branching, unconditional branching, information search / replacement, etc., as described throughout this disclosure and specified by the instruction sequences of the programs, and write the results back to RAM 1214. CPU 1212 may also search for information in files, databases, etc. on the recording medium. For example, if multiple entries, each having an attribute value of a first attribute associated with an attribute value of a second attribute, are stored on the recording medium, CPU 1212 may search for an entry whose attribute value of the first attribute matches a specified condition from among the multiple entries, read the attribute value of the second attribute stored in the entry, and thereby obtain the attribute value of the second attribute associated with the first attribute that satisfies a predetermined condition.

[0102] The above-described programs or software modules may be stored in a computer-readable storage medium on or near the computer 1200. A recording medium such as a hard disk or RAM provided in a server system connected to a dedicated communication network or the Internet can also be used as a computer-readable storage medium, thereby providing the programs to the computer 1200 via the network.

[0103] The blocks in the flowcharts and block diagrams in the present embodiments may represent stages of a process in which an operation is performed or "parts" of an apparatus responsible for performing the operation. Particular stages and "parts" may be implemented by dedicated circuitry, programmable circuitry provided with computer-readable instructions stored on a computer-readable storage medium, and / or a processor provided with computer-readable instructions stored on a computer-readable storage medium. The dedicated circuitry may include digital and / or analog hardware circuits, including integrated circuits (ICs) and / or discrete circuits. The programmable circuitry may include reconfigurable hardware circuits, such as field programmable gate arrays (FPGAs) and programmable logic arrays (PLAs), including AND, OR, XOR, NAND, NOR, and other logical operations, flip-flops, registers, and memory elements.

[0104] A computer-readable storage medium may include any tangible device capable of storing instructions that are executed by an appropriate device, such that a computer-readable storage medium having instructions stored thereon comprises an article of manufacture, including instructions that can be executed to create means for performing the operations specified in the flowcharts or block diagrams. Examples of computer-readable storage media may include electronic storage media, magnetic storage media, optical storage media, electromagnetic storage media, semiconductor storage media, etc. More specific examples of computer-readable storage media may include floppy disks, diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), electrically erasable programmable read-only memory (EEPROM), static random access memory (SRAM), compact disc read-only memory (CD-ROM), digital versatile disc (DVD), Blu-ray disc, memory stick, integrated circuit card, etc.

[0105] The computer readable instructions may include either assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state-setting data, or source or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk®, JAVA®, C++, etc., and conventional procedural programming languages ​​such as the “C” programming language or similar programming languages.

[0106] Computer-readable instructions may be provided locally or over a wide area network (WAN) such as a local area network (LAN), the Internet, etc. to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus, or programmable circuitry, such that the processor or programmable circuitry executes the computer-readable instructions to generate means for performing the operations specified in the flowcharts or block diagrams. Examples of processors include computer processors, processing units, microprocessors, digital signal processors, controllers, microcontrollers, etc.

[0107] (Other embodiments) The robot 100 described above may be mounted on a stuffed toy, or may be applied to a control device connected wirelessly or by wire to a control target device (speaker or camera) mounted on a stuffed toy.

[0108] The emotion determining unit 232 may determine the emotion of the user in accordance with a specific mapping. Specifically, the emotion determining unit 232 may determine the emotion of the user in accordance with an emotion map (see FIG. 5), which is a specific mapping.

[0109] FIG. 5 is a diagram illustrating an emotion map 700 on which multiple emotions are mapped. In emotion map 700, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both emotions and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in emotion map 700, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[0110] (1) For example, if the emotion engine, which is the emotion determination unit 232 of the robot 100, detects emotions in about 100 msec, the frequency of determining the reaction action of the robot 100 (for example, a backchannel) may be set at a timing at least as frequent as the emotion engine's detection frequency (100 msec), or may be set at a timing earlier than this. The emotion engine's detection frequency may be interpreted as a sampling rate.

[0111] By detecting emotions in about 100 msec and immediately performing a corresponding reaction (e.g., nodding), unnatural reactions are eliminated, enabling a natural, well-read dialogue. The reaction (e.g., nodding) is performed according to the direction and degree (strength) of the mandala in the robot 100 emotion map 700. Note that the detection frequency (sampling rate) of the emotion engine is not limited to 100 ms and may be changed depending on the situation (e.g., when playing sports), the user's age, etc.

[0112] (2) The directionality and intensity of emotions may be set in advance in reference to the emotion map 700, and the movement of the back-channel and the strength of the back-channel may be set. For example, if the robot 100 feels a sense of stability, security, etc., the robot 100 may nod and continue listening. If the robot 100 feels anxious, confused, or suspicious, the robot 100 may tilt its head or stop shaking its head.

[0113] These emotions are distributed in the 3 o'clock direction of the emotion map 700, and usually fluctuate between relief and anxiety. In the right half of the emotion map 700, situational awareness dominates over internal sensations, resulting in a sense of calm.

[0114] (3) If the robot 100 receives a compliment and feels good, the filler "ah" may precede the line, and if the robot 100 receives harsh words and feels pain, the filler "ugh!" may precede the line. A physical reaction, such as the robot 100 crouching down while saying "ugh!", may also be included. These emotions are distributed around the 9 o'clock position on the emotion map 700.

[0115] (4) In the left half of the emotion map 700, internal sensations (reactions) are more important than situational awareness. This can give the impression of an unconscious reaction.

[0116] When the robot 100 feels a positive feeling in its situational awareness while experiencing an internal sensation (reaction) of understanding, the robot 100 may nod deeply while looking at the other person, or may say "uh-huh." In this way, the robot 100 may generate a behavior that shows a balanced positive feeling toward the other person, that is, tolerance and tolerance toward the other person. Such emotions are distributed around 12 o'clock on the emotion map 700.

[0117] Conversely, when the robot 100 is aware of an internal sensation (reaction) of discomfort, it may shake its head when it feels disgust, or may turn the LEDs in its eyes red and glare at the other person when it feels hatred. These emotions are distributed around the 6 o'clock position on the emotion map 700.

[0118] (5) The inside of emotion map 700 represents the mind, and the outside of emotion map 700 represents behavior, so the further outside emotion map 700 you go, the more visible the emotion becomes (is expressed in behavior).

[0119] (6) When listening to someone while feeling a sense of security, which is distributed around 3 o'clock on the emotion map 700, the robot 100 may nod its head lightly and say "hmm." However, when listening to someone while feeling a sense of love, which is distributed around 12 o'clock, the robot 100 may nod its head strongly, as if deeply nodding its head.

[0120] The emotion determination unit 232 inputs the information analyzed by the sensor module unit 210 and the recognized state of the user 10 into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 700, and determines the emotion of the user 10. This neural network is pre-trained based on multiple pieces of training data that are combinations of the information analyzed by the sensor module unit 210, the recognized state of the user 10, and emotion values ​​indicating each emotion shown in the emotion map 700. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 6. FIG. 6 is a diagram showing another example of an emotion map. FIG. 6 shows an example in which multiple emotions, such as "relieved," "calm," and "reassuring," have similar emotion values.

[0121] Furthermore, the emotion determination unit 232 may determine the emotion of the robot 100 according to a specific mapping. Specifically, the emotion determination unit 232 inputs the information analyzed by the sensor module unit 210, the state of the user 10 recognized by the user state recognition unit 230, and the state of the robot 100 into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 700, and determines the emotion of the robot 100. This neural network is pre-trained based on a plurality of training data that are combinations of the information analyzed by the sensor module unit 210, the recognized state of the user 10, the state of the robot 100, and emotion values ​​indicating each emotion shown in the emotion map 700. For example, the neural network is trained based on training data indicating that when it is recognized from the output of the touch sensor 207 that the robot 100 is being stroked by the user 10, the emotion value of "happy" is "3," and based on training data indicating that when it is recognized from the output of the acceleration sensor 205 that the robot 100 is being hit by the user 10, the emotion value of "anger" is "3." Furthermore, this neural network is trained so that emotions that are placed close to each other have similar values, as in the emotion map 900 shown in FIG.

[0122] Furthermore, the emotion determination unit 232 may determine the emotion of the robot 100 based on the behavioral content of the robot 100 generated by the sentence generation model. Specifically, the emotion determination unit 232 inputs the behavioral content of the robot 100 generated by the sentence generation model into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 700, and integrates the obtained emotion values ​​indicating each emotion with emotion values ​​indicating each current emotion of the robot 100 to update the emotion of the robot 100. For example, the emotion values ​​indicating each obtained emotion and emotion values ​​indicating each current emotion of the robot 100 are averaged and integrated. This neural network is trained in advance based on multiple learning data that are combinations of text indicating the behavioral content of the robot 100 generated by the sentence generation model and emotion values ​​indicating each emotion shown in the emotion map 700.

[0123] For example, if the utterance of the robot 100, "That's great. You're lucky," is obtained as the behavior of the robot 100 generated by the sentence generation model, when the text representing this utterance is input into the neural network, a high emotional value is obtained for the emotion "happy," and the emotion of the robot 100 is updated so that the emotional value of the emotion "happy" becomes higher.

[0124] The behavior determination unit 236 generates the robot's behavior content by adding fixed sentences to ask about the robot's behavior content corresponding to the user's behavior to text representing the user's behavior, the user's emotions, and the robot's emotions, and inputting the added sentences into a sentence generation model with an interactive function.

[0125] For example, the behavior determining unit 236 acquires text representing the state of the robot 100 from the emotion of the robot 100 determined by the emotion determining unit 232, using an emotion table such as that shown in Fig. 7. Fig. 7 is a diagram showing an example of an emotion table. Here, in the emotion table, an index number is assigned to each emotion value for each type of emotion, and text representing the state of the robot 100 is stored for each index number.

[0126] When the emotion of the robot 100 determined by the emotion determination unit 232 corresponds to the index number "2", the text "very happy state" is obtained. When the emotions of the robot 100 correspond to multiple index numbers, multiple texts representing the state of the robot 100 are obtained.

[0127] An emotion table as shown in FIG. 8 is also prepared for the emotions of the user 10. FIG. 8 is a diagram showing an example of an emotion table. Here, if the user's behavior is "talking to AAA", the emotion of the robot 100 is index number "2", and the emotion of the user 10 is index number "3", then "The robot is in a very happy state. The user is in a normal happy state. The user spoke to you as 'AAA'. How would you respond as the robot?" is input into the sentence generation model to obtain the content of the robot's behavior. The behavior determination unit 236 determines the robot's behavior from this content of the behavior. Note that "AAA" is the name (nickname) given to the robot 100 by the user.

[0128] In this way, the robot 100 can change its behavior according to the index number corresponding to the robot's emotion, so the user gets the impression that the robot 100 has a heart, and is encouraged to take actions such as talking to the robot.

[0129] Furthermore, the behavior determination unit 236 may generate the robot's behavior content by adding not only text representing the user's behavior, the user's emotions, and the robot's emotions, but also text representing the contents of the history data 222, adding a fixed sentence for asking about the robot's behavior content corresponding to the user's behavior, and inputting the result into a sentence generation model with a dialogue function. This allows the robot 100 to change its behavior according to the history data representing the user's emotions and behavior, so that the user has the impression that the robot has individuality and is encouraged to take actions such as talking to the robot. Furthermore, the history data may further include the robot's emotions and behavior.

[0130] Although the present invention has been described above using the embodiments, the technical scope of the present invention is not limited to the scope described in the above embodiments. It will be apparent to those skilled in the art that various modifications and improvements can be made to the above embodiments. It is clear from the claims that such modifications and improvements can also be included within the technical scope of the present invention.

[0131] It should be noted that the execution order of each process, such as operations, procedures, steps, and stages, in the devices, systems, programs, and methods shown in the claims, specifications, and drawings is not specifically stated as "before," "prior to," etc., and that the processes can be performed in any order unless the output of a previous process is used in a later process. Even if the operational flow in the claims, specifications, and drawings is described using "first," "next," etc. for convenience, this does not mean that the processes must be performed in this order. [Explanation of symbols]

[0132] 1. Control System 20. Communication Network 100 robots 200 Sensor unit 201 Mike 202 3D depth sensor 203 2D Camera 204 Distance Sensor 205 Acceleration Sensor 206 Thermosensor 207 Touch Sensor 210 Sensor module section 211 Voice Emotion Recognition Unit 212 Speech Understanding Unit 213 Facial expression recognition unit 214 Face Recognition Unit 220 Storage Unit 221 Reaction Rules 222 Historical Data 230 User state recognition unit 232 Emotion Determination Department 234 Behavior Recognition Department 236 Action Decision-Making Department 238 Memory control unit 250 Behavior Control Unit 252 Control Object 280 Communication Processing Unit 300 servers 400 Related Devices

Claims

1. a control unit that recognizes an action of a user giving a presentation, determines its own action corresponding to the recognized action of the user, and controls a control target based on the determined action of its own; Equipped with the control unit analyzes the content of the presentation, and if a preset index is less than a predetermined threshold, generates action content that improves a predetermined level of completeness of the content of the user's presentation using a sentence generation model; when receiving a voice from the user requesting that the content of the presentation be made more complete, generating action content for making the content of the presentation more complete using the sentence generation model; electronic equipment.

2. The control unit When the user practices the presentation, the user performs an action to improve the content of the presentation to a predetermined level of completion. The electronic device according to claim 1 .

3. The control unit When the presentation by the user is recognized as a practice presentation, determining an action to enhance the emotion of the user giving the presentation. The electronic device according to claim 2 .

4. The control unit identifying content of the presentation and determining a predetermined feedback action tailored to the identified content of the presentation; The electronic device according to claim 2 .

5. The control unit When the user has finished practicing the presentation, he or she makes a speech related to the content of the presentation. The electronic device according to claim 2 .

6. The control unit analyzing the content of the presentation in the presentation practice, extracting portions where a predetermined indicator of completion is less than a threshold, and determining an action regarding predetermined feedback for the extracted portions; The electronic device according to claim 2 .

7. The control unit when receiving a voice from the user requesting that the content of the presentation be improved, determining an action to improve the content of the presentation; The electronic device according to claim 2 .

8. The control unit extracting portions where the indicator of completeness is less than a threshold value using one or a combination of typos, omissions, errors in content, completeness of content, the user's voice volume, presentation speed, line of sight, and changes in the user's emotions as the predetermined indicator of completeness of the presentation; 10. The electronic device according to claim 2 or 6.

9. The electronic device includes: It is installed in the plush toy or connected wirelessly or by wire to the controlled device installed in the plush toy. The electronic device according to claim 1 .

Images