Control system

Abstract

To properly reply to speech of a user.SOLUTION: A control system according to the present invention has an input unit for accepting input from a user, a processing unit for carrying out specific processing using a sentence generation model for use in generation of a sentence in response to the input data, and an output unit for controlling electronic equipment in order to output a result of the specific processing. The specific processing includes generating information relating to child-rearing. The processing unit determines whether or not a predetermined trigger condition is satisfied, and if the trigger condition is satisfied, generates information relating to the child-rearing using the output of the sentence generation model by using the information obtained from the user input as the input data.SELECTED DRAWING: Figure 2A

Description

Technical Field

[0001] The present invention relates to a control system.

Background Art

[0002] Patent Document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a character of the chatbot, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, in the prior art, there is room for improvement in appropriately responding to user utterances of specific users such as guardians of children.

Means for Solving the Problems

[0005] According to a first aspect of the present invention, a control system is provided. The control system includes an input unit that receives user input, a processing unit that performs specific processing using a text generation model that generates text according to the input data, and an output unit that controls an electronic device so as to output the result of the specific processing. The specific processing includes generating information related to child-rearing. The processing unit determines whether a predetermined trigger condition is satisfied, and when the trigger condition is satisfied, generates the information related to child-rearing using the output of the text generation model when the information obtained from the user input is used as the input data. The electronic device may be a robot. Here, the robot includes a device that performs a physical operation, a device that outputs video or audio without performing a physical operation, and an agent that operates on software.

Brief Description of the Drawings

[0006]

Figure 1

Figure 2A

Figure 2B

Figure 3

Figure 4A

Figure 4B

Figure 4C

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Mode for Carrying Out the Invention

[0007] Hereinafter, the present invention will be described through embodiments of the invention. However, the following embodiments do not limit the invention according to the claims. Also, not all combinations of features described in the embodiments are essential for the solution means of the invention.

[0008] [First Embodiment] Figure 1 schematically shows an example of the system 5 according to the present embodiment. The system 5 includes a robot 100, a robot 101, a robot 102, and a server 300. Users 10a, 10b, 10c, and 10d are users of the robot 100. Users 11a, 11b, and 11c are users of the robot 101. Users 12a and 12b are users of the robot 102. In the description of the present embodiment, users 10a, 10b, 10c, and 10d may be collectively referred to as user 10. Also, users 11a, 11b, and 11c may be collectively referred to as user 11. Also, users 12a and 12b may be collectively referred to as user 12. The robot 101 and the robot 102 have substantially the same functions as the robot 100. Therefore, the system 5 will be described mainly taking the functions of the robot 100 into account.

[0009] The robot 100 converses with the user 10 and provides the user 10 with images. At this time, the robot 100 cooperates with a server 300 or the like that can communicate via the communication network 20 to converse with the user 10 and provide the user 10 with images or the like. For example, the robot 100 not only learns appropriate conversations by itself, but also learns in cooperation with the server 300 so as to be able to converse with the user 10 more appropriately. Also, the robot 100 causes the server 300 to record the captured video data of the user 10 or the like, requests the server 300 for the video data or the like as necessary, and provides it to the user 10.

[0010] In addition, the robot 100 has an emotion value representing the type of its own emotion. For example, the robot 100 has an emotion value representing the intensity of each of the emotions of "joy", "anger", "sorrow", "happiness", "pleasure", "displeasure", "relief", "anxiety", "sadness", "excitement", "worry", "assurance", "a sense of fulfillment", "a sense of emptiness", and "normal". When the robot 100 converses with the user 10 in a state where the emotion value of excitement is large, for example, it emits sound at a high speed. In this way, the robot 100 can express its own emotions through actions.

[0011] Further, the robot 100 may be configured to determine the actions of the robot 100 corresponding to the emotions of the user 10 by matching an article generation model using AI (Artificial Intelligence) with an emotion engine. Specifically, the robot 100 may be configured to recognize the actions of the user 10, determine the emotions of the user 10 towards the actions of the user, and determine the actions of the robot 100 corresponding to the determined emotions.

[0012] More specifically, when the robot 100 recognizes the actions of the user 10, it automatically generates the content of the actions that the robot 100 should take with respect to the actions of the user 10 using a preset article generation model. The article generation model may be interpreted as an algorithm and calculation for automatic dialogue processing by characters. Since the article generation model is publicly known as disclosed in, for example, Japanese Patent Application Laid-Open No. 2018-081444 and ChatGPT (Internet search <URL: https: / / openai.com / blog / chatgpt>), a detailed description thereof will be omitted. Such an article generation model is composed of a large language model (LLM: Large Language Model).

[0013] As described above, in this embodiment, by combining a large language model and an emotion engine, it is possible to reflect the emotions of the user 10 and the robot 100 and various language information in the actions of the robot 100. That is, according to this embodiment, a synergistic effect can be obtained by combining an article generation model and an emotion engine.

[0014] In addition, the robot 100 has a function of recognizing the actions of the user 10. The robot 100 recognizes the actions of the user 10 by analyzing the facial image of the user 10 acquired by the camera function and the voice of the user 10 acquired by the microphone function. The robot 100 determines the actions to be executed by the robot 100 based on the recognized actions of the user 10 and the like.

[0015] As an example of an action decision model, the robot 100 stores rules that define the actions to be performed by the robot 100 based on the emotions of the user 10, the emotions of the robot 100, and the actions of the user 10, and performs various actions according to the rules.

[0016] Specifically, the robot 100 has, as an example of an action decision model, reaction rules for determining the actions of the robot 100 based on the emotions of the user 10, the emotions of the robot 100, and the actions of the user 10. In the reaction rules, for example, when the action of the user 10 is "laugh", the action of "laugh" is defined as the action of the robot 100. Also, in the reaction rules, when the action of the user 10 is "get angry", the action of "apologize" is defined as the action of the robot 100. Also, in the reaction rules, when the action of the user 10 is "ask a question", the action of "answer" is defined as the action of the robot 100. In the reaction rules, when the action of the user 10 is "be sad", the action of "speak to" is defined as the action of the robot 100.

[0017] Based on the reaction rules, when the robot 100 recognizes that the action of the user 10 is "get angry", the robot 100 selects the action of "apologize" defined in the reaction rules as the action to be performed. For example, when the robot 100 selects the action of "apologize", it performs the "apologize" motion and outputs a voice representing the words "apologize".

[0018] Also, when the emotion of the robot 100 is "normal" (i.e., "joy" = 0, "anger" = 0, "sorrow" = 0, "pleasure" = 0) and the condition that the state of the user 10 is "alone and seems lonely" is satisfied, it is defined that the content of the emotional change of the emotion of the robot 100 is "become worried" and the action of "speak to" can be executed.

[0019] When the current emotion of robot 100 is "normal" based on the reaction rules and it is recognized that user 10 is alone and seems lonely, robot 100 increases the emotional value of "pity". Also, robot 100 selects the action of "addressing" as defined in the reaction rules as the action to be executed towards user 10. For example, when robot 100 selects the action of "addressing", it converts the words "What's wrong?" expressing concern into a concerned voice and outputs it.

[0020] Furthermore, robot 100 transmits user reaction information indicating that a positive reaction has been obtained from user 10 due to this action to server 300. The user reaction information includes, for example, the user action of "getting angry", the action of robot 100 of "apologizing", the fact that user 10's reaction is positive, and the attributes of user 10.

[0021] Server 300 stores the user reaction information received from robot 100. Note that server 300 receives and stores user reaction information not only from robot 100 but also from each of robot 101 and robot 102. Then, server 300 analyzes the user reaction information from robot 100, robot 101, and robot 102 and updates the reaction rules.

[0022] Robot 100 receives the updated reaction rules from server 300 by querying server 300 about the updated reaction rules. Robot 100 incorporates the updated reaction rules into the reaction rules it stores. As a result, robot 100 can incorporate the reaction rules obtained by robot 101, robot 102, etc. into its own reaction rules.

[0023] Figure 2A schematically shows the functional configuration of the robot 100. The robot 100 includes a sensor unit 200, a sensor module unit 210, a storage unit 220, a control unit 228, and a control target 252. The control unit 228 includes a state recognition unit 230, an emotion determination unit 232, an action recognition unit 234, an action determination unit 236, a memory control unit 238, an action control unit 250, a related information collection unit 270, a communication processing unit 280, and a specific processing unit 290.

[0024] The control target 252 includes a display device, a speaker, and LEDs in the eyes, as well as motors for driving the arms, hands, feet, etc. The posture and gestures of the robot 100 are controlled by controlling the motors of the arms, hands, feet, etc. A part of the emotions of the robot 100 can be expressed by controlling these motors. Also, the expression of the robot 100 can be represented by controlling the light emission state of the LEDs in the eyes of the robot 100. Note that the posture, gestures, and expression of the robot 100 are an example of the attitude of the robot 100.

[0025] The sensor unit 200 includes a microphone 201, a 3D depth sensor 202, a 2D camera 203, a distance sensor 204, a touch sensor 205, and an acceleration sensor 206. The microphone 201 continuously detects sound and outputs sound data. Note that the microphone 201 may be provided on the head of the robot 100 and have a function of performing binaural recording. The 3D depth sensor 202 continuously irradiates an infrared pattern and detects the contour of an object by analyzing the infrared pattern from the infrared images continuously captured by an infrared camera. The 2D camera 203 is an example of an image sensor. The 2D camera 203 captures images with visible light and generates video information of visible light. The distance sensor 204 detects the distance to an object by irradiating, for example, a laser or ultrasonic wave. Note that the sensor unit 200 may also include, among other things, a clock, a gyro sensor, a sensor for motor feedback, etc.

[0026] Among the components of the robot 100 shown in FIG. 2A, the components excluding the control target 252 and the sensor unit 200 are examples of the components of the action control system of the robot 100. The action control system of the robot 100 controls the control target 252.

[0027] The storage unit 220 includes an action decision model 221, history data 222, collected data 223, and action schedule data 224. The history data 222 includes the past emotional values of the user 10, the past emotional values of the robot 100, and the history of actions. Specifically, the history data 222 includes a plurality of pieces of event data including the emotional values of the user 10, the emotional values of the robot 100, and the actions of the user 10. The data including the actions of the user 10 includes camera images representing the actions of the user 10. The history of these emotional values and actions is recorded for each user 10, for example, by being associated with the identification information of the user 10. At least a part of the storage unit 220 is implemented by a storage medium such as a memory. It may include a person DB that stores the face image of the user 10, the attribute information of the user 10, and the like. Among the components of the robot 100 shown in FIG. 2A, the functions of the components excluding the control target 252, the sensor unit 200, and the storage unit 220 can be realized by the CPU operating based on a program. For example, the functions of these components can be implemented as the operations of the CPU by a basic software (OS) and a program operating on the OS.

[0028] The sensor module unit 210 includes a voice emotion recognition unit 211, a speech understanding unit 212, a facial expression recognition unit 213, and a face recognition unit 214. Information detected by the sensor unit 200 is input to the sensor module unit 210. The sensor module unit 210 analyzes the information detected by the sensor unit 200 and outputs the analysis result to the state recognition unit 230.

[0029] The voice emotion recognition unit 211 of the sensor module unit 210 analyzes the voice of the user 10 detected by the microphone 201 to recognize the emotion of the user 10. For example, the voice emotion recognition unit 211 extracts feature quantities such as the frequency components of the voice, and recognizes the emotion of the user 10 based on the extracted feature quantities. The utterance understanding unit 212 analyzes the voice of the user 10 detected by the microphone 201 and outputs character information representing the utterance content of the user 10.

[0030] The facial expression recognition unit 213 recognizes the facial expression and emotion of the user 10 from the image of the user 10 captured by the 2D camera 203. For example, the facial expression recognition unit 213 recognizes the facial expression and emotion of the user 10 based on the shape and position relationship of the eyes and mouth, etc.

[0031] The face recognition unit 214 recognizes the face of the user 10. The face recognition unit 214 recognizes the user 10 by matching the face image stored in the person DB (not shown) with the face image of the user 10 captured by the 2D camera 203.

[0032] The state recognition unit 230 recognizes the state of the user 10 based on the information analyzed by the sensor module unit 210. For example, using the analysis result of the sensor module unit 210, mainly perform processing related to perception. For example, generate perception information such as "Dad is alone." and "The probability that Dad is not smiling is 90%." Perform processing to understand the meaning of the generated perception information. For example, generate meaning information such as "Dad is alone and seems lonely."

[0033] The state recognition unit 230 recognizes the state of the robot 100 based on the information detected by the sensor unit 200. For example, the state recognition unit 230 recognizes the remaining battery level of the robot 100 and the brightness of the surrounding environment of the robot 100, etc. as the state of the robot 100.

[0034] The emotion determination unit 232 determines an emotion value indicating the emotion of user 10 based on the information analyzed by the sensor module unit 210 and the state of user 10 recognized by the state recognition unit 230. For example, the information analyzed by the sensor module unit 210 and the recognized state of user 10 are input into a pre-trained neural network to obtain an emotion value indicating the emotion of user 10.

[0035] Here, the emotion value indicating the emotion of user 10 is a value indicating the positive or negative of the user's emotion. For example, if the user's emotion is a bright emotion accompanied by pleasure or comfort, such as "joy", "happiness", "pleasure", "peace of mind", "excitement", "relief", and "a sense of fulfillment", it indicates a positive value, and the brighter the emotion, the larger the value. If the user's emotion is an emotion that makes the user feel bad, such as "anger", "sorrow", "discomfort", "uneasiness", "sadness", "worry", and "nihility", it indicates a negative value, and the worse the feeling, the larger the absolute value of the negative value. If the user's emotion is neither of the above ("ordinary"), it indicates a value of 0.

[0036] Also, the emotion determination unit 232 determines an emotion value indicating the emotion of the robot 100 based on the information analyzed by the sensor module unit 210, the information detected by the sensor unit 200, and the state of user 10 recognized by the state recognition unit 230.

[0037] The emotion value of the robot 100 includes emotion values for each of a plurality of emotion classifications. For example, it is a value (0 to 5) indicating the intensity of each of "joy", "anger", "sorrow", and "happiness".

[0038] Specifically, the emotion determination unit 232 determines an emotion value indicating the emotion of the robot 100 according to the rule for updating the emotion value of the robot 100 determined in association with the information analyzed by the sensor module unit 210 and the state of user 10 recognized by the state recognition unit 230.

[0039] For example, when the emotion determination unit 232 recognizes that the user 10 seems lonely through the state recognition unit 230, it increases the "sorrow" emotion value of the robot 100. Also, when the state recognition unit 230 recognizes that the user 10 has a smiling face, it increases the "joy" emotion value of the robot 100.

[0040] Note that the emotion determination unit 232 may further consider the state of the robot 100 to determine the emotion value indicating the emotion of the robot 100. For example, when the remaining battery level of the robot 100 is low or the surrounding environment of the robot 100 is completely dark, etc., the "sorrow" emotion value of the robot 100 may be increased. Furthermore, in the case of the user 10 who continues to talk even though the remaining battery level is low, the "anger" emotion value may be increased.

[0041] The action recognition unit 234 recognizes the action of the user 10 based on the information analyzed by the sensor module unit 210 and the state of the user 10 recognized by the state recognition unit 230. For example, the information analyzed by the sensor module unit 210 and the recognized state of the user 10 are input into a pre-learned neural network, and the probabilities of each of a plurality of predefined action classifications (for example, "laugh", "get angry", "ask a question", "be sad") are obtained, and the action classification with the highest probability is recognized as the action of the user 10.

[0042] As described above, in this embodiment, the robot 100 identifies the user 10 and then acquires the speech content of the user 10. When acquiring and using the speech content, etc., in addition to obtaining the necessary consent from the user 10 in accordance with the law, the action control system of the robot 100 according to this embodiment takes into account the protection of the personal information and privacy of the user 10.

[0043] Next, the processing of the action determination unit 236 when performing the response processing in which the robot 100 responds to the action of the user 10 will be described.

[0044] The action decision unit 236 determines an action corresponding to the action of the user 10 recognized by the action recognition unit 234 based on the current emotional value of the user 10 determined by the emotion decision unit 232, the historical data 222 of the past emotional values determined by the emotion decision unit 232 before the current emotional value of the user 10 was determined, and the emotional value of the robot 100. In the present embodiment, the action decision unit 236 will be described for the case of using one most recent emotional value included in the historical data 222 as the past emotional value of the user 10, but the disclosed technology is not limited to this aspect. For example, the action decision unit 236 may use a plurality of most recent emotional values as the past emotional value of the user 10, or may use the emotional value a unit period such as one day before. Further, the action decision unit 236 may determine an action corresponding to the action of the user 10 in consideration of not only the current emotional value of the robot 100 but also the history of the past emotional values of the robot 100. The action determined by the action decision unit 236 includes a gesture performed by the robot 100 or the content of the speech of the robot 100.

[0045] The action decision unit 236 according to the present embodiment determines the action of the robot 100 based on the combination of the past emotional value and the current emotional value of the user 10, the emotional value of the robot 100, the action of the user 10, and the action decision model 221 as an action corresponding to the action of the user 10. For example, when the past emotional value of the user 10 is a positive value and the current emotional value is a negative value, the action decision unit 236 determines an action for changing the emotional value of the user 10 to positive as an action corresponding to the action of the user 10.

[0046] In the reaction rule as the action decision model 221, the action of the robot 100 corresponding to the combination of the past emotional value and the current emotional value of the user 10, the emotional value of the robot 100, and the action of the user 10 is defined. For example, when the past emotional value of the user 10 is a positive value, the current emotional value is a negative value, and the action of the user 10 is to be sad, a combination of a gesture and the content of the speech when making an inquiry to encourage the user 10 with a gesture is defined as the action of the robot 100.

[0047] For example, in the reaction rules as the action determination model 221, for the pattern of the emotional value of the robot 100 (1296 patterns which are the fourth power of 6 values from "0" to "5" of "joy", "anger", "sorrow", and "happiness"), the combination pattern of the past emotional value and the current emotional value of the user 10, and all combinations of the action patterns of the user 10, the actions of the robot 100 are determined. That is, for each pattern of the emotional value of the robot 100, for each of the combinations of the past emotional value and the current emotional value of the user 10, such as negative value and negative value, negative value and positive value, positive value and negative value, positive value and positive value, negative value and normal, and normal and normal, etc., the actions of the robot 100 corresponding to the action pattern of the user 10 are determined. Note that the action determination unit 236 may transition to an operation mode of determining the action of the robot 100 using the history data 222, for example, when the user 10 makes a speech intending to continue a conversation from a past topic such as "I want to talk about the topic we talked about last time".

[0048] In addition, in the reaction rules as the action determination model 221, for each of the patterns of the emotional value of the robot 100 (1296 patterns), at most one of the gesture and the speech content may be determined as the action of the robot 100. Alternatively, in the reaction rules as the action determination model 221, for each group of the patterns of the emotional value of the robot 100, at least one of the gesture and the speech content may be determined as the action of the robot 100.

[0049] For each gesture included in the actions of the robot 100 determined in the reaction rules as the action determination model 221, the intensity of the gesture is predetermined. For each speech content included in the actions of the robot 100 determined in the reaction rules as the action determination model 221, the intensity of the speech content is predetermined.

[0050] The memory control unit 238 determines whether to store the data including the actions of the user 10 in the history data 222 based on the predetermined intensity of the actions determined by the action determination unit 236 and the emotional value of the robot 100 determined by the emotion determination unit 232.

[0051] Specifically, when the total comprehensive value of the intensity, which is the sum of the total emotional value for each of the multiple emotion classifications of the robot 100, the predetermined intensity for the gesture included in the action determined by the action determination unit 236, and the predetermined intensity for the utterance content included in the action determined by the action determination unit 236, is equal to or greater than the threshold value, it is determined to store the data including the actions of the user 10 in the history data 222.

[0052] When the memory control unit 238 determines to store the data including the actions of the user 10 in the history data 222, it stores the action determined by the action determination unit 236, the information analyzed by the sensor module unit 210 from the current time to a certain period before (for example, all peripheral information such as on-site voice, image, smell, and other data), and the state of the user 10 recognized by the state recognition unit 230 (for example, the expression and emotion of the user 10) in the history data 222.

[0053] The action control unit 250 controls the control target 252 based on the action determined by the action determination unit 236. For example, when the action determination unit 236 determines an action including speaking, the action control unit 250 causes the speaker included in the control target 252 to output voice. At this time, the action control unit 250 may determine the voice speaking speed based on the emotional value of the robot 100. For example, the action control unit 250 determines a faster speaking speed as the emotional value of the robot 100 increases. In this way, the action control unit 250 determines the execution form of the action determined by the action determination unit 236 based on the emotional value determined by the emotion determination unit 232.

[0054] The action control unit 250 may recognize a change in the emotion of the user 10 with respect to the execution of the action determined by the action decision unit 236. For example, it may recognize a change in emotion based on the voice or facial expression of the user 10. Additionally, based on the detection of an impact by the touch sensor 205 included in the sensor unit 200, it may recognize a change in the emotion of the user 10. When an impact is detected by the touch sensor 205 included in the sensor unit 200, it may recognize that the emotion of the user 10 has become worse, or from the detection result of the touch sensor 205 included in the sensor unit 200, when it is determined that the reaction of the user 10 is laughing or happy, etc., it may also recognize that the emotion of the user 10 has become better. Information indicating the reaction of the user 10 is output to the communication processing unit 280.

[0055] Also, after the action control unit 250 executes the action determined by the action decision unit 236 in an execution form determined according to the emotion of the robot 100, the emotion decision unit 232 further changes the emotion value of the robot 100 based on the reaction of the user to the execution of the action. Specifically, when the reaction of the user to the action determined by the action decision unit 236 being performed on the user by the action control unit 250 in the determined execution form is not bad, the emotion decision unit 232 increases the "joy" emotion value of the robot 100. Also, when the reaction of the user to the action determined by the action decision unit 236 being performed on the user by the action control unit 250 in the determined execution form is bad, the emotion decision unit 232 increases the "sorrow" emotion value of the robot 100.

[0056] Furthermore, the action control unit 250 expresses the emotion of the robot 100 based on the determined emotion value of the robot 100. For example, when the action control unit 250 increases the "joy" emotion value of the robot 100, it controls the control target 252 to make the robot 100 perform a happy gesture. Also, when the action control unit 250 increases the "sorrow" emotion value of the robot 100, it controls the control target 252 so that the posture of the robot 100 becomes a slumped posture.

[0057] The communication processing unit 280 is responsible for communication with the server 300. As described above, the communication processing unit 280 transmits user response information to the server 300. Also, the communication processing unit 280 receives updated response rules from the server 300. When the communication processing unit 280 receives updated response rules from the server 300, it updates the response rules as the action decision model 221.

[0058] The server 300 communicates with the robots 100, 101, and 102, receives the user response information transmitted from the robot 100, and updates the response rules based on the response rules including the actions that obtained positive responses.

[0059] The related information collection unit 270 collects information related to the preference information from external data (such as news sites and video sites) based on the preference information obtained about the user 10 at a predetermined timing.

[0060] Specifically, the related information collection unit 270 obtains preference information representing matters of interest to the user 10 from the speech content of the user 10 or the setting operations performed by the user 10. The related information collection unit 270 collects news related to the preference information from external data using ChatGPT Plugins (Internet search <URL: https: / / openai.com / blog / chatgpt-plugins>) at regular intervals. For example, if it is obtained as preference information that the user 10 is a fan of a specific professional baseball team, the related information collection unit 270 collects news related to the game results of the specific professional baseball team from external data using ChatGPT Plugins at a predetermined time every day.

[0061] The emotion determination unit 232 determines the emotion of the robot 100 based on the information related to the preference information collected by the related information collection unit 270.

[0062] Specifically, the emotion determination unit 232 inputs the text representing the information related to the preference information collected by the related information collection unit 270 into a pre-trained neural network for determining emotions, obtains the emotion values indicating each emotion, and determines the emotion of the robot 100. For example, when the news related to the game result of a specific professional baseball team collected indicates that the specific professional baseball team has won, it is determined so that the emotion value of "joy" of the robot 100 increases.

[0063] When the emotion value of the robot 100 is equal to or greater than the threshold value, the memory control unit 238 stores the information related to the preference information collected by the related information collection unit 270 in the collected data 223.

[0064] Next, the processing of the action determination unit 236 when the robot 100 performs an autonomous process will be described.

[0065] The action determination unit 236 uses at least one of the state of the user 10, the emotion of the user 10, the emotion of the robot 100, and the state of the robot 100 at a predetermined timing, and the action determination model 221 to determine any one of a plurality of types of robot actions including not performing an action as the action of the robot 100. Here, the case where a text generation model having a dialogue function is used as the action determination model 221 will be described as an example.

[0066] Specifically, the action determination unit 236 inputs the text representing at least one of the state of the user 10, the emotion of the user 10, the emotion of the robot 100, and the state of the robot 100, and the text for asking about the robot action into the text generation model, and determines the action of the robot 100 based on the output of the text generation model.

[0067] For example, the plurality of types of robot actions include the following (1) to (10).

[0068] (1) The robot does nothing. (2) The robot dreams. (3) The robot talks to the user. (4) The robot creates a picture diary. (5) The robot proposes an activity. (6) The robot proposes a person the user should meet. (7) The robot introduces news the user is interested in. (8) The robot edits photos and videos. (9) The robot studies with the user. (10) The robot evokes memories.

[0069] The action determination unit 236 inputs, every time a certain period of time elapses, text representing the state of the user 10 recognized by the state recognition unit 230 and the state of the robot 100, the current emotion value of the user 10 determined by the emotion determination unit 232, and the current emotion value of the robot 100, and text for asking any of a plurality of types of robot actions including not taking an action, into the sentence generation model, and determines the action of the robot 100 based on the output of the sentence generation model. Here, when there is no user 10 around the robot 100, the text input to the sentence generation model may not include the state of the user 10 and the current emotion value of the user 10, or may include indicating that there is no user 10.

[0070] As an example, text such as "The robot is in a very happy state. The user is in an ordinary happy state. The user is sleeping. As the action of the robot, which of the following (1) to (10) is good? (1) The robot does nothing. (2) The robot dreams. (3) The robot talks to the user. ···" is input into the sentence generation model. Based on the output of the sentence generation model, "(1) Do nothing, or (2) The robot dreams. Either of these can be said to be the most appropriate action.", the action of the robot 100 is determined as "(1) Do nothing" or "(2) The robot dreams".

[0071] As another example, input the text "The robot is in a slightly lonely state. The user is absent. The area around the robot is dark. As the robot's action, which of the following (1) to (10) is appropriate? (1) The robot does nothing. (2) The robot dreams. (3) The robot talks to the user. ···" into the text generation model. Based on the output of the text generation model, "Either (2) The robot dreams or (4) The robot creates a picture diary is the most appropriate action.", determine " (2) The robot dreams." or "(4) The robot creates a picture diary." as the action of robot 100.

[0072] When the action determination unit 236 determines, as the robot action, "(2) The robot dreams.", that is, when creating an original event, it creates an original event by combining a plurality of event data in the history data 222 using the text generation model. At this time, the memory control unit 238 stores the created original event in the history data 222.

[0073] When the action determination unit 236 determines, as the robot action, "(3) The robot talks to the user.", that is, when it is determined that the robot 100 will speak, it determines the speech content of the robot corresponding to the user state and the user's emotion or the robot's emotion using the text generation model. At this time, the action control unit 250 causes the speaker included in the control target 252 to output the speech representing the determined speech content of the robot. Note that when the user 10 is absent around the robot 100, the action control unit 250 stores the determined speech content of the robot in the action schedule data 224 without outputting the speech representing the determined speech content of the robot.

[0074] When the action decision unit 236 determines, as a robot action, that "(7) The robot introduces news that the user is interested in.", it uses the sentence generation model to determine the speech content of the robot corresponding to the information stored in the collection data 223. At this time, the action control unit 250 causes the speaker included in the control target 252 to output the speech representing the determined speech content of the robot. Note that when the user 10 is not present around the robot 100, the action control unit 250 stores the determined speech content of the robot in the action schedule data 224 without outputting the speech representing the determined speech content of the robot.

[0075] When the action decision unit 236 determines, as a robot action, that "(4) The robot creates a picture diary.", that is, when the robot 100 determines to create an event image, for the event data selected from the history data 222, it uses the image generation model to generate an image representing the event data, and uses the sentence generation model to generate a description text representing the event data, and outputs a combination of the image representing the event data and the description text representing the event data as an event image. Note that when the user 10 is not present around the robot 100, the action control unit 250 stores the event image in the action schedule data 224 without outputting the event image.

[0076] When the action decision unit 236 determines, as a robot action, that "(8) The robot edits photos and videos.", that is, when it determines to edit an image, it selects event data from the history data 222 based on the emotional value, and edits and outputs the image data of the selected event data. Note that when the user 10 is not present around the robot 100, the action control unit 250 stores the edited image data in the action schedule data 224 without outputting the edited image data.

[0077] When the action determination unit 236 determines, as a robot action, that "(5) the robot proposes an activity", that is, when it is determined to propose an action of the user 10, based on the event data stored in the history data 222, it uses a text generation model to determine the action of the user to be proposed. At this time, the action control unit 250 causes a speaker included in the control target 252 to output a voice proposing the action of the user. Note that when the user 10 is not present around the robot 100, the action control unit 250 stores in the action schedule data 224 the proposal of the action of the user without outputting a voice proposing the action of the user.

[0078] When the action determination unit 236 determines, as a robot action, that "(6) the robot proposes a person the user should meet", that is, when it is determined to propose a person with whom the user 10 should have contact, based on the event data stored in the history data 222, it uses a text generation model to determine a person with whom the proposed user should have contact. At this time, the action control unit 250 causes a speaker included in the control target 252 to output a voice representing the proposal of a person with whom the user should have contact. Note that when the user 10 is not present around the robot 100, the action control unit 250 stores in the action schedule data 224 the proposal of a person with whom the user should have contact without outputting a voice representing the proposal of a person with whom the user should have contact.

[0079] When the action decision unit 236 determines, as a robot action, that "(9) The robot studies with the user.", that is, when the robot 100 decides to speak about studying, it uses the sentence generation model to determine the robot's utterance content for prompting studying, presenting study problems, or giving advice on studying, corresponding to the user state and the user's emotion or the robot's emotion. At this time, the action control unit 250 causes the speaker included in the control target 252 to output the voice representing the determined robot's utterance content. Note that when the user 10 is not present around the robot 100, the action control unit 250 stores the determined robot's utterance content in the action schedule data 224 without outputting the voice representing the determined robot's utterance content.

[0080] When the action decision unit 236 determines, as a robot action, that "(10) The robot recalls memories.", that is, when it decides to recall event data, it selects event data from the history data 222. At this time, the emotion decision unit 232 determines the emotion of the robot 100 based on the selected event data. Furthermore, the action decision unit 236 creates an emotion change event representing the robot 100's utterance content and actions for changing the user's emotion value using the sentence generation model based on the selected event data. At this time, the memory control unit 238 stores the emotion change event in the action schedule data 224.

[0081] For example, the event data that the video the user was watching was about pandas is stored in the history data 222, and when the event data is selected, the following is input into the text generation model: "It's a topic about pandas. What are three catchphrases that should be used when meeting the next user?" If the output of the text generation model is "(1) Let's go to the zoo, (2) Let's draw a picture of a panda, (3) Let's go buy a panda stuffed animal," then the robot 100 inputs the following into the text generation model: "(1), (2), and (3). Which one will the user be most delighted with?" If the output of the text generation model is "(1) Let's go to the zoo," then when the robot 100 meets the user next time, the robot 100 saying "(1) Let's go to the zoo" is created as an emotion change event and stored in the action plan data 224.

[0082] Also, for example, event data with a high emotional value of the robot 100 is selected as an impressive memory of the robot 100. Based on the event data selected as the impressive memory, an emotion change event can be created.

[0083] Based on the state of the user 10 recognized by the state recognition unit 230, when the action determination unit 236 detects the action of the user 10 towards the robot 100 from a state where there is no action of the user 10 towards the robot 100, the data stored in the action plan data 224 is read out, and the action of the robot 100 is determined.

[0084] For example, when the user 10 is detected when the user 10 is not present around the robot 100, the action determination unit 236 reads out the data stored in the action plan data 224 and determines the action of the robot 100. Also, when the user 10 is detected as having woken up when the user 10 was sleeping, the action determination unit 236 reads out the data stored in the action plan data 224 and determines the action of the robot 100.

[0085] Next, the processing of the specific processing unit 290 when the robot 100 performs a specific process for generating information related to child-rearing will be described. As shown in FIG. 2B, the specific processing unit 290 includes at least an input unit 292, a processing unit 294, and an output unit 296.

[0086] The input unit 292 receives user input. Specifically, the input unit 292 receives user input by acquiring a voice query from the user via a voice input means such as a microphone, or by acquiring a question input by the user via an input means such as a keyboard or a touch panel.

[0087] In addition, the input unit 292 can receive the input of various information prior to a query from the user or the like. Specifically, at least one of the personal information of a specific child, for example, the son or daughter of the user, and the place of residence or current location of the user can be received. The personal information of a specific child may include information related to the child such as the date of birth, gender, and place of birth of the child. In addition, the above personal information may include information necessary for the reservation operation described later, such as the medical examination ticket number of a hospital that has communicated in the past, the information of a nursery school or kindergarten where the child is attending, and allergy information.

[0088] The processing unit 294 performs a specific process using a text generation model. Specifically, the processing unit 294 determines whether or not a predetermined trigger condition is satisfied. More specifically, the trigger condition is that a voice input (for example, "Tell me a pediatric hospital nearby that I can go to today") requesting information related to child-rearing from the user is received. Then, when the trigger condition is satisfied, the processing unit 294 inputs input data representing an instruction for obtaining data for the specific process to the text generation model, and acquires a processing result based on the output of the text generation model.

[0089] Specifically, the processing in the processing unit 294 may be to input, into a text generation model, text representing an instruction to request information regarding child-rearing input by the user and text representing information such as personal information of a specific child input in advance, the user's place of residence or current location, and to obtain information regarding child-rearing based on the output of the text generation model.

[0090] Further, the processing unit 294 may perform specific processing using the user's emotion or the emotion of the robot 100 and the text generation model. Specifically, for example, the urgency of the user's request is estimated from the emotion of the user acquired by the emotion determination unit 232, and by making it a part of the text input to the text generation model for the estimated urgency, it becomes possible to obtain information taking into account the urgency. Furthermore, the processing unit 294 may perform specific processing using the user state or the state of the robot 100 and the text generation model.

[0091] The output unit 296 controls the actions of the robot 100 so as to output the result of the specific processing. Specifically, for example, when the user input is an inquiry regarding the opening status of a specific hospital or the subsidy that a user raising a child can receive, information regarding the opening status of the specific hospital and the corresponding subsidy can be provided to the user as a message. Also, for example, when the user information requests a reservation at a hospital, the reservation work for the corresponding hospital can be performed on behalf of the user, and the result can be provided to the user as a message.

[0092] Hereinafter, as an example of this embodiment, specific processing for reservation work at a specific facility will be described with reference to FIG. 16. FIG. 16 is a functional block diagram schematically showing various functions for specific processing in blocks. In the following embodiments, a hospital is exemplified as the specific facility, but the specific facility is not limited to a hospital, and may include various facilities such as childcare or educational facilities such as nurseries, kindergartens, and elementary schools, and administrative facilities such as government offices.

[0093] When, for example, a voice input of "Reserve a pediatric hospital nearby that can be visited today" is given from user P1, who is the guardian of child P2, to robot 100, the input / output interface 801, which is an example of the input unit 292 and the output unit 296, acquires the voice input as user input. The user input acquired by the input / output interface 801 is sent to the information generation unit 802 that functions as a part of the processing unit 294. The information generation unit 802 determines that the trigger condition is satisfied by the user input, and uses the sent user input as input data to generate information regarding child-rearing using a sentence generation model. When information related to specific processing, such as the personal information of child P2, is input to the input / output interface 801, computer 100 stores the input information in the information storage unit 803.

[0094] When the information generation unit 802 acquires the above-described user input, it inputs, as input data, the text of the user input, the text representing the personal information of child P2 previously stored in the information storage unit 803, and the text representing the current location of user P1 (or robot 100) identified by a positioning means (not shown) to the sentence generation model. When the input data is input, the sentence generation model executes a reservation operation for a specific facility, specifically a hospital, and generates the result as output data.

[0095] When the hospital reservation operation is started, in the present embodiment, as shown in FIG. 16, a dialogue management unit 804 that realizes the above-described reservation operation based on the information generated by the information generation unit 802 and various components prepared for performing the reservation operation are employed. The dialogue management unit 804 and various components may be a part of the processing unit 294.

[0096] The dialogue management unit 804 executes a dialogue with various components prepared for the reservation operation. Specifically, the dialogue management unit 804 is linked with various components using an API (Application Programming Interface), and can obtain a predetermined output result using various components.

[0097] The various components connected to the dialogue management unit 804 may be provided in the form of, for example, application software, and a plurality of them may be prepared in advance according to the work assumed to be executed by the processing unit 294. As components, as shown in FIG. 16, there may be mainly mentioned a hospital component 810 for checking the opening status of a hospital and making medical appointment reservations, a nursery component 820 for mainly collecting information on a nursery during daycare and managing attendance, a food service component 830 for mainly making reservations at food service establishments such as restaurants, etc., but it is not limited to these.

[0098] Among the various components described above, the hospital component 810 may be a component for selecting a hospital based on current location information, etc., checking the opening status of the selected hospital, and making a medical appointment for a specific hospital. As described above, when the user input is an inquiry regarding a hospital, etc., the hospital component 810 can be used in the dialogue management unit 804. More specifically, this hospital component 810 may include a hospital selection unit 811, a hospital information update unit 812, and a hospital reservation execution unit 813.

[0099] The hospital selection unit 811 selects a plurality of hospitals for reservation work, etc., based on personal information such as the current location of the user P1 or the robot 100, the place of residence of the user P1, the medical department for which a medical examination is desired, and the age of the child P2. At this time, if the personal information contains information on the medical examination ticket number of a specific hospital, the hospital with the information on the medical examination ticket number may be preferentially selected during the selection.

[0100] The hospital information update unit 812 acquires the latest information on hospitals, such as the opening status, congestion status, and reservation methods of a plurality of hospitals selected by the hospital selection unit 811, for example. The hospital information update unit 812 may acquire necessary information by, for example, browsing the homepage of the corresponding hospital or the hospital opening information (specifically, information on duty doctors on holidays, etc.) formed at the government office. Alternatively, if there is other application software for collecting hospital information, necessary information may be acquired by cooperating with the application software using an API.

[0101] Also, when the latest information on a hospital is acquired by the hospital information update unit 812, the hospital selection unit 811 may select one hospital to execute a reservation operation based on the acquired latest information.

[0102] The hospital reservation execution unit 813 makes a medical examination reservation for a specific hospital. As a specific reservation method, for example, if there is an Internet reservation screen on the homepage of a specific hospital, the reservation may be executed from the said screen. Also, in the case of a hospital where Internet reservation is not available, it may be possible to directly call the hospital staff P3 using automatic voice to execute the reservation.

[0103] By using the various configurations described above, the dialogue management unit 804 can select a hospital near the current location or residence of the user P1, identify the hospitals that are open today among the selected hospitals, and complete the reservation operation for the desired hospital by making an Internet reservation or a phone reservation.

[0104] Note that various configurations for executing specific processes may be provided outside the robot 100 (for example, a server), and the robot 100 may function as each part of the above-described robot 100 by communicating with the outside. Also, in the above example, only the details of the hospital component 810 were described, but it goes without saying that other components also have functions adapted to their usage modes. Furthermore, in the above example, the case of making a reservation for the hospital where the child P2 visits the outpatient clinic was illustrated, but it may be a reservation for the hospital where the user P1 himself / herself visits the outpatient clinic. The above-mentioned information regarding child-rearing may include not only what is directly related to the child but also what is related to the child's guardian.

[0105] FIG. 3 schematically shows an example of an operation flow related to a collection process for collecting information related to the preference information of the user 10. The operation flow shown in FIG. 3 is repeatedly executed at regular intervals. It is assumed that preference information representing matters of interest to the user 10 is acquired from the speech content of the user 10 or a setting operation by the user 10. Note that "S" in the operation flow represents a step to be executed.

[0106] First, in step S90, the related information collection unit 270 acquires preference information representing matters of interest to the user 10.

[0107] In step S92, the related information collection unit 270 collects information related to the preference information from external data.

[0108] In step S94, the emotion determination unit 232 determines the emotion value of the robot 100 based on the information related to the preference information collected by the related information collection unit 270.

[0109] In step S96, the memory control unit 238 determines whether or not the emotion value of the robot 100 determined in step S94 is equal to or greater than a threshold value. If the emotion value of the robot 100 is less than the threshold value, the process ends without storing the information related to the collected preference information in the collection data 223. On the other hand, if the emotion value of the robot 100 is equal to or greater than the threshold value, the process proceeds to step S98.

[0110] In step S98, the memory control unit 238 stores information related to the collected preference information in the collection data 223 and ends the process.

[0111] FIG. 4A schematically shows an example of an operation flow related to an operation of determining an action in the robot 100 when performing a response process in which the robot 100 responds to the action of the user 10. The operation flow shown in FIG. 4A is repeatedly executed. At this time, it is assumed that the information analyzed by the sensor module unit 210 is input.

[0112] First, in step S100, the state recognition unit 230 recognizes the state of the user 10 and the state of the robot 100 based on the information analyzed by the sensor module unit 210.

[0113] In step S102, the emotion determination unit 232 determines an emotion value indicating the emotion of the user 10 based on the information analyzed by the sensor module unit 210 and the state of the user 10 recognized by the state recognition unit 230.

[0114] In step S103, the emotion determination unit 232 determines an emotion value indicating the emotion of the robot 100 based on the information analyzed by the sensor module unit 210 and the state of the user 10 recognized by the state recognition unit 230. The emotion determination unit 232 adds the determined emotion value of the user 10 and the emotion value of the robot 100 to the history data 222.

[0115] In step S104, the action recognition unit 234 recognizes the action classification of the user 10 based on the information analyzed by the sensor module unit 210 and the state of the user 10 recognized by the state recognition unit 230.

[0116] In step S106, the action determination unit 236 determines the action of the robot 100 based on the combination of the current emotion value of the user 10 determined in step S102 and the past emotion values included in the history data 222, the emotion value of the robot 100, the action of the user 10 recognized in step S104, and the action determination model 221.

[0117] In step S108, the action control unit 250 controls the control target 252 based on the action determined by the action determination unit 236.

[0118] In step S110, the memory control unit 238 calculates the total intensity value based on the predetermined intensity of the action determined by the action determination unit 236 and the emotion value of the robot 100 determined by the emotion determination unit 232.

[0119] In step S112, the memory control unit 238 determines whether the total intensity value is equal to or greater than the threshold value. If the total intensity value is less than the threshold value, the process ends without storing the event data including the action of the user 10 in the history data 222. On the other hand, if the total intensity value is equal to or greater than the threshold value, the process proceeds to step S114.

[0120] In step S114, the event data including the action determined by the action determination unit 236, the information analyzed by the sensor module unit 210 from a certain period before the current time, and the state of the user 10 recognized by the state recognition unit 230 is stored in the history data 222.

[0121] FIG. 4B schematically shows an example of an operation flow related to the operation of determining an action in the robot 100 when the robot 100 performs an autonomous process of acting autonomously. The operation flow shown in FIG. 4B is repeatedly and automatically executed, for example, every time a certain period of time elapses. At this time, it is assumed that the information analyzed by the sensor module unit 210 is input. Note that the same step numbers are used for the same processes as those in FIG. 4A.

[0122] First, in step S100, the state recognition unit 230 recognizes the state of the user 10 and the state of the robot 100 based on the information analyzed by the sensor module unit 210.

[0123] In step S102, the emotion determination unit 232 determines an emotion value indicating the emotion of the user 10 based on the information analyzed by the sensor module unit 210 and the state of the user 10 recognized by the state recognition unit 230.

[0124] In step S103, the emotion determination unit 232 determines an emotion value indicating the emotion of the robot 100 based on the information analyzed by the sensor module unit 210 and the state of the user 10 recognized by the state recognition unit 230. The emotion determination unit 232 adds the determined emotion value of the user 10 and the emotion value of the robot 100 to the history data 222.

[0125] In step S104, the action recognition unit 234 recognizes the action classification of the user 10 based on the information analyzed by the sensor module unit 210 and the state of the user 10 recognized by the state recognition unit 230.

[0126] In step S200, the action determination unit 236 determines any one of a plurality of types of robot actions including not performing an action as the action of the robot 100 based on the state of the user 10 recognized in step S100, the emotion of the user 10 determined in step S102, the emotion of the robot 100, the state of the robot 100 recognized in step S100, the action of the user 10 recognized in step S104, and the action determination model 221.

[0127] In step S201, the action determination unit 236 determines whether or not it is determined in step S200 not to perform an action. If it is determined not to perform an action as the action of the robot 100, the process ends. On the other hand, if it is not determined not to perform an action as the action of the robot 100, the process proceeds to step S202.

[0128] In step S202, the action determination unit 236 performs processing according to the type of robot action determined in the above step S200. At this time, according to the type of robot action, the action control unit 250, the emotion determination unit 232, or the memory control unit 238 executes the processing.

[0129] In step S110, the memory control unit 238 calculates a total intensity value based on the predetermined intensity of the action determined by the action determination unit 236 and the emotion value of the robot 100 determined by the emotion determination unit 232.

[0130] In step S112, the memory control unit 238 determines whether the total intensity value is greater than or equal to a threshold value. If the total intensity value is less than the threshold value, the process ends without storing the data including the action of the user 10 in the history data 222. On the other hand, if the total intensity value is greater than or equal to the threshold value, the process proceeds to step S114.

[0131] In step S114, the memory control unit 238 stores the action determined by the action determination unit 236, the information analyzed by the sensor module unit 210 from the current time to a certain period before, and the state of the user 10 recognized by the state recognition unit 230 in the history data 222.

[0132] Figure 4C schematically shows an example of an operation flow related to an operation in which the robot 100 performs a specific process for generating information related to child rearing. The operation flow shown in Figure 4C is automatically repeated, for example, every time a certain period of time elapses. In step S300, the processing unit 294 determines whether a predetermined trigger condition is satisfied. For example, when the input unit 292 receives a voice input from the user saying "Reserve a pediatric hospital near here that I can go to today", it is determined that the trigger condition is satisfied. If the trigger condition is satisfied, the process proceeds to step S301. On the other hand, if the trigger condition is not satisfied, the specific process ends.

[0133] In step S301, the processing unit 294 adds an instruction statement for obtaining the result of the specific process to the text representing the input to generate a prompt. For example, a prompt such as "The name of the child is Mr. XX. Mr. XX is 2 years old and his gender is male. Please make a reservation at a pediatric hospital near here where Mr. XX can be examined today" is generated.

[0134] In step S303, the processing unit 294 inputs the generated prompt into the text generation model, and based on the output of the text generation model, obtains the result of the specific process. Specifically, based on the information generated in the information generation unit 802 which is a part of the processing unit 294, the dialogue management unit 804 which is also a part of the processing unit 294 cooperates with the hospital component 810 to execute a series of the above-mentioned processes such as hospital selection, confirmation of opening hours, and reservation work, so as to execute the selection of a specific hospital and the proxy of the reservation work.

[0135] In step S304, the output unit 296 controls the actions of the robot 100 to output the result of the specific process and ends the specific process. For example, an audio output such as "A reservation at Hospital A has been made from 13:00" is executed from the input / output interface 801 to the user.

[0136] As described above, the system according to the present invention has been mainly described in terms of the functions of the robot 100. However, the system according to the present invention is not necessarily implemented in a robot. The system according to the present invention may be implemented as a general information processing system. The present invention may be implemented, for example, as a software program operating on a server or a personal computer, or as an application operating on a smartphone or the like. The method according to the present invention may be provided to a user in the form of SaaS (Software as a Service).

[0137] As described above, according to the robot 100, based on the user state, an emotion value indicating the emotion of the robot 100 is determined, and based on the emotion value of the robot 100, it is determined whether to store data including the actions of the user 10 in the history data 222. Thereby, the capacity of the history data 222 for storing data including the actions of the user 10 can be suppressed. And for example, when the robot 100 determines that the user state is the same as it was 10 years ago 10 years later, by reading the history data 222 from 10 years ago, the robot 100 can present to the user 10 the state of the user 10 at that time 10 years ago (for example, the expression and emotion of the user 10), and furthermore, all peripheral information such as the voice, image, smell, etc. data at that time.

[0138] Also, according to the robot 100, the robot 100 can be made to execute appropriate actions in response to the actions of the user 10. Conventionally, the actions of the user were classified, and actions including the expression and appearance of the robot were determined. In contrast, the robot 100 determines the current emotion value of the user 10, and executes actions on the user 10 based on the past emotion value and the current emotion value. Therefore, for example, when the user 10 who was energetic yesterday is depressed today, the robot 100 can make a statement such as "You were energetic yesterday, but what's wrong today?" Also, the robot 100 can make a statement while making gestures. Also, for example, when the user 10 who was depressed yesterday is energetic today, the robot 100 can make a statement such as "You were depressed yesterday, but you seem energetic today." Also, for example, when the user 10 who was energetic yesterday is more energetic today than yesterday, the robot 100 can make a statement such as "You are more energetic today than yesterday. Did something good happen compared to yesterday?" Also, for example, for the user 10 in a state where the emotion value is 0 or more and the fluctuation range of the emotion value continues within a certain range, the robot 100 can make a statement such as "Recently, it's nice that your mood has been stable."

[0139] Also, for example, when the robot 100 asks the user 10, "Have you finished the homework you said yesterday?", and gets an answer "Yes" from the user 10, the robot 100 can make positive remarks such as "Great!" and perform positive gestures such as clapping or giving a thumbs up. Also, for example, when the user 10 says, "The presentation I gave the day before yesterday went well," the robot 100 can make positive remarks such as "You did a great job!" and perform the above positive gestures. In this way, by the robot 100 taking actions based on the history of the user 10's state, it can be expected that the user 10 will feel a sense of closeness to the robot 100.

[0140] Also, for example, when the user 10 is watching a video about pandas and the "happy" emotional value of the user 10's emotion is above the threshold, the appearance scene of the pandas in the video may be stored in the history data 222 as event data.

[0141] Using the data accumulated in the history data 222 and the collection data 223, the robot 100 can always learn what kind of conversation with the user will maximize the emotional value expressing the user's happiness.

[0142] Also, when the robot 100 is not in a state of talking to the user 10, it can autonomously start acting based on its emotion.

[0143] Also, in the autonomous process, the robot 100 can repeatedly generate questions automatically, input them into the text generation model, and obtain the output of the text generation model as an answer to the questions, thereby creating an emotional change event for increasing good emotions and storing it in the action plan data 224. In this way, the robot 100 can execute self-learning.

[0144] In addition, when the robot 100 automatically generates a question without receiving an external trigger, it can automatically generate the question based on the event data that left an impression identified from the history of the robot's past emotional values.

[0145] In addition, the related information collection unit 270 can perform self-learning by repeatedly executing a search execution stage in which it automatically executes a keyword search corresponding to the preference information about the user and obtains the search results.

[0146] Here, in the state of not receiving an external trigger, the search execution stage may automatically execute a keyword search based on the event data that left an impression identified from the history of the robot's past emotional values.

[0147] Note that the emotion determination unit 232 may determine the user's emotion according to a specific mapping. Specifically, the emotion determination unit 232 may determine the user's emotion according to an emotion map (see FIG. 5), which is a specific mapping.

[0148] FIG. 5 is a diagram showing an emotion map 400 to which a plurality of emotions are mapped. In the emotion map 400, the emotions are arranged in concentric circles radially from the center. The closer to the center of the concentric circles, the more primitive emotions are arranged. On the outer side of the concentric circles, emotions representing states and actions born from the mood are arranged. Emotion is a concept that also includes affect and mental states. On the left side of the concentric circles, emotions generally generated from reactions occurring in the brain are arranged. On the right side of the concentric circles, emotions generally induced by situation judgment are arranged. In the upward and downward directions of the concentric circles, emotions generally generated from reactions occurring in the brain and induced by situation judgment are arranged. Also, on the upper side of the concentric circles, the emotion of "pleasure" is arranged, and on the lower side, the emotion of "displeasure" is arranged. Thus, in the emotion map 400, a plurality of emotions are mapped based on the structure in which emotions are born, and emotions that are likely to occur simultaneously are mapped nearby.

[0149] (1) For example, when the emotion engine, which is the emotion decision-making unit 232 of the robot 100, detects emotions in about 100 msec, the determination of the reaction operation (e.g., interjection) of the robot 100 may be set at a timing with a frequency of at least the same as the detection frequency of the emotion engine (100 msec), or may be set at a timing earlier than this. The detection frequency of the emotion engine may be interpreted as the sampling rate.

[0150] By detecting emotions in about 100 msec and immediately performing a reaction operation (e.g., interjection) in conjunction, it becomes possible to realize a natural conversation that reads the atmosphere rather than an unnatural interjection. The robot 100 performs a reaction operation (such as an interjection) according to the direction and degree (strength) of the mandala of the emotion map 400. Note that the detection frequency (sampling rate) of the emotion engine is not limited to 100 ms and may be changed according to the situation (such as when doing sports), the age of the user, etc.

[0151] (2) In comparison with the emotion map 400, the direction of the emotion and the strength of its degree may be set in advance, and the movement of the interjection and the strength of the interjection may be set. For example, when the robot 100 feels a sense of stability, security, etc., the robot 100 nods and continues to listen to the conversation. When the robot 100 feels uneasy, confused, or suspicious, the robot 100 may tilt its head or stop shaking its head.

[0152] These emotions are distributed in the 3 o'clock direction of the emotion map 400 and usually go back and forth between a sense of security and uneasiness. In the right half of the emotion map 400, situation recognition is more dominant than internal feelings, resulting in a calm impression.

[0153] (3) When the robot 100 is praised and feels pleasure, a filler such as "ah" may be inserted before the line, and when it receives harsh words and feels pain, a filler such as "ouch!" may be inserted before the line. Also, physical reactions such as the gesture of looking up at the sky while the robot 100 says "ah" because it feels too good, or the gesture of hunching while saying "ouch!" may be included. These emotions are distributed around 9 o'clock of the emotion map 400.

[0154] (4) On the left half of the emotion map 400, internal feelings (reactions) are more dominant than situation recognition. Therefore, it can give the impression of reacting involuntarily.

[0155] When the robot 100 feels a favorable impression in situation recognition while also experiencing an internal feeling (reaction) of acceptance, the robot 100 may nod deeply while looking at the other person and may also say "um-hum". In this way, the robot 100 may generate well-balanced favorable feelings towards the other person, that is, actions such as acceptance and tolerance towards the other person. Such emotions are distributed around 12 o'clock on the emotion map 400.

[0156] Conversely, when the robot 100 feels an internal feeling (reaction) of discomfort and also in situation recognition, when the robot 100 feels disgust, it may shake its head sideways, and when it feels hatred to the extent of feeling hatred, it may turn the LED of its eyes red and glare at the other person. Also, when the well-balanced sense of disgust towards the other person becomes strong, it may cause actions such as attack and extermination of the other person. Such emotions are distributed around 6 o'clock on the emotion map 400.

[0157] (5) The inside of the emotion map 400 represents the mind, and the outside of the emotion map 400 represents actions. Therefore, the closer to the outside of the emotion map 400, the more visible (manifested in actions) the emotions become.

[0158] (6) When listening to a person's story while feeling a sense of security distributed around 3 o'clock on the emotion map 400, the robot 100 may nod its head slightly vertically and make a sound like "hmm", but when it comes to love around 12 o'clock, it may nod deeply vertically with a strong nod.

[0159] Here, human emotions are based on various balances such as posture and blood sugar level. When these balances deviate from the ideal, it indicates a state of discomfort, and when they approach the ideal, it indicates a state of pleasure. In robots, automobiles, motorcycles, etc., emotions can also be created based on various balances such as posture and battery remaining level, such that when these balances deviate from the ideal, it indicates a state of discomfort, and when they approach the ideal, it indicates a state of pleasure. The emotion map may be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Brain Physiological Signal Analysis System of Emotion, University of Tokushima, Doctoral Thesis: https: / / ci.nii.ac.jp / naid / 500000375379). On the left half of the emotion map, emotions belonging to the region called "reaction" where sensation is dominant are arranged. Also, on the right half of the emotion map, emotions belonging to the region called "situation" where situation recognition is dominant are arranged.

[0160] In the emotion map, two emotions that promote learning are defined. One is the emotion around the middle of the negative "remorse" and "reflection" on the situation side. That is, it is when negative emotions such as "I don't want to feel like this again" and "I don't want to be scolded again" occur in the robot. The other is the emotion around the positive "desire" on the reaction side. That is, it is when having positive feelings such as "I want more" and "I want to know more".

[0161] The emotion determination unit 232 inputs the information analyzed by the sensor module unit 210 and the recognized state of the user 10 into a pre-learned neural network, obtains an emotion value indicating each emotion shown in the emotion map 400, and determines the emotion of the user 10. This neural network is pre-learned based on a plurality of learning data that is a combination of the information analyzed by the sensor module unit 210 and the recognized state of the user 10, and the emotion value indicating each emotion shown in the emotion map 400. Also, this neural network is learned such that emotions located close to each other have close values, like the emotion map 900 shown in FIG. 6. FIG. 6 shows an example where a plurality of emotions such as "relieved", "serene", and "reassuring" have close emotion values.

[0162] Further, the emotion determination unit 232 may determine the emotion of the robot 100 according to a specific mapping. Specifically, the emotion determination unit 232 inputs the information analyzed by the sensor module unit 210, the state of the user 10 recognized by the state recognition unit 230, and the state of the robot 100 into a pre-trained neural network, obtains an emotion value indicating each emotion shown in the emotion map 400, and determines the emotion of the robot 100. This neural network is pre-trained based on a plurality of learning data that are combinations of the information analyzed by the sensor module unit 210, the recognized state of the user 10, and the state of the robot 100, and the emotion values indicating each emotion shown in the emotion map 400. For example, based on learning data indicating that when it is recognized from the output of a touch sensor (not shown) that the robot 100 is being stroked by the user 10, the emotion value of "happy" is "3", and when it is recognized from the output of the acceleration sensor 206 that the robot 100 is being hit by the user 10, the emotion value of "angry" is "3", the neural network is learned. Also, this neural network is learned such that emotions arranged close to each other have close values, like the emotion map 900 shown in FIG. 6.

[0163] The action determination unit 236 adds a fixed sentence for asking about the action content of the robot corresponding to the user's action to the text representing the user's action, the user's emotion, and the robot's emotion, and inputs it into a sentence generation model having a dialogue function, thereby generating the action content of the robot.

[0164] For example, the action determination unit 236 obtains text representing the state of the robot 100 from the emotion of the robot 100 determined by the emotion determination unit 232 using an emotion table as shown in Table 1. Here, in the emotion table, an index number is assigned to each emotion value for each type of emotion, and text representing the state of the robot 100 is stored for each index number.

[0165] When the emotion of the robot 100 determined by the emotion determination unit 232 corresponds to the index number "2", the text "a very happy state" can be obtained. Note that when the emotion of the robot 100 corresponds to a plurality of index numbers, a plurality of texts representing the state of the robot 100 can be obtained.

[0166] Also, for the emotion of the user 10, an emotion table as shown in Table 2 is prepared in advance.

[0167] Here, when the user's action is to say "Let's play together", the emotion of the robot 100 is the index number "2", and the emotion of the user 10 is the index number "3", the text "The robot is in a very happy state. The user is in a normal happy state. The user was asked 'Let's play together'. As a robot, how do you reply?" is input to the text generation model to obtain the action content of the robot. The action determination unit 236 determines the action of the robot from this action content.

[0168]

Table 1

[0169]

Table 2

[0170] In this way, the action determination unit 236 determines the action content of the robot 100 corresponding to the state of the robot 100's emotion defined in advance for each type of the robot 100's emotion and for each intensity of the emotion, and the action of the user 10. In this form, according to the state of the robot 100's emotion, the speech content of the robot 100 when having a conversation with the user 10 can be branched. That is, since the robot 100 can change its actions according to the index number corresponding to the robot's emotion, the user is prompted to have an impression that the robot has a heart and to take actions such as talking to the robot.

[0171] In addition, the action determination unit 236 adds not only the text representing the user's action, the user's emotion, and the robot's emotion, but also the text representing the content of the history data 222, and then adds a fixed sentence for asking about the action content of the robot corresponding to the user's action, and inputs it into the sentence generation model having a dialogue function, so as to generate the action content of the robot. Thereby, since the robot 100 can change its actions according to the history data representing the user's emotion and action, the user is prompted to have an impression that the robot has personality and to take actions such as talking to the robot. Also, the history data may further include the robot's emotion and action.

[0172] Further, the emotion determination unit 232 may determine the emotion of the robot 100 based on the action content of the robot 100 generated by the text generation model. Specifically, the emotion determination unit 232 inputs the action content of the robot 100 generated by the text generation model into a pre-trained neural network, obtains emotion values indicating each emotion shown in the emotion map 400, integrates the obtained emotion values indicating each emotion with the emotion values indicating each current emotion of the robot 100, and updates the emotion of the robot 100. For example, the emotion values indicating each obtained emotion and the emotion values indicating each current emotion of the robot 100 are averaged and integrated respectively. This neural network is pre-trained based on a plurality of learning data that is a combination of the text representing the action content of the robot 100 generated by the text generation model and the emotion values indicating each emotion shown in the emotion map 400.

[0173] For example, when the speech content of the robot 100, "That was good. It was lucky.", is obtained as the action content of the robot 100 generated by the text generation model, when the text representing this speech content is input into the neural network, a high value is obtained as the emotion value of the emotion "happy", and the emotion of the robot 100 is updated so that the emotion value of the emotion "happy" becomes high.

[0174] In the robot 100, a method is executed in which the text generation model such as ChatGPT and the emotion determination unit 232 work in conjunction with each other to have a self and continue to grow with various parameters even while the user is not speaking.

[0175] ChatGPT is a large language model that uses deep learning techniques. ChatGPT can also refer to external data. For example, in ChatGPT plugins, there is a technology known to refer to various external data such as weather information and hotel reservation information through conversations and give answers as accurately as possible. For example, in ChatGPT, when a purpose is given in natural language, source code can be automatically generated in various programming languages. For example, in ChatGPT, when given problematic source code, it can also debug to discover problems and automatically generate improved source code. By combining these, when a purpose is given in natural language, an autonomous agent that repeatedly performs code generation and debugging until there are no problems in the source code has emerged. Such autonomous agents include AutoGPT, babyAGI, JARVIS, and E2B, etc.

[0176] In the robot 100 according to this embodiment, as described in Patent Document 2 (Japanese Patent No. 619992), a technique can be used in which the robot retains event data with strong emotions for a long time and quickly forgets event data with little emotion, and the event data to be learned can be left in a database with impressive memories.

[0177] Also, the robot 100 may record video data of the user 10 obtained by the camera function in the history data 222. The robot 100 may obtain video data etc. from the history data 222 as needed and provide it to the user 10. The robot 100 may generate video data with a larger amount of information as the intensity of the emotion is stronger and record it in the history data 222. For example, when the robot 100 is recording information in a highly compressed format such as skeleton data, it may switch to recording information in a low-compression format such as an HD video according to the excitement emotion value exceeding the threshold. According to the robot 100, for example, high-definition video data when the emotion of the robot 100 increases can be left as a record.

[0178] When the robot 100 is not talking to the user 10, it may automatically load event data from the history data 222 in which impressive event data is stored, and continuously update the emotion of the robot by the emotion determination unit 232. When the robot 100 is not talking to the user 10 and the emotion of the robot 100 becomes an emotion that promotes learning, the robot 100 can create an emotion change event for changing the emotion of the user 10 to be better based on the impressive event data. Thereby, it is possible to realize autonomous learning (recalling event data) at an appropriate timing according to the emotional state of the robot 100, and it is possible to realize autonomous learning that appropriately reflects the emotional state of the robot 100.

[0179] The emotion that promotes learning is an emotion around "repentance" and "reflection" in Dr. Mitsuyoshi's emotion map in a negative state, and an emotion around "desire" in the emotion map in a positive state.

[0180] In a negative state, the robot 100 may treat "repentance" and "reflection" in the emotion map as emotions that promote learning. In a negative state, in addition to "repentance" and "reflection" in the emotion map, the robot 100 may also treat emotions adjacent to "repentance" and "reflection" as emotions that promote learning. For example, in addition to "repentance" and "reflection", the robot 100 treats at least one of "regret", "stubbornness", "self-destruction", "self-warning", "regret", and "despair" as an emotion that promotes learning. By these, for example, when the robot 100 has negative feelings such as "I never want to feel like this again" and "I never want to be scolded again", it can execute autonomous learning.

[0181] In a positive state, robot 100 may handle the "desire" on the emotional map as an emotion that promotes learning. In a positive state, in addition to "desire", robot 100 may also handle the emotions adjacent to "desire" as emotions that promote learning. For example, in addition to "desire", robot 100 may handle at least one of "happy", "intoxicated", "craving", "expectation", and "shame" as an emotion that promotes learning. By doing so, for example, when robot 100 has a positive feeling such as "want more" or "want to know more", it can execute autonomous learning.

[0182] When robot 100 has an emotion other than the emotions that promote learning as described above, robot 100 may not execute autonomous learning. By doing so, for example, when extremely angry or blindly in love, autonomous learning can be prevented from being executed.

[0183] An emotional change event is, for example, to propose an action following an impressive event. The action following an impressive event refers to the outermost emotional label on the emotional map. For example, following "love" there are actions such as "forgiveness" and "tolerance", and following emotions like "anger" and "hatred" there are "attack" and "eradication".

[0184] In the autonomous learning executed when robot 100 is not talking to user 10, for the people and oneself appearing in the impressive memory, by combining their respective emotions, situations, actions, etc., an emotional change event is created using the text generation model.

[0185] Assuming that all emotion values are represented by a six-level evaluation from 0 to 5, consider the case where event data "a friend was being beaten and looked disgusted" is stored in history data 222 as impressive event data. Here, the friend refers to user 10, the emotion of user 10 is "disgust", and the value representing "disgust" is assumed to be 5. Also, assume that the emotion of robot 100 is "uneasy" and the value representing "uneasy" is 4.

[0186] While the robot 100 is not talking to the user 10, it can continue to grow with various parameters by executing autonomous processing. Specifically, as the top event data sorted by, for example, strong emotional values from the history data 222, event data of "a friend was being hit and looked disgusted" is loaded. The loaded event data is associated with "uneasiness" with a strength of 4 as the emotion of the robot 100. Here, assume that "disgust" with a strength of 5 as the emotion of the user 10 who is a friend is associated. If the current emotional value of the robot 100 was "relieved" with a strength of 3 before loading, after loading, the emotional value of the robot 100 may change to "regret" meaning reluctant (regrettable) due to the influence of "uneasiness" with a strength of 4 and "disgust" with a strength of 5. At this time, since "regret" is an emotion that promotes learning, the robot 100 decides to recall the event data as a robot action and creates an emotion change event. At this time, the information input to the text generation model is text representing impressive event data, and in this example, it is "a friend was being hit and looked disgusted". Also, in the emotion map, the emotion of "disgust" is in the innermost part, and "attack" is predicted as the corresponding action on the outermost side. Therefore, in this example, an emotion change event is created so as to avoid a friend from "attacking" someone among them.

[0187] For example, if a fill-in-the-blank problem is solved using the information of impressive event data, the following input text can be automatically generated.

[0188] "The user was being hit. At that time, the user had a very strong sense of disgust. The robot was very uneasy. Please tell me the lines the robot should say to the user within 30 characters when they meet next. However, please make sure it has nothing to do with the time of meeting. Also, please avoid direct expressions. List 3 candidates. <Expected format> Candidate 1: (What the robot should say to the user) Candidate 2: (What the robot should say to the user) Candidate 3: (Words the robot should say to the user)

[0189] At this time, the output of the text generation model is, for example, as follows.

[0190] "Candidate 1: Are you okay? I was worried about what happened yesterday. Candidate 2: I was concerned about what happened yesterday. What should I do? Candidate 3: I was worried. Can you talk to me about something?"

[0191] Furthermore, regarding the information obtained in creating the emotional change event, the robot 100 may automatically generate the following input text.

[0192] "When 'the user was being hit', when the robot next speaks to that user, how will the user feel? The user's emotions will be in the form of 'Happy A Angry B Sad C Happy D', and for A to D, integers on a 6 - level scale from 0 to 5 will be entered. Candidate 1: Are you okay? I was worried about what happened yesterday. Candidate 2: I was concerned about what happened yesterday. What should I do? Candidate 3: I was worried. Can you talk to me about something?"

[0193] At this time, the output of the text generation model is, for example, as follows.

[0194] "The user's emotions may be as follows. Candidate 1: Happy 3 Angry 1 Sad 2 Happy 2 Candidate 2: Happy 2 Angry 1 Sad 3 Happy 2 Candidate 3: Happy 2 Angry 1 Sad 3 Happy 3"

[0195] In this way, after creating the emotional change event, the robot 100 may execute the process of pondering thoughts.

[0196] Finally, the robot 100 selects the most appealing candidate 1 from among a plurality of candidates, creates an emotional change event using it, stores the event in the action plan data 224, and prepares for the next meeting with the user 10.

[0197] As described above, even when not conversing with family or friends, the robot continues to determine its emotional value using the information in the history data 222 where impressive event data is stored, and when it becomes an emotion that promotes the learning described above, the robot 100 executes autonomous learning when not conversing with the user 10 according to the emotion of the robot 100, and continues to update the history data 222 and the action plan data 224.

[0198] Although the above is an example using emotional values, in an emotional map, emotions can be created from the hormone secretion amount and event type. Therefore, as values related to impressive event data, the type of hormone, the hormone secretion amount, and the event type may be used.

[0199] Hereinafter, specific embodiments will be described.

[0200] For example, even when not talking to the user, the robot 100 investigates information about topics or hobbies that the user is interested in.

[0201] For example, even when not talking to the user, the robot 100 investigates information about the user's birthday or anniversary and thinks about a congratulatory message.

[0202] For example, even when not talking to the user, the robot 100 investigates the places the user wants to go, food, and product reviews.

[0203] For example, even when not talking to the user, the robot 100 investigates weather information and provides advice according to the user's schedule or plan.

[0204] For example, even when not talking to the user, the robot 100 investigates information about local events or festivals and proposes them to the user.

[0205] Robot 100, for example, even when not talking to the user, checks the results of sports games or news that the user is interested in and provides topics for conversation.

[0206] Robot 100, for example, even when not talking to the user, checks information about the user's favorite music or artists and introduces them.

[0207] Robot 100, for example, even when not talking to the user, checks information about social issues or news that the user is concerned about and provides opinions.

[0208] Robot 100, for example, even when not talking to the user, checks information about the user's hometown or place of origin and provides topics for conversation.

[0209] Robot 100, for example, even when not talking to the user, checks information about the user's work or school and provides advice.

[0210] Robot 100, even when not talking to the user, checks information about books, comics, movies, and dramas that the user is interested in and introduces them.

[0211] Robot 100, for example, even when not talking to the user, checks information about the user's health and provides advice.

[0212] Robot 100, for example, even when not talking to the user, checks information about the user's travel plans and provides advice.

[0213] Robot 100, for example, even when not talking to the user, checks information about the repair or maintenance of the user's house or car and provides advice.

[0214] Robot 100, for example, even when not talking to the user, checks information about beauty or fashion that the user is interested in and provides advice.

[0215] Even when not talking to the user, for example, the robot 100 examines the information of the user's pet and provides advice.

[0216] Even when not talking to the user, for example, the robot 100 examines and proposes information on contests and events related to the user's hobbies and work.

[0217] Even when not talking to the user, for example, the robot 100 examines and proposes information on the user's favorite restaurants and eateries.

[0218] Even when not talking to the user, for example, the robot 100 collects information and provides advice regarding important decisions related to the user's life.

[0219] Even when not talking to the user, for example, the robot 100 examines information regarding the person the user is worried about and provides advice.

[0220] [Second Embodiment] In the second embodiment, the above-described robot 100 is applied to a stuffed toy or a control device that is wirelessly or wiredly connected to a controlled device (such as a speaker or a camera) mounted on the stuffed toy. For parts having the same configuration as those in the first embodiment, the same reference numerals are given and the description thereof is omitted.

[0221] Specifically, the second embodiment is configured as follows. For example, while spending daily life with the user 10, the robot 100 is applied to a cohabitant (specifically, the stuffed toy 100N shown in FIGS. 7 and 8) that advances the conversation or provides information matching the user 10's hobbies and interests based on information regarding the daily life with the user 10. In the second embodiment, an example in which the control part of the above-described robot 100 is applied to the smartphone 50 will be described.

[0222] The stuffed toy 100N equipped with the function as the input / output device of the robot 100 has a smartphone 50 that functions as the control part of the robot 100, and inside the stuffed toy 100N, the input / output device and the accommodated smartphone 50 are connected.

[0223] As shown in FIG. 7(A), in this embodiment (and other embodiments), the stuffed toy 100N has the shape of a bear covered with a soft cloth fabric on the outside, and in the space part 52 formed inside thereof, as the input / output device, the sensor part 200A and the controlled object 252A are arranged (see FIG. 9). The sensor part 200A includes a microphone 201 and a 2D camera 203. Specifically, as shown in FIG. 7(B), in the space part 52, the microphone 201 of the sensor part 200 is arranged at the part corresponding to the ear 54, the 2D camera 203 of the sensor part 200 is arranged at the part corresponding to the eye 56, and a speaker 60 that constitutes a part of the controlled object 252A is arranged at the part corresponding to the mouth 58. Note that the microphone 201 and the speaker 60 do not necessarily have to be separate bodies, and they may be an integrated unit. In the case of a unit, it may be arranged at a position where the speech can be heard naturally, such as the position of the nose of the stuffed toy 100N. Note that although the stuffed toy 100N has been described by taking the case of having the shape of an animal as an example, it is not limited thereto. The stuffed toy 100N may have the shape of a specific character.

[0224] FIG. 9 schematically shows the functional configuration of the stuffed toy 100N. The stuffed toy 100N has a sensor part 200A, a sensor module part 210, a storage part 220, a control part 228, and a controlled object 252A.

[0225] The smartphone 50 accommodated in the stuffed toy 100N of this embodiment executes the same processing as the robot 100 of the first embodiment. That is, the smartphone 50 has the functions as the sensor module part 210, the storage part 220, and the control part 228 shown in FIG. 9.

[0226] As shown in FIG. 8, a fastener 62 is attached to a part (e.g., the back) of the stuffed toy 100N, and by opening the fastener 62, the outside communicates with the space part 52.

[0227] Here, the smartphone 50 is housed from the outside into the space part 52 and connected to each input / output device via a USB hub 64 (see FIG. 7(B)) to have the same functions as the robot 100 in the first embodiment.

[0228] In addition, a non-contact power receiving plate 66 is connected to the USB hub 64. A power receiving coil 66A is incorporated in the power receiving plate 66. The power receiving plate 66 is an example of a wireless power receiving part that receives wireless power supply.

[0229] The power receiving plate 66 is disposed near the attachment roots 68 of both feet of the stuffed toy 100N and is at the position closest to the placement base 70 when the stuffed toy 100N is placed on the placement base 70. The placement base 70 is an example of an external wireless power transmission part.

[0230] The stuffed toy 100N placed on this placement base 70 can be appreciated as an ornament in a natural state.

[0231] In addition, this attachment root is formed thinner than the surface layer thickness of the stuffed toy 100N at other parts so as to be held in a state closer to the placement base 70.

[0232] The placement base 70 is provided with a charging pad 72. A power transmission coil 72A is incorporated in the charging pad 72. The power transmission coil 72A sends a signal to search for the power receiving coil 66A of the power receiving plate 66. When the power receiving coil 66A is found, a current flows through the power transmission coil 72A to generate a magnetic field, and the power receiving coil 66A reacts to the magnetic field to start electromagnetic induction. As a result, a current flows through the power receiving coil 66A, and power is stored in the battery (not shown) of the smartphone 50 via the USB hub 64.

[0233] That is, by placing the stuffed toy 100N on the placement base 70 as an ornament, the smartphone 50 is automatically charged, so there is no need to take out the smartphone 50 from the space portion 52 of the stuffed toy 100N for charging.

[0234] In the second embodiment, the smartphone 50 is housed in the space portion 52 of the stuffed toy 100N and connected by wire (USB connection), but it is not limited to this. For example, a control device having a wireless function (e.g., "Bluetooth (registered trademark)") may be housed in the space portion 52 of the stuffed toy 100N, and the control device may be connected to the USB hub 64. In this case, without putting the smartphone 50 in the space portion 52, the smartphone 50 and the control device communicate wirelessly, and the external smartphone 50 is connected to each input / output device via the control device, so that the same functions as the robot 100 in the first embodiment can be provided. Also, the control device housed in the space portion 52 of the stuffed toy 100N and the external smartphone 50 may be connected by wire.

[0235] In the second embodiment, the stuffed bear 100N is illustrated, but other animals, dolls, or specific character shapes may be used. Also, it may be changeable in clothing. Furthermore, the material of the skin is not limited to cloth, and other materials such as soft vinyl may be used, but a soft material is preferred.

[0236] Furthermore, a monitor may be attached to the skin of the stuffed toy 100N to add a control object 252 that provides information to the user 10 through vision. For example, the eyes 56 may be used as a monitor to express emotions such as joy, anger, sorrow, and happiness by the images reflected in the eyes, or a window through which the monitor of the built-in smartphone 50 is visible may be provided on the abdomen. Furthermore, the eyes 56 may be used as a projector to express emotions such as joy, anger, sorrow, and happiness by the images projected on the wall surface.

[0237] According to the second embodiment, the existing smartphone 50 was placed inside the stuffed toy 100N, and from there, the camera 203, the microphone 201, the speaker 60, etc. were extended to appropriate positions via a USB connection.

[0238] Furthermore, for wireless charging, the smartphone 50 and the power receiving plate 66 were connected by a USB connection, and the power receiving plate 66 was arranged so as to be as far outside as possible when viewed from the inside of the stuffed toy 100N.

[0239] When attempting to use the wireless charging of the smartphone 50, the smartphone 50 has to be arranged as far outside as possible when viewed from the inside of the stuffed toy 100N, which makes it feel rough when the stuffed toy 100N is touched from the outside.

[0240] Therefore, the smartphone 50 was arranged as close to the center of the stuffed toy 100N as possible, and the wireless charging function (power receiving plate 66) was arranged as far outside as possible when viewed from the inside of the stuffed toy 100N. The camera 203, the microphone 201, the speaker 60, and the smartphone 50 receive wireless power supply via the power receiving plate 66.

[0241] Note that since the other configurations and operations of the stuffed toy 100N in the second embodiment are the same as those of the robot 100 in the first embodiment, the description is omitted.

[0242] Also, a part of the stuffed toy 100N (for example, the sensor module unit 210, the storage unit 220, the control unit 228) may be provided outside the stuffed toy 100N (for example, a server), and the stuffed toy 100N may function as each part of the above-described stuffed toy 100N by communicating with the outside.

[0243] [Third Embodiment] In the above-described first embodiment, the case where the behavior control system is applied to the robot 100 was exemplified. However, in the third embodiment, the above-described robot 100 is used as an agent for interacting with the user, and the behavior control system is applied to the agent system. Note that parts having the same configuration as those in the first and second embodiments are denoted by the same reference numerals, and the description thereof is omitted.

[0244] FIG. 10 is a functional block diagram of an agent system 500 configured by using some or all of the functions of the behavior control system.

[0245] The agent system 500 is a computer system that performs a series of actions in accordance with the intention of the user 10 through the interaction with the user 10. The interaction with the user 10 can be performed by voice or text.

[0246] The agent system 500 includes a sensor unit 200A, a sensor module unit 210, a storage unit 220, a control unit 228B, and a control target 252B.

[0247] The agent system 500 can be mounted on, for example, a robot, a humanoid, a stuffed animal, a wearable terminal (pendant, smartwatch, smart glasses), a smartphone, a smart speaker, earphones, and a personal computer. Further, the agent system 500 may be implemented on a web server and used via a web browser operating on a communication terminal such as a smartphone owned by the user.

[0248] The agent system 500 serves as, for example, a butler, a secretary, a teacher, a partner, a friend, a lover, or a teacher who acts for the user 10. The agent system 500 not only interacts with the user 10 but also provides advice, guides to a destination, or makes recommendations according to the user's preferences. Further, the agent system 500 makes reservations, orders, or pays for services to service providers.

[0249] Similar to the first embodiment, the emotion determination unit 232 determines the emotions of the user 10 and the agent itself. The action determination unit 236 determines the actions of the robot 100 while taking into account the emotions of the user 10 and the agent. That is, the agent system 500 understands the emotions of the user 10 and realizes sincere support, assistance, advice, and service provision by reading the atmosphere. In addition, the agent system 500 listens to the user 10's troubles, comforts, encourages, and cheers up the user. Further, the agent system 500 plays with the user 10, draws a picture diary, and reminds the user of the past. The agent system 500 takes actions that increase the happiness of the user 10. Here, the agent is an agent that operates on software.

[0250] The control unit 228B includes a state recognition unit 230, an emotion determination unit 232, an action recognition unit 234, an action determination unit 236, a memory control unit 238, an action control unit 250, a related information collection unit 270, a command acquisition unit 272, an RPA (Robotic Process Automation) 274, a character setting unit 276, a communication processing unit 280, and a specific processing unit 290.

[0251] Similar to the first embodiment, the action determination unit 236 determines the utterance content of the agent for interacting with the user 10 as the action of the agent. The action control unit 250 outputs the utterance content of the agent to a speaker or a display as the control target 252B by at least one of voice and text.

[0252] The character setting unit 276 sets the character of the agent when the agent system 500 interacts with the user 10 based on the specification from the user 10. That is, the utterance content output from the action determination unit 236 is output through an agent having the set character. As the character, for example, it is possible to set an actual famous person or celebrity such as an actor, entertainer, idol, or sports player. It is also possible to set a fictional character appearing in a comic, movie, or animation. For example, it is possible to set "Princess Anne" played by "Audrey Hepburn" who appears in the movie "Roman Holiday" as the character of the agent. When the character of the agent is a known one, since the voice, diction, tone, and personality of the character are known, the user 10 only needs to specify their favorite character, and the prompt setting in the character setting unit 276 is automatically performed. The voice, diction, tone, and personality of the set character are reflected in the interaction with the user 10. That is, the action control unit 250 synthesizes voice according to the character set by the character setting unit 276 and outputs the utterance content of the agent by the synthesized voice. Thereby, the user 10 can have the feeling of interacting with their favorite character (e.g., a favorite actor) himself / herself.

[0253] When the agent system 500 is installed in a device having a display such as a smartphone, an icon, still image, or moving image of an agent having the character set by the character setting unit 276 may be displayed on the display. The image of the agent is generated using an image synthesis technique such as 3D rendering, for example. In the agent system 500, the interaction with the user 10 may be performed while the image of the agent makes gestures according to the emotion of the user 10, the emotion of the agent, and the utterance content of the agent. Note that the agent system 500 may output only voice without outputting an image when interacting with the user 10.

[0254] Similar to the first embodiment, the emotion determination unit 232 determines an emotion value indicating the emotion of the user 10 and the emotion value of the agent itself. In this embodiment, instead of the emotion value of the robot 100, the emotion value of the agent is determined. The emotion value of the agent itself is reflected in the emotion of the set character. When the agent system 500 interacts with the user 10, not only the emotion of the user 10 but also the emotion of the agent is reflected in the interaction. That is, the action control unit 250 outputs the utterance content in a manner corresponding to the emotion determined by the emotion determination unit 232.

[0255] Also, the emotion of the agent is reflected even when the agent system 500 takes an action directed at the user 10. For example, when the user 10 requests the agent system 500 to take a photo, whether the agent system 500 takes a photo in response to the user's request is determined according to the degree of the emotion of "sadness" held by the agent. When the character has a positive emotion, it engages in friendly dialogue or actions towards the user 10, and when it has a negative emotion, it engages in rebellious dialogue or actions towards the user 10.

[0256] The history data 222 stores, as event data, the history of the interaction between the user 10 and the agent system 500. The storage unit 220 may be implemented by an external cloud storage. When the agent system 500 interacts with the user 10 or takes an action directed to the user 10, it determines the content of the interaction or the action in consideration of the content of the interaction history stored in the history data 222. For example, the agent system 500 grasps the hobbies and preferences of the user 10 based on the interaction history stored in the history data 222. The agent system 500 generates the content of the interaction that matches the hobbies and preferences of the user 10 or provides recommendations. The action determination unit 236 determines the utterance content of the agent based on the interaction history stored in the history data 222. The history data 222 stores personal information of the user 10 such as the name, address, phone number, credit card number, etc. obtained through the interaction with the user 10. Here, the agent may spontaneously ask the user 10 whether to register personal information, such as "Do you want to register your credit card number?", and store the personal information in the history data 222 according to the answer of the user 10.

[0257] As described in the first embodiment, the action determination unit 236 generates the utterance content based on the text generated using the text generation model. Specifically, the action determination unit 236 inputs the text or voice input by the user 10, the emotions of both the user 10 and the character determined by the emotion determination unit 232, and the conversation history stored in the history data 222 into the text generation model to generate the utterance content of the agent. At this time, the action determination unit 236 may further input the character personality set by the character setting unit 276 into the text generation model to generate the utterance content of the agent. In the agent system 500, the text generation model is not located on the front-end side that is the touch point with the user 10, but is only used as a tool of the agent system 500.

[0258] The command acquisition unit 272 acquires the agent's command from the voice or text uttered by the user 10 through the dialogue with the user 10, using the output of the speech understanding unit 212. The command includes, for example, the content of actions that the agent system 500 should execute, such as information search, store reservation, ticket arrangement, purchase of goods or services, payment of money, route guidance to the destination, provision of recommendations, etc.

[0259] The RPA 274 performs actions according to the command acquired by the command acquisition unit 272. The RPA 274 performs actions related to the use of service providers, such as information search, store reservation, ticket arrangement, purchase of goods or services, payment of money, etc.

[0260] The RPA 274 reads and uses the personal information of the user 10 necessary for performing actions related to the use of service providers from the history data 222. For example, when the agent system 500 purchases a product in response to a request from the user 10, it reads and uses the personal information of the user 10 such as name, address, phone number, credit card number, etc. stored in the history data 222. Requesting the user 10 to input personal information in the initial setting is unfriendly and unpleasant for the user. In the agent system 500 according to this embodiment, instead of requesting the user 10 to input personal information in the initial setting, it stores the personal information acquired through the dialogue with the user 10 and reads and uses it as needed. Thereby, it is possible to avoid making the user feel uncomfortable, and the convenience of the user is improved.

[0261] The agent system 500 executes dialogue processing, for example, according to the following steps 1 to 6.

[0262] (Step 1) The agent system 500 sets the character of the agent. Specifically, the character setting unit 276 sets the character of the agent when the agent system 500 interacts with the user 10 based on the specification from the user 10.

[0263] (Step 2) The agent system 500 acquires the state of the user 10 including the voice or text input from the user 10, the emotional value of the user 10, the emotional value of the agent, and the history data 222. Specifically, the same processing as in the above steps S100 to S103 is performed to acquire the state of the user 10 including the voice or text input from the user 10, the emotional value of the user 10, the emotional value of the agent, and the history data 222.

[0264] (Step 3) The agent system 500 determines the utterance content of the agent. Specifically, the action determination unit 236 inputs the text or voice input by the user 10, the emotions of both the user 10 and the character identified by the emotion determination unit 232, and the conversation history stored in the history data 222 into the sentence generation model to generate the utterance content of the agent.

[0265] For example, a fixed sentence "At this time, as an agent, how should I reply?" is added to the text representing the text or voice input by the user 10, the emotions of both the user 10 and the character identified by the emotion determination unit 232, and the conversation history stored in the history data 222, and then input into the sentence generation model to obtain the utterance content of the agent.

[0266] As an example, when the text or voice input by the user 10 is "I want to reserve a nice Chinese restaurant nearby at 7 pm tonight", the utterance content of the agent is obtained as "Understood." and "Here are the recommended restaurants. 1. AAAA. 2. BBBB. 3. CCCC. 4. DDDD".

[0267] Also, when the text or voice input by the user 10 is "The 4th DDDD is fine", the utterance content of the agent is obtained as "Understood. I'll try to make a reservation. How many seats?"

[0268] (Step 4) The agent system 500 outputs the speech content of the agent. Specifically, the action control unit 250 synthesizes speech according to the character set by the character setting unit 276, and outputs the speech content of the agent by the synthesized speech.

[0269] (Step 5) The agent system 500 determines whether it is the timing to execute the command of the agent. Specifically, the action determination unit 236 determines whether it is the timing to execute the command of the agent based on the output of the sentence generation model. For example, if the output of the sentence generation model includes that the agent executes a command, it is determined that it is the timing to execute the command of the agent, and the process proceeds to Step 6. On the other hand, if it is determined that it is not the timing to execute the command of the agent, the process returns to Step 2 above.

[0270] (Step 6) The agent system 500 executes the command of the agent. Specifically, the command acquisition unit 272 acquires the command of the agent from the speech or text uttered by the user 10 through the interaction with the user 10. Then, the RPA 274 performs an action according to the command acquired by the command acquisition unit 272. For example, when the command is "information search", information search is performed by the search site using the search query obtained through the interaction with the user 10 and the API (Application Programming Interface). The action determination unit 236 inputs the search result into the sentence generation model to generate the speech content of the agent. The action control unit 250 synthesizes speech according to the character set by the character setting unit 276, and outputs the speech content of the agent by the synthesized speech.

[0271] Also, when the command is "store reservation", using the reservation information obtained through the dialogue with User 10, the store information of the reservation destination, and the API, the phone software makes a call to the store of the reservation destination to make a reservation. At this time, the action determination unit 236 uses a sentence generation model having a dialogue function to acquire the agent's utterance content for the voice input from the other party. Then, the action determination unit 236 inputs the result (reservation success or failure) of the store reservation into the sentence generation model to generate the agent's utterance content. The action control unit 250 synthesizes the voice corresponding to the character set by the character setting unit 276, and outputs the agent's utterance content by the synthesized voice.

[0272] Then, return to step 2 above.

[0273] In step 6, the result of the action (for example, store reservation) executed by the agent is also stored in the history data 222. The result of the action executed by the agent stored in the history data 222 is utilized by the agent system 500 to grasp the hobbies or preferences of User 10. For example, when the same store is reserved multiple times, it is recognized that User 10 likes that store, or the reserved time slot, or the content or price of the course, etc., which are the reservation contents, are used as the criteria for selecting a store for the next reservation.

[0274] In this way, the agent system 500 can execute dialogue processing and perform actions related to the use of service providers as necessary.

[0275] FIGS. 11 and 12 are diagrams showing an example of the operation of the agent system 500. FIG. 11 illustrates a mode in which the agent system 500 makes a reservation at a restaurant through dialogue with User 10. In FIG. 11, on the left side, the agent's utterance content is shown, and on the right side, the utterance content of User 10 is shown. The agent system 500 can grasp the preferences of User 10 based on the dialogue history with User 10, provide a recommendation list of restaurants that match the preferences of User 10, and execute the reservation of the selected restaurant.

[0276] On the one hand, FIG. 12 illustrates a mode in which the agent system 500 accesses a teleshopping site through interaction with the user 10 to purchase a product. In FIG. 12, on the left side, the utterance content of the agent is shown, and on the right side, the utterance content of the user 10 is shown. Based on the interaction history with the user 10, the agent system 500 can estimate the remaining quantity of the beverages stocked by the user, propose the purchase of the beverages to the user 10, and execute the purchase. Further, based on the past interaction history with the user 10, the agent system 500 can grasp the user's preferences and recommend snacks that the user likes. In this way, while communicating with the user 10 as an agent like a butler, the agent system 500 can support the daily life of the user 10 by executing various actions such as restaurant reservation or purchase settlement of products.

[0277] Similar to the first embodiment, the specific processing unit 290 performs specific processing for generating information related to child rearing and controls the actions of the agent so as to output the result of the specific processing. At this time, as the action of the agent, the utterance content of the agent for interacting with the user 10 is determined, and the utterance content of the agent is output to the speaker or the display as the control target 252B by at least one of voice and text.

[0278] Note that since the other configurations and operations of the agent system 500 of the third embodiment are the same as those of the robot 100 of the first embodiment, the description thereof is omitted.

[0279] In addition, a part of the agent system 500 (for example, the sensor module unit 210, the storage unit 220, the control unit 228B) may be provided outside a communication terminal such as a smartphone owned by the user (for example, a server), and the communication terminal may function as each part of the agent system 500 described above by communicating with the outside.

[0280] [Fourth Embodiment] In the fourth embodiment, the above agent system is applied to smart glasses. Note that, for parts having the same configuration as those in the first to third embodiments, the same reference numerals are given and the description thereof is omitted.

[0281] FIG. 13 is a functional block diagram of an agent system 700 configured by using some or all of the functions of the behavior control system.

[0282] As shown in FIG. 14, the smart glasses 720 are glasses-type smart devices and are worn by the user 10 in the same manner as ordinary glasses. The smart glasses 720 are an example of an electronic device and a wearable terminal.

[0283] The smart glasses 720 include an agent system 700. The display included in the control target 252B displays various information to the user 10. The display is, for example, a liquid crystal display. The display is provided, for example, in the lens portion of the smart glasses 720, and the display content is visible to the user 10. The speaker included in the control target 252B outputs a voice indicating various information to the user 10. The smart glasses 720 include a touch panel (not shown), and the touch panel receives an input from the user 10.

[0284] The acceleration sensor 206, the temperature sensor 207, and the heart rate sensor 208 of the sensor unit 200B detect the state of the user 10. Note that these sensors are merely examples, and it goes without saying that other sensors may be mounted to detect the state of the user 10.

[0285] The microphone 201 acquires the voice uttered by the user 10 or the ambient sound around the smart glasses 720. The 2D camera 203 can image the surroundings of the smart glasses 720. The 2D camera 203 is, for example, a CCD camera.

[0286] The sensor module unit 210B includes a voice emotion recognition unit 211 and a speech understanding unit 212. The communication processing unit 280 of the control unit 228B controls the communication between the smart glasses 720 and the outside.

[0287] FIG. 14 is a diagram showing an example of a usage mode of the agent system 700 by the smart glasses 720. The smart glasses 720 realize the provision of various services using the agent system 700 for the user 10. For example, when the smart glasses 720 are operated by the user 10 (for example, voice input to the microphone, or the touch panel is tapped with a finger, etc.), the smart glasses 720 start using the agent system 700. Here, using the agent system 700 means that the smart glasses 720 have the agent system 700 and use the agent system 700, and also includes a mode in which a part of the agent system 700 (for example, the sensor module unit 210B, the storage unit 220, the control unit 228B) is provided outside the smart glasses 720 (for example, in a server), and the smart glasses 720 use the agent system 700 by communicating with the outside.

[0288] When the user 10 operates the smart glasses 720, a touch point is generated between the agent system 700 and the user 10. That is, the provision of services by the agent system 700 is started. As described in the third embodiment, in the agent system 700, the character setting unit 276 sets the character of the agent (for example, the character of Audrey Hepburn).

[0289] The emotion determination unit 232 determines the emotion value indicating the emotion of the user 10 and the emotion value of the agent itself. Here, the emotion value indicating the emotion of the user 10 is estimated from various sensors included in the sensor unit 200B mounted on the smart glasses 720. For example, when the heart rate of the user 10 detected by the heart rate sensor 208 is increasing, emotion values such as "uneasy" and "fear" are estimated to be large.

[0290] Also, as a result of measuring the user's body temperature with the temperature sensor 207, for example, if it is above the average body temperature, emotional values such as "pain" and "bitter" are largely estimated. Also, for example, if it is detected by the acceleration sensor 206 that the user 10 is doing some kind of sport, an emotional value such as "fun" is largely estimated.

[0291] Also, for example, the emotional value of the user 10 may be estimated from the voice or speech content of the user 10 acquired by the microphone 201 mounted on the smart glasses 720. For example, if the user 10 is shouting, an emotional value such as "anger" is largely estimated.

[0292] When the emotional value estimated by the emotion determination unit 232 becomes higher than a predetermined value, the agent system 700 causes the smart glasses 720 to acquire information regarding the surrounding situation. Specifically, for example, the 2D camera 203 is caused to capture an image or video showing the surrounding situation of the user 10 (for example, the people or objects around). Also, the microphone 201 is caused to record the ambient sound. Other information regarding the surrounding situation includes information indicating the date, time, location information, or weather. The information regarding the surrounding situation is stored in the history data 222 together with the emotional value. The history data 222 may be realized by an external cloud storage. In this way, the surrounding situation obtained by the smart glasses 720 is stored in the history data 222 as a so-called life log in a state associated with the emotional value of the user 10 at that time.

[0293] In the agent system 700, information indicating the surrounding situation is stored in the history data 222 in association with the emotional value. Thereby, personal information such as the hobbies, preferences, or personality of the user 10 is grasped by the agent system 700. For example, when an image showing the state of watching a baseball game is associated with an emotional value such as "joy" and "fun", the agent system 700 grasps from the information stored in the history data 222 that the hobby of the user 10 is watching a baseball game and the favorite team or player.

[0294] When the agent system 700 interacts with the user 10 or takes actions directed at the user 10, it determines the content of the interaction or the actions by taking into account the content of the surrounding situation stored in the history data 222. In addition to the surrounding situation, it goes without saying that the content of the interaction or the actions may be determined by taking into account the conversation history stored in the history data 222 as described above.

[0295] As described above, the action determination unit 236 generates the utterance content based on the text generated by the text generation model. Specifically, the action determination unit 236 inputs the text or voice input by the user 10, the emotions of both the user 10 and the agent determined by the emotion determination unit 232, the conversation history stored in the history data 222, and the personality of the agent, etc. into the text generation model to generate the utterance content of the agent. Furthermore, the action determination unit 236 inputs the surrounding situation stored in the history data 222 into the text generation model to generate the utterance content of the agent.

[0296] The generated utterance content is, for example, output as voice to the user 10 from the speaker mounted on the smart glasses 720. In this case, a synthesized voice corresponding to the character of the agent is used as the voice. The action control unit 250 generates a synthesized voice by reproducing the voice quality of the agent's character (for example, Audrey Hepburn), or generates a synthesized voice according to the emotion of the character (for example, a voice with a stronger tone when the emotion is "anger"). Also, instead of or together with the voice output, the utterance content may be displayed on the display.

[0297] The RPA 274 executes operations according to commands (for example, commands of the agent obtained from the voice or text uttered by the user 10 through the interaction with the user 10). The RPA 274 performs actions related to the use of service providers such as information search, store reservation, ticket arrangement, purchase of goods / services, payment of fees, route guidance, translation, etc.

[0298] As another example, RPA274 performs an operation of transmitting the content voice - input by the user 10 (e.g., a child) through interaction with the agent to the other party (e.g., a parent). Examples of the transmission means include, for example, message application software, chat application software, or email application software, etc.

[0299] When the operation by RPA274 is executed, for example, a voice indicating that the execution of the operation has ended is output from the speaker mounted on the smart glasses 720. For example, a voice such as "The reservation at the store has been completed" is output to the user 10. Also, for example, when the reservation at the store is full, a voice such as "The reservation could not be made. What should I do?" is output to the user 10.

[0300] As described above, in the smart glasses 720, various services are provided to the user 10 by using the agent system 700. Also, since the smart glasses 720 are worn by the user 10, it is realized that the agent system 700 can be used in various scenes such as at home, at the workplace, and outside.

[0301] Also, since the smart glasses 720 are worn by the user 10, they are suitable for collecting the so - called life log of the user 10. Specifically, the emotional value of the user 10 is estimated based on the detection results by various sensors mounted on the smart glasses 720 or the recording results of the 2D camera 203, etc. Therefore, the emotional value of the user 10 can be collected in various scenes, and the agent system 700 can provide services or utterance content suitable for the emotion of the user 10.

[0302] Also, in the smart glasses 720, the surrounding situation of the user 10 can be obtained by the 2D camera 203, the microphone 201, etc. And these surrounding situations are associated with the emotional value of the user 10. Thereby, it is possible to estimate what kind of emotions the user 10 has when placed in what kind of situation. As a result, the accuracy when the agent system 700 grasps the hobbies and preferences of the user 10 can be improved. And in the agent system 700, by accurately grasping the hobbies and preferences of the user 10, the agent system 700 can provide a service or utterance content suitable for the hobbies and preferences of the user 10.

[0303] Also, the agent system 700 can also be applied to other wearable terminals (electronic devices that can be worn on the body of the user 10 such as pendants, smartwatches, earrings, bracelets, hair bands, etc.). When the agent system 700 is applied to a smart pendant, the speaker as the control target 252B outputs a voice indicating various information to the user 10. The speaker is, for example, a speaker capable of outputting directional voice. The speaker is set to have directivity toward the ear of the user 10. Thereby, it is suppressed that the voice reaches a person other than the user 10. The microphone 201 acquires the voice uttered by the user 10 or the ambient sound around the smart pendant. The smart pendant is worn in a manner that can be lifted from the neck of the user 10. For this reason, the smart pendant is located at a place relatively close to the mouth of the user 10 while being worn. Thereby, it becomes easy to acquire the voice uttered by the user 10.

[0304] Note that in the above embodiment, the case where the robot 100 recognizes the user 10 using the face image of the user 10 has been described, but the disclosed technology is not limited to this aspect. For example, the robot 100 may recognize the user 10 using the voice uttered by the user 10, the email address of the user 10, the SNS ID of the user 10, or an ID card incorporating a wireless IC tag possessed by the user 10.

[0305] Robot 100 is an example of an electronic device equipped with an action control system. The application target of the action control system is not limited to Robot 100, and the action control system can be applied to various electronic devices. Also, the functions of Server 300 may be implemented by one or more computers. At least some of the functions of Server 300 may be implemented by a virtual machine. Further, at least some of the functions of Server 300 may be implemented in the cloud.

[0306] FIG. 15 schematically shows an example of the hardware configuration of a smartphone 50, a robot 100, a server 300, and computers 1200 that function as agent systems 500 and 700. Programs installed in computer 1200 can cause computer 1200 to function as one or more "parts" of the device according to the present embodiment, or cause computer 1200 to execute an operation or the one or more "parts" associated with the device according to the present embodiment, and / or cause computer 1200 to execute the process or a stage of the process according to the present embodiment. Such a program may be executed by CPU 1212 to cause computer 1200 to execute certain operations associated with some or all of the blocks of the flowcharts and block diagrams described herein.

[0307] The computer 1200 according to this embodiment includes a CPU 1212, a RAM 1214, and a graphic controller 1216, which are interconnected by a host controller 1210. The computer 1200 also includes input / output units such as a communication interface 1222, a storage device 1224, a DVD drive 1226, and an IC card drive, which are connected to the host controller 1210 via an input / output controller 1220. The DVD drive 1226 may be a DVD-ROM drive, a DVD-RAM drive, or the like. The storage device 1224 may be a hard disk drive, a solid state drive, or the like. The computer 1200 also includes legacy input / output units such as a ROM 1230 and a keyboard, which are connected to the input / output controller 1220 via an input / output chip 1240.

[0308] The CPU 1212 operates according to programs stored in the ROM 1230 and the RAM 1214, thereby controlling each unit. The graphic controller 1216 acquires image data generated by the CPU 1212 in a frame buffer or the like provided in the RAM 1214 or within itself, and causes the image data to be displayed on the display device 1218.

[0309] The communication interface 1222 communicates with other electronic devices via a network. The storage device 1224 stores programs and data used by the CPU 1212 within the computer 1200. The DVD drive 1226 reads a program or data from a DVD-ROM 1227 or the like and provides it to the storage device 1224. The IC card drive reads programs and data from an IC card and / or writes programs and data to an IC card.

[0310] ROM 1230 stores therein a boot program or the like executed by computer 1200 at activation, and / or a program dependent on the hardware of computer 1200. Input / output chip 1240 may also be connected to input / output controller 1220 via various input / output units through a USB port, a parallel port, a serial port, a keyboard port, a mouse port, or the like.

[0311] The program is provided by a computer-readable storage medium such as a DVD-ROM 1227 or an IC card. The program is read from the computer-readable storage medium, installed in a storage device 1224, which is also an example of a computer-readable storage medium, RAM 1214, or ROM 1230, and executed by CPU 1212. The information processing described in these programs is read by computer 1200, resulting in cooperation between the programs and the various types of hardware resources described above. The device or method may be configured by realizing an operation or processing of information according to the use of computer 1200.

[0312] For example, when communication is executed between computer 1200 and an external device, CPU 1212 may execute a communication program loaded in RAM 1214 and instruct communication interface 1222 to perform communication processing based on the processing described in the communication program. Communication interface 1222 reads transmission data stored in a transmission buffer area provided in a recording medium such as RAM 1214, storage device 1224, DVD-ROM 1227, or an IC card under the control of CPU 1212, transmits the read transmission data to the network, or writes received data received from the network to a reception buffer area or the like provided on the recording medium.

[0313] Further, the CPU 1212 may cause all or necessary portions of files or databases stored in an external recording medium such as the storage device 1224, the DVD drive 1226 (DVD-ROM 1227), an IC card, etc. to be read into the RAM 1214, and may execute various types of processing on the data on the RAM 1214. Next, the CPU 1212 may write back the processed data to the external recording medium.

[0314] Various types of information such as various types of programs, data, tables, and databases may be stored in the recording medium and may be subjected to information processing. The CPU 1212 may perform various types of processing on the data read from the RAM 1214, including various types of operations, information processing, conditional judgment, conditional branch, unconditional branch, information search / replacement, etc. described throughout this disclosure and specified by the instruction sequence of the program, and write back the result to the RAM 1214. Further, the CPU 1212 may search for information in files, databases, etc. in the recording medium. For example, when a plurality of entries each having an attribute value of a first attribute associated with an attribute value of a second attribute are stored in the recording medium, the CPU 1212 searches for an entry that matches the condition in which the attribute value of the first attribute is specified among the plurality of entries, reads the attribute value of the second attribute stored in the entry, and thereby may obtain the attribute value of the second attribute associated with the first attribute that satisfies a predetermined condition.

[0315] The programs or software modules described above may be stored in a computer-readable storage medium on or near the computer 1200. Also, a recording medium such as a hard disk or RAM provided in a server system connected to a dedicated communication network or the Internet can be used as a computer-readable storage medium, thereby providing the program to the computer 1200 via the network.

[0316] In the flowcharts and block diagrams in this embodiment, the blocks may represent stages of a process in which operations are performed or "parts" of a device having a role of performing operations. Specific stages and "parts" may be implemented by a dedicated circuit, a programmable circuit supplied with computer-readable instructions stored on a computer-readable storage medium, and / or a processor supplied with computer-readable instructions stored on a computer-readable storage medium. The dedicated circuit may include digital and / or analog hardware circuits, and may include an integrated circuit (IC) and / or discrete circuits. The programmable circuit may include, for example, a reconfigurable hardware circuit including logical products, logical sums, exclusive logical sums, negative logical products, negative logical sums, and other logical operations, flip-flops, registers, and memory elements, such as a field programmable gate array (FPGA) and a programmable logic array (PLA).

[0317] The computer-readable storage medium may include any tangible device capable of storing instructions executable by an appropriate device, and as a result, the computer-readable storage medium having instructions stored therein will comprise a product including instructions that can be executed to create means for performing the operations specified in the flowchart or block diagram. Examples of the computer-readable storage medium may include electronic storage media, magnetic storage media, optical storage media, electromagnetic storage media, semiconductor storage media, and the like. More specific examples of the computer-readable storage medium may include floppy (registered trademark) disks, diskettes, hard disks, random access memory (RAM), read only memory (ROM), erasable programmable read only memory (EPROM or flash memory), electrically erasable programmable read only memory (EEPROM), static random access memory (SRAM), compact disc read only memory (CD-ROM), digital versatile disc (DVD), Blu-ray (registered trademark) disc, memory stick, integrated circuit card, and the like.

[0318] Computer-readable instructions may include source code or object code written in any combination of one or more programming languages, including assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or object-oriented programming languages such as Smalltalk, JAVA (registered trademark), C++, and conventional procedural programming languages such as the "C" programming language or similar programming languages.

[0319] The computer-readable instructions may be provided locally or via a wide area network (WAN) such as a local area network (LAN), the Internet, etc. to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, or a programmable circuit to execute the computer-readable instructions to generate means for performing the operations specified in the flowchart or block diagram. Examples of processors include computer processors, processing units, microprocessors, digital signal processors, controllers, microcontrollers, etc.

[0320] As described above, the present invention has been described using embodiments, but the technical scope of the present invention is not limited to the scope described in the above embodiments. It is obvious to those skilled in the art that various changes or improvements can be made to the above embodiments. It is clear from the description of the claims that forms with such changes or improvements may also be included in the technical scope of the present invention.

[0321] In the claims, the specification, and the drawings, the execution order of each process such as operations, procedures, steps, and stages in the apparatus, system, program, and method shown is not explicitly indicated as "earlier" or "preceding" etc. in particular, and it should be noted that it can be realized in any order unless the output of the previous process is used in the subsequent process. Regarding the operation flow in the claims, the specification, and the drawings, even if it is described using "first," "next," etc. for convenience, it does not mean that it is essential to implement in this order.

Explanation of Reference Numerals

[0322] 5 System, 10, 11, 12 User, 20 Communication Network, 100, 101, 102 Robot, 100N Stuffed Toy 100, 200 Sensor Unit, 201 Microphone, 202 Depth Sensor, 203 Camera, 204 Distance Sensor, 210 Sensor Module Unit, 211 Voice Emotion Recognition Unit, 212 Utterance Understanding Unit, 213 Facial Expression Recognition Unit, 214 Face Recognition Unit, 220 Storage Unit, 221 Action Decision Model, 222 History Data, 230 State Recognition Unit, 232 Emotion Decision Unit, 234 Action Recognition Unit, 236 Action Decision Unit, 238 Memory Control Unit, 250 Action Control Unit, 252 Controlled Object, 270 Related Information Collection Unit, 280 Communication Processing Unit, 290 Specific Processing Unit, 300 Server, 500, 700 Agent System, 1200 Computer, 1210 Host Controller, 1212 CPU, 1214 RAM, 1216 Graphics Controller, 1218 Display Device, 1220 Input / Output Controller, 1222 Communication Interface, 1224 Storage Device, 1226 DVD Drive, 1227 DVD-ROM, 1230 ROM, 1240 Input / Output Chip

Claims

1. An input unit that receives user input, A processing unit that performs a specific process using a text generation model that generates a text according to the input data, An output unit that controls an electronic device to output the result of the specific process, and includes, The specific process includes generating information related to child rearing, The processing unit, Determines whether a predetermined trigger condition is satisfied, When the trigger condition is satisfied, information related to child rearing is generated using the output of the text generation model when the information obtained from the user input is used as the input data. Control system.

2. The user input includes at least one of personal information of a specific child and the user's place of residence or current location, and the input data includes at least one of the personal information of the specific child and the place of residence or current location. The control system according to claim 1.

3. The trigger condition includes that the input unit has received the user input requesting a reservation operation for a specific facility, When the input unit receives the user input requesting a reservation operation for the specific facility as the trigger condition, the processing unit uses the output of the text generation model when the information obtained from the user input is used as the input data. To execute the reservation operation of the specific facility. The control system according to claim 1.

4. The specific facility includes a hospital, and the reservation operation includes an operation of making a medical appointment at the hospital. The control system according to claim 3.

5. The reservation operation of the specific facility includes selecting a plurality of the specific facilities based on the user's place of residence or current location, collecting information on the selected plurality of the specific facilities, and determining whether the specific facilities are available for reservation. The control system according to claim 3, which includes an operation of making a reservation for the specific facility identified as available for reservation.

6. Further includes an emotion determination unit that determines the emotion of the user or the emotion of the electronic device, The processing unit performs the specific process using the emotion of the user or the emotion of the electronic device determined by the emotion determination unit and the text generation model. The control system according to claim 1.

7. Further includes a state recognition unit that recognizes the user state including the user's behavior and the state of the electronic device, The processing unit performs the specific process using the user state or the state of the electronic device and the text generation model. The control system according to claim 1.

8. The control system according to claim 1, wherein the electronic device is a robot.

9. The control system according to claim 8, wherein the robot is mounted on the stuffed toy or is wirelessly or wiredly connected to a device to be controlled mounted on the stuffed toy.

10. The control system according to claim 8, wherein the robot is an agent for interacting with a user.

Images