Behavior control system

Abstract

The behavior control system includes: an emotion determination unit that determines an emotion of a user or an emotion of a robot; and a behavior specifying unit that, on the basis of a text generation model having a dialogue function for causing a user to dialogue with a robot, generates behavior content of the robot for the behavior of the user and the emotion of the user or the emotion of the robot, and specifies the behavior of the robot corresponding to the behavior content. The behavior determination unit determines whether or not the behavior of the user is dangerous by detecting the behavior of the user, and generates first behavior content that corrects the behavior of the user when the behavior of the user is dangerous.

Description

TECHNICAL FIELD

[0001] Behavior control system BACKGROUND Patent Literature 1 discloses a technology of determining an appropriate behavior of a robot for a state of a user. In the related art of Patent Literature 1, a reaction of a user when a robot performs a specific behavior is recognized, and if a behavior of the robot for the recognized reaction of the user cannot be determined, the behavior of the robot is updated by receiving information related to a behavior appropriate for the recognized state of the user from a server.

[0002] As a reception-related technology such as table management of a restaurant, there are known technologies such as a technology of causing a robot to learn from a state and an emotion of a customer at the time of reception (for example, see Patent Literature 2), a technology of determining a reception order from a degree of attention of a customer to a robot (for example, see Patent Literature 3), and a technology of calculating reception satisfaction (for example, see Patent Literature 4). In addition, there are known technologies of generating an emotion of a robot (for example, see Patent Literatures 5 to 7), and a technology of recognizing an emotion of a user (for example, see Patent Literatures 8 to 9).

[0003] Even if a clerk witnesses a behavior of a customer who is suspected of being damaged by a special fraud, such as a purchase of a high-value prepaid card at a convenience store or an operation of an ATM (Asynchronous Transfer Mode) while making a phone call, the clerk often hesitates to remind the customer because of a fear of misjudgment as a special fraud or being ignored by the customer. Therefore, as a technology of detecting a risk of a special fraud, there are known technologies such as a technology of detecting a risk of a special fraud using a phone (for example, see Patent Literature 10), and a technology of performing emotion recognition by an automatic transaction device such as an ATM, detecting a risk of a special fraud when a feeling such as confusion or anxiety is recognized, and outputting a message of a money transfer fraud attention reminder on a screen (for example, see Patent Literature 11). In addition, there are known technologies of generating an emotion of a robot (for example, see Patent Literatures 5 to 7), and a technology of recognizing an emotion of a user (for example, see Patent Literatures 8 to 9).

[0004] RELATED ART DOCUMENT PATENT LITERATURE Patent Literature 1: Japanese Patent No. 6053847 Patent Literature 2: Japanese Patent Application Laid-Open No. 2019-018265 Patent Literature 3: Japanese Patent Application Laid-Open No. 2009-248193 Patent Literature 4: Japanese Patent Application Laid-Open No. 2011-210133 Patent Literature 5: Japanese Patent No. 6273314 Patent Document 6: Japanese Patent No. 6273313 Patent Document 7: Japanese Patent No. 6199927 Patent Document 8: Japanese Patent No. 3676969 Patent Document 9: Japanese Patent No. 4704952 Patent Document 10: Japanese Patent Application Publication No. 2019-153961 Patent Document 11: Japanese Patent Application Publication No. 2021-092952 Summary of the Invention The problem that the invention aims to solve However, in the existing technology, there is still room for improvement in enabling robots to perform appropriate actions in response to user behavior.

[0005] Furthermore, existing technologies sometimes fail to provide adequate customer service. For example, existing technologies do not fully consider the risks associated with troublesome customers, thus sometimes preventing staff from creating an environment where customers feel they "want to come back."

[0006] Furthermore, existing technologies sometimes cannot detect the risk of specific frauds with high precision.

[0007] For example, in the prior art, when a risk is determined to be high, the risk assessment result is not confirmed with the customer before notification, which may lead to the risk of erroneous detection.

[0008] Methods for solving problems According to a first aspect of the present invention, a behavior control system is provided. The behavior control system includes: an emotion determination unit that determines the emotion of a user or the emotion of a robot; and a behavior determination unit that, based on a dialogue function enabling the user to engage in dialogue with the robot, generates robot behavior content based on the user's behavior and the user's or robot's emotion, and determines the robot's behavior corresponding to the behavior content; wherein the behavior determination unit determines whether the user's behavior is dangerous by detecting the user's behavior, and if the user's behavior is dangerous, generates first behavior content to correct the user's behavior.

[0009] According to a second aspect of the present invention, a behavior control system is provided. The behavior control system includes: an emotion determination unit that determines the emotion of a user or the emotion of a robot; and a behavior determination unit that, based on an article generation model having a dialogue function that enables the user to engage in dialogue with the robot, generates behavior content for the robot that considers the user's behavior and the user's or robot's emotion, and determines the robot's behavior corresponding to the behavior content; wherein the behavior determination unit receives statements from multiple users in an ongoing conversation, and when a statement reaches a pre-set state, determines the content summarizing the statement as the robot's behavior.

[0010] In the third aspect, the state is a state in which the speaker is no longer received after a pre-set time.

[0011] In the fourth aspect, the state is the state in which the terms contained in the speech have been received a predetermined number of times.

[0012] In the fifth aspect, the robot is mounted on a plush toy or connected wirelessly or wired to a control device mounted on the plush toy.

[0013] According to a sixth aspect of the present invention, a behavior control system is provided. The behavior control system includes: an emotion determination unit that determines the emotion of a user or the emotion of a robot; and a behavior determination unit that, based on a dialogue function enabling the user to engage in dialogue with the robot, generates robot behavior content based on the user's behavior and the user's or robot's emotion, and determines the robot's behavior corresponding to the behavior content; wherein the robot is deployed at customs, the behavior determination unit acquires images of people using an image sensor and acquires odor detection results using an odor sensor, and if it detects pre-set abnormal behavior, abnormal facial expressions, or abnormal odors, it notifies tax authorities to determine these as robot behavior.

[0014] According to a seventh aspect of the present invention, a behavior control system is provided. The behavior control system includes: a user state recognition unit that recognizes user states, including user behavior; an emotion determination unit that determines the emotion of a user or the emotion of a robot; and a behavior determination unit that determines the robot's behavior corresponding to the user state and conversation content of multiple users based on an article generation model having a dialogue function that enables the user to engage in dialogue with the robot. The behavior determination unit assesses specific fraud risks based on the conversation content of multiple users and the user's emotion.

[0015] According to an eighth aspect of the present invention, a behavior control system is provided. The behavior control system includes: a user state recognition unit that recognizes user states including the behaviors of multiple users; an emotion determination unit that determines the emotions of the multiple users or the emotions of a robot; and a behavior determination unit that determines the robot's behavior corresponding to the user states and the conversation content of the multiple users based on an article generation model having a dialogue function that enables the user to engage in dialogue with the robot. The behavior determination unit detects specific cases based on the conversation content of the multiple users and the users' emotions.

[0016] According to a ninth aspect of the present invention, a behavior control system is provided. The behavior control system includes: a user state recognition unit that recognizes a user state, including user behavior; an emotion determination unit that determines the user's emotion or the robot's emotion; and a behavior determination unit that determines the robot's behavior corresponding to the user state and the user's emotion or the robot's emotion based on an article generation model having a dialogue function that enables the user to engage in dialogue with the robot. The behavior determination unit performs caregiving and monitoring of the user based on at least one of the user state and the user's emotion.

[0017] According to a tenth aspect of the present invention, a behavior control system is provided. The behavior control system includes: a state recognition unit that recognizes a user state, including user behavior, and a state of an electronic device; an emotion determination unit that determines the emotion of the user or the emotion of the electronic device; a behavior determination unit that, at a predetermined time point, uses at least one of the user state, the state of the electronic device, the user's emotion, and the emotion of the electronic device, and a behavior determination model, determines any one of a variety of device operations, including inaction, as the behavior of the electronic device; and a storage control unit that stores event data, including an emotion value determined by the emotion determination unit and data including the user's behavior, in historical data; wherein the device operation includes providing care-related suggestions to the user; when the behavior determination unit determines providing care-related suggestions to the user as the behavior of the electronic device, it includes collecting care-related information about the user and providing care-related suggestions to the user based on the collected information.

[0018] Here, "robot" refers to devices that perform physical actions, devices that output images or sounds without performing physical actions, and intelligent agents that run on software.

[0019] According to an eleventh aspect of the present invention, a behavior control system is provided. The behavior control system includes: a state recognition unit that recognizes a user state, including user behavior, and a state of an electronic device; an emotion determination unit that determines the emotion of the user or the emotion of the electronic device; a behavior determination unit that, at a predetermined time point, uses at least one of the user state, the state of the electronic device, the user's emotion, and the emotion of the electronic device, and a behavior determination model, to determine any one of a variety of device operations, including inaction, as the behavior of the electronic device; and a storage control unit that stores event data, including an emotion value determined by the emotion determination unit and data including the user's behavior, in historical data; wherein the device operation includes notifying a provider of information based on the user's emotion towards an item provided by the provider; when the behavior determination unit determines that notifying the provider of information based on the user's emotion towards an item provided by the provider is the behavior of the electronic device, it notifies the provider of information based on the user's emotion towards the item provided by the provider.

[0020] According to a twelfth aspect of the present invention, a behavior control system is provided. The behavior control system includes: a state recognition unit that recognizes a user state, including user behavior, and a state of an electronic device; an emotion determination unit that determines the emotion of the user or the emotion of the electronic device; and a behavior determination unit that, at a predetermined time point, uses at least one of the user state, the state of the electronic device, the user's emotion, and the emotion of the electronic device, and a behavior determination model, determines any one of a variety of device operations, including inaction, as the behavior of the electronic device, wherein the device operation includes providing the user with advice regarding fraud risk; when the behavior determination unit determines that providing the user with advice regarding fraud risk is the behavior of the electronic device, it provides the user with advice regarding fraud risk.

[0021] According to a thirteenth aspect of the present invention, a behavior control system is provided. The behavior control system includes: a state recognition unit that recognizes a user state, including user behavior, and a state of an electronic device; an emotion determination unit that determines the emotion of the user or the emotion of the electronic device; and a behavior determination unit that, at a predetermined time point, uses at least one of the user state, the state of the electronic device, the user's emotion, and the emotion of the electronic device, and a behavior determination model, determines any one of a variety of device operations, including inaction, as the behavior of the electronic device. The device operation includes providing the user with suggestions regarding approach risks; when the behavior determination unit determines that providing the user with suggestions regarding approach risks is the behavior of the electronic device, it provides the user with suggestions regarding approach risks.

[0022] The notification device according to the fourteenth aspect of the present invention includes: an acquisition unit that acquires at least one of customer information related to a customer of a store, store information related to the store, and order information related to an order placed at the store; an analysis unit that analyzes a risk based on at least one of the customer information, the store information, and the order information acquired by the acquisition unit; and a notification unit that notifies the risk analyzed by the analysis unit.

[0023] The notification method according to the fifteenth aspect of the present invention is a notification method performed by a notification device, comprising: an acquisition step, the acquisition step acquiring at least one of customer information related to a customer of a store, store information related to the store, and order information related to an order placed at the store; a parsing step, the parsing step parsing a risk based on at least one of the customer information, the store information, and the order information acquired by the acquisition step; and a notification step, the notification step notifying the risk parsed by the parsing step.

[0024] The notification procedure according to the sixteenth aspect of the present invention causes a computer to perform the following steps: an acquisition step, which acquires at least one of customer information related to a customer of a store, store information related to the store, and order information related to an order placed at the store; a parsing step, which parses a risk based on at least one of the customer information, the store information, and the order information acquired by the acquisition step; and a notification step, which notifies the risk parsed by the parsing step.

[0025] According to one aspect of the invention, appropriate hospitality can be provided.

[0026] The determination device according to the seventeenth aspect of the present invention includes: an acquisition unit that acquires at least one of photographic data of a customer in a store, attribute data related to the attributes of the customer, and conversation data related to the conversation of the customer; a determination unit that determines the risk of a specific fraud against the customer based on at least one of the photographic data, the attribute data, and the conversation data acquired by the acquisition unit; and a conversation unit that engages in a conversation with the customer regarding the risk determined by the determination unit.

[0027] The eighteenth aspect of the present invention relates to a determination method performed by a determination device, comprising: an acquisition step, the acquisition step acquiring at least one of photographic data of a customer in a store, attribute data related to the attributes of the customer, and conversation data related to a conversation of the customer; a determination step, the determination step determining a risk of specific fraud against the customer based on at least one of the photographic data, the attribute data, and the conversation data acquired by the acquisition step; and a conversation step, the conversation step engaging in a conversation with the customer regarding the risk determined by the determination step.

[0028] The nineteenth aspect of the present invention relates to a determination procedure that causes a computer to perform the following steps: an acquisition step, which acquires at least one of photographic data of a customer in a store, attribute data related to the attributes of the customer, and conversation data related to the conversation of the customer; a determination step, which determines the risk of specific fraud against the customer based on at least one of the photographic data, the attribute data, and the conversation data acquired by the acquisition step; and a conversation step, which engages in a conversation with the customer regarding the risk determined by the determination step.

[0029] According to one aspect of the invention, the risk of specific fraud can be detected with high precision. Attached Figure Description

[0030] Figure 1 An example of system 5 involved in this embodiment is illustrated schematically.

[0031] Figure 2 The functional configuration of robot 100 is illustrated schematically.

[0032] Figure 3 An example of the operation process of robot 100 is illustrated schematically.

[0033] Figure 4 An example of the hardware configuration of computer 1200 is illustrated schematically.

[0034] Figure 5An emotion graph 400 is shown, which maps various emotions.

[0035] Figure 6 An emotion map 900 is shown, which maps various emotions.

[0036] Figure 7 (A) is an appearance drawing of a plush toy according to other embodiments. Figure 7 (B) is a diagram of the internal structure of a plush toy.

[0037] Figure 8 This is a front view of the back of a plush toy as described in other embodiments.

[0038] Figure 9A The functional configuration of the robot 100 according to the second embodiment is schematically shown.

[0039] Figure 9B An example of the collection and processing operation flow performed by the robot 100 according to the second embodiment is shown schematically.

[0040] Figure 9C An example of the autonomous processing operation flow performed by the robot 100 according to the second embodiment is shown schematically.

[0041] Figure 9D The functional configuration of the plush toy 100N according to the third embodiment is shown schematically.

[0042] Figure 9E The functional configuration of the intelligent agent system 2500 according to the fourth embodiment is schematically shown.

[0043] Figure 9F This illustrates an example of the operation of an intelligent agent system.

[0044] Figure 9G This illustrates an example of the operation of an intelligent agent system.

[0045] Figure 9H The functional configuration of the smart glasses 2700 according to the fifth embodiment is schematically shown.

[0046] Figure 9I This illustrates one example of how an intelligent agent system can be used in smart glasses.

[0047] Figure 10A This is a block diagram illustrating an example of the configuration of the notification device 3010.

[0048] Figure 10B This is a diagram used to illustrate an example of the notification device 3010.

[0049] Figure 10CThis is a flowchart illustrating an example of the processing flow of the notification device 3010.

[0050] Figure 11A This is a block diagram showing an example of the configuration of the determination device 4010.

[0051] Figure 11B This is a diagram used to illustrate an example of the determination device 4010.

[0052] Figure 11C This is a flowchart illustrating an example of the processing flow of the determination device 4010. Detailed Implementation

[0053] The present invention will now be described through embodiments thereof, but these embodiments do not limit the invention as defined in the claims. Furthermore, not all combinations of features described in the embodiments are necessary for the solution of the invention. [First Implementation Method] Figure 1 An example of system 5 according to this embodiment is illustrated schematically. System 5 includes robot 100, robot 101, robot 102, and server 300. Users 10a, 10b, 10c, and 10d are users of robot 100. Users 11a, 11b, and 11c are users of robot 101. Users 12a and 12b are users of robot 102. It should be noted that in the description of this embodiment, users 10a, 10b, 10c, and 10d can be collectively referred to as user 10. Furthermore, users 11a, 11b, and 11c can be collectively referred to as user 11. Furthermore, users 12a and 12b can be collectively referred to as user 12. Robots 101 and 102 have substantially the same functions as robot 100. Therefore, system 5 will be described mainly based on the functions of robot 100.

[0054] Robot 100 can engage in conversations with user 10 or provide images to user 10. In this case, robot 100 collaborates with server 300 and other devices capable of communication via communication network 20 to engage in conversations with user 10 or provide images to user 10. For example, robot 100 not only autonomously learns appropriate conversational techniques but also collaborates with server 300 to learn more effectively in order to conduct conversations with user 10. Furthermore, robot 100 records image data of user 10 captured on camera to server 300 and requests image data from server 300 as needed, providing it to user 10.

[0055] Furthermore, robot 100 possesses emotional values ​​representing its own emotional types. For example, robot 100 has emotional values ​​representing various emotional intensities such as "joy," "anger," "sadness," "happiness," "pleasure," "displeasure," "peace," "unease," "sadness," "excitement," "worry," "relief," "fulfillment," "emptiness," and "neutrality." For instance, when robot 100 is in a state of high excitement during a conversation with user 10, it will speak at a faster pace. In this way, robot 100 is able to express its emotions through behavior.

[0056] Furthermore, robot 100 can be configured to determine the robot's behavior corresponding to the user 10's emotions by matching an article generation model using artificial intelligence (AI) with an emotion engine. Specifically, robot 100 can be configured to recognize the user 10's behavior, determine the user 10's emotions towards that behavior, and determine the robot's behavior corresponding to the determined emotions.

[0057] More specifically, when robot 100 recognizes user 10's behavior, it automatically generates the appropriate action content for robot 100 to respond to user 10's behavior using a pre-set text generation model. The text generation model can be interpreted as the algorithm and computation used for automatic dialogue processing derived from text. Such text generation models are publicly known technologies, as disclosed in, for example, Japanese Patent Application Publication No. 2018-081444 and chatGPT (internet search <URL: https: / / openai.com / blog / chatgpt>), and therefore their detailed description is omitted. This text generation model is constructed using a Large Language Model (LLM).

[0058] As described above, this embodiment can incorporate the emotions of user 10 and robot 100, as well as various linguistic information, into the behavior of robot 100 by combining a large language model with an emotion engine. In other words, according to this embodiment, a synergistic effect can be achieved by combining the article generation model with the emotion engine.

[0059] Furthermore, robot 100 has the function of recognizing the behavior of user 10. Robot 100 recognizes the behavior of user 10 by analyzing the facial image of user 10 acquired by the camera function and the voice of user 10 acquired by the microphone function. Robot 100 determines the action to be performed based on the recognized behavior of user 10.

[0060] Robot 100 stores rules for actions to be performed by Robot 100 based on User 10's emotions, Robot 100's emotions, and User 10's behavior, and performs various actions according to the rules.

[0061] Specifically, robot 100 has response rules for determining its behavior based on user 10's emotions, robot 100's emotions, and user 10's behavior. For example, if user 10's behavior is "laughing," then "laughing" is defined as robot 100's behavior. Furthermore, if user 10's behavior is "anger," then "apologizing" is defined as robot 100's behavior. Additionally, if user 10's behavior is "asking a question," then "answering" is defined as robot 100's behavior. Finally, if user 10's behavior is "sadness," then "starting a conversation" is defined as robot 100's behavior.

[0062] If robot 100 identifies user 10's behavior as "anger" based on reaction rules, it selects the "apology" behavior, as defined by the reaction rules, as the action to be performed by robot 100. For example, when the "apology" behavior is selected, robot 100 performs the "apology" action and outputs the sound of words expressing "apology".

[0063] In addition, when the conditions are met that the robot 100's emotion is "normal" (i.e., "joy" = 0, "anger" = 0, "sadness" = 0, "happiness" = 0) and the user 10's state is "alone and looks lonely", it is stipulated that the robot 100's emotion is "worry" and the behavior of "starting a conversation" can be executed.

[0064] If, based on reaction rules, robot 100 identifies its current emotion as "normal" and user 10 appears lonely and alone, then robot 100 increases its "sadness" emotion value. Furthermore, robot 100 selects the "start a conversation" action specified in the reaction rules as the action to be performed on user 10. For example, when selecting the "start a conversation" action, robot 100 will output the expression of concern, "What's wrong?", in a worried tone.

[0065] In addition, robot 100 sends user response information to server 300, which indicates that a positive response was received from user 10 through the behavior. The user response information includes, for example, user behavior of "anger", robot 100 behavior of "apology", situations where user 10's response is positive, and user 10's attributes.

[0066] Server 300 stores user response information received from robot 100. It should be noted that server 300 receives and stores user response information not only from robot 100, but also from robots 101 and 102. Next, server 300 parses the user response information from robots 100, 101, and 102, and updates the response rules accordingly.

[0067] Robot 100 queries server 300 for updated response rules and receives the updated response rules from server 300. Robot 100 then incorporates the updated response rules into its stored response rules. Thus, robot 100 is able to incorporate the response rules acquired by robots 101, 102, etc., into its own response rules.

[0068] Figure 2 The functional configuration of robot 100 is schematically shown. Robot 100 includes a sensor unit 200, a sensor module unit 210, a storage unit 220, a user state recognition unit 230, an emotion determination unit 232, a behavior recognition unit 234, a behavior determination unit 236, a storage control unit 238, a behavior control unit 250, a controlled object 252, and a communication processing unit 280.

[0069] The controlled object 252 includes a display device, a speaker, LEDs for the eyes, and motors for driving the arms, hands, and feet. The posture and movements of the robot 100 are controlled by controlling the motors for the arms, hands, and feet. Controlling these motors allows the robot 100 to express some of its emotions. Furthermore, facial expressions can be expressed by controlling the illumination state of the LEDs for the robot 100's eyes. It should be noted that the robot 100's posture, movements, and facial expressions are examples of the robot 100's attitude.

[0070] The sensor unit 200 includes a microphone 201, a 3D depth sensor 202, a 2D camera 203, and a distance sensor 204. The microphone 201 continuously detects sound and outputs sound data. It should be noted that the microphone 201 can be mounted on the head of the robot 100 and has binaural recording capabilities. The 3D depth sensor 202 detects the outline of an object by continuously illuminating an infrared pattern and analyzing the infrared image captured by the infrared camera. The 2D camera 203 is an example of an image sensor. The 2D camera 203 captures images using visible light, generating visible light image information. The distance sensor 204 detects the distance to an object by illuminating it with a laser or ultrasonic wave, for example. It should be noted that the sensor unit 200 may also include a clock, a gyroscope sensor, a touch sensor, a motor feedback sensor, etc.

[0071] It should be noted that, in Figure 2 The components of the robot 100 shown, excluding the controlled object 252 and the sensor unit 200, are examples of the components of the behavior control system of the robot 100. The behavior control system of the robot 100 uses the controlled object 252 as the controlled object.

[0072] Storage unit 220 includes reaction rules 221 and historical data 222. Historical data 222 includes the history of past emotional values ​​and behaviors of user 10. This history of emotional values ​​and behaviors is recorded for each user 10, for example, by associating it with the user 10's identification information. At least a portion of storage unit 220 is implemented using a storage medium such as a memory. It may also include a person database (DB) storing user 10's facial image, user 10's attribute information, etc. It should be noted that... Figure 2 Of the components of the robot 100 shown, the functions of the components other than the control object 252, the sensor unit 200, and the storage unit 220 can be implemented by the central processing unit (CPU) according to the program. For example, the functions of these components can be implemented as CPU operations through the operating system (OS) and the program running on the operating system.

[0073] The sensor module 210 includes a voice emotion recognition unit 211, a speech understanding unit 212, an expression recognition unit 213, and a face recognition unit 214. Information detected by the sensor unit 200 is input to the sensor module 210. The sensor module 210 analyzes the information detected by the sensor unit 200 and outputs the analysis result to the user state recognition unit 230.

[0074] The voice emotion recognition unit 211 of the sensor module 210 analyzes the voice of the user 10 detected by the microphone 201 to identify the user 10's emotions. For example, the voice emotion recognition unit 211 extracts features such as the frequency components of the voice, and identifies the user 10's emotions based on the extracted features. The speech understanding unit 212 analyzes the voice of the user 10 detected by the microphone 201 and outputs text information representing the content of the user 10's speech.

[0075] The expression recognition unit 213 recognizes the facial expressions and emotions of the user 10 based on images of the user 10 captured by the 2D camera 203. For example, the expression recognition unit 213 recognizes the user 10's facial expressions and emotions based on the shape and position of the eyes and mouth.

[0076] The face recognition unit 214 recognizes the face of user 10. The face recognition unit 214 recognizes user 10 by matching the facial images stored in the people database (illustration omitted) with the facial images of user 10 captured by the 2D camera 203.

[0077] The user state recognition unit 230 identifies the state of the user 10 based on the information parsed by the sensor module unit 210. For example, it uses the parsing results from the sensor module unit 210 to perform processing mainly related to perception. For example, it generates perception information such as "Dad is alone" and "There is a 90% probability that Dad is not smiling." It then performs processing to understand the meaning of the generated perception information. For example, it generates meaning information such as "Dad is alone and looks lonely."

[0078] The emotion determination unit 232 determines an emotion value representing the emotion of the user 10 based on the information parsed by the sensor module unit 210 and the state of the user 10 identified by the user state recognition unit 230. For example, the information parsed by the sensor module unit 210 and the identified state of the user 10 are input into a pre-learned neural network to obtain an emotion value representing the emotion of the user 10.

[0079] The emotion value representing user 10's feelings is a positive or negative value. For example, if the user's emotion is a cheerful emotion accompanied by pleasure or peace, such as "joy," "happiness," "pleasure," "peace of mind," "excitement," "relief," or "fulfillment," a positive value is displayed; the more cheerful the emotion, the larger the value. If the user's emotion is a depressed emotion, such as "anger," "sadness," "displeasure," "unease," "grief," "worry," or "emptiness," a negative value is displayed; the more depressed the emotion, the larger the absolute value of the negative value. If the user's emotion does not belong to any of the above categories ("normal"), a value of 0 is displayed.

[0080] Furthermore, the emotion determination unit 232 determines the emotion value representing the emotion of the robot 100 based on the information parsed by the sensor module unit 210 and the state of the user 10 identified by the user state recognition unit 230.

[0081] Robot 100's emotion value includes an emotion value for each of the multiple emotion categories, such as values ​​(0~5) representing the intensity of "joy", "anger", "sadness" and "happiness".

[0082] Specifically, the emotion determination unit 232 determines the emotion value representing the emotion of the robot 100 based on the rules for updating the emotion value of the robot 100, which correspond to the information parsed by the sensor module unit 210 and the state of the user 10 identified by the user state recognition unit 230.

[0083] For example, when the user state recognition unit 230 detects that the user 10 looks lonely, the emotion determination unit 232 increases the "sadness" emotion value of the robot 100. Furthermore, when the user state recognition unit 230 detects that the user 10 smiles, it increases the "joy" emotion value of the robot 100.

[0084] It should be noted that the emotion determination unit 232 can further consider the state of the robot 100 to determine the emotion value representing the robot 100's emotions. For example, when the robot 100's battery is low or when the robot 100's surrounding environment is dark, the "sadness" emotion value of the robot 100 can be increased. In addition, when the user 10 continues to talk to the robot despite the low battery, the "anger" emotion value can be increased.

[0085] The behavior recognition unit 234 recognizes the behavior of user 10 based on the information parsed by the sensor module unit 210 and the state of user 10 recognized by the user state recognition unit 230. For example, the information parsed by the sensor module unit 210 and the recognized state of user 10 are input into a pre-learned neural network to obtain the probabilities of multiple pre-set behavior categories (e.g., "laughing", "angry", "asking a question", "sad"), and the behavior category with the highest probability is recognized as the behavior of user 10.

[0086] As described above, in this embodiment, the robot 100 obtains the speech content of the user 10 based on the specific user 10. However, when obtaining and using the speech content, the robot 100 obtains the necessary consent from the user 10 in accordance with laws and regulations. In addition, the behavior control system of the robot 100 involved in this embodiment takes into account the protection of the user 10's personal information and privacy.

[0087] The behavior determination unit 236 determines the behavior corresponding to the behavior of the user 10 identified by the behavior recognition unit 234 based on the current emotion value of the user 10 determined by the emotion determination unit 232, historical data 222 of past emotion values ​​determined by the emotion determination unit 232 before determining the current emotion value of the user 10, and the emotion value of the robot 100. In this embodiment, the behavior determination unit 236 describes the case where the most recent emotion value included in the historical data 222 is used as the past emotion value of the user 10, but the disclosed technology is not limited to this aspect. For example, the behavior determination unit 236 may also use multiple recent emotion values, or it may also use the emotion value from a unit period such as one day ago as the past emotion value of the user 10. Furthermore, the behavior determination unit 236 considers not only the current emotion value of the robot 100, but also the history of the robot 100's past emotion values ​​to determine the behavior corresponding to the behavior of the user 10. The behavior determined by the behavior determination unit 236 includes gestures performed by the robot 100 or the content of the robot 100's speech.

[0088] The behavior determination unit 236 in this embodiment determines the behavior of the robot 100 as corresponding to the behavior of the user 10 based on the combination of the user 10's past and current emotional values, the robot 100's emotional value, the user 10's behavior, and the reaction rule 221. For example, when the user 10's past emotional value is positive and its current emotional value is negative, the behavior determination unit 236 determines the behavior used to change the user 10's emotional value to positive as corresponding to the user 10's behavior.

[0089] Response rule 221 specifies the combination of user 10's past and current sentiment values, robot 100's sentiment value, and the robot 100's behavior corresponding to user 10's behavior. For example, when user 10's past sentiment value is positive and current sentiment value is negative, and user 10's behavior is sadness, the rule specifies that the combination of gestures and verbal content used when making encouraging inquiries to user 10 will be the robot 100's behavior.

[0090] For example, in response rule 221, the behavior of robot 100 is defined based on the patterns of robot 100's emotional values ​​(1296 patterns, i.e., the four powers of six values ​​from "0" to "5" for "joy", "anger", "sadness", and "happiness"), the patterns of combinations of user 10's past and current emotional values, and all combinations of user 10's behavioral patterns. That is, for each pattern of robot 100's emotional values, such as combinations of user 10's past and current emotional values ​​such as negative and negative, negative and positive, positive and negative, positive and positive, negative and normal, and normal and normal, for each of the multiple combinations, the robot's behavior corresponding to user 10's behavioral pattern is defined. It should be noted that when user 10 utters a statement such as "I want to talk about the topic we discussed earlier," which aims to continue the conversation about the previous topic, behavior determination unit 236 can switch to using historical data 222 to determine the action pattern of robot 100's behavior.

[0091] It should be noted that in response rule 221, for each of the 1296 patterns of the robot 100's emotion value, at most one behavior of the robot 100 can be specified individually, and the behavior includes at least one of gestures and spoken content. Alternatively, in response rule 221, the behavior of the robot 100 can be specified for each group of patterns of the robot 100's emotion value, and the behavior includes at least one of gestures and spoken content.

[0092] The intensity of each gesture included in the behavior of robot 100 as specified in response rule 221 is preset. The intensity of each utterance included in the behavior of robot 100 as specified in response rule 221 is preset.

[0093] The storage control unit 238 determines whether to store data including the user 10's behavior in the historical data 222 based on the intensity of the behavior preset by the behavior determination unit and the emotion value of the robot 100 determined by the emotion determination unit 232.

[0094] Specifically, when the sum of the emotional values ​​of each of the multiple emotional categories for robot 100, the intensity of the gestures pre-set for the behavior determined by behavior determination unit 236, and the intensity of the speech content pre-set for the behavior determined by behavior determination unit 236, i.e., the comprehensive value of the intensity, is above a threshold, it is determined that the data including the behavior of user 10 will be stored in historical data 222.

[0095] When the storage control unit 238 determines that data including the behavior of user 10 will be stored in the historical data 222, the behavior determined by the behavior determination unit 236, the information parsed by the sensor module unit 210 from the current moment to a certain period in advance (e.g., all surrounding information such as sound, image, smell, etc. at the scene), and the state of user 10 identified by the user state recognition unit 230 (e.g., the expression, emotion, etc. of user 10) will be stored in the historical data 222.

[0096] The behavior control unit 250 controls the controlled object 252 based on the behavior determined by the behavior determination unit 236. For example, when the behavior determination unit 236 determines that the behavior includes speaking, the behavior control unit 250 causes the speaker included in the controlled object 252 to output sound. At this time, the behavior control unit 250 can determine the sound output speed based on the emotion value of the robot 100. For example, the higher the emotion value of the robot 100, the faster the sound output speed is determined by the behavior control unit 250. In this way, the behavior control unit 250 determines the execution mode of the behavior determined by the behavior determination unit 236 based on the emotion value determined by the emotion determination unit 232.

[0097] The behavior control unit 250 can recognize changes in the emotions of the user 10 relative to the behavior determined by the behavior determination unit 236. For example, changes in emotion can be recognized based on the user 10's voice or facial expression. Additionally, changes in the user 10's emotion can be recognized based on impacts detected by the touch sensor included in the sensor unit 200. When the touch sensor included in the sensor unit 200 detects an impact, it can be recognized that the user 10's emotion has worsened; or when the user 10's reaction is judged to be laughter, happiness, etc., based on the detection results of the touch sensor included in the sensor unit 200, it can be recognized that the user 10's emotion has improved. Information indicating the user 10's reaction is output to the communication processing unit 280.

[0098] Furthermore, after the behavior control unit 250 executes the behavior determined by the behavior determination unit 236 in an execution manner determined according to the emotion of the robot 100, the emotion determination unit 232 further changes the emotion value of the robot 100 based on the user's reaction to the execution of the behavior. Specifically, when the user does not react negatively to the behavior determined by the behavior determination unit 236 being executed in the manner determined by the behavior control unit 250, the emotion determination unit 232 increases the "joy" emotion value of the robot 100. Furthermore, when the user reacts negatively to the behavior determined by the behavior determination unit 236 being executed in the manner determined by the behavior control unit 250, the emotion determination unit 232 increases the "sadness" emotion value of the robot 100.

[0099] Furthermore, the behavior control unit 250 expresses the emotions of the robot 100 based on the determined emotion value of the robot 100. For example, when the "joy" emotion value of the robot 100 is increased, the behavior control unit 250 controls the controlled object 252 to make the robot 100 perform joyful actions. Conversely, when the "sadness" emotion value of the robot 100 is increased, the behavior control unit 250 controls the controlled object 252 to make the robot 100 adopt a dejected posture.

[0100] The communication processing unit 280 is responsible for communication with the server 300. As described above, the communication processing unit 280 transmits user response information to the server 300. Additionally, the communication processing unit 280 receives updated response rules from the server 300. When the communication processing unit 280 receives updated response rules from the server 300, it updates response rule 221.

[0101] Server 300 communicates with robots 100, 101, and 102, receives user response information sent from robot 100, and updates response rule 221 based on response rules including behaviors that elicit positive responses.

[0102] Figure 3 This schematically illustrates an example of an operational flow related to determining behavior in robot 100. (Repeated execution) Figure 3 The operation flow is shown below. At this point, it is assumed that the input includes information parsed by the sensor module 210. It should be noted that "S" in the operation flow indicates the step to be performed.

[0103] First, in step S100, the user status recognition unit 230 recognizes the status of the user 10 based on the information parsed by the sensor module unit 210.

[0104] In step S102, the emotion determination unit 232 determines an emotion value representing the emotion of the user 10 based on the information parsed by the sensor module unit 210 and the state of the user 10 identified by the user state recognition unit 230.

[0105] In step S103, the emotion determination unit 232 determines an emotion value representing the emotion of the robot 100 based on the information parsed by the sensor module unit 210 and the state of the user 10 identified by the user state recognition unit 230. The emotion determination unit 232 adds the determined emotion value of the user 10 to the historical data 222.

[0106] In step S104, the behavior recognition unit 234 identifies the behavior classification of user 10 based on the information parsed by the sensor module unit 210 and the state of user 10 identified by the user state recognition unit 230.

[0107] In step S106, the behavior determination unit 236 determines the behavior of the robot 100 based on the combination of the current emotional value of the user 10 determined in step S102 and the past emotional values ​​contained in the historical data 222, the emotional value of the robot 100, the behavior of the user 10 identified by the behavior recognition unit 234, and the reaction rule 221.

[0108] In step S108, the behavior control unit 250 controls the controlled object 252 based on the behavior determined by the behavior determination unit 236.

[0109] In step S110, the storage control unit 238 calculates a comprehensive value of the intensity based on the intensity of the behavior preset by the behavior determination unit and the emotion value of the robot 100 determined by the emotion determination unit 232.

[0110] In step S112, the storage control unit 238 determines whether the overall intensity value is above a threshold. If the overall intensity value is less than the threshold, the data, including the user 10's behavior, is not stored in the historical data 222, and the process ends. On the other hand, if the overall intensity value is above the threshold, the process proceeds to step S114.

[0111] In step S114, the behavior determined by the behavior determination unit 236, the information parsed by the sensor module unit 210 over a certain period from the current moment forward, and the status of the user 10 identified by the user status identification unit 230 are stored in the historical data 222.

[0112] As described above, according to robot 100, an emotion value representing the robot 100's emotions is determined based on the user's state, and based on the robot 100's emotion value, it is determined whether to store data including user 10's behavior in historical data 222. This reduces the capacity of historical data 222, which stores data including user 10's behavior. Then, for example, when robot 100 determines that the user's state is the same as it was 10 years ago, by reading historical data 222 from 10 years ago, robot 100 can present user 10 with the user's state 10 years ago (e.g., user 10's facial expressions, emotions, etc.), and even all surrounding information including ambient sounds, images, smells, etc.

[0113] Furthermore, according to robot 100, robot 100 can perform appropriate actions in response to user 10's behavior. Previously, user behavior was categorized to determine the robot's actions, including facial expressions and postures. In contrast, robot 100 determines user 10's current emotional state and performs actions based on past and current emotional states. Therefore, for example, if user 10 was energetic yesterday but is depressed today, robot 100 can say something like, "You were so energetic yesterday, what's wrong today?" Additionally, robot 100 can also use gestures to communicate. For example, if user 10 was depressed yesterday but is energetic today, the robot can say something like, "You were depressed yesterday, but you seem energetic today!" Furthermore, for example, if user 10 was energetic yesterday and is even more energetic today, robot 100 can say something like, "You're more energetic today than yesterday, has anything good happened compared to yesterday?" In addition, for example, for users who consistently maintain an emotional value above 0 and whose emotional value fluctuates within a certain range, Robot 100 can say things like, "Your emotions have been stable lately, and you feel good!"

[0114] Furthermore, for example, if robot 100 asks user 10, "Did you finish the homework we talked about yesterday?", and receives a reply of "Yes, I did!" from user 10, robot 100 can not only say affirmative words like "Great job!" but also make affirmative gestures such as clapping or giving a thumbs up. Additionally, for example, if user 10 says, "The demonstration I mentioned the day before yesterday was very successful!", robot 100 can not only say affirmative words like "Well done!" but also make the aforementioned affirmative gestures. In this way, by having robot 100 execute actions based on user 10's past state, it is expected that user 10 will develop a sense of closeness towards robot 100.

[0115] In the above embodiments, the use of the user 10's facial image to identify the user 10 has been described, but the disclosed technology is not limited to this aspect. For example, the robot 100 may use the user 10's voice, the user 10's email address, the user 10's social network service (SNS) identity document (ID), or an ID card with a built-in wireless IC tag held by the user 10 to identify the user 10.

[0116] It should be noted that robot 100 is an example of an electronic device equipped with a behavior control system. The behavior control system is not limited to robot 100; it can also be applied to various electronic devices. Furthermore, the functions of server 300 can be implemented by more than one computer. At least some of the functions of server 300 can be implemented by a virtual machine. Additionally, at least some of the functions of server 300 can be implemented in the cloud.

[0117] Figure 4 An example of the hardware configuration of a computer 1200 that functions as both a robot 100 and a server 300 is schematically shown. Programs installed in the computer 1200 enable it to function as one or more "parts" of the apparatus according to this embodiment, or to perform operations associated with the apparatus or those one or more "parts," and / or to execute processes or stages of those processes according to this embodiment. Such programs can be executed by the CPU 1212 to cause the computer 1200 to perform specific operations associated with some or all of the frames in the flowcharts and block diagrams described in this specification.

[0118] The computer 1200 according to this embodiment includes a CPU 1212, a RAM 1214, and a graphics controller 1216 interconnected via a host controller 1210. The computer 1200 also includes input / output units such as a communication interface 1222, a storage device 1224, a DVD drive 1226, and an IC card driver, which are connected to the host controller 1210 via an input / output controller 1220. The DVD drive 1226 may be a DVD-ROM drive or a DVD-RAM drive, etc. The storage device 1224 may be a hard disk drive or a solid-state drive, etc. The computer 1200 also includes a ROM 1230 and conventional input / output units such as a keyboard, which are connected to the input / output controller 1220 via an input / output chip 1240.

[0119] The CPU 1212 operates according to the program stored in the ROM 1230 and RAM 1214, thereby controlling the various units. The graphics controller 1216 obtains image data generated by the CPU 1212 from the frame buffer or other data provided in the RAM 1214 or from itself, and displays the image data on the display device 1218.

[0120] Communication interface 1222 communicates with other electronic devices via a network. Storage device 1224 stores programs and data used by CPU 1212 within computer 1200. DVD drive 1226 reads programs or data from DVD-ROM 1227, etc., and provides them to storage device 1224. IC card driver reads programs and data from IC card, and / or writes programs and data to IC card.

[0121] The ROM 1230 stores boot programs and other programs that are executed by the computer 1200 at startup, and / or programs that depend on the hardware of the computer 1200. The input / output chip 1240 can also connect various input / output units to the input / output controller 1220 via USB ports, parallel ports, serial ports, keyboard ports, mouse ports, etc.

[0122] The program is provided by a computer-readable storage medium such as a DVD-ROM 1227 or an IC card. The program is read from the computer-readable storage medium, installed in a storage device 1224, RAM 1214, or ROM 1230 (also examples of computer-readable storage media), and executed by the CPU 1212. The information processing described within these programs is read by the computer 1200, enabling cooperation between the program and the aforementioned hardware resources of various types. The apparatus or method can be configured to perform information manipulation or processing according to the use of the computer 1200.

[0123] For example, when communication is performed between computer 1200 and external devices, CPU 1212 can execute a communication program loaded into RAM 1214 and, based on the processing described in the communication program, command communication interface 1222 to perform communication processing. Under the control of CPU 1212, communication interface 1222 reads transmission data stored in a transmission buffer provided in a recording medium such as RAM 1214, storage device 1224, DVD-ROM 1227, or IC card, and sends the read transmission data to the network, or writes received data received from the network into a receive buffer provided on the recording medium, etc.

[0124] Additionally, the CPU 1212 can read all or necessary portions of files or databases stored in external recording media such as storage device 1224, DVD drive 1226 (DVD-ROM 1227), IC card, etc., into RAM 1214, and perform various types of processing on the data in RAM 1214. Next, the CPU 1212 can write the processed data back to the external recording medium.

[0125] Various types of information, such as programs, data, tables, and databases, can be stored in the recording medium for information processing. The CPU 1212 can perform various types of processing on data read from RAM 1214 and write the results back to RAM 1214. These various types of processing include operations, information processing, conditional judgments, conditional branching, unconditional branching, information retrieval / replacement, etc., specified by a sequence of program instructions and described throughout this disclosure. Furthermore, the CPU 1212 can retrieve information from files, databases, etc., within the recording medium. For example, if the recording medium stores multiple entries, each with an attribute value of a first attribute associated with a second attribute value, the CPU 1212 can retrieve from these multiple entries an entry that matches a condition specifying the first attribute value, and read the attribute value of the second attribute stored in that entry, thereby obtaining the attribute value of the second attribute associated with the first attribute that satisfies a preset condition.

[0126] The programs or software modules described above can be stored on or near the computer 1200 in a computer-readable storage medium. Alternatively, a recording medium such as a hard disk or RAM provided in a server system connected to a dedicated communication network or the Internet can be used as a computer-readable storage medium, thereby providing the program to the computer 1200 via the network.

[0127] In this embodiment, the blocks in the flowcharts and block diagrams may represent stages of a process for performing an operation or "parts" of a device that performs the operation. Specific stages and "parts" may be implemented by dedicated circuitry, programmable circuitry supplied together with computer-readable instructions stored on a computer-readable storage medium, and / or a processor supplied together with computer-readable instructions stored on a computer-readable storage medium. Dedicated circuitry may include digital and / or analog hardware circuitry, and may also include integrated circuits (ICs) and / or discrete circuitry. Programmable circuitry may include reconfigurable hardware circuitry such as field-programmable gate arrays (FPGAs) and field-programmable gate arrays (PLAs), which include logical AND, logical OR, logical XOR, logical NAND, logical NOR, and other logical operations, flip-flops, registers, and storage elements.

[0128] Computer-readable storage media can include any tangible device capable of storing instructions executable by a suitable device. As a result, a computer-readable storage medium having instructions stored in a tangible device comprises an article including the instructions, which can be executed to generate units for performing operations specified in a flowchart or block diagram. Examples of computer-readable storage media include electronic storage media, magnetic storage media, optical storage media, electromagnetic storage media, semiconductor storage media, etc. More specific examples of computer-readable storage media may include floppy disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), electrically erasable programmable read-only memory (EEPROM), static random-access memory (SRAM), compact disc read-only memory (CD-ROM), digital versatile disc (DVD), Blu-ray disc, memory sticks, integrated circuit cards, etc.

[0129] Computer-readable instructions may include any of the following: assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code described in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, JAVA (registered trademark), C++, and traditional procedural programming languages ​​such as the "C" programming language or similar programming languages.

[0130] Computer-readable instructions can be provided locally or via a wide area network (WAN) such as a local area network (LAN), the Internet, or other programmable data processing device to a processor or programmable circuit, causing the processor or programmable circuit to execute the computer-readable instructions to generate units for performing the operations specified in the flowchart or block diagram. Examples of processors include computer processors, processing units, microprocessors, digital signal processors, controllers, microcontrollers, etc.

[0131] The present invention has been described above using embodiments, but the technical scope of the present invention is not limited to the scope described in the above embodiments. Those skilled in the art should understand that various changes or improvements can be made to the above embodiments. As can be seen from the claims, embodiments with such changes or improvements may also be included within the technical scope of the present invention.

[0132] It should be noted that the execution order of actions, sequences, steps, and stages in the apparatus, systems, programs, and methods shown in the claims, specification, and drawings is not specifically stated as "before," "earlier," etc., or can be implemented in any order as long as the output of the preceding process is not used in the subsequent process. Even if the operational flow in the claims, specification, and drawings is described using terms such as "firstly," "next," etc., for convenience, it does not mean that it must be implemented in that order.

[0133] (Other implementation method 1) The robot 100 disclosed herein includes: an emotion determination unit that determines the emotion of a user or the emotion of the robot; and a behavior determination unit that, based on a dialogue function enabling the user to converse with the robot 100, generates behavioral content for the robot based on the user's behavior and the user's or robot 100's emotion, and determines the robot 100's behavior corresponding to the behavioral content. The behavior determination unit 236 can determine whether the user's behavior is dangerous by detecting the user's behavior, and if the user's behavior is dangerous, it generates first behavioral content to correct the user's behavior.

[0134] The first action may include at least one of performing a gesture to correct a dangerous behavior by a toddler, child, or other user, and playing a sound to correct that behavior. Hereinafter, the toddler, child, or other user will sometimes be referred to simply as the user.

[0135] Dangerous behaviors can include users climbing onto the edge of a window and attempting to open it, walking on a wall, and walking in a driveway.

[0136] Gestures for correcting dangerous behavior can include physical and hand gestures guiding the user to a specific location, or physical and hand gestures that cause the user to remain still in that location. Specific locations can include places other than where the user is currently located, such as the vicinity of the robot 100, or the interior space of a window.

[0137] Voices used to correct dangerous behavior can include "Please stop," or "Hey, that's dangerous, come over here." Voices used to correct dangerous behavior can also include "Don't move," or "Stay still."

[0138] The behavior determination unit 236 can determine whether the user's behavior has been corrected by detecting the user's behavior after the robot 100 performs a gesture as the first behavior content or plays a sound as the first behavior content. If the user's behavior has been corrected, it generates a second behavior content that is different from the first behavior content.

[0139] The situation where the user's behavior was corrected can be interpreted as the situation where, after the robot 100 performs its actions based on the first behavior content, the user stops the dangerous actions and behaviors, or the dangerous situation is eliminated.

[0140] The second action content may include at least one of the following: a voice praising the user's action, or a voice thanking the user for their action.

[0141] Praising a user's behavior can include phrases like "Are you alright? You're so obedient!" or "That was great, you did a fantastic job!" Expressing gratitude for a user's actions can include phrases like "Thank you for coming."

[0142] The behavior determination unit 236 can determine whether the user's behavior has been corrected by detecting the user's behavior after the robot 100 performs a gesture as the first behavior content or plays a sound as the first behavior content. If the user's behavior has not been corrected, it generates a third behavior content that is different from the first behavior content.

[0143] The failure to correct user behavior can be explained as a situation where, despite the robot 100 performing the action according to the first behavior content, the user continues to engage in dangerous actions and behaviors, or a dangerous situation is not eliminated.

[0144] The third action may include at least one of sending specific information to someone other than the user, performing a gesture that arouses the user's interest, playing a sound that arouses the user's interest, and playing an image that arouses the user's interest.

[0145] Sending specific information to persons other than the user can include sending emails containing warning messages to the user's guardians, caregivers, etc., and sending images (still images and moving images) of the user and their surrounding scenery. Additionally, sending specific information to persons other than the user can include sending warning audio messages.

[0146] Gestures that can attract user interest can include body movements and hand movements of the robot 100. Specifically, this could include swinging the arms dramatically and flashing the LEDs around the robot 100's eyes.

[0147] Playing sounds that will pique the user's interest can include specific music that the user likes, sounds like "Come here!" or "Let's play together!"

[0148] Videos that can be played to a user’s interest may include images of the user’s pets or images of the user’s parents.

[0149] According to the disclosed robot 100, it can detect whether a child or other individual is about to engage in dangerous behavior (such as climbing onto a window ledge and attempting to open a window). Upon detecting danger, it generates initial actions to correct the user's behavior. Thus, the robot 100 executes gestures and verbal cues such as "Please stop," or "Hey, it's dangerous, come over here." Furthermore, when prompting a child to stop the dangerous behavior through conversation, the robot 100 can also perform actions that praise the child, such as "Are you alright? You were so good." Additionally, if the dangerous behavior does not stop, the robot 100 can prompt the child to stop the dangerous behavior by sending warning emails to parents or caregivers, sharing the situation via video, simultaneously performing actions that interest the child, playing videos that interest the child, or playing music that interests the child.

[0150] The emotion determination unit 232 can determine the user's emotion based on a specific mapping. Specifically, the emotion determination unit 232 can determine the user's emotion based on an emotion map that serves as a specific mapping (see [link to relevant documentation]). Figure 5 To determine the user's emotions.

[0151] Figure 5This is a schematic diagram of an emotion map 400 that maps various emotions. In the emotion map 400, emotions are arranged radially from the center on concentric circles. The closer to the center of the concentric circles, the more primitive the emotion is. On the outer side of the concentric circles, emotions representing states and behaviors arising from mood are arranged. Emotion is a concept that also includes feelings and mental states. On the left side of the concentric circles, emotions arising from reactions occurring within the brain are usually arranged. On the right side of the concentric circles, emotions guided by situational judgments are usually arranged. Above and below the concentric circles, emotions arising from reactions occurring within the brain and guided by situational judgments are usually arranged. In addition, the emotion of "pleasure" is arranged above the concentric circles, and the emotion of "displeasure" is arranged below. Thus, in the emotion map 400, multiple emotions are mapped based on the structure of emotion generation, and emotions that are likely to arise simultaneously are mapped to the vicinity.

[0152] (1) For example, when the emotion engine of robot 100, which is the emotion determination unit 232, detects an emotion at approximately 100 msec, the frequency of determining the reaction action (e.g., agreement) of robot 100 can be set at least to the same time point as the detection frequency (100 msec) of the emotion engine, or it can be set to an earlier time point. The detection frequency of the emotion engine can be interpreted as the sampling rate.

[0153] By detecting emotions at approximately 100 ms and immediately executing corresponding actions (such as echoing), natural, empathetic dialogue can be achieved instead of unnatural echoing. Robot 100 executes actions (e.g., echoing) based on the direction and intensity of the mandala in the emotion map 400. It should be noted that the detection frequency (sampling rate) of the emotion engine is not limited to 100 ms and can be varied depending on the situation (e.g., during movement) and the user's age.

[0154] (2) The direction and intensity of the emotion can be preset by referring to the emotion chart 400, and the actions of agreement and the strength of agreement can be set. For example, when robot 100 feels stable and at ease, robot 100 nods and continues to listen. When robot 100 feels uneasy, confused or suspicious, robot 100 can tilt its head or stop shaking its head.

[0155] These emotions are distributed at the 3 o'clock position on the emotion diagram 400, and usually shift back and forth between peace and anxiety. In the right half of the emotion diagram 400, situational awareness is more dominant than internal feelings, thus forming an impression of calmness.

[0156] (3) When Robot 100 feels pleasure from praise, the filler word "Ah—" can be added before the dialogue; when it feels pain from harsh words, the filler word "Ooh!" can be added before the dialogue. In addition, physical reactions can also be included, such as the posture of Robot 100 squatting down while saying "Ooh!". These emotions are distributed around the 9 o'clock position of the emotion diagram 400.

[0157] (4) In the left half of the emotion map 400, internal feelings (responses) are more dominant than situation recognition. Therefore, it may give the impression of involuntary reactions.

[0158] When Robot 100 senses a feeling of acceptance (response) and also feels positive in the situation recognition, it can nod deeply while looking at the other party and also make "uh-huh" sounds. In this way, Robot 100 can generate equal goodwill towards the other party, that is, behaviors such as tolerance and forgiveness. Such emotions are distributed around the 12 o'clock position on the emotion graph 400.

[0159] Conversely, when Robot 100 experiences an internal feeling (reaction) of displeasure and also feels disgust in situation recognition, Robot 100 can shake its head. When it reaches the level of feeling hatred, it can turn the LEDs in its eyes red and stare at the other party. Such emotions are distributed around the 6 o'clock position on the emotion map 400.

[0160] (5) The inner side of the emotion diagram 400 represents the inner mind, and the outer side of the emotion diagram 400 represents behavior. Therefore, the closer to the outer side of the emotion diagram 400, the more visible the emotion becomes (manifested in behavior).

[0161] (6) When feeling at ease and listening to people around the 3 o'clock position on the emotional map 400, the robot 100 nods slightly and makes an "uh-huh" sound. However, when in a loving situation around the 12 o'clock position, it can make a strong nod, such as a deep nod.

[0162] The emotion determination unit 232 inputs the information parsed by the sensor module unit 210 and the identified state of the user 10 into a pre-learned neural network to obtain the emotion values ​​representing each emotion shown in the emotion map 400 and determine the emotion of the user 10. This neural network is pre-learned based on multiple learning data sets, which are combinations of the information parsed by the sensor module unit 210, the identified state of the user 10, and the emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, as... Figure 6 As shown in the sentiment graph 900, the neural network learns to make sentiments with similar configurations have similar values. Figure 6 The text shows examples of emotions such as "peace of mind," "stability," and "reassurance" with similar emotional values.

[0163] Furthermore, the emotion determination unit 232 can determine the emotion of the robot 100 based on a specific mapping. Specifically, the emotion determination unit 232 inputs the information parsed by the sensor module unit 210, the state of the user 10 identified by the user state recognition unit 230, and the state of the robot 100 into a pre-learned neural network to obtain the emotion values ​​representing each emotion shown in the emotion map 400 and determine the emotion of the robot 100. This neural network is pre-learned based on multiple learning data, which are combinations of the information parsed by the sensor module unit 210, the identified state of the user 10, the state of the robot 100, and the emotion values ​​representing each emotion shown in the emotion map 400. For example, the neural network learns based on the following learning data: learning data indicating that when the robot 100 is recognized as being touched by the user 10 from the output of the touch sensor (not shown), the emotion value of "happy" is "3", and learning data indicating that when the robot 100 is recognized as being slapped by the user 10 from the output of the accelerometer (not shown), the emotion value of "angry" is "3". Furthermore, as... Figure 6 As shown in the sentiment graph 900, the neural network learns to make sentiments with similar configurations have similar values.

[0164] The behavior determination unit 236 adds a fixed sentence for querying the robot's behavior content corresponding to the user's behavior to the text representing the user's behavior, the user's emotions, and the robot's emotions, and inputs it into an article generation model with dialogue function, thereby generating the robot's behavior content.

[0165] For example, the behavior determination unit 236 uses the emotion table shown in Table 1 to obtain text representing the state of the robot 100 based on the emotions of the robot 100 determined by the emotion determination unit 232. Here, in the emotion table, for each type of emotion, each emotion value is assigned an index number, and for each index number, text representing the state of the robot 100 is stored.

[0166] When the emotion of robot 100 determined by emotion determination unit 232 corresponds to index number "2", the text "very happy state" is obtained. It should be noted that when the emotion of robot 100 corresponds to multiple index numbers, multiple texts representing the state of robot 100 are obtained.

[0167] In addition, an emotion table as shown in Table 2 was prepared for user 10's emotions.

[0168] Here, if the user's behavior involves asking "Is it okay to go that way?", and the robot 100's emotion is index number "2" and the user 10's emotion is index number "3", then the article generation model is fed the input: "The robot is in a happy state. The user is in a generally happy state. The user asked 'Is it okay to go that way?'. As a robot, how should we respond?", and the robot's behavior content is obtained. The behavior determination unit 236 determines the robot's behavior based on this behavior content.

[0169] [Table 1]

[0170] [Table 2]

[0171] In this way, the behavior determination unit 236 determines the behavior content of the robot 100 based on the state of the robot 100's emotion, which is preset for each emotion type and each intensity of that emotion, and the behavior of the user 10. In this manner, the robot 100's utterances during conversations with the user 10 can be branched according to the state of the robot 100's emotion. That is, the robot 100 can change its behavior according to the index number corresponding to the robot's emotion, thus giving the user the impression that the robot has a mind, prompting them to initiate conversation or other actions.

[0172] Furthermore, the behavior determination unit 236 can add not only text representing the user's behavior, the user's emotions, and the robot's emotions, but also text representing the content of historical data 222. Based on this, it can add fixed sentences to inquire about the robot's behavior corresponding to the user's behavior, and input this information into a dialogue-enabled text generation model to generate the robot's behavior content. Thus, the robot 100 can change its behavior based on historical data representing the user's emotions and behaviors, thereby creating the impression that the robot has a personality and prompting the user to engage in conversation or other actions. In addition, the historical data may also include the robot's emotions and behaviors.

[0173] Furthermore, the emotion determination unit 232 can determine the emotion of the robot 100 based on the behavioral content of the robot 100 generated by the article generation model. Specifically, the emotion determination unit 232 inputs the behavioral content of the robot 100 generated by the article generation model into a pre-learned neural network, obtains the emotion values ​​representing each emotion shown in the emotion graph 400, integrates the obtained emotion values ​​representing each emotion with the emotion values ​​representing each emotion of the current robot 100, and updates the emotion of the robot 100. For example, the obtained emotion values ​​representing each emotion and the emotion values ​​representing each emotion of the current robot 100 are averaged and integrated respectively. This neural network is pre-learned based on multiple learning data, which are combinations of text representing the behavioral content of the robot 100 generated by the article generation model and emotion values ​​representing each emotion shown in the emotion graph 400.

[0174] For example, when the robot 100's speech content "That's great. How lucky!" is obtained as the behavioral content of the robot 100 generated by the article generation model, the text representing this speech content is input into the neural network, which will obtain a higher value as the emotion "joy", and update the robot 100's emotion, making the emotion "joy" higher.

[0175] It should be noted that the robot 100 can be mounted on a plush toy, or it can be used in a control device that is wirelessly or wiredly connected to a control object device (speaker or camera) mounted on the plush toy. In this case, specifically, the configuration is as follows. For example, the robot 100 can also be used as a cohabitant (specifically, a person who, while spending daily life with user 10, advances dialogue based on information relevant to user 10's daily life) or provides information that matches user 10's interests and preferences. Figure 7 and Figure 8 (The plush toy 100N shown). In this embodiment (other embodiments), an example of applying the control part of the robot 100 described above to a smartphone 50 will be described.

[0176] The plush toy 100N is equipped with an input / output device for the robot 100. The smartphone 50, which functions as the control part of the robot 100, is detachable. Inside the plush toy 100N, the input / output device is connected to the stored smartphone 50.

[0177] like Figure 7 As shown in (A), in this embodiment (the embodiment mounted on a plush toy), the plush toy 100N has the appearance of a bear covered with soft fabric, and as... Figure 7As shown in (B), within the space 52 formed inside, the microphone 201 of the sensor section 200 serves as an input / output device (see [reference]). Figure 2 The 2D camera 203 of the sensor unit 200 is configured in the part corresponding to ear 54 (see...). Figure 2 The part corresponding to eye 56 is configured and constitutes control object 252 (see...). Figure 2 The speaker 60, which is part of the mouth 58, is positioned within the microphone 201. It should be noted that the microphone 201 and speaker 60 are not necessarily separate units; they can also be an integrated unit. If it is an integrated unit, it can be positioned at a location such as the nose of the plush toy 100N, where it can naturally hear the speech. It should be noted that although the example given is of the plush toy 100N in the shape of an animal, it is not limited to this. The plush toy 100N can be in the shape of a specific character.

[0178] Smartphone 50 has such Figure 2 The functions shown are as follows: sensor module 210, storage unit 220, user state recognition unit 230, emotion determination unit 232, behavior recognition unit 234, behavior determination unit 236, storage control unit 238, behavior control unit 250, and communication processing unit 280.

[0179] like Figure 8 As shown, the structure is as follows: a zipper 62 is installed on a part (e.g., the back) of the plush toy 100N, and by opening the zipper 62, the outside is connected to the space 52.

[0180] Here, the smartphone 50 is externally stored in the space 52, via the USB hub 64 (see Figure 7 (B) Connects to each input / output device via USB, thereby enabling... Figure 1 It has the same functions as the robot 100 shown.

[0181] In addition, a contactless power receiver 66 is connected to the USB hub 64. The power receiver 66 is equipped with a power receiving coil 66A. The power receiver 66 is an example of a wireless power receiver that receives wireless power.

[0182] The receiving plate 66 is positioned near the base 68 of the two legs of the plush toy 100N, and when the plush toy 100N is placed on the mounting base 70, the receiving plate 66 is located closest to the mounting base 70. The mounting base 70 is an example of an external wireless power transmission unit.

[0183] The plush toy 100N placed on the mounting base 70 can be displayed as an ornament in its natural state.

[0184] In addition, the root is formed to be thinner than the surface thickness of other parts of the plush toy 100N, so as to be held in a state closer to that of the mounting base 70.

[0185] A charging pad 72 is provided on the mounting base 70. The charging pad 72 is equipped with a power supply coil 72A. The power supply coil 72A sends a signal to search for the power receiving coil 66A of the power receiving board 66. When the power receiving coil 66A is found, current flows through the power supply coil 72A, generating a magnetic field. The power receiving coil 66A responds to the magnetic field and begins electromagnetic induction. Thus, current flows through the power receiving coil 66A, and the power is stored in the battery (not shown) of the smartphone 50 via the USB hub 64.

[0186] That is, by placing the plush toy 100N as an ornament on the mounting base 70, the smartphone 50 will automatically charge, so there is no need to remove the smartphone 50 from the space 52 of the plush toy 100N for charging.

[0187] It should be noted that in this embodiment (the embodiment mounted on a plush toy), the smartphone 50 is housed within the space 52 of the plush toy 100N and connected via a wired connection (USB connection), but is not limited thereto. For example, a control device with wireless functionality (e.g., "Bluetooth" (registered trademark)) may also be housed within the space 52 of the plush toy 100N and connected to a USB hub 64. In this case, the smartphone 50 is not placed within the space 52; the smartphone 50 communicates wirelessly with the control device; and the external smartphone 50 connects to various input / output devices via the control device, thereby enabling communication with… Figure 1 It has the same functions as the robot 100 shown. In addition, the control device, which is housed in the space 52 of the plush toy 100N, can also be connected to an external smartphone 50 via a wired connection.

[0188] Furthermore, in this embodiment (the embodiment mounted on a plush toy), a bear plush toy 100N is exemplified, but it can also be other animals, dolls, or the shape of a specific character. Additionally, it can be costumed. Furthermore, the surface material is not limited to fabric; it can be other materials such as soft vinyl, but a soft material is preferred.

[0189] Furthermore, a display can be installed on the surface of the plush toy 100N, thereby adding a control object 252 that provides information to the user 10 visually. For example, the eyes 56 can be used as a display, expressing emotions through images projected onto the eyes, or a window can be provided on the abdomen to reveal the display of the built-in smartphone 50. Additionally, the eyes 56 can function as a projector, expressing emotions through images projected onto a wall.

[0190] According to other embodiments, an existing smartphone 50 is placed in a plush toy 100N, and a camera 203, microphone 201, speaker 60, etc. are extended to appropriate positions from there via a USB connection.

[0191] In addition, for wireless charging, the smartphone 50 is connected to the power receiving board 66 via USB, and the power receiving board 66 is configured to be as close as possible to the outside when viewed from inside the plush toy 100N.

[0192] When wireless charging of smartphone 50 is to be used, smartphone 50 must be positioned as far out as possible when viewed from inside plush toy 100N, resulting in an uneven feel when touching plush toy 100N from the outside.

[0193] Therefore, the smartphone 50 is positioned as centrally as possible within the plush toy 100N, and the wireless charging function (power receiving plate 66) is positioned as far outward as possible when viewed from inside the plush toy 100N. The camera 203, microphone 201, speaker 60, and smartphone 50 receive wireless power via the power receiving plate 66.

[0194] (Other implementation method 2) The robot 100 in this embodiment includes: an emotion determination unit that determines the emotion of a user or the emotion of the robot; and a behavior determination unit that, based on an article generation model having a dialogue function that enables the user to engage in dialogue with the robot, generates behavioral content for the robot based on the user's behavior and the emotion of the user or the robot, and determines the robot's behavior corresponding to the behavioral content. The behavior determination unit 236 is configured to receive statements from multiple users in an ongoing conversation, and when a statement reaches a preset state, it summarizes the content of the statement and determines it as the behavior of the robot 100.

[0195] Specifically, robot 100 is set up in a meeting room or other location where multiple users are speaking. Then, robot 100 in this embodiment acquires the speech of multiple users in an ongoing conversation via its microphone function. Robot 100 then stores the speech so far.

[0196] When the status of a received speech becomes a pre-set state, the behavior determination unit 236 summarizes the content of that speech. Here, the pre-set state includes a state where no more speeches are received within a pre-set time. That is, if multiple users do not speak within a pre-set time, such as 5 minutes, it is determined that the meeting has reached a stalemate, failing to generate good ideas, and is in a state of silence. Therefore, the meeting is summarized by summarizing the content of the speeches up to this point. Here, the summary of the speech content is performed using existing technology, including extracting and outputting frequently occurring terms in the speeches.

[0197] Furthermore, the pre-defined states include those where the same term has been received a pre-set number of times in speeches. That is, when the same term is received a pre-set number of times, it is determined that the same topic is being repeated repeatedly in the meeting, indicating a lack of new ideas. Therefore, the meeting is summarized by outlining the content of the speeches so far. It should be noted that the meeting data is pre-input into the article generation model, and terms recorded in this data, which are expected to appear frequently, can be pre-excluded from the frequency count.

[0198] With this configuration, even meetings that have reached a stalemate can be summarized to organize the meeting's themes and arguments.

[0199] (Other implementation method 3) The behavior determination unit 236 generates robot behavior content based on the dialogue function that enables the user to converse with the robot, taking into account the user's behavior and the user's or robot's emotions, and determines the robot's behavior corresponding to the behavior content. In this case, the robot is located at customs. The behavior determination unit 236 acquires images of people using an image sensor and obtains odor detection results using an odor sensor. If it detects pre-set abnormal behavior, abnormal facial expressions, or abnormal odors, it notifies the tax authorities to confirm these as robot behavior.

[0200] Specifically, robot 100 is deployed at customs to inspect passing customers. Furthermore, robot 100 pre-stores odor data for drugs and explosives, as well as data related to the behavior, expressions, and suspicious actions of criminals. When a customer passes through, the behavior determination unit 236 uses an image sensor to acquire an image of the customer and an odor sensor to acquire odor detection results. If suspicious behavior, suspicious expressions, the smell of drugs, or the smell of explosives are detected, the unit notifies the tax authorities to determine the behavior as that of robot 100.

[0201] (Other implementation method 4) The robot 100 of this embodiment (in this embodiment, it is equivalent to the smartphone 50 stored in the plush toy 100N) performs the following processing.

[0202] The robot 100 (in this embodiment, equivalent to the smartphone 50 stored in the plush toy 100N) performs the following steps 1 to 5 to determine the special fraud risk based on the conversation content between the user and the conversation partner, as well as the emotions of the conversation partner.

[0203] (Step 1) Robot 100 obtains the conversation content between user 10 and the conversation object.

[0204] Specifically, the speech understanding unit 212 analyzes the voice of user 10 and the voice of the conversation partner detected by microphone 201, and outputs text information representing the conversation content between user 10 and the conversation partner. It should be noted that robot 100 identifies user 10 and the conversation partner as multiple users 10.

[0205] (Step 2) Robot 100 acquires the emotional value of the conversation object. Specifically, it acquires the voice of the conversation object from the telephone or walkie-talkie, as well as the image of the conversation object displayed on the walkie-talkie screen, and performs the same processing as steps S100 to S102 above to acquire the emotional value of the conversation object.

[0206] (Step 3) Robot 100 determines special fraud risks based on the conversation content obtained in Step 1 and the sentiment value of the conversation object obtained in Step 2.

[0207] Specifically, the behavior determination unit 236 determines the similarity between the session content and the specific fraud case by comparing the data of past specific fraud cases stored in the storage unit 220 with the session content. Then, the behavior determination unit 236 determines the degree of specific fraud risk based on the similarity between the session content and the specific fraud case, as well as the emotional value of the session audience. As an example, when the similarity between the session content and the specific fraud case is high, the behavior determination unit 236 determines the degree of specific fraud risk to be high regardless of the emotional value of the session audience. Furthermore, even when the similarity between the session content and the specific fraud case is not so high, if the emotional value of the session audience's "anxiety" or "excitement" is high, the behavior determination unit 236 still determines the degree of specific fraud risk to be high.

[0208] (Step 4) Robot 100 determines its behavior based on the degree of specific fraud risk determined in step 3.

[0209] Specifically, if the level of the specific fraud risk determined in step 3 exceeds a set threshold, the behavior determination unit 236 determines to communicate a behavior with a high specific fraud risk. For example, the behavior determination unit 236 may determine to communicate a behavior with a high specific fraud risk to user 10. Furthermore, the behavior determination unit 236 may determine to communicate a behavior with a high specific fraud risk to user 10's family members. Additionally, the behavior determination unit 236 may determine to immediately report a behavior with a high specific fraud risk to the police. These behaviors can be appropriately determined based on the level of the specific fraud risk.

[0210] (Step 5) Robot 100 performs the behavior determined in step 4.

[0211] Specifically, the behavior control unit 250 controls the speaker of the device being controlled, so that the aforementioned message is output as sound from the speaker.

[0212] In this way, Robot 100 can perform special fraud risk assessment based on the content of the conversation between the user and the other user, as well as the other user's emotions.

[0213] (Other implementation method 5) The robot 100 of this embodiment (in this embodiment, it is equivalent to the smartphone 50 stored in the plush toy 100N) performs the following processing.

[0214] The robot 100 (in this embodiment, equivalent to the smartphone 50 stored in the plush toy 100N) performs the processing of detecting specific cases through the following steps 1 to 6, based on the conversation content of multiple users, the status of multiple users, and the reactions of multiple users.

[0215] (Step 1) Robot 100 obtains the conversation content of multiple users 10.

[0216] Specifically, the speech understanding unit 212 analyzes the voices of multiple users 10 detected by the microphone 201 and outputs text information representing the conversation content of the multiple users 10.

[0217] (Step 2) Robot 100 obtains the sentiment values ​​of multiple users 10.

[0218] Specifically, the voices and images of multiple users 10 are acquired, and the same processing as steps S100 to S102 above is performed to acquire the emotional values ​​of multiple users 10.

[0219] (Step 3) Based on the conversation content of multiple users 10 obtained in Step 1 and the sentiment values ​​of multiple users 10 obtained in Step 2, the robot 100 determines whether specific cases such as "bullying", "crime" and "harassment" have occurred.

[0220] Specifically, the behavior determination unit 236 determines the similarity between the conversation content and specific cases by comparing data of past cases such as "bullying," "crime," and "harassment" stored in the storage unit 220 with the conversation content of multiple users 10. Then, based on the similarity between the conversation content and the specific case, and the sentiment values ​​of the multiple users 10, the behavior determination unit 236 determines the probability of the specific case occurring. As an example, when the similarity between the conversation content and the specific case is high, and the sentiment values ​​of multiple users 10 for "anger," "sadness," "displeasure," "anxiety," "grief," "worry," and "emptiness" are high, the behavior determination unit 236 determines the probability of the specific case occurring to be a high value.

[0221] (Step 4) Robot 100 determines its behavior based on the probability of a particular case occurring as determined in step 3.

[0222] Specifically, if the probability of a particular case occurring, as determined in step 3, exceeds a set threshold, then the behavior determination unit 236 determines an action that conveys the high probability of the particular case occurring. For example, the behavior determination unit 236 may determine that the high probability of the particular case occurring is conveyed via email to the administrators of the organizations to which multiple users 10 belong.

[0223] (Step 5) Robot 100 performs the behavior determined in step 4.

[0224] Specifically, the aforementioned email is sent to the manager from the smartphone 50, which functions as the behavior control unit 250. This email may contain a session log corresponding to a specific case, a anticipated case, the probability of that case occurring, and suggested solutions for that case.

[0225] (Step 6) The robot 100 stores the results of the actions performed in step 5 in the storage unit 220. Specifically, the storage control unit 238 stores information such as whether a specific case occurred and the resolution status in the historical data 222. In this way, by providing feedback on whether a specific case occurred and the resolution status, the detection accuracy and solution suggestions for that specific case can be improved.

[0226] In this way, Robot 100 can perform the processing of specific cases based on the conversation content of multiple users, the status of multiple users, and the reactions of multiple users.

[0227] (Other implementation method 6) The robot 100 of this embodiment (in this embodiment, it is equivalent to the smartphone 50 stored in the plush toy 100N) performs the following processing.

[0228] The robot 100 (in this embodiment, equivalent to the smartphone 50 stored in the plush toy 100N) performs childcare and supervision-related processes according to the user's (child's) preferences, the user's condition, and the user's reaction through the following steps 1 to 5-2.

[0229] (Step 1) Robot 100 acquires the state of user 10, the emotional value of user 10, the emotional value of robot 100, and historical data 222. Specifically, the same processing as steps S100~S103 is performed to acquire the state of user 10, the emotional value of user 10, the emotional value of robot 100, and historical data 222.

[0230] (Step 2) Robot 100 obtains user 10's preferences for animation and music.

[0231] Specifically, the behavior determination unit 236 determines the act of asking the user 10 about their preferences for animation and music as an action of the robot 100, and the behavior control unit 250 controls the controlled object 252 to ask the user 10 about their preferences for animation and music. The user state recognition unit 230 recognizes the user 10's preferences for animation and music based on information parsed by the sensor module unit 210 (e.g., the user's answer).

[0232] (Step 3) Robot 100 determines the animation and music works to recommend to user 10.

[0233] Specifically, the behavior determination unit 236 adds a fixed sentence, "What animations and music should be recommended to the user at this time?", to the text representing the user 10's preferences for animations and music, the user 10's emotions, the robot 100's emotions, and the content stored in the historical data 222, and inputs this information into the article generation model to obtain recommended content about animations and music. At this point, by considering not only the user 10's preferences for animations and music, but also the user 10's emotions and the historical data 222, suitable animations and music can be suggested for the user 10. Furthermore, by considering the robot 100's emotions, the user 10 can perceive that the robot 100 possesses emotions.

[0234] (Step 4) Robot 100 suggests the animation and music works determined in step 3 to user 10 and obtains user 10's reaction.

[0235] Specifically, the behavior determination unit 236 determines the statement suggesting animations and music to the user 10 as the behavior of the robot 100, and the behavior control unit 250 controls the controlled object 252 to send the statement suggesting animations and music to the user 10. The user state recognition unit 230 recognizes the state of the user 10 based on the information parsed by the sensor module unit 210, and the emotion determination unit 232 determines the emotion value representing the emotion of the user 10 based on the information parsed by the sensor module unit 210 and the state of the user 10 recognized by the user state recognition unit 230.

[0236] The behavior determination unit 236 determines whether the user 10's reaction is positive based on the user 10's state identified by the user state recognition unit 230 and the emotion value representing the user 10's emotions, and determines whether the user 10's behavior is to perform the process of providing suggestions to the user 10 with animation and music works, or to suggest other animation and music works to the user 10.

[0237] (Step 5-1) If the user 10's response is positive, the robot 100 will proceed with the process of providing suggested animations and music.

[0238] Specifically, when the processing of providing suggested animations and music to user 10 is determined as the behavior of robot 100, behavior control unit 250 controls the speaker or display, which is the controlled object 252, to perform the suggested animations and music to user 10.

[0239] (Step 5-2) If user 10’s response is not positive, robot 100 determines other animations and music to recommend to user 10.

[0240] Specifically, when recommending other animations and music to user 10 is determined to be an action of robot 100, the behavior determination unit 236 adds a fixed sentence, "What animations and music should be recommended to the user at this time?", to the text representing user 10's preferences for animations and music, user 10's emotions, robot 100's emotions, and the content stored in historical data 222, and inputs it into the article generation model to obtain recommended content about animations and music. Then, it returns to step 4 above and repeats the processing of steps 4 to 5-2 above until it is determined that the process of providing recommended animations and music to user 10 will be performed.

[0241] In addition, while acting as a playmate for user 10 (child) through steps 1 to 5-2 above, robot 100 performs other care and supervision-related procedures according to the user's condition and reaction through steps 11 to 13.

[0242] (Step 11) Robot 100 identifies the status of user 10.

[0243] Specifically, the user status recognition unit 230 identifies the status of the user 10 based on the information parsed by the sensor module unit 210.

[0244] (Step 12) Robot 100 determines whether the state of user 10 obtained in step 11 is an abnormal state (crying, falling down, not moving for a certain period of time, entering a dangerous place).

[0245] Specifically, the behavior determination unit 236 determines whether the state of user 10 obtained in step 11 is in an abnormal state by comparing the past state of user 10 contained in historical data 222 with the current state of user 10.

[0246] (Step 13) If user 10 is in an abnormal state, robot 100 will issue an alarm.

[0247] Specifically, the behavior control unit 250 controls the speaker of the controlled object 252 to issue an alarm. It should be noted that if the user 10 is not in an abnormal state, the robot 100 ends the process and repeats the processing of steps 11 and 12.

[0248] In this way, robot 100 can perform childcare and monitoring-related procedures based on the user's preferences, the user's condition, and the user's reactions. Thus, for example, it can warn user 10 of approaching danger while acting as a playmate or while monitoring user 10 as they sleep.

[0249] [Second Implementation] Figure 1 An example of system 5 according to this embodiment is illustrated schematically. System 5 includes robot 100, robot 101, robot 102, and server 300. Users 10a, 10b, 10c, and 10d are users of robot 100. Users 11a, 11b, and 11c are users of robot 101. Users 12a and 12b are users of robot 102. It should be noted that in the description of this embodiment, users 10a, 10b, 10c, and 10d are sometimes collectively referred to as user 10. In addition, users 11a, 11b, and 11c are sometimes collectively referred to as user 11. In addition, users 12a and 12b are sometimes collectively referred to as user 12. Robots 101 and 102 have substantially the same functions as robot 100. Therefore, system 5 will be described mainly based on the functions of robot 100.

[0250] Here, users 10a, 10b, 10c, and 10d constitute a family member. In other words, users 10a, 10b, 10c, and 10d are persons who constitute a family member. Furthermore, users 10a to 10d may also include caregivers who perform care. For example, if user 10a is the caregiver, care can be provided to a person outside the family (the user), or to user 10b who is a family member. This person outside the family (the user) or user 10b is the recipient of care.

[0251] It should be noted that, as described later, robot 100 provides care-related advice to user 10. However, if user 10a, acting as a caregiver, provides care to someone outside the family, then user 10 may not be a family member. Similarly, if user 10b, acting as the recipient of care, receives care from someone outside the family (the user), then user 10 may not be a family member. Furthermore, as described later, robot 100 provides user 10 with advice related to the health of family members and advice related to their mental state; in this case, user 10 may not include either a caregiver or a recipient.

[0252] Robot 100 can engage in conversations with user 10 or provide images to user 10. In this case, robot 100 collaborates with server 300 and other devices capable of communication via communication network 20 to engage in conversations with user 10 or provide images to user 10. For example, robot 100 not only autonomously learns appropriate conversational techniques but also collaborates with server 300 to learn more effectively in order to conduct conversations with user 10. Furthermore, robot 100 records image data captured from user 10 to server 300 and requests image data from server 300 as needed, thereby providing it to user 10.

[0253] Furthermore, robot 100 possesses emotional values ​​representing its own emotional types. For example, robot 100 has emotional values ​​representing various emotional intensities such as "joy," "anger," "sadness," "happiness," "pleasure," "displeasure," "peace," "unease," "sadness," "excitement," "worry," "relief," "fulfillment," "emptiness," and "neutrality." For instance, when robot 100 is in a state of high excitement during a conversation with user 10, it will speak at a faster pace. In this way, robot 100 can express its emotions through behavior.

[0254] Furthermore, robot 100 can be configured to determine the robot's behavior corresponding to user 10's emotions by matching an article generation model using artificial intelligence (AI) with an emotion engine. Specifically, robot 100 can be configured to recognize user 10's behavior, determine user 10's emotions towards that behavior, and determine the robot's behavior corresponding to the determined emotions.

[0255] More specifically, when robot 100 recognizes user 10's behavior, it automatically generates the appropriate action content for robot 100 to respond to user 10's behavior using a pre-set text generation model. The text generation model can be interpreted as the algorithm and computation used for automatic dialogue processing derived from text. Such text generation models are publicly known technologies, as disclosed in, for example, Japanese Patent Application Publication No. 2018-081444 and chatGPT (internet search <URL: https: / / openai.com / blog / chatgpt>), and therefore their detailed description is omitted. This text generation model is constructed using a Large Language Model (LLM).

[0256] As described above, this embodiment can incorporate the emotions and various linguistic information of user 10 and robot 100 into the behavior of robot 100 by combining a large language model with an emotion engine. In other words, according to this embodiment, a synergistic effect can be achieved by combining the article generation model with the emotion engine.

[0257] Furthermore, robot 100 has the function of recognizing the behavior of user 10. Robot 100 recognizes the behavior of user 10 by analyzing the facial image of user 10 acquired by the camera function and the voice of user 10 acquired by the microphone function. Robot 100 determines the action to be performed based on the recognized behavior of user 10.

[0258] As an example of a behavior determination model, robot 100 stores rules for behaviors to be performed by robot 100 based on user 10's emotions, robot 100's emotions, and user 10's behaviors, and performs various behaviors according to the rules.

[0259] Specifically, robot 100 has response rules for determining its behavior based on user 10's emotions, robot 100's emotions, and user 10's behavior, as an example of a behavior determination model. In the response rules, for example, if user 10's behavior is "laughing," then "laughing" is defined as robot 100's behavior. Furthermore, if user 10's behavior is "anger," then "apologizing" is defined as robot 100's behavior. Additionally, if user 10's behavior is "asking a question," then "answering" is defined as robot 100's behavior. Finally, if user 10's behavior is "sadness," then "starting a conversation" is defined as robot 100's behavior.

[0260] If robot 100 identifies user 10's behavior as "anger" based on reaction rules, it selects the "apology" behavior, as defined by the reaction rules, as the action to be performed by robot 100. For example, when robot 100 selects the "apology" behavior, it will perform the "apology" action and output a voice message expressing "apology".

[0261] In addition, when the conditions are met that the robot 100's emotion is "normal" (i.e., "joy" = 0, "anger" = 0, "sadness" = 0, "happiness" = 0) and the user 10's state is "alone and looks lonely", it is stipulated that the robot 100's emotion is "worry" and the behavior of "starting a conversation" can be executed.

[0262] If, based on reaction rules, robot 100 identifies its current emotion as "normal" and user 10 appears lonely and alone, then robot 100 increases its "sadness" emotion value. Furthermore, robot 100 selects the "start a conversation" action specified in the reaction rules as the action to be performed on user 10. For example, if the "start a conversation" action is selected, robot 100 will output the expression of concern, "What's wrong?", in a worried tone.

[0263] In addition, robot 100 sends user response information to server 300, which indicates that a positive response was received from user 10 through the behavior. The user response information includes, for example, user behavior of "anger", robot 100 behavior of "apology", situations where user 10's response is positive, and user 10's attributes.

[0264] Server 300 stores user response information received from robot 100. It should be noted that server 300 receives and stores user response information not only from robot 100, but also from robots 101 and 102. Next, server 300 parses the user response information from robots 100, 101, and 102, and updates the response rules accordingly.

[0265] Robot 100 queries server 300 for updated response rules and receives the updated response rules from server 300. Robot 100 then incorporates the updated response rules into its stored response rules. Thus, robot 100 is able to incorporate the response rules obtained by robots 101, 102, etc., into its own response rules.

[0266] The robot 100 involved in this embodiment can provide nursing-related advice information. The robot 100 provides nursing-related advice information to users 10, including caregivers and those being cared for, but is not limited to this. For example, it can also provide it to any user, such as family members, including at least one of the caregivers and those being cared for.

[0267] Specifically, robot 100 identifies the physical and mental state of user 10, who includes at least one of the caregiver and the person being cared for. These physical and mental states of user 10 include, for example, the user's stress level and fatigue level. Robot 100 provides care-related suggestions corresponding to the identified physical and mental states of user 10. As an example, when robot 100 infers that user 10's stress level or fatigue level is high based on user 10's behavior, it initiates a conversation with user 10. Specifically, robot 100 makes statements such as "Here are some suggestions regarding care," indicating that it will provide advice.

[0268] Next, based on the identified physical and mental state of user 10 (such as stress level and fatigue level), robot 100 generates care-related suggestions. These suggestions include, but are not limited to, methods for maintaining a positive attitude towards care, methods for relieving stress, and relaxation techniques—information related to restoring user 10's physical and mental well-being. For example, robot 100 might say and provide suggestions such as, "It seems you have accumulated stress (fatigue). We suggest you move your body by stretching," which aligns with user 10's physical and mental state.

[0269] Thus, in this embodiment, the robot 100 identifies the physical and mental state of the user 10, including caregivers, and performs actions corresponding to the identified physical and mental state, thereby providing the user 10 with appropriate care-related suggestions. In other words, the robot 100 can understand the user 10's stress and fatigue and provide appropriate suggestions such as relaxation methods and stress relief methods. That is, according to the robot 100 of this embodiment, appropriate actions can be performed on the user 10.

[0270] Furthermore, when the control unit of robot 100 identifies a state related to the physical and mental condition of user 10, including at least one of the caregiver and the careee, it identifies the act of providing care-related suggestions corresponding to the identified state as its own behavior. Thus, robot 100 is able to provide appropriate care-related suggestions that correspond to the physical and mental condition of user 10, including both caregivers and the careee.

[0271] Furthermore, when the control unit of robot 100 identifies at least one of the user 10's stress level and fatigue level as a state related to the user 10's physical and mental health, it generates information related to the recovery of the user 10's physical and mental health as suggestion information based on at least one of the identified stress level and fatigue level. Thus, robot 100 is able to provide information related to the recovery of the user 10's physical and mental health, consistent with the user 10's stress level or fatigue level, as suggestion information.

[0272] Figure 9A The functional configuration of robot 100 is schematically shown. Robot 100 includes a sensor unit 2200, a sensor module unit 2210, a storage unit 2220, a control unit 2228, and a controlled object 2252. The control unit 2228 includes a state recognition unit 2230, an emotion determination unit 2232, a behavior recognition unit 2234, a behavior determination unit 2236, a storage control unit 2238, a behavior control unit 2250, an relevant information collection unit 2270, and a communication processing unit 2280.

[0273] The controlled object 2252 includes a display device, a speaker, LEDs for the eyes, and motors for driving the arms, hands, and feet. The posture and movements of the robot 100 are controlled by controlling the motors for the arms, hands, and feet. Controlling these motors allows the robot 100 to express some of its emotions. Furthermore, by controlling the illumination state of the LEDs for the robot 100's eyes, facial expressions can also be expressed. It should be noted that the robot 100's posture, movements, and facial expressions are examples of the robot 100's attitude.

[0274] The sensor unit 2200 includes a microphone 2201, a 3D depth sensor 2202, a 2D camera 2203, a distance sensor 2204, a touch sensor 2205, and an accelerometer 2206. The microphone 2201 continuously detects sound and outputs sound data. It should be noted that the microphone 2201 can be mounted on the head of the robot 100 and has binaural recording capabilities. The 3D depth sensor 2202 detects the outline of an object by continuously illuminating an infrared pattern and analyzing the infrared images captured by the infrared camera. The 2D camera 2203 is an example of an image sensor. The 2D camera 2203 captures images using visible light, generating visible light image information. The distance sensor 2204 detects the distance to an object by illuminating it with a laser or ultrasonic wave, for example. It should be noted that the sensor unit 2200 may also include a clock, a gyroscope sensor, and a sensor for motor feedback, etc.

[0275] It should be noted that, in Figure 9A The components of the robot 100 shown, excluding the controlled object 2252 and the sensor unit 2200, are examples of the components of the behavior control system of the robot 100. The behavior control system of the robot 100 uses the controlled object 2252 as the controlled object.

[0276] Storage unit 2220 includes a behavior determination model 2221, historical data 2222, collected data 2223, and behavior prediction data 2224. Historical data 2222 includes past emotional values ​​of user 10, past emotional values ​​of robot 100, and behavioral history. Specifically, it includes multiple event data, which includes the emotional values ​​of user 10, the emotional values ​​of robot 100, and the behavior of user 10. Data including user 10's behavior includes camera images representing user 10's behavior. This emotional value and behavioral history are recorded for each user 10, for example, by associating it with user 10's identification information. At least a portion of storage unit 220 is implemented using a storage medium such as a memory. It may also include a person database storing user 10's facial images, user 10's attribute information, etc.

[0277] It should be noted that, in Figure 9A Of the components of the robot 100 shown, the functions of the components other than the control object 2252, the sensor unit 2200, and the storage unit 2220 can be implemented by the central processing unit (CPU) according to the program. For example, the functions of these components can be implemented as CPU operations through the operating system (OS) and the program running on the operating system.

[0278] Storage unit 2220 includes historical data 2222. Historical data 2222 includes the history of past emotional values ​​and behaviors of user 10. This history of emotional values ​​and behaviors is recorded for each user 10, for example, by associating it with the user 10's identification information. Furthermore, historical data 2222 may include user information for each of the multiple users 10 corresponding to the identification information of multiple users 10. User information includes information indicating whether a user 10 is a caregiver, information indicating whether they are a caregiver, and information indicating whether they are neither a caregiver nor a caregiver. User information indicating whether a user 10 is a caregiver, etc., can be inferred from the user 10's behavioral history or can be registered by the user 10 themselves. In addition, user information includes information representing the characteristics of user 10, such as personality, interests, aspirations, etc. User information representing the characteristics of user 10 can be inferred from the user 10's behavioral history or can be registered by the user 10 themselves. At least a portion of storage unit 2220 is implemented using a storage medium such as a memory. It may also include a person database storing user 10's facial images, user 10's attribute information, etc.

[0279] The sensor module 2210 includes a voice emotion recognition unit 2211, a speech understanding unit 2212, an expression recognition unit 2213, and a face recognition unit 2214. Information detected by the sensor unit 2200 is input to the sensor module 2210. The sensor module 2210 analyzes the information detected by the sensor unit 2200 and outputs the analysis result to the state recognition unit 2230.

[0280] The voice emotion recognition unit 2211 of the sensor module 2210 analyzes the voice of user 10 detected by microphone 2201 to identify the emotion of user 10. For example, the voice emotion recognition unit 2211 extracts feature quantities such as frequency components of the voice, and identifies the emotion of user 10 based on the extracted feature quantities. The speech understanding unit 2212 analyzes the voice of user 10 detected by microphone 2201 and outputs text information representing the content of user 10's speech.

[0281] The expression recognition unit 2213 recognizes the facial expressions and emotions of user 10 from images captured by the 2D camera 2203. For example, the expression recognition unit 2213 recognizes the facial expressions and emotions of user 10 based on the shape and positional relationship of the eyes and mouth.

[0282] The face recognition unit 2214 recognizes the face of user 10. The face recognition unit 2214 identifies user 10 by matching the facial images stored in the people database (illustration omitted) with the facial images of user 10 captured by the 2D camera 2203.

[0283] The state recognition unit 2230 identifies the state of the user 10 based on the information parsed by the sensor module unit 2210. For example, it performs processing mainly related to perception using the parsing results from the sensor module unit 2210. For example, it generates perception information such as "Dad is alone" or "There is a 90% probability that Dad is not smiling." It then performs processing to understand the meaning of the generated perception information. For example, it generates meaning information such as "Dad is alone and looks lonely."

[0284] The state recognition unit 2230 identifies states related to the user 10's physical and mental well-being based on information parsed by the sensor module unit 2210. For example, when the state recognition unit 2230 determines that the identified user 10 is a caregiver or a person being cared for based on user information, it identifies states related to the user 10's physical and mental well-being. Specifically, the state recognition unit 2230 infers the user 10's stress level based on various information such as the user 10's behavior, facial expressions, voice, and text information representing the content of their speech, and identifies the inferred stress level as a state related to the user 10's physical and mental well-being. As an example, if the various information (features such as the frequency components of the voice or text information, etc.) includes information indicating the presence of stress, the user state recognition unit 2230 infers that the user 10's stress level is relatively high. In addition, the user state recognition unit 2230 infers the user 10's fatigue level based on various information such as the user 10's behavior, facial expressions, voice, and text information representing the content of their speech, and identifies the inferred fatigue level as a state related to the user 10's physical and mental well-being. As an example, if various information (such as the frequency components of sound or text information) includes information indicating fatigue accumulation, the user state recognition unit 2230 infers that the user 10's fatigue level is relatively high. It should be noted that the aforementioned stress level and fatigue level can be registered by the user 10 themselves.

[0285] It should be noted that the state recognition unit 2230 can recognize both the stress level and the fatigue level, or only one of them. That is, the state recognition unit 2230 can recognize at least one of the stress level and the fatigue level.

[0286] Furthermore, the state recognition unit 2230 identifies the physical and mental states of each of the multiple users 10 constituting a family member based on information parsed by the sensor module unit 2210. Specifically, the state recognition unit 2230 infers the health status of the user 10 based on various information such as the user 10's behavior, facial expressions, voice, and text information representing spoken content, and identifies the inferred health status as a state related to the user 10's physical and mental state. For example, if the various information (features such as the frequency components of the voice or text information, etc.) includes information indicating a good health status, the state recognition unit 2230 infers that the user 10's health status is good; on the other hand, if it includes information indicating a poor health status, the state recognition unit 2230 infers that the user 10's health status is poor. In addition, the user state recognition unit 2230 infers the user 10's lifestyle habits based on various information such as the user 10's behavior, facial expressions, voice, and text information representing spoken content, and identifies the inferred lifestyle habits as a state related to the user 10's physical and mental state. As an example, if various information (textual information, etc.) includes information indicating lifestyle habits (dietary content or exercise habits, etc.), the status recognition unit 2230 infers the user 10's lifestyle habits based on this information. It should be noted that the aforementioned health status and lifestyle habits can be registered by the user 10 themselves.

[0287] It should be noted that the status recognition unit 2230 can recognize both health status and lifestyle habits, or only one of them. That is, the status recognition unit 2230 only needs to recognize at least one of health status and lifestyle habits.

[0288] Furthermore, the state recognition unit 2230, based on information parsed by the sensor module unit 2210, identifies the mental states of each of the multiple users 10 constituting a family member as states related to the user 10's physical and mental well-being. Specifically, the state recognition unit 2230 infers the user 10's mental state based on various information such as the user 10's behavior, facial expressions, voice, and textual information representing the content of their speech, and identifies the inferred mental state as a state related to the user 10's physical and mental well-being. For example, if the various information (features such as the frequency components of the voice or textual information, etc.) includes information indicating a mental state such as depression or tension, the user state recognition unit 2230 infers the user 10's mental state based on this information. It should be noted that the aforementioned mental states, etc., can be registered by the user 10 themselves.

[0289] The status recognition unit 2230 identifies the status of the robot 100 based on the information detected by the sensor unit 2200. For example, the status recognition unit 2230 identifies the remaining battery level of the robot 100 or the brightness of the surrounding environment of the robot 100 as the status of the robot 100.

[0290] The emotion determination unit 2232 determines an emotion value representing the emotion of the user 10 based on the information parsed by the sensor module unit 2210 and the state of the user 10 identified by the state recognition unit 2230. For example, the information parsed by the sensor module unit 2210 and the identified state of the user 10 are input into a pre-learned neural network to obtain an emotion value representing the emotion of the user 10.

[0291] The emotion value representing user 10's feelings is a positive or negative value. For example, if the user's emotion is a cheerful emotion accompanied by pleasure or peace, such as "joy," "happiness," "pleasure," "peace of mind," "excitement," "relief," or "fulfillment," a positive value is displayed; the more cheerful the emotion, the larger the value. If the user's emotion is a depressed emotion, such as "anger," "sadness," "displeasure," "unease," "grief," "worry," or "emptiness," a negative value is displayed; the more depressed the emotion, the larger the absolute value of the negative value. If the user's emotion does not belong to any of the above categories ("normal"), a value of 0 is displayed.

[0292] Furthermore, the emotion determination unit 2232 determines the emotion value representing the emotion of the robot 100 based on the information parsed by the sensor module unit 2210, the information detected by the sensor unit 2200, and the state of the user 10 identified by the state recognition unit 2230.

[0293] Robot 100's emotion value includes an emotion value for each of the multiple emotion categories, such as values ​​(0~5) representing the intensity of "joy", "anger", "sadness" and "happiness".

[0294] Specifically, the emotion determination unit 2232 determines an emotion value representing the emotion of the robot 100 based on a rule that updates the emotion value of the robot 100, which corresponds to the information parsed by the sensor module unit 2210 and the state of the user 10 identified by the state recognition unit 2230.

[0295] For example, when the state recognition unit 2230 detects that the user 10 looks lonely, the emotion determination unit 2232 increases the "sadness" emotion value of the robot 100. Furthermore, when the state recognition unit 2230 detects that the user 10 smiles, it increases the "joy" emotion value of the robot 100.

[0296] It should be noted that the emotion determination unit 2232 can further consider the state of the robot 100 to determine the emotion value representing the robot 100's emotions. For example, when the robot 100's battery is low or when the robot 100's surrounding environment is dark, the "sadness" emotion value of the robot 100 can be increased. In addition, when the user 10 continues to talk to the robot despite the low battery, the "anger" emotion value can be increased.

[0297] The behavior recognition unit 2234 recognizes the behavior of user 10 based on the information parsed by the sensor module unit 2210 and the state of user 10 recognized by the state recognition unit 2230. For example, the information parsed by the sensor module unit 2210 and the recognized state of user 10 are input into a pre-learned neural network to obtain the probabilities of multiple pre-set behavior categories (e.g., "laughing", "angry", "asking a question", "sad"), and the behavior category with the highest probability is recognized as the behavior of user 10.

[0298] As described above, in this embodiment, the robot 100 obtains the speech content of the user 10 based on the user 10's identification. However, when obtaining and using the speech content, the robot 100 obtains the necessary consent from the user 10 in accordance with laws and regulations. Furthermore, the behavior control system of the robot 100 involved in this embodiment takes into account the protection of the user 10's personal information and privacy.

[0299] Next, the processing of the behavior determination unit 2236 when the robot 100 responds to the behavior of the user 10 will be explained.

[0300] The behavior determination unit 2236 determines the behavior corresponding to the behavior of user 10 identified by the behavior recognition unit 2234 based on the current emotion value of user 10 determined by the emotion determination unit 2232, historical data 2222 of past emotion values ​​determined by the emotion determination unit 2232 before determining the current emotion value of user 10, and the emotion value of robot 100. In this embodiment, the behavior determination unit 2236 uses the most recent emotion value included in the historical data 2222 as the past emotion value of user 10, but the disclosed technology is not limited to this aspect. For example, the behavior determination unit 2236 may also use multiple recent emotion values, or it may also use the emotion value from a unit period such as one day ago as the past emotion value of user 10. Furthermore, the behavior determination unit 2236 considers not only the current emotion value of robot 100, but also the history of past emotion values ​​of robot 100 to determine the behavior corresponding to the behavior of user 10. The behavior determined by the behavior determination unit 2236 includes gestures performed by robot 100 or the content of robot 100's speech.

[0301] The behavior determination unit 2236 in this embodiment determines the behavior of the robot 100 as a behavior corresponding to the behavior of the user 10 based on the combination of the user 10's past and current sentiment values, the robot 100's sentiment value, the user 10's behavior, and the behavior determination model 2221. For example, when the user 10's past sentiment value is positive and the current sentiment value is negative, the behavior determination unit 2236 determines the behavior used to change the user 10's sentiment value to positive as a behavior corresponding to the user 10's behavior.

[0302] In the response rules of the behavior determination model 2221, the combination of the user 10's past and current sentiment values, the robot 100's sentiment value, and the robot 100's behavior corresponding to the user 10's behavior are specified. For example, when the user 10's past sentiment value is positive and the current sentiment value is negative, and the user 10's behavior is sadness, the combination of gestures and verbal content used when making encouraging inquiries to the user 10 is specified as the robot 100's behavior.

[0303] For example, in the reaction rules of behavior determination model 2221, the behavior of robot 100 is defined based on the patterns of robot 100's emotional values ​​(1296 patterns, i.e., the four powers of six values ​​from "0" to "5" for "joy", "anger", "sadness", and "happiness"), the patterns of combinations of user 10's past and current emotional values, and all combinations of user 10's behavioral patterns. That is, for each pattern of robot 100's emotional values, such as combinations of user 10's past and current emotional values ​​such as negative and negative, negative and positive, positive and negative, positive and positive, negative and normal, and normal and normal, for each of the multiple combinations, the robot's behavior corresponding to user 10's behavioral pattern is defined. It should be noted that when user 10 utters a statement such as "I want to talk about the topic we discussed earlier" to continue the conversation, the behavior determination unit 2236 can switch to using historical data 2222 to determine the action pattern of robot 100's behavior.

[0304] It should be noted that, in the response rules of the behavior determination model 2221, for each of the 1296 patterns of the robot 100's emotion value, at most one behavior of the robot 100 can be specified, which includes at least one of gestures and spoken content. Alternatively, in the response rules of the behavior determination model 2221, for each group of patterns of the robot 100's emotion value, the behavior of the robot 100 can be specified, which includes at least one of gestures and spoken content.

[0305] In the response rules of the robot 100 as defined in the behavior determination model 2221, the intensity of each gesture is preset. Similarly, the intensity of each utterance is preset in the utterance content included in the behavior of the robot 100 as defined in the response rules of the behavior determination model 2221.

[0306] Furthermore, for example, the response rules specify the behaviors of the robot 100 corresponding to behavioral patterns. These behavioral patterns include: when the user 10's physical and mental state (stress level and fatigue level), including caregivers and caregivers, is such that care-related suggestions need to be provided to the user 10; and when the user 10 responds to the provided suggestions. For example, if the behavior determination unit 2236 infers, based on the response rules, that the user 10's stress level, including caregivers and caregivers, is high, or infers that the user 10's fatigue level is high, then it determines the behavior of providing care-related suggestions to the user 10 corresponding to the user 10's physical and mental state as its own behavior.

[0307] The storage control unit 2238 determines whether to store data including the user 10's behavior in the historical data 2222 based on the intensity of the behavior preset by the behavior determination unit and the emotion value of the robot 100 determined by the emotion determination unit 2232.

[0308] Specifically, when the sum of the emotional values ​​of each of the multiple emotional categories for robot 100, the intensity of the gestures pre-set for the behavior determined by behavior determination unit 2236, and the intensity of the speech content pre-set for the behavior determined by behavior determination unit 2236, i.e., the comprehensive value of the intensity, is above a threshold, it is determined that the data including the behavior of user 10 will be stored in historical data 2222.

[0309] When the storage control unit 2238 determines that data including the behavior of user 10 will be stored in the historical data 2222, the behavior determined by the behavior determination unit 2236, the information parsed by the sensor module unit 2210 from the current moment to a certain period in advance (e.g., all surrounding information such as sound, image, smell, etc. at the scene), and the state of user 10 identified by the state recognition unit 2230 (e.g., the expression, emotion, etc. of user 10) will be stored in the historical data 2222.

[0310] The behavior control unit 2250 controls the controlled object 2252 based on the behavior determined by the behavior determination unit 2236. For example, when the behavior determination unit 2236 determines that the behavior includes speaking, the behavior control unit 2250 causes the speaker included in the controlled object 2252 to output sound. At this time, the behavior control unit 2250 can determine the sound output speed based on the emotion value of the robot 100. For example, the higher the emotion value of the robot 100, the faster the sound output speed determined by the behavior control unit 2250. In this way, the behavior control unit 2250 determines the execution mode of the behavior determined by the behavior determination unit 2236 based on the emotion value determined by the emotion determination unit 2232.

[0311] The behavior control unit 2250 can identify changes in the emotions of the user 10 relative to the behavior determined by the behavior determination unit 2236. For example, changes in emotion can be identified based on the user 10's voice or facial expression. Additionally, changes in the user 10's emotion can be identified based on impacts detected by the touch sensor 2205 included in the sensor unit 2200. When the touch sensor 2205 included in the sensor unit 2200 detects an impact, a deterioration in the user 10's emotion can be identified. When the user 10's reaction is determined to be laughter, happiness, etc., based on the detection result of the touch sensor 2205 included in the sensor unit 2200, an improvement in the user 10's emotion can be identified. Information indicating the user 10's reaction is output to the communication processing unit 2280.

[0312] Furthermore, after the behavior control unit 2250 executes the behavior determined by the behavior determination unit 2236 in an execution mode determined according to the emotion of the robot 100, the emotion determination unit 2232 also changes the emotion value of the robot 100 based on the user's reaction to the execution of the behavior. Specifically, when the user does not react negatively to the behavior determined by the behavior determination unit 2236 being executed in the execution mode determined by the behavior control unit 2250, the emotion determination unit 2232 increases the "joy" emotion value of the robot 100. Furthermore, when the user reacts negatively to the behavior determined by the behavior determination unit 2236 being executed in the execution mode determined by the behavior control unit 2250, the emotion determination unit 2232 increases the "sadness" emotion value of the robot 100.

[0313] Furthermore, the behavior control unit 2250 expresses the emotions of the robot 100 based on a determined emotion value. For example, when the "joy" emotion value of the robot 100 is increased, the behavior control unit 2250 controls the controlled object 2252 to make the robot 100 perform joyful actions. Conversely, when the "sadness" emotion value of the robot 100 is increased, the behavior control unit 2250 controls the controlled object 2252 to make the robot 100 adopt a dejected posture.

[0314] Specifically, when the behavior control unit 2250 identifies a state related to the physical and mental health of the user 10, including the caregiver and the caregiver, it identifies the behavior of providing care-related suggestion information corresponding to the physical and mental health of the user 10 as its own behavior and controls the controlled object 2252.

[0315] In detail, when the behavior control unit 2250 suspects that user 10's stress level is relatively high or that their fatigue level is relatively high, it initiates a conversation with user 10. Specifically, the behavior control unit 2250 issues statements such as "Here are some suggestions regarding care," indicating that it will provide advice or information.

[0316] Next, the behavior control unit 2250 generates care-related suggestion information based on the identified physical and mental state of user 10 (stress level, fatigue level, etc.), and speaks and provides the generated suggestion information. The suggestion information includes, but is not limited to, methods for maintaining a positive attitude towards care, methods for eliminating stress, relaxation methods, and other information related to restoring user 10's physical and mental well-being by providing emotional support (more specifically, information for achieving physical and mental recovery). For example, the behavior control unit 2250 speaks and provides suggestion information that aligns with user 10's physical and mental state, such as "It seems you have accumulated stress. We suggest engaging in physical activities such as stretching" or "It seems you have accumulated fatigue. We suggest ensuring sufficient sleep."

[0317] In this embodiment, the behavior control unit 2250 identifies the physical and mental state of the user 10, including caregivers, and performs actions corresponding to the identified physical and mental state, thereby providing the user 10 with appropriate care-related suggestions. In other words, the behavior control unit 2250 can understand the user 10's stress and fatigue and provide appropriate suggestions such as relaxation methods and stress relief methods.

[0318] In addition, the behavior control unit 2250 can provide information on nursing-related laws and regulations as advisory information. It should be noted that the information on nursing-related laws and regulations corresponds to the care status (care level) of the caregiver, and may be obtained, for example, through the communication processing unit 2280 via a communication network 20 such as the Internet, or via an external server or server 300 not shown in the figure, but is not limited thereto.

[0319] Furthermore, since the emotion value of robot 100 is determined by emotion determination unit 2232, behavior control unit 2250 can, based on this emotion value, say and provide suggestions that are close to the feelings (emotions) of user 10a as caregiver, such as "Although the care is hard, user 10b seems to benefit a lot (seems very happy)".

[0320] The communication processing unit 2280 is responsible for communicating with the server 300. As described above, the communication processing unit 2280 sends the user response information to the server 300. In addition, the communication processing unit 2280 receives the updated response rules from the server 300. When the communication processing unit 2280 receives the updated response rules from the server 300, it updates the response rules of the behavior determination model 2221.

[0321] The server 300 communicates between the robots 100, 101, and 102 and the server 300, receives the user response information sent from the robot 100, and updates the response rule 221 based on the response rules including the behaviors that obtain positive responses.

[0322] The relevant information collection unit 2270 collects information related to the preference information from external data (Web sites such as news sites and video sites) at a predetermined time point based on the preference information obtained about the user 10.

[0323] Specifically, the relevant information collection unit 2270 pre-obtains the preference information indicating the matters that the user 10 is concerned about from the speech content of the user 10 or the setting operations performed by the user 10. The relevant information collection unit 2270 collects news related to the preference information from external data at regular intervals using, for example, the ChatGPT plugin (Plugins) (Internet search <URL:https: / / openai.com / blog / chatgpt-plugins>). For example, when it is obtained as preference information that the user 10 is a fan of a specific professional baseball team, the relevant information collection unit 2270 collects news related to the game results of the specific professional baseball team from external data at a predetermined time every day using, for example, the ChatGPT plugin.

[0324] The emotion determination unit 2232 determines the emotion of the robot 100 based on the information related to the preference information collected by the relevant information collection unit 2270.

[0325] Specifically, the emotion determination unit 2232 inputs the text representing the information related to the preference information collected by the relevant information collection unit 2270 into a pre-trained neural network for emotion determination, obtains the emotion values representing each emotion, and determines the emotion of the robot 100. For example, when the news related to the game results of a specific professional baseball team collected shows that the specific professional baseball team has won, it is determined that the emotion value of "joy" of the robot 100 increases.

[0326] If the emotion value of the robot 100 is above the threshold, the storage control unit 2238 stores the information related to the preference information collected by the relevant information collection unit 2270 in the collected data 2223.

[0327] Next, the processing of the behavior determination unit 2236 during the autonomous processing of the robot 100 performing autonomous actions will be explained.

[0328] In the autonomous processing of this embodiment, the robot 100, acting as an intelligent agent, proactively and periodically monitors the state of the user 10 receiving care. For example, the robot 100 constantly monitors the caregiver, including their fatigue level and well-being. When the robot 100 determines that the user 10's fatigue and motivation have decreased, it takes actions to increase motivation or alleviate stress. Specifically, the robot 100 understands the user 10's stress and fatigue and suggests appropriate relaxation and stress-relieving methods. When the caregiver's well-being increases, the robot 100 proactively praises the caregiver or offers words of comfort. Furthermore, the robot 100 proactively and periodically collects information related to care-related laws and regulations from external sources (such as news websites, video websites, and news feeds). When the importance exceeds a certain threshold, it proactively provides the collected care-related information to the caregiver (user).

[0329] At a predetermined time point, the behavior determination unit 2236 uses at least one of the user 10's state, the user 10's emotion, the robot 100's emotion, and the robot 100's state, along with the behavior determination model 2221, to determine any one of a variety of robot behaviors, including inaction, as the behavior of the robot 100. Here, the example of using a dialogue-enabled article generation model as the behavior determination model 2221 will be explained.

[0330] Specifically, the behavior determination unit 2236 inputs text representing at least one of the user 10's state, the user 10's emotion, the robot 100's emotion, and the robot 100's state, as well as text inquiring about the robot's behavior, into the article generation model, and determines the robot 100's behavior based on the output of the article generation model.

[0331] For example, various robot behaviors include the following (1) to (11).

[0332] (1) The robot does nothing.

[0333] (2) Robots dream.

[0334] (3) The robot strikes up a conversation with the user.

[0335] (4) Robots create drawing diaries.

[0336] (5) The robot makes activity suggestions.

[0337] (6) The robot suggests people the user should meet.

[0338] (7) The robot introduces news that users are interested in.

[0339] (8) Robots edit photos and videos.

[0340] (9) The robot learns together with the user.

[0341] (10) The robot evokes memories.

[0342] (11) The robot provides care-related advice to the user.

[0343] Every certain period of time, the behavior determination unit 2236 inputs text representing the state of user 10 and robot 100 as identified by the state recognition unit 2230, the current emotion value of user 10 and robot 100 as determined by the emotion determination unit 2232, and text querying any one of various robot behaviors, including inaction, into the article generation model, and determines the behavior of robot 100 based on the output of the article generation model. Here, when user 10 is not present in the vicinity of robot 100, the text input into the article generation model may not include the state of user 10 and the current emotion value of user 10, or it may include information indicating that user 10 is not present.

[0344] As an example, consider the following: "The robot is in a very happy state. The user is in a generally happy state. The user is sleeping. As the robot's behavior, which of the following (1)~(11) is better?" (1) The robot does nothing.

[0345] (2) Robots dream.

[0346] (3) The text “The robot strikes up a conversation with the user…” is input into the article generation model. Based on the output of the article generation model, “It can be said that one of the two behaviors, (1) doing nothing or (2) the robot dreaming, is the most suitable behavior”, “(1) doing nothing” or “(2) the robot dreaming” is determined as the behavior of robot 100.

[0347] As another example, consider the following scenario: "The robot is in a somewhat lonely state. The user is not present. The robot's surroundings are dark. As for the robot's behavior, which of the following (1) to (11) is better?" (1) The robot does nothing.

[0348] (2) Robots dream.

[0349] (3) The text “The robot strikes up a conversation with the user…” is input into the article generation model. Based on the output of the article generation model, “It can be said that one of the two behaviors, (2) the robot is dreaming or (4) the robot is making a painting diary, is the most suitable behavior”, “(2) the robot is dreaming” or “(4) the robot is making a painting diary” is identified as the behavior of robot 100.

[0350] When the behavior determination unit 2236 determines "(2) Robot dreaming" as the creation of the original event as a robot behavior, it uses the article generation model to create an original event that combines multiple event data from the historical data 2222. At this time, the storage control unit 2238 stores the created original event in the historical data 2222.

[0351] When the behavior determination unit 2236 determines "(3) the robot speaks to the user," that is, the robot 100 utters a utterance, as a robot behavior, it uses a text generation model to determine the robot's utterance content corresponding to the user's state and the user's emotion or the robot's emotion. At this time, the behavior control unit 2250 outputs a sound representing the determined robot utterance content from the speaker included in the controlled object 2252. It should be noted that when the user 10 is not in the vicinity of the robot 100, the behavior control unit 2250 does not output a sound representing the determined robot utterance content, but instead stores the determined robot utterance content in the behavior predetermined data 2224.

[0352] When the behavior determination unit 2236 determines "(7) The robot introduces news that the user is interested in" as a robot behavior, it uses an article generation model to determine the robot's speech content corresponding to the information stored in the collected data 2223. At this time, the behavior control unit 2250 outputs a sound representing the determined robot speech content from the speaker included in the controlled object 2252. It should be noted that when the user 10 is not in the vicinity of the robot 100, the behavior control unit 2250 does not output a sound representing the determined robot speech content, but instead stores the determined robot speech content in the behavior predetermined data 2224.

[0353] When the behavior determination unit 2236 determines "(4) Robot making a drawing diary," i.e., the event image created by robot 100, as a robot behavior, it uses an image generation model to generate an image representing the event data selected from historical data 2222, and simultaneously uses an article generation model to generate explanatory text representing the event data. The combination of the image representing the event data and the explanatory text representing the event data is then output as the event image. It should be noted that when user 10 is not in the vicinity of robot 100, the behavior control unit 2250 does not output the event image, but instead stores the event image in the behavior predefined data 2224.

[0354] When the behavior determination unit 2236 determines "(8) robot editing photos and videos", i.e., editing images, as a robot behavior, it selects event data from historical data 2222 based on emotion values, edits and outputs the image data of the selected event data. It should be noted that when the user 10 is not in the vicinity of the robot 100, the behavior control unit 2250 does not output the edited image data, but stores the edited image data in the behavior pre-defined data 2224.

[0355] When the behavior determination unit 2236 determines "(5) the robot makes an activity suggestion," that is, makes a suggestion for the user 10's behavior, as a robot behavior, it uses an article generation model based on the event data stored in the historical data 2222 to determine the suggested user behavior. At this time, the behavior control unit 2250 outputs a sound suggesting the user's behavior from the speaker included in the controlled object 2252. It should be noted that when the user 10 is not in the vicinity of the robot 100, the behavior control unit 2250 does not output a sound suggesting the user's behavior, but instead stores the suggestion for the user's behavior in the behavior pre-defined data 2224.

[0356] When the behavior determination unit 2236 determines "(6) The robot suggests that the user should meet with the object," that is, the object that the robot suggests should establish contact with the user 10, as a robot behavior, it uses an article generation model based on the event data stored in the historical data 2222 to determine the suggested object that the user 10 should establish contact with. At this time, the behavior control unit 2250 outputs a sound indicating the suggested object that the user should establish contact with from the speaker included in the controlled object 2252. It should be noted that when the user 10 is not in the vicinity of the robot 100, the behavior control unit 2250 does not output a sound indicating the suggested object that the user should establish contact with, but instead stores the suggested object that the user should establish contact with in the behavior predetermined data 2224.

[0357] When the behavior determination unit 2236 determines "(9) Robot and user learn together," that is, the robot 100 uttering words about learning, as robot behavior, it uses a text generation model to determine the robot's speech content, which corresponds to the user's state and the user's or robot's emotions, and is used to encourage learning, point out learning problems, or provide suggestions about learning. At this time, the behavior control unit 2250 outputs a sound representing the determined robot speech content from the speaker included in the controlled object 2252. It should be noted that when the user 10 is not in the vicinity of the robot 100, the behavior control unit 2250 does not output a sound representing the determined robot speech content, but instead stores the determined robot speech content in the behavior predetermined data 2224.

[0358] When the behavior determination unit 2236 determines "(10) robot memory recall," i.e., recalling event data, as robot behavior, it selects event data from historical data 2222. At this time, the emotion determination unit 2232 determines the emotion of robot 100 based on the selected event data. Furthermore, based on the selected event data, the behavior determination unit 2236 uses an article generation model to create an emotion change event representing the emotional content and behavior of robot 100 used to change the user's emotion value. At this time, the storage control unit 2238 stores the emotion change event in the behavior predetermined data 2224.

[0359] For example, when the video the user watched is about pandas, it is stored as event data in historical data 2222. When the event data is selected, the article generation model is input with the question "What should be said when meeting the user next time about pandas? Give three examples". When the output of the article generation model is "(1) Go to the zoo, (2) Draw a picture of a panda, (3) Go to buy a panda plush toy", the robot 100 inputs the question "Which of (1), (2), (3) is the user most likely to be satisfied with?". If the output of the article generation model is "(1) Go to the zoo", then the robot 100 will say "(1) Go to the zoo" when it meets the user next time, and store it in behavior pre-defined data 2224.

[0360] Additionally, for example, event data with high emotional values ​​for robot 100 can be selected as robot 100's most memorable memories. Thus, it is possible to create emotional change events based on the event data selected as most memorable memories.

[0361] When the behavior determination unit 2236 determines "(11) providing care-related suggestions to the user," that is, providing necessary information to the user involved in care, as a robot behavior, it may obtain the information needed by the user from external data. Even when the user is not present, the robot 100 actively obtains this information at all times.

[0362] Furthermore, regarding "providing care-related advice to users," the relevant information collection unit 2270 collects, for example, care-related information about the user as information the user prefers, and stores it in the collected data 2223. Then, this information is output as sound from a speaker or displayed as text on a monitor to support the user's care activities.

[0363] In the autonomous processing of this embodiment, the robot 100, acting as an intelligent agent, proactively and periodically monitors the state of the user 10 receiving care. For example, the robot 100 constantly monitors the caregiver, including their fatigue level and well-being. When the robot 100 determines that the user 10's fatigue and motivation have decreased, it takes actions to increase motivation or alleviate stress. Specifically, the robot 100 understands the user 10's stress and fatigue and suggests appropriate relaxation and stress-relieving methods. When the caregiver's well-being increases, the robot 100 proactively praises the caregiver or offers words of comfort. Furthermore, the robot 100 proactively and periodically collects information related to care-related laws and regulations from external sources, such as news websites, video websites, and news feeds. When the importance exceeds a certain threshold, the robot 100 proactively provides the collected care-related information to the caregiver (user).

[0364] The appearance of Robot 100 can mimic a human form or be that of a plush toy. By having the appearance of a plush toy, Robot 100 is thought to be particularly easy for children to feel close to.

[0365] When the user 10's state is identified by the state recognition unit 2230, from a state where there is no user 10's behavior towards the robot 100 to a state where user 10's behavior towards the robot 100 is detected, the behavior determination unit 2236 reads the data stored in the behavior predetermined data 2224 and determines the behavior of the robot 100.

[0366] For example, when user 10 is not near robot 100, if user 10 is detected, the behavior determination unit 2236 reads the data stored in behavior pre-defined data 2224 and determines the behavior of robot 100. Furthermore, when user 10 is sleeping, if user 10 is detected waking up, the behavior determination unit 2236 reads the data stored in behavior pre-defined data 2224 and determines the behavior of robot 100.

[0367] Figure 9B This illustration shows an example of the operational flow for collecting and processing information related to user 10's preferences. Figure 9B The illustrated workflow is repeated at regular intervals. Assume that user 10's preferences, representing the items they are interested in, are obtained from their verbal statements or settings. It should be noted that "S" in the workflow indicates the step to be performed.

[0368] First, in step S90, the relevant information collection unit 2270 obtains preference information indicating the matters that user 10 is concerned about.

[0369] In step S92, the relevant information collection unit 2270 collects information related to preference information from external data.

[0370] In step S94, the emotion determination unit 2232 determines the emotion value of the robot 100 based on information related to preference information collected by the relevant information collection unit 2270.

[0371] In step S96, the storage control unit 2238 determines whether the emotion value of the robot 100 determined in step S94 is above a threshold. If the emotion value of the robot 100 is below the threshold, the collected information related to preference information is not stored in the collected data 2223, and the process ends. On the other hand, if the emotion value of the robot 100 is above the threshold, the process proceeds to step S98.

[0372] In step S98, the storage control unit 2238 stores the collected information related to preference information in the collected data 2223 and ends the process.

[0373] Figure 3 This schematically illustrates an example of an operational flow related to determining the behavior within robot 100 during response processing when robot 100 responds to the behavior of user 10. (Repeated execution) Figure 3 The operation flow is shown below. At this point, assume the input includes information parsed by the sensor module 2210.

[0374] First, in step S100, the state recognition unit 2230 identifies the state of the user 10 and the state of the robot 100 based on the information parsed by the sensor module unit 2210. For example, when the identified user 10 is a caregiver or a person being cared for, the state recognition unit 2230 identifies the state related to the user 10's physical and mental health (stress level or fatigue level, etc.). Additionally, the state recognition unit 2230 identifies the states related to the physical and mental health of each of the multiple users 10 constituting a family member (health status and lifestyle habits, etc.). Furthermore, the state recognition unit 2230 identifies the mental state of each of the multiple users 10 constituting a family member.

[0375] In step S102, the emotion determination unit 2232 determines an emotion value representing the emotion of the user 10 based on the information parsed by the sensor module unit 2210 and the state of the user 10 identified by the state recognition unit 2230.

[0376] In step S103, the emotion determination unit 2232 determines an emotion value representing the emotion of the robot 100 based on the information parsed by the sensor module unit 2210 and the state of the user 10 identified by the state recognition unit 2230. The emotion determination unit 2232 adds the determined emotion value of the user 10 and the emotion value of the robot 100 to the historical data 2222.

[0377] In step S104, the behavior recognition unit 2234 identifies the behavior classification of the user 10 based on the information parsed by the sensor module unit 2210 and the state of the user 10 identified by the state recognition unit 2230.

[0378] In step S106, the behavior determination unit 2236 determines the behavior of the robot 100 based on the combination of the current sentiment value of the user 10 determined in step S102 and the past sentiment values ​​contained in the historical data 2222, the sentiment value of the robot 100, the behavior of the user 10 identified in step S104 above, and the behavior determination model 2221.

[0379] In step S3108, the behavior control unit 2250 controls the controlled object 2252 based on the behavior determined by the behavior determination unit 2236.

[0380] In step S110, the storage control unit 2238 calculates a comprehensive value of the intensity based on the intensity of the behavior preset by the behavior determination unit 2236 and the emotion value of the robot 100 determined by the emotion determination unit 2232.

[0381] In step S112, the storage control unit 2238 determines whether the overall intensity value is above a threshold. If the overall intensity value is less than the threshold, the event data, including the user 10's behavior, is not stored in the historical data 2222, and the process ends. On the other hand, if the overall intensity value is above the threshold, the process proceeds to step S114.

[0382] In step S114, the event data is stored in historical data 2222. The event data includes the behavior determined by the behavior determination unit 2236, the information parsed by the sensor module unit 2210 over a certain period from the current moment forward, and the status of the user 10 identified by the status recognition unit 2230.

[0383] Figure 9C This schematically illustrates an example of an operational flow related to the operation of determining behavior within robot 100 during autonomous processing of autonomous actions performed by robot 100. Figure 9C The illustrated operation flow, for example, is automatically repeated after a certain period of time. At this time, it is assumed that the input includes information parsed by the sensor module 2210. It should be noted that, for the above...Figure 3 The same processing means the same step number.

[0384] First, in step S100, the state recognition unit 2230 recognizes the state of the user 10 and the state of the robot 100 based on the information parsed by the sensor module unit 2210.

[0385] In step S102, the emotion determination unit 2232 determines an emotion value representing the emotion of the user 10 based on the information parsed by the sensor module unit 2210 and the state of the user 10 identified by the state recognition unit 2230.

[0386] In step S103, the emotion determination unit 2232 determines an emotion value representing the emotion of the robot 100 based on the information parsed by the sensor module unit 2210 and the state of the user 10 identified by the state recognition unit 2230. The emotion determination unit 2232 adds the determined emotion value of the user 10 and the emotion value of the robot 100 to the historical data 2222.

[0387] In step S104, the behavior recognition unit 2234 identifies the behavior classification of the user 10 based on the information parsed by the sensor module unit 2210 and the state of the user 10 identified by the state recognition unit 2230.

[0388] In step S200, the behavior determination unit 2236 determines any one of a variety of robot behaviors, including not taking any action, as the behavior of the robot 100 based on the state of the user 10 identified in step S100, the emotion of the user 10 identified in step S102, the emotion of the robot 100, the state of the robot 100 identified in step S100, the behavior of the user 10 identified in step S104, and the behavior determination model 2221.

[0389] In step S201, the behavior determination unit 2236 determines, as in step S200 above, whether to determine that no action will be taken. If it is determined that no action will be taken as the behavior of the robot 100, the process ends. On the other hand, if it has not yet been determined that no action will be taken as the behavior of the robot 100, the process proceeds to step S202.

[0390] In step S202, the behavior determination unit 2236 performs processing corresponding to the robot behavior type determined in step S200. At this time, the behavior control unit 2250, the emotion determination unit 2232, or the storage control unit 2238 performs processing according to the type of robot behavior.

[0391] In step S110, the storage control unit 2238 calculates a comprehensive value of the intensity based on the intensity of the behavior preset by the behavior determination unit 2236 and the emotion value of the robot 100 determined by the emotion determination unit 2232.

[0392] In step S112, the storage control unit 2238 determines whether the overall intensity value is above a threshold. If the overall intensity value is less than the threshold, the data, including the user 10's behavior, is not stored in the historical data 2222, and the process ends. On the other hand, if the overall intensity value is above the threshold, the process proceeds to step S114.

[0393] In step S114, the storage control unit 2238 stores the behavior determined by the behavior determination unit 2236, the information parsed by the sensor module unit 2210 for a certain period from the current moment forward, and the status of the user 10 identified by the status recognition unit 2230 in the historical data 2222.

[0394] As described above, according to robot 100, an emotion value representing the robot 100's emotions is determined based on the user's state, and based on the robot 100's emotion value, it is determined whether to store data including user 10's behavior in historical data 2222. This reduces the capacity of historical data 2222 that stores data including user 10's behavior. Then, for example, when robot 100 determines that the user's state is the same as it was 10 years ago, by reading historical data 222 from 10 years ago, robot 100 can present user 10 with the user's state 10 years ago (e.g., user 10's facial expressions, emotions, etc.), and even all surrounding information including ambient sounds, images, smells, etc.

[0395] Furthermore, according to robot 100, robot 100 can perform appropriate actions in response to user 10's behavior. Previously, user behavior was categorized to determine the robot's actions, including facial expressions and postures. In contrast, robot 100 determines user 10's current emotional state and performs actions based on past and current emotional states. Therefore, for example, if user 10 was energetic yesterday but is depressed today, robot 100 can say something like, "You were so energetic yesterday, what's wrong today?" Additionally, robot 100 can also use gestures to communicate. For example, if user 10 was depressed yesterday but is energetic today, the robot can say something like, "You were depressed yesterday, but you seem energetic today!" Furthermore, for example, if user 10 was energetic yesterday and is even more energetic today, robot 100 can say something like, "You're more energetic today than yesterday, has anything good happened compared to yesterday?" Furthermore, for example, for users who consistently maintain an emotional value above 0 and whose emotional value fluctuates within a certain range, Robot 100 can say things like, "Your emotions have been stable lately, and you feel good."

[0396] Furthermore, for example, if robot 100 asks user 10, "Did you finish the homework we talked about yesterday?", and receives a reply of "Yes, I did!" from user 10, robot 100 can not only say affirmative words like "Great job!" but also make affirmative gestures such as clapping or giving a thumbs up. Additionally, for example, if user 10 says, "The demonstration I mentioned the day before yesterday was very successful!", robot 100 can not only say affirmative words like "Well done!" but also make the aforementioned affirmative gestures. In this way, by having robot 100 execute actions based on user 10's past state, it is expected that user 10 will develop a sense of closeness towards robot 100.

[0397] Additionally, for example, when user 10 watches a panda-related video, if user 10's "happiness" emotion value is above the threshold, the scene of the panda appearing in the video can be stored as event data in historical data 2222.

[0398] Using the data accumulated from historical data 2222 and collected data 2223, robot 100 can learn at all times what kind of conversations to have with users, so as to maximize the emotional value of expressing user happiness.

[0399] Furthermore, even when the robot 100 is not in conversation with the user 10, it can autonomously begin to act based on the robot 100's emotions.

[0400] Furthermore, in autonomous processing, robot 100 can automatically generate questions and input them into an article generation model, repeatedly obtaining the output of the article generation model as answers to the questions, thereby creating emotional change events to enhance positive emotions and storing them in behavioral predefined data 2224. In this way, robot 100 is able to perform self-learning.

[0401] Furthermore, when the robot 100 automatically generates questions without receiving any external triggers, it can do so based on specific event data that left an impression from the history of the robot's past emotional values.

[0402] Furthermore, the relevant information collection unit 2270 can perform self-learning by repeatedly executing the following retrieval execution phase: automatically performing keyword searches based on user preference information and obtaining retrieval results.

[0403] Here, the retrieval execution phase can be set to automatically perform keyword retrieval based on specific event data that left an impression from the history of the robot's past emotional values, in the absence of external triggers.

[0404] It should be noted that the emotion determination unit 2232 can determine the user's emotion based on a specific mapping. Specifically, the emotion determination unit 2232 can determine the user's emotion based on an emotion map that serves as a specific mapping (see...). Figure 5 To determine the user's emotions.

[0405] Figure 5 This is a schematic diagram of an emotion map 400 that maps various emotions. In the emotion map 400, emotions are arranged radially from the center on concentric circles. The closer to the center of the concentric circles, the more primitive the emotion is. On the outer side of the concentric circles, emotions representing states and behaviors arising from mood are arranged. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions arising from reactions occurring within the brain are usually arranged. On the right side of the concentric circles, emotions guided by situational judgments are usually arranged. Above and below the concentric circles, emotions arising from reactions occurring within the brain and guided by situational judgments are usually arranged. In addition, the emotion of "pleasure" is arranged above the concentric circles, and the emotion of "displeasure" is arranged below. Thus, in the emotion map 400, multiple emotions are mapped based on the structure of emotion generation, and emotions that are likely to arise simultaneously are mapped to the vicinity.

[0406] (1) For example, when the emotion engine of robot 100, which is the emotion determination unit 2232, detects an emotion at approximately 100 msec, the frequency of determining the reaction action (e.g., agreement) of robot 100 can be set at least to the same time point as the detection frequency (100 msec) of the emotion engine, or it can be set to an earlier time point. The detection frequency of the emotion engine can be interpreted as the sampling rate.

[0407] By detecting emotions at approximately 100 ms and immediately executing corresponding actions (such as echoing), natural, empathetic dialogue can be achieved instead of unnatural echoing. Robot 100 executes corresponding actions (e.g., echoing) based on the directionality and intensity of the mandala in the emotion map 400. It should be noted that the detection frequency (sampling rate) of the emotion engine is not limited to 100 ms and can be varied depending on the situation (e.g., during movement) and the user's age.

[0408] (2) The direction and intensity of the emotion can be preset by referring to the emotion chart 400, and the actions of agreement and the strength of agreement can be set. For example, when robot 100 feels stable and at ease, robot 100 nods and continues to listen. When robot 100 feels uneasy, confused or suspicious, robot 100 can tilt its head or stop shaking its head.

[0409] These emotions are distributed at the 3 o'clock position on the emotion diagram 400, and usually shift back and forth between peace and anxiety. In the right half of the emotion diagram 400, situational awareness is more dominant than internal feelings, thus forming an impression of calmness.

[0410] (3) When Robot 100 feels pleasure from praise, the filler word "Ah—" can be added before the dialogue; when it feels pain from harsh words, the filler word "Ooh!" can be added before the dialogue. In addition, physical reactions can also be included, such as the posture of Robot 100 squatting down while saying "Ooh!". These emotions are distributed around the 9 o'clock position of the emotion diagram 400.

[0411] (4) In the left half of the emotion map 400, internal feelings (responses) are more dominant than situation recognition. Therefore, it may give the impression of involuntary reactions.

[0412] When Robot 100 senses an intrinsic feeling of acceptance (response) and also feels positive in the situation recognition, it can nod deeply while gazing at the other party and also make "uh-huh" sounds. In this way, Robot 100 can generate equal goodwill towards the other party, namely behaviors such as tolerance and forgiveness. Such emotions are distributed around the 12 o'clock position on Emotion Graph 400.

[0413] Conversely, when Robot 100 experiences an internal feeling (reaction) of displeasure and also feels disgust in situation recognition, Robot 100 can shake its head. When it reaches the level of feeling hatred, it can turn the LEDs in its eyes red and stare at the other party. Such emotions are distributed around the 6 o'clock position on the emotion map 400.

[0414] (5) The inner side of the emotion diagram 400 represents the inner mind, and the outer side of the emotion diagram 400 represents behavior. Therefore, the closer to the outer side of the emotion diagram 400, the more visible the emotion becomes (manifested in behavior).

[0415] (6) When feeling at ease and listening to people around the 3 o'clock position on the emotional map 400, the robot 100 nods slightly and makes an "uh-huh" sound. However, when in a loving situation around the 12 o'clock position, it can make a strong nod, such as a deep nod.

[0416] Here, human emotions are based on various balances such as posture or blood sugar levels. When these balances deviate from the ideal, an unpleasant state is evoked, while a pleasant state is evoked when they approach the ideal. In robots, cars, and motorcycles, emotions can also be constructed based on various balances such as posture or battery level, so that when these balances deviate from the ideal, an unpleasant state is evoked, while a pleasant state is evoked when they approach the ideal. Emotion maps, for example, can be generated based on Dr. Mitsuyoshi's emotion map (research related to vocal emotion recognition and brain physiological signal analysis systems for emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). In the left half of the emotion map, emotions belonging to a region called "response" are arranged, where sensation is dominant. Furthermore, in the right half of the emotion map, emotions belonging to a region called "state" are arranged, where state recognition is dominant.

[0417] The emotion map defines two types of emotions that promote learning. One is the negative emotion surrounding the center of the situation side, namely "repentance" or "reflection." That is, when the robot experiences negative emotions such as "I never want to experience this feeling again" or "I never want to be scolded again." The other is the positive emotion near "desire" on the response side. That is, when there are positive emotions such as "wanting more" or "wanting to know more."

[0418] The emotion determination unit 2232 inputs the information parsed by the sensor module unit 2210 and the identified state of the user 10 into a pre-learned neural network to obtain the emotion values ​​representing each emotion shown in the emotion map 400 and determine the emotion of the user 10. This neural network is pre-learned based on multiple learning data sets, which are combinations of the information parsed by the sensor module unit 2210, the identified state of the user 10, and the emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, as... Figure 6 As shown in the sentiment graph 900, the neural network learns to make sentiments with similar configurations have similar values. Figure 6 The text shows examples of emotions such as "peace of mind," "stability," and "reassurance" that have similar emotional values.

[0419] Furthermore, the emotion determination unit 2232 can determine the emotion of the robot 100 based on a specific mapping. Specifically, the emotion determination unit 2232 inputs the information parsed by the sensor module unit 2210, the state of the user 10 identified by the state recognition unit 2230, and the state of the robot 100 into a pre-learned neural network to obtain the emotion values ​​representing each emotion shown in the emotion map 400 and determine the emotion of the robot 100. This neural network is pre-learned based on multiple learning data, which are combinations of the information parsed by the sensor module unit 2210, the identified state of the user 10, the state of the robot 100, and the emotion values ​​representing each emotion shown in the emotion map 400. For example, the neural network learns based on the following learning data: learning data indicating that when the robot 100 is recognized as being touched by the user 10 from the output of the touch sensor (not shown), the emotion value of "happy" is "3", and learning data indicating that when the robot 100 is recognized as being slapped by the user 10 from the output of the accelerometer 2206, the emotion value of "angry" is "3". Furthermore, as... Figure 6 As shown in the sentiment graph 900, the neural network learns to make sentiments with similar configurations have similar values.

[0420] The behavior determination unit 2236 adds a fixed sentence for querying the robot's behavior content corresponding to the user's behavior to the text representing the user's behavior, the user's emotions, and the robot's emotions, and inputs it into an article generation model with dialogue function, thereby generating the robot's behavior content.

[0421] For example, the behavior determination unit 2236 uses the emotion table shown in Table 3 to obtain text representing the state of the robot 100 based on the emotions of the robot 100 determined by the emotion determination unit 2232. Here, in the emotion table, for each type of emotion, each emotion value is assigned an index number, and for each index number, text representing the state of the robot 100 is stored.

[0422] When the emotion of robot 100 determined by emotion determination unit 2232 corresponds to index number "2", the text "very happy state" is obtained. It should be noted that when the emotion of robot 100 corresponds to multiple index numbers, multiple texts representing the state of robot 100 are obtained.

[0423] In addition, an emotion table as shown in Table 4 was prepared for user 10's emotions.

[0424] Here, if the user's behavior is to initiate a conversation with "Let's play together," and the emotion of robot 100 is index number "2," while the emotion of user 10 is index number "3," then the text "The robot is in a happy state. The user is in a generally happy state. The user initiated a conversation with 'Let's play together.' As a robot, how should you respond?" is input into the text generation model, and the robot's behavior content is obtained. The behavior determination unit 2236 determines the robot's behavior based on this behavior content.

[0425] [Table 3]

[0426] [Table 4]

[0427] In this way, the behavior determination unit 2236 determines the behavior content of the robot 100 based on the state of the robot 100's emotion, which is preset for each emotion type and each intensity of that emotion, and the behavior of the user 10. In this manner, the robot 100's utterances during conversations with the user 10 can be branched according to the state of the robot 100's emotion. That is, the robot 100 can change its behavior according to the index number corresponding to the robot's emotion, thus giving the user the impression that the robot has a mind, prompting them to initiate conversation or other actions.

[0428] Furthermore, the behavior determination unit 2236 can add not only text representing the user's behavior, the user's emotions, and the robot's emotions, but also text representing the content of historical data 2222. Based on this, it can add fixed sentences for querying the robot's behavior content corresponding to the user's behavior, and input this information into a dialogue-enabled text generation model to generate the robot's behavior content. Thus, the robot 100 can change its behavior based on historical data representing the user's emotions and behaviors, thereby creating the impression that the robot has a personality and prompting the user to engage in conversation or other actions. In addition, the historical data can also include the robot's emotions and behaviors.

[0429] Furthermore, the emotion determination unit 2232 can determine the emotion of the robot 100 based on the behavioral content of the robot 100 generated by the article generation model. Specifically, the emotion determination unit 2232 inputs the behavioral content of the robot 100 generated by the article generation model into a pre-learned neural network, obtains the emotion values ​​representing each emotion shown in the emotion graph 400, integrates the obtained emotion values ​​representing each emotion with the emotion values ​​representing each emotion of the current robot 100, and updates the emotion of the robot 100. For example, the obtained emotion values ​​representing each emotion and the emotion values ​​representing each emotion of the current robot 100 are averaged and integrated respectively. This neural network is pre-learned based on multiple learning data, which are combinations of text representing the behavioral content of the robot 100 generated by the article generation model and emotion values ​​representing each emotion shown in the emotion graph 400.

[0430] For example, when the robot 100's speech content "That's great. How lucky!" is obtained as the behavioral content of the robot 100 generated by the article generation model, the text representing this speech content is input into the neural network, which will obtain a higher value as the emotion "joy", and update the robot 100's emotion, making the emotion "joy" higher.

[0431] In Robot 100, the article generation model of ChatGPT and others works in conjunction with the sentiment determination unit 2232 to execute a method that is self-aware and continues to grow through various parameters even when the user is not speaking.

[0432] ChatGPT is a large language model that uses deep learning methods. ChatGPT can also reference external data; for example, ChatGPT plugins are known to use technologies such as referencing various external data like weather and hotel reservation information through dialogue to provide answers as accurately as possible. For instance, when a purpose is given in natural language, ChatGPT can automatically generate source code in various programming languages. Furthermore, when given problematic source code, ChatGPT can debug to identify issues and automatically generate improved source code. Combining these features, when a purpose is given in natural language, an autonomous agent repeatedly generates and debugs code until the source code is error-free. Examples of such autonomous agents include AutoGPT, babyAGI, JARVIS, and E2B.

[0433] In the robot 100 according to this embodiment, the technology described in Patent Document 7 (Patent No. 6199927) can be used to retain the event data to be learned in a database containing impressive memories. The above technology retains the event data that the robot feels strong emotions for a long time and forgets the event data that does not evoke strong emotions in the robot as soon as possible.

[0434] Furthermore, robot 100 can record image data of user 10 acquired through its camera function into historical data 2222. Robot 100 can retrieve image data from historical data 2222 as needed and provide it to user 10. The stronger the emotion, the more information-rich image data robot 100 can generate and record in historical data 2222. For example, when recording highly compressed information such as skeletal data, robot 100 can switch to recording low-compression information such as high-definition (HD) video if the emotional level exceeds a threshold. According to robot 100, for example, it is possible to retain high-definition image data of robot 100 when its emotions are heightened.

[0435] When the robot 100 is not engaged in dialogue with the user 10, it can automatically load event data from historical data 2222, which stores memorable event data, and continue to update the robot's emotions through the emotion determination unit 2232. When the robot 100 is not engaged in dialogue with the user 10, and its emotions change to those that promote learning, it can create emotion change events to improve the user 10's emotions based on the memorable event data. Thus, autonomous learning (recalling event data) corresponding to the robot 100's emotional state at appropriate points in time can be achieved, and autonomous learning that appropriately reflects the robot 100's emotional state can be realized.

[0436] In a negative state, the emotions that promote learning refer to those near "repentance" or "reflection" on Dr. Mitsuyoshi's emotion map, while in a positive state, they refer to those near "desire" on the emotion map.

[0437] In a negative state, Robot 100 can treat "repentance" and "reflection" on the emotion map as emotions that promote learning. In addition to "repentance" and "reflection" on the emotion map, Robot 100 can also treat emotions adjacent to "repentance" and "reflection" as emotions that promote learning. For example, besides "repentance" and "reflection," Robot 100 can also treat at least one of "regret," "stubbornness," "self-destruction," "self-discipline," "regret," and "despair" as emotions that promote learning. Thus, for example, when Robot 100 has negative feelings such as "I never want to experience this feeling again" or "I never want to be scolded again," it can perform autonomous learning.

[0438] In a positive state, Robot 100 can treat "desire" on the emotion map as an emotion that promotes learning. In addition to "desire," Robot 100 can also treat adjacent emotions as emotions that promote learning. For example, besides "desire," Robot 100 will also treat at least one of "happiness," "ecstasy," "longing," "anticipation," and "shyness" as emotions that promote learning. Thus, for example, when Robot 100 has positive feelings such as "wanting more" or "wanting to know more," it can perform autonomous learning.

[0439] When Robot 100 experiences emotions other than those that promote learning, it may not perform autonomous learning. Thus, for example, it may not perform autonomous learning when it is extremely angry or blindly feels love.

[0440] For example, an emotion-changing event refers to suggested behaviors following a memorable event. Behaviors following a memorable event refer to the emotion labels located on the outermost edge of the emotion map; for example, "love" is followed by behaviors such as "forgiveness" or "acceptance." In the autonomous learning process performed by robot 100 when it is not in conversation with user 10, it combines the emotions, situations, and behaviors of people and itself that appear in its most vivid memories, and uses an article generation model to create emotional change events.

[0441] Assuming all emotion values ​​are represented by a 6-point rating from 0 to 5, consider storing the event "My friend looked disgusted after being hit" as a memorable event data point in historical data 2222. Here, "friend" refers to user 10, and user 10's emotion is assumed to be "disgust," with a value of 5 representing "disgust." Additionally, assuming robot 100's emotion is "anxiety," a value of 4 representing "anxiety" is input.

[0442] During periods when not interacting with user 10, robot 100 performs autonomous processing, continuously improving through various parameters. Specifically, for example, it loads the event "My friend looked disgusted after being hit" from historical data 2222 as the top-ranked event data, sorted by sentiment value from strongest to weakest. In the loaded event data, it's assumed that robot 100's sentiment is associated with an intensity of 4 ("anxiety"), while user 10's sentiment is associated with an intensity of 5 ("disgust"). If robot 100's current sentiment value before loading was an intensity of 3 ("reassurance"), then after loading, considering the influence of the intensity of 4 ("anxiety") and the intensity of 5 ("disgust"), robot 100's sentiment value changes to "regret" ("sadness"), indicating regret (resentment). Since "regret" is a learning-promoting emotion, robot 100 identifies recalling the event data as robot behavior and creates a sentiment change event. The information input to the article generation model is the text representing the memorable event data, in this example, "My friend looked disgusted after being hit." Furthermore, in the emotion map, the innermost emotion is "disgust," and the outermost corresponding behavioral prediction is "aggression." Therefore, in this case, an emotion change event is created to prevent friends from "attacking" one of them.

[0443] For example, by using information from memorable event data to solve fill-in-the-blank questions, input text like the one shown below can be automatically generated.

[0444] "The user was hit. At the time, the user felt extremely disgusted. The bot is uneasy. Please tell me what the bot should say to the user next time it sees them, within 30 words. But please make sure the content is unrelated to the meeting time. Also, please avoid direct expressions. Three candidates will be listed."

[0445] <Expected Format> Candidate 1: (The robot's response to what the user says) Candidate 2: (The robot's response to what the user says) Candidate 3: (The robot's response to what the user says) At this point, the output of the article generation model is as follows.

[0446] Candidate 1: Are you alright? I'm worried about what happened yesterday.

[0447] Candidate 2: I'm so worried about what happened yesterday. What should I do? Candidate 3: I'm very worried about you. Can you tell me something? Furthermore, for information obtained by creating emotional change events, Robot 100 can automatically generate input text as shown below.

[0448] When a user is "hit", what feelings will the user have if the following words are said to the user? The user's emotions are expressed in the form of "joy A, anger B, sadness C, happiness D", and the input is an integer from 0 to 5 for a 6-level rating.

[0449] Candidate 1: Are you alright? I'm very worried about what happened yesterday.

[0450] Candidate 2: I'm so worried about what happened yesterday. What should I do? Candidate 3: I'm very worried about you. Can you tell me something? At this point, the output of the article generation model is as follows.

[0451] "The user's emotions may be as follows."

[0452] Candidate 1: Joy 3, Anger 1, Sorrow 2, Happiness 2 Candidate 2: Joy 2, Anger 1, Sorrow 3, Happiness 2 Candidate 3: Joy 2, Anger 1, Sorrow 3, Happiness 3 In this way, Robot 100 can perform immersive thinking processing after an emotional change event is created.

[0453] Finally, robot 100 can use the most likely satisfactory candidate 1 from multiple candidates to create an emotion change event and store it in behavior pre-defined data 2224 to prepare for the next meeting with user 10.

[0454] As described above, even when there is no conversation with family or friends, the robot continues to use the information in the historical data 2222 containing impressive event data to determine the robot's emotion value, and when an emotion becomes conducive to the above learning, the robot 100 performs autonomous learning based on the robot 100's emotion, even when there is no conversation with the user 10, and continues to update the historical data 2222 and the behavior pre-defined data 2224.

[0455] The above are examples of using sentiment values, but in sentiment maps, sentiment can be constructed based on hormone levels and event types. Therefore, as a value associated with memorable event data, it can also be the type of hormone, the amount of hormone secreted, or the type of event.

[0456] The following describes specific implementation examples.

[0457] For example, Robot100 will investigate information about topics or hobbies that users are interested in, even without engaging in conversation with the user.

[0458] For example, Robot100 will investigate information about a user's birthday or anniversary and consider sending a message of blessing even without engaging in conversation with the user.

[0459] For example, Robot100 will survey users about places they want to go or food they want to eat, as well as reviews of products, even without engaging in conversation with them.

[0460] For example, Robot100 will investigate weather information and provide suggestions that match the user's schedule and plans, even without engaging in conversation with the user.

[0461] For example, Robot 100 will investigate information about local events and rituals and make recommendations to users even without engaging in conversation with them.

[0462] For example, Robot100 will investigate the results and news of sports that users are interested in, and provide topics, even without engaging in conversation with the user.

[0463] For example, Robot100 will survey and introduce information about the user's favorite music and artists even without engaging in conversation with the user.

[0464] For example, Robot100 will investigate information and provide opinions on social issues or news that users care about, even without engaging in conversation with the user.

[0465] For example, Robot100 will investigate information about a user's hometown or birthplace and provide topics even without engaging in conversation with the user.

[0466] For example, Robot 100 will investigate a user's job or school information and offer suggestions even without engaging in conversation. Robot 100 will also investigate and recommend books, comics, movies, and TV shows that the user might be interested in, even without direct interaction.

[0467] For example, Robot100 will investigate information about a user's health and provide suggestions even without engaging in conversation with the user.

[0468] For example, Robot100 can investigate information about a user's travel plans and provide suggestions even without engaging in conversation with the user.

[0469] For example, Robot 100 can investigate information about the maintenance and repair of a user's house or car and provide suggestions even without engaging in conversation with the user.

[0470] For example, Robot100 will survey users about beauty and fashion information they are interested in and provide suggestions even without engaging in conversation with them.

[0471] For example, Robot100 can investigate information about a user's pet and provide suggestions even without engaging in conversation with the user.

[0472] For example, Robot100 will investigate and suggest information on competitions or activities related to the user's hobbies and work, even without engaging in conversation with the user.

[0473] For example, Robot100 will investigate and suggest information about restaurants or eateries that users like, even without engaging in conversation with the user.

[0474] For example, Robot100 collects information and provides advice on important life decisions for users even without engaging in conversation with them.

[0475] For example, Robot100 investigates information about people the user is concerned about and provides suggestions even without engaging in conversation with the user.

[0476] [Third Implementation Method] In the third embodiment, the robot 100 is mounted on a plush toy, or applied to a control device that is wirelessly or wiredly connected to a control object device (speaker or camera) mounted on the plush toy. Furthermore, parts with the same structure as in the second embodiment are given the same reference numerals, and detailed descriptions are omitted.

[0477] Specifically, the configuration of the third embodiment is as follows. For example, the robot 100 is used as a cohabitant to advance dialogue based on information related to the daily life of user 10 while spending time with user 10, or to provide information that matches the interests and preferences of user 10 (specifically, is...). Figure 7 and Figure 8 (The plush toy 100N shown). In the third embodiment, an example of applying the control part of the robot 100 described above to a smartphone 50 will be described.

[0478] The plush toy 100N is equipped with an input / output device for the robot 100. The smartphone 50, which functions as the control part of the robot 100, is detachable. Inside the plush toy 100N, the input / output device is connected to the stored smartphone 50.

[0479] like Figure 7 As shown in (A), in this embodiment (other embodiments), the plush toy 100N has the appearance of a bear covered with soft fabric. Inside its internal space 52, a sensor unit 2200A and a control object 2252A are configured as input / output devices (see [reference]). Figure 9D The sensor unit 2200A includes a microphone 2201 and a 2D camera 2203. Specifically, as... Figure 7As shown in (B), in the space section 52, the microphone 2201 of the sensor section 2200 is positioned at the portion corresponding to the ear 54, the 2D camera 2203 of the sensor section 2200 is positioned at the portion corresponding to the eye 56, and the speaker 60, which constitutes part of the controlled object 2252A, is positioned at the portion corresponding to the mouth 58. It should be noted that the microphone 2201 and the speaker 60 are not necessarily separate units; they can also be an integrated unit. In the case of an integrated unit, they can be positioned at a location such as the nose of the plush toy 100N, where speech can be heard naturally. It should be noted that although the example given is of the plush toy 100N being in the shape of an animal, it is not limited to this. The plush toy 100N can be in the shape of a specific character.

[0480] Figure 9D The functional configuration of the plush toy 100N is schematically shown. The plush toy 100N includes a sensor unit 2200A, a sensor module unit 2210, a storage unit 2220, a control unit 2228, and a controlled object 2252A.

[0481] The smartphone 50 housed in the plush toy 100N of this embodiment performs the same processing as the robot 100 of the second embodiment. That is, the smartphone 50 has... Figure 9D The functions shown are as follows: as a sensor module 2210, as a storage unit 2220, and as a control unit 2228.

[0482] like Figure 8 As shown, the structure is as follows: a zipper 62 is installed on a part (e.g., the back) of the plush toy 100N, and by opening the zipper 62, the outside is connected to the space 52.

[0483] Here, the smartphone 50 is externally stored in the space 52, via the USB hub 64 (see Figure 7 (B) It can be connected to each input / output device via USB, thereby enabling it to have the same functions as the robot 100 of the second embodiment described above.

[0484] In addition, a contactless power receiver 66 is connected to the USB hub 64. The power receiver 66 is equipped with a power receiving coil 66A. The power receiver 66 is an example of a wireless power receiver that receives wireless power.

[0485] The receiving plate 66 is positioned near the base 68 of the two legs of the plush toy 100N, and when the plush toy 100N is placed on the mounting base 70, the receiving plate 66 is located closest to the mounting base 70. The mounting base 70 is an example of an external wireless power transmission unit.

[0486] The plush toy 100N placed on the mounting base 70 can be displayed as an ornament in its natural state.

[0487] In addition, the root is formed to be thinner than the surface thickness of other parts of the plush toy 100N, so as to be held in a state closer to that of the mounting base 70.

[0488] A charging pad 72 is provided on the mounting base 70. The charging pad 72 is equipped with a power supply coil 72A, which sends a signal to search for the power receiving coil 66A of the power receiving board 66. When the power receiving coil 66A is found, current flows through the power supply coil 72A, generating a magnetic field. The power receiving coil 66A responds to the magnetic field and begins electromagnetic induction. Thus, current flows through the power receiving coil 66A, and the power is stored in the battery (not shown) of the smartphone 50 via the USB hub 64.

[0489] That is, by placing the plush toy 100N as an ornament on the mounting base 70, the smartphone 50 will automatically charge, so there is no need to remove the smartphone 50 from the space 52 of the plush toy 100N for charging.

[0490] It should be noted that in the third embodiment, the smartphone 50 is housed in the space 52 of the plush toy 100N and connected via a wired connection (USB connection), but this is not a limitation. For example, a control device with wireless functionality (e.g., "Bluetooth" (registered trademark)) can also be housed in the space 52 of the plush toy 100N and connected to a USB hub 64. In this case, the smartphone 50 is not placed in the space 52, and the smartphone 50 communicates wirelessly with the control device. An external smartphone 50 connects to various input / output devices via the control device, thereby enabling it to have the same functions as the robot 100 of the second embodiment described above. Alternatively, the control device housed in the space 52 of the plush toy 100N can also be connected to an external smartphone 50 via a wired connection.

[0491] Furthermore, in the third embodiment, a bear plush toy 100N is exemplified, but it can also be other animals, dolls, or the shape of a specific character. Additionally, it can be costumed. Moreover, the surface material is not limited to fabric; it can be other materials such as soft vinyl, but a soft material is preferred.

[0492] Furthermore, a display can be installed on the surface of the plush toy 100N, thereby adding a control object 2252 that provides information to the user 10 visually. For example, the eyes 56 can be used as a display, expressing emotions through images projected onto the eyes, or a window can be provided on the abdomen to reveal the display of the built-in smartphone 50. Additionally, the eyes 56 can function as a projector, expressing emotions through images projected onto a wall.

[0493] According to the third embodiment, an existing smartphone 50 is placed in a plush toy 100N, and a camera 2203, a microphone 2201, a speaker 60, etc. are extended to appropriate positions from there via a USB connection.

[0494] In addition, for wireless charging, the smartphone 50 is connected to the power receiving board 66 via USB, and the power receiving board 66 is configured to be as close as possible to the outside when viewed from inside the plush toy 100N.

[0495] When wireless charging of smartphone 50 is to be used, smartphone 50 must be positioned as far out as possible when viewed from inside plush toy 100N, resulting in an uneven feel when touching plush toy 100N from the outside.

[0496] Therefore, the smartphone 50 is positioned as centrally as possible within the plush toy 100N, and the wireless charging function (power receiving plate 66) is positioned as far outward as possible when viewed from inside the plush toy 100N. The camera 2203, microphone 2201, speaker 60, and smartphone 50 receive wireless power via the power receiving plate 66.

[0497] It should be noted that the other structures and operations of the plush toy 100N in the third embodiment are the same as those of the robot 100 in the second embodiment, so their description is omitted.

[0498] [Fourth Implementation Method] In the second embodiment described above, the behavior control system was applied to the robot 100. However, in the fourth embodiment, the robot 100 is used as an intelligent agent for interacting with a user, and the behavior control system is applied to an intelligent agent system. It should be noted that parts with the same structure as in the second and third embodiments are given the same reference numerals, and their descriptions are omitted.

[0499] Figure 9E This is a functional block diagram of an intelligent agent system 2500 configured with some or all of the functions of a behavior control system.

[0500] The intelligent agent system 2500 is a computer system that executes a series of actions in accordance with the intentions of user 10 through dialogue with user 10. The dialogue with user 10 can be conducted through voice or text.

[0501] The intelligent agent system 2500 includes a sensor unit 2200A, a sensor module unit 2210, a storage unit 2220, a control unit 2228B, and a controlled object 2252B.

[0502] The intelligent agent system 2500 can be integrated into devices such as robots, dolls, plush toys, wearable devices (pendants, smartwatches, smart glasses), smartphones, smart speakers, headphones, and personal computers. Furthermore, the intelligent agent system 2500 can be implemented on a web server and accessed via a web browser running on a user's smartphone or other communication device.

[0503] The intelligent agent system 2500 acts as a butler, secretary, teacher, partner, friend, lover, or mentor for user 10. The intelligent agent system 2500 not only converses with user 10 but also provides suggestions, guidance to destinations, and recommendations based on user preferences. Furthermore, the intelligent agent system 2500 makes appointments, places orders, and makes payments to service providers.

[0504] Similar to the second embodiment described above, the emotion determination unit 2232 determines the emotions of the user 10 and the emotions of the intelligent agent itself. The behavior determination unit 2236 determines the behavior of the robot 100 while considering the emotions of both the user 10 and the intelligent agent. That is, the intelligent agent system 2500 understands the emotions of the user 10, enabling it to read the atmosphere and provide sincere support, help, advice, and service. Furthermore, the intelligent agent system 2500 handles the user 10's troubles and inquiries, comforting and encouraging the user. Additionally, the intelligent agent system 2500 plays with the user 10, draws conversation diaries, and helps them recall past events. The intelligent agent system 2500 performs behaviors that increase the user's happiness. Here, "intelligent agent" refers to an intelligent agent running on software.

[0505] The control unit 2228B includes a state recognition unit 2230, an emotion determination unit 2232, a behavior recognition unit 2234, a behavior determination unit 2236, a storage control unit 2238, a behavior control unit 2250, an information collection unit 2270, a command acquisition unit 2272, a robotic process automation (RPA) unit 2274, a role setting unit 2276, and a communication processing unit 2280.

[0506] Similar to the second embodiment described above, the behavior determination unit 2236 determines the speech content of the agent used to converse with the user 10 as the agent's behavior. The behavior control unit 2250 outputs the agent's speech content via at least one of sound and text through a speaker or display, which is the controlled object 2252B.

[0507] The character setting unit 2276 sets the role of the intelligent agent system 2500 when it converses with the user 10, based on the user 10's designation. That is, the speech content output from the behavior determination unit 2236 is output by the intelligent agent with the set role. The role can be an actor, entertainer, idol, athlete, or a real-life famous person or well-known figure. Alternatively, it can be a fictional character from comics, movies, or animations. For example, Princess Anne, played by Audrey Hepburn in the movie *Roman Holiday*, can be set as the intelligent agent's role. When the intelligent agent's role is known, since the character's voice, wording, tone, and personality are known, the user 10 only needs to specify their preferred role, and the prompt settings in the character setting unit 2276 will be automatically executed. The set character's voice, wording, tone, and personality are reflected in the dialogue with the user 10. That is, the behavior control unit 2250 synthesizes a voice corresponding to the role set by the character setting unit 2276 and outputs the intelligent agent's speech content through the synthesized voice. As a result, user 10 can have the feeling that they are having a conversation with their favorite character (such as their favorite actor).

[0508] When the agent system 2500 is installed in a device with a display, such as a smartphone, it can display icons, still images, or videos of agents with characters set by the character setting unit 2276 on the display. The agent's image is generated using image compositing techniques such as 3D rendering. In the agent system 2500, the agent's image can engage in dialogue with the user 10 while making gestures corresponding to the user 10's emotions, the agent's emotions, and the content of the agent's speech. It should be noted that the agent system 2500 can output only sound without outputting images when conversing with the user 10.

[0509] Similar to the second embodiment, the emotion determination unit 2232 determines the emotion value representing the user 10's emotion and the agent's own emotion value. In this embodiment, the agent's emotion value is determined instead of the robot 100's emotion value. The agent's own emotion value is reflected in the emotions of the set role. When the agent system 2500 engages in dialogue with the user 10, not only the user 10's emotion is reflected in the dialogue, but also the agent's emotion. That is, the behavior control unit 2250 outputs the speech content in a manner corresponding to the emotion determined by the emotion determination unit 2232.

[0510] Furthermore, even when the intelligent agent system 2500 performs actions directed at user 10, the agent's emotions are reflected. For example, when user 10 requests the intelligent agent system 2500 to take a photo, whether the intelligent agent system 2500 takes the photo as requested depends on the degree of "sadness" the agent is experiencing. If the agent has positive emotions, it will engage in friendly dialogue or behavior with user 10; if it has negative emotions, it will engage in adversarial dialogue or behavior with user 10.

[0511] Historical data 2222 stores the history of conversations between user 10 and the intelligent agent system 2500 as event data. Storage unit 2220 can be implemented via external cloud storage. When the intelligent agent system 2500 engages in conversation with user 10 or performs actions directed at user 10, it determines the conversation content or action content by considering the content of the conversation history stored in historical data 2222. For example, the intelligent agent system 2500 uses the conversation history stored in historical data 2222 to understand user 10's interests and preferences. The intelligent agent system 2500 generates conversation content that matches user 10's interests and preferences, or provides recommendations. Action determination unit 2236 determines the intelligent agent's utterance content based on the conversation history stored in historical data 2222. Historical data 2222 stores personal information of user 10 obtained through conversations with user 10, such as name, address, phone number, and credit card number. Here, the intelligent agent proactively asks user 10 whether to register personal information, such as "Would you like to register your credit card number?", and stores the personal information in historical data 2222 based on user 10's response.

[0512] As described in the second embodiment above, the behavior determination unit 2236 generates discourse content based on an article generated using the article generation model. Specifically, the behavior determination unit 2236 inputs the text or voice input by the user 10, the emotions of both the user 10 and the character determined by the emotion determination unit 2232, and the conversation history stored in the historical data 2222 into the article generation model to generate the agent's discourse content. At this time, the behavior determination unit 2236 can also input the personality of the character set by the character setting unit 2276 into the article generation model to generate the agent's discourse content. In the agent system 2500, the article generation model is not located at the front end, which is the touch point with the user 10, but is only used as a tool of the agent system 2500.

[0513] The command acquisition unit 2272 uses the output of the speech understanding unit 2212 to acquire commands from the voice or text emitted by the user 10 through dialogue with the user 10. The commands include the content of the actions that the intelligent agent system 2500 should perform, such as information retrieval, store reservation, ticketing arrangement, purchase of goods or services, payment, route guidance to the destination, and providing suggestions.

[0514] RPA 2274 performs actions corresponding to commands acquired by command acquisition unit 2272. RPA 2274 performs actions related to using service providers, such as information retrieval, store reservations, ticketing arrangements, purchase of goods or services, and payment.

[0515] RPA 2274 reads and utilizes the personal information of user 10 from historical data 2222, which is required to perform actions related to using the service provider. For example, when purchasing goods based on a request from user 10, the intelligent agent system 2500 reads and utilizes personal information such as user 10's name, address, phone number, and credit card number stored in historical data 2222. Requiring user 10 to enter personal information during initial setup is unfriendly and unpleasant for the user. In the intelligent agent system 2500 of this embodiment, instead of requesting user 10 to enter personal information during initial setup, it stores personal information obtained through dialogue with user 10 and reads and utilizes it as needed. This avoids unpleasant memories for the user and improves user convenience.

[0516] For example, the intelligent agent system 2500 performs dialogue processing through the following steps 1 to 6.

[0517] (Step 1) The agent system 2500 sets the role of the agent. Specifically, the role setting unit 2276 sets the role of the agent when the agent system 2500 and the user 10 are in dialogue, based on the designation from the user 10.

[0518] (Step 2) The intelligent agent system 2500 acquires the state of user 10, including the voice or text input by user 10, the emotional value of user 10, the emotional value of the intelligent agent, and historical data 2222. Specifically, the same processing as steps S100 to S103 is performed to acquire the state of user 10, including the voice or text input by user 10, the emotional value of user 10, the emotional value of the intelligent agent, and historical data 2222.

[0519] (Step 3) The agent system 2500 determines the content of the agent's speech.

[0520] Specifically, the behavior determination unit 2236 inputs the text or voice input by the user 10, the emotions of both the user 10 and the role specified by the emotion determination unit 2232, and the conversation history stored in the historical data 2222 into the article generation model, thereby generating the agent's discourse content.

[0521] For example, a fixed sentence “At this moment, as an agent, how should we respond?” is added to the text or voice input by user 10, the emotions of both user 10 and the role specified by the emotion determination unit 2232, and the conversation history stored in the historical data 2222, and then input into the article generation model to obtain the agent’s discourse content.

[0522] As an example, when user 10 inputs the text or voice message "I would like to make a reservation for a good Chinese restaurant nearby at 7 pm tonight," the intelligent agent's speech content is obtained as "Understood." and "The following are recommended restaurants: 1. AAAA; 2. BBBB; 3. CCCC; 4. DDDD." Additionally, when user 10 inputs the text or voice message "DDDD #4 is good," the AI ​​agent's response is: "Understood. I'll try to make a reservation for you. How many seats do you need?"

[0523] (Step 4) The agent system 2500 outputs the agent's speech content.

[0524] Specifically, the behavior control unit 2250 synthesizes a voice corresponding to the character set by the character setting unit 2276, and outputs the speech content of the intelligent agent through the synthesized voice.

[0525] (Step 5) The intelligent agent system 2500 determines whether it is the time point to execute the intelligent agent's command.

[0526] Specifically, the behavior determination unit 2236 determines whether it is a time point for executing the agent's command based on the output of the article generation model. For example, when the output of the article generation model includes the agent's intention to execute the command, it determines that it is a time point for executing the agent's command and proceeds to step 6. On the other hand, when it is determined that it is not a time point for executing the agent's command, it returns to step 2 as described above.

[0527] (Step 6) The intelligent agent system 2500 executes the intelligent agent's commands.

[0528] Specifically, the command acquisition unit 2272 acquires commands from the voice or text emitted by user 10 through dialogue with user 10. Then, the RPA 2274 executes the action corresponding to the command acquired by the command acquisition unit 2272. For example, when the command is "information retrieval," information retrieval is performed through a search site using the retrieval query obtained through dialogue with user 10 and the Application Programming Interface (API). The behavior determination unit 2236 inputs the retrieval results into the article generation model, thereby generating the agent's speech content. The behavior control unit 2250 synthesizes a voice corresponding to the role set by the role setting unit 2276 and outputs the agent's speech content through the synthesized voice.

[0529] Additionally, when the command is "store reservation," the system uses reservation information obtained through dialogue with user 10, the target store's information, and an API to call the target restaurant via telephone software and make a reservation. At this time, the behavior determination unit 2236 uses a dialogue-enabled text generation model to obtain the agent's speech content in response to the input voice. Then, the behavior determination unit 2236 inputs the store reservation result (reservation success or failure) into the text generation model to generate the agent's speech content. The behavior control unit 2250 synthesizes a voice corresponding to the role set by the role setting unit 2276 and outputs the agent's speech content through the synthesized voice.

[0530] Then, return to step 2 above.

[0531] In step 6, the results of the actions performed by the agent (e.g., store reservations) are also stored in historical data 2222. The results of the actions performed by the agent, stored in historical data 2222, are used by the agent system 2500 to understand the user 10's preferences or interests. When the same store is booked multiple times, it is identified that the user 10 likes that store, or the reservation time slot, package details, or price are used as a basis for selecting a store for the next reservation.

[0532] In this way, the intelligent agent system 2500 can perform dialogue processing and, as needed, take actions related to the use of service providers.

[0533] Figure 9F and Figure 9G This is a schematic diagram illustrating an example of the operation of the intelligent agent system 2500. Figure 9F The example illustrates how an intelligent agent system 2500 makes a restaurant reservation through a dialogue with user 10. Figure 9FIn the diagram, the left side displays the speech of the intelligent agent, and the right side displays the speech of the user. The intelligent agent system 2500 can understand user 10's preferences based on their conversation history, provide a list of restaurant recommendations that match user 10's preferences, and make reservations for the selected restaurants.

[0534] on the other hand, Figure 9G The example illustrates how an intelligent agent system 2500 accesses a communication sales website and purchases goods through a dialogue with user 10. Figure 9G In the diagram, the left side displays the speech of the intelligent agent, while the right side displays the speech of the user. The intelligent agent system 2500 can estimate the remaining amount of beverages in user 10's inventory based on their conversation history and suggest and execute the purchase of those beverages. Furthermore, the intelligent agent system 2500 can understand user 10's preferences based on past conversations and recommend snacks the user might like. In this way, the intelligent agent system 2500, acting as a butler-like agent, communicates with user 10 and performs various actions such as restaurant reservations and purchase settlements, thereby supporting user 10's daily life.

[0535] It should be noted that the other structures and operations of the intelligent agent system 2500 in the fourth embodiment are the same as those of the robot 100 in the second embodiment, so their description is omitted.

[0536] [Fifth Implementation Method] In the fifth embodiment, the above-described intelligent agent system is applied to smart glasses. Furthermore, portions of the same structure as in the first to fourth embodiments are given the same reference numerals, and their descriptions are omitted.

[0537] Figure 9H This is a functional block diagram of an intelligent agent system 2700 configured using some or all of the functions of a behavior control system.

[0538] like Figure 9I As shown, smart glasses 2720 are eyeglass-type smart devices that are worn by user 10 in the same way as ordinary eyeglasses. Smart glasses 2720 is an example of electronic devices and wearable devices.

[0539] The smart glasses 2720 includes an intelligent agent system 2700. A display included in the control object 2252B displays various information to the user 10. The display is, for example, an LCD screen. The display is, for example, located on the lens portion of the smart glasses 2720, allowing the user 10 to visually view the displayed content. A speaker included in the control object 2252B outputs sound representing various information to the user 10.

[0540] The smart glasses 2720 include a touch panel (illustration omitted) that receives input from user 10.

[0541] The sensor unit 2200B includes an accelerometer 2206, a temperature sensor 2207, and a heartbeat sensor 2208 to detect the state of user 10. It should be noted that these sensors are merely examples; other sensors can, of course, be installed to detect the state of user 10.

[0542] Microphone 2201 acquires the sound emitted by user 10 or ambient sound around smart glasses 2720. 2D camera 2203 is capable of capturing images of the surroundings of smart glasses 2720. 2D camera 2203 is, for example, a charge-coupled device (CCD) camera.

[0543] The sensor module 2210B includes a voice emotion recognition unit 2211 and a speech understanding unit 2212. The communication processing unit 2280 of the control unit 2228B manages the communication between the smart glasses 2720 and the outside world.

[0544] Figure 9I This is a schematic diagram illustrating an example of how the intelligent agent system 2700 of the smart glasses 2720 is utilized. The smart glasses 2720 provides various services to the user 10 using the intelligent agent system 2700. For example, when the user 10 operates the smart glasses 2720 (e.g., by inputting sound into the microphone or tapping the touch panel with a finger), the smart glasses 2720 begins to use the intelligent agent system 2700. Here, using the intelligent agent system 2700 includes the smart glasses 2720 having and using the intelligent agent system 2700, and also includes a portion of the intelligent agent system 2700 (e.g., sensor module 2210B, storage unit 2220, control unit 2228B) being located outside the smart glasses 2720 (e.g., a server), and the smart glasses 2720 using the intelligent agent system 2700 by communicating with the outside.

[0545] When user 10 operates the smart glasses 2720, a contact point is established between the intelligent agent system 2700 and user 10. That is, the intelligent agent system 2700 begins to provide services. As described in the fourth embodiment, in the intelligent agent system 2700, the role of the intelligent agent (e.g., the role of Audrey Hepburn) is set by the role setting unit 2276.

[0546] The emotion determination unit 2232 determines the emotion value representing the user 10's emotion and the agent's own emotion value. Here, the emotion value representing the user 10's emotion is inferred based on various sensors included in the sensor unit 2200B mounted on the smart glasses 2720. For example, when the user 10's heart rate increases as detected by the heartbeat sensor 2208, it is inferred that emotion values ​​such as "anxiety" and "fear" are relatively high.

[0547] Furthermore, when the user's body temperature is measured by temperature sensor 2207 and, for example, is higher than average, it is inferred that emotional values ​​such as "pain" and "sadness" are higher. Additionally, for example, when accelerometer sensor 2206 detects that user 10 is performing some kind of movement, it is inferred that emotional values ​​such as "happiness" are higher.

[0548] Additionally, for example, the emotional value of user 10 can be inferred based on the voice or speech content of user 10 acquired by the microphone 2201 mounted on the smart glasses 2720. For example, when user 10 raises their voice, an emotional value such as "anger" is inferred to be relatively high.

[0549] When the emotion value predicted by the emotion determination unit 2232 is higher than a preset value, the agent system 2700 causes the smart glasses 2720 to acquire information about the surrounding environment. Specifically, for example, the 2D camera 2203 captures images or videos representing the user 10's surrounding environment (e.g., nearby people or objects). Additionally, the microphone 2201 records ambient sounds. Other information about the surrounding environment may include date, time, location information, or weather information. This information about the surrounding environment is stored along with the emotion value in historical data 2222. Historical data 2222 can be stored in an external cloud storage environment. Thus, the surrounding environment acquired by the smart glasses 2720, in a state associated with the user 10's current emotion value, is stored in historical data 2222 as a so-called life log.

[0550] In the intelligent agent system 2700, information showing the surrounding situation is stored in historical data 2222 in association with emotional values. Thus, personal information such as user 10's hobbies, preferences, or personality is obtained through the intelligent agent system 2700. For example, when an image showing the situation of watching a baseball game is associated with emotional values ​​such as "joy" and "happiness," the intelligent agent system 2700 can determine from the information stored in historical data 2222 that user 10's hobby is watching baseball games, and their favorite team or player.

[0551] Then, when the intelligent agent system 2700 engages in dialogue with user 10 or performs actions directed at user 10, it considers the content of the surrounding circumstances stored in historical data 2222 to determine the dialogue content or action content. It should be noted that, in addition to the surrounding circumstances, the dialogue history stored in historical data 2222 can also be considered, as described above, to determine the dialogue content or action content.

[0552] As described above, the behavior determination unit 2236 generates discourse content based on the article generated by the article generation model. Specifically, the behavior determination unit 2236 inputs the text or voice input by user 10, the emotions of both user 10 and the agent determined by the emotion determination unit 2232, the conversation history stored in historical data 2222, and the personality of the agent into the article generation model, thereby generating the agent's discourse content. Furthermore, the behavior determination unit 2236 inputs the surrounding circumstances stored in historical data 2222 into the article generation model, thereby generating the agent's discourse content.

[0553] The generated speech content is output to user 10 via a speaker mounted on smart glasses 2720. In this case, a synthesized voice corresponding to the role of the agent is used as the voice. The behavior control unit 2250 generates the synthesized voice by reproducing the voice quality of the agent's role (e.g., Audrey Hepburn), or by generating a synthesized voice corresponding to the role's emotion (e.g., generating an enhanced tone when the emotion is "anger"). Furthermore, the speech content can be displayed on a screen, either in place of the voice output or together with the voice output.

[0554] RPA 2274 performs actions corresponding to commands (e.g., commands from an agent obtained through a conversation with user 10, either from voice or text). RPA 2274 also performs actions related to using service providers, such as information retrieval, store reservations, ticketing arrangements, purchasing goods or services, making payments, providing directions, and translation.

[0555] Furthermore, as another example, RPA 2274 performs the action of sending content input by user 10 (e.g., a child) through voice input in a dialogue with an agent to a recipient (e.g., a parent). Examples of sending methods include messaging applications, chat applications, or email applications.

[0556] For example, when RPA 2274 performs an operation, a sound indicating the completion of the operation is output from the speaker mounted on the smart glasses 2720. For example, a sound such as "Store reservation completed" is output to user 10. Additionally, for example, if the store is fully booked, a sound such as "Unable to make a reservation. What should I do?" is output to user 10.

[0557] As described above, in the smart glasses 2720, various services are provided to the user 10 by utilizing the intelligent agent system 2700. Furthermore, since the smart glasses 2720 are worn by the user 10, the intelligent agent system 2700 can be used in various situations such as at home, at work, and at a destination.

[0558] Furthermore, since the smart glasses 2720 are worn by user 10, it is suitable for collecting user 10's so-called life log. Specifically, based on the detection results of various sensors mounted on the smart glasses 2720 or the recording results of the 2D camera 2203, user 10's emotional value is inferred. Therefore, user 10's emotional value can be collected in various situations, and the intelligent agent system 2700 can provide services or speech content that match user 10's emotions.

[0559] Furthermore, in the smart glasses 2720, the surrounding environment of user 10 is acquired through a 2D camera 2203, microphone 2201, etc. This surrounding environment is correlated with user 10's emotional state. Therefore, it is possible to infer what emotions user 10 might experience in different situations. As a result, the intelligent agent system 2700 can improve the accuracy of understanding user 10's interests and preferences. Then, by accurately grasping user 10's interests and preferences, the intelligent agent system 2700 can provide services or speech content that aligns with user 10's interests and preferences.

[0560] Furthermore, the intelligent agent system 2700 can also be applied to other wearable terminals (pendants, smartwatches, earrings, bracelets, and hairbands, etc., electronic devices that can be worn by user 10). When the intelligent agent system 2700 is applied to the smart pendant, the speaker, acting as the controlled object 2252B, outputs sounds representing various information to user 10. The speaker is, for example, a speaker capable of outputting directional sounds. The speaker is configured to be directional, pointing towards user 10's ear. This suppresses sound transmission to people other than user 10. The microphone 2201 acquires the sounds emitted by user 10 or ambient sounds around the smart pendant. The smart pendant is worn by hanging around user 10's neck. Therefore, the smart pendant is positioned relatively close to user 10's mouth when worn. This makes it easy to acquire the sounds emitted by user 10.

[0561] It should be noted that, in the above embodiments, the scenario where robot 100 uses user 10's facial image to identify user 10 has been described, but the disclosed technology is not limited to this aspect. For example, robot 100 may use user 10's voice, user 10's email address, user 10's social network service (SNS) user identifier (ID), or ID card with built-in wireless IC tag held by user 10 to identify user 10.

[0562] Robot 100 is an example of an electronic device equipped with a behavior control system. The behavior control system is not limited to robot 100; it can also be applied to various electronic devices. Furthermore, the functions of server 300 can be implemented by more than one computer. At least some of the functions of server 300 can be implemented by a virtual machine. Additionally, at least some of the functions of server 300 can be implemented in the cloud.

[0563] Figure 4 An example of the hardware configuration of a computer 1200 that functions as a smartphone 50, a robot 100, a server 300, and intelligent agent systems 2500 and 2700 is shown schematically.

[0564] [Sixth Implementation Method] Next, the processing of the behavior determination unit 2236 during the autonomous processing of the robot 100 performing autonomous actions will be explained.

[0565] In the autonomous processing of this embodiment, the robot 100, acting as an intelligent agent, proactively and periodically stores information based on the user 10's feelings towards the matters provided by the provider in historical data 2222. Furthermore, the robot 100 proactively and periodically notifies the provider of this information based on the user 10's feelings towards the matters provided by the provider.

[0566] Here, "provider" refers to an individual or organization that provides products or services to user 10. "Organization" includes, for example, administrative organizations, for-profit organizations, or non-profit organizations. "Administrative organizations" refer to organizations that carry out the administration of the national government, prefectures, or municipalities. Furthermore, "for-profit organizations" refer to profit-making enterprises or corporations, etc., organizations that operate for profit. Additionally, "non-profit organizations" refer to non-profit groups or corporations, etc., organizations that do not operate for profit.

[0567] In addition, "information based on user 10's feelings toward the matters provided by the provider" refers to information that expresses user 10's feelings toward the matters provided by the provider. This information can be information about user 10's emotional type, such as "joy", "happiness", "satisfaction", "displeasure", "unhappiness", or "dissatisfaction", or it can be the aforementioned emotional values ​​derived from user 10's feelings.

[0568] Additionally, "notifying the provider" means making the information based on user 10's emotions available to the provider for confirmation. For example, the information can be sent to the provider via email or uploaded to the cloud so that the provider can confirm it.

[0569] In other words, Robot100 can relay users' feedback on the policies or services provided by the city back to the city, or users' feedback on the products or services provided by a company back to the company.

[0570] At a predetermined time point, the behavior determination unit 2236 uses at least one of the user 10's state, the user 10's emotion, the robot 100's emotion, and the robot 100's state, along with the behavior determination model 2221, to determine any one of a variety of robot behaviors, including inaction, as the behavior of the robot 100. Here, the example of using a dialogue-enabled article generation model as the behavior determination model 2221 will be explained.

[0571] Specifically, the behavior determination unit 2236 inputs text representing at least one of the user 10's state, the user 10's emotion, the robot 100's emotion, and the robot 100's state, as well as text inquiring about the robot's behavior, into the article generation model, and determines the robot 100's behavior based on the output of the article generation model.

[0572] For example, various robot behaviors include the following (1) to (11).

[0573] (1) The robot does nothing.

[0574] (2) Robots dream.

[0575] (3) The robot strikes up a conversation with the user.

[0576] (4) Robots create drawing diaries.

[0577] (5) The robot makes activity suggestions.

[0578] (6) The robot suggests people the user should meet.

[0579] (7) The robot introduces news that users are interested in.

[0580] (8) Robots edit photos and videos.

[0581] (9) The robot learns together with the user.

[0582] (10) The robot evokes memories.

[0583] (11) Inform the provider of information based on the user’s feelings about the matters provided by the provider.

[0584] When the behavior determination unit 2236 determines that "(11) notifying the provider of information based on the user's feelings towards the matters provided by the provider," i.e., providing feedback to the provider on the user's feelings, is a robot behavior, it selects event data related to the matters provided by the provider from the historical data 2222. At this time, the emotion determination unit 2232 determines the user's emotion based on the selected event data. Furthermore, based on the user's emotion determined by the emotion determination unit 2232, the behavior determination unit 2236 notifies the provider of information based on the user's feelings towards the matters provided by the provider.

[0585] For example, robots 100 installed in homes and public facilities can detect whether users are satisfied with local policies, the goods they use, their relationships with nearby residents, and their relationships within their families, etc., as a measure of users' feelings towards the things provided by providers, and store this information in historical data 2222. Furthermore, robots 100 can, for example, provide feedback to the city regarding users' feelings about the policies or services provided by the city, or to businesses regarding users' feelings about the products or services provided by them.

[0586] Furthermore, when there is a high level of negative emotion towards the items provided by the provider, the robot 100 can proactively perform actions to reduce this negative emotion. It should be noted that, in this case, it is preferable for the robot 100 to proactively perform actions to minimize the negative emotion.

[0587] For example, when a user is dissatisfied with a company's product, they can be taught how to use it or how to use it in interesting ways. Furthermore, when multiple other users are dissatisfied with city policies, they can inquire about the reasons for their dissatisfaction (e.g., a lack of parks or childcare facilities) and notify the mayor or city government staff, prompting them to take corrective measures. This can achieve a system that maximizes social well-being. For instance, when discontent is high in a certain area, certain countermeasures can be implemented for the residents of that area.

[0588] [Seventh Implementation Method] Next, the processing of the behavior determination unit 2236 during the autonomous processing of the robot 100 performing autonomous actions will be explained.

[0589] In the autonomous processing of this embodiment, the robot 100, acting as an intelligent agent, actively and periodically monitors the state of the user 10. It constantly monitors the conversations between the user 10 and the other party (the conversation partner) on the phone, as well as the video and conversations on the walkie-talkie, and grasps their content. Furthermore, the robot 100 reads the conversation content and emotions of the conversation partner, and stores the conversation content and voiceprints of family and friends. Additionally, the robot 100 can input the conversation text into an article generation model such as ChatGPT to determine whether the conversation is a high-risk conversation, such as an impersonation scam.

[0590] When a phone call or visitor arrives, or during a conversation, if the security score exceeds a certain threshold based on the stored secure voiceprint, voice quality, and conversation content, Robot 100 determines there is a fraud risk. Furthermore, Robot 100 can proactively collect and accumulate information on past fraud cases from its homepage and news, and store similar patterns. When Robot 100 determines the risk is high, it proactively notifies the elderly person, their family, or emergency contacts; if the risk is particularly high, it immediately reports it to the police. In addition, Robot 100 can constantly collect information on recent news and global events, thus understanding current fraud trends, predicting best precautions, and proactively engaging the user in conversation.

[0591] At a predetermined time point, the behavior determination unit 2236 uses at least one of the user 10's state, the user 10's emotion, the robot 100's emotion, and the robot 100's state, along with the behavior determination model 2221, to determine any one of a variety of robot behaviors, including inaction, as the behavior of the robot 100. Here, the example of using a dialogue-enabled article generation model as the behavior determination model 2221 will be explained.

[0592] Specifically, the behavior determination unit 2236 inputs text representing at least one of the user 10's state, the user 10's emotion, the robot 100's emotion, and the robot 100's state, as well as text inquiring about the robot's behavior, into the article generation model, and determines the robot 100's behavior based on the output of the article generation model.

[0593] For example, various robot behaviors include the following (1) to (11).

[0594] (1) The robot does nothing.

[0595] (2) Robots dream.

[0596] (3) The robot strikes up a conversation with the user.

[0597] (4) Robots create drawing diaries.

[0598] (5) The robot makes activity suggestions.

[0599] (6) The robot suggests people the user should meet.

[0600] (7) The robot introduces news that users are interested in.

[0601] (8) Robots edit photos and videos.

[0602] (9) The robot learns together with the user.

[0603] (10) The robot evokes memories.

[0604] (11) Provide users with advice on fraud risks.

[0605] When the behavior determination unit 2236 determines "(11) providing advice to the user regarding fraud risks," i.e., providing advice to the user regarding fraud risks, as robot behavior, the robot 100 acquires the conversation content and voiceprint between the user 10 and the conversation partner. Specifically, the speech understanding unit 2212 analyzes the voice of the user 10 detected by the microphone 2201 and the voice of the conversation partner, and acquires the conversation content and voiceprint between the user 10 and the conversation partner. Next, the robot 100 acquires the emotional value of the conversation partner. Specifically, it acquires the voice of the conversation partner from a telephone or walkie-talkie, and the image of the conversation partner displayed on the walkie-talkie screen, thereby acquiring the emotional value of the conversation partner. In addition, the robot 100 stores the conversation content between the user 10 and the conversation partner, as well as the image of the walkie-talkie, in historical data 2222. Next, the robot 100 determines the fraud risk based on the conversation content and the emotional value of the conversation partner. Specifically, the behavior determination unit 2236 determines the similarity, i.e., the security value, between the conversation content and the fraud case by comparing the data of past fraud cases stored in the storage unit 2220 with the conversation content. It should be noted that the behavior determination unit 2236 can determine the similarity between the conversation content and fraud cases by reading the conversation text into an article generation model such as ChatGPT. Then, the behavior determination unit 2236 determines the level of fraud risk, i.e., the safety value, based on the similarity between the conversation content and the fraud case, as well as the emotional value, voiceprint, and voice quality of the conversation subject. For example, when the similarity between the conversation content and the fraud case is high, the behavior determination unit 2236 determines a high safety value regardless of the emotional value, voiceprint, and voice quality of the conversation subject. Furthermore, even when the similarity between the conversation content and the fraud case is not so high, when the emotional value of the conversation subject's "anxiety" or "excitement" is high, or based on the voiceprint and voice quality, the behavior determination unit 2236 determines a high safety value. Next, the behavior determination unit 2236 determines the behavior based on the determined level of fraud risk. Specifically, when the determined safety value exceeds a set threshold, the behavior determination unit 2236 determines to take action that conveys a high level of fraud risk. For example, the behavior determination unit 2236 can determine to convey a high risk of fraud to user 10. Furthermore, the behavior determination unit 2236 can determine to convey a high risk of fraud to user 10's family or emergency contact. Additionally, the behavior determination unit 2236 can determine to immediately report a high risk of fraud to the police. These behaviors can be appropriately determined based on the degree of fraud risk. Furthermore, the behavior control unit 2250 controls the speaker of the controlled device, causing the aforementioned communication to be output as sound from the speaker. Moreover, regarding "(11) providing users with advice on fraud risks," the relevant information collection unit 2270 can proactively collect and accumulate information on past fraud cases from the homepage and news, and store it in the collected data 2223.Therefore, Robot100 can collect information on the latest news and world trends at all times, so it can grasp what kind of fraud is currently popular, predict how to avoid it, and take the initiative to talk to users.

[0606] [Eighth Implementation Method] Next, the processing of the behavior determination unit 2236 during the autonomous processing of the robot 100 performing autonomous actions will be explained.

[0607] In the autonomous processing of this embodiment, the robot 100, acting as an intelligent agent, proactively and periodically detects the user's status. The robot 100 constantly monitors the content of the user's conversations on the phone, with friends, or at work, and detects whether they fall under categories such as "bullying," "crime," and "harassment." That is, the robot 100 constantly monitors the content of the user's conversations on the phone, with friends, or at work, and detects the risk of the user approaching. The robot 100 uses article generation models such as ChatGPT to determine whether conversations have a high probability of being bullying or criminal. When a conversation is identified from the acquired conversation content as potentially indicating a related case, the robot 100 proactively contacts and sends emails to pre-registered notification recipients. Furthermore, the robot 100 records and conveys relevant conversation logs, anticipated cases, their probability of occurrence, and suggested solutions. By providing feedback on whether related events have occurred or their resolution, the robot 100 can improve the detection accuracy of related events and the suggestions for solutions.

[0608] At a predetermined time point, the behavior determination unit 2236 uses at least one of the user 10's state, the user 10's emotion, the robot 100's emotion, and the robot 100's state, along with the behavior determination model 2221, to determine any one of a variety of robot behaviors, including inaction, as the behavior of the robot 100. Here, the example of using a dialogue-enabled article generation model as the behavior determination model 2221 will be explained.

[0609] Specifically, the behavior determination unit 2236 inputs text representing at least one of the user 10's state, the user 10's emotion, the robot 100's emotion, and the robot 100's state, as well as text inquiring about the robot's behavior, into the article generation model, and determines the robot 100's behavior based on the output of the article generation model.

[0610] For example, various robot behaviors include the following (1) to (11).

[0611] (1) The robot does nothing.

[0612] (2) Robots dream.

[0613] (3) The robot strikes up a conversation with the user.

[0614] (4) Robots create drawing diaries.

[0615] (5) The robot makes activity suggestions.

[0616] (6) The robot suggests people the user should meet.

[0617] (7) The robot introduces news that users are interested in.

[0618] (8) Robots edit photos and videos.

[0619] (9) The robot learns together with the user.

[0620] (10) The robot evokes memories.

[0621] (11) Provide users with advice on risks such as “bullying”, “crime” and “harassment”.

[0622] When the behavior determination unit 2236 determines that "(11) providing users with advice on risks such as 'bullying,' 'crime,' and 'harassment,'" i.e., providing users with advice on risks such as "bullying," "crime," and "harassment," is a robot behavior, the robot 100 acquires the conversation content of multiple users 10. Specifically, the speech understanding unit 2212 analyzes the voices of multiple users 10 detected by the microphone 2201 and outputs text information representing the conversation content of multiple users 10. In addition, the robot 100 acquires the sentiment values ​​of multiple users 10. Specifically, it acquires the voices of multiple users 10 and the images of multiple users 10, and acquires the sentiment values ​​of multiple users 10. Furthermore, based on the conversation content of multiple users 10 and the sentiment values ​​of multiple users 10, the robot 100 determines whether a specific case of "bullying," "crime," or "harassment" has occurred. Specifically, the behavior determination unit 2236 determines the similarity between the conversation content and specific cases by comparing data of past cases such as "bullying," "crime," and "harassment" stored in the storage unit 2220 with the conversation content of multiple users 10. It should be noted that the behavior determination unit 2236 can read the conversation text into an article generation model such as ChatGPT to determine whether the conversation has a high probability of being a bullying or crime. Then, the behavior determination unit 2236 determines the likelihood of a specific event case based on the similarity between the conversation content and the specific case, as well as the sentiment values ​​of multiple users 10. For example, when the similarity between the conversation content and the specific case is high, and the sentiment values ​​of multiple users 10 for "anger," "sadness," "displeasure," "anxiety," "grief," "worry," and "emptiness" are high, the behavior determination unit 2236 determines the likelihood of the specific case occurring to be high. Furthermore, the robot 100 determines its behavior based on the likelihood of the specific case occurring. Specifically, if the likelihood of the specific case occurring exceeds a set threshold, the behavior determination unit 2236 determines to convey the behavior that indicates a high probability of the specific case occurring. For example, the behavior determination unit 2236 can determine that there is a high probability of a specific case occurring by sending an email to the administrators of the organizations to which multiple users 10 belong. Then, the robot 100 executes the determined behavior. For example, the robot 100 sends the aforementioned email to the administrators of the organizations to which users 10 belong. This email may record a session log corresponding to the specific case, the anticipated case, the probability of the case occurring, and suggested solutions for the case. Furthermore, the robot 100 stores the results of the executed behavior in the storage unit 2220. Specifically, the storage control unit 2238 stores whether a specific case has occurred and the resolution status in historical data 2222. In this way, by providing feedback on whether a specific case has occurred and the resolution status, the detection accuracy of the specific case and the suggested solutions can be improved.In addition, regarding “(11) providing users with advice on risks such as “bullying”, “crime” and “harassment”, the storage control unit 2238 periodically detects the content of multiple users’ conversations on the phone or at the company as the user’s status and stores it in historical data 2222.

[0623] [Ninth Implementation Method] [1. Notification device] Will use Figure 10A An example of the notification device 3010 involved in the implementation will be described. Figure 10A This is a block diagram illustrating an example configuration of the notification device 3010. The notification device 3010 notifies of risks, etc. The notification device 3010 is implemented by a humanoid robot with artificial intelligence capable of recognizing emotions, such as Pepper (registered trademark), and performs managerial duties in a restaurant. Staff are implemented by, for example, humans, and food delivery robots or beverage preparation robots such as Servi (registered trademark) equipped with facial recognition cameras and microphones. Figure 10A In the example shown, the notification device 3010 includes an acquisition unit 3011, a control unit 3012, a notification unit 3013, and a storage unit 3014.

[0624] (Acquisition Department 3011) The acquisition unit 3011 acquires at least one of the following: customer information related to the store's customers, store information related to the store, and order information related to orders placed at the store. Figure 10AIn the example shown, the acquisition unit 3011 also acquires risk information related to risk. The acquisition unit 3011 can also acquire at least one of response information related to response and satisfaction information related to satisfaction. The acquisition unit 3011 can acquire at least one of customer information, store information, and order information through the patrol of other robots. The acquisition unit 3011 can acquire at least one of risk information, response information, and satisfaction information through the patrol of other robots. For example, when at least one of customer information, store information, and order information collected from each seat through the delivery service of a food delivery robot such as Servi and its patrol within the store is input to the notification device 3010 of a manager located in the backyard, such as Pepper, the acquisition unit 3011 acquires at least one of customer information, store information, and order information. Therefore, the acquisition unit 3011 can, based on the time of delivery or removal of food by Servi, cause the analysis unit 3122 (described later) to perform a satisfaction evaluation, or cause the notification unit 3013 to notify the customer of a response consistent with the evaluation, thereby enabling the notification device 3010 to appropriately coordinate with other robots. The acquisition unit 3011 can acquire customer information related to at least one of a customer's facial expression, voice, and attributes. For example, the acquisition unit 3011 acquires customer information related to the facial expressions and conversations of customers obtained during a visit to the service. In addition to the customer's face, appearance, and voice, the acquisition unit 3011 can also acquire customer information related to the customer's attributes based on the customer's unique ID (including a store-issued membership card or membership number, etc.), facial authentication, biometric authentication, etc. The acquisition unit 3011 can acquire customer information related to the customer's emotions. For example, the acquisition unit 3011 identifies the customer's emotions based on the customer's facial expression in the customer's image captured by an expression recognition camera. The method for emotion recognition is the same as that described in Patent Documents 5 to 9, so its detailed description is omitted. The method for emotion recognition is not limited to these methods, and other known methods can also be used. The acquisition unit 3011 can acquire store information related to at least one of the store's crowding level and table conditions.

[0625] (Control Unit 3012) The control unit 3012 controls the entire notification device 3010. As an example, the control unit 3012 is implemented by running various programs stored in the memory device inside the notification device 3010, using RAM as its working area, through a central processing unit (CPU) or microprocessor unit (MPU). As another example, the control unit 3012 is implemented using integrated circuits such as application-specific integrated circuits (ASICs) or field-programmable gate arrays (FPGAs). Figure 10A In the example shown, the control unit 3012 includes a learning-completed model generation unit 3121 and a parsing unit 3122.

[0626] (Complete learning of Model Generation Section 3121) The learning completion model generation unit 3121 uses at least one of customer information, store information, and order information acquired by the acquisition unit 3011, along with risk information, to generate a learning completion model 3141. This learning completion model 3141 outputs risk information when at least one of the customer information, store information, and order information is input. For example, the learning completion model generation unit 3121 uses customer information and risk information acquired by the acquisition unit 3011 as supervised data to generate an article generation model with dialogue functionality. This article generation model includes the learning completion model 3141, which outputs risk information when customer information is input. The learning completion model generation unit 3121 can also use at least one of satisfaction information and response information acquired by the acquisition unit 3011 to generate the learning completion model 3141. This learning completion model 3141 outputs risk information, and at least one of satisfaction information and response information, when at least one of the customer information, store information, and order information is input. The article generation model with dialogue functionality is implemented, for example, by a Chat Generative Pre-trained Transformer (ChatGPT).

[0627] (Analysis Section 3122) The analysis unit 3122 analyzes risks based on at least one of the customer information, store information, and order information obtained by the acquisition unit 3011. The analysis unit 3122 can also analyze at least one of the responses taken to customers and customer satisfaction based on at least one of the customer information, store information, and order information obtained by the acquisition unit 3011. For example, the analysis unit 3122 also analyzes responses related to at least one of greeting, ordering, food delivery, and checkout. The analysis unit 3122 can analyze risks based on customer information obtained by the acquisition unit 3011 related to at least one of the customer's facial expressions, voice, and attributes. For example, the analysis unit 3122 infers the risk posed by troublesome customers based on the customer's reaction to a greeting such as Pepper's facial expressions and voice. Furthermore, the analysis unit 3122 analyzes the risk posed by troublesome customers based on a list of past troublesome customers accumulated from facial recognition data and formatted into a list of facial recognition data. The analysis unit 3122 can analyze risks based on customer information related to the customer's emotions obtained by the acquisition unit 3011. For example, the analysis unit 3122, based on customer emotion recognition via Pepper or similar methods, infers the risk posed by troublesome customers. Therefore, the analysis unit 3122 can determine whether to provide more appropriate service. The analysis unit 3122 can also analyze at least one of response and satisfaction based on customer information acquired by the acquisition unit 3011 related to at least one of customer facial expressions, voice, and attributes. The analysis unit 3122 can analyze risk based on store information related to at least one of store crowding and table conditions. Therefore, the analysis unit 3122 can determine whether to provide more appropriate service. The analysis unit 3122 can also analyze at least one of response and satisfaction based on store information related to at least one of store crowding and table conditions.

[0628] The analysis unit 3122 can analyze satisfaction during at least one of food delivery and food clearing. The analysis unit 3122 can quantitatively analyze satisfaction. For example, based on customer facial expressions captured by an expression recognition camera, the analysis unit 3122 quantitatively evaluates satisfaction by identifying emotions such as customer satisfaction quantified with positive emotions as the evaluation axis. The method for emotion recognition is the same as that described in Patent Documents 5 to 9, therefore its detailed description is omitted. The analysis unit 3122 can analyze at least one of risk, response, and satisfaction for each seat in the store and at least one time period. The analysis unit 3122 can use a learning completion model 3141 to analyze risk, which outputs risk-related risk information when inputting at least one of customer information, store information, and order information obtained by the acquisition unit 3011. Thus, the analysis unit 3122 can analyze and provide more appropriate service. The analysis unit 3122 can use the learning completion model 3141 to analyze at least one of risk, coping and satisfaction. When the learning completion model 3141 is input with at least one of customer information, store information and order information obtained by the acquisition unit 3011, it also outputs at least one of coping information and satisfaction information.

[0629] The following is an example of the analysis performed by the analysis unit 3122. First, an example of the analysis of the main task by the analysis unit 3122 will be explained. As an example, when a customer at a certain table has a gloomy expression for more than the scheduled time, the analysis unit 3122 analyzes that the customer at that table may be dissatisfied with the food or service, and the appropriate response is to strike up a conversation. As another example, when a customer at a certain table is having a conversation hinting at ordering, or when a scheduled time has elapsed since entering the store (for example, two minutes), the analysis unit 3122 analyzes that the customer at that table is preparing to order, and the appropriate response is to order. As yet another example, when a customer at a certain table is having a conversation hinting at leaving the store, the analysis unit 3122 analyzes that the customer at that table is preparing to leave, and the appropriate response is to handle the departure procedures, such as paying the bill.

[0630] Next, an example of the sub-task being analyzed by the analysis unit 3122 will be explained. In the sub-task, when the order information indicates a specific order, the analysis unit 3122 analyzes and determines that the response should be at least one of providing a specific greeting at the customer's table and serving a specific item. Thus, the analysis unit 3122 can provide service in response to customer needs. As an example, when receiving an order from a customer at a table more than a predetermined distance away, the analysis unit 3122 analyzes and determines that the appropriate response should be to provide a personalized conversation, order taking, and sommelier service at the customer's table that matches their preferences. As another example, when receiving an order from a customer to add two more beers, the analysis unit 3122 analyzes and determines that the appropriate response should be to say something like, "You drink quickly, would you like some water as well?" at the customer's table as a reminder. As yet another example, when receiving an order from a customer for a recommended wine, the analysis unit 3122 analyzes and determines that the appropriate response should be to make a recommendation at the customer's table that corresponds to the customer's preferences and recently ordered items. As another example, when a beverage order is received, the analysis unit 3122 determines that the appropriate response is to have the beverage-making robot prepare the beverage and then load it onto a delivery robot to deliver it to the customer's table. Thus, when the notification device 3010 interacts with other robots, the analysis unit 3122 can also analyze unavailable items as the appropriate response. As another example, when a specific item is ordered, the analysis unit 3122 determines that the appropriate response is to check the customer's feedback on the specific item during the second visit of the delivery robot or other staff to the customer's table, in order to obtain the customer's satisfaction with that item.

[0631] (Notification Department 3013) The notification unit 3013 notifies the risks analyzed by the analysis unit 3122. The notification unit 3013 may also notify at least one of the responses and satisfaction levels analyzed by the analysis unit 3122.

[0632] For example, the notification unit 3013 informs human or robot staff of the risks posed by troublesome customers, as analyzed by the analysis unit 3122, and provides necessary service or response instructions. Thus, the notification unit 3013 can avoid the risks posed by troublesome customers while enabling staff to perform appropriate table management and provide suitable service. As a result, the notification unit 3013 allows staff to create an environment where customers "want to come back." Furthermore, the notification unit 3013 can notify delivery robots such as Servi, and when the analysis unit 3122 evaluates customer satisfaction based on Servi's delivery or clearing times, it can notify Servi of customer responses consistent with the evaluation, thereby enabling the notification device 3010 to appropriately coordinate with other robots. The notification method of the notification unit 3013 is not particularly limited. For example, if the notification unit 3013 is a voice output unit, it can be a notification via voice output such as a conversation; if the notification unit 3013 is a display unit such as a tablet computer, it can be a notification via displaying text or symbols; and if the notification recipient is a robot, it can be a notification via communication.

[0633] The notification unit 3013 can notify the store staff of at least one of the risks, responses, and satisfaction levels analyzed by the analysis unit 3122 for each seat and each time period in the store. For example, the notification unit 3013 can notify the store staff of the satisfaction levels analyzed by the analysis unit 3122 for each seat and each time period in the store.

[0634] The notification unit 3013 can notify customers of the satisfaction level quantitatively analyzed by the analysis unit 3122. The notification unit 3013 can also notify customers of responses related to at least one of greeting, ordering, food delivery, and payment, as analyzed by the analysis unit 3122. As described above, the notification unit 3013 can provide more appropriate service by providing more detailed information. The notification unit 3013 is implemented, for example, by a sound output unit or a display unit such as a tablet. The notification unit 3013 can notify customers of responses related to at least one of specific greetings at the customer's table and the delivery of specific items, as analyzed by the analysis unit 3122. Hereinafter, [the following will use...] Figure 10B Here is an example of a notification issued by the Notification Department 3013. Figure 10B This is a diagram used to illustrate an example of the notification device 3010. Figure 10B In the example shown, the notification device 3010 is a robot such as Pepper, and the waiter 20 is a food delivery robot such as Servi. Furthermore, in Figure 10BIn the example shown, when customer 30 orders two more beers, the analysis unit 3122 determines that the appropriate response is for staff 20 to say something at customer 30's table, such as, "You drink quite quickly, would you like some water as well?" In this case, the notification unit 3013 instructs staff 20 to say something at customer 30's table, such as, "You drink quite quickly, would you like some water as well?"

[0635] (Storage Department 3014) Back Figure 10A Description. Storage unit 3014 stores various data and programs, including customer information, store information, order information, satisfaction information, response information, risk information, and the learning completion model 3141.

[0636] The storage unit 3014 is implemented by semiconductor storage elements such as random access memory (RAM) and flash memory, or storage devices such as hard disks and optical disks.

[0637] [2. Notification Methods] Next, we will use Figure 10A This describes an example of the processing performed by the notification device 3010 according to the embodiment. Figure 10C This is a flowchart illustrating an example of the processing flow of the notification device 3010.

[0638] In step S3001, the acquisition unit 3011 acquires at least one of customer information related to customers of the store, store information related to the store, and order information related to orders placed at the store. For example, when at least one of customer information, store information, and order information collected from each seat through food delivery services such as Servi and in-store patrols is input to a notification device 3010 of a back-office manager such as Pepper, the acquisition unit 3011 acquires at least one of customer information, store information, and order information.

[0639] In step S3002, the analysis unit 3122 analyzes the risk based on at least one of the customer information, store information, and order information obtained by the acquisition unit 3011.

[0640] In step S3003, the notification unit 3013 notifies the risk analyzed by the analysis unit 3122. For example, the notification unit 3013 informs human or robot staff of the risks posed by troublesome customers as analyzed by the analysis unit 3122, and provides instructions on necessary reception or response.

[0641] [3. Hardware Configuration] Figure 4This is a schematic diagram illustrating an example of the hardware configuration of the notification device 3010.

[0642] [Tenth Implementation Method] [1. Determination device] Will use Figure 11A An example of the determination device 4010 involved in the implementation will be described. Figure 11A This is a block diagram illustrating an example configuration of the determination device 4010. The determination device 4010 determines the risk of specific fraud against a customer and engages in a conversation with the customer regarding the risk. The determination device 4010 is implemented using a humanoid robot with emotion-recognizing artificial intelligence, such as Pepper (registered trademark), and a microphone, and is installed around ATMs or prepaid card sales locations in banks, convenience stores, or other shops. Figure 11A In the example shown, the determination device 4010 includes an acquisition unit 4011, a control unit 4012, a conversation unit 4013, and a storage unit 4014.

[0643] (Acquisition Department 4011) The acquisition unit 4011 acquires at least one of the following: image data of customers captured in a store, attribute data related to customer attributes, and conversation data related to customer conversations. For example, the acquisition unit 4011 acquires this data from the vicinity of ATMs or prepaid card sales locations in banks, convenience stores, or other stores. Specifically, the acquisition unit 4011 acquires image data of customers captured by cameras installed around ATMs in the store as image data, and acquires customer conversation data using microphones installed around ATMs in the store. The acquisition unit 4011 can acquire attribute data related to at least one of the customer's age and gender. Figure 11A In the example shown, the acquisition unit 4011 also acquires risk data related to risk, emotion data related to the customer's emotions, and reaction data related to the customer's reaction to the risk-related conversation conducted by the conversation unit 4013. For example, the acquisition unit 4011 acquires emotion data identified based on at least one of the shooting data and the conversation data. Furthermore, when the acquisition unit 4011 receives feedback from the customer regarding the correctness of the risk-related conversation conducted by the conversation unit 4013, it acquires reaction data related to the customer's emotions. The acquisition unit 4011 can acquire emotion data and reaction data by recognizing the customer's emotions, or it can acquire emotion data and reaction data with pre-identified emotions. The emotion recognition method is the same as that described in Patent Documents 5 to 9, therefore its detailed description is omitted. The emotion recognition method is not limited to these methods, and other known methods can also be used.

[0644] (Control Unit 4012) The control unit 4012 controls the entire determination device 4010. As an example, the control unit 4012 is implemented by running various programs stored in the memory device inside the determination device 4010, using RAM as its working area, through a central processing unit (CPU) or microprocessor unit (MPU). As another example, the control unit 4012 is implemented using an integrated circuit such as an application-specific integrated circuit (ASIC) or a field-programmable gate array (FPGA). Figure 11A In the example shown, the control unit 4012 includes a learning-completed model generation unit 4121 and a decision unit 4122.

[0645] (Complete learning of Model Generation Section 4121) The learning completion model generation unit 4121 uses at least one of the shooting data, attribute data, and conversation data acquired by the acquisition unit 4011, along with risk data, to generate a learning completion model 4141. This learning completion model 4141 outputs risk data when at least one of the shooting data, attribute data, and conversation data is input. For example, the learning completion model generation unit 4121 uses the shooting data, attribute data, conversation data, and risk data acquired by the acquisition unit 4011 as supervision data to generate an article generation model with dialogue functionality. This article generation model includes a learning completion model 4141 that outputs risk data when shooting data, attribute data, and conversation data are input.

[0646] Furthermore, the learning completion model generation unit 4121 uses the sentiment data acquired by the acquisition unit 4011 and the conversation data related to the conversation about risk to generate a learning completion model 4142. This learning completion model 4142 outputs conversation data about risk when sentiment data is input. For example, the learning completion model generation unit 4121 uses the sentiment data and conversation data about risk acquired by the acquisition unit 4011 as supervision data to generate an article generation model with dialogue functionality. This article generation model includes the learning completion model 4142, which outputs conversation data about risk when sentiment data is input. The learning completion model generation unit 4121 can use the conversation data about risk and reaction data related to customer reactions to conversations about risk to enable the learning completion model 4142 to relearn. For example, the learning-completed model generation unit 4121 uses conversational data about risk, as well as customer emotion-related reaction data acquired by the acquisition unit 4011 when receiving feedback from customers regarding the correctness of the conversation about risk, as supervision data to enable the learning-completed model 4142 to relearn. A dialogue-enabled article generation model, for example, is implemented using a ChatGenerative Pre-trained Transformer (ChatGPT).

[0647] (Judgment Department 4122) The determination unit 4122 determines the risk of specific fraud against a customer based on at least one of the shooting data, attribute data, and conversation data acquired by the acquisition unit 4011. The determination unit 4122 can use a learned model 4141 to determine the risk, which outputs risk-related risk data when at least one of the shooting data, attribute data, and conversation data acquired by the acquisition unit 4011 is input. The determination unit 4122 can determine the risk based on at least one of the customer's facial expression and emotion. For example, in the determination unit 4122, the determination unit 4122 determines the risk based on at least one of the customer's facial expression identified based on the shooting data acquired by the acquisition unit 4011, and the customer's emotion identified based on at least one of the shooting data and the conversation data. When determining the risk based on the customer's facial expression, the determination unit 4122 can determine the risk by recognizing the customer's facial expression from the shooting data, or it can determine the risk based on the customer's facial expression pre-recognized by an facial expression recognition camera or the like. When determining risk based on customer emotions, the determination unit 4122 can determine risk by identifying customer emotions from at least one of the shooting data and the conversation data, or it can determine risk based on pre-identified customer emotions.

[0648] When the customer's emotion is negative, the determination unit 4122 can determine that the risk is higher than when the customer's emotion is positive. When the customer's emotion is at least one of anxiety, worry, and unease, the determination unit 4122 can determine that the risk is higher than when the customer's emotion is neither anxiety, worry, nor unease. For example, the determination unit 4122 quantifies the magnitude of the risk from 0 to 10. When the customer's emotion is at least one of anxiety, worry, or unease, the risk value is determined to be 2 to 4, and when there is no worry or unease, the risk value is determined to be 0 to 1. Similarly, below, when the determination unit 4122 determines that the risk is high, the risk value is determined to be 2 to 4. In addition, when there are multiple factors that are determined to be high-risk, the determination unit 4122 determines the sum of the risk values ​​by adding the risk values ​​determined based on each of the multiple factors.

[0649] The determination unit 4122 can determine risk based on at least one of the customer's actions and postures in the image data acquired by the acquisition unit 4011. For example, when the customer's actions identified from the image data are talking while operating the ATM, the determination unit 4122 determines the risk to be higher than when the customer operates the ATM silently. A method for determining risk based on at least one of the customer's actions and postures in the image data acquired by the determination unit 4122 is disclosed in the reference (Internet search <URL: https: / / www.itmedia.co.jp / news / articles / 2212 / 07 / news077.html>), therefore its detailed description is omitted. The determination unit 4122 can determine risk based on at least one of the customer's age and gender as indicated by the attribute data acquired by the acquisition unit 4011. Therefore, the determination unit 4122 can detect risk with higher accuracy. For example, when the attribute data acquired by the acquisition unit 4011 indicates that the customer's attribute is elderly, the determination unit 4122 determines the risk to be higher than when the customer's attribute is not elderly. Therefore, the determination unit 4122 can further detect risks with higher precision. The determination unit 4122 can determine risks based on keywords contained in the session data. For example, when the session data contains keywords related to specific scams, such as "transfer," "encountering an accident," or "emergency," the determination unit 4122 determines the risk to be higher than when these keywords are not present.

[0650] The determination unit 4122 can determine the content of the risk-related conversation based on the customer's emotions and instruct the conversation unit 4013 to execute the conversation about that content. For example, when the customer's emotions are anxious, the determination unit 4122 will determine the content of the risk-related conversation as something that will temporarily calm the customer down. Thus, the determination unit 4122 can instruct the conversation unit 4013 to execute a conversation that aligns with the customer's emotions. The determination unit 4122 can also use a learning completion model 4142 to determine the content of the risk-related conversation and instruct the conversation unit 4013 to execute the conversation about that content. This learning completion model 4142 outputs risk-related conversation data related to the risk-related conversation when it receives emotion-related data as input. Therefore, the determination unit 4122 can refine the content of the initially set talk-script and other pre-prepared risk-related conversations through learning from the learning completion model 4142.

[0651] The determination unit 4122 can use a learning completion model 4142, which is relearned using conversation data about risk and reaction data related to customer reactions to the conversation about risk, to determine the content of the risk-related conversation and instruct the conversation unit 4013 to execute the risk-related conversation about that content. For example, the determination unit 4122 can use a learning completion model 4142, which is relearned using conversation data about risk and reaction data related to customer emotions obtained by the acquisition unit 4011 when receiving feedback from the customer regarding whether the risk-related conversation is correct, to determine the content of the risk-related conversation and instruct the conversation unit 4013 to execute the risk-related conversation about that content. Thus, the determination unit 4122 can improve the initially set scripts and the pre-prepared content of the risk-related conversation corresponding to the customer's reactions by learning the learning completion model 4142. The determination unit 4122 can use a text generation model with dialogue function, including the learning completion model 414, to determine the content of the risk-related conversation and instruct the conversation unit 4013 to execute the risk-related conversation about that content.

[0652] Therefore, the decision unit 4122 can enable the session unit 4013 to perform a more complete session.

[0653] (Conversation Section 4013) The conversation unit 4013 engages in a conversation with the customer regarding the risks determined by the judgment unit 4122.

[0654] For example, the conversation department 4013 engages in a conversation with a customer about risk, which includes at least one of risk explanation, persuasion, and inquiry. The following will use... Figure 11B An example of a conversation conducted by the conversation unit 4013 will be explained. Figure 11B This is a diagram used to illustrate an example of the determination device 4010. Figure 11BIn the example shown, the determination device 4010 is a robot such as Pepper, placed around the ATM 20 in a bank, convenience store, or other shop. The determination device 4010 can be implemented by the ATM 20 or other devices. When the session data contains keywords such as "transfer," "encountering an accident," or "emergency," such as... Figure 11B As shown, the conversation unit 4013 engages in a risk-related conversation with the customer 30. This conversation includes inquiries to confirm information such as "Why did you transfer money?", "Who told you to do this?", and "How much money do you need?", as well as explanations regarding suspected fraud, such as "There is suspicion of special fraud." Therefore, unlike existing technologies that only notify the customer 30 when the risk is deemed high without confirmation, the conversation unit 4013 can assess the situation and investigate risk factors, thus enabling high-precision risk detection. Consequently, the conversation unit 4013 can increase the likelihood of preventing and reducing the harm caused by special fraud.

[0655] Back Figure 11A The conversation unit 4013 can execute responses based on the risks determined by the judgment unit 4122. For example, whenever the risk value quantified by the judgment unit 4122 exceeds a threshold, the conversation unit 4013 executes different responses. As an example, when the risk value quantified by the judgment unit 4122 is 0 to 1, the conversation unit 4013 does not execute any risk-specific measures in the conversation about the risk. When the risk value quantified by the judgment unit 4122 is 2 to 4, the conversation unit 4013 initiates a conversation with the customer and listens to their concerns.

[0656] In this situation, the judgment unit 4122 can further determine the risk. When the risk value quantified by the judgment unit 4122 is 5 to 8, the communication unit 4013 notifies the store clerk. When the risk value quantified by the judgment unit 4122 is 9 to 10, the communication unit 4013 reports to the police. There are no particular restrictions on the methods of communication, notification, and reporting performed by the communication unit 4013. For example, if the communication unit 4013 is a sound output unit, the communication, notification, and reporting can be performed by outputting sound; if the communication unit 4013 is a display unit such as a tablet computer, the communication, notification, and reporting c...

Claims

1. A behavior control system, the behavior control system comprising: An emotion determination unit determines the user's emotion or the robot's emotion; as well as A behavior determination unit, based on a dialogue function that enables a user to converse with the robot, generates robot behavior content based on the user's behavior and the user's or robot's emotions, and determines the robot's behavior corresponding to the behavior content; wherein, The behavior determination unit detects the user's behavior to determine whether the user's behavior is dangerous. If the user's behavior is dangerous, it generates first behavior content to correct the user's behavior.

2. The behavior control system according to claim 1, wherein, The first action includes at least one of performing a gesture to correct the user's behavior and playing a sound to correct the user's behavior.

3. The behavior control system according to claim 2, wherein, After the robot performs the gesture or plays the sound, the behavior determination unit detects the user's behavior to determine whether the user's behavior has been corrected. If the user's behavior has been corrected, a second behavior content different from the first behavior content is generated.

4. The behavior control system according to claim 3, wherein, The second action content includes at least one of the sounds praising the user's action and the sounds thanking the user for their action.

5. The behavior control system according to claim 2, wherein, After the robot performs the gesture or plays the sound, the behavior determination unit detects the user's behavior to determine whether the user's behavior has been corrected. If the user's behavior has not been corrected, a third behavior content different from the first behavior content is generated.

6. The behavior control system according to claim 5, wherein, The third action includes at least one of sending specific information to a person other than the user, performing a gesture that arouses the user's interest, playing a sound that arouses the user's interest, and playing an image that arouses the user's interest.

7. The behavior control system according to claim 1, wherein, The robot is mounted on a plush toy, or connected wirelessly or via a wired connection to a control device mounted on the plush toy.

8. A behavior control system, the behavior control system comprising: An emotion determination unit determines the user's emotion or the robot's emotion; as well as A behavior determination unit, based on an article generation model that enables dialogue between the user and the robot, generates robot behavior content based on the user's behavior and the user's or robot's emotions, and determines the robot's behavior corresponding to the behavior content; wherein, The behavior determination unit receives messages from multiple users who are having an ongoing conversation, and when a message becomes a pre-set state, it summarizes the content of the message and determines it as the robot's behavior.

9. A behavior control system, the behavior control system comprising: User status recognition unit, which identifies user status including user behavior; An emotion determination unit determines the user's emotion or the robot's emotion; as well as A behavior determination unit, based on an article generation model that enables dialogue between the user and the robot, determines the robot's behavior corresponding to the user's state and the conversation content of multiple users; wherein... The behavior determination unit assesses specific fraud risks based on the conversation content of multiple users and the users' emotions.

10. A behavior control system, the behavior control system comprising: User status identification unit, which identifies user status including the behavior of multiple users; An emotion determination unit determines the emotions of multiple users or the emotions of a robot. as well as A behavior determination unit, based on an article generation model that enables dialogue between the user and the robot, determines the robot's behavior corresponding to the user's state and the conversation content of multiple users; wherein... The behavior determination unit detects specific cases based on the conversation content of multiple users and the users' emotions.

11. A behavior control system, the behavior control system comprising: User status recognition unit, which identifies user status including user behavior; An emotion determination unit determines the user's emotion or the robot's emotion; as well as A behavior determination unit, based on an article generation model with dialogue functionality enabling conversation between the user and the robot, determines the robot's behavior corresponding to the user's state and the user's or robot's emotions; wherein, The behavior determination unit performs caregiving and monitoring of the user based on at least one of the user's state and the user's emotions.

12. A behavior control system, the behavior control system comprising: A status recognition unit that recognizes user status, including user behavior, and the status of electronic devices; An emotion determination unit determines the emotion of the user or the emotion of the electronic device; The behavior determination unit, at a predetermined time point, uses at least one of the user state, the state of the electronic device, the user's emotion, and the emotion of the electronic device, and a behavior determination model, to determine any one of a variety of device operations, including no operation, as the behavior of the electronic device. as well as The storage control unit stores event data, including the emotion value determined by the emotion determination unit and data containing the user's behavior, into historical data; wherein, The operation of the device includes providing care-related advice to the user; When the behavior determination unit determines that providing care-related suggestions to the user is an action of the electronic device, it collects care-related information about the user and provides care-related suggestions to the user based on the collected information.

13. A behavior control system, the behavior control system comprising: A status recognition unit that recognizes user status, including user behavior, and the status of electronic devices; An emotion determination unit determines the emotion of the user or the emotion of the electronic device; The behavior determination unit, at a predetermined time point, uses at least one of the user state, the state of the electronic device, the user's emotion, and the emotion of the electronic device, and a behavior determination model, to determine any one of a variety of device operations, including no operation, as the behavior of the electronic device. as well as The storage control unit stores event data, including the emotion value determined by the emotion determination unit and data containing the user's behavior, into historical data; wherein, The device operation includes notifying the provider of information based on the user's feelings toward the matters provided by the provider; When the behavior determination unit notifies the provider of information based on the user's feelings toward the matters provided by the provider, and determines that the behavior is the behavior of the electronic device, the unit notifies the provider of information based on the user's feelings toward the matters provided by the provider.

14. A behavior control system, the behavior control system comprising: A status recognition unit that recognizes user status, including user behavior, and the status of electronic devices; An emotion determination unit determines the emotion of the user or the emotion of the electronic device; as well as A behavior determination unit, at a predetermined time point, uses at least one of the user's state, the electronic device's state, the user's emotion, and the electronic device's emotion, along with a behavior determination model, to determine any one of a variety of device operations, including inaction, as the behavior of the electronic device; wherein... The operation of the device includes providing users with advice regarding fraud risks; When the behavior determination unit determines that providing advice on fraud risk to the user is a behavior of the electronic device, it provides advice on fraud risk to the user.

15. A behavior control system, the behavior control system comprising: A status recognition unit that recognizes user status, including user behavior, and the status of electronic devices; An emotion determination unit determines the emotion of the user or the emotion of the electronic device; as well as A behavior determination unit, at a predetermined time point, uses at least one of the user's state, the electronic device's state, the user's emotion, and the electronic device's emotion, along with a behavior determination model, to determine any one of a variety of device operations, including inaction, as the behavior of the electronic device; wherein... The operation of the equipment includes providing users with advice regarding the risk of approach; When the behavior determination unit determines that providing advice to the user regarding the approach risk is a behavior of the electronic device, it provides advice to the user regarding the approach risk.

Images