Agent system and program

The agent system addresses the challenge of understanding user emotions and preferences by using a reception unit, history acquisition, and product determination to provide personalized product recommendations, improving user satisfaction.

JP2024159680APending Publication Date: 2024-11-08SOFTBANK GROUP CORP

Patent Information

Application Number
JP2024072046
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-08-24
Filing Date
2024-04-26
Publication Date
2024-11-08

AI Technical Summary

Technical Problem

Existing technologies fail to effectively understand and respond to user emotions and preferences, limiting the ability of robots to provide personalized interactions and product recommendations.

Method used

An agent system that includes a reception unit for budget acceptance, a history acquisition unit for past purchases, a product determination unit based on user emotions and preferences, and an operation execution unit to obtain products, utilizing a user state recognition unit and emotion determination unit to tailor interactions and purchases.

Benefits of technology

The system enables robots to understand user emotions and preferences, allowing for personalized product recommendations and interactions, enhancing user satisfaction and convenience.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024159680000001_ABST
    Figure 2024159680000001_ABST
Patent Text Reader

Abstract

To provide an agent system and a program that understand a feeling of a user and oneself to acquire a commodity.SOLUTION: An agent system 50 comprises: a reception unit 502 that receives a budget of settlement means which can be used for acquirement of a commodity; a history acquisition unit 504 that acquires history information being information associated with acquisition history in which a commodity acquired by a user in the past is recorded; a commodity determination unit 506 that determines an acquired commodity on the basis of the budget received by the reception unit 502 and the history information acquired from the history acquisition unit 504; and a RPA 57 that executes an action to acquire the commodity determined by the commodity determination unit at predetermined timing.SELECTED DRAWING: Figure 13
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to an agent system and a program. [Background technology]

[0002] Patent Document 1 discloses a technology for determining an appropriate robot behavior for a user's state. The conventional technology in Patent Document 1 recognizes the user's reaction when the robot executes a specific action, and if the robot is unable to determine an action for the recognized user's reaction, it updates the robot's behavior by receiving information about an action suitable for the recognized user's state from a server. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 6053847 Summary of the Invention [Problem to be solved by the invention]

[0004] An object of the present invention is to provide an agent system and program that understands the emotions of a user and the user's own emotions and acquires a product. [Means for solving the problem]

[0005] The agent system according to the first aspect includes a reception unit that receives a budget for a payment method that can be used to acquire a product, a history acquisition unit that acquires history information that is information relating to a acquisition history of products that a user has acquired in the past, a product determination unit that determines a product to acquire based on the budget received by the reception unit and the history information obtained from the history acquisition unit, and an operation execution unit that executes an operation to acquire the product determined by the product determination unit at a predetermined timing.

[0006] The agent system of the second aspect is the agent system of the first aspect, further comprising a user state recognition unit capable of acquiring information relating to the user's state, and an emotion determination unit that determines the user's emotion based on the information about the user's state acquired by the user state recognition unit, and the history information includes a acquisition history that records products acquired by the operation execution unit in association with the user's emotion toward the products determined by the emotion determination unit.

[0007] The agent system of the third aspect is the agent system of the second aspect, wherein the emotion determination unit further determines the emotion of the agent based on the state of the user acquired via the user state recognition unit, and the product determination unit determines the product to be acquired based on the emotion of the agent determined by the emotion determination unit.

[0008] The agent system of the fourth aspect is the agent system of the second aspect, wherein the user state recognition unit further acquires information regarding the surrounding situation of the user, and the product determination unit determines the product to be acquired based on the surrounding situation acquired by the user state recognition unit.

[0009] The agent system of the fifth aspect is the agent system of the fourth aspect, wherein the information regarding the surrounding situation is information regarding the inventory of each product inferred from image information representing the situation surrounding the user, and the product determination unit determines the product to obtain based on the information regarding the inventory.

[0010] The agent system of the sixth aspect is the agent system of the fifth aspect, wherein the action execution unit executes an action to acquire the product when the inventory falls below a predetermined threshold for a predetermined consecutive period of time.

[0011] The agent system of the seventh aspect is the agent system of the second aspect, wherein the reception unit further receives at least one setting information among information regarding a product category and information regarding a user's preferences, and the product determination unit determines the product to be acquired based on the setting information received by the reception unit.

[0012] A program according to an eighth aspect causes a computer to operate as the agent system according to any one of the first to seventh aspects. [Brief description of the drawings]

[0013] [Figure 1] 1 illustrates a schematic diagram of an example of a system 5 according to a first embodiment. [Diagram 2] 1 illustrates a schematic functional configuration of a robot 100 according to a first embodiment. [Diagram 3] 2 shows an example of an operation flow of the robot 100 according to the first embodiment. [Figure 4] 1 illustrates an example of a hardware configuration of a computer 1200 according to the first embodiment. [Diagram 5] 1 shows an emotion map 300 onto which multiple emotions are mapped according to a first embodiment. [Figure 6] 9 shows an emotion map 900 onto which multiple emotions are mapped according to the first embodiment. [Figure 7] 1 illustrates an outline of a functional configuration of an agent system according to a first embodiment. [Figure 8] 4 shows an example of the operation of the agent system according to the first embodiment. [Figure 9] 4 shows an example of the operation of the agent system according to the first embodiment. [Figure 10] 2 illustrates an example of a hardware configuration of smart glasses according to a first embodiment. [Figure 11] 2 shows an example of a usage mode of an agent system using smart glasses according to the first embodiment. [Figure 12]1 illustrates an example of a hardware configuration of a smart pendant according to a first embodiment. [Figure 13] FIG. 11 is a diagram illustrating an example of a functional configuration of an agent system according to a second embodiment. [Figure 14] 13 is a flowchart showing an example of the flow of a product determination process in which the agent system according to the second embodiment determines an item to be purchased. [Figure 15] 13 is a flowchart showing an example of the flow of a product acquisition process in which an agent system according to a second embodiment purchases an item. [Figure 16] 13 is a flowchart showing an example of the flow of an acquisition determination process in which the agent system according to the first modified example of the second embodiment determines whether or not to purchase an item. [Figure 17] 13 is a flowchart showing an example of the flow of a product determination process in which an agent system according to a second modification of the second embodiment determines a service to be contracted. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] The present invention will be described below through embodiments of the invention, but the following embodiments do not limit the scope of the invention according to the claims. Furthermore, not all of the combinations of features described in the embodiments are necessarily essential to the solution of the invention.

[0015] (First embodiment) FIG. 1 is a schematic diagram of an example of a system 5 according to the present embodiment. The system 5 includes a robot 100, a robot 101, a robot 102, and a server 300. A user 10a, a user 10b, a user 10c, and a user 10d are users of the robot 100. A user 11a, a user 11b, and a user 11c are users of the robot 101. A user 12a and a user 12b are users of the robot 102. In the description of the present embodiment, the user 10a, the user 10b, the user 10c, and the user 10d may be collectively referred to as the user 10. The user 11a, the user 11b, and the user 11c may be collectively referred to as the user 11. The user 12a and the user 12b may be collectively referred to as the user 12. The robot 101 and the robot 102 have substantially the same functions as the robot 100. Therefore, the system 5 will be described by mainly focusing on the functions of the robot 100.

[0016] The robot 100 converses with the user 10 and provides images to the user 10. At this time, the robot 100 cooperates with a server 300 or the like with which it can communicate via a communication network 20 to converse with the user 10 and provide images, etc. to the user 10. For example, the robot 100 not only learns appropriate conversation by itself, but also cooperates with the server 300 to learn how to have a more appropriate conversation with the user 10. The robot 100 also causes the server 300 to record captured image data, etc. of the user 10, and requests the server 300 for the image data, etc. as necessary, and provides the data, etc. to the user 10.

[0017] The robot 100 also has an emotion value that indicates the type of its own emotion. For example, the robot 100 has emotion values ​​that indicate the intensity of each of the emotions of "joy", "anger", "sorrow", "pleasure", "discomfort", "relief", "anxiety", "sadness", "excitement", "concern", "relief", "fulfillment", "emptiness", and "neutral". When the robot 100 converses with the user 10 in a state where the emotion value of excitement is high, for example, the robot 100 utters sounds at a fast speed. In this way, the robot 100 can express its own emotion through actions.

[0018] Furthermore, the robot 100 may be configured to match the sentence generation model with the emotion engine, thereby determining the behavior of the robot 100 corresponding to the emotion of the user 10. Specifically, the robot 100 may be configured to recognize the behavior of the user 10, determine the emotion of the user 10 regarding the user's behavior, and determine the behavior of the robot 100 corresponding to the determined emotion.

[0019] More specifically, when the robot 100 recognizes the behavior of the user 10, the robot 100 automatically generates the behavioral content that the robot 100 should take in response to the behavior of the user 10, using a preset sentence generation model. The sentence generation model may be interpreted as an algorithm and calculation for automatic dialogue processing by text. The sentence generation model is described in, for example, JP 2018-081444 A and chatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ) and thus detailed description thereof will be omitted. Such a sentence generation model is configured using a large language model (LLM).

[0020] As described above, in this embodiment, by combining a large-scale language model and an emotion engine, it is possible to reflect the emotions of the user 10 and the robot 100 and various linguistic information in the behavior of the robot 100. In other words, according to this embodiment, by combining a sentence generation model and an emotion engine, a synergistic effect can be obtained.

[0021] The robot 100 also has a function of recognizing the behavior of the user 10. The robot 100 recognizes the behavior of the user 10 by analyzing a facial image of the user 10 acquired by a camera function and a voice of the user 10 acquired by a microphone function. The robot 100 determines the behavior to be performed by the robot 100 based on the recognized behavior of the user 10, etc.

[0022] The robot 100 stores rules that define actions to be taken by the robot 100 based on the emotions of the user 10, the emotions of the robot 100, and the actions of the user 10, and performs various actions according to the rules.

[0023] Specifically, the robot 100 has reaction rules for determining the behavior of the robot 100 based on the emotions of the user 10, the emotions of the robot 100, and the behavior of the user 10. For example, the reaction rules define the behavior of the robot 100 as "laughing" when the behavior of the user 10 is "laughing". The reaction rules also define the behavior of the robot 100 as "apologizing" when the behavior of the user 10 is "angry". The reaction rules also define the behavior of the robot 100 as "answering" when the behavior of the user 10 is "asking a question". The reaction rules also define the behavior of the robot 100 as "calling out" when the behavior of the user 10 is "sad".

[0024] When the robot 100 recognizes the behavior of the user 10 as "anger" based on the reaction rules, the robot 100 selects the behavior of "apologizing" defined in the reaction rules as the behavior to be executed by the robot 100. For example, when the robot 100 selects the behavior of "apologizing", the robot 100 performs the motion of "apologizing" and outputs a voice expressing the word "apologize".

[0025] In addition, when the emotion of the robot 100 is "normal" (i.e., "happy" = 0, "anger" = 0, "sad" = 0, and "happy" = 0) and the condition that the state of the user 10 is "alone and looks lonely" is satisfied, it is defined that the emotion of the robot 100 will change to "worried" and that the robot 100 will be able to execute the action of "calling out."

[0026] When the robot 100 recognizes based on the reaction rule that the current emotion of the robot 100 is "normal" and that the user 10 is alone and seems lonely, the robot 100 increases the emotion value of "sadness" of the robot 100. In addition, the robot 100 selects the action of "calling out" defined in the reaction rule as the action to be performed toward the user 10. For example, when the robot 100 selects the action of "calling out", the robot 100 converts the words "What's wrong?", which indicate concern, into a worried voice and outputs it.

[0027] The robot 100 also transmits to the server 300 user reaction information indicating that this action has elicited a positive reaction from the user 10. The user reaction information includes, for example, the user action of "getting angry," the robot 100 action of "apologizing," the fact that the reaction of the user 10 was positive, and attributes of the user 10.

[0028] The server 300 stores the user reaction information received from the robot 100. The server 300 receives and stores user reaction information not only from the robot 100, but also from each of the robots 101 and 102. The server 300 analyzes the user reaction information from the robots 100, 101, and 102, and updates the reaction rules.

[0029] The robot 100 receives the updated reaction rules from the server 300 by inquiring about the updated reaction rules from the server 300. The robot 100 incorporates the updated reaction rules into the reaction rules stored in the robot 100. This allows the robot 100 to incorporate the reaction rules acquired by the robot 101, the robot 102, etc. into its own reaction rules.

[0030] 2 shows a schematic functional configuration of the robot 100. The robot 100 includes a sensor unit 200, a sensor module unit 210, a storage unit 220, a user state recognition unit 230, an emotion determination unit 232, a behavior recognition unit 234, a behavior determination unit 236, a memory control unit 238, a behavior control unit 250, a control target 252, and a communication processing unit 280.

[0031] The control target 252 includes a display device, a speaker, LEDs in the eyes, and motors for driving the arms, hands, legs, etc. The posture and gestures of the robot 100 are controlled by controlling the motors of the arms, hands, legs, etc. Some of the emotions of the robot 100 can be expressed by controlling these motors. In addition, the facial expressions of the robot 100 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 100. The posture, gestures, and facial expressions of the robot 100 are examples of the attitude of the robot 100.

[0032] The sensor unit 200 includes a microphone 201, a 3D depth sensor 202, a 2D camera 203, and a distance sensor 204. The microphone 201 continuously detects sound and outputs sound data. The microphone 201 may be provided on the head of the robot 100 and may have a function of performing binaural recording. The 3D depth sensor 202 detects the contour of an object by continuously irradiating an infrared pattern and analyzing the infrared pattern from infrared images continuously captured by the infrared camera. The 2D camera 203 is an example of an image sensor. The 2D camera 203 captures an image using visible light and generates visible light video information. The distance sensor 204 detects the distance to an object by irradiating, for example, a laser or ultrasonic waves. The sensor unit 200 may also include a clock, a gyro sensor, a touch sensor, a sensor for motor feedback, and the like.

[0033] 2, the components other than the control target 252 and the sensor unit 200 are examples of components included in a behavior control system included in the robot 100. The behavior control system of the robot 100 controls the control target 252.

[0034] The storage unit 220 includes a reaction rule 221 and history data 222. The history data 222 includes the emotion value and behavior history of the user 10 in the past. The emotion value and behavior history are recorded for each user 10, for example, by being associated with the identification information of the user 10. At least a part of the storage unit 220 is implemented by a storage medium such as a memory. It may include a person DB that stores the face image of the user 10, attribute information of the user 10, and the like. Note that, among the components of the robot 100 shown in FIG. 2, the functions of the components other than the control target 252, the sensor unit 200, and the storage unit 220 can be realized by the CPU operating based on a program. For example, the functions of these components can be implemented as the operation of the CPU by the operating system (OS) and a program operating on the OS.

[0035] The sensor module unit 210 includes a voice emotion recognition unit 211, a speech understanding unit 212, a facial expression recognition unit 213, and a face recognition unit 214. Information detected by the sensor unit 200 is input to the sensor module unit 210. The sensor module unit 210 analyzes the information detected by the sensor unit 200, and outputs the analysis result to the user state recognition unit 230.

[0036] The voice emotion recognition unit 211 of the sensor module unit 210 analyzes the voice of the user 10 detected by the microphone 201 and recognizes the emotion of the user 10. For example, the voice emotion recognition unit 211 extracts feature amounts such as frequency components of the voice and recognizes the emotion of the user 10 based on the extracted feature amounts. The speech understanding unit 212 analyzes the voice of the user 10 detected by the microphone 201 and outputs text information representing the content of the utterance of the user 10.

[0037] The facial expression recognition unit 213 recognizes the facial expression and emotion of the user 10 from the image of the user 10 captured by the 2D camera 203. For example, the facial expression recognition unit 213 recognizes the facial expression and emotion of the user 10 based on the shapes, positional relationship, etc. of the eyes and mouth.

[0038] The face recognition unit 214 recognizes the face of the user 10. The face recognition unit 214 recognizes the user 10 by matching a face image stored in a person DB (not shown) with a face image of the user 10 captured by the 2D camera 203.

[0039] The user state recognition unit 230 recognizes the state of the user 10 based on the information analyzed by the sensor module unit 210. For example, the user state recognition unit 230 mainly performs processing related to perception using the analysis results of the sensor module unit 210. For example, it generates perceptual information such as "Daddy is alone" or "There is a 90% chance that Daddy is not smiling." It performs processing to understand the meaning of the generated perceptual information. For example, it generates semantic information such as "Daddy is alone and looks lonely."

[0040] The emotion determination unit 232 determines an emotion value indicating the emotion of the user 10 based on the information analyzed by the sensor module unit 210 and the state of the user 10 recognized by the user state recognition unit 230. For example, the information analyzed by the sensor module unit 210 and the recognized state of the user 10 are input to a pre-trained neural network to obtain an emotion value indicating the emotion of the user 10.

[0041] Here, the emotion value indicating the emotion of the user 10 is a value indicating the positive or negative emotion of the user. For example, if the emotion of the user is a cheerful emotion accompanied by a sense of pleasure or comfort, such as "joy", "pleasure", "comfort", "relief", "excitement", "relief", and "fulfillment", the value indicates a positive value, and the more cheerful the emotion, the larger the value. If the emotion of the user is an unpleasant emotion, such as "anger", "sorrow", "discomfort", "anxiety", "sorrow", "worry", and "emptiness", the value indicates a negative value, and the more unpleasant the emotion, the larger the absolute value of the negative value. If the emotion of the user is not any of the above ("neutral"), the value indicates 0.

[0042] Furthermore, the emotion determining unit 232 determines an emotion value indicating the emotion of the robot 100 based on the information analyzed by the sensor module unit 210 and the state of the user 10 recognized by the user state recognizing unit 230 . The emotion value of the robot 100 includes emotion values ​​for each of a plurality of emotion categories, and is, for example, a value (0 to 5) indicating the strength of each of "happiness," "anger," "sorrow," and "pleasure."

[0043] Specifically, the emotion determination unit 232 determines an emotion value indicating the emotion of the robot 100 in accordance with a rule for updating the emotion value of the robot 100, which rule is determined in association with the information analyzed by the sensor module unit 210 and the state of the user 10 recognized by the user state recognition unit 230.

[0044] For example, when the user state recognition unit 230 recognizes that the user 10 looks lonely, the emotion determination unit 232 increases the emotion value of "sadness" of the robot 100. Also, when the user state recognition unit 230 recognizes that the user 10 is smiling, the emotion determination unit 232 increases the emotion value of "joy" of the robot 100.

[0045] The emotion determining unit 232 may determine the emotion value indicating the emotion of the robot 100 by further considering the state of the robot 100. For example, when the battery level of the robot 100 is low or when the surrounding environment of the robot 100 is pitch black, the emotion value of "sadness" of the robot 100 may be increased. Furthermore, when the user 10 continues to talk to the robot 100 despite the battery level being low, the emotion value of "anger" may be increased.

[0046] The behavior recognition unit 234 recognizes the behavior of the user 10 based on the information analyzed by the sensor module unit 210 and the state of the user 10 recognized by the user state recognition unit 230. For example, the information analyzed by the sensor module unit 210 and the recognized state of the user 10 are input to a pre-trained neural network, the probability of each of a plurality of predetermined behavior classifications (for example, "laughing", "getting angry", "asking a question", "getting sad") is obtained, and the behavior classification with the highest probability is recognized as the behavior of the user 10.

[0047] As described above, in this embodiment, the robot 100 identifies the user 10 and acquires the contents of the user's utterance. When acquiring and using the contents of the utterance, the robot 100 obtains the necessary consent in accordance with laws and regulations from the user 10. The behavior control system of the robot 100 in this embodiment takes into consideration the protection of the personal information and privacy of the user 10.

[0048] The behavior determining unit 236 determines a behavior corresponding to the behavior of the user 10 recognized by the behavior recognizing unit 234 based on the current emotion value of the user 10 determined by the emotion determining unit 232, the history data 222 of past emotion values ​​determined by the emotion determining unit 232 before the current emotion value of the user 10 was determined, and the emotion value of the robot 100. In this embodiment, the behavior determining unit 236 uses one most recent emotion value included in the history data 222 as the past emotion value of the user 10, but the disclosed technology is not limited to this aspect. For example, the behavior determining unit 236 may use a plurality of most recent emotion values ​​as the past emotion value of the user 10, or may use an emotion value from a unit period ago, such as one day ago. In addition, the behavior determining unit 236 may determine a behavior corresponding to the behavior of the user 10 by further considering not only the current emotion value of the robot 100 but also the history of the past emotion values ​​of the robot 100. The behavior determined by the behavior determining unit 236 includes a gesture performed by the robot 100 or the contents of the speech of the robot 100 .

[0049] The behavior decision unit 236 according to this embodiment decides on the behavior of the robot 100 as a behavior corresponding to the behavior of the user 10, based on a combination of the past emotion value and the current emotion value of the user 10, the emotion value of the robot 100, the behavior of the user 10, and the reaction rules 221. For example, when the past emotion value of the user 10 is a positive value and the current emotion value is a negative value, the behavior decision unit 236 decides on a behavior for changing the emotion value of the user 10 to a positive value as a behavior corresponding to the behavior of the user 10.

[0050] The reaction rules 221 prescribe the behavior of the robot 100 according to a combination of the past emotion value and the current emotion value of the user 10, the emotion value of the robot 100, and the behavior of the user 10. For example, when the past emotion value of the user 10 is a positive value and the current emotion value is a negative value, and the behavior of the user 10 is sad, a combination of gestures and speech content when asking a question to encourage the user 10 with gestures is prescribe as the behavior of the robot 100.

[0051] For example, the reaction rule 221 defines behaviors of the robot 100 for all combinations of patterns of the emotion values ​​of the robot 100 (1296 patterns, which are the fourth power of six values ​​of "joy", "anger", "sorrow", and "pleasure", from "0" to "5"); combination patterns of the past emotion values ​​and the current emotion values ​​of the user 10; and behavior patterns of the user 10. That is, for each pattern of the emotion values ​​of the robot 100, behaviors of the robot 100 are defined according to the behavior patterns of the user 10 for each of a plurality of combinations of the past emotion values ​​and the current emotion values ​​of the user 10, such as negative values ​​and negative values, negative values ​​and positive values, positive values ​​and negative values, positive values ​​and positive values, negative values ​​and normal values, and normal values ​​and normal values. Note that the behavior determining unit 236 may transition to an operation mode in which the behavior of the robot 100 is determined using the history data 222, when the user 10 makes an utterance intending to continue a conversation from a past topic, such as "I want to talk about that topic we talked about last time." The reaction rules 221 may prescribe at least one of a gesture and a statement as the behavior of the robot 100 for each of the patterns (1296 patterns) of the emotion value of the robot 100. Alternatively, the reaction rules 221 may prescribe at least one of a gesture and a statement as the behavior of the robot 100 for each group of patterns of the emotion value of the robot 100.

[0052] The strength of each gesture included in the behavior of the robot 100 defined in the reaction rule 221 is determined in advance. The strength of each utterance included in the behavior of the robot 100 defined in the reaction rule 221 is determined in advance.

[0053] The memory control unit 238 determines whether or not to store data including the behavior of the user 10 in the history data 222 based on the predetermined behavior strength for the behavior determined by the behavior determination unit 236 and the emotion value of the robot 100 determined by the emotion determination unit 232. Specifically, when a total intensity value, which is the sum of the sum of the emotion values ​​for each of the multiple emotion classifications of the robot 100, the predetermined intensity for the gesture included in the behavior determined by the behavior determination unit 236, and the predetermined intensity for the speech content included in the behavior determined by the behavior determination unit 236, is equal to or greater than a threshold value, it is determined that data including the behavior of the user 10 is to be stored in the history data 222.

[0054] When the memory control unit 238 decides to store data including the behavior of the user 10 in the history data 222, it stores in the history data 222 the behavior determined by the behavior determination unit 236, information analyzed by the sensor module unit 210 from the present time up to a certain period of time ago (e.g., all peripheral information such as data on the sound, images, smells, etc. of the scene), and the state of the user 10 recognized by the user state recognition unit 230 (e.g., the facial expression, emotions, etc. of the user 10).

[0055] The behavior control unit 250 controls the control target 252 based on the behavior determined by the behavior determination unit 236. For example, when the behavior determination unit 236 determines an behavior including speaking, the behavior control unit 250 outputs a sound from a speaker included in the control target 252. At this time, the behavior control unit 250 may determine the speaking speed of the sound based on the emotion value of the robot 100. For example, the behavior control unit 250 determines a faster speaking speed as the emotion value of the robot 100 increases. In this way, the behavior control unit 250 determines the execution form of the behavior determined by the behavior determination unit 236 based on the emotion value determined by the emotion determination unit 232.

[0056] The behavior control unit 250 may recognize a change in the user 10's emotion in response to the execution of the behavior determined by the behavior determination unit 236. For example, the change in emotion may be recognized based on the voice or facial expression of the user 10. Alternatively, the change in emotion may be recognized based on the detection of an impact by a touch sensor included in the sensor unit 200. When an impact is detected by the touch sensor included in the sensor unit 200, the emotion of the user 10 may be recognized as having worsened, or when the reaction of the user 10 is determined to be laughing or happy based on the detection result of the touch sensor included in the sensor unit 200, the emotion of the user 10 may be recognized as having improved. Information indicating the reaction of the user 10 is output to the communication processing unit 280.

[0057] In addition, after the behavior control unit 250 executes the behavior determined by the behavior determination unit 236 in the execution form determined according to the emotion of the robot 100, the emotion determination unit 232 further changes the emotion value of the robot 100 based on the user's reaction to the execution of the behavior. Specifically, the emotion determination unit 232 increases the emotion value of "happiness" of the robot 100 when the user's reaction to the behavior determined by the behavior determination unit 236 being performed on the user in the execution form determined by the behavior control unit 250 is not bad. In addition, the emotion determination unit 232 increases the emotion value of "sadness" of the robot 100 when the user's reaction to the behavior determined by the behavior determination unit 236 being performed on the user in the execution form determined by the behavior control unit 250 is bad.

[0058] Furthermore, the behavior control unit 250 expresses the emotion of the robot 100 based on the determined emotion value of the robot 100. For example, when the behavior control unit 250 increases the emotion value of "happiness" of the robot 100, the behavior control unit 250 controls the control object 252 to make the robot 100 perform a happy gesture. When the behavior control unit 250 increases the emotion value of "sadness" of the robot 100, the behavior control unit 250 controls the control object 252 to make the robot 100 assume a droopy posture.

[0059] The communication processing unit 280 is responsible for communication with the server 300. As described above, the communication processing unit 280 transmits user reaction information to the server 300. In addition, the communication processing unit 280 receives updated reaction rules from the server 300. When the communication processing unit 280 receives the updated reaction rules from the server 300, it updates the reaction rules 221.

[0060] The server 300 communicates between the robots 100, 101, and 102 and the server 300, receives user reaction information transmitted from the robot 100, and updates the reaction rules based on reaction rules that include actions that have received positive reactions.

[0061] Fig. 3 shows an example of an operation flow for determining an action of the robot 100. The operation flow shown in Fig. 3 is repeatedly executed. At this time, it is assumed that information analyzed by the sensor module unit 210 is input. Note that "S" in the operation flow indicates a step to be executed.

[0062] First, in step S100 , the user state recognition unit 230 recognizes the state of the user 10 based on the information analyzed by the sensor module unit 210 .

[0063] In step S102 , the emotion determination unit 232 determines an emotion value indicating the emotion of the user 10 based on the information analyzed by the sensor module unit 210 and the state of the user 10 recognized by the user state recognition unit 230 .

[0064] In step S103, the emotion determination unit 232 determines an emotion value indicating the emotion of the robot 100, based on the information analyzed by the sensor module unit 210 and the state of the user 10 recognized by the user state recognition unit 230. The emotion determination unit 232 adds the determined emotion value of the user 10 to the history data 222.

[0065] In step S104 , the behavior recognition unit 234 recognizes a behavior classification of the user 10 based on the information analyzed by the sensor module unit 210 and the state of the user 10 recognized by the user state recognition unit 230 .

[0066] In step S106, the behavior determination unit 236 determines the behavior of the robot 100 based on a combination of the current emotional value of the user 10 determined in step S102 and the past emotional values ​​included in the history data 222, the emotional value of the robot 100, the behavior of the user 10 recognized by the behavior recognition unit 234, and the reaction rules 221.

[0067] In step S 108 , the behavior control unit 250 controls the control target 252 based on the behavior determined by the behavior determining unit 236 .

[0068] In step S110, the memory control unit 238 calculates a total intensity value based on the predetermined action intensities for the actions determined by the action determination unit 236 and the emotion value of the robot 100 determined by the emotion determination unit 232.

[0069] In step S112, the storage control unit 238 determines whether the total intensity value is equal to or greater than the threshold. If the total intensity value is less than the threshold, the process ends without storing data including the behavior of the user 10 in the history data 222. On the other hand, if the total intensity value is equal to or greater than the threshold, the process proceeds to step S114.

[0070] In step S114, the behavior determined by the behavior determination unit 236, the information analyzed by the sensor module unit 210 from the present time up to a certain period of time ago, and the state of the user 10 recognized by the user state recognition unit 230 are stored in the history data 222.

[0071] As described above, according to the robot 100, an emotion value indicating the emotion of the robot 100 is determined based on the user state, and whether or not to store data including the behavior of the user 10 in the history data 222 is determined based on the emotion value of the robot 100. This makes it possible to reduce the capacity of the history data 222 that stores data including the behavior of the user 10. For example, when the robot 100 determines that the user state 10 years from now is the same as that 10 years ago, the robot 100 can present to the user 10 all kinds of peripheral information, such as the state of the user 10 10 years ago (e.g., the facial expression, emotion, etc. of the user 10), and data on the sound, image, smell, etc. of the place.

[0072] Moreover, according to the robot 100, it is possible to make the robot 100 perform an appropriate action in response to the action of the user 10. Conventionally, the action of the user is classified, and the action including the facial expression and the appearance of the robot is determined. In contrast, the robot 100 determines the current emotional value of the user 10, and performs an action on the user 10 based on the past emotional value and the current emotional value. Therefore, for example, if the user 10 who was cheerful yesterday is depressed today, the robot 100 can make an utterance such as "You were cheerful yesterday, but what's wrong with you today?" The robot 100 can also make an utterance with gestures. For example, if the user 10 who was depressed yesterday is cheerful today, the robot 100 can make an utterance such as "You were depressed yesterday, but you look cheerful today, don't you?" For example, if the user 10 who was cheerful yesterday is more cheerful today than yesterday, the robot 100 can make an utterance such as "You're more cheerful today than yesterday. Has anything better happened than yesterday?" In addition, for example, the robot 100 can say to a user 10 whose emotional value is equal to or greater than 0 and whose emotional value fluctuation range continues to be within a certain range, "You've been feeling stable lately, which is good."

[0073] Also, for example, the robot 100 can ask the user 10, "Did you finish the homework you told me about yesterday?", and when the user 10 answers, "I did it," it can utter a positive utterance such as "Great!" and perform a positive gesture such as clapping or a thumbs up. Also, for example, when the user 10 utters, "The presentation you gave the day before yesterday went well," the robot 100 can utter a positive utterance such as "You did a good job!" and perform the above-mentioned positive gesture. In this way, it is expected that the user 10 will feel a sense of affinity with the robot 100 by the robot 100 performing an action based on the state history of the user 10.

[0074] In the above embodiment, the robot 100 recognizes the user 10 using a facial image of the user 10, but the disclosed technology is not limited to this aspect. For example, the robot 100 may recognize the user 10 using a voice emitted by the user 10, an email address of the user 10, an SNS ID of the user 10, an ID card with a built-in wireless IC tag that the user 10 possesses, or the like.

[0075] The robot 100 is an example of an electronic device equipped with a behavior control system. The application of the behavior control system is not limited to the robot 100, and the behavior control system can be applied to various electronic devices. The functions of the server 300 may be implemented by one or more computers. At least a part of the functions of the server 300 may be implemented by a virtual machine. At least a part of the functions of the server 300 may be implemented in the cloud.

[0076] 4 shows a schematic example of a hardware configuration of a computer 1200 functioning as the robot 100 and the server 300. A program installed on the computer 1200 can cause the computer 1200 to function as one or more "parts" of an apparatus according to the present embodiment, or cause the computer 1200 to execute operations or one or more "parts" associated with the apparatus according to the present embodiment, and / or cause the computer 1200 to execute a process or steps of the process according to the present embodiment. Such a program can be executed by the CPU 1212 to cause the computer 1200 to execute specific operations associated with some or all of the blocks of the flowcharts and block diagrams described herein.

[0077] The computer 1200 according to this embodiment includes a CPU 1212, a RAM 1214, and a graphics controller 1216, which are connected to each other by a host controller 1210. The computer 1200 also includes input / output units such as a communication interface 1222, a storage device 1224, a DVD drive 1226, and an IC card drive, which are connected to the host controller 1210 via an input / output controller 1220. The DVD drive 1226 may be a DVD-ROM drive, a DVD-RAM drive, or the like. The storage device 1224 may be a hard disk drive, a solid state drive, or the like. The computer 1200 also includes a legacy input / output unit such as a ROM 1230 and a keyboard, which are connected to the input / output controller 1220 via an input / output chip 1240.

[0078] The CPU 1212 operates according to a program stored in the ROM 1230 and the RAM 1214, thereby controlling each unit. The graphic controller 1216 acquires image data generated by the CPU 1212 into a frame buffer or the like provided in the RAM 1214 or into itself, and causes the image data to be displayed on the display device 1218.

[0079] The communication interface 1222 communicates with other electronic devices via a network. The storage device 1224 stores programs and data used by the CPU 1212 in the computer 1200. The DVD drive 1226 reads programs or data from a DVD-ROM 1227 or the like and provides the programs or data to the storage device 1224. The IC card drive reads programs and data from an IC card and / or writes programs and data to an IC card.

[0080] The ROM 1230 stores therein a boot program or the like that is executed by the computer 1200 upon activation, and / or a program that depends on the hardware of the computer 1200. The input / output chip 1240 may also connect various input / output units to the input / output controller 1220 via a USB port, a parallel port, a serial port, a keyboard port, a mouse port, and the like.

[0081] The programs are provided by a computer-readable storage medium such as a DVD-ROM 1227 or an IC card. The programs are read from the computer-readable storage medium, installed in the storage device 1224, the RAM 1214, or the ROM 1230, which are also examples of computer-readable storage media, and executed by the CPU 1212. Information processing described in these programs is read by the computer 1200, and brings about cooperation between the programs and the various types of hardware resources described above. An apparatus or method may be constructed by implementing operations or processing of information according to the use of the computer 1200.

[0082] For example, when communication is performed between the computer 1200 and an external device, the CPU 1212 may execute a communication program loaded in the RAM 1214 and instruct the communication interface 1222 to perform communication processing based on the processing described in the communication program. Under the control of the CPU 1212, the communication interface 1222 reads transmission data stored in a transmission buffer area provided in the RAM 1214, the storage device 1224, the DVD-ROM 1227, or a recording medium such as an IC card, and transmits the read transmission data to a network, or writes reception data received from the network to a reception buffer area or the like provided on the recording medium.

[0083] Furthermore, the CPU 1212 may cause all or a necessary portion of a file or database stored in an external recording medium such as the storage device 1224, the DVD drive 1226 (DVD-ROM 1227), an IC card, etc. to be read into the RAM 1214, and perform various types of processing on the data on the RAM 1214. The CPU 1212 may then write back the processed data to the external recording medium.

[0084] Various types of information, such as various types of programs, data, tables, and databases, may be stored in the recording medium and undergo information processing. The CPU 1212 may perform various types of processing on the data read from the RAM 1214, including various types of operations, information processing, conditional judgment, conditional branching, unconditional branching, information search / replacement, etc., as described throughout this disclosure and specified by the instruction sequence of the program, and write back the results to the RAM 1214. The CPU 1212 may also search for information in a file, database, etc. in the recording medium. For example, when a plurality of entries each having an attribute value of a first attribute associated with an attribute value of a second attribute are stored in the recording medium, the CPU 1212 may search for an entry whose attribute value of the first attribute matches a specified condition from among the plurality of entries, read the attribute value of the second attribute stored in the entry, and thereby obtain the attribute value of the second attribute associated with the first attribute that satisfies a predetermined condition.

[0085] The above-described programs or software modules may be stored in a computer-readable storage medium on the computer 1200 or in the vicinity of the computer 1200. In addition, a recording medium such as a hard disk or a RAM provided in a server system connected to a dedicated communication network or the Internet can be used as a computer-readable storage medium, thereby providing the programs to the computer 1200 via the network.

[0086] The blocks in the flowcharts and block diagrams in the present embodiment may represent stages of a process in which an operation is performed or "parts" of an apparatus responsible for performing the operation. Particular stages and "parts" may be implemented by dedicated circuitry, programmable circuitry provided with computer-readable instructions stored on a computer-readable storage medium, and / or a processor provided with computer-readable instructions stored on a computer-readable storage medium. The dedicated circuitry may include digital and / or analog hardware circuits, and may include integrated circuits (ICs) and / or discrete circuits. The programmable circuitry may include reconfigurable hardware circuits, such as, for example, field programmable gate arrays (FPGAs), programmable logic arrays (PLAs), and the like, including AND, OR, XOR, NAND, NOR, and other logical operations, flip-flops, registers, and memory elements.

[0087] A computer-readable storage medium may include any tangible device capable of storing instructions that are executed by a suitable device, such that a computer-readable storage medium having instructions stored thereon comprises an article of manufacture that includes instructions that can be executed to create means for performing the operations specified in the flowcharts or block diagrams. Examples of computer-readable storage media may include electronic storage media, magnetic storage media, optical storage media, electromagnetic storage media, semiconductor storage media, and the like. More specific examples of computer-readable storage media may include floppy disks, diskettes, hard disks, random access memories (RAMs), read-only memories (ROMs), erasable programmable read-only memories (EPROMs or flash memories), electrically erasable programmable read-only memories (EEPROMs), static random access memories (SRAMs), compact disk read-only memories (CD-ROMs), digital versatile disks (DVDs), Blu-ray disks, memory sticks, integrated circuit cards, and the like.

[0088] The computer readable instructions may include either assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state-setting data, or source or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, JAVA, C++, etc., and conventional procedural programming languages ​​such as the "C" programming language or similar programming languages.

[0089] Computer readable instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus, or to a programmable circuit, either locally or over a local area network (LAN), a wide area network (WAN), such as the Internet, etc., to cause the processor of the general purpose computer, special purpose computer, or other programmable data processing apparatus, or to a programmable circuit, to execute the computer readable instructions to generate means for performing the operations specified in the flowcharts or block diagrams. Examples of processors include computer processors, processing units, microprocessors, digital signal processors, controllers, microcontrollers, etc.

[0090] Although the present invention has been described above using the embodiments, the technical scope of the present invention is not limited to the scope described in the above embodiments. It is clear to those skilled in the art that various modifications and improvements can be made to the above embodiments. It is clear from the description of the claims that such modifications and improvements can also be included in the technical scope of the present invention.

[0091] It should be noted that the order of execution of each process, such as operations, procedures, steps, and stages, in the devices, systems, programs, and methods shown in the claims, specifications, and drawings is not specifically stated as "before" or "prior to," and that the process may be performed in any order unless the output of a previous process is used in a later process. Even if the operational flow in the claims, specifications, and drawings is described using "first," "next," etc. for convenience, it does not mean that the process must be performed in this order.

[0092] The emotion determining unit 232 may determine the emotion of the user according to a specific mapping. Specifically, the emotion determining unit 232 may determine the emotion of the user according to an emotion map (see FIG. 5), which is a specific mapping.

[0093] FIG. 5 is a diagram showing an emotion map 300 on which multiple emotions are mapped. In the emotion map 300, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive emotions are arranged. The more outside the concentric circles, the more emotions that represent states and actions that arise from a state of mind are arranged. Emotions are a concept that includes emotions and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions that occur in the brain are arranged. On the right side of the concentric circles, emotions that are generally induced by situational judgment are arranged. On the upper and lower sides of the concentric circles, emotions that are generally generated from reactions that occur in the brain and are induced by situational judgment are arranged. In addition, on the upper side of the concentric circles, emotions of "pleasure" are arranged, and on the lower side, emotions of "discomfort" are arranged. In this way, in the emotion map 300, multiple emotions are mapped based on the structure in which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[0094] (1) For example, if the emotion engine, which is the emotion determination unit 232 of the robot 100, detects emotions in about 100 msec, the frequency of the determination of the reaction action (e.g., a backchannel) of the robot 100 may be set at least at the same timing as the detection frequency (100 msec) of the emotion engine, or may be set at a timing earlier than this. The detection frequency of the emotion engine may be interpreted as a sampling rate.

[0095] By detecting emotions in about 100 msec and immediately performing a corresponding reaction (e.g., a backchannel), unnatural backchannels are eliminated, and a natural dialogue that reads the atmosphere can be realized. The reaction (backchannel, etc.) is performed according to the directionality and the degree (strength) of the mandala in the robot 100 emotion map 300. Note that the detection frequency (sampling rate) of the emotion engine is not limited to 100 ms, and may be changed according to the situation (e.g., when playing sports), the age of the user, etc.

[0096] (2) The directionality of emotions and the strength of their intensity may be preset in reference to the emotion map 300, and the movement of the interjections and the strength of the interjections may be set. For example, if the robot 100 feels a sense of stability and security, the robot 100 may nod and continue listening. If the robot 100 feels anxious, confused, or suspicious, the robot 100 may tilt its head or stop shaking its head.

[0097] These emotions are distributed in the three o'clock direction of emotion map 300 and usually fluctuate between relief and anxiety. In the right half of emotion map 300, situational awareness dominates over internal sensations, resulting in a sense of calm.

[0098] (3) If the robot 100 feels good after being praised, the filler "ah" may be inserted before the line, and if the robot 100 feels hurt after receiving harsh words, the filler "ugh!" may be inserted before the line. Also, a physical reaction such as the robot 100 crouching down while saying "ugh!" may be included. These emotions are distributed around 9 o'clock on the emotion map 300.

[0099] (4) In the left half of the emotion map 300, internal sensations (reactions) are more important than situational awareness. This can give the impression that the person is reacting unconsciously.

[0100] When the robot 100 feels a favorable feeling in its situation recognition while feeling an internal sensation (reaction) of a sense of satisfaction, the robot 100 may nod deeply while looking at the other person, or may say "uh-huh." In this way, the robot 100 may generate a behavior that shows a balanced favorable feeling toward the other person, that is, a tolerance or tolerance toward the other person. Such emotions are distributed around 12 o'clock on the emotion map 300.

[0101] Conversely, even when the robot 100 is aware of a situation while experiencing an internal sensation (reaction) of discomfort, the robot 100 may shake its head when it feels disgust, or turn the LEDs in its eyes red and glare at the other person when it feels hatred. These emotions are distributed around the 6 o'clock position on the emotion map 300.

[0102] (5) The inside of emotion map 300 represents what is going on in one's mind, while the outside of emotion map 300 represents behavior. Therefore, the further out you go on the emotion map 300, the more visible the emotions become (the more they are expressed in behavior).

[0103] (6) When listening to someone while feeling a sense of security, which is distributed around the 3 o'clock position on the emotion map 300, the robot 100 may lightly nod its head and say "hmm." However, when listening to someone while feeling a sense of love, which is distributed around 12 o'clock, the robot 100 may nod vigorously, nodding its head deeply.

[0104] The emotion determination unit 232 inputs the information analyzed by the sensor module unit 210 and the recognized state of the user 10 into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 300, and determines the emotion of the user 10. This neural network is pre-trained based on a plurality of learning data that are combinations of the information analyzed by the sensor module unit 210 and the recognized state of the user 10, and emotion values ​​indicating each emotion shown in the emotion map 300. In addition, this neural network is trained so that emotions that are arranged close to each other have similar values, as in the emotion map 900 shown in FIG. 6. FIG. 6 shows an example in which multiple emotions, such as "peace of mind," "calm," and "reassuring," have similar emotion values.

[0105] Furthermore, the emotion determination unit 232 may determine the emotion of the robot 100 according to a specific mapping. Specifically, the emotion determination unit 232 inputs the information analyzed by the sensor module unit 210, the state of the user 10 recognized by the user state recognition unit 230, and the state of the robot 100 to a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 300, and determines the emotion of the robot 100. This neural network is pre-trained based on a plurality of learning data that are combinations of the information analyzed by the sensor module unit 210, the recognized state of the user 10, and the state of the robot 100, and emotion values ​​indicating each emotion shown in the emotion map 300. For example, the neural network is trained based on learning data indicating that when it is recognized from the output of a touch sensor (not shown) that the robot 100 is being stroked by the user 10, the emotional value of "happy" is "3," and based on learning data indicating that when it is recognized from the output of an acceleration sensor (not shown) that the robot 100 is being hit by the user 10, the emotional value of "anger" is "3." Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 6.

[0106] The behavior decision unit 236 generates the robot's behavior content by adding fixed sentences for asking about the robot's behavior content corresponding to the user's behavior to text representing the user's behavior, the user's emotions, and the robot's emotions, and inputting the added text into the dialogue function.

[0107] For example, the behavior determining unit 236 obtains text representing the state of the robot 100 from the emotion of the robot 100 determined by the emotion determining unit 232, using an emotion table such as that shown in Table 1. Here, in the emotion table, an index number is assigned to each emotion value for each type of emotion, and text representing the state of the robot 100 is stored for each index number.

[0108] When the emotion of the robot 100 determined by the emotion determination unit 232 corresponds to the index number "2", the text "very happy state" is obtained. When the emotions of the robot 100 correspond to multiple index numbers, multiple texts representing the state of the robot 100 are obtained.

[0109] In addition, an emotion table as shown in Table 2 is prepared for the emotions of the user 10. In this case, if the user's action is speaking "***", the emotion of the robot 100 is index number "2", and the emotion of the user 10 is index number "3", "The robot is in a very happy state. The user is in a normal happy state. The user says "***". As the robot, how would you respond?" The sentence generation model is input to obtain the robot's action content. The action decision unit 236 decides the robot's action from the action content.

[0110] [Table 1]

[0111] [Table 2]

[0112] In this way, the robot 100 can change its behavior according to the index number that corresponds to the robot's emotion, so that the user gets the impression that the robot has a heart, and is encouraged to take actions such as talking to the robot.

[0113] The behavior determining unit 236 may generate the behavioral content of the robot by adding not only texts representing the user's behavior, the user's emotions, and the robot's emotions, but also texts representing the contents of the history data 222, adding a fixed sentence for asking about the behavioral content of the robot corresponding to the user's behavior, and inputting the result into a sentence generation model having a dialogue function. This allows the robot 100 to change its behavior according to the history data representing the user's emotions and behavior, so that the user has the impression that the robot has individuality, and is encouraged to take actions such as talking to the robot. The history data may further include the emotions and behavior of the robot.

[0114] The robot 100 may be mounted on a stuffed toy, or may be applied to a control device connected wirelessly or by wire to a control target device (speaker or camera) mounted on the stuffed toy.

[0115] In the above explanation, the case where the behavior control system is applied to the robot 100 has been exemplified, but the present invention is not limited to this aspect. Fig. 7 is a functional block diagram of an agent system 50 configured using some or all of the functions of the behavior control system. The agent system 50 is a computer system that performs a series of actions in accordance with the intentions of a user 60 through a dialogue with the user 60. The dialogue with the user 60 can be performed by voice or text.

[0116] The agent system 50 may be installed in, for example, a robot, a doll, a stuffed animal, a pendant, a smart watch, a smartphone, a smart speaker, earphones, a personal computer, etc. The agent system 50 may also be implemented in a web server and used via a web browser running on a communication terminal such as a smartphone owned by a user.

[0117] The agent system 50 plays a role of, for example, a butler, secretary, teacher, partner, friend, lover, or teacher acting for the user 60. The agent system 50 not only converses with the user 60, but also provides advice, guides to a destination, or makes recommendations according to the user's preferences. The agent system 50 also makes reservations, orders, or makes payments to the service provider 70.

[0118] The agent system 50, equipped with an emotion engine 53, determines the emotions of the user 60 and the agent's own emotions, and acts for the user 60 while taking into account the emotions of the user 60 and the agent. Here, the agent is an agent that runs on software. That is, the agent system 50 understands the emotions of the user 60, reads the mood, and provides heartfelt support, assistance, advice, and services. The agent system 50 also listens to the worries of the user 60, comforts, encourages, and cheers up the user. The agent system 50 also plays with the user 60, draws picture diaries, and reminds the user of the past. The agent system 50 acts in a way that increases the user 60's sense of happiness.

[0119] The agent system 50 includes a dialogue processing unit 51, a character setting unit 52, an emotion engine 53, a dialogue history storage unit 54, a sentence generation model 55, a command acquisition unit 56, and an RPA (Robotic Process Automation) 57.

[0120] The dialogue processing unit 51 includes a behavior decision unit 236 that performs processing related to utterances, and a behavior control unit 250. The dialogue processing unit 51 generates utterance contents for an agent to have a dialogue with a user 60, and outputs the utterance contents in the form of at least one of voice and text.

[0121] The character setting unit 52 sets the character of an agent (corresponding to the agent 52A shown in FIG. 9) when the agent system 50 has a dialogue with the user 60, based on a designation from the user 60. That is, the speech content output from the dialogue processing unit 51 is output through the agent having the set character. For example, it is possible to set real celebrities or famous people such as actors, entertainers, idols, and athletes as the characters. It is also possible to set fictional characters that appear in comics, movies, or animations. For example, it is possible to set "Princess Anne" played by "Audrey Hepburn" in the movie "Roman Holiday" as the agent character. When the character of the agent is known, the voice, speech, tone, and personality of the character are known, so that the user 60 only needs to designate a character of his / her choice, and the prompt setting in the character setting unit 52 is automatically performed. The voice, speech, tone, and personality of the set character are reflected in the dialogue with the user 60. That is, the dialogue processing unit 51 synthesizes a voice according to the character set by the character setting unit 52, and outputs the speech content of the agent using the synthesized voice. This allows the user 60 to feel as if he or she is interacting with his or her favorite character (for example, a favorite actor) in person.

[0122] When the agent system 50 is mounted on a device having a display such as a smartphone, an icon, still image, or video of the agent having a character set by the character setting unit 52 may be displayed on the display. The image of the agent is generated using an image synthesis technique such as 3D rendering. In the agent system 50, a dialogue with the user 60 may be performed while the image of the agent makes gestures according to the emotions of the user 60, the emotions of the agent, and the contents of the agent's speech. Note that the agent system 50 may output only audio without outputting an image when engaging in a dialogue with the user 60.

[0123] The emotion engine 53 includes an emotion determination unit 232, and determines an emotion value indicating the emotion of the user 60 and an emotion value of the agent itself. The emotion value of the agent itself is reflected in the emotion of the set character. When the agent system 50 converses with the user 60, not only the emotion of the user 60 but also the emotion of the character is reflected in the dialogue. In other words, the dialogue processing unit 51 outputs the speech content in a manner according to the emotion determined by the emotion engine 53.

[0124] The agent's emotions are also reflected when the agent system 50 behaves toward the user 60. For example, when the user 60 requests the agent system 50 to take a photo, whether the agent system 50 will take a photo in response to the user's request is determined according to the degree of "sadness" the agent is feeling. When the agent feels positive emotions, it will talk or behave in a friendly manner toward the user 60, and when the agent feels negative emotions, it will talk or behave in a hostile manner toward the user 60.

[0125] The dialogue history storage unit 54 is a storage medium for storing the history of the dialogue between the user 60 and the agent system 50. The dialogue history storage unit 54 may be realized by the history data 222 of the storage unit 220, or may be realized by an external cloud storage. When the agent system 50 dialogues with the user 60 or takes an action toward the user 60, the agent system 50 determines the dialogue content or the action content by taking into account the content of the dialogue history stored in the dialogue history storage unit 54. For example, the agent system 50 grasps the hobbies and preferences of the user 60 based on the dialogue history stored in the dialogue history storage unit 54. The agent system 50 generates dialogue content that matches the hobbies and preferences of the user 60, or provides recommendations. The dialogue processing unit 51 generates utterance content based on the dialogue history stored in the dialogue history storage unit 54. The dialogue history storage unit 54 stores personal information of the user 60, such as the name, address, telephone number, and credit card number, acquired through dialogue with the user 60. Here, the agent may spontaneously ask the user 60 whether or not to register personal information, such as "Would you like to register your credit card number?", and depending on the user's answer, the personal information may be stored in the dialogue history storage unit 54.

[0126] The sentence generation model 55 generates sentences that are responses to input text or voice. The sentence generation model 55 may be interpreted as algorithms and calculations for automatic dialogue processing by characters. The sentence generation model is described in, for example, JP 2018-081444 A and chatGPT (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ) and thus detailed description thereof will be omitted. Such a sentence generation model is configured using a large language model (LLM).

[0127] The dialogue processing unit 51 generates the utterance contents based on the sentences generated by the sentence generation model 55. Specifically, the dialogue processing unit 51 inputs the text or voice input by the user 60, the emotions of both the user 60 and the agent specified by the emotion engine 53, and the conversation history stored in the dialogue history storage unit 54 to the sentence generation model 55 to generate the utterance contents of the agent. At this time, the dialogue processing unit 51 may further input the personality of the character set by the character setting unit 52 to the sentence generation model 55 to generate the utterance contents of the agent. In the agent system 50, the sentence generation model 55 is not located on the front end side which is the touch point with the user 60, but is used merely as a tool of the agent system 50.

[0128] The command acquisition unit 56 is configured by the speech understanding unit 212, and acquires commands for the agent from voice or text uttered by the user 60 through dialogue with the user 60. The commands include the contents of actions to be performed by the agent system 50, such as information search, store reservation, ticket arrangement, purchase of goods and services, payment, route guidance to a destination, provision of recommendations, etc.

[0129] The RPA 57 performs an action according to the command acquired by the command acquisition unit 56. The RPA 57 performs an action related to the use of the service provider 70, such as information search, store reservation, ticket arrangement, purchase of goods and services, payment of the price, etc. The RPA 57 is an example of the "action execution unit" in the present invention.

[0130] The RPA 57 reads out and uses personal information of the user 60, which is necessary to execute an action related to the use of the service provider 70, from the dialogue history storage unit 54. For example, when the agent system 50 purchases a product at the request of the user 60, it reads out and uses personal information of the user 60, such as the name, address, telephone number, and credit card number, stored in the dialogue history storage unit 54. It is unkind and unpleasant for the user to request the user 60 to input personal information in the initial setting. In the agent system 50 according to this embodiment, instead of requesting the user 60 to input personal information in the initial setting, the personal information acquired through the dialogue with the user 60 is stored and read out and used as necessary. This makes it possible to avoid making the user 60 feel uncomfortable, and improves user convenience.

[0131] The agent system 50 executes the dialogue process through the following steps 1 to 6. (Step 1) The agent system 50 sets an agent character. Specifically, the character setting unit 52 sets an agent character when the agent system 50 interacts with the user 60, based on a designation from the user 60.

[0132] (Step 2) The agent system 50 acquires the state of the user 10 including the voice or text input from the user 10, the emotional value of the user 10, the emotional value of the robot 100, and the history data 222. Specifically, the agent system 50 performs the same processes as steps S100 to S103 described above, and acquires the state of the user 10 including the voice or text input from the user 10, the emotional value of the user 10, the emotional value of the robot 100, and the history data 222.

[0133] (Step 3) The agent system 50 determines the content of the agent's utterance. Specifically, the dialogue processor 51 inputs the text or voice input by the user 60, the emotions of both the user 60 and the character specified by the emotion engine 53, and the conversation history stored in the dialogue history storage unit 54 to the sentence generation model 55, and generates the content of the agent's utterance. For example, a fixed sentence such as "How would you respond as an agent in this situation?" is added to the text or voice input by the user 60, the emotions of both the user 60 and the character identified by the emotion engine 53, and the text representing the conversation history stored in the dialogue history storage unit 54, and this is input into the sentence generation model 55 to obtain the content of the agent's speech. As an example, if the text or voice input by user 60 is "Please make a reservation at a delicious Chinese restaurant nearby for tonight at 7pm," the agent's speech may be retrieved as "Understood," and "Here are some recommended restaurants: 1. AAAA. 2. BBBB. 3. CCCC. 4. DDDD." Also, if the text or voice input by the user 60 is "Number 4, DDDD, would be good," the agent's speech content acquired is "Understood. I will try to make a reservation. How many seats are there?".

[0134] (Step 4) The agent system 50 outputs the contents of the agent's utterance. Specifically, the dialogue processor 51 synthesizes a voice according to the character set by the character setting unit 52, and outputs the contents of the agent's utterance in the synthesized voice.

[0135] (Step 5) The agent system 50 judges whether it is time to execute an agent command. Specifically, the dialogue processor 51 judges whether it is time to execute an agent command based on the output of the sentence generation model 55. For example, if the output of the sentence generation model includes information indicating that the agent will execute a command, it is judged that it is time to execute the agent command, and the process proceeds to step 6. On the other hand, if it is judged that it is not time to execute the agent command, the process returns to step 2 above.

[0136] (Step 6) The agent system 50 executes the command of the agent. Specifically, the command acquisition unit 56 acquires the command of the agent from the voice or text uttered by the user 60 through a dialogue with the user 60. Then, the RPA 57 performs an action according to the command acquired by the command acquisition unit 56. For example, if the command is "information search", an information search is performed on a search site using a search query obtained through a dialogue with the user 60 and an API (Application Programming Interface). The dialogue processing unit 51 inputs the search results to the sentence generation model 55 to generate the utterance content of the agent. The dialogue processing unit 51 synthesizes a voice according to the character set by the character setting unit 52, and outputs the utterance content of the agent using the synthesized voice. Also, if the command is "make a reservation for a restaurant", the reservation information obtained through the dialogue with the user 60, the restaurant information, and an API (Application Programming Interface) are used to make a phone call to the restaurant by telephone software to make the reservation. At this time, the dialogue processing unit 51 uses a sentence generation model with a dialogue function to obtain the agent's utterance in response to the voice input from the other party. Then, the dialogue processing unit 51 inputs the result of the restaurant reservation (whether the reservation was successful or not) to the sentence generation model 55 to generate the agent's utterance. The dialogue processing unit 51 synthesizes a voice according to the character set by the character setting unit 52, and outputs the agent's utterance in the synthesized voice. Then, the process returns to step 2 above.

[0137] In step 6, the results of the actions taken by the agent (e.g., making a restaurant reservation) are also stored in the dialogue history storage unit 54. The results of the actions taken by the agent stored in the dialogue history storage unit 54 are utilized by the agent system 50 to understand the hobbies or preferences of the user 60. For example, if the same restaurant has been reserved multiple times, the agent system 50 may recognize that the user 60 likes that restaurant, and may use the reserved time period, or the details of the reservation such as the course content or price, as a criterion for choosing a restaurant the next time the reservation is made.

[0138] In this way, the agent system 50 executes dialogue processing, and can take actions related to the use of the service provider 70 as necessary. While the user is sleeping, the agent system 50 may search for news of high interest to the user based on the user's preferences acquired in advance, and when the user wakes up, the agent system 50 may deliver the news to the user. For example, the agent may say, "It seems that the Yankees won the game yesterday. Let's go to the baseball stadium tonight to cheer them on." When the user says, "I have to ask my dad," the dad may say, "I don't know if I can get tickets." The agent may then say, "I'll check if I can get tickets," and obtain seat requests and other information from the dialogue with the user, and secure the tickets.

[0139] 8 and 9 are diagrams showing an example of the operation of the agent system 50. Fig. 8 illustrates an example in which the agent system 50 makes a restaurant reservation through a dialogue with the user 60. The agent system 50 is able to grasp the preferences of the user 60 based on a dialogue history with the user 60, provide a recommended list of restaurants that match the preferences of the user 60, and make a reservation at the selected restaurant.

[0140] On the other hand, FIG. 9 illustrates an example in which the agent system 50 accesses a mail-order site through a dialogue with the user 60 to purchase a product. The agent system 50 can estimate the remaining amount of a drink stocked by the user 60 based on the dialogue history with the user 60, and can suggest and execute the purchase of the drink to the user 60. The agent system 50 can also understand the user's preferences based on the past dialogue history with the user 60, and can recommend snacks that the user likes. In this way, the agent system 50 supports the user 60's daily life by communicating with the user 60 as a butler-like agent and performing various actions such as making restaurant reservations or purchasing and paying for products.

[0141] The agent system 50 described above can be applied to smart glasses 72. FIG. 10 is a block diagram showing an example of a hardware configuration of the smart glasses 72. The smart glasses 72 are glasses-type smart devices, and are worn by a user 60 in the same manner as general glasses. The smart glasses 72 include a control device 74, a display 76, a touch panel 78, a speaker 80, a sensor group 82, a microphone 84, a camera 86, and a communication I / F 88. The smart glasses 72 are an example of the "electronic device" and the "wearable terminal" of the present invention.

[0142] The control device 74 is a device that controls the entire smart glasses 72, and is configured, for example, by a processor, memory, and storage (not shown). The display 76 displays various information to the user 60. The display 76 is, for example, a liquid crystal display. The display 76 is provided, for example, in the lens portion of the smart glasses 72, and the display contents are visible to the user 60. The touch panel 78 accepts input from the user 60. The speaker 80 outputs sound indicating various information to the user 60.

[0143] The sensor group 82 detects the state of the user 60. The sensor group 82 is configured to include a plurality of sensors. For example, the sensor group 82 includes an acceleration sensor 82A, a temperature sensor 82B, and a heart rate sensor 82C. Note that these sensors are merely examples, and it goes without saying that other sensors may be mounted to detect the state of the user 60.

[0144] The microphone 84 captures the voice of the user 60 or environmental sounds around the smart glasses 72. The camera 86 is capable of capturing images of the surroundings of the smart glasses 72. The camera 86 is also capable of capturing images of the face of the user 60. The camera 86 is, for example, a CCD camera. The communication I / F 88 controls communication between the smart glasses 72 and the outside. The control device 74, the display 76, the touch panel 78, the speaker 80, the sensor group 82, the microphone 84, the camera 86, and the communication I / F 88 are connected to a bus 90.

[0145] 11 is a diagram showing an example of a usage mode of the agent system 50 by the smart glasses 72. The smart glasses 72 provide various services to the user 60 using the agent system 50. For example, when the user 60 operates the smart glasses 72 (for example, by inputting voice into a microphone or tapping a touch panel with a finger), the smart glasses 72 start using the agent system 50. Here, using the agent system 50 includes a case where the control device 74 has a function similar to that of the agent system 50 and uses the function, and also includes a case where the smart glasses 72 communicate with an external device (for example, a server) via the communication I / F 88 to use the agent system 50 stored outside the smart glasses 72.

[0146] When the user 60 operates the smart glasses 72, a touch point is generated between the agent system 50 and the user 60. That is, the agent system 50 starts providing a service. As described above, in the agent system 50, the character setting unit 52 sets the character of the agent 52A (for example, the character of Audrey Hepburn).

[0147] As described above, the emotion engine 53 includes the emotion determining unit 232, and determines an emotion value indicating the emotion of the user 60 and an emotion value of the agent itself. Here, the emotion value indicating the emotion of the user 60 may be estimated from various sensors included in the sensor group 82 mounted on the smart glasses 72. For example, when the heart rate of the user 60 detected by the heart rate sensor 82C mounted on the smart glasses 72 is increasing, emotion values ​​such as "anxiety" and "fear" are estimated to be large. The heart rate sensor 82C is an example of the "acquisition unit" in the present invention.

[0148] Furthermore, when the temperature sensor 82B measures the user's body temperature and, for example, the temperature is higher than the average body temperature, the emotion value such as "pain" or "distress" is estimated to be large. Furthermore, when the acceleration sensor 82A detects that the user 60 is playing some kind of sport, the emotion value such as "fun" is estimated.

[0149] Also, for example, the emotion value of the user 60 (e.g., the emotion value of "sad") may be estimated based on the facial expression (e.g., a sad facial expression) of the user 60 captured by the camera 86 mounted on the smart glasses 72. In this case, the camera 86 is an example of the "acquisition unit" in the present invention.

[0150] Also, for example, the emotion value of the user 60 may be estimated from the voice of the user 60 or the content of the speech acquired by the microphone 84 mounted on the smart glasses 72. For example, if the user 60 is raising his / her voice, an emotion value such as "anger" is estimated. In this case, the microphone 84 is an example of the "acquisition unit" in the present invention.

[0151] When the emotion value estimated by the emotion engine 53 becomes higher than a predetermined value, the agent system 50 causes the smart glasses 72 to acquire information about the surrounding situation. Specifically, for example, the smart glasses 72 are caused to capture an image or video showing the surrounding situation of the user 60 (for example, people or objects in the vicinity). In addition, the smart glasses 72 are caused to record the surrounding environmental sound with a microphone. Other information about the surrounding situation includes information showing the date, time, location information, or weather. The agent system 50 acquires information about the surrounding situation when the emotion value becomes high from the smart glasses 72. The information about the surrounding situation is stored in the surrounding situation storage unit 54A together with the emotion value. The surrounding situation storage unit 54A is a storage medium for storing information about the surrounding situation acquired by the agent system 50. The surrounding situation storage unit 54A may be realized by the history data 222 of the storage unit 220, similar to the dialogue history storage unit 54, or may be realized by an external cloud storage. In this way, the surrounding situation obtained by the smart glasses 72 is stored in the surrounding situation storage unit 54A as a so-called life log in a state where it is associated with the emotion value of the user 60 at that time.

[0152] In the agent system 50, information indicating the surrounding situation is stored in the surrounding situation storage unit 54A in association with an emotional value. This allows the agent system 50 to grasp personal information of the user 60, such as the hobbies, preferences, or personality. For example, if an image showing a baseball game is associated with an emotional value such as "joy" or "fun," the agent system 50 will understand that the user 60's hobby is watching baseball games and that the user's favorite team or player is based on the information stored in the surrounding situation storage unit 54A.

[0153] When the agent system 50 converses with the user 60 or takes an action toward the user 60, the agent system 50 determines the content of the dialogue or the content of the action by taking into account the content of the surrounding circumstances stored in the surrounding circumstances storage unit 54A. It goes without saying that the content of the dialogue or the content of the action may be determined by taking into account the dialogue history stored in the dialogue history storage unit 54 as described above, in addition to the surrounding circumstances.

[0154] As described above, the dialogue processing unit 51 generates the utterance contents based on the sentences generated by the sentence generation model 55. Specifically, the dialogue processing unit 51 inputs the text or voice input by the user 60, the emotions of both the user 60 and the agent identified by the emotion engine 53, the conversation history stored in the dialogue history storage unit 54, the agent's personality, etc., to the sentence generation model 55 to generate the utterance contents of the agent. Furthermore, the dialogue processing unit 51 inputs the surrounding circumstances stored in the surrounding circumstances storage unit 54A to the sentence generation model 55 to generate the utterance contents of the agent.

[0155] The generated speech content is outputted to the user 60, for example, as a voice from the speaker 80 mounted on the smart glasses 72. In this case, a synthetic voice corresponding to the character of the agent is used as the voice. The dialogue processing unit 51 generates a synthetic voice by reproducing the voice quality of the agent character (for example, Audrey Hepburn), or generates a synthetic voice corresponding to the emotion of the character (for example, a voice with a stronger tone in the case of the emotion of "anger"). Also, instead of or together with the voice output, the speech content may be displayed on the display 76. The speaker 80 and the display 76 are examples of the "output unit" in the present invention.

[0156] The RPA 57 executes an operation according to a command (for example, an agent command obtained from a voice or text issued by the user 60 through a dialogue with the user 60). The RPA 57 performs actions related to the use of the service provider 70, such as information search, store reservation, ticket arrangement, purchase of goods and services, payment, route guidance, translation, etc.

[0157] As another example, the RPA 57 executes an operation of transmitting the contents of voice input by the user 60 (e.g., a child) through a dialogue with the agent to a destination (e.g., a parent). Examples of the transmission means include message application software, chat application software, and email application software.

[0158] When an operation is executed by the RPA 57, for example, a sound indicating that execution of the operation has been completed is output from the speaker 80 mounted on the smart glasses 72. For example, a sound such as "Your restaurant reservation has been completed" is output to the user 60. Also, for example, if the restaurant is fully booked, a sound such as "We were unable to make a reservation. What would you like to do?" is output to the user 60.

[0159] As described above, the smart glasses 72 provide various services to the user 60 by using the agent system 50. In addition, since the smart glasses 72 are worn by the user 60, the agent system 50 can be used in various situations, such as at home, at work, and outside the home.

[0160] In addition, since the smart glasses 72 are worn by the user 60, they are suitable for collecting a so-called life log of the user 60. Specifically, the emotion value of the user 60 is estimated based on the detection results of various sensors mounted on the smart glasses 72 or the recording results of the camera 86. Therefore, the emotion value of the user 60 can be collected in various situations, and the agent system 50 can provide services or speech contents suitable for the emotion of the user 60.

[0161] Moreover, in the smart glasses 72, the camera 86, the microphone 84, etc. obtain the surrounding conditions of the user 60. These surrounding conditions are associated with the emotion values ​​of the user 60. This makes it possible to estimate what emotion the user 60 felt in what situation. As a result, the accuracy of the agent system 50 in grasping the hobbies and preferences of the user 60 can be improved. By accurately grasping the hobbies and preferences of the user 60 in the agent system 50, the agent system 50 can provide services or speech contents that are suited to the hobbies and preferences of the user 60.

[0162] The agent system 50 can also be applied to a smart pendant 92. Fig. 12 is a block diagram showing an example of the hardware configuration of the smart pendant 92. The smart pendant 92 is a pendant-type smart device, and is worn by the user 60 in the same manner as a general pendant. The smart pendant 92 includes a control device 93, a speaker 94, a microphone 95, and a communication I / F 96. The smart pendant 92 is an example of the "electronic device" and "wearable terminal" of the present invention.

[0163] The control device 93 is a device that controls the entire smart pendant 92, and is composed of, for example, a processor, memory, and storage (not shown). The speaker 94 outputs sound indicating various information to the user 60. The speaker 94 is, for example, a speaker that can output sound with directionality (i.e., a directional speaker). The speaker 94 is set to have directionality toward the ears of the user 60. This prevents the sound from reaching people other than the user 60. The speaker 94 is an example of an "output unit" in the present invention.

[0164] The microphone 95 acquires the voice uttered by the user 60 or the environmental sound around the smart pendant 92. The smart pendant 92 is worn by the user 60 in a manner that it is hung from the neck of the user 60. Therefore, the smart pendant 92 is located relatively close to the mouth of the user 60 while being worn. This makes it easy to acquire the voice uttered by the user 60. The microphone 95 is an example of an "acquisition unit" in the present invention. The communication I / F 96 manages communication between the smart pendant 92 and the outside. The control device 93, the speaker 94, the microphone 95, and the communication I / F 96 are connected to a bus 97.

[0165] The agent system 50 can also be used in the smart pendant 92. That is, the services provided by using the agent system 50 in the smart glasses 72 described above (see FIG. 11) are also provided in the smart pendant 92. For example, the emotional values ​​of the user 60 and the agent are estimated through a dialogue with an agent with a set character, and the agent's speech content or the content of the service provided changes depending on the emotional value. In addition, for example, the situation around the user 60 and the emotional value at that time are recorded as a life log, and are reflected in the estimation of the emotional value of the user 60 or the content of the service provided.

[0166] The smart pendant 92 may also be provided with a sensor group (corresponding to the sensor group 82 in the smart glasses 72) in the same manner as the smart glasses 72. This allows information indicating the physical condition of the user 60 or the surrounding situation to be detected, and allows the agent system 50 to estimate an emotion value or record the surrounding situation.

[0167] As described above, the smart pendant 92 can also provide the same effect as the smart glasses 72 described above. The smart pendant 92 is lighter and easier to carry than a stuffed toy, a smart speaker, a smartphone, or smart glasses. The smart pendant 92 is lighter and more portable than other smart terminals such as a smartphone or smart glasses, and is usually worn. This allows the agent system 50 to be used in various situations, such as at home, at work, and on the go. The smart pendant 92 is more suitable for collecting the life log of the user 60, and can therefore collect the emotion value of the user 60 in various situations, and the agent system 50 can provide services or speech content that is appropriate for the emotion of the user 60.

[0168] Second Embodiment Next, a second embodiment will be described while omitting or simplifying the overlapping parts with the above embodiment. The second embodiment is characterized in that the agent system 50 purchases products based on the budget received from the user 60 and the history of products acquired by the user 60.

[0169] Here, the "budget" refers to a budget for a payment method such as cash, electronic money, virtual currency, or points. The "budget" may be a budget for any payment method, including prepaid, postpaid, and debit. "Acquisition" includes purchase, rental, and contract, and "goods" includes both goods and services.

[0170] In the second embodiment, a case will be described in which the agent system 50 purchases an item with prepaid electronic money.

[0171] 13 is a diagram illustrating an example of a functional configuration of an agent system 50 according to the second embodiment. As described above, the agent system 50 is configured using some or all of the functions of the behavior control system shown in FIG.

[0172] As shown in FIG. 13, the agent system 50 of this embodiment includes a reception unit 502, an acquisition unit 504, and a product determination unit 506 in addition to the functional configuration of the above-described embodiment.

[0173] The reception unit 502 in this embodiment has a function of receiving the amount of electronic money usable for purchasing goods. Specifically, the reception unit 502 receives the amount of electronic money charged by the user 60. As an example, the reception unit 502 receives the amount of "10,000 yen" charged by the user 60 as the usable electronic money. The amount of electronic money is an example of the "budget" in the present invention.

[0174] The history acquisition unit 504 of this embodiment has a function of acquiring a purchase history that records items purchased in the past by the user 60. Specifically, the acquisition unit 504 acquires the purchase history from the history data 222 (see FIG. 2) in the storage unit 220. As an example, the history acquisition unit 504 acquires a list of items purchased by the user 60, the purchase date, the purchased quantity, the emotional value of the user 60, etc. The purchase history is an example of "history information" in the present invention.

[0175] The product determination unit 506 of this embodiment has a function of determining an item to be purchased based on the amount of electronic money and the purchase history. As an example, the product determination unit 506 determines the items to be purchased in order of the most frequently purchased items so that the total amount falls within the range of 10,000 yen. In addition, when the emotion value of the user 60 for the purchased items is recorded in the purchase history, the product determination unit 506 may determine the item to be purchased based on the emotion value of the user 60. The product determination unit 506 may also determine the quantity of the items to be purchased.

[0176] Furthermore, the product determination unit 506 has a function of determining an item to be purchased based on the emotion value of the agent. Specifically, the product determination unit 506 determines an item to be purchased based on the emotion value of the agent determined by the emotion engine 53. As an example, the product determination unit 506 determines an item to be purchased by using the reaction rules 221 (see FIG. 2) in the storage unit 220.

[0177] In the reaction rules 221 in this embodiment, an item to be purchased is defined according to a combination of the past and current emotion values ​​of the user 60, the emotion value of the agent, and the behavior of the user 60. That is, the reaction rules 221 in this embodiment are characterized in that an item to be purchased is defined instead of the behavior of the robot 100 defined in the first embodiment. For example, when the past emotion value of the user 60 is a positive value, the current emotion value is a negative value, and the behavior of the user 60 is sad, an item to encourage the user 60 is defined as an item to be purchased by the agent. The item to encourage the user 60 is, for example, food that the user 60 likes, a doll of a character that the user 60 likes, goods of an artist that the user 60 likes, etc.

[0178] For example, the products to be purchased defined in the reaction rules 221 include the following (1) to (10).

[0179] (1) Items to celebrate the user. (2) An item to apologize to the user. (3) Gifts to show appreciation to users. (4) Items designed to make users laugh. (5) Items intended to distract the user. (6) Items intended to comfort the user. (7) Goods to solve users' problems. (8) Items for the entertainment of users. (9) Any item intended to harass users. (10) Items to reassure users.

[0180] For example, specific names of items determined by analyzing the history data 222 and the dialogue history storage unit 54 are registered as items defined in the reaction rules 221. For example, a "strawberry shortcake" is registered as "(1) an item for congratulating the user", and a "box of sweets" is registered as "(2) an item for apologizing to the user". The registered names of items may be updated at regular intervals.

[0181] The product determination unit 506 may not determine an item to be purchased if the item to be purchased is not registered in the reaction rule 221 or if the emotion value of the agent is a negative value. For example, the product determination unit 506 does not determine an item to be purchased if the emotion of the agent is "dislike," "anger," "unforgiveness," "hatred," or the like. In addition, when the agent is praised by the user 60 and the emotion of the agent becomes "happy," the product determination unit 506 determines an "item for thanking the user" as the item to be purchased. In addition, when the agent is scolded by the user 60 and the emotion of the agent becomes "sad," the product determination unit 506 determines an "item for apologizing to the user" as the item to be purchased.

[0182] The RPA 57 of this embodiment has a function of performing an action related to the purchase of the determined item. Specifically, the RPA 57 purchases the item from the mail order site using the amount of the received electronic money, the item name registered in the reaction rule 221 of the item to be purchased, and the API. Note that the RPA 57 does not need to purchase all the determined items in one purchase.

[0183] Furthermore, the RPA 57 of this embodiment has a function of performing an action related to purchasing an item at a predetermined timing. Specifically, the RPA 57 performs this action when the emotion value of the agent becomes equal to or greater than a predetermined value. As an example, the RPA 57 performs an action related to purchasing an item for entertaining the user when the emotion value of the agent, which is "fun," becomes equal to or greater than 4. When the emotion value of the agent becomes equal to or greater than a predetermined value, this is an example of a "predetermined timing." The predetermined timing includes, for example, a timing when the emotion value of the user becomes equal to or greater than a predetermined value, a timing when a purchase instruction is given from the user, and the like.

[0184] The RPA 57 of this embodiment has a function of communicating the reason for deciding the product to be purchased to the user 60. Specifically, the RPA 57 inputs information on which the product decision unit 506 decided the product to be purchased to the sentence generation model 55, and communicates the reason to the user 60 using the generated text as a message for the user 60. As an example, a case will be described in which the agent's emotion becomes "happy" because the user 60 is happy about his / her birthday, and the product determination unit 506 decides to purchase an "item to celebrate the user." In this case, the RPA 57 inputs the text "Happy Birthday!" into the gift message field of the mail order site. As another example, a case will be described in which the agent is scolded by the user 60, the agent's emotion becomes "sad", and the product determination unit 506 decides to purchase "an item to apologize to the user". In this case, the RPA 57 sends a message saying "I'm sorry. This is an apology item" by writing it in the text of an email, or conveys it to the user 60 through a dialogue.

[0185] FIG. 14 is a flowchart showing an example of the flow of a product determination process in which the agent system 50 according to the second embodiment determines an item to be purchased.

[0186] 14, the agent system 50 accepts the amount of money that the user 60 can use. As an example, the agent system 50 accepts the amount of electronic money that the user 60 has charged.

[0187] In step S201, the agent system 50 acquires the purchase history of the user 60. Specifically, the agent system 50 acquires the purchase history of the user 60 from the history data 222.

[0188] In step S202, the agent system 50 acquires the emotion of the agent. Specifically, in the agent system 50, the emotion determination unit 232 determines the emotion of the agent based on the state of the user 60 acquired by the user state recognition unit 230.

[0189] In step S203, the agent system 50 determines the item to be purchased based on the accepted amount, the acquired purchase history, and the agent's emotion. Then, the agent system 50 ends the product determination process. Note that, after the product determination process, the agent system 50 may proceed to a product acquisition process shown in FIG. 15, which will be described later.

[0190] 15 is a flowchart showing an example of the flow of a product acquisition process in which the agent system 50 according to the second embodiment purchases an item. The agent system 50 executes the product acquisition process at a predetermined timing after the product determination process.

[0191] In step S300 of Fig. 15, the agent system 50 executes an operation of purchasing the determined item. Specifically, the agent system 50 purchases the item determined in step S203 of Fig. 14. Specifically, the agent system 50 purchases the item from the mail order site using the amount of electronic money, the determined item, and the API by the RPA 57.

[0192] 15, the agent system 50 adds information about the purchased item to the purchase history. Specifically, the agent system 50 adds information about the purchased item to the purchase history of the history data 222 via the storage control unit 238.

[0193] In step S302, the agent system 50 acquires the state of the user 60 when the purchased item is recognized. Specifically, the agent system 50 acquires the state of the user 60 by the user state recognition unit 230.

[0194] In step S303, the agent system 50 determines the emotion of the user 60 regarding the purchased item from the acquired state of the user 60. Specifically, the agent system 50 determines the emotion of the user 60 by the emotion determining unit 232.

[0195] In step S304, the agent system 50 adds the emotion of the user 60 to the information of the purchased item stored in the purchase history. Specifically, the agent system 50 causes the storage control unit 238 to associate the information of the purchased item with the emotion value of the user 60 and add them to the purchase history of the history data 222. Then, the agent system 50 ends the product acquisition process.

[0196] (Summary of the second embodiment) The agent system 50 of the second embodiment purchases an item determined based on electronic money received from the user 60 and the purchase history of the user 60. That is, the agent system 50 can purchase an item that the user 60 has purchased in the past. In other words, the agent system 50 purchases an item that the user 60 regularly purchases on behalf of the user 60, thereby saving the user 60 the trouble of purchasing the item. On the other hand, the agent system 50 can purchase an item that the user 60 has never purchased before. In other words, the agent system 50 can purchase an item that the user 60 has never purchased on behalf of the user 60, thereby providing the user 60 with a new experience.

[0197] The agent system 50 of the second embodiment determines an article to be purchased based on a purchase history that records purchased articles and the user's 60 feelings toward the purchased articles. Therefore, the agent system 50 can reflect the user's 60's past feelings toward purchased articles in the decision of an article to be purchased. In other words, the agent system 50 can purchase an article that the user 60 felt favorably about and avoid purchasing an article in which the user 60 had no interest.

[0198] The agent system 50 of the second embodiment determines an item to be purchased based on the agent's emotion. Therefore, the agent system 50 can determine whether an item can be purchased or not depending on the agent's emotion, and can change the item to be purchased. Furthermore, the agent system 50 can express the agent's emotion by the item to be purchased. In other words, the agent system 50 can communicate with the user 60 through the purchased item.

[0199] The agent system 50 of the second embodiment transmits a message regarding the purchased item to the user 60. Therefore, the user 60 can know the agent's emotion through the message. In addition, the agent system 50 can inform the user 60 that an item matching the emotion of the user 60 has been purchased.

[0200] (Modification 1 of the second embodiment) In the second embodiment, the agent's emotions were used to determine the item to be purchased. In the first modification, the situation around the user 60 is used to determine the item to be purchased. In the second embodiment, prepaid electronic money was used to purchase the item, but in the first modification of the second embodiment, a case where a credit card is used to purchase the item will be described. In the above description, in the first modification, the part described as "amount of electronic money" will be replaced with "credit card payment limit" unless otherwise specified. The following describes the contents unique to the first modification.

[0201] In the first modification, the user state recognition unit 230 shown in FIG. 2 has a function of acquiring and recognizing an image of the situation around the user 60. Specifically, the user state recognition unit 230 acquires the image via the 2D camera 203 of the sensor unit 200. The user state recognition unit 230 also estimates the quantity of each item shown in the image by applying a known image recognition technique to the image. The image of the situation around the user 60 is an example of "information relating to the situation around the user" in the present invention, and the quantity of each item is an example of "information relating to the inventory of each product" in the present invention.

[0202] For example, when the user state recognition unit 230 recognizes a place where items that the user 60 regularly uses are stored, such as a refrigerator or a pantry, the user state recognition unit 230 acquires an image of the place and estimates the quantity of each item stored in the place. The image of the refrigerator or pantry is an example of the "image information" in the present invention.

[0203] In the first modification, the reception unit 502 shown in Fig. 13 has a function of receiving the credit card payment limit that can be used to purchase an item. More specifically, the reception unit 502 receives the limit input by the user 60. As an example, the reception unit 502 receives the amount that can be paid without a signature. The credit card payment limit is an example of the "budget" in the present invention.

[0204] The product determination unit 506 in the first modification has a function of determining an item to be purchased based on an image of the situation around the user 60. Specifically, the product determination unit 506 has a function of determining an item to be purchased based on the quantity of each item estimated by the user state recognition unit 230. As an example, when the estimated quantity of an item is below the minimum quantity of the item set by the user 60, the product determination unit 506 determines the item to be purchased.

[0205] The RPA 57 in the first modification performs an action related to the purchase of an item when the estimated quantity of the item falls below the minimum quantity set by the user 60 for a predetermined consecutive period. Specifically, the RPA 57 determines whether or not to purchase the item using the quantity of each item estimated by the user state recognition unit 230, the estimated date, and the minimum quantity set for each item. As an example, when the number of 500 mL PET bottles of carbonated water falls below 12 bottles for seven consecutive days, the RPA 57 purchases 24 bottles of the carbonated water PET bottles (one cardboard box). The minimum quantity of 12 bottles set by the user 60 is an example of the "predetermined threshold" in the present invention, and seven days is an example of the "predetermined period" in the present invention. After performing an action related to the purchase of an item, the RPA 57 may refrain from purchasing the item until the purchased item arrives.

[0206] FIG. 16 is a flowchart showing an example of the flow of an acquisition determination process in which the agent system 50 according to the first modification of the second embodiment determines whether or not to purchase an item.

[0207] 16, the agent system 50 accepts the amount of money that the user 60 can spend. As an example, the agent system 50 accepts the credit card limit set by the user 60.

[0208] In step S401, the agent system 50 acquires the purchase history of the user 60. Specifically, the agent system 50 acquires the purchase history of the user 60 from the history data 222.

[0209] In step S402, the agent system 50 acquires an image showing the surroundings of the user 60. Specifically, the agent system 50 acquires the image from the sensor unit 200.

[0210] In step S403, the agent system 50 estimates information on the quantity of each item from the acquired image. Specifically, the agent system 50 estimates the quantity of each item shown in the image by the user state recognition unit 230.

[0211] In step S404, the agent system 50 determines the item to be purchased based on the received amount, the user's purchase history, and information on the quantity of each item.

[0212] In step S405, the agent system 50 determines whether the quantity of the item is below the minimum quantity set by the user for a predetermined consecutive period. If the agent system 50 determines that the quantity of the item is below the minimum quantity set by the user for a predetermined consecutive period (step S405: YES), the agent system 50 proceeds to step S406. If the agent system 50 determines that the quantity of the item is not below the minimum quantity set by the user for a predetermined consecutive period (step S405: NO), the acquisition determination process ends.

[0213] In step S406, the agent system 50 executes the product acquisition process shown in Fig. 15. Then, the agent system 50 ends the acquisition determination process.

[0214] (Summary of Modification 1 of the Second Embodiment) The agent system 50 of the first modification determines an item to be purchased based on the situation around the user 60. Therefore, the agent system 50 can change the item to be purchased depending on the change in the situation around the user 60. Furthermore, since the agent system 50 does not need an instruction from the user 60 when purchasing an item, even if the user 60 does not notice a change in the situation around the user 60, the agent system 50 can purchase an item in accordance with the change.

[0215] The agent system 50 of the first modification determines the item to be purchased based on the quantity of items estimated from an image captured of the surroundings of the user 60. Therefore, the agent system 50 can change the item to be purchased depending on the quantity of items owned by the user 60. In other words, the agent system 50 can avoid purchasing an item that is in sufficient quantity.

[0216] The agent system 50 of the first modification purchases an item when the quantity of the item estimated from the video of the situation around the user 60 falls below a predetermined threshold for a predetermined consecutive period of time. That is, the agent system 50 can purchase the item so that the stock of the item does not run out. That is, the agent system 50 can save the user 60 the trouble of checking the quantity of the item and purchasing the item that seems to be in short supply.

[0217] (Modification 2 of the second embodiment) In the second embodiment, the agent's emotion is used to determine the item to be purchased. In the second modification, information received from the user 60 is used to determine the service to be contracted. In the second embodiment, the item is purchased using prepaid electronic money, but in the second modification of the second embodiment, a case where a service is contracted with the point balance given to the user 60 will be described. In the second modification, the above description, except where otherwise specified, refers to the "amount of electronic money" and the "credit card payment limit" and the "point balance" are replaced. In the second modification, the above description, except where otherwise specified, refers to the "item" and the "purchase" are replaced with the "contract". The following describes the contents specific to the second modification.

[0218] In the second modification, the reception unit 502 shown in Fig. 13 has a function of receiving the balance of points that can be used to sign up for a service. More specifically, the reception unit 502 receives the balance input by the user 60 from the total amount of points owned by the user 60. As an example, the reception unit 502 receives a balance of "10,000 pt" out of the points owned by the user 60. The service contract is an example of "obtaining a product" in the present invention, and the balance of points is an example of "budget" in the present invention.

[0219] Furthermore, the reception unit 502 in the second modification has a function of receiving at least one of information related to a service category and information related to the preferences of the user 60. Specifically, the reception unit 502 in the second modification presents a plurality of service categories and a plurality of preferences to the user 60 as options, and receives the service category and preference selected by the user 60.

[0220] The multiple service categories presented by the reception unit 502 include, for example, the following categories:

[0221] (1) Video (2) Music (3) Books and e-books (4) Games (5) Home appliances and digital devices (6) Sports (7) Beauty (8) Healthcare and diet (9) Fashion (10) Housing-related matters (11) Food and drink (12) Leisure

[0222] The multiple preferences presented by the reception unit 502 are items into which each category is subdivided. The items into which each category is subdivided will be described using "(1) Video" as an example. Items into which the category "(1) Video" is subdivided are, for example, "movies," "dramas," "anime," and "variety shows." The reception unit 502 may also present items into which "movies" are further subdivided, such as "comedy," "action," "horror," "romance," "fantasy," "sci-fi," and "documentary." Furthermore, the reception unit 502 may present items into which "movies" are subdivided from a different perspective, such as "film director," "year of release," "cast," and "target age group."

[0223] The history acquisition unit 504 in the second modification has a function of acquiring a contract history that records services for which the user 60 has previously subscribed. Specifically, the acquisition unit 504 acquires the contract history from the history data 222 in the storage unit 220. As an example, the history acquisition unit 504 acquires a list of the items of services for which the user 60 has subscribed, the contract date, the contract quantity, the emotion value of the user 60, and the like. The contract history is an example of "history information" in the present invention.

[0224] The product determination unit 506 in the second modification has a function of determining a service to be contracted for based on the information received by the reception unit 502. Specifically, the product determination unit 506 determines a service that suits the tastes of the user 60 from among the categories selected by the user 60. As an example, the product determination unit 506 determines a service for viewing movies by a movie director that the user 60 likes from the video category. A video is an example of "information related to a product category" in the present invention, and a director that the user 60 likes is an example of "information related to a user's tastes" in the present invention.

[0225] The product determination unit 506 also has a function of determining an item to be purchased based on the point balance and the contract history. As an example, the product determination unit 506 determines the service to be contracted for in order of the frequency of contracting so that the total points are within the range of 10,000 points. In addition, if the emotion value of the user 60 for the contracted service is recorded in the contract history, the product determination unit 506 may determine the service to be contracted for based on the emotion value of the user 60.

[0226] The RPA 57 in the second modification performs actions related to the contract for the determined service. As an example, the RPA 57 uses the point balance input by the user 60, the determined service, and an API to make a contract for the service at a provider site that provides the service. As an example, the RPA 57 concludes a contract that allows the user 60 to watch movies by his / her favorite director on a site that provides a movie viewing service. Note that if the contract requires the consent of the user 60, the RPA 57 performs actions related to the contract excluding the procedure for obtaining the consent of the user 60.

[0227] FIG. 17 is a flowchart showing an example of the flow of a product determination process in which the agent system 50 according to the second modification of the second embodiment determines a service to be contracted.

[0228] 17, the agent system 50 accepts the amount of money that can be used by the user 60. As an example, the agent system 50 accepts the remaining amount of points that can be used by the agent system 50 from the total amount of points owned by the user 60.

[0229] In step S501, the agent system 50 acquires the contract history of the user 60. Specifically, the agent system 50 acquires the contract history of the user 60 from the history data 222.

[0230] In step S502, the agent system 50 receives information on the category of the service from the user 60. Specifically, the agent system 50 receives the information when the user 60 selects a category presented by the receiving unit 502.

[0231] In step S503, the agent system 50 accepts information on the preferences of the user 60 from the user 60. Specifically, the agent system 50 accepts the information by the user 60 selecting items into which each category presented by the accepting unit 502 is subdivided.

[0232] In step S504, the agent system 50 determines the service to be contracted based on the received amount, the contract history of the user 60, and the information received from the user 60. Then, the agent system 50 ends the product determination process. Note that the agent system 50 may proceed to the product acquisition process shown in FIG. 15 after the product determination process.

[0233] (Summary of Modification 2 of Second Embodiment) The agent system 50 of the second modification determines a service to be contracted for based on at least one of information on the service category and information on the preferences of the user 60. That is, the agent system 50 can contract for a service based on the settings of the user 60. That is, the agent system 50 can contract for a service that matches the preferences of the user 60.

[0234] [remarks] In the first and second embodiments, the emotion determination unit 232 determines an emotion value indicating the emotion of the user 60 and an emotion value of the agent itself based on the information analyzed by the sensor module unit 210 and the state of the user 60 recognized by the user state recognition unit 230. However, it is not always possible to acquire the state of the user 60. Therefore, the emotion determination unit 232 may determine an emotion value indicating the emotion of the user 60 and an emotion value of the agent itself based on a dialogue history between the user 60 and the agent. The dialogue history may be stored in either the dialogue history storage unit 54 or the history data 222.

[0235] In the second embodiment, the reception unit 502 receives the amount of electronic money charged by the user 60 as an example, but the method of receiving the budget is not limited to this. For example, the reception unit 502 may automatically receive points acquired by the user through shopping or the like, or may automatically receive an amount set by the user each month.

[0236] In the second embodiment, the product determination unit 506 determines the product to be purchased based on the reaction rule 221, but the method of determining the product is not limited to this. The product determination unit 506 determines the product according to, for example, the level of the emotional value of the agent. As an example, the product determination unit 506 determines the product to be purchased to be a more expensive product as the emotional value of the agent increases.

[0237] In the second embodiment, the agent system 50 makes the user 60 perform a procedure that the user 60 must perform. For example, with respect to a settlement, contract, or the like that requires the consent or input of the user 60, the agent system 50 confirms with the user 60 whether or not to consent or input.

[0238] In the first modification of the second embodiment, the agent system 50 acquires an image of the user's surroundings from the sensor unit 200, but the means for acquiring the image is not limited to this. When the agent system 50 is mounted on the smart glasses 72 shown in FIG. 10, the image may be acquired via the camera 86.

[0239] In the first modification of the second embodiment, the agent system 50 purchases an item when the quantity of the item is below the minimum quantity set by the user 60 for a predetermined consecutive period of time, but the timing of purchasing the item is not limited to this. The agent system 50 may purchase the item immediately when the quantity of the item falls below the minimum quantity.

[0240] In the first modification of the second embodiment, the agent system 50 purchases an item when the quantity of the item falls below a predetermined threshold for seven consecutive days, but it is not necessary to recognize the quantity of the item for seven consecutive days. For example, if the agent system 50 recognizes that the quantity of the item is insufficient on the first day and then recognizes that the quantity of the item is insufficient on the eighth day, it can be considered that the quantity has been below the predetermined threshold for seven consecutive days. [Explanation of symbols]

[0241] 5 system, 10, 11, 12, 60 user, 20 communication network, 100, 101, 102 robot, 200 sensor unit, 201 microphone, 202 depth sensor, 203 camera, 204 distance sensor, 210 sensor module unit, 211 voice emotion recognition unit, 212 speech understanding unit, 213 facial expression recognition unit, 214 face recognition unit, 220 storage unit, 221 reaction rule, 222 history data, 230 user state recognition unit, 232 emotion determination unit, 234 behavior recognition unit, 236 behavior determination unit, 238 memory control unit, 250 behavior control unit, 252 control target, 280 communication processing unit, 300 server, 1200 computer, 1210 host controller, 1212 CPU, 1214 RAM, 1216 graphic controller, 1218 Display device, 1220 I / O controller, 1222 Communication interface, 1224 Storage device, 1226 DVD drive, 1227 DVD-ROM, 1230 ROM, 1240 I / O chip, 50 Agent system, 51 Dialogue processor, 52 Character setting unit, 52A Agent, 53 Emotion engine, 54 Dialogue history storage unit, 54A Surrounding situation storage unit, 55 Sentence generation model, 56 Command acquisition unit, 57 RPA, 70 Service provider, 72 Smart glasses, 74 Control device, 76 Display, 78 Touch panel, 80 Speaker, 82 Sensor group, 82A acceleration sensor, 82B temperature sensor, 82C heart rate sensor, 84 microphone, 86 camera, 88 communication I / F, 90 bus, 92 smart pendant, 93 control device, 94 speaker, 95 microphone, 96 communication I / F, 97 bus

Claims

1. a reception unit that receives a budget for a payment method that can be used to purchase the product; a history acquisition unit that acquires history information that is information regarding a purchase history that records products that a user has previously purchased; a product determination unit that determines a product to be acquired based on the budget accepted by the acceptance unit and the history information obtained from the history acquisition unit; an action execution unit that executes an action of acquiring the product determined by the product determination unit at a predetermined timing; An agent system comprising:

2. A user state recognition unit capable of acquiring information regarding the user's state; and an emotion determination unit that determines an emotion of the user based on information about the state of the user acquired by the user state recognition unit, the history information includes a purchase history in which a product obtained by the operation execution unit and the user's emotion toward the product determined by the emotion determination unit are recorded in association with each other; The agent system according to claim 1 .

3. The emotion determining unit further determines an emotion of the agent based on the state of the user acquired via the user state recognizing unit; the product determination unit determines a product to be acquired based on the emotion of the agent determined by the emotion determination unit; The agent system according to claim 2 .

4. The user state recognition unit further acquires information regarding a situation surrounding the user, The product determination unit determines a product to be acquired based on the surrounding situation acquired by the user state recognition unit. The agent system according to claim 2 .

5. The information about the surrounding situation is information about inventory of each product inferred from image information representing the surrounding situation of the user, The product determination unit determines a product to be acquired based on the information about the inventory. The agent system according to claim 4.

6. the operation execution unit executes an operation to acquire the product when the inventory falls below a predetermined threshold for a predetermined consecutive period of time. The agent system according to claim 5 .

7. The receiving unit further receives at least one setting information among information on a product category and information on a user's preferences, The product determination unit determines a product to be acquired based on the setting information received by the reception unit. The agent system according to claim 2 .

8. A program for causing a computer to operate as the agent system according to any one of claims 1 to 7.

Citation Information

Patent Citations

  • Determination of analite in medium containing particle

    JP1985053847A

Cited By

  • Information processing device, information processing method, and program

    JP2026142014A

  • Information Processing Apparatus, Information Processing Method, and Program

    JP7711334B1