Control system, system, and program
The control system addresses the issue of inadequate emotional data recording by dynamically adjusting recording formats based on emotional states, ensuring comprehensive data capture and improved interaction analysis.
Patent Information
- Application Number
- PCT/JP2024/018115
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-06-09
- Filing Date
- 2024-05-16
- Publication Date
- 2024-11-28
AI Technical Summary
Conventional systems fail to appropriately record data based on the emotions of users or robots, leading to incomplete or inaccurate data capture.
A control system that includes a recording control section and a switching control section to adjust the recording format based on the emotional state of users or robots, determining emotion values and switching between high and low compression formats to ensure detailed data recording when emotional thresholds are exceeded.
Enables appropriate and detailed recording of data according to emotional states, improving the accuracy of data capture and learning from interactions, particularly during heightened emotional events.
Smart Images

Figure JP2024018115_28112024_PF_FP_ABST
Abstract
Description
Control system, system and program
[0001] The present invention relates to a control system, a system, and a program.
[0002] A terminal that learns conversations between a user and a call partner and stores the call partner's responses to user questions in a response table is known (see, for example, Patent Document 1). Also, an emotion generation device is known that includes a neural network that inputs user information, device information, and its own current emotional state and outputs its next emotional state (see, for example, Patent Document 2). Also, a technology is known that stores spatiotemporal patterns in an associative memory that includes multiple electronic neurons having a layer-neural network relationship with directional artificial synaptic connectivity (see, for example, Patent Document 3).
[0003] JP 2011-253389, JP 10-254592, Special publication 2013-535067
[0004] However, conventional techniques have had the problem that data may not be recorded appropriately depending on the emotions of the user or robot.
[0005] The present invention has been made in view of the above, and has an object to provide a control system, a system, and a program that can appropriately record data according to emotions.
[0006] According to one embodiment, a control system for controlling the behavior of an electronic device includes a recording control unit and a switching control unit. The recording control unit processes at least a portion of information continuously detected by a sensor to generate and record information in a first recording format or information in a second recording format having a larger amount of information than the information in the first recording format. When the switching control unit is generating information in the first recording format, if an emotion of the electronic device or a user exceeds a threshold, the recording format used by the recording control unit is switched from the first recording format to the second recording format.
[0007] Furthermore, a control system according to one aspect of the embodiment includes an emotion determination unit that determines an emotion value indicating the intensity of the emotion of the electronic device or a user, or a type of emotion, based on information detected by a sensor provided in the electronic device; an identification unit that identifies a manner in which to record data indicating information about the user detected by the sensor, based on the information determined by the emotion determination unit; and a recording control unit that records the previous data based on the manner identified by the identification unit.
[0008] Furthermore, a control system according to one aspect of the embodiment includes an emotion determination unit that determines the emotion of one or both of a user and an electronic device based on at least a portion of information detected by a sensor that continuously detects a user; a memory unit that stores information obtained by processing at least a portion of the information detected by the sensor, in association with the determined emotion of one or both of the user and the electronic device; and an output control unit that, when the information stored in the memory unit is output by the electronic device, controls to change one or both of the behavior of the electronic device and the output timing of the information according to the emotion associated with the information.
[0009] Furthermore, a control system according to one aspect of the embodiment is a control system for controlling the storage of an electronic device, and includes an emotion determination unit that determines the emotions of a user and the electronic device based on at least a portion of information detected by a sensor that continuously detects the user, a memory unit that stores information obtained by processing at least a portion of the information detected by the sensor and associating it with the determined emotions of one or both of the user and the electronic device, and a storage control unit that controls the storage period of the information stored in the memory unit according to the emotion associated with the information.
[0010] Furthermore, a control system according to one aspect of the embodiment is a system for controlling the memory of an electronic device, and includes: an emotion determination unit that determines an emotion of the electronic device based on information detected by a sensor provided in the electronic device; a memory unit that stores the information detected by the sensor provided in the electronic device in association with the emotion determined by the emotion determination unit; and a recording control unit that updates, based on the emotion newly determined by the emotion determination unit, an emotion previously determined by the emotion determination unit that is associated with information previously detected by the sensor provided in the electronic device and that includes elements common to the information newly detected by the sensor provided in the electronic device, based on elements related to the emotion newly determined by the emotion determination unit.
[0011] Furthermore, a control system according to one aspect of the embodiment includes an emotion determination unit that determines the intensity of a user's emotion based on at least a portion of information detected by a sensor that continuously detects the user, and a memory control unit that records information obtained by processing at least a portion of the information detected by the sensor and switches the information to be recorded depending on whether the intensity of the user's emotion determined by the emotion determination unit exceeds a threshold.
[0012] Furthermore, a control system according to one aspect of the embodiment includes an information acquisition unit that acquires event information relating to an event experienced by a target object whose memory is to be controlled, an emotion determination unit that determines the intensity of the emotion of the target object based on the event information acquired by the information acquisition unit, and a memory control unit that stores the intensity of the emotion determined by the emotion determination unit in association with the event information.
[0013] Furthermore, a control system according to one aspect of the embodiment is a control system for controlling the behavior of an electronic device, and includes a recording control unit that processes at least a portion of information continuously detected by a sensor to generate and record information in a first recording format or information in a second recording format that has a larger amount of information than the information in the first recording format, and a switching control unit that switches the recording format used by the recording control unit from the first recording format to the second recording format in response to frequent changes in the emotions of the electronic device or the user when information in the first recording format is being generated.
[0014] In addition, a control system according to one aspect of the embodiment includes a recording control unit that processes at least a portion of the information continuously detected by the sensor to generate information in a first recording format or information in a second recording format that has a larger amount of information than the information in the first recording format, and tags and records the information in the second recording format; and an output control unit that controls the output of information in the second recording format selected from the information in the second recording format recorded by the recording control unit based on the tagged information.
[0015] According to one aspect of the embodiment, data can be appropriately recorded according to emotions.
[0016] FIG. 1 is a diagram schematically illustrating an example of a system according to the first embodiment. FIG. 2 is a diagram illustrating an example of emotional value changes. FIG. 3 is a diagram schematically illustrating the block configuration of a server and a robot. FIG. 4 is a diagram schematically illustrating a neural network. FIG. 5 is a diagram schematically illustrating neural network parameters in table format. FIG. 6 is a diagram schematically illustrating the operation flow of the server 200 when the robot is started or reset. FIG. 7 is a diagram schematically explaining the calculation of the coupling coefficient of an artificial synapse. FIG. 8 is a diagram illustrating the calculation of the coupling coefficient using a function h as an increase / decrease parameter for the coupling coefficient. t ijFIG. 9 is a diagram showing the time evolution of the coupling coefficient when simultaneous firing occurs at time t2. FIG. 10 is a diagram showing the influence definition information defining the chemical influence given to the parameters. FIG. 11 is a diagram showing a flowchart for calculating the internal state and status. FIG. 12 is a diagram showing an example of calculation of the internal state when the artificial neuron does not fire. FIG. 13 is a diagram showing an example of calculation of the output when the artificial neuron fires. FIG. 14 is a diagram showing the time evolution of the coupling coefficient when a function is defined as the increase / decrease parameter of the artificial neuron. FIG. 15 is a diagram showing an example of a rule stored in the recording format switching rule in table format. FIG. 16 is a diagram showing an example of a system according to the second embodiment. FIG. 17 is a diagram showing an example of emotional value changes. FIG. 18 is a diagram showing the block configuration of the server and robot. FIG. 19 is a diagram showing a flowchart related to robot operation control. FIG. 20 is a diagram showing a flowchart related to data storage period control. FIG. 21 is a diagram schematically illustrating block configurations of a server and a robot according to a third embodiment. FIG. 22 is a diagram schematically illustrating an example of a system according to a fourth embodiment. FIG. 23 is a diagram schematically illustrating an action performed by a robot toward a user. FIG. 24 is a diagram schematically illustrating a scene in which a robot switches the recording format of data to be recorded in a server. FIG. 25 is a diagram schematically illustrating the functional configuration of a robot. FIG. 26 is a diagram schematically illustrating the functional configuration of a server. FIG. 27 is a diagram schematically illustrating information perceived by a perception processing unit. FIG. 28 is a diagram schematically illustrating information understood by a semantic understanding unit. FIG. 29 is a diagram showing information included in reaction rules in a table format. FIG. 30 is a diagram schematically illustrating processing based on reaction rules. FIG. 31 is a diagram schematically illustrating an example of an operation flow related to an operation of determining an action in a robot. FIG. 32 is a diagram illustrating an example of reaction rule statistical information stored in a server. FIG. 33 is a diagram schematically illustrating an action performed by a robot to determine a label to be attached to recorded data. FIG. 34 is a diagram illustrating an example of a recording format switching rule.FIG. 35 is a diagram schematically showing an example of an operational flow related to switching of the recording format. FIG. 36 is a diagram showing an example of a pattern table. FIG. 37 is a diagram showing an example of a recording mode table. FIG. 38 is a diagram showing an example of a process flow for recording data in a recording mode corresponding to an emotion. FIG. 39 is a diagram schematically showing an example of a system according to a fifth embodiment. FIG. 40 is a diagram schematically showing the functional configuration of a robot. FIG. 41 is a diagram schematically showing an example of an operational flow related to an operation for determining information to be recorded in a robot. FIG. 42 is a diagram showing an emotion map to which multiple emotions are mapped. FIG. 43 is a diagram showing another example of an emotion map. FIG. 44 is a diagram showing an example of an emotion table. FIG. 45 is a diagram showing an example of an emotion table. FIG. 46 is an external view of a stuffed animal according to another embodiment. FIG. 47 is a rear front view of a stuffed animal according to another embodiment. FIG. 48 is a diagram schematically showing an example of the overall configuration of a system according to a sixth embodiment. FIG. 49 is a diagram schematically showing the functional block configuration of a robot and a server. FIG. 50 is a diagram schematically showing an emotion map. FIG. 51 is a diagram schematically showing a part of a neural network used by the system. FIG. 52 is a diagram showing an example of correspondence information correlating the remaining capacity of a storage battery with endocrine substances. FIG. 53 is a diagram showing an example of coupling coefficient correspondence information correlating the secretion amount of noradrenaline with the coupling coefficient BS. FIG. 54 is a diagram showing a schematic diagram of a change in excitement level over time in a robot. FIG. 55 is a diagram showing a schematic diagram of an example of a change in memory strength of memory information stored in the storage unit over time. FIG. 56 is a diagram showing another example of a change in memory strength of memory information stored in the storage unit over time. FIG. 57 is a diagram for explaining an acquisition sequence of information acquired from a robot. FIG. 58 is a diagram showing an example of information stored in the storage unit. FIG. 59 is a diagram for explaining an acquisition sequence of incidental memory information attached to memory information. FIG. 60 is a diagram explaining an endocrine substance reflected when an event different from the recollection information or an event negating the recollection information occurs. FIG. 61 is a diagram showing selection allowance information indicating recollection information that is allowed to be selected as recollection information. FIG. 62 is a diagram showing a schematic diagram of an example of a relationship between the number of identical event information and the intensity of emotion.Fig. 63 is a diagram schematically showing an example of the relationship between emotional intensity and the amount of memory retention. Fig. 64 is a diagram schematically showing an example of the relationship between attention and the amount of memory retention. Fig. 65 is a diagram schematically showing an example of the relationship between the number of attention targets and the amount of memory retention. Fig. 66 is a diagram schematically showing an example of the relationship between the number of identical event information and the amount of memory retention. Fig. 67 is a diagram schematically showing an example of the operation flow of the server.
[0017] The present invention will be described below through embodiments of the invention, but the following embodiments do not limit the scope of the invention according to the claims. Furthermore, not all of the combinations of features described in the embodiments are necessarily essential to the solution of the invention.
[0018] First Embodiment Fig. 1 schematically illustrates an example of a system 20 according to a first embodiment. Fig. 2 is a diagram illustrating an example of changes in emotional values. The system 20 includes a server 200, and robots 40a and 40b. The robots 40a and 40b communicate with the server 200 via a communication network 90 to exchange information. The robots 40a and 40b are examples of electronic devices.
[0019] The user 30a is the user of the robot 40a. The user 30b is the user of the robot 40b. The robot 40b has substantially the same functions as the robot 40a. Therefore, the system 20 will be described with the robots 40a and 40b collectively referred to as the robots 40.
[0020] The robot 40 performs various actions depending on the situation, such as moving its head or limbs, conversing with the user 30, or providing video to the user 30. At this time, the robot 40 decides its actions in cooperation with the server 200. For example, the robot 40 transmits to the server 200 detection information such as a facial image of the user 30 acquired using a camera function or the voice of the user 30 acquired using a microphone function. The server 200 analyzes the detection information received from the robot 40, determines the action the robot 40 should take, and transmits action information representing the decided action to the robot 40. The robot 40 operates in accordance with the action information received from the server 200.
[0021] The robot 40 has an emotional value that represents its own emotions. For example, the robot 40 has an emotional value that represents the strength of each emotion, such as "happy," "fun," "sad," "fear," and "excitement." The emotional value of the robot 40 is determined by the server 200. The server 200 causes the robot 40 to perform an action according to the determined emotion. For example, when the robot 40 is having a conversation with the user 30 in a state where the emotional value of excitement is high, the server 200 causes the robot 40 to speak quickly. In this way, the robot 40 can express its own emotions through actions, etc.
[0022] The server 200 uses a neural network to update the current state of the robot 40 based on the detection information received from the robot 40. The state of the robot 40 includes the emotion of the robot 40. Therefore, the server 200 uses the neural network to determine the emotion of the robot 40.
[0023] The robot 40 also records video data of the user 30 acquired using the camera function on the server 200. The robot 40 acquires video data from the server 200 as needed and provides it to the user 30. The stronger the robot's emotion, the greater the amount of information the robot generates and records on the server 200. For example, when the robot 40 is recording information in a highly compressed format, such as skeletal data, the robot 40 switches to recording information in a low-compression format, such as HD video, when the robot's excitement level exceeds a threshold. The system 20 allows high-definition video data to be recorded when the robot 40's emotion is heightened. The stronger the user's emotion, the greater the amount of information the robot 40 generates and records on the server 200. Specifically, when the robot 40 is recording information in a highly compressed format, such as skeletal data, the robot 40 switches to recording information in a low-compression format, such as HD video, when the robot's excitement level exceeds a threshold. According to the system 20, high-definition video data of the user 30 when he / she is feeling emotional can be recorded.
[0024] Furthermore, when the intensity of the emotion of the robot 40 or the user exceeds a threshold, the robot 40 generates video data with a large amount of information and records it on the server 200. For example, when the robot 40 is recording information in a highly compressed format such as skeletal data, the robot 40 switches to recording information in a low-compression format such as HD video when the emotional value of the robot 40 or the user 30 exceeds a threshold, which is a trigger. According to the system 20, high-definition video data can be recorded when the intensity of the emotion of the robot 40 or the user 30 exceeds a threshold.
[0025] For example, suppose that the robot 40 interacts with a user (e.g., a conversation) and the user's anger level exceeds a threshold. In other words, suppose that the robot 40's behavior angers the user. In such a case, the robot 40 records a large amount of information in the server 200, and the system 20 can learn based on detailed information about when the robot 40 angered the user.
[0026] In this case, the system 20 learns information about the flow of conversation between the user and the robot 40, statements, and expressions, as well as the facial expressions and attitudes of the user and the robot 40. Then, the system 20 can control the robot 40 to refrain from the behavior (conversation, gestures, etc.) that made the user uncomfortable in the same situation from next time onwards.
[0027] As a specific example, the system 20 records the flow of conversation, remarks, facial expressions, etc. in a situation where the robot 40 is calm and the user becomes emotional, and based on this information, in a similar situation, the system 20 restricts the behavior of the robot 40 when the user becomes emotional, and controls it to take a different action.
[0028] On the other hand, when performing such control, it is conceivable that, for example, the user may not be particularly concerned about the restricted behavior of the robot 40, and therefore, it is possible that the behavior of the robot 40 may be excessively restricted.
[0029] Therefore, the system 20 may, for example, weight the behavior of the robot 40 that makes the user feel uncomfortable over time. For example, the system 20 may gradually remove restrictions on the behavior of the robot 40 that makes the user feel uncomfortable as time passes.
[0030] Furthermore, the system 20 may record actions that the robot 40 has taken toward the user in association with the emotional value of the user at that time, and may determine whether the robot 40 will take the same action after leveling out the emotional value of the user when the robot 40 has taken the same action. In this case, when the robot 40 has taken a specific action (for example, an action that angers the user), the system 20 may cause the server 200 to record data with a large amount of information, regardless of the magnitude of the emotional value.
[0031] Furthermore, the robot 40 generates video data with a large amount of information and records it in the server 200 when the user's emotion is transmitted to the robot, or when the robot's emotion is transmitted to the user.
[0032] Here, emotion propagation refers to the emotion value of a specific emotion of the user or robot 40 that is the source of emotion exceeding a threshold, and then the emotion value of the same emotion increases in the user or robot 40 that is the destination of emotion. For example, emotion propagation refers to the emotion value of the anger of the robot increasing after the emotion value of the anger of the user exceeds a threshold. Note that when emotion propagation occurs, the emotion value of the anger of the user or robot 40 that is the destination of emotion is assumed to be below a predetermined value, that is, the user or robot 40 that is the destination of emotion is not angry.
[0033] Specifically, as shown in FIG. 2, after the user's anger emotion value A1 exceeds a threshold, the user's emotion is transmitted to the robot 40 through an interaction between the user and the robot (e.g., a conversation), causing the robot 40's emotion to change from calm to anger, and the anger emotion value A2 exceeds the threshold.
[0034] In such a case, the robot 40 causes the server 200 to record data with a large amount of information, triggered by the user's emotional value A1 exceeding the threshold and then the robot's emotional value A2 exceeding the threshold.
[0035] In such a case, the server 200 can record detailed information on when the user's emotions are transmitted to the robot 40. By collecting detailed information on such cases, the system 20 can improve the behavior of the robot 40 toward the user when the user becomes emotional, for example, when the user's emotional value exceeds a threshold.
[0036] 2 shows a case where the user's emotion value and the robot's emotion value exceed their respective thresholds, but the present invention is not limited to this. Emotion propagation may occur when the user's anger emotion value exceeds the threshold, and then when the robot's anger emotion value exceeds the threshold, the user's anger emotion value may be equal to or less than the threshold. Emotion propagation also includes a case where the robot's anger emotion value increases after the user's anger emotion value exceeds the threshold, but does not exceed the threshold.
[0037] Furthermore, when a strong emotion of the robot 40 is transmitted to the user, the robot 40 also records a video or the like with a large amount of information in the server 200. As a specific example, suppose that after the emotion value of the robot 40, anger, exceeds the threshold, the user's emotion changes from calm to anger through a conversation between the robot 40 and the user, and the emotion value of anger exceeds the threshold.
[0038] In such a case, the system 20 can collect detailed data when the robot 40's anger is transmitted to the user, and can use such data to improve the robot 40's behavior.
[0039] For example, the system 20 can control the emotions of the robot 40 so that the emotions of the robot 40 do not change to anger due to the user's comments, or change the behavior (tone of voice, wording, gestures, etc.) of the robot 40 when it is angry to behavior that does not change the user's emotions to anger.
[0040] Furthermore, the system 20 may switch the detection trigger for starting recording of data with a large amount of information depending on the situation. For example, the system 20 switches the detection trigger between the emotion value of the robot 40 or the user exceeding a threshold, the transmission of the user's emotion to the robot 40, and the transmission of the robot 40's emotion to the user depending on the situation.
[0041] For example, the system 20 switches the detection trigger for starting recording using the topic of the conversation between the user and the robot 40, specific keywords or phrases used in the conversation, etc. as keys. For example, when the system 20 detects a phrase or phrase in an utterance by the robot 40 that caused the user's emotional value to exceed a threshold in the past, the system 20 switches the detection trigger to the user's emotional value exceeding the threshold. In other words, the system 20 switches the detection trigger using the user or the robot 40 rehashing a past conversation as a key.
[0042] In such a state, recording may not be started even if the anger emotion value of the robot 40 exceeds the threshold. In other words, in this case, the system 20 starts recording by focusing on the emotion value of the user after the robot 40 uses a specific phrase or expression.
[0043] In addition, when the system 20 detects a phrase or expression in a user's speech that has caused the robot 40's anger emotion value to exceed a threshold in the past, the system 20 switches to a detection trigger using the fact that the robot 40's anger emotion value has exceeded the threshold as a key.
[0044] In addition, the system 20 switches the detection trigger to the fact that the user's emotions have been transmitted to the robot 40, using as a key when the conversation is on a specific topic, such as when the user and the robot 40 are conversing about the topic of a conversation when the user's emotions were transmitted to the robot 40 in the past.
[0045] For example, in this case, the system 20 starts recording when it detects that the emotion value of the robot 40 has exceeded the threshold after the emotion value of the user's anger has exceeded the threshold. Additionally, when switching the detection trigger, the system 20 may switch the detection trigger by combining conditions such as weather, temperature, the number of people in the vicinity, location, time of day, facial expression, and emotion.
[0046] Furthermore, the robot 40 tags information to be recorded when it detects a current detection trigger, such as when the emotional value of itself or the robot 40 exceeds a threshold. Specifically, the robot 40 tags video data with information detected by sensors provided in the robot 40 and information on the emotional value of the user or itself. The tagged information includes, for example, surrounding information (such as information on surrounding objects), smell, temperature, humidity, location information, weather, the number of people in the vicinity, their facial expressions, and emotions. This allows the user 30 to clearly recall the situation at the time, etc., from the tagged information when reviewing the recorded information. Furthermore, tagging makes it easier to find information related to a specific situation from the recorded information.
[0047] 3 is a schematic block diagram of the server 200 and the robot 40b. The robot 40b has a sensor unit 156, a processing unit 152, a control target 155, a communication unit 158, and a display unit 157. The server 200 has a processing unit 202, a storage unit 172, and a communication unit 208. The processing unit 202 includes an initial value setting unit 210, an external input data generation unit 230, a parameter processing unit 240, an action determination unit 250, a switching control unit 260, and a recording control unit 270. The storage unit 172 stores an action determination rule 282, definition information 284, parameter initial values 286, latest parameters 288, a recording format switching rule 290, and recorded data 292.
[0048] In the robot 40, the sensor unit 156 includes sensors such as a microphone 161, a 2D camera 163, a 3D depth sensor 162, and a distance sensor 164. Each sensor in the sensor unit 156 continuously detects information. The sensor information detected by the sensor unit 156 is output to the processing unit 152. The 2D camera 163 is an example of an image sensor that continuously captures images of an object using visible light to generate video information. The 3D depth sensor 162 continuously emits an infrared pattern and detects the contour of the object by analyzing the infrared pattern from the infrared images continuously captured by the infrared camera. In addition, the sensor unit 156 may include various other sensors, such as a clock, a gyro sensor, a touch sensor, a sensor for motor feedback, a sensor for detecting remaining battery capacity, an odor sensor, a temperature sensor, a humidity sensor, and a GPS (Global Positioning System) sensor.
[0049] The processing unit 152 is formed by a processor such as a CPU. The processing unit 152 transmits sensor information continuously detected by each sensor of the sensor unit 156 to the server 200 via the communication unit 158. The processing unit 152 also processes at least a portion of the sensor information continuously detected by each sensor of the sensor unit 156 to generate information for recording. The processing unit 152 generates information in a first recording format or information in a second recording format that contains more information than the information in the first recording format. The information in the first recording format is, for example, information in a high-compression format, and the information in the second recording format is, for example, information in a low-compression format. For example, the processing unit 152 generates shape data such as skeletal data of an object as information in the first recording format based on skeletal information continuously detected by the 3D depth sensor 162. The processing unit 152 also generates full HD video data and audio data from video information captured by the 2D camera 163 and audio information detected by the microphone 161. The full HD video data is an example of video data that contains more information than object shape data.
[0050] The communication unit 158 transmits the information in the first recording format or the information in the second recording format generated by the processing unit 152 to the server 200. In the server 200, the recording control unit 270 stores the information in the first recording format or the information in the second recording format received by the communication unit 208 from the robot 40 in the recording data 292. The recording control unit 270 stores the information received from each robot 40 in the recording data 292 in association with information identifying each robot 40.
[0051] Furthermore, in the robot 40, the communication unit 158 acquires information stored in the recording data 292 from the server 200. The communication unit 158 functions as a recording information receiving unit that acquires information in the second recording format, including video data, that has been recorded by the recording control unit 270. The processing unit 152 generates video to be presented to the user 30, based on the video data included in the information in the second recording format received by the communication unit 158. The processing unit 152 functions as an image generating unit that generates video to be presented to the user 30.
[0052] Furthermore, the communication unit 158 receives operation information indicating operation content from the server 200. The processing unit 152 controls the control target 155 based on the operation content received by the communication unit 158. The control target 155 includes a speaker, motors that drive each part of the robot 40 such as the limbs, a light-emitting device, etc. When information indicating the vocal content is received from the server 200, the processing unit 152 outputs sound from the speaker in accordance with the received vocal content. Furthermore, the processing unit 152 can control some of the behaviors of the robot 40 by controlling the drive motors of the limbs. Furthermore, the processing unit 152 can express some of the emotions of the robot 40 by controlling these motors.
[0053] In the server 200, the communication unit 208 outputs the information received from the robot 40 to the processing unit 202. The initial value setting unit 210 stores the initial values of the parameters indicating the initial state of the neural network received by the communication unit 208 in the parameter initial values 286 in the storage unit 172. Note that the initial values of the neural network parameters may be predetermined in the server 200, or may be changeable by the user 30 via the communication network 90.
[0054] The external input data generation unit 230 processes at least a portion of the sensor information received by the communication unit 208 to generate input information from outside the neural network, and outputs the generated information to the parameter processing unit 240. The parameter processing unit 240 performs calculations on the neural network based on the input information and the current parameters 288 and definition information 284 of the neural network stored in the storage unit 172.
[0055] The artificial neurons in the neural network include multiple artificial neurons defining the robot 40's situation, multiple emotional artificial neurons defining the robot 40's own emotions, and multiple endocrine artificial neurons defining the robot 40's own endocrine substance production state. The parameter processing unit 240 calculates parameters representing the internal states of the multiple artificial neurons in the neural network based on the input information generated by the external input data generation unit 230. For example, the parameter processing unit 240 updates the current internal state parameters of the multiple artificial neurons defining the robot 40's situation, etc., based on the input information generated by the external input data generation unit 230. The parameter processing unit 240 also calculates internal state parameters of other artificial neurons in the neural network. As a result, for example, the internal state parameters of an emotional artificial neuron defining the emotion "happy" are calculated. The internal state parameters of this emotional artificial neuron are an example of an index representing the intensity of the emotion "happy." Therefore, the parameter processing unit 240 can determine the intensity of emotions in the control system based on the internal states of the emotional artificial neurons. In this way, the parameter processing unit 240 functions as an emotion determination unit that determines the intensity of an emotion using a neural network based on at least a part of the information detected by the sensors of the sensor unit 156 .
[0056] The neural network parameters calculated by the parameter processing unit 240 are supplied to the switching control unit 260 and the action determination unit 250. The switching control unit 260 determines the recording format of the information generated by the processing unit 152 of the robot 40 based on the parameters supplied from the parameter processing unit 240. When it is necessary to switch the recording format of the information generated by the processing unit 152, the switching control unit 260 causes the robot 40 to transmit an instruction to switch the recording format via the communication unit 208. In the robot 40, the processing unit 152 switches the recording format in accordance with the instruction received from the server 200.
[0057] For example, when the switching control unit 260 is causing the processing unit 152 to generate information in the first recording format, the switching control unit 260 transmits to the robot 40 an instruction to switch the recording format of the information generated by the processing unit 152 from the first recording format to the second recording format in response to an increase in the intensity of the emotion determined by the parameter processing unit 240. In this way, the switching control unit 260 switches the recording format of the information recorded by the recording control unit 270 from the first recording format to the second recording format in response to an increase in the intensity of the emotion determined by the parameter processing unit 240. This allows detailed information about when the robot 40's emotions become heightened to be recorded. Furthermore, in the robot 40, the processing unit 152 obtains video data in the second recording format from the server 200 and generates a video to be presented to the user 30. Therefore, the user 30 can enjoy information about when the robot 40's emotions become heightened as a video.
[0058] Furthermore, in response to an increase in the intensity of the user 30's emotion, the switching control unit 260 transmits to the robot 40 an instruction to switch the recording format of the information generated by the processing unit 152 from the first recording format to the second recording format. In this way, in response to an increase in the intensity of the emotion determined by the parameter processing unit 240, the switching control unit 260 switches the recording format of the information recorded by the recording control unit 270 from the first recording format to the second recording format. This allows detailed information about when the user 30's emotion is heightened to be recorded. In addition, in the robot 40, the processing unit 152 acquires video data in the second recording format from the server 200 and generates a video to be presented to the user 30. Therefore, the user 30 can enjoy information about when their emotion is heightened as a video. The user's emotion can be detected, for example, based on the voice or facial expression detected from the user 30. The switching control unit 260 may also recognize a change in the user 30's emotion based on the detection of an impact by a touch sensor (not shown) included in the sensor unit 156. For example, when a touch sensor included in the sensor unit 156 detects an impact, the switching control unit 260 may recognize that the user 30 is feeling bad.
[0059] When the switching control unit 260 is causing the processing unit 152 to generate information in the second recording format, the switching control unit 260 transmits to the robot 40 an instruction to switch the recording format of the information to be generated by the processing unit 152 from the second recording format to the first recording format in response to a decrease in the intensity of the emotion of the robot 40 or the emotion of the user determined by the parameter processing unit 240. In this way, the switching control unit 260 switches the recording format of the information recorded by the recording control unit 270 from the second recording format to the first recording format in response to a decrease in the intensity of the emotion determined by the parameter processing unit 240 or the emotion of the user.
[0060] Furthermore, when the emotion of the user 30 is transmitted to the robot 40, or when the emotion of the robot 40 is transmitted to the user 30, the switching control unit 260 transmits to the robot 40 an instruction to switch the recording format of the information generated by the processing unit 152 from the first recording format to the second recording format. This allows the information transmitted by the user 30 or the robot 40 to be recorded in detail.
[0061] The switching control unit 260 switches the trigger for switching from the first recording format to the second recording format depending on the situation. For example, the switching control unit 260 switches the detection trigger for switching from the first recording format to the second recording format depending on the situation.
[0062] The switching control unit 260 switches the detection trigger when it detects, based on data such as past conversations, a conversation topic, a keyword or phrase included in the conversation, or a gesture of the user 30 or the robot 40 when the emotional value of the user 30 or the robot 40 exceeds a threshold.
[0063] For example, the switching control unit 260 sets a detection trigger based on past data when the emotional value of the user 30 or the robot 40, or both the user 30 and the robot 40, exceeds a threshold, and when the detection trigger is detected, switches from the first recording format to the second recording format.
[0064] Furthermore, in the robot 40, the processing unit 152 acquires the video data in the second recording format from the server 200 and generates a video to be presented to the user 30. Therefore, the user 30 can respond to the stronger change in emotion.
[0065] When the switching control unit 260 is causing the processing unit 152 to generate information in the second recording format, the switching control unit 260 transmits to the robot 40 an instruction to switch the recording format of the information to be generated by the processing unit 152 from the second recording format to the first recording format in response to the change in the emotion of the robot 40 or the emotion of the user subsiding (the amount of change becoming equal to or less than the threshold) or the emotion of the robot 40 or the emotion of the user returning to the emotion before the change. In this way, the switching control unit 260 switches the recording format of the information recorded by the recording control unit 270 from the second recording format to the first recording format in response to the change in the emotion of the robot 40 or the emotion of the user subsiding.
[0066] Furthermore, when the recording control unit 270 switches the recording format of the information generated by the processing unit 152 from the first recording format to the second recording format, it sends an instruction to the robot 40 to tag the information in the second recording format. This allows the user 30 to clearly recall the situation at the time, etc., from the tagged information when checking the recorded information. Furthermore, tagging makes it easier to find information related to a specific situation from the recorded information.
[0067] The action decision rule 282 defines an action to be performed by the robot 40 in association with the state of the robot 40. For example, the action decision rule 282 defines an action to be performed by the robot 40 in association with the internal state of an artificial neuron of a neural network. For example, the action decision rule 282 defines an action to be performed by the robot 40, such as uttering words expressing happiness, in association with the condition that an emotional artificial neuron defining the emotion of "happy" is high. The action decision rule 282 also defines an action to be performed when the robot becomes sleepy in association with the condition that an internal state of an endocrine artificial neuron defining an endocrine substance corresponding to sleepiness is high.
[0068] Note that endocrine substances refer to substances such as neurotransmitters and hormones that are secreted within the body and transmit signals. Furthermore, endocrine means that endocrine substances are secreted within the body. However, the endocrine substances of the robot 40 itself are one piece of information that affects the behavior of the robot 40, and do not mean that the robot 40 actually produces endocrine substances. Similarly, the emotions of the robot 40 itself are one piece of information that affects the behavior of the robot 40, and do not mean that the robot 40 actually has emotions.
[0069] The action determination unit 250 determines the action of the robot 40 based on the action defined in the action determination rules 282 associated with the activation state or internal state of each artificial neuron determined by the parameter processing unit 240. Action information indicating the action determined by the action determination unit 250 is transmitted from the communication unit 208 to the robot 40. In the robot 40, the processing unit 152 controls the control target 155 to cause the robot 40 to perform the action indicated by the information received from the server 200. This allows the robot 40 to perform an appropriate action according to the current emotion of the robot 40.
[0070] 4 is a schematic diagram of a neural network 300. The neural network 300 is an exemplary neural network for explaining the operation of the parameter processing unit 240. The neural network 300 includes a plurality of artificial neurons, including artificial neuron 1, artificial neuron 2, artificial neuron 3, artificial neuron 4, artificial neuron 5, artificial neuron 6, artificial neuron 7, artificial neuron 8, artificial neuron 9, artificial neuron a, artificial neuron b, and artificial neuron c. The neural network 300 includes a plurality of artificial synapses, including artificial synapse 301, artificial synapse 302, artificial synapse 303, artificial synapse 304, artificial synapse 305, artificial synapse 306, artificial synapse 307, artificial synapse 308, artificial synapse 309, artificial synapse 310, artificial synapse 311, artificial synapse 312, artificial synapse 313, artificial synapse 314, artificial synapse 315, artificial synapse 316, artificial synapse 317, artificial synapse 318, and artificial synapse 319. The artificial neurons correspond to neurons in a living organism. The artificial synapses correspond to synapses in a living organism.
[0071] Artificial synapse 301 connects artificial neuron 4 and artificial neuron 1. Artificial synapse 301 is a unidirectional artificial synapse, as indicated by the arrow of artificial synapse 301. Artificial neuron 4 is an artificial neuron connected to the input of artificial neuron 1. Artificial synapse 302 connects artificial neuron 1 and artificial neuron 2. Artificial synapse 302 is a bidirectional artificial synapse, as indicated by the arrows at both ends of artificial synapse 302. Artificial neuron 1 is an artificial neuron connected to the input of artificial neuron 2. Artificial neuron 2 is an artificial neuron connected to the input of artificial neuron 1.
[0072] In this embodiment, an artificial neuron may be represented by N, and an artificial synapse may be represented by S. When identifying each artificial neuron, a superscripted reference symbol may be used as an identification character. When identifying an arbitrary artificial neuron, i or j may be used as an identification character. For example, Ni represents an arbitrary artificial neuron.
[0073] An artificial synapse may also be identified using the respective identification numbers i and j of the two artificial neurons connected to it. For example, S 41 is N 1 and N 4 In general, S represents an artificial synapse that connects ij is N i The output of N j represents an artificial synapse input to S ji is N j The output of N i represents an artificial synapse input to
[0074] In FIG. 4, A to J indicate the defined states of the robot 40. The states of the robot 40 include the emotions of the robot 40, the state of production of endocrine substances, the situation of the robot 40, etc. 4 , N 6 , and N 7 is a conceptual artificial neuron in which a concept representing the situation of the robot 40 is defined. For example, N 4 is a conceptual artificial neuron that has been assigned the situation "the bell rang." 6 is a conceptual artificial neuron that has been assigned the status "charging has started." 7 is a conceptual artificial neuron to which the state of "charge level is below the threshold" is assigned.
[0075] N 1 , N 3 , N b and N c is an emotional artificial neuron in which the emotion of the robot 40 is defined. 1 is an emotional artificial neuron to which the emotion "happy" is assigned. 3 is an emotional artificial neuron to which the emotion "sad" is assigned. b is an emotional artificial neuron to which the emotion "fear" is assigned. c is an emotional artificial neuron to which the emotion "fun" is assigned.
[0076] N 2 , N 5 and N a is an endocrine artificial neuron in which the endocrine state of the robot 40 is defined. 5 is an endocrine artificial neuron assigned the developmental state of dopamine. Dopamine is an example of an endocrine substance involved in the reward system. 5 is an example of an endocrine artificial neuron involved in the reward system. 2 is an endocrine artificial neuron to which the generation state of serotonin is assigned. Serotonin is an example of an endocrine substance involved in the sleep system. 2 is an example of an endocrine artificial neuron involved in the sleep system. a is an endocrine artificial neuron assigned the generation state of noradrenaline. Noradrenaline is an example of an endocrine substance involved in the sympathetic nervous system. a is an endocrine artificial neuron involved in the sympathetic nervous system.
[0077] The definition information 284 in the storage unit 172 stores information defining the state of the robot 40 as described above for each of the multiple artificial neurons that make up the neural network. In this way, the neural network 300 includes conceptual artificial neurons, emotional artificial neurons, and endocrine artificial neurons. Conceptual artificial neurons, emotional artificial neurons, and endocrine artificial neurons are artificial neurons in which the meanings of concepts, emotions, endocrine, etc. are explicitly defined. In contrast, N 8 YaN 9 is an artificial neuron whose state is not defined for the robot 40. 8 YaN 9 is an artificial neuron whose meanings, such as concepts, emotions, and endocrine secretions, are not explicitly defined.
[0078] The parameters of the neural network 300 are i I, which is the input to t i and N from outside the neural network i E, which is the input to ti And, N i The parameters of and S i and the parameters of
[0079] N i The parameters of N i S indicates the status of t i And, N i represents the internal state of the artificial neuron. i m t And, N i T represents the threshold for firing i t And, N i represents the final firing time, t f and the artificial neuron N at the final firing time i V represents the internal state of i m tf and the output increase / decrease parameter a t i , b t i , h t i The output increase / decrease parameter is an example of a parameter that determines the time evolution of the output when the artificial neuron fires. In this embodiment, the subscript t indicates that the parameter can be updated as time progresses. i m t is information corresponding to the membrane potential of the artificial neuron, and is an example of a parameter representing the internal state or output of the artificial neuron.
[0080] S ij The parameters of S ij BS represents the coupling coefficient of the artificial synapse t ij And, S. ij N connected to i and N j The time when the two fire simultaneously is the last time that they fired simultaneously. cf and BS, which represents the coupling coefficient at the time of the final simultaneous firing. ij tcf and a, which is a parameter for increasing or decreasing the coupling coefficient t ij , b tij , h t ij The coupling coefficient increase / decrease parameter is an example of a parameter that determines the time evolution of the coupling coefficient after the two artificial neurons connected by the artificial synapse last fired simultaneously.
[0081] The parameter processing unit 240 updates the above parameters based on inputs from the external input data generation unit 230 and the neural network to determine the activation state of each artificial neuron. The action determination unit 250 determines the action of the robot 40 based on the internal states or activation states of at least some of the artificial neurons in the neural network, which are determined by the parameter values of at least some of the artificial neurons, and the states defined for at least some of the artificial neurons by the definition information 284. Note that the activation state can be either an activated state or an inactivated state. In this embodiment, activation may be referred to as "firing," and inactivation may be referred to as "non-firing." Note that, as described below, the "firing" state is divided into an "upward phase" and a "downward phase" depending on whether the internal state is rising. "Non-firing," "upward phase," and "downward phase" are represented by the status Sti.
[0082] 5 shows the neural network parameters in a table format. Each neuron N has a threshold T t and the increase / decrease parameter h t , a t and b t Each artificial synapse has a coupling coefficient BS t and the increase / decrease parameter h t , a t and b t and as parameters. i Every time, N i Each parameter of all the artificial neurons directly connected to the corresponding artificial synapse is shown in a single line.
[0083] 6 shows an outline of the operation flow of the server 200 when the robot 40 is activated or reset. When the server 200 receives a notification that the robot 40 has been activated or reset, the parameter processing unit 240 initializes the parameters of the neural network. For example, the parameter processing unit 240 acquires the initial values of the parameters from the storage unit 172 and generates parameter data of the neural network in a predetermined data structure (S502). Also, at time t 0 When the initial setting is completed, a loop for time t is started in S504.
[0084] In S510, the parameter processing unit 240 calculates the time step t n+1 Calculate the parameters corresponding to the changes due to the electrical influence of the artificial synapse in S. Specifically, t ij Calculate.
[0085] In S520, the parameter processing unit 240 calculates the time step t n+1 Calculate the parameters corresponding to the changes due to the chemical influence of the endocrine substance in the i and S ij More specifically, we calculate the change in the parameters of n+1 Artificial neuron N affected by endocrine artificial neuron i The internal state of the S ij Calculate the coupling coefficient increase / decrease parameters and coupling coefficients.
[0086] In S530, the parameter processing unit 240 acquires an input from outside the neural network. Specifically, the parameter processing unit 240 acquires the output of the external input data generation unit 230.
[0087] In S540, the parameter processing unit 240 calculates the time step t n+1 In i Specifically, we calculate the internal state of V i mtn+1 and Status S tt i Then, in S550, the time t n+1 The values of the parameters at time t are stored in the parameter 288 of the storage unit 172. n+1 The values of the parameters in the above are output to the operation determination unit 250 and the switching control unit 260.
[0088] In S560, the switching control unit 260 n+1 N in i It is determined whether the parameters satisfy the conditions for switching the recording format of the data stored in the recording data 292. n+1 N in i If the parameters satisfy the conditions for switching the recording format, the switching control unit 260 instructs the robot 40 to switch the recording format (S570). Next, when switching from the first recording format to the second recording format, the recording control unit 270 instructs the robot 40 to tag information in the second recording format (S580), and the process proceeds to S506. Meanwhile, in S560, n+1 N in i If the parameters do not satisfy the conditions for switching the recording format, the process proceeds to S506.
[0089] In S506, the parameter processing unit 240 determines whether to end the loop. For example, it determines to end the loop when the time represented by the time step reaches a predetermined time or when sensor information from the robot 40 has not been received for a predetermined time. If the loop is not to be ended, the process returns to S510 and the next time step is calculated. If the loop is to be ended, this flow ends.
[0090] 7 is a diagram for explaining the calculation of the coupling coefficient of the artificial synapse. Here, the constant a is used as the initial value of the increase / decrease parameter. ij and b ij This section explains the case where is defined.
[0091] Time t n At the time step S ij N on both ends of i and N jare both firing, the parameter processing unit 240 n+1 in B.S. tn+1 ij ,BS tn+1 ij =BS tn ij +a tn ij × (t n+1 -t n ) is calculated. On the other hand, at time t n At the time step of S i and S j If none of the above fires at time t n+1 Coupling coefficient BS tn+1 ij ,BS tn+1 ij =BS tn ij +b tn ij × (t n+1 -t n ) is calculated. Also, BS tn+1 ij If is negative, then BS tn+1 ij is set to 0. Note that BS ij is a positive value of S ij So, a t ij is a positive value, and b t ij is a negative value. ij is a negative value of S ij So, a t ij is a positive value, and b t ij is a negative value.
[0092] As shown in FIG. 7, at time t 0 Since the artificial neurons on both ends fire simultaneously, t ij is a per unit time t0 ij Also, at time t 1 Since they do not fire simultaneously, BS t ij is |b per unit time t1 ij| Also, at time t 4 Due to simultaneous fires at BS t ij is a per unit time t4 ij increases with.
[0093] FIG. 8 shows the function h t ij The time evolution of the coupling coefficient when is defined is shown schematically. t ij is t cf The elapsed time Δt from cf ) ≧0. t ij is a function of at least Δt and takes on real values.
[0094] The function 700 shown in FIG. t ij The function 700 is an example of cf Coupling coefficient BS tcf ij and Δt. The function 700 monotonically increases when Δt is less than a predetermined value, and monotonically decreases and gradually decreases toward 0 when Δt is greater than the predetermined value. The function 700 has a value BS at Δt=0. tcf ij Take.
[0095] FIG. 8 shows a function 700 defined as a parameter for increasing or decreasing the coupling coefficient, and 0 In i and N j The parameter processing unit 240 calculates the coupling coefficient at time t based on the function 700 and Δt. 1 ~Time t 6 BS at each time t ij Calculate at time t 1 ~Time t 6 Within the time range of i and N j are not firing simultaneously. 2 Thereafter, the coupling coefficient decreases monotonically.
[0096] FIG. 9 shows the time t2 And N i and N j The graph shows the time evolution of the coupling coefficient when the neurons fire simultaneously. 0 From time t 2 The calculation is the same as in FIG. 2 In N i and N j When these two fire simultaneously, the parameter processing unit 240 t ij (t-t 2 , B.S. t2 ij ) at time t 3 ~t 6 The coupling coefficient is calculated for each time point. In this way, the coupling coefficient increases each time simultaneous firing occurs. This has the effect of strengthening the artificial synaptic connection, similar to Hebb's law in living organisms. On the other hand, as shown in Figures 7 and 8, when the time without simultaneous firing becomes long, the effect of weakening the artificial synaptic connection is obtained.
[0097] 10 is a schematic diagram of effect definition information that defines the chemical effect applied to a parameter. This effect definition information is used to calculate the change in the parameter in S520 of FIG. 6. The definition information includes conditions related to the internal state of the endocrine artificial neuron, information specifying the influencing artificial neuron or artificial synapse, and an equation that defines the effect content.
[0098] In the example of FIG. 10, the endocrine artificial neuron N 2 is an endocrine artificial neuron to which the endocrine substance responsible for sleepiness is assigned. 2 The definition information regarding "Vm tn 2 >T tn 2 "Conditions for Endocrine Artificial Neuron N 2 The artificial neuron that is affected by the 1 and N 3 ", and "T" is the formula that determines the impact content. tn+1 i =T tn iParameter processing unit 240 determines Vm tn 2 T tn 2 If it exceeds t n+1 Emotional Artificial Neuron N 1 and N 3 The threshold is increased by 10%. This makes it difficult for the emotional artificial neuron to fire when, for example, drowsiness occurs. For example, the conceptual artificial neuron N is defined as "the amount of stored power is equal to or less than the threshold." 7 The output of the endocrine artificial neuron N 2 By defining a neural network connected to the input of the above, it becomes possible to embody the phenomenon in which emotions become less likely to rise when the battery level is low.
[0099] In addition, the endocrine artificial neuron N 5 is an endocrine artificial neuron to which dopamine is assigned. 5 The first definition information regarding "Vm tn 5 >T tn 5 and Vm tn 4 >T tn 4 "Conditions for Endocrine Artificial Neuron N 5 As an artificial synapse that is affected by 49 and S 95 ", and "a" as the formula for determining the impact content. tn+1 ij = a tn ij The parameter processing unit 240 calculates Vm tn 5 T tn 5 and Vm tn 4 T tn 4 If it exceeds t n+1 Artificial Synapse S 49 and S 95 Increases the increase / decrease parameter by 10%.
[0100] This allows the concept artificial neuron N to be generated, which defines the situation of "the bell has rung" when the endocrine artificial neuron in the reward system fires. 4 is firing, the implicit artificial neuron N 9 Conceptual artificial neuron N via 4 and N 5 This allows the endocrine artificial neuron N of the reward system to be activated when the "bell" is rung. 5 becomes more likely to ignite.
[0101] In addition, the endocrine artificial neuron N 5 The second definition information regarding "Vm tn 5 >T tn 5 "Conditions for Endocrine Artificial Neuron N 5 As an artificial neuron that is influenced by 1 ", and "T" is the formula that determines the impact content. tn+1 i =T tn i The parameter processing unit 240 calculates Vm tn 5 T tn 5 If it exceeds t n+1 Artificial neuron N 1 This reduces the increase / decrease parameter of the reward system endocrine artificial neuron N 5 When this emotion is ignited, it becomes easier for the emotion of happiness to be ignited.
[0102] According to this definition of the effect of the reward system on the endocrine artificial neuron, it is possible to implement an implementation in which, if the action of charging the robot 40 while ringing a bell is repeated, the robot 40 will behave in a way that expresses happiness just by ringing the bell.
[0103] The influence definition information is not limited to the example shown in FIG. 10 . For example, a condition may be defined that the internal state of the artificial neuron is equal to or less than a threshold. A condition related to the status of the artificial neuron, such as an ascending phase, a descending phase, or non-firing, may also be defined. The influence range can be defined by directly specifying an artificial neuron or an artificial synapse, or by defining "all artificial synapses connected to a specific artificial neuron." Regarding the influence formula, when the target is an artificial neuron, in addition to multiplying the threshold by a constant, a formula may be defined that adds a constant to the threshold or multiplies the increase / decrease parameter of the internal state by a constant. When the target is an artificial synapse, in addition to multiplying the increase / decrease parameter by a constant, a formula may be defined that multiplies the coupling coefficient by a constant.
[0104] The influence definition information is stored in the definition information 284 of the storage unit 172. In this way, the storage unit 172 stores influence definition information that defines the influence of at least one of the internal state and firing state of an endocrine artificial neuron on the parameters of at least one of other artificial neurons and artificial synapses that are not directly connected to the endocrine artificial neuron via an artificial synapse. The parameter processing unit 240 then updates the parameters of at least one of other artificial neurons and artificial synapses that are not directly connected to the endocrine artificial neuron via an artificial synapse based on at least one of the internal state and firing state of the endocrine artificial neuron and the influence definition information. Furthermore, the parameters of other artificial neurons that are influenced by at least one of the internal state and firing state of the endocrine artificial neuron can include at least one of the parameters that determine the threshold, firing state, and time evolution of the output at the time of firing of the other artificial neuron. Furthermore, the parameters of the artificial synapse influenced by at least one of the internal state and firing state of the endocrine artificial neuron can include at least one of the coupling coefficient of the artificial synapse and a parameter that determines the time evolution of the coupling coefficient after the last simultaneous firing of the two artificial neurons connected by the artificial synapse. Furthermore, the influence definition information includes information that determines the influence that the firing state of the endocrine artificial neuron associated with the reward system has on the threshold of the emotional artificial neuron, and the parameter processing unit 240 updates the threshold of the emotional artificial neuron in accordance with the influence definition information when the endocrine artificial neuron fires.
[0105] FIG. 11 shows the V tn+1 i and S tn+1 i 6. The process of this flowchart can be applied to part of the process in S540 in FIG. 6. In S1100, the parameter processing unit 240 calculates S tn i indicates non-firing.
[0106] S tn i indicates non-firing, the parameter processing unit 240 i Input I to tn+1 iSpecifically, when an input from outside the neural network is N i If not connected to tn+1 i =Σ j B.S. tn+1 ji ×Vm tn j ×f(S tn j ) is calculated by the input from outside the neural network. i If connected to tn+1 i =Σ j B.S. tn+1 ji × Vm tn j ×f(S tn j ) + E tn+1 i Here, E tn i is the time t from outside the neural network n This is the input in
[0107] Furthermore, f(S) returns 0 when S is a value representing no firing, and returns 1 when S is a value representing the rising or falling phase. This model corresponds to a model in which a synapse transmits an action potential only when a neuron fires. Note that f(S) = 1 may also be returned. This corresponds to a model in which a membrane potential is transmitted regardless of the firing state of the neuron.
[0108] In S1112, the parameter processing unit 240 tn+1 i T tn+1 i Determine whether it exceeds I. tn+1 i T tn+1 i If Vm tn+1 i is calculated based on the increase / decrease parameters, and Vm tn+1 i Depending on S tn+1 iis set to a value indicating the rising phase or the falling phase (S1114), and this flow ends.
[0109] In S1100, S tn i is in the rising phase or the falling phase, the parameter processing unit 240 tn+1 i (S1120). Then, the parameter processing unit 240 calculates t n+1 By Vm t i When Vmin is reached, S tn+1 i Set t to the unfired value, n+1 By Vm t i If Vmin has not been reached, S tn+1 i The parameter processing unit 240 sets t to the value of the rising phase or the falling phase, and then ends this flow. n+i By Vm t i When Vmax is reached, S tn+1 i Set the value of the falling phase to t n+1 By Vm t i If Vmax is not reached, S tn+1 i Set the value of the rising phase to
[0110] In this way, N i If N is firing, even if the output falls below the threshold, i The output of is independent of the input. Such a period corresponds to the absolute refractory period in biological neurons.
[0111] FIG. 12 shows the N i V when does not ignite t i FIG. 10 is a diagram for explaining an example of calculation of
[0112] Time t 0 At time steps of N i is not fired. 1 I t1 i T t1 iIf the time t 1 V in t1 i V t1 i =I t1 i At time t 0 From t 1 V for the period up to t i V t i =I t0 i Similarly, the parameter processing unit 240 calculates the time step t n V calculated by t n The value of is maintained until the next time step, and V tn+1 In tn+1 Change it to.
[0113] FIG. 13 shows the N i V when ignition occurs i t 13 is a diagram for explaining an example of calculation of the constant a i and b i This is a calculation example when is defined.
[0114] Time t 0 At time steps N i is not fired. 1 I t1 i T t1 i If the time t 1 V in t1 i V t1 i =I t1 i At time t 0 From t 1 V for the period up to t i V t i =I t0 i Here, the calculation is performed at time t 1 I t1i is equal to or less than Vmax. 1 I t1 i If exceeds Vmax, I t1 i = Vmax.
[0115] As shown in FIG. 13, the parameter processing unit 240 1 From then on, V t i Until the time when V reaches Vmax, V t i per unit time t ij The parameter processing unit 240 also increases the N i Status S t i is determined to be the ascending phase.
[0116] Also, V t i When V reaches Vmax, V t i Until V reaches Vmin, V t i per unit time |b t i In addition, the parameter processing unit 240 decreases N i The status of V is determined to be in the descending phase. t i When V reaches Vmin, V at the next time t6 i V t6 i =I t6 i It is calculated by V t i After Vmin is reached, the status is determined to be non-fired.
[0117] In addition, N i If the status of is in a falling phase, the calculated Vm t i T t i Even if it falls below Vm t i is I t iThe parameter processing unit 240 does not depend on Vm t i T t i Even if it falls below Vm t i Vm according to the increase / decrease parameters until it reaches Vmin t i Calculate.
[0118] FIG. 14 shows the N i The function h t i The time evolution of the coupling coefficient when is defined is shown below. t i is the firing time t f The elapsed time Δt from f ) ≧0. t i is a function of at least Δt. t i takes a real value, and h t i The value range is equal to or greater than Vmin and equal to or less than Vmax.
[0119] The function 1300 shown in FIG. t i The function 1300 is an example of f Vm in tf i and Δt. The function 1300 monotonically increases when Δt is less than a predetermined value, and monotonically decreases when Δt is greater than the predetermined value. The function 1300 has a value Vm at Δt=0. tf i Take.
[0120] FIG. 14 shows a function 1400 defined as an increase / decrease parameter of the internal state, and the time t 1 In N i The parameter processing unit 240 calculates the output when the function 1400, Δt, and Vm f i Based on this, at time t 1 Calculate Vmti at each time from time t to time t5. t i is time t 5Since Vmin is reached at time t 6 So Vm i =I t6 i This becomes:
[0121] 15 shows an example of a rule 1400 stored in the recording format switching rule 290 in table format. 1 , N 3 , N b and N c When at least the first condition that Vmti of any of the above exceeds the threshold is satisfied, the operation of "switching the information recording format to a low compression format" is defined. As a result, when information is recorded in a high compression format, the switching control unit 260 determines to switch the information recording format to a low compression format when the first condition is satisfied after not being satisfied. Note that the thresholds are set to N j The threshold value may be higher than Tit.
[0122] Also, in rule 1400, N 5 and N a Vm t i When at least the second condition that the sum of the values of N and N exceeds the threshold is satisfied, the operation of "switching the data recording format to a low compression format" is defined. As a result, when information is recorded in a high compression format, the switching control unit 260 determines to switch the information recording format to a low compression format when the second condition is satisfied after not being satisfied. Note that as the threshold, each N j The threshold value is calculated by multiplying the total value of Vmax of each N by a constant 0.9. j T i t It may be higher than the total value of
[0123] N 1 , N 3 , N b and N care emotion artificial neurons in which the emotions of "happy," "sad," "fearful," and "fun" are defined, respectively. Therefore, the parameter processing unit 240 determines the intensity of the emotion based on the internal state of the emotion artificial neuron, and when the determined intensity of the emotion exceeds a predetermined threshold, the recording format can be switched to a low-compression format.
[0124] N5 and Na are endocrine artificial neurons for which the endocrine substances "dopamine" and "noradrenaline" are defined, respectively. The sum of the parameters of the internal states of these endocrine artificial neurons is an example of an index representing the intensity of the emotion "excitement." Therefore, the parameter processing unit 240 determines the intensity of the emotion based on the internal state of the endocrine artificial neuron, and when the determined intensity of the emotion exceeds a predetermined threshold, the recording format can be switched to a low-compression format.
[0125] Also, in rule 1400, N 1 , N 3 , N b and N c Any Vm t i is also equal to or less than the first threshold, and N 5 and N a Vm t i When a third condition is satisfied, that is, the sum of the values of the first and second thresholds is equal to or less than the second threshold, the switching control unit 260 determines to switch the data recording format to a high-compression format when the third condition is satisfied after the third condition is no longer satisfied, in a case where information is recorded in a low-compression format. In this way, the switching control unit 260 can switch the recording format to a high-compression format when the intensity of the emotion falls below a predetermined threshold.
[0126] The first threshold value of the third condition is j The second threshold value of the third condition is the value obtained by multiplying the Vmax of each N jThe value is obtained by multiplying the total value of Vmax of the above conditions by a constant 0.8. In this way, the case where the first threshold value of the third condition is smaller than the threshold value of the first condition, and the second threshold value of the third condition is smaller than the threshold value of the second condition is exemplified. However, the first threshold value may be the same as the threshold value of the first condition, and the second threshold value may be the same as the threshold value of the second condition. Furthermore, the first threshold value of the third condition is set to a value obtained by multiplying the total value of Vmax of the above conditions by a constant 0.8. j T i t The second threshold value of the third condition may be higher than the respective N j T i t The threshold values for each condition are not limited to these examples, and various values can be applied.
[0127] According to the system 20, the robot 40 continuously transmits highly compressed information, such as skeletal data, to the server 200 and records it there while the robot 40 is not in a significantly emotional state. The continuous information, such as skeletal data, recorded in the server 200 can be used to analyze the robot 40's memories. When the robot 40's emotions become significantly elevated, the robot 40 starts transmitting full HD video data and audio data, and while the robot 40's emotions remain strong above a certain level, the robot 40 records low-compression information, including the full HD video data and audio data, in addition to the skeletal data, in the server 200. When the user 30 requests the robot 40 to provide a video of the robot 40's memories, the robot 40 requests the server 200 to transmit the full HD video data and audio data, and provides the video data and audio data received from the server 200 to the user 30.
[0128] In this way, according to the system 20, high-quality video data of scenes in which the robot 40 feels strong emotions can be stored in the server 200. On the other hand, when the robot 40 does not feel strong emotions, simplified information such as skeletal data can be stored in the server 200. In this way, the robot 40, like a human, can retain vivid memories of times when it felt strong emotions, while retaining simplified memories of times when it did not feel strong emotions.
[0129] In this embodiment, emotions such as "happy," "sad," "fearful," "fun," and "excited" have been described, but the emotions handled by the system 20 are not limited to these. In addition, in this embodiment, endocrine substances such as "dopamine," "serotonin," and "noradrenaline" have been described, but the endocrine substances handled by the system 20 are not limited to these.
[0130] The functions of the server 200 may be implemented by one or more computers. At least some of the functions of the server 200 may be implemented by a virtual machine. At least some of the functions of the server 200 may be implemented in the cloud. The functions of the components of the server 200, excluding the storage unit 172, may be realized by a CPU operating based on a program. For example, at least some of the processes described as the operation of the server 200 may be realized by a processor controlling each piece of hardware (e.g., a hard disk, memory, etc.) of the computer in accordance with a program. In this way, at least some of the processes of the server 200 may be realized by the processor operating based on a program to control each piece of hardware, thereby enabling each piece of hardware, including the processor, hard disk, memory, etc., to operate in cooperation with the program. In other words, the program can cause the computer to function as each component of the server 200. Similarly, the functions of the components of the robot 40, excluding the control target 155 and the sensor unit 156, may be realized by a CPU operating based on a program. In other words, the program can cause the computer to function as each component of the robot 40. The computer may load a program that controls the execution of the above-described processes and operate in accordance with the loaded program to execute the processes. The computer may load the program from a computer-readable recording medium that stores the program. The program may also be supplied to the computer via a communication line, and the computer may load the program supplied via the communication line.
[0131] In the embodiment described above, the server 200 separate from the robot 40 is responsible for neural network processing. Also, the server 200 separate from the robot 40 stores information such as video data. However, the robot 40 itself may perform the functions of the server 200, such as neural network processing. Also, the robot 40 itself may store information such as video data. Also, the robot 40 is an example of a device that is to be controlled by the server 200. The device to be controlled is not limited to the robot 40, and various devices such as home appliances, vehicles, toys, etc. may be used as the controlled device.
[0132] Second Embodiment Next, a second embodiment will be described. Here, the description will focus on the differences from the first embodiment. Description of the same configuration as the first embodiment will be omitted. FIG. 16 shows an example of a system 20 according to the second embodiment. FIG. 17 is a diagram showing an example of emotional value changes. The system 20 includes a server 200, and robots 40a and 40b. The robots 40a and 40b communicate with the server 200 via a communication network 90 to exchange information.
[0133] The robot 40 has an emotional value that represents its own emotions. For example, the robot 40 has an emotional value that represents the strength of each emotion, such as "happy," "fun," "sad," "fear," and "excitement." The emotional value of the robot 40 is determined by the server 200. The server 200 causes the robot 40 to perform an action according to the determined emotion. For example, when the robot 40 is having a conversation with the user 30 in a state where the emotional value of excitement is high, the server 200 causes the robot 40 to speak quickly. In this way, the robot 40 can express its own emotions through actions, etc.
[0134] The server 200 uses a neural network to update the current state of the robot 40 based on the detection information received from the robot 40. The state of the robot 40 includes the emotion of the robot 40. Therefore, the server 200 uses the neural network to determine the emotion of the robot 40.
[0135] The robot 40 also acquires various types of information related to the user 30. For example, the robot 40 uses a camera function to take a photo and acquire video of the user 30. The robot 40 also acquires various types of information related to the surroundings using sensors provided in the robot 40. For example, the robot 40 acquires surrounding information such as information about objects present in the surroundings, as well as smells, temperature, humidity, location information, weather, the number of people in the surroundings, and their expressions. The robot 40 also detects events from various types of information related to the user 30 and the surroundings.
[0136] The server 200 stores various pieces of information about the user 30 and the surroundings received from the robot 40 in association with the emotions of the robot 40. For example, the server 200 stores various pieces of information about the user 30 and the surroundings by tagging them with information about the emotional value of the robot 40. This allows the server 200 to record the emotions at the time the information was acquired in association with the various pieces of information about the user 30 and the surroundings.
[0137] Furthermore, when an event is detected from various information about the user 30 and the surroundings, the server 200 associates the level of emotion of the robot 40 with the event and stores the associated information. The server 200 stores the event information and the emotion value information of the robot 40 by tagging them. This allows the server 200 to record the emotion at the time the information was acquired, in association with the event.
[0138] When outputting stored information from the robot 40, the server 200 controls to change one or both of the behavior of the robot 40 and the timing of outputting the information according to the emotion associated with the information. This allows the user 30 to sense the emotion associated with the information from the behavior and output timing of the robot 40 when outputting the information. In this way, the server 200 can cause the robot 40 to output stored information according to the emotion associated with the information.
[0139] Furthermore, the server 200 controls the storage period of the stored information according to the emotion associated with the information. For example, the server 200 controls the storage period so that information associated with a strong emotion is stored for a long period of time, and information associated with a weak emotion is stored for a short period of time. This allows information that leaves an impression on the user 30 to be stored for a long period of time, and information that leaves a less lasting impression on the user 30 to be deleted quickly. In this way, the server 200 can appropriately store information according to the emotion associated with the information.
[0140] The robot 40 also records video data of the user 30 acquired using the camera function on the server 200. The robot 40 acquires video data from the server 200 as needed and provides it to the user 30. The stronger the robot's emotion, the greater the amount of information the robot generates and records on the server 200. For example, when the robot 40 is recording information in a highly compressed format, such as skeletal data, the robot 40 switches to recording information in a low-compression format, such as HD video, when the robot's excitement level exceeds a threshold. The system 20 allows high-definition video data to be recorded when the robot 40's emotion is heightened. The stronger the user's emotion, the greater the amount of information the robot 40 generates and records on the server 200. Specifically, when the robot 40 is recording information in a highly compressed format, such as skeletal data, the robot 40 switches to recording information in a low-compression format, such as HD video, when the robot's excitement level exceeds a threshold. According to the system 20, high-definition video data of the user 30 when he / she is feeling emotional can be recorded.
[0141] Furthermore, even when the robot 40's or the user's emotion is not strong, as in State 2 of FIG. 17 , if the robot 40's or the user's emotion changes suddenly, the robot 40 generates video data with a large amount of information and records it on the server 200. For example, when the robot 40 is recording information in a highly compressed format, such as skeletal data, the robot 40 may switch to recording information in a low-compression format, such as HD video, in response to a change in the emotion value exceeding a threshold, even if the emotion value of the robot 40's or the user's excitement is below the threshold. The system 20 can record high-definition video data when the emotion of the robot 40 or the user 30 changes drastically. The robot 40 may also change the amount of information in the video data depending on the degree of change in emotion. Specifically, the robot 40 may lower the compression rate of the video data as the degree of change in the robot's or the user's emotion increases. As another example, the robot 40 may change the format of the recorded data in ascending order of the degree of change in the robot's or the user's emotion, such as text only → image only → audio and image → low-quality video → high-quality video. Furthermore, the robot 40 may change the length or amount of video data, not just the compression rate or information format, so that when the emotions of the robot 40 or the user 30 change more dramatically, higher-definition video data can be recorded.
[0142] Furthermore, the robot 40 tags the information it records when its own or the robot 40's emotional value exceeds a threshold or when a change in the emotional value exceeds a threshold. Specifically, the robot 40 tags the video data with information detected by sensors provided in the robot 40 and information on the user's or the robot's own emotional value. The tagged information includes, for example, surrounding information (such as information on surrounding objects), smells, temperature, humidity, location information, weather, the number of people in the vicinity, their expressions, and emotions. This allows the user 30 to clearly recall the situation at the time, etc., from the tagged information when reviewing the recorded information. Furthermore, tagging makes it easier to find information related to a specific situation from the recorded information.
[0143] Furthermore, even when the robot 40's or the user's emotion is not strong, as in State 2 of FIG. 17 , if the robot 40's or the user's emotion changes suddenly, the robot 40 generates video data with a large amount of information and records it on the server 200. For example, when the robot 40 is recording information in a highly compressed format, such as skeletal data, the robot 40 may switch to recording information in a low-compression format, such as HD video, in response to a change in the emotion value exceeding a threshold, even if the emotion value of the robot 40's or the user's excitement is below the threshold. The system 20 can record high-definition video data when the emotion of the robot 40 or the user 30 changes drastically. The robot 40 may also change the amount of information in the video data depending on the degree of change in emotion. Specifically, the robot 40 may lower the compression rate of the video data as the degree of change in the robot's or the user's emotion increases. As another example, the robot 40 may change the format of the recorded data in ascending order of the degree of change in the robot's or the user's emotion, such as text only → image only → audio and image → low-quality video → high-quality video. Furthermore, the robot 40 may change the length or amount of video data, not just the compression rate or information format, so that when the emotions of the robot 40 or the user 30 change more dramatically, higher-definition video data can be recorded.
[0144] Furthermore, the robot 40 may change the information to be stored according to the continuity of changes in the emotions of the robot 40 or the user 30, as in State 1 in FIG. 17 . For example, when the cumulative emotional value within a predetermined time period exceeds a threshold, the robot 40 generates video data with a large amount of information and records it in the server 200. The robot 40 may also change the information to be stored according to the combination of the speed and speed of changes (gradients) in emotions within a predetermined time period or the degree of the gradient. For example, the robot 40 generates video data with a large amount of information and records it in the server 200 when the gradient indicating the change in emotional value within a predetermined time period is a sudden rise, followed by a gradual decline for a certain period of time. This makes it possible to record video data of a situation where, for example, the absolute value of the emotion is not relatively large but the duration feels long (the emotional value continues for a certain period of time above a certain value), such as in a tense situation due to pressure.
[0145] Furthermore, the robot 40 updates a previously determined emotion associated with previously detected information that includes elements common to the newly detected information based on the newly determined emotion, based on elements related to the newly determined emotion. For example, when the strength of the newly determined emotion exceeds a predetermined threshold, the robot 40 updates a previously determined emotion associated with previously detected information that includes elements common to the newly detected information, based on elements related to the newly determined emotion. This makes it possible to generate a variety of emotions by taking into account changes in the relationship with the target and using the determined emotion strength in accordance with the situations of people and events learned in the past and the current situation.
[0146] Furthermore, for example, when the type of newly determined emotion changes in a predetermined manner, the robot 40 updates the previously determined emotion associated with previously detected information that includes elements common to the newly detected information, based on elements related to the newly determined emotion. This makes it possible to generate a variety of emotions by taking into account changes in the relationship with the target and using changes in the determined emotion in accordance with the situations of people and events learned in the past and the current situation.
[0147] Furthermore, for example, the robot 40 updates a previously determined emotion to an emotion that corresponds to the strength of a newly determined emotion. This enables the robot 40 to update emotions according to the strength of the determined emotion, take into account changes in the relationship with the target, and generate a variety of emotions.
[0148] Furthermore, for example, the robot 40 updates a previously determined emotion to an emotion corresponding to the amount of information stored in association with a newly determined emotion. This allows the emotion to be updated according to the amount of information stored in association with the determined emotion, taking into account changes in the relationship with the target, and makes it possible to generate a variety of emotions.
[0149] Furthermore, for example, the robot 40 updates a previously determined emotion to an emotion that corresponds to the time when the previously detected information was stored, and that includes elements that are associated with the newly determined emotion and that are common to the newly detected information. This allows the robot 40 to update emotions according to the time when the information to be updated was stored, and to generate a variety of emotions by taking into account changes in the relationship with the target.
[0150] Furthermore, for example, the robot 40 can perform update processing by combining the strength and type of a newly determined emotion. This enables the robot 40 to generate a variety of emotions by taking into account changes in the relationship with the target and using the determined change and strength of the emotion in accordance with the situation of people and events learned in the past and the current situation.
[0151] Furthermore, for example, information detected by the sensors provided on the robot 40 includes elements such as surrounding information, location information, date and time, weather, temperature, scenery, smells, sounds, the number of people in the vicinity, people, their facial expressions, and people's words and actions. This makes it possible to identify commonalities between the situations of people and events learned in the past and the current situation, and then use the determined emotions to generate a variety of emotions, taking into account changes in the relationship with the target.
[0152] Fig. 18 is a schematic block diagram of the server 200 and the robot 40. In Fig. 18, processing related to the processing unit 152 and the processing unit 202 will be mainly described.
[0153] The processing unit 152 detects an event from various information about the user 30 and the surroundings detected by the sensors of the sensor unit 156. When an event is detected, the processing unit 152 transmits event information about the detected event to the server 200 via the communication unit 158. Note that the event may be detected by the processing unit 202 of the server 200.
[0154] The communication unit 158 transmits the information in the first recording format or the information in the second recording format generated by the processing unit 152 to the server 200. Furthermore, when an event is detected, the communication unit 158 transmits event information related to the detected event to the server 200.
[0155] Furthermore, in the robot 40, the communication unit 158 acquires information stored in the recording data 292 from the server 200. The communication unit 158 functions as a recording information receiving unit that acquires information in the second recording format, including video data, that has been recorded by the recording control unit 270. The processing unit 152 generates video to be presented to the user 30, based on the video data included in the information in the second recording format received by the communication unit 158. The processing unit 152 functions as an image generating unit that generates video to be presented to the user 30.
[0156] Furthermore, the communication unit 158 receives operation information indicating operation content from the server 200. The processing unit 152 controls the control target 155 based on the operation content received by the communication unit 158. The control target 155 includes a speaker, motors that drive each part of the robot 40 such as the limbs, a light-emitting device, etc. When information indicating the vocal content is received from the server 200, the processing unit 152 outputs sound from the speaker in accordance with the received vocal content. Furthermore, the processing unit 152 can control some of the behaviors of the robot 40 by controlling the drive motors of the limbs. Furthermore, the processing unit 152 can express some of the emotions of the robot 40 by controlling these motors.
[0157] In the server 200 , the communication unit 208 outputs the information received from the robot 40 to the processing unit 202 .
[0158] In the processing unit 202, the initial value setting unit 210 stores the initial values of the parameters indicating the initial state of the neural network received by the communication unit 208 in the parameter initial value 286 in the storage unit 172. Note that the initial values of the neural network parameters may be predetermined in the server 200, or may be changeable by the user 30 via the communication network 90.
[0159] The external input data generation unit 230 processes at least a portion of the sensor information received by the communication unit 208 to generate input information from outside the neural network, and outputs the generated information to the parameter processing unit 240. The parameter processing unit 240 performs calculations on the neural network based on the input information and the current parameters 288 and definition information 284 of the neural network stored in the storage unit 172.
[0160] The parameter processing unit 240 calculates parameters representing the internal states of multiple artificial neurons in the neural network based on the input information generated by the external input data generation unit 230. For example, the parameter processing unit 240 updates the current internal state parameters of multiple artificial neurons, etc., that define the status of the robot 40, based on the input information generated by the external input data generation unit 230. The parameter processing unit 240 also calculates the internal state parameters of other artificial neurons in the neural network. This calculates the internal state parameters of an emotional artificial neuron that defines, for example, the emotion "happy." The internal state parameters of this emotional artificial neuron are an example of an index representing the intensity of the emotion "happy." Therefore, the parameter processing unit 240 can determine the intensity of an emotion in the control system based on the internal state of the emotional artificial neuron. In this way, the parameter processing unit 240 functions as an emotion determination unit that determines an emotion using a neural network based on at least a portion of the information detected by the sensors in the sensor unit 156. For example, the parameter processing unit 240 determines the type and intensity of an emotion based on at least a portion of the information detected by the sensors in the sensor unit 156.
[0161] Furthermore, the parameter processing unit 240 determines an emotion based on information newly detected by a sensor included in the sensor unit 156 provided in the robot 40 and an emotion previously determined by the parameter processing unit 240 that is associated with information previously detected by a sensor included in the sensor unit 156 provided in the robot 40. For example, based on the information previously detected by the sensor included in the sensor unit 156 provided in the robot 40, such as "Person A" and "patted on the head," and the emotion previously determined by the parameter processing unit 240, such as "happy," the parameter processing unit 240 can newly determine an emotion similar to the previously determined emotion of "happy" for the information newly detected by the sensor included in the sensor unit 156 provided in the robot 40.
[0162] That is, when newly detected information includes a specific element (e.g., "Person A"), the parameter processing unit 240 determines the emotion for the newly detected information based on the emotion associated with the specific element. In other words, the parameter processing unit 240 suppresses, to some extent, the strong expression of emotions that are completely opposite to emotions that occurred in the past in a situation that has something in common with a situation experienced in the past. This makes it possible to generate a variety of emotions according to the information on people and events learned in the past and the current situation.
[0163] Furthermore, the parameter processing unit 240 determines the emotion using the emotion information updated by the recording control unit 270. For example, the parameter processing unit 240 determines the emotion based on information newly detected by the sensor of the sensor unit 156 included in the robot 40 and the emotion updated by the recording control unit 270 and associated with information previously detected by the sensor of the sensor unit 156 included in the robot 40. More specifically, the parameter processing unit 240 can determine a new emotion similar to the emotion "fear" updated by the recording control unit 270 and associated with the information "Person A" "patted" newly detected by the sensor of the sensor unit 156 included in the robot 40 and the element "Person A" included in the information "Person A" "hit" previously detected by the sensor of the sensor unit 156 included in the robot 40. In other words, the parameter processing unit 240, in a situation that has something in common with a previously experienced situation, takes into account changes in the relationship and to some extent suppresses the strong expression of an emotion completely opposite to the updated emotion. This makes it possible to generate a variety of emotions based on the current situation and the people and events learned in the past, while also taking into account changes in the relationship with the subject.
[0164] The neural network parameters calculated by the parameter processing unit 240 are supplied to the switching control unit 260 , the recording control unit 270 and the operation determination unit 250 .
[0165] The recording control unit 270 associates the emotion of the robot 40 with information obtained by processing at least a portion of the information detected by each sensor of the sensor unit 156, and stores the information in the storage unit 172. For example, the recording control unit 270 associates the emotion of the robot 40 with various types of information received by the communication unit 208 and stores the information in the recorded data 292. For example, the recording control unit 270 associates the emotion of the robot 40 with various types of information related to the user 30, peripheral information, and event information, and stores the information in the recorded data 292. For example, the recording control unit 270 associates the emotion of the robot 40 with an emotion value of the robot 40 as the emotion of the robot 40 and stores the information in the recorded data 292.
[0166] For example, the recording control unit 270 associates information detected by a sensor included in the sensor unit 156 provided in the robot 40 with an emotion determined by the parameter processing unit 240, and stores the associated information in the storage unit 172. For example, the recording control unit 270 associates the information "Person A" and "patted on the head" detected by the sensor included in the sensor unit 156 provided in the robot 40 with the emotion "happy" determined by the parameter processing unit 240, and stores the associated information in the recorded data 292. In other words, the recording control unit 270 associates information about people, events, etc. with information about emotions, and stores the associated information.
[0167] Furthermore, the recording control unit 270 stores the information in the first recording format or the information in the second recording format received by the communication unit 208 from the robot 40 in the recording data 292. The recording control unit 270 stores the information received from each robot 40 in the recording data 292 in association with information identifying each robot 40.
[0168] Furthermore, based on the emotion newly determined by the parameter processing unit 240, the recording control unit 270 updates the emotion previously determined by the parameter processing unit 240, which is associated with information previously detected by a sensor provided in the robot 40 and which includes elements common to the information newly detected by the sensor provided in the robot 40, to the emotion newly determined by the parameter processing unit 240.
[0169] For example, when the strength of an emotion newly determined by the parameter processing unit 240 exceeds a predetermined threshold, the recording control unit 270 updates the emotion previously determined by the parameter processing unit 240, which is associated with information previously detected by a sensor provided in the robot 40 and which includes elements common to information newly detected by a sensor provided in the robot 40, to the emotion newly determined by the parameter processing unit 240.
[0170] More specifically, when the emotion "very happy" newly determined by the parameter processing unit 240 exceeds a predetermined threshold, the recording control unit 270 updates the emotion "happy" previously determined by the parameter processing unit 240, which is associated with the element "person A" included in the information "person A" and "petted" previously detected by the sensor of the sensor unit 156 provided in the robot 40, and which includes the element "person A" common to the information "person A" and "hug" newly detected by the sensor provided in the robot 40, to the emotion "very happy" newly determined by the parameter processing unit 240. That is, in the above example, the recording control unit 270 updates the emotion associated with "person A" from "happy" to "very happy".
[0171] The predetermined threshold can be a fixed value, or any value or rate of change depending on the purpose, such as any value or rate of change relative to the strength of the current emotion, any value or rate of change relative to the type of the current emotion, etc. This makes it possible to generate a variety of emotions by taking into account changes in the relationship with the target and using the determined strength of the emotion based on the situation of people and events learned in the past and the current situation.
[0172] Furthermore, in the above example, when the recording control unit 270 updates the emotion associated with a specific element included in the detected information, it may also update emotions associated with other elements detected together with the common element. More specifically, when the emotion "very fun" newly determined by the parameter processing unit 240 exceeds a predetermined threshold, the recording control unit 270 updates the emotion "boredom" previously determined by the parameter processing unit 240, which is associated with the element "amusement park" included in the information "person A" and "amusement park" previously detected by the sensor of the sensor unit 156 provided in the robot 40, including the element "person A" common to the information "person A," "hugged," and "park" newly detected by the sensor provided in the robot 40, to the emotion "very fun" newly determined by the parameter processing unit 240. That is, in the above example, the recording control unit 270 updates the emotion associated with the "amusement park" related to "person A" from "boredom" to "very fun" using the common element "person A" as a key. That is, when updating the emotion associated with the common element "Person A" from "bored" to "very fun," the recording control unit 270 can also update the emotion associated with the element "Amusement Park" previously detected together with the common element "Person A" from "bored" to "very fun." In this way, for example, when the emotion toward a person changes from negative to positive, the emotion toward an event related to that person also changes from negative to positive. This makes it possible to propagate the change in the relationship with the target to other targets in accordance with the situation of the person or event learned in the past and the current situation, and to generate a variety of emotions taking the changed relationship into consideration.
[0173] Furthermore, for example, when the type of emotion determined by the parameter processing unit 240 undergoes a predetermined change, the recording control unit 270 updates the emotion previously determined by the parameter processing unit 240, which is associated with information previously detected by the sensor of the sensor unit 156 provided in the robot 40 and which includes an element common to information newly detected by the sensor of the sensor unit 156 provided in the robot 40, to the emotion newly determined by the parameter processing unit 240. More specifically, when the emotion "fearful" newly determined by the parameter processing unit 240 is the predetermined change, the recording control unit 270 updates the emotion "happy" previously determined by the parameter processing unit 240, which is associated with the element "person A" included in the information "person A" "patted" previously detected by the sensor of the sensor unit 156 provided in the robot 40 and which includes the element "person A" common to the information "person A" "hit" newly detected by the sensor of the sensor unit 156 provided in the robot 40, to the emotion "fearful" newly determined by the parameter processing unit 240.
[0174] That is, in the above example, the recording control unit 270 updates the emotion associated with "Person A" from "happy" to "fearful." Note that the predetermined change can be any change according to the purpose, in addition to the change to opposing emotions mentioned in the above example. This makes it possible to generate a variety of emotions by taking into account changes in the relationship with the target and using a determined change in emotion based on the situation of people and events learned in the past and the current situation.
[0175] Furthermore, for example, the recording control unit 270 can perform an update process by combining the strength and type of emotion determined by the parameter processing unit 240. For example, when the type of emotion determined by the parameter processing unit 240 undergoes a predetermined change and the strength of the emotion exceeds a predetermined threshold, the recording control unit 270 updates the emotion previously determined by the parameter processing unit 240, which is associated with information previously detected by the sensor of the sensor unit 156 provided in the robot 40 and which includes elements common to information newly detected by the sensor of the sensor unit 156 provided in the robot 40, to the emotion newly determined by the parameter processing unit 240. More specifically, when the emotion "very scary" newly determined by the parameter processing unit 240 is a predetermined change and exceeds a predetermined threshold, the recording control unit 270 updates the emotion "happy" previously determined by the parameter processing unit 240, which is associated with the element "Person A" included in the information "Person A" and "Pat" previously detected by the sensor unit 156 provided in the robot 40, including the element "Person A" common to the information "Person A" and "Hit" newly detected by the sensor provided in the sensor unit 156 provided in the robot 40, to the emotion "very scary" newly determined by the parameter processing unit 240. That is, in the above example, the recording control unit 270 updates the emotion associated with "Person A" from "happy" to "very scary." This makes it possible to generate a variety of emotions by taking into account changes in the relationship with the target using the change and intensity of the determined emotion in accordance with the situation of people and events learned in the past and the current situation.
[0176] The target of updating by the recording control unit 270 is an emotion associated with information that includes elements common to information newly detected by the sensor of the sensor unit 156 provided in the robot 40. In other words, emotions associated with information that shares at least one element among information such as surrounding information, location information, date and time, weather, temperature, scenery, smells, sounds, the number of people in the vicinity, people, their facial expressions, their words and actions, and other situations can be updated by the recording control unit 270. This makes it possible to identify commonalities between the situations of people or events learned in the past and the current situation, and then generate a variety of emotions by taking into account changes in the relationship with the target using the determined emotion.
[0177] The switching control unit 260 determines the recording format of the information generated by the processing unit 152 of the robot 40 based on the parameters supplied from the parameter processing unit 240. When it is necessary to switch the recording format of the information generated by the processing unit 152, the switching control unit 260 causes the robot 40 to transmit an instruction to switch the recording format via the communication unit 208. In the robot 40, the processing unit 152 switches the recording format in accordance with the instruction received from the server 200.
[0178] For example, when the switching control unit 260 is causing the processing unit 152 to generate information in the first recording format, the switching control unit 260 transmits to the robot 40 an instruction to switch the recording format of the information generated by the processing unit 152 from the first recording format to the second recording format in response to an increase in the intensity of the emotion determined by the parameter processing unit 240. In this way, the switching control unit 260 switches the recording format of the information recorded by the recording control unit 270 from the first recording format to the second recording format in response to an increase in the intensity of the emotion determined by the parameter processing unit 240. This allows detailed information about when the robot 40's emotions become heightened to be recorded. Furthermore, in the robot 40, the processing unit 152 obtains video data in the second recording format from the server 200 and generates a video to be presented to the user 30. Therefore, the user 30 can enjoy information about when the robot 40's emotions become heightened as a video.
[0179] Furthermore, in response to an increase in the intensity of the user 30's emotion, the switching control unit 260 transmits to the robot 40 an instruction to switch the recording format of the information generated by the processing unit 152 from the first recording format to the second recording format. In this way, in response to an increase in the intensity of the emotion determined by the parameter processing unit 240, the switching control unit 260 switches the recording format of the information recorded by the recording control unit 270 from the first recording format to the second recording format. This allows detailed information about when the user 30's emotion is heightened to be recorded. In addition, in the robot 40, the processing unit 152 acquires video data in the second recording format from the server 200 and generates a video to be presented to the user 30. Therefore, the user 30 can enjoy information about when their emotion is heightened as a video. The user 30's emotion can be detected, for example, based on the voice or facial expression detected from the user 30. Furthermore, the switching control unit 260 may recognize a change in the emotion of the user 30 based on the detection of an impact by a touch sensor (not shown) included in the sensor unit 156. For example, when an impact is detected by the touch sensor included in the sensor unit 156, the switching control unit 260 may recognize that the emotion of the user 30 has worsened.
[0180] When the switching control unit 260 is causing the processing unit 152 to generate information in the second recording format, the switching control unit 260 transmits to the robot 40 an instruction to switch the recording format of the information to be generated by the processing unit 152 from the second recording format to the first recording format in response to a decrease in the intensity of the emotion of the robot 40 or the emotion of the user determined by the parameter processing unit 240. In this way, the switching control unit 260 switches the recording format of the information recorded by the recording control unit 270 from the second recording format to the first recording format in response to a decrease in the intensity of the emotion determined by the parameter processing unit 240 or the emotion of the user.
[0181] Furthermore, in response to an increase in the emotional change of the user 30 or the robot 40, the switching control unit 260 transmits to the robot 40 an instruction to switch the recording format of the information generated by the processing unit 152 from the first recording format to the second recording format. This allows information when the user 30 or the robot 40 becomes emotional to be recorded in detail. Furthermore, in the robot 40, the processing unit 152 obtains video data in the second recording format from the server 200 and generates video to be presented to the user 30. Therefore, the user 30 can enjoy information when the user 30 or the robot 40 becomes emotional as video.
[0182] When the switching control unit 260 is causing the processing unit 152 to generate information in the second recording format, the switching control unit 260 transmits to the robot 40 an instruction to switch the recording format of the information to be generated by the processing unit 152 from the second recording format to the first recording format in response to the change in the emotion of the robot 40 or the emotion of the user subsiding (the amount of change becoming equal to or less than the threshold) or the emotion of the robot 40 or the emotion of the user returning to the emotion before the change. In this way, the switching control unit 260 switches the recording format of the information recorded by the recording control unit 270 from the second recording format to the first recording format in response to the change in the emotion of the robot 40 or the emotion of the user subsiding.
[0183] The switching control unit 260 may also switch the recording format depending on the continuity of changes in the emotions of the robot 40 or the user 30. For example, when the cumulative value of the emotion value within a predetermined time period exceeds a threshold, the switching control unit 260 sends to the robot 40 an instruction to switch the recording format of the information generated by the processing unit 152 from the first recording format to the second recording format. The switching control unit 260 may also switch the recording format depending on the combination of the speed and speed of changes (gradients) in emotions within a predetermined time period or the degree of the gradient. For example, when the gradient indicating the change in emotion value within a predetermined time period shows a combination of a sudden increase, a gradual increase for a certain period of time, and a sudden decrease, the switching control unit 260 sends to the robot 40 an instruction to switch the recording format of the information generated by the processing unit 152 from the first recording format to the second recording format. This makes it possible to record video data of situations where, for example, the absolute value of the emotion is not relatively large but the duration is perceived as long (emotional values above a certain value continue for a predetermined period of time), such as in a tense situation due to pressure or the like.
[0184] Furthermore, when the recording control unit 270 switches the recording format of the information generated by the processing unit 152 from the first recording format to the second recording format, it sends an instruction to the robot 40 to tag the information in the second recording format. This allows the user 30 to clearly recall the situation at the time, etc., from the tagged information when checking the recorded information. Furthermore, tagging makes it easier to find information related to a specific situation from the recorded information.
[0185] However, people may remember information for different periods depending on the emotions they felt when they memorized the information. For example, people tend to remember information associated with strong emotions for a longer period of time, and information associated with weak emotions for a longer period of time.
[0186] Therefore, the recording control unit 270 controls the storage period of information stored in the recording data 292 of the storage unit 172 according to the emotion associated with the information. For example, the recording control unit 270 controls the storage period so that information associated with a strong emotion is stored for a long period of time, and information associated with a weak emotion is stored for a short period of time. For example, the recording control unit 270 determines whether the emotion value associated with the information is equal to or greater than a predetermined threshold for determining that the emotion is strong, and determines that the emotion is strong if the emotion value is equal to or greater than the threshold, and determines that the emotion is strong if the emotion value is less than the threshold. Note that the recording control unit 270 may set multiple thresholds and determine emotions in multiple stages, such as strong, normal, and weak.
[0187] The recording control unit 270 sets a storage period according to the strength of the emotion. For example, the recording control unit 270 sets a longer storage period for information associated with a strong emotion, and a shorter storage period for information associated with a weak emotion. The recording control unit 270 periodically determines whether the storage period has elapsed for each piece of information stored in the recording data 292 since the date and time the information was stored in the recording data 292. The recording control unit 270 deletes information whose storage period has elapsed from the recording data 292. This allows information to be appropriately stored according to the emotion at the time the information was stored. For example, information associated with a strong emotion can be stored longer than information associated with a weak emotion.
[0188] The recording control unit 270 may change the storage period for information associated with weak emotions if similar information is stored consecutively, rather than consecutively. For example, the recording control unit 270 may change the storage period for information about the same person, event, etc. if the information is stored a predetermined number of times within a predetermined period, or for a long period of time. This allows information associated with weak emotions to be stored for a long time if the information is not consecutively stored.
[0189] When switching the recording format of the information generated by the processing unit 152 from the first recording format to the second recording format, the recording control unit 270 transmits to the robot 40 an instruction to tag the information in the second recording format. This allows the user 30 to clearly recall the situation at the time, etc., from the tagged information when checking the recorded information. Furthermore, tagging makes it easier to find information related to a specific situation from the recorded information.
[0190] However, over time, people tend to generalize the information they have memorized. For example, over time, people tend to gradually forget the details of the information they have memorized, and the information becomes general. The memories that remain in the end are more generalized.
[0191] Therefore, the recording control unit 270 may update the information stored in the storage unit 172 to generalize the information over time since the information was stored. For example, the recording control unit 270 updates the information stored in the recording data 292 to information of a higher concept over time since the information was stored. For example, the recording control unit 270 reads out the information stored in the recording data 292 every time a predetermined period of time has elapsed since the information was stored. The predetermined period of time may be a fixed period of time or may be determined to correspond to the period of time over which memory fades. For example, the predetermined period of time may be set to increase in length, such as one month, three months, six months, one year, three years, etc. The recording control unit 270 identifies a general concept of the read out information. For example, the recording control unit 270 identifies general information about detailed information included in the information. For example, the recording control unit 270 identifies a broader area including the location, or a broader time period, date, or period including the date and time, for date and time or location information included in the information. The recording control unit 270 also erases some of the detailed content of the read information. The recording control unit 270 then updates the information stored in the recording data 292 corresponding to the read information to information of a specific higher concept. For example, if the recording data 292 stores "I went to a theme park with a character," the recording control unit 270 updates it over time, first to "I went to a theme park," and then to "I went to an amusement park somewhere." For example, if the recording data 292 stores "Sunny with about 30% clouds," the recording control unit 270 updates it over time, first to "Sunny with light clouds," then to "Sunny," and then to "Good weather." For example, if the recording data 292 stores "I was petted by a child," the recording control unit 270 updates it over time, first to "I was petted by someone," and then to "I was petted." Also, for example, if "traveled to Thailand" is stored, the recording control unit 270 may update it over time to "traveled to Thailand," then to "traveled to Southeast Asia," then to "traveled abroad," and then to "traveled." This allows the stored information to be generalized, compressed, or deleted over time.
[0192] Additionally, people tend to experience less emotion associated with memorized information over time.
[0193] Therefore, the recording control unit 270 may update the emotion associated with the information stored in the storage unit 172 over time. For example, the recording control unit 270 updates the information stored in the recording data 292 so as to weaken the intensity of the emotion associated with the information stored in the recording data 292 over time from the time the information was stored. For example, the recording control unit 270 updates the emotion value associated with the information stored in the recording data 292 to a smaller value by a predetermined percentage or a predetermined value each time a predetermined period of time has elapsed since the information was stored. The predetermined period of time may be a fixed period of time, or may be set to correspond to a period of time during which memory fades. In this way, the intensity of the emotion associated with the stored information can be weakened over time.
[0194] The action decision rule 282 defines an action to be performed by the robot 40 in association with the state of the robot 40. For example, the action decision rule 282 defines an action to be performed by the robot 40 in association with the internal state of an artificial neuron of a neural network. For example, the action decision rule 282 defines an action to be performed by the robot 40, such as uttering words expressing happiness, in association with the condition that an emotional artificial neuron defining the emotion of "happy" is high. The action decision rule 282 also defines an action to be performed when the robot becomes sleepy in association with the condition that an internal state of an endocrine artificial neuron defining an endocrine substance corresponding to sleepiness is high.
[0195] Note that endocrine substances refer to substances such as neurotransmitters and hormones that are secreted within the body and transmit signals. Furthermore, endocrine means that endocrine substances are secreted within the body. However, the endocrine substances of the robot 40 itself are one piece of information that affects the behavior of the robot 40, and do not mean that the robot 40 actually produces endocrine substances. Similarly, the emotions of the robot 40 itself are one piece of information that affects the behavior of the robot 40, and do not mean that the robot 40 actually has emotions.
[0196] The movement determination unit 250 determines the movement of the robot 40 based on the movement defined in the movement determination rules 282 in association with the activation state or internal state of each artificial neuron determined by the parameter processing unit 240 .
[0197] The action determination unit 250 reads information stored in the record data 292 in the storage unit 172 in accordance with the determined action. For example, when it is necessary to output information stored in the record data 292 for an action defined in the action determination rule 282, the action determination unit 250 reads necessary information from the record data 292. Furthermore, for example, when the user 30 issues an output instruction regarding the information stored in the record data 292, the action determination unit 250 reads the instructed information to be output from the record data 292. Furthermore, when the action determination unit 250 is to have a conversation with the user 30, respond to an inquiry from the user 30, or ask a question to the user 30, the action determination unit 250 reads information indicating the content of the utterance from the record data 292.
[0198] The action determination unit 250 generates action information for the robot 40 so that the robot 40 performs the determined action, and outputs the information to the robot 40 together with the read information. When the robot 40 outputs information stored in the record data 292 of the storage unit 172, the action determination unit 250 controls to change one or both of the behavior of the robot 40 and the output timing of the information according to the emotion associated with the information. For example, the action determination unit 250 associates the level of emotion with an event and provides a lag in the timing of outputting information according to the level of emotion. For example, the action determination unit 250 controls to output information associated with a weak emotion over a longer period of time, or to output information with a filler word, compared to information associated with a strong emotion. For example, the action determination unit 250 controls to output information associated with a weak emotion over a longer period of time. For example, the action determination unit 250 controls to output information associated with a weak emotion by inserting filler words such as "umm" or "hmm" into the conversation to create a pause in the output. This allows the stored information to be output according to the emotion associated with the information. It also allows for natural conversation.
[0199] Furthermore, for example, the action determination unit 250 controls the output of information in small amounts, even when the information is associated with a strong emotion. The action determination unit 250 controls the output of information associated with a strong emotion to be output little by little, compared to information associated with a weak emotion. Furthermore, the action determination unit 250 detects the surrounding circumstances when the information associated with a strong emotion is output from the robot 40 using a sensor, and controls one or both of the behavior of the robot 40 and the timing of outputting the information according to the detected surrounding circumstances. For example, when a strong emotion is associated, the action determination unit 250 controls the robot 40 to insert filler words such as "eh..." or "that's..." into the conversation, carefully reading and outputting the surrounding information while considering the next line to be spoken. This allows the stored information to be output according to the emotion associated with the information. Furthermore, it is possible to buy time to grasp the surrounding information. Furthermore, since the robot 40 can grasp the surrounding information and output the next line to be spoken, it is possible to lead to a natural conversation.
[0200] The action information indicating the action determined by the action determination unit 250 is transmitted from the communication unit 208 to the robot 40. In the robot 40, the processing unit 152 controls the control target 155 to cause the robot 40 to perform the action indicated by the information received from the server 200. This allows the robot 40 to perform an appropriate action according to the current emotion of the robot 40.
[0201] However, people may remember information for different periods depending on the emotions they felt when they memorized the information. For example, people tend to remember information associated with strong emotions for a longer period of time, and information associated with weak emotions for a longer period of time.
[0202] Therefore, the recording control unit 270 controls the storage period of information stored in the recording data 292 of the storage unit 172 according to the emotion associated with the information. For example, the recording control unit 270 controls the storage period so that information associated with a strong emotion is stored for a long period of time, and information associated with a weak emotion is stored for a short period of time. For example, the recording control unit 270 determines whether the emotion value associated with the information is equal to or greater than a predetermined threshold for determining that the emotion is strong, and determines that the emotion is strong if the emotion value is equal to or greater than the threshold, and determines that the emotion is strong if the emotion value is less than the threshold. Note that the recording control unit 270 may set multiple thresholds and determine emotions in multiple stages, such as strong, normal, and weak.
[0203] The recording control unit 270 sets a storage period according to the strength of the emotion. For example, the recording control unit 270 sets a longer storage period for information associated with a strong emotion, and a shorter storage period for information associated with a weak emotion. The recording control unit 270 periodically determines whether the storage period has elapsed for each piece of information stored in the recording data 292 since the date and time the information was stored in the recording data 292. The recording control unit 270 deletes information whose storage period has elapsed from the recording data 292. This allows information to be appropriately stored according to the emotion at the time the information was stored. For example, information associated with a strong emotion can be stored longer than information associated with a weak emotion.
[0204] The recording control unit 270 may change the storage period for information associated with weak emotions if similar information is stored consecutively, rather than consecutively. For example, the recording control unit 270 may change the storage period for information about the same person, event, etc. if the information is stored a predetermined number of times within a predetermined period, or for a long period of time. This allows information associated with weak emotions to be stored for a long time if the information is not consecutively stored.
[0205] When switching the recording format of the information generated by the processing unit 152 from the first recording format to the second recording format, the recording control unit 270 transmits to the robot 40 an instruction to tag the information in the second recording format. This allows the user 30 to clearly recall the situation at the time, etc., from the tagged information when checking the recorded information. Furthermore, tagging makes it easier to find information related to a specific situation from the recorded information.
[0206] However, over time, people tend to generalize the information they have memorized. For example, over time, people tend to gradually forget the details of the information they have memorized, and the information becomes general. The memories that remain in the end are more generalized.
[0207] Therefore, the recording control unit 270 may update the information stored in the storage unit 172 to generalize the information over time since the information was stored. For example, the recording control unit 270 updates the information stored in the recording data 292 to information of a higher concept over time since the information was stored. For example, the recording control unit 270 reads out the information stored in the recording data 292 every time a predetermined period of time has elapsed since the information was stored. The predetermined period of time may be a fixed period of time or may be determined to correspond to the period of time over which memory fades. For example, the predetermined period of time may be set to increase in length, such as one month, three months, six months, one year, three years, etc. The recording control unit 270 identifies a general concept of the read out information. For example, the recording control unit 270 identifies general information about detailed information included in the information. For example, the recording control unit 270 identifies a broader area including the location, or a broader time period, date, or period including the date and time, for date and time or location information included in the information. The recording control unit 270 also erases some of the detailed content of the read information. The recording control unit 270 then updates the information stored in the recording data 292 corresponding to the read information to information of a specific higher concept. For example, if the recording data 292 stores "I went to a theme park with a character," the recording control unit 270 updates it over time, first to "I went to a theme park," and then to "I went to an amusement park somewhere." For example, if the recording data 292 stores "Sunny with about 30% clouds," the recording control unit 270 updates it over time, first to "Sunny with light clouds," then to "Sunny," and then to "Good weather." For example, if the recording data 292 stores "I was petted by a child," the recording control unit 270 updates it over time, first to "I was petted by someone," and then to "I was petted." Also, for example, if "traveled to Thailand" is stored, the recording control unit 270 may update it over time to "traveled to Thailand," then to "traveled to Southeast Asia," then to "traveled abroad," and then to "traveled." This allows the stored information to be generalized, compressed, or deleted over time.
[0208] Additionally, people tend to experience less emotion associated with memorized information over time.
[0209] Therefore, the recording control unit 270 may update the emotion associated with the information stored in the storage unit 172 over time. For example, the recording control unit 270 updates the information stored in the recording data 292 so as to weaken the intensity of the emotion associated with the information stored in the recording data 292 over time from the time the information was stored. For example, the recording control unit 270 updates the emotion value associated with the information stored in the recording data 292 to a smaller value by a predetermined percentage or a predetermined value each time a predetermined period of time has elapsed since the information was stored. The predetermined period of time may be a fixed period of time, or may be set to correspond to a period of time during which memory fades. In this way, the intensity of the emotion associated with the stored information can be weakened over time.
[0210] Furthermore, the recording control unit 270 updates an emotion previously determined by the parameter processing unit 240 to an emotion corresponding to the time when information previously detected by the sensor of the sensor unit 156 provided in the robot 40, which includes elements common to information newly detected by the sensor of the sensor unit 156 provided in the robot 40 and which is associated with an emotion newly determined by the parameter processing unit 240, was stored. For example, the more recently the information previously detected by the sensor of the sensor unit 156 provided in the robot 40, which includes elements common to information newly detected by the sensor of the sensor unit 156 provided in the robot 40 and which is associated with an emotion newly determined by the parameter processing unit 240, the more the recording control unit 270 updates the emotion to an emotion closer to the emotion newly determined by the parameter processing unit 240;
[0211] More specifically, consider an example in which a person is treated kindly by person A when they first meet, and the emotion "very happy" is associated with "person A." However, several days later, it is discovered that person A behaved in that way not out of kindness but because of an ulterior motive, and the new emotion "very sad" is determined. When updating the emotion associated with "person A," the recording control unit 270 updates the emotion "very happy" previously determined by the parameter processing unit 240 to the same emotion "very sad" as the emotion "very sad" newly determined by the parameter processing unit 240, because the information previously detected by the sensor unit 156 of the robot 40, which includes elements common to the information newly detected by the sensor unit 156 of the robot 40 and which is associated with the emotion newly determined by the parameter processing unit 240, was stored recently.
[0212] In another example, consider a case where a person was treated kindly by person A several years ago and the emotion "very happy" was associated with "person A," but when the person meets person A again several years later, it is discovered that person A had behaved in that way not out of kindness but because of an ulterior motive, and the new emotion determined is "very sad." When updating the emotion associated with "person A," the recording control unit 270 updates the emotion previously determined by the parameter processing unit 240, "very happy," to "happy," which is slightly closer to the emotion newly determined by the parameter processing unit 240, "very sad," because the information previously detected by the sensor unit 156 of the robot 40, which includes elements common to the information newly detected by the sensor unit 156 of the robot 40 and which is associated with the emotion newly determined by the parameter processing unit 240, was stored several years ago.
[0213] In another example, consider a case where a robot was treated kindly by person A several decades ago and associated the emotion of "very happy" with "person A." However, when the robot reunites with person A several decades later, person A behaves rudely, resulting in a newly determined emotion of "very sad." When updating the emotion associated with "person A," the recording control unit 270 does not update the emotion previously determined by the parameter processing unit 240 to "very happy" without moving it toward the newly determined emotion of "very sad" or updates it to the same emotion as the previously determined emotion of "very happy." That is, in the above example, the recording control unit 270 updates the emotion to one within a range between the previously determined emotion and the newly determined emotion, depending on when the information to be updated was stored. This allows emotions to be updated according to the time when the information to be updated was stored, taking into account changes in the relationship with the subject, making it possible to generate a variety of emotions.
[0214] Furthermore, the recording control unit 270 updates an emotion previously determined by the parameter processing unit 240 to an emotion corresponding to the amount of information stored in association with an emotion newly determined by the parameter processing unit 240. For example, the greater the amount of information stored in association with an emotion newly determined by the parameter processing unit 240, the closer the recording control unit 270 updates the emotion to an emotion closer to the emotion newly determined by the parameter processing unit 240, and the less the amount of information stored, the closer the emotion to the emotion previously determined by the parameter processing unit 240. More specifically, consider an example in which a person has been treated kindly by person A in the past and has associated the emotion of "very happy" with "person A," and it is later discovered that person A behaved in that way not out of kindness but because of an ulterior motive, and the new emotion determined is "very sad." When updating the emotion associated with "person A," the recording control unit 270 updates the emotion previously determined by the parameter processing unit 240 to the same emotion as the emotion newly determined by the parameter processing unit 240, "very sad," because the emotion newly determined by the parameter processing unit 240, "very sad," is a very strong emotion and the amount of information stored in association with it is large.
[0215] On the other hand, consider an example in which a person was treated kindly by person A in the past and thus associated with "person A" an emotion of "very happy." Later, person A is slightly mean to the person, and thus a new emotion of "slightly sad" is determined. When updating the emotion associated with "person A," the recording control unit 270 updates the emotion previously determined by the parameter processing unit 240 from "very happy" to "happy," which is slightly closer to the newly determined emotion of "slightly sad" because the newly determined emotion, "slightly sad," is a weak emotion and the amount of information associated with it is small. That is, in the above example, the recording control unit 270 updates the emotion to one within a range from the previously determined emotion to the newly determined emotion, depending on the amount of information associated with the newly determined emotion. This enables the emotion to be updated depending on the amount of information associated with the determined emotion, thereby taking into account changes in the relationship with the target and generating a variety of emotions.
[0216] Returning now to the example in which the recording control unit 270 performs an update when the strength of the emotion newly determined by the parameter processing unit 240 exceeds a predetermined threshold. More specifically, when the emotion "very happy" newly determined by the parameter processing unit 240 exceeds a predetermined threshold, the recording control unit 270 updates the emotion "happy" previously determined by the parameter processing unit 240, which is associated with the element "Person A" included in the information "Person A" and "Patted" previously detected by the sensor of the sensor unit 156 provided in the robot 40, including the element "Person A" common to the information "Person A" and "Hugged" newly detected by the sensor provided in the robot 40, to "very happy" based on the elements related to the emotion "very happy" newly determined by the parameter processing unit 240. That is, in the above example, the recording control unit 270 updates the emotion associated with "Person A" from "happy" to "very happy."
[0217] The predetermined threshold can be a fixed value, or any value or rate of change depending on the purpose, such as any value or rate of change relative to the strength of the current emotion, any value or rate of change relative to the type of the current emotion, etc. This makes it possible to generate a variety of emotions by taking into account changes in the relationship with the target and using the determined strength of the emotion based on the situation of people and events learned in the past and the current situation.
[0218] Furthermore, in the above example, when the recording control unit 270 updates the emotion associated with a specific element included in the detected information, it may also update emotions associated with other elements detected along with the common element. More specifically, when the emotion "very fun" newly determined by the parameter processing unit 240 exceeds a predetermined threshold, the recording control unit 270 updates the emotion "boredom" previously determined by the parameter processing unit 240, which is associated with the element "amusement park" included in the information "person A" and "amusement park" previously detected by the sensor of the sensor unit 156 provided in the robot 40, including the element "person A" common to the information "person A," "hugged," and "park" newly detected by the sensor provided in the robot 40, to "fun" based on the elements related to the emotion "very fun" newly determined by the parameter processing unit 240. That is, in the above example, the recording control unit 270 updates the emotion associated with the "amusement park" related to "person A" from "boredom" to "fun" using the common element "person A" as a key. That is, when updating the emotion associated with the common element "Person A" from "boredom" to "fun," the recording control unit 270 can also update the emotion associated with the element "Amusement Park" previously detected together with the common element "Person A" from "boredom" to "fun." In this way, for example, when the emotion toward a person changes from negative to positive, the emotion toward an event related to that person also changes from negative to positive. This makes it possible to propagate the change in the relationship with the target to other targets in accordance with the situation of the person or event learned in the past and the current situation, and to generate a variety of emotions taking the changed relationship into consideration.
[0219] Furthermore, for example, when the type of emotion determined by the parameter processing unit 240 undergoes a predetermined change, the recording control unit 270 updates the emotion previously determined by the parameter processing unit 240, which is associated with information previously detected by the sensor of the sensor unit 156 provided in the robot 40 and which includes an element common to information newly detected by the sensor of the sensor unit 156 provided in the robot 40, based on the element related to the emotion newly determined by the parameter processing unit 240. More specifically, when the emotion "fearful" newly determined by the parameter processing unit 240 is the predetermined change, the recording control unit 270 updates the emotion "happy" previously determined by the parameter processing unit 240, which is associated with the element "person A" included in the information "person A" "patted" previously detected by the sensor of the sensor unit 156 provided in the robot 40, which includes the element "person A" common to the information "person A" "hit" newly detected by the sensor of the sensor unit 156 provided in the robot 40, to "a little scared" based on the element related to the emotion "fearful" newly determined by the parameter processing unit 240.
[0220] That is, in the above example, the recording control unit 270 updates the emotion associated with "Person A" from "happy" to "slightly scared." Note that the predetermined change can be any change according to the purpose, in addition to the change to opposing emotions mentioned in the above example. This makes it possible to generate a variety of emotions by taking into account changes in the relationship with the target and using a determined change in emotion based on the situation of people and events learned in the past and the current situation.
[0221] Furthermore, for example, the recording control unit 270 can perform an update process by combining the strength and type of emotion determined by the parameter processing unit 240. For example, when the type of emotion determined by the parameter processing unit 240 undergoes a predetermined change and the strength of the emotion exceeds a predetermined threshold, the recording control unit 270 updates the emotion previously determined by the parameter processing unit 240, which is associated with information previously detected by the sensor unit 156 provided in the robot 40 and which includes elements common to information newly detected by the sensor provided in the sensor unit 156 provided in the robot 40, based on elements related to the emotion newly determined by the parameter processing unit 240. More specifically, when the emotion "very scary" newly determined by the parameter processing unit 240 is a predetermined change and exceeds a predetermined threshold, the recording control unit 270 updates the emotion "happy" previously determined by the parameter processing unit 240, which is associated with the element "Person A" included in the information "Person A" and "Pat" previously detected by the sensor unit 156 of the robot 40, including the element "Person A" common to the information "Person A" and "Hit" newly detected by the sensor unit 156 of the robot 40, to "scary" based on the element related to the emotion "very scary" newly determined by the parameter processing unit 240. That is, in the above example, the recording control unit 270 updates the emotion associated with "Person A" from "happy" to "scary." This makes it possible to generate a variety of emotions by taking into account changes in the relationship with the target using the change and intensity of the determined emotion in accordance with the previously learned situations of people and events and the current situation.
[0222] The target of updating by the recording control unit 270 is an emotion associated with information that includes elements common to information newly detected by the sensor of the sensor unit 156 provided in the robot 40. In other words, emotions associated with information that shares at least one element among information such as surrounding information, location information, date and time, weather, temperature, scenery, smells, sounds, the number of people in the vicinity, people, their facial expressions, their words and actions, and other situations can be updated by the recording control unit 270. This makes it possible to identify commonalities between the situations of people or events learned in the past and the current situation, and then generate a variety of emotions by taking into account changes in the relationship with the target using the determined emotion.
[0223] Next, we will explain the movement control of the robot 40. Fig. 19 is a diagram showing a flowchart related to the movement control of the robot 40. When the movement determination unit 250 determines the movement of the robot 40, it reads out information stored in the recorded data 292 of the storage unit 172 in accordance with the determined movement (S600).
[0224] In S610, the action determination unit 250 changes one or both of the behavior of the robot 40 and the timing of outputting the information in accordance with the emotion associated with the read information. Then, in S620, the action determination unit 250 generates action information for the robot 40 so that the robot 40 performs the determined action, outputs this to the robot 40 together with the read information, and ends this flow. In this way, the system 20 can cause the robot 40 to output the stored information in accordance with the emotion associated with the information.
[0225] Next, the deletion of information stored in the recorded data 292 will be described. FIG. 20 is a flowchart showing control of the data storage period. The recording control unit 270 determines the strength of the emotion associated with the information stored in the recorded data 292 of the storage unit 172 (S700). For example, the recording control unit 270 determines whether the emotion value associated with the information is equal to or greater than a predetermined threshold value that indicates a strong emotion, and determines that the emotion is strong if the emotion value is equal to or greater than the threshold, and determines that the emotion is strong if the emotion value is less than the threshold.
[0226] In S710, the recording control unit 270 sets a storage period depending on the strength of the emotion. For example, the recording control unit 270 sets a long storage period for storing information associated with a strong emotion, and a short storage period for information associated with a weak emotion.
[0227] In S720, the recording control unit 270 deletes from the recorded data 292 any information stored in the recorded data 292 for which the retention period has elapsed since the date and time the information was stored in the recorded data 292, and ends this flow. This allows the system 20 to appropriately store information according to the associated emotion. For example, information associated with a strong emotion can be stored longer than information associated with a weak emotion.
[0228] According to the system 20, the robot 40 continuously transmits highly compressed information, such as skeletal data, to the server 200 and records it there while the robot 40 is not in a significantly emotional state. The continuous information, such as skeletal data, recorded in the server 200 can be used to analyze the robot 40's memories. When the robot 40's emotions become significantly elevated, the robot 40 starts transmitting full HD video data and audio data, and while the robot 40's emotions remain strong above a certain level, the robot 40 records low-compression information, including the full HD video data and audio data, in addition to the skeletal data, in the server 200. When the user 30 requests the robot 40 to provide a video of the robot 40's memories, the robot 40 requests the server 200 to transmit the full HD video data and audio data, and provides the video data and audio data received from the server 200 to the user 30.
[0229] In this way, according to the system 20, high-quality video data of scenes in which the robot 40 feels strong emotions can be stored in the server 200. On the other hand, when the robot 40 does not feel strong emotions, simplified information such as skeletal data can be stored in the server 200. In this way, the robot 40, like a human, can retain vivid memories of times when it felt strong emotions, while retaining simplified memories of times when it did not feel strong emotions.
[0230] Third Embodiment Next, a third embodiment will be described. Here, the description will focus on differences from the first and second embodiments. Description of configurations similar to those of the first and second embodiments will be omitted. Figure 21 is a diagram schematically showing a block configuration of a server 200 and a robot 40 according to the third embodiment.
[0231] 21 , the server 200 in this embodiment differs from the above-described embodiment in that it includes an output control unit 271. When outputting recorded information in the second recording format, the output control unit 271 selects the information in the second recording format to be output based on the information tagged to the information in the second recording format. Specifically, the output control unit 271 selects, as the output target, information in the second recording format that is associated with the current status of the robot 40 and the tagged information.
[0232] For example, the output control unit 271 selects and outputs information in the second recording format in which the attributes of the surrounding people and the attributes of the tagged person are related. Specifically, the output control unit 271 selects and outputs video data in which the ages of the surrounding people and the tagged person match or are similar, and does not select (output) video data in which the ages of the surrounding people and the tagged person match or are similar. Furthermore, the output control unit 271 selects and outputs video data in which the affiliation of the surrounding people and the affiliation of the tagged person match or are similar. In this way, by selecting and filtering the information to be output based on the tagged information, it is possible to narrow down the output to information that is likely to gain sympathy, and to avoid outputting information that is unlikely to gain sympathy.
[0233] Furthermore, even if there is information in the second recording format that correlates the attributes of people around and the attributes of the tagged person, the output control unit 271 will not output (prohibit output of) the information in the second recording format if the emotions of the people around are negative (anxiety, sadness, anger, etc.). This makes it possible to refrain from outputting information in situations where the user is not emotionally comfortable with the information to be output.
[0234] (Fourth Embodiment) Next, a fourth embodiment will be described. Here, differences from the previous embodiments will be mainly described. Descriptions of configurations similar to those of the previous embodiments will be omitted. FIG. 22 schematically illustrates an example of a system 5 according to a fourth embodiment. The system 5 includes a robot 100, a robot 101, a robot 102, and a server 200. User 10a, user 10b, user 10c, and user 10d are users of the robot 100. User 11a, user 11b, and user 11c are users of the robot 101. User 12a and user 12b are users of the robot 102. In the description of this embodiment, user 10a, user 10b, user 10c, and user 10d may be collectively referred to as user 10. User 11a, user 11b, and user 11c may be collectively referred to as user 11. User 12a and user 12b may be collectively referred to as user 12. The robots 101 and 102 have substantially the same functions as the robot 100. Therefore, the system 5 will be described mainly focusing on the functions of the robot 100.
[0235] The robot 100 converses with the user 10 and provides images to the user 10. At this time, the robot 100 cooperates with a server 200 or the like with which it can communicate via a communication network 90 to converse with the user 10 and provide images, etc. to the user 10. For example, the robot 100 not only learns appropriate conversation by itself, but also cooperates with the server 200 to learn how to have a more appropriate conversation with the user 10. The robot 100 also records captured image data, etc. of the user 10 in the server 200, and requests the video data, etc. from the server 200 as needed and provides it to the user 10.
[0236] The robot 100 also has an emotional value that indicates the type of its own emotion. For example, the robot 100 has emotional values that indicate the intensity of each of the following emotions: "joy," "anger," "sorrow," "pleasure," "discomfort," "relief," "anxiety," "sadness," "excitement," "worry," "relief," "fulfillment," "emptiness," and "neutral." For example, when the robot 100 is in a state where the emotional value of excitement is high, the robot 100 speaks at a fast speed when conversing with the user 10. In this way, the robot 100 can express its own emotions through actions.
[0237] The robot 100 also has a function of recognizing the emotions of the user 10. The robot 100 recognizes the emotions of the user 10 by analyzing a facial image of the user 10 acquired by a camera function and a voice of the user 10 acquired by a microphone function. The robot 100 determines an action to be performed by the robot 100 based on the recognized emotions of the user 10. For example, if the robot 100 recognizes that the user 10 looks lonely, it performs an action such as calling out to the user 10.
[0238] The robot 100 stores rules that define actions that the robot 100 will take based on the emotions of the user 10, its own emotions, etc., and performs various actions in accordance with the rules. An example of a rule-based action will be outlined with reference to Figures 23 and 24, etc.
[0239] 23 is a schematic diagram showing actions that the robot 100 performs toward the user 10. The robot 100 has reaction rules for determining actions based on the emotions of the user 10. According to the reaction rules shown in the figure, when the conditions that the emotion of the robot 100 is "normal" and the state of the user is "alone and looks lonely" are satisfied, the robot 100 can change its emotion to "worried" and perform an action of "calling out." Here, each action is associated with a consideration point that indicates the strength of the consideration of the user 10.
[0240] When the robot 100 recognizes based on the reaction rule that the current emotion of the robot 100 is "normal" and that the user 10 appears lonely, the robot 100 increases the emotion value of "worry" of the robot 100. Furthermore, the robot 100 selects the action of "calling out" defined in the reaction rule as the action to be performed on the user 10. For example, when the robot 100 selects the action of "calling out," the robot 100 converts the phrase "What's wrong?", which indicates concern, into a worried voice and outputs it.
[0241] After outputting a voice to the user 10, the robot 100 recognizes a change in the user 10's emotions. For example, if the robot 100 outputs a voice saying "What's wrong?" and the user 10 replies "Thank you. I'm fine!", the robot 100 recognizes that the user 10's emotions have improved by analyzing the user 10's response. Since the user 10's emotions have improved as a result of the action of "calling out," the robot 100 determines that the action of "calling out" was a highly considerate action and increases the consideration points for the action of "calling out." This allows the robot 100 to learn appropriate behavior for the user 10.
[0242] The robot 100 also transmits to the server 200 user reaction information indicating that this action has elicited a positive reaction from the user 10. The user reaction information includes, for example, the condition "looking lonely and alone," the action of "calling out," the fact that the reaction of the user 10 was positive, and the attributes of the user 10.
[0243] The server 200 stores the user reaction information received from the robot 100. The server 200 receives and stores user reaction information not only from the robot 100 but also from each of the robots 101 and 102. The server 200 then analyzes the user reaction information from the robots 100, 101, and 102 to generate recommended reaction rules.
[0244] The robot 100 receives the recommended reaction rules from the server 200 by inquiring about the recommended reaction rules from the server 200. The robot 100 incorporates the recommended reaction rules into the reaction rules stored in the robot 100. This allows the robot 100 to incorporate the reaction rules acquired by the robot 101, the robot 102, etc. into its own reaction rules.
[0245] 24 is a schematic diagram showing a scene in which the robot 100 switches the recording format of data to be recorded in the server 200. The robot 100 has a recording format switching rule for switching the recording format based on the emotion of the user 10. According to the illustrated recording format switching rule, when the conditions that "there is a certain number of family members or more" and "a certain number of smiling family members or more" are satisfied, the robot 100 will be required to "switch the data recording format to a low-compression format."
[0246] During the period when the conditions of the recording format switching rule are not satisfied, the robot 100 continuously transmits highly compressed data such as skeletal data of the user 10 to the server 200, and causes the server 200 to record the data. Note that the skeletal data is an example of shape data that represents the shape of the user. When the robot 100 determines that the conditions of the recording format switching rule are satisfied, it starts transmitting full HD video data and audio data, and causes the server 200 to record low-compression data including the full HD video data and audio data in addition to the skeletal data.
[0247] When the robot 100 wants to provide some entertaining video to the user 10, it requests the server 200 to transmit full HD video data and audio data, and provides the video data received from the server 200 to the user 10. In this way, when the robot 100 encounters a scene that is strongly memorable to the user 10, it can store high-quality video data of that scene in the server 200.
[0248] 25 schematically shows the functional configuration of the robot 100. The robot 100 has a sensor unit 400, a sensor module unit 410, a storage unit 420, a user state recognition unit 430, a condition determination unit 434, a user reaction recognition unit 432, a behavior determination unit 440, an emotion determination unit 442, a rule update unit 422, a communication processing unit 480, a record data generation unit 470, a switching control unit 477, a label generation unit 478, and a control target 452.
[0249] The control object 452 includes a display device, a speaker, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 100 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 100 can be expressed by controlling these motors. Some of the emotions of the robot 100 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 100. The posture, gestures, and facial expressions of the robot 100 are examples of the attitude of the robot 100.
[0250] The sensor unit 400 includes a microphone 401, a 3D depth sensor 402, a 2D camera 403, and a distance sensor 404. In addition to these, the sensor unit 400 may also include a clock, a gyro sensor, a touch sensor, a sensor for motor feedback, and the like.
[0251] 25, the components of the robot 100 excluding the control target 452 and the sensor unit 400 are examples of components included in the behavior control system of the robot 100. The behavior control system of the robot 100 controls the control target 452.
[0252] The storage unit 420 includes a reaction rule 421, a perception confirmation rule 417, a semantic understanding rule 419, a recording format switching rule 471, and a person DB 424. At least a portion of the storage unit 420 is implemented by a storage medium such as a memory. The person DB 424 stores facial images of the user 10, attribute information of the user 10, and the like. Note that, among the components of the robot 100 shown in FIG. 518, the functions of the components excluding the control target 452, the sensor unit 400, and the storage unit 420 can be realized by the CPU operating based on a program. For example, the functions of these components can be implemented as CPU operations by operating an operating system (OS) and a program running on the OS.
[0253] The sensor module unit 410 includes a voice emotion recognition unit 411, a speech understanding unit 412, a facial expression recognition unit 413, and a face recognition unit 414. Information detected by the sensor unit 400 is input to the sensor module unit 410. The sensor module unit 410 analyzes the information detected by the sensor unit 400 and outputs the analysis result to the user state recognition unit 430.
[0254] The voice emotion recognition unit 411 of the sensor module unit 410 analyzes the voice of the user 10 detected by the microphone 401 and recognizes the emotion of the user 10. For example, the voice emotion recognition unit 411 extracts feature quantities such as frequency components of the voice and recognizes the emotion of the user 10 based on the extracted feature quantities. The speech understanding unit 412 analyzes the voice of the user 10 detected by the microphone 401 and outputs text information representing the content of the utterance of the user 10.
[0255] The facial expression recognition unit 413 recognizes the facial expression and emotion of the user 10 from the image of the user 10 captured by the 2D camera 403. For example, the facial expression recognition unit 413 recognizes the facial expression and emotion of the user 10 based on the shape, positional relationship, etc. of the eyes and mouth.
[0256] The face recognition unit 414 recognizes the face of the user 10. The face recognition unit 414 recognizes the user 10 by matching a face image stored in the person DB 424 with a face image of the user 10 captured by the 2D camera 403.
[0257] The user state recognition unit 430 recognizes the state of the user 10. The user state recognition unit 430 includes a perception processing unit 416 and a semantic understanding unit 418. Information analyzed by the sensor module unit 410 is output to the perception processing unit 416. The perception processing unit 416 mainly performs processing related to perception using the analysis results of the sensor module unit 410 and the perception confirmation rules 417 in the storage unit 420. For example, the perception processing unit 416 generates perception information such as "Dad is alone" or "There is a 90% chance that Dad is not smiling."
[0258] The semantic understanding unit 418 performs processing to understand the meaning of the perceptual information generated by the perceptual processing unit 416 using semantic understanding rules 419 in the storage unit 420. For example, the semantic understanding unit 418 generates semantic information such as "Dad is alone and looks lonely." Note that perceptual information and semantic information are examples of information representing the state of the user 10. Here, the information representing the state of the user 10 may include information representing the emotions of the user 10.
[0259] The semantic information generated by the semantic understanding unit 418 is output to the condition determination unit 434 , the user reaction recognition unit 432 , and the label generation unit 478 .
[0260] The condition determination unit 434 determines whether the conditions defined in the reaction rules 421 are satisfied based on the semantic information and the reaction rules 421 in the storage unit 420. The reaction rules 421 are an example of determination information for determining the behavior of the robot 100. As described above, the reaction rules 421 associate multiple behaviors with consideration points for each of the multiple behaviors. The reaction rules 421 also include information for determining behavior based on an emotional value indicating the emotion of the robot 100 and the state of the user 10. In this way, the storage unit 420 functions as an example of a behavior information storage unit that stores information for determining behavior based on the state of the user.
[0261] The condition determination unit 434 determines that the conditions stored in the reaction rules 421 are satisfied. The behavior determination unit 440 determines an behavior to be executed from among a plurality of behaviors stored in the reaction rules 421 in association with the conditions determined to be satisfied by the condition determination unit 434. Specifically, the behavior determination unit 440 determines an behavior to be executed based on the consideration points of each of the plurality of behaviors stored in the reaction rules 421 in association with the conditions determined to be satisfied by the condition determination unit 434. In this way, the behavior determination unit 440 determines an behavior to be executed based on the state of the user 10 recognized by the user state recognition unit 430 and the information in the reaction rules 421.
[0262] The emotion determination unit 442 determines an emotion value indicating the current emotion of the robot 100 based on the semantic information generated by the semantic understanding unit 418. In this way, the emotion determination unit 442 determines an emotion value indicating the current emotion of the robot 100 based on the state of the user recognized by the user state recognition unit 430.
[0263] The behavior control unit 450 controls the control target 452 based on the behavior determined by the behavior determination unit 440. For example, when the behavior determination unit 440 determines to speak, the behavior control unit 450 causes a speaker included in the control target 452 to output a voice. At this time, the behavior control unit 450 determines the speech rate of the voice based on the emotion value updated by the emotion determination unit 442. For example, the behavior control unit 450 determines a faster speech rate the greater the emotion value of excitement. In this way, the behavior control unit 450 determines the execution form of the behavior determined by the behavior determination unit 440 based on the emotion value determined by the emotion determination unit 442.
[0264] The user reaction recognition unit 432 recognizes a change in the user 10's emotions in response to the execution of the action determined by the action determination unit 440. The change in the user 10's emotions is an example of the user 10's reaction. That is, the user reaction recognition unit 432 recognizes the user 10's reaction to the execution of the action determined by the action determination unit 440. The user reaction recognition unit 432 may recognize a change in emotions based on the user 10's voice or facial expression. Alternatively, the user reaction recognition unit 432 may recognize a change in the user 10's emotions based on the detection of an impact by a touch sensor included in the sensor unit 400. For example, the user reaction recognition unit 432 may recognize that the user 10's emotions have worsened when an impact is detected by a touch sensor included in the sensor unit 400.
[0265] Information indicating the reaction of the user 10 recognized by the user reaction recognition unit 432 is output to the emotion determination unit 442, the communication processing unit 480, and the label generation unit 478. As described above, when the user state recognition unit 430 recognizes that the user 10 looks lonely, the emotion determination unit 442 may increase the sadness emotion value in accordance with the reaction rule 421. Furthermore, when the user reaction recognition unit 432 recognizes that the user 10 is smiling, the emotion determination unit 442 may increase the joy emotion value in accordance with the reaction rule 421. In this way, the emotion determination unit 442 updates the emotion value based on at least one of the user's state recognized by the user state recognition unit 430 and the user's reaction to the execution of the action determined by the action determination unit 440.
[0266] Furthermore, the rule update unit 422 changes the consideration points stored by the reaction rules 421 in association with the behavior determined by the behavior determination unit 440, based on the change in the user 10's emotions recognized by the user reaction recognition unit 432. Specifically, the rule update unit 422 changes the consideration points of the behavior determined by the behavior determination unit 440, among the consideration points of the multiple behaviors stored by the reaction rules 421 in association with the conditions determined to be satisfied by the condition determination unit 434, based on the change in the user 10's emotions recognized by the user reaction recognition unit 432. For example, when the user reaction recognition unit 432 recognizes that the user 10's emotions have improved, the rule update unit 422 increases the consideration points stored by the reaction rules 421 in association with the conditions determined to be satisfied by the condition determination unit 434 and the behavior determined by the behavior determination unit 440. On the other hand, when the user reaction recognition unit 432 recognizes that the user 10's emotions have worsened, the rule update unit 422 reduces the consideration points stored by the reaction rule 421 in association with the condition determined to be satisfied by the condition judgment unit 434 and the action determined by the action determination unit 440.
[0267] The communication processing unit 480 is responsible for communication with the server 200. As described above, the communication processing unit 480 transmits user reaction information to the server 200. In this way, the communication processing unit 480 transmits to the server 200 the state of the user 10 recognized by the user state recognition unit 430, the action determined by the action determination unit 440, and the reaction of the user 10 recognized by the user reaction recognition unit 432. Furthermore, when the action determination unit 440 is unable to determine the action to be performed based on the state of the user 10 and the reaction rule 421, the communication processing unit 480 receives a recommended reaction rule from the server 200. In this way, when the action determination unit 440 is unable to determine the action to be performed based on the state of the user 10 and the reaction rule 421, the communication processing unit 480 receives from the server 200 an action appropriate to the state of the user 10 recognized by the user state recognition unit 430.
[0268] The rule update unit 422 updates the reaction rules 421 based on the recommended reaction rules that the communication processing unit 480 receives from the server 200. In this way, the rule update unit 422 updates the reaction rules 421 based on the actions that the communication processing unit 480 receives from the server 200.
[0269] In addition, if the behavior decision unit 440 is unable to decide on an action to take toward the user based on the state of the user 10 recognized by the user state recognition unit 430, the emotion decision unit 442 may change the emotion value representing the intensity of sadness to a value representing a greater intensity of sadness.
[0270] The recording data generation unit 470 generates data to be recorded in the server 200 based on the information detected by the sensor unit 400. The communication processing unit 480 transmits the information generated by the recording data generation unit 470 to an external server and causes the information to be recorded in the server.
[0271] For example, the recording data generation unit 470 continuously generates skeletal data for recording based on skeletal information continuously detected by the 3D depth sensor 402. The recording data generation unit 470 also generates video data for recording based on video information captured by the 2D camera 403. For example, the recording data generation unit 470 generates full HD video data. The recording data generation unit 470 also generates audio data for recording based on audio information continuously detected by the microphone 401.
[0272] Here, the video data is, for example, full HD video data, and contains more information than the skeletal data. Therefore, the skeletal data can be considered to be highly compressed data, and the video data can be considered to be low-compression data. In this way, the skeletal data is an example of information in a first recording format, and the video data is an example of information in a second recording format that contains more information than the information in the first recording format. In this way, the recording data generation unit 470 continuously processes at least a portion of the information detected by a sensor that continuously detects the user 10, and generates information in the first recording format or information in a second recording format that contains more information than the information in the first recording format.
[0273] Here, the perception processing unit 416 perceives the number of recognized family members and the number of family members whose smiles are detected, based on at least a portion of the information detected by the sensor unit 400. As described above, the perception processing unit 416 perceives smiling users based on at least one of the images of the users 10 and the voices of the users 10 detected by the sensor unit 400. Here, a smile is an example of information representing the intensity of the emotion of the user 10. Therefore, the number of family members whose smiles are detected is an example of the total value of the intensity of the emotions of the multiple users 10. In this way, the perception processing unit 416 detects the intensity of the user's emotion based on at least a portion of the information detected by the sensor unit 400.
[0274] Then, the perception processing unit 416 perceives whether or not a certain number of family members have been recognized and whether or not a certain number of smiles of the family members have been detected, based on the recording format switching rule 471. Then, when the switching control unit 477 is causing the recording data generation unit 470 to generate skeletal data, the switching control unit 477 causes the recording data generation unit 470 to start generating video data in response to the perception processing unit 416 recognizing and detecting a certain number of smiles of the family members. In this way, when the switching control unit 477 is causing the recording data generation unit 470 to generate information in the first recording format, the switching control unit 477 switches the recording format of the information generated by the recording data generation unit 470 from the first recording format to the second recording format, which has a larger amount of information than the information in the first recording format, in response to the number of multiple users 10 exceeding a predetermined number and the total value of the intensity of the detected emotions exceeding a predetermined threshold.
[0275] The label generation unit 478 generates labels to be recorded and attached to the record data. A label is an example of additional information attached to the record data. The label generation unit 478 generates labels including, for example, words such as "Yoshiko's birthday" and "Yoshiko's birthday party." Other examples of labels include "celebration of graduation," "Christmas," and "New Year's." The label generation unit 478 determines a situation, such as a birthday or a birthday party, based on date and time information and the response content recognized by the user reaction recognition unit 432 in response to a question for the user 10, and generates a label based on the determined situation. In this way, the label generation unit 478 generates a label including one or more words that describe the current situation based on at least a portion of the information detected by the sensor unit 400. The communication processing unit 480 associates the labels generated by the label generation unit 478 with the record data generated by the record data generation unit 470, transmits the data to the server 200, and stores the data in the server 200.
[0276] 26 shows a schematic functional configuration of the server 200. The server 200 includes a storage unit 520, a reaction rule generation unit 530, a reaction rule search unit 540, a statistical information control unit 550, an access control unit 560, and a communication processing unit 580.
[0277] The storage unit 520 stores reaction rules 521, reaction rule statistical information 522, and recording data 561. The reaction rules 521 include information similar to that of the reaction rules 421. The reaction rule statistical information 522 includes, in addition to information similar to that of the reaction rules 421, attribute information of users 10, 11, and 12, and information on the proportion of positive reactions obtained. The recording data 561 is data for recording, such as the skeletal data, video data, and audio data described above.
[0278] The communication processing unit 580 is responsible for communication between the robot 100, the robot 101, and the robot 102 and the server 200. The communication processing unit 580 receives user reaction information transmitted from the robot 100. As described above, the robots 101 and 102 also have functions similar to those of the robot 100. Therefore, for example, the robot 101 transmits user reaction information including a reaction to an action performed on the user 11 to the server 200. The communication processing unit 580 receives user reaction information transmitted from the robot 101 and user reaction information transmitted from the robot 102. In this way, the communication processing unit 580 receives from the robot 101 the state of the user 11 recognized by the robot 101, the action determined by the robot 101 based on the state of the user 11, and the reaction of the user 11 to the execution of the action.
[0279] The statistical information control unit 550 accumulates the user reaction information received by the communication processing unit 580 as reaction rule statistical information 522 in the storage unit 520. In this way, the storage unit 520 stores the user states, actions, and user reactions received by the communication processing unit 580 in association with each other. The user reaction information accumulated in the statistical information control unit 550 includes user reaction information regarding user 10, user reaction information regarding user 11, and user reaction information regarding user 12.
[0280] The reaction rule generation unit 530 generates recommended reaction rules based on the user reaction information accumulated in the reaction rule statistical information 522, and stores them as reaction rules 521. Based on the information included in the reaction rule statistical information 522, reaction rules including actions that have received positive reactions are generated as recommended reaction rules.
[0281] Here, an operation will be described when the robot 100 inquires of the server 200 about recommended reaction rules. The robot 100 inquires of the server 200 about recommended reaction rules by transmitting the state of the user 10 recognized by the robot 100 to the server 200. In this case, the reaction rule search unit 540 searches at least one of the reaction rules 521 and the reaction rule statistical information 522 for reaction rules having conditions that are satisfied by the state of the user 10 received by the communication processing unit 580. The communication processing unit 580 transmits the reaction rules searched by the reaction rule search unit 540 to the robot 100 as recommended reaction rules.
[0282] In this way, the reaction rule search unit 540 selects an action that is appropriate for the state of the user 10 received by the communication processing unit 580 from the actions that are stored in the reaction rule statistical information 522 in association with the state that matches the state of the user 10 received by the communication processing unit 580, based on the reaction of the user 10 that is stored in the action result storage unit in association with the state that matches the state of the user 10 received by the communication processing unit 580. The communication processing unit 580 transmits the action selected by the reaction rule search unit 540 to the robot 100.
[0283] The access control unit 560 stores the recording data that the communication processing unit 580 receives from the robot 100, the robot 101, and the robot 102 in the recording data 561 of the storage unit 520. Furthermore, when the communication processing unit 580 receives a request to transmit recording data, the access control unit 560 reads the requested recording data from the recording data 561 and causes the communication processing unit 580 to transmit it.
[0284] 26, the functions of the components of the robot 100 except for the storage unit 420 can be realized by a CPU operating based on a program. For example, the functions of these components can be implemented as CPU operations by operating an operating system (OS) and a program running on the OS.
[0285] 27 schematically illustrates information perceived by the perception processing unit 416. When person detection information 601 from the 3D depth sensor 402, face detection information 602 from the face recognition unit 414, and user recognition information 603 are input to the perception processing unit 416, the perception processing unit 416 integrates the person detection information 601, face detection information 602, and user recognition information 603 to generate perception information 611, perception information 612, and perception information 613. Next, when smile detection information 604 from the facial expression recognition unit 413 is input, the perception processing unit 416 converts the smile detection information 604 to generate perception information 614. Furthermore, based on the probability that the perception information 614 is not a smile, the perception processing unit 416 perceives, "Are you lonely, Dad?" (perception information 615).
[0286] Here, the perception confirmation rule 417 includes a perception confirmation rule 630, which is a perception confirmation rule for when the user perceives "I feel lonely?" and has a condition part that "someone is alone" and "the probability of smiling is 20% or less," and a conclusion part that "checks whether there is any speech from the microphone for 30 seconds." The perception processing unit 416 determines that the condition part of the perception confirmation rule 630 is satisfied based on the perception information 613 and the smile detection information 604, and executes confirmation of the detection result by the microphone 401 based on the conclusion part of the perception confirmation rule 630. Specifically, the perception processing unit 416 causes the speech understanding unit 412 to analyze the speech output from the microphone 401.
[0287] Here, if no voice is detected by microphone 401 for 30 seconds, speech understanding unit 412 outputs timeout information indicating that no speech has been detected for 30 seconds. When timeout information 605 from speech understanding unit 412 is input, perception processing unit 416 converts timeout information 605 into perception information 616 that says, "Dad isn't saying anything."
[0288] In this way, the perceptual processing unit 416 generates a perceptual history 610 including perceptual information 611, perceptual information 612, perceptual information 613, perceptual information 614, perceptual information 615, and perceptual information 616. The perceptual processing unit 416 outputs the generated perceptual information to the semantic understanding unit 418.
[0289] 28 schematically shows information understood by the semantic understanding unit 418. The semantic understanding unit 418 receives the perceptual history 610 generated by the perceptual processing unit 416. The semantic understanding unit 418 generates semantic information 711, semantic information 712, semantic information 713, semantic information 714, and semantic information 715 from perceptual information 611, perceptual information 612, perceptual information 613, perceptual information 614, and perceptual information 615, respectively.
[0290] When generating the semantic information 714 from the perceptual information 614, the semantic understanding unit 418 vagues the numerical information "90% probability" contained in the perceptual information 614 to information "not very likely." In this way, the semantic understanding unit 418 generates the semantic information 714 by vaguely interpreting the perceptual information 614.
[0291] Here, the meaning understanding rule 419 includes a meaning understanding rule 730, which is a meaning confirmation rule for when it understands "Maybe he's lonely?" and has a condition part of "someone is alone," "not smiling much," and "not saying anything," and a conclusion part of understanding that "someone is alone and looks lonely." When the meaning understanding unit 418 determines, based on the meaning understanding rule 730, that the semantic information 713, the semantic information 714, and the perceptual information 616 satisfy the condition part of the meaning understanding rule 730, it generates semantic information 717, "Daddy is alone and looks lonely." In this way, the meaning understanding unit 418 uses the meaning understanding rule 730 to integrate the semantic information 714 and the perceptual information 616 to generate the semantic information 717.
[0292] In this way, the semantic understanding unit 418 generates a semantic history 710 including semantic information 711, semantic information 712, semantic information 713, semantic information 714, semantic information 715, and semantic information 717. The semantic information generated by the semantic understanding unit 418 is output to the condition determination unit 434, the user reaction recognition unit 432, and the label generation unit 478.
[0293] As explained in relation to Figures 27 and 28, the perception processing unit 416 and the semantic understanding unit 418 vaguely store the meaning of the perception result using patterned language, even though the accuracy of the perception result is lost.
[0294] 29 shows in table form the information contained in the reaction rule 421. The reaction rule 421 includes a condition part including the conditions of "own emotions" and "user state", a conclusion part including "emotional changes" and "behavior", and "attached information" including "consideration points" and "praise points".
[0295] The "Own emotion" condition is a condition related to the robot 100's own emotion. The "User state" is a condition related to the state of the user 10. The "Emotion change" indicates how the emotion of the robot 100 is changed. The "Action" indicates the action that the robot 100 can perform. The "Consideration point" indicates the degree of consideration for the user 10. The "Consideration point" is an example of appropriateness.
[0296] As shown in the figure, the condition part of the reaction rule 421 includes a "self-emotion" condition of "normal" and a "user state" of "someone looks lonely and alone." The conclusion part corresponding to this condition part includes an "emotional change" that indicates an emotional change of "worried." The conclusion part corresponding to this condition part also includes "actions" indicating actions of "calling out," "only returning worried gestures," and "giving a greeting." The "attached information" also associates 30 points, 20 points, and 10 points as "consideration points" for the actions of "calling out," "only returning worried gestures," and "giving a greeting," respectively. The "attached information" also associates 50 points, 30 points, and 30 points as "praise points" for the actions of "calling out," "only returning worried gestures," and "giving a greeting," respectively.
[0297] In this way, the reaction rule 421 includes information indicating a condition indicating a state of the user that should be satisfied and an action associated with the condition. The reaction rule 421 also includes information indicating a condition indicating an emotion of the robot 100 that should be satisfied and an action associated with the condition. The reaction rule 421 also includes emotion change information for changing the emotion value based on the emotion value of the robot 100 and the state of the user 10. The emotion determination unit 442 then updates the emotion value based on the emotion change information stored by the reaction rule 421 in association with the emotion value determined by the emotion determination unit 442 and the state of the user 10 recognized by the user state recognition unit 430.
[0298] 30 schematically shows processing based on the reaction rules 421. When the semantic information 717 "Daddy is alone and looks lonely" is input from the semantic understanding unit 418, the condition determination unit 434 searches the reaction rules 421 for a condition part that is satisfied by the state of the user 10, "Daddy is alone and looks lonely," and the current emotion of the robot 100, which is "normal."
[0299] The condition determination unit 434 determines that the state of the user 10, "Daddy looks lonely and alone," and the emotion of "normal" satisfy the condition of the reaction rule 810 in the reaction rule 421. As a result, the action determination unit 440 identifies "calling out," "only responding with worried gestures," and "greeting" defined in the reaction rule 810 as candidate actions 910. Then, the action determination unit 440 selects an action to actually be performed from the identified candidate actions 910. For example, the action determination unit 440 selects one action with a selection probability according to the size of the consideration point. Here, it is assumed that the action determination unit 440 selects the action of "calling out."
[0300] Furthermore, the emotion determination unit 442 updates the emotion value based on the change in the emotion of "worried" defined in the reaction rule 810. For example, the emotion determination unit 442 increases the emotion value of "worried," which represents the intensity of anxiety. In this way, the emotion determination unit 442 updates the emotion value based on the state of the user 10 recognized by the user state recognition unit 430, based on the reaction rule 421. For example, when the user state recognition unit 430 recognizes that the user 10 is in a state of feeling lonely, the emotion determination unit 442 changes the emotion value representing the intensity of anxiety to a value representing a greater intensity of anxiety.
[0301] Next, the behavior control unit 450 outputs a voice of the utterance content "What's wrong?" from the speaker of the control target 452. At this time, the behavior control unit 450 outputs a voice having voice characteristics that express an anxious emotion based on the current emotion value. For example, it generates a voice with a relatively slow speed. Note that the behavior control unit 450 may determine the utterance content "What's wrong?" based on the emotion of "worrying."
[0302] Next, the user reaction recognition unit 432 waits for detection of the reaction of the user 10. For example, when the semantic information 940 "Dad is smiling a little" is output from the semantic understanding unit 418 based on the voice "It's okay. Thank you!" and the facial expression of the user 10 at that time, the user reaction recognition unit 432 determines that the user 10 has changed from feeling "lonely" to feeling "happy." As a result, the rule update unit 422 determines that the user 10's emotions have improved due to the action of "calling out." As a result, the consideration points for the action of "calling out" are increased from 30 points to 40 points. The rule update unit 422 also increases the praise points from 50 points to 60 points.
[0303] Furthermore, the emotion determination unit 442 increases the emotion value of joy because the user 10 feels better due to the action performed by the robot 100. Furthermore, the behavior control unit 450 expresses the joyful emotion through an action based on the increased emotion value of joy. For example, the behavior control unit 450 causes the robot 100 to make a gesture that expresses joy.
[0304] In addition, the communication processing unit 480 transmits to the server 200 user reaction information 960, which includes information contained in the reaction rules 810 excluding the "caring points" and "praise points," information indicating that the reaction of the user 10 was positive, and the attributes of the user 10 recognized as "Dad."
[0305] In this way, the behavior determination unit 440 determines the behavior to be executed based on the emotion value determined by the emotion determination unit 442, the user's state recognized by the user state recognition unit 430, and the reaction rule 421. Specifically, the behavior determination unit 440 determines, as the behavior to be executed, with a higher frequency, from among the plurality of behaviors stored in the reaction rule 421 in association with the condition determined to be satisfied by the condition determination unit 434. Note that the behavior determination unit 440 may select the behavior with the highest reaction point as the behavior to be actually executed. In this case, if there are multiple behaviors with the highest reaction point, the behavior determination unit 440 may randomly select one behavior to be actually executed from the plurality of behaviors with the highest reaction point. In this case, the behavior determination unit 440 may select one behavior to be actually executed with a selection probability corresponding to the praise point.
[0306] 31 schematically shows an example of an operational flow relating to an operation for determining an action in the robot 100. The operational flow shown in FIG. 31 is started when, for example, semantic information including the emotion of the user 10 is input.
[0307] The condition determination unit 434 searches for a condition part that is satisfied by the emotion of the robot 100 and the input semantic information among the condition parts included in the reaction rule 421 (S1002). The condition determination unit 434 determines whether or not there is a reaction rule whose condition part is satisfied (S1004). If it determines that there is a reaction rule whose condition part is satisfied, the behavior determination unit 440 selects one behavior to be performed on the user 10 from the behaviors defined in the conclusion parts of the searched reaction rules (S1006). Furthermore, the emotion determination unit 442 updates the emotion value based on the emotion change defined in the conclusion part of the searched reaction rule (S1008).
[0308] Next, the behavior control unit 450 controls the control object 452 to execute the behavior selected in S1006 (S1010). At this time, the behavior control unit 450 controls the control object 452 based on the emotion value updated in S1008. For example, when expressing the emotion of "worry," the behavior control unit 450 controls the control object 452 so that the posture of the robot 100 assumes a worried posture.
[0309] Next, the user reaction recognition unit 432 monitors for input of semantic information representing the reaction of the user 10 (S1012). When the semantic information representing the reaction of the user 10 is input, the user reaction recognition unit 432 determines whether the user's reaction is positive or not based on the input semantic information (S1016).
[0310] If the determination in S1016 determines that the reaction is positive, the emotion determination unit 442 increases the emotion value of "joy" (S1018). The rule update unit 422 also increases the consideration points and praise points corresponding to the performed action. On the other hand, if the determination in S1016 determines that the reaction is negative, the emotion determination unit 442 increases the emotion value of "sadness" (S1022). The rule update unit 422 also decreases the consideration points and praise points corresponding to the performed action (S1024).
[0311] In this way, when the user's reaction recognized by the user reaction recognition unit 432 is not bad, the rule update unit 422 increases the consideration points stored by the reaction rules 421 in association with the conditions determined to be satisfied by the condition determination unit 434 and the actions determined by the action determination unit 440. On the other hand, when the user's reaction recognized by the user reaction recognition unit 432 is bad, the rule update unit 422 decreases the consideration points stored by the reaction rules 421 in association with the conditions determined to be satisfied by the condition determination unit 434 and the actions determined by the action determination unit 440.
[0312] Furthermore, after the behavior control unit 450 executes the behavior determined by the behavior determination unit 440 in the execution mode determined according to the emotion, the emotion determination unit 442 further changes the emotion value based on the user's reaction to the execution of the behavior. Specifically, if the user's reaction to the behavior determined by the behavior determination unit 440 being performed on the user in the execution mode determined by the behavior control unit 450 is not negative, the emotion determination unit 442 changes the emotion value representing the intensity of joy to a value representing a greater intensity of joy. Furthermore, if the user's reaction to the behavior determined by the behavior determination unit 440 being performed on the user in the execution mode determined by the behavior control unit 450 is negative, the emotion determination unit 442 changes the emotion value representing the intensity of sadness to a value representing a greater intensity of sadness.
[0313] The behavior control unit 450 expresses the emotion of the robot 100 based on the emotion value determined in S1018 or S1022 (S1026). For example, when the emotion value of "happiness" is increased, the behavior control unit 450 controls the control object 452 to make the robot 100 perform a happy gesture. When the emotion value of "sadness" is increased, the behavior control unit 450 controls the control object 452 to make the robot 100 assume a drooping posture. Next, the communication processing unit 480 transmits user reaction information to the server 200 (S1028), and the operation related to determining the behavior of the robot 100 is completed.
[0314] If it is determined in S1004 that no reaction rule exists whose condition part is satisfied, the communication processing unit 480 queries the server 200 for a recommended reaction rule. For example, the communication processing unit 480 transmits to the server 200 information indicating the emotion of the robot 100, information indicating the state of the user represented by the input semantic information, and information indicating the attributes of the user 10 (S1030). The server 200 transmits to the robot 100 a reaction rule suited to the information transmitted in the query as a recommended reaction rule. When the communication processing unit 480 receives the recommended reaction rule from the server 200, the rule update unit 422 incorporates the recommended reaction rule into the reaction rules 421 (S1032), and the process proceeds to S1006. In this case, in S1006 and S1008, an action is selected and an emotion value is updated based on the reaction rule received from the server 200. In this way, when the condition judgment unit 434 determines that none of the conditions included in the reaction rule 421 are satisfied, the communication processing unit 480 receives from the server 200 an action appropriate to the user's state recognized by the user state recognition unit 430.
[0315] According to this operation flow, if there is no reaction rule whose condition part is satisfied, an inquiry about a recommended reaction rule is made. Even if there is a reaction rule whose condition part is satisfied, if the consideration points associated with any of the actions are less than a predetermined value, it may be determined that an action cannot be determined. In this case, the communication processing unit 480 may make an inquiry about a recommended reaction rule. That is, the action determination unit 440 determines, as an action to be executed, an action whose consideration points included in the determination information are higher than a predetermined value from among one or more actions associated with the conditions determined to be satisfied by the condition determination unit 434. If the consideration points of the one or more actions associated with the conditions determined to be satisfied by the condition determination unit 434 are all equal to or less than a predetermined value, the communication processing unit 480 may receive from the server 200 an action appropriate for the state of the user 10 recognized by the user state recognition unit 430. Then, the rule update unit 422 may add the behavior received by the communication processing unit 480 from the server 200 to the reaction rule in association with the condition determined to be satisfied by the condition determination unit 434 .
[0316] The inquiry about the recommended reaction rule may be made periodically by batch processing or the like, in addition to when the behavior decision unit 440 cannot decide on a behavior. The inquiry about the recommended reaction rule may also be made when it is determined that the user 10 is asleep, or when the robot 100 transitions to an idle state or a sleep state. In addition, the inquiry about the recommended reaction rule may be made at any timing when the robot 100 is about to take an action.
[0317] Furthermore, the reaction rule 421 may include a rule that defines an action to be taken when no action to be taken for the user 10 is found. For example, the reaction rule 421 may include a rule that includes a condition part that defines the condition "no action to be taken for the user 10 is found," and a conclusion part that defines the action "query the server 200 for recommended reaction rules." In this way, the action determination part 440 may determine to query the server 200 for recommended reaction rules in accordance with the reaction rule 421.
[0318] 32 shows an example of reaction rule statistical information 522 stored in the server 200. The reaction rule statistical information 522 includes a "reaction rule," an "attribute," a "number of people," and a "reaction result." A "reaction rule" includes a "condition part," a "conclusion part," and a "consideration point." The "condition part," "conclusion part," and "consideration point" of the reaction rule statistical information 522 store information corresponding to the "condition part," "conclusion part," and "consideration point" of the reaction rule 421.
[0319] The "attributes" of the reaction rule statistical information 522 stores the attributes of the user 10. The attributes of the user 10 include information such as age group, gender, role in the household, and area of residence. The "number of people" stores the number of people who contributed to the information accumulated in the reaction rule statistical information 522. The "reaction results" stores information indicating the number of times there were positive reactions and the number of times there were positive and negative reactions.
[0320] When the communication processing unit 580 receives user reaction information from the robot 100, the statistical information control unit 550 stores the "condition part" and the "conclusion part" of the reaction rule included in the user reaction information in the reaction rule statistical information 522, in association with attributes that match the attributes included in the user reaction information. The statistical information control unit 550 also increments the value of the "number of positive reactions" or the value of the "number of negative reactions" based on the good / bad reaction information included in the user reaction information. The statistical information control unit 550 also increments the value of the "number of people" as necessary. For example, if the currently received user reaction information is user reaction information of a user whose user reaction information has been reflected in the same record of the reaction rule statistical information 522 in the past, the value of the "number of people" is not incremented.
[0321] Furthermore, the statistical information control unit 550 calculates recommended consideration points and stores them in the "consideration points" of the "reaction rule." For example, the statistical information control unit 550 calculates a higher value as the recommended consideration points as the value of the "positive count" is larger. Specifically, the statistical information control unit 550 may calculate a higher value as the recommended consideration points as the ratio of the value of the "positive count" to the value of the "negative count" is larger.
[0322] When the communication processing unit 580 receives an inquiry about a recommended reaction rule from the robot 100, the reaction rule searching unit 540 searches for the recommended reaction rule from among the reaction rules stored in the reaction rules 521. For example, the reaction rule searching unit 540 searches the reaction rules 521 for a reaction rule having a "condition part" that matches the "condition part" included in the inquiry. The reaction rule searched by the reaction rule searching unit 540 is transmitted from the communication processing unit 580 to the robot 100 as a response to the inquiry.
[0323] The reaction rule searching unit 540 may search for a recommended reaction rule from among the reaction rules stored in the reaction rule statistical information 522. Specifically, the reaction rule searching unit 540 may search for a reaction rule that has a "condition part" that matches a "condition part" included in the query, and has an "attribute" that matches an "attribute" included in the query, from among the reaction rules stored in the reaction rule statistical information 522. This allows the server 200 to provide the robot 100 with an action that is suited to the attributes of the user 10.
[0324] 32, reaction results are tallied for each user attribute in the reaction rule statistical information 522. As another example of reaction rule statistical information, reaction results may be tallied for each user.
[0325] 33 schematically shows the behavior executed by the robot 100 to determine a label to be attached to the recorded data. When the behavior determination unit 440 determines that it is the birthday of the user 10 based on the date and time information and the attribute information of the user 10, it decides to ask the user 10 a question to confirm whether it is their birthday. In this case, the behavior control unit 450 causes the speaker of the control target 452 to output a voice utterance such as "Mr. / Ms. XX, today is your birthday. Happy birthday!"
[0326] Thereafter, the user reaction recognition unit 432 recognizes whether the reaction of the user 10 was positive or negative based on the content of the utterance of the user 10. For example, if the user 10 replies, "Thank you, I'm happy!", the user reaction recognition unit 432 recognizes that the reaction of the user 10 was positive. If the recognized name of the user 10 is "Yoshiko-chan," the label generation unit 478 generates a label, "Yoshiko-chan's birthday."
[0327] The label generation unit 478 generates a label including information indicating the name of the recognized user 10 and the information "Yoshiko's birthday" as a label to be attached to the record data. The communication processing unit 480 continuously transmits to the server 200 the record data in which the label generated by the label generation unit 478 has been attached to the skeleton data based on the skeleton information acquired by the 3D depth sensor 402. The server 200 stores the record data received from the robot 100 in the record data 561 of the storage unit 520.
[0328] The semantic understanding unit 418 calculates the number of people currently recognized based on current skeletal data based on skeletal information acquired from the 3D depth sensor 402. Furthermore, skeletal data acquired in the same time period in the past is acquired from the server 200, and the average number of people recognized in the same room in the same time period in the past is calculated. Here, the semantic understanding unit 418 recognizes that the situation is unusual if it recognizes that the currently recognized number of people is higher than the average number of people recognized in the same room in the same time period in the past by a predetermined threshold or more. Furthermore, the semantic understanding unit 418 compares the labels attached to the skeletal data acquired in the same time period in the past with the currently recognized user 10, and recognizes that the currently recognized user 10 is a user 10 who is usually not at home. If the behavior control unit 450 recognizes that the situation is unusual, it may control the sensor unit 400 and the sensor module unit 410 to perform more detailed detection operations. For example, the behavior control unit 450 may control the utterance understanding unit 412 to recognize the content of the utterance with higher accuracy. Furthermore, the voice emotion recognition unit 411 may be controlled to recognize emotions with higher accuracy, and the facial expression recognition unit 413 may be controlled to recognize facial expressions with higher accuracy.
[0329] If the meaning understanding unit 418 recognizes that the situation is different from usual, the behavior decision unit 440 decides to ask a question to confirm the situation to the user 10. For example, if the label "Yoshiko's birthday" has been determined and it is determined that many people have been recognized, the behavior control unit 450 causes the speaker of the control target 452 to output a voice message stating "Are you having a birthday party for Yoshiko?"
[0330] In this case, the user reaction recognition unit 432 recognizes whether the reaction of the user 10 was positive or negative based on the content of the utterance of the user 10. For example, if the user 10 replies "That's right!", the user reaction recognition unit 432 recognizes that the reaction of the user 10 was positive. In this case, the label generation unit 478 generates a label "Yoshiko's birthday party." The labels "Yoshiko's birthday" and "Yoshiko's birthday party" are examples of labels that represent the current situation of the user 10.
[0331] In this way, the robot 100 interacts with the user 10 to confirm the current situation based on at least a part of the information detected by the sensor unit 400. Then, the label generation unit 478 determines the current situation of the user 10 based on the result of the interaction with the user 10, and generates a label including one or more pieces of character information representing the current situation of the user 10 based on the determined information.
[0332] 34 shows an example of a recording format switching rule 471. The recording format switching rule 471 includes a rule 1300 having a condition part that defines the conditions that "a certain number of family members are in front" and "a certain number of smiling family members have been detected," and a conclusion part that defines the action of "switching to a low-compression recording format." The recording format switching rule 471 also includes a rule having a condition part that defines the conditions that "the number of family members in front has fallen below a certain number" and "a certain number of angry family members have been detected," and a conclusion part that defines the action of "switching to a high-compression recording format."
[0333] The value of the "predetermined number" may be larger as the number of people in a family increases. For example, if a family has four members, the value of the "predetermined number" may be 3. If a family has five members, the value of the "predetermined number" may be 4. The value of the "predetermined number" may be determined based on the history of the number of users 10 recognized by the meaning understanding unit 418. For example, even if a family has five members, if the meaning understanding unit 418 often only recognizes four members, the value of the "predetermined number" may be determined to be 3.
[0334] 35 shows an example of an operation flow for switching the recording format. This operation flow is initiated when the recording data generation unit 470 starts generating recording data in a highly compressed format.
[0335] In S1402, the perception processing unit 416 starts detecting the number of users 10 belonging to a specific family and the number of users 10 whose smile levels exceed a predetermined value, based on the information output by the sensor unit 400 and the sensor module unit 410. Here, the perception processing unit 416 counts the number of users 10 who are registered as family members in the person DB 424, among the users whose presence is recognized based on the information output by the sensor unit 400 and the sensor module unit 410. The perception processing unit 416 also counts the number of users 10 who are registered as family members in the person DB 424, among the users whose smile levels exceed a predetermined value.
[0336] In S1404, the switching control unit 477 determines whether the condition part of the rule 1300 that specifies switching the recording format to a low-compression format is satisfied. If it is determined that the condition part of the rule 1300 is satisfied, the behavior control unit 450 causes the speaker of the controlled object 452 to output a voice utterance saying, "This looks like fun, so I'll take a picture!" (S1406). This notifies the user 10 that video will be recorded.
[0337] In S1408, the switching control unit 477 determines whether or not there was a negative reaction from the user 10. For example, if no negative words or phrases are recognized in the user 10's voice, the switching control unit 477 determines that there was no negative reaction from the user 10. If it is determined that there was no negative reaction from the user 10, the switching control unit 477 controls the recording data generation unit 470 to start generating full HD video data and audio data in addition to the skeleton data (S1410). In this way, the switching control unit 477 switches the recording format from the high-compression format to the low-compression format. If it is determined in S1408 that there was a negative reaction from the user 10, the processing proceeds to S1404 after a predetermined wait time has elapsed (S1420).
[0338] After switching to the low compression format in S1410, it is determined in S1412 whether the conditional part of rule 1310, which stipulates switching to the high compression format, is satisfied. If it is determined that the conditional part of rule 1310 is satisfied, the switching control unit 477 switches the recording format from the low compression format to the high compression format (S1414). If it is determined in S1412 that the conditional part of rule 1310 is not satisfied, the determination of S1412 is repeated until the conditional part of rule 1310 is satisfied.
[0339] In addition to the determination based on rule 1310 in S1412, the recording format may be switched from the low compression format to the high compression format when an utterance instructing to stop recording in the low compression format, such as "end recording," is recognized. Furthermore, after switching the recording format to the low compression format, the recording format may be switched from the low compression format to the high compression format when a predetermined maximum recording time has elapsed.
[0340] The switching control unit 477 may switch the recording format not only in response to the number of smiles but also in response to a drastic change in the intensity of the emotion of the user 10. For example, the switching control unit 477 may switch the recording format to a low-compression format when a touch sensor included in the sensor unit 400 detects an impact stronger than a predetermined value.
[0341] [Recording Data in a Recording Format According to the Emotion of the User or the Robot] Up to now, it has been explained that the robot 100 switches the recording format of the data to be recorded in the server 200 based on the emotion of the user. For example, the recording format may be a high-compression format or a low-compression format.
[0342] On the other hand, the robot 100 may change the data recording mode based on the emotion of the user or the robot 100. For example, the recording mode may be the type of data, such as text, still images, audio, or moving images.
[0343] As described above, the robot 100 records data (e.g., audio data, skeletal data, image data) obtained by processing information detected by the continuous detection sensors. The robot 100 may also record text data of the information detected by the continuous detection sensors.
[0344] As described above, the robot 100 can express its own emotions using emotion values corresponding to a plurality of types of emotions, such as "joy," "anger," "sorrow," and "happiness."
[0345] The robot 100 can also detect changes and strengths of the user's emotions. Here, the robot 100 also represents the user's emotions using emotion values for each type of emotion.
[0346] That is, the robot 100 can convert emotions of both the user and the robot 100 into structured data based on the type and emotional value of the emotion. For example, the emotion determination unit 442 determines the type of emotion or the emotional value indicating the strength of the emotion of the robot or the user based on information detected by a sensor provided in the robot. Furthermore, the robot 100 specifies the recording mode based on the emotion (type and emotional value) converted into structured data.
[0347] The same method as that described in this paper for converting the emotions of the user and the robot 100 into structured data is disclosed in Japanese Patent No. 6058053. The robot 100 may also identify the emotions of the user or the robot 100 itself by known methods such as a method using an emotion map (see, for example, Japanese Patent No. 6273313) or a method using a neural network (see, for example, Japanese Patent No. 6199927).
[0348] The data to be recorded here includes the data described above. That is, the robot 100 can record information (hereinafter referred to as main information) such as still images of the user (e.g., facial images), moving images, recorded voice, and text (e.g., labels of supplementary information) as data. Furthermore, the robot 100 can record surrounding information such as the smell, temperature, humidity, location, and weather of the location where the robot 100 is located, as well as the number and details of nearby users.
[0349] For example, if the emotion value of the robot 100's negative emotions (such as "anger" and "sadness") increases, there is a possibility that the robot 100 or the user is suffering from a disadvantage or danger. For example, if the user is seriously injured, the emotion value of "sadness" of the robot 100 may increase significantly. Also, for example, if the robot 100 is physically hit, the emotion value of "anger" of the robot 100 may increase significantly.
[0350] In order to be able to later consider countermeasures and avoidance measures for such events that cause harm or danger to the robot 100 or the user, it is desirable that the data be recorded in a (rich) recording format (e.g., long video images, detailed text) that provides more information about the event.
[0351] Furthermore, for example, if the emotion value of the robot 100's positive emotion (such as "joy" or "fun") increases, the user may want to recall the situation at that time in detail later.
[0352] Therefore, when the user's positive emotion value increases, it is desirable that the data be recorded in a format that makes it easier to recall the situation (for example, audio or high-quality video images).This allows the user to refer to the recorded data and recall, for example, a situation when they had fun.
[0353] The relationship between the emotions of the robot 100 and the user and the recording mode is not limited to that described here, but can be set arbitrarily in advance. Here, the relationship between emotions and recording mode is stored in a pattern table and a recording mode table. The pattern table and the recording mode table are stored in the storage unit 520.
[0354] The record data generation section 470 specifies a recording format for recording data indicating information about the user detected by the sensor based on the emotion value. The record data generation section 470 is an example of a specification section.
[0355] The record data generation unit 470 refers to the pattern table and the record mode table stored in the storage unit 520 and specifies a record mode according to the emotion of the robot 100 or the user. The record data generation unit 470 generates data based on the specified record mode. Note that the record data generation unit 470 may specify the record mode based on both the emotion value and the type of emotion, or may specify the record mode based on either one of them.
[0356] The communication processing unit 480 records the generated data in the server 200. That is, the communication processing unit 480 records the data based on the recording mode specified by the recording data generating unit 470. The communication processing unit 480 is an example of a recording control unit.
[0357] An example of a pattern table is shown in Fig. 36. As shown in Fig. 36, the pattern table has columns such as "subject," "type," "emotional value range," and "pattern." "Subject," "type," and "emotional value range" correspond to conditions. The recorded data generation unit 470 identifies a pattern that matches the conditions based on the emotional value of the user or robot 100 and the pattern table.
[0358] For example, if the emotion value of the emotion of "happiness" of the robot 100 is 0.7 (0.5 or more), the recorded data generating unit 470 identifies pattern "P11" that matches the condition based on the record shown in the first row of the pattern table in Fig. 29. Note that the emotion value range here is assumed to be from 0 to 1.
[0359] Also, for example, if the emotion value of the user's emotion of "easy" is 0.9 (0.8 or more), the recorded data generation unit 470 identifies pattern "P14" that matches the conditions based on the record shown in the fourth row of the pattern table in FIG. 29.
[0360] An example of the recording mode table is shown in Fig. 37. The recording data generating unit 470 specifies the recording mode based on the pattern specified based on the pattern table and the recording mode table.
[0361] 37, the recording mode table has a column for "pattern" and a column for indicating the recording mode. The column for indicating the recording mode includes "text," "audio," "still image," "moving image (low quality)," "moving image (high quality)," "length," "capacity," and "peripheral information."
[0362] "Text," "Audio," "Still Image," "Video (Low Quality)," and "Video (High Quality)" are columns indicating the type of data. A value of 1 in the column indicating the type of data means that the corresponding type of data is recorded. The recording data generation unit 470 can specify, as the recording mode, whether the data includes text, audio, still images, or video.
[0363] "Length" is the maximum length of data for which time can be defined (for example, audio and video). Also, "capacity" is the maximum amount of data to be recorded. The recording data generation unit 470 can specify the maximum amount of data as the recording mode. Also, the recording data generation unit 470 can specify the maximum length of data for which time can be defined as the recording mode.
[0364] The "surrounding information" is information that is recorded as data together with the main information (text, audio, image). The robot 100 acquires surrounding information such as odor, temperature, humidity, location, weather, the number of nearby users, and details thereof, using the sensor unit 400. The details of nearby users are information equivalent to the main information (e.g., facial images) related to users around the user whose main information is being recorded.
[0365] The recording data generation unit 470 can specify the recording format to include in the data, which of the following is to be included: the smell, temperature, humidity, location, weather, number and details of nearby users in the location where the robot is present.
[0366] For example, if the recording data generation unit 470 identifies the pattern "P11," it will identify the recording mode as recording up to 500 MB of data including "text" and "still images" along with the temperature, humidity, and position (position information of the robot 100).
[0367] Furthermore, for example, when the recording data generation unit 470 identifies "P12" as the pattern, it identifies a recording mode in which "text," "audio," "still image," "moving image (low quality)," and "moving image (high quality)" are recorded with no limit on length or capacity, along with odor, temperature, humidity, location, weather, the number of nearby users, and their details. Note that the recording mode of pattern "P12" is the recording mode that can include the most information (richest) of all the recording modes that can be represented in the recording mode table.
[0368] The "video (low quality)" and the "video (high quality)" have different resolutions and frame rates. For example, the "video (low quality)" and the "video (high quality)" are full HD images at 30 FPS and 4K images at 60 FPS, respectively.
[0369] The recording mode may vary not only depending on the strength of the emotion (the magnitude of the emotion value) but also on the type of emotion. That is, the recorded data may differ depending on the type of emotion felt by the robot 100 or the user. This can be achieved by setting the pattern table and the recording mode table.
[0370] For example, for the emotion type "joy", a setting may be made so that a still image is recorded, and no moving image is recorded. Also, for the emotion type "pleasure", a setting may be made so that a moving image is recorded.
[0371] Furthermore, for image and audio data, targets may be set for each type of emotion. For example, for the emotion type "happy," a paper-cut image of the face and audio of another user (partner user) who is having a conversation with the target user may be set to be recorded. For example, for the emotion type "anger," a paper-cut image of the face of the other user may be set to be recorded.
[0372] Furthermore, as shown in the recording mode table of Fig. 37, the peripheral information to be recorded differs depending on the type of emotion. For example, when the type of emotion of the robot 100 is "sadness" (pattern "P11"), "weather" is not recorded as peripheral information, whereas when the type of emotion of the robot 100 is "anger" (pattern "P12"), "weather" is recorded as peripheral information.
[0373] The robot 100 records the data based on the specified recording format. That is, the recording data generating unit 470 generates the data in accordance with the specified recording format.
[0374] The communication processing unit 480 causes the generated data to be recorded in the server 200 .
[0375] An example of a process flow for recording data in a recording format according to an emotion is shown in Fig. 38. As shown in Fig. 41, first, the robot 100 acquires the type and emotion value of the emotion of the user or the robot 100 itself (step S2101).
[0376] Next, the robot 100 identifies a pattern that matches the conditions of the type and emotion value of the user or the robot 100 (step S2102). For example, the robot 100 identifies the pattern by referring to a pattern table.
[0377] Next, the robot 100 identifies a recording mode corresponding to the identified pattern (step S2103). For example, the robot 100 identifies the pattern by referring to a recording mode table.
[0378] The robot 100 generates data based on the recording mode (step S2104). Then, the robot 100 records the generated data (step S2105). For example, the robot 100 causes the server 200 to record the data.
[0379] In this way, the robot 100 can record data in a recording mode according to the emotion, thereby appropriately recording data according to the emotion of the user or the robot 100. Furthermore, the robot 100 can assist in dealing with disadvantages and dangers that may arise for the robot 100 or the user. In addition, the robot 100 can thereby assist in remembering situations when the user wants to remember.
[0380] Fifth Embodiment Next, a fifth embodiment will be described. Here, the differences from the above-described embodiments will be mainly described. Descriptions of the same configurations as the above-described embodiments will be omitted.
[0381] 39 schematically illustrates an example of a system 5 according to the fifth embodiment. The system 5 includes a robot 100, a robot 101, a robot 102, and a server 200. A user 10a, a user 10b, a user 10c, and a user 10d are users of the robot 100. A user 11a, a user 11b, and a user 11c are users of the robot 101. A user 12a and a user 12b are users of the robot 102.
[0382] The appearance of the robot may be a human-like appearance like the robots 100 and 101, or may be a stuffed toy like the robot 102. The appearance of the robot 102 is that of a stuffed toy, which is thought to make it particularly easy for children to relate to it.
[0383] The robot 100 converses with the user 10 and provides images to the user 10. At this time, the robot 100 cooperates with a server 200 or the like with which it can communicate via a communication network 90 to converse with the user 10 and provide images, etc. to the user 10. For example, the robot 100 not only learns appropriate conversation by itself, but also cooperates with the server 200 to learn how to have a more appropriate conversation with the user 10. The robot 100 also records captured image data, etc. of the user 10 in the server 200, and requests the video data, etc. from the server 200 as needed and provides it to the user 10.
[0384] Furthermore, the robot 100 may be configured to match a sentence generation model (so-called AI (Artificial Intelligence) chat engine) with an emotion engine, thereby determining an action of the robot 100 corresponding to the emotion of the user 10. Specifically, the robot 100 may be configured to recognize the action of the user 10, determine the emotion of the user 10 regarding the user's action, and determine an action of the robot 100 corresponding to the determined emotion.
[0385] More specifically, when the robot 100 recognizes the behavior of the user 10, it automatically generates behavioral content that the robot 100 should take in response to the behavior of the user 10 using a preset sentence generation model. The sentence generation model may be interpreted as an algorithm and calculation for automatic dialogue processing using text. The sentence generation model is publicly known, as disclosed in, for example, Japanese Patent Application Laid-Open No. 2018-081444, and therefore a detailed description thereof will be omitted. Such a sentence generation model is configured using a large-scale language model (LLM). As described above, this embodiment combines a large-scale language model and an emotion engine to reflect the emotions of the user 10 and the robot 100, as well as various linguistic information, in the behavior of the robot 100. In other words, according to this embodiment, a synergistic effect can be obtained by combining the sentence generation model and the emotion engine.
[0386] The robot 100 also has a function of recognizing the behavior of the user 10. The robot 100 recognizes the behavior of the user 10 by analyzing a facial image of the user 10 acquired by a camera function and a voice of the user 10 acquired by a microphone function. The robot 100 determines the behavior to be performed by the robot 100 based on the recognized behavior of the user 10, etc.
[0387] The robot 100 stores rules that define the actions that the robot 100 will take based on the emotions of the user 10, the emotions of the robot 100, and the actions of the user 10, and performs various actions in accordance with the rules.
[0388] Specifically, the robot 100 has reaction rules for determining the behavior of the robot 100 based on the emotions of the user 10, the emotions of the robot 100, and the behavior of the user 10. For example, the reaction rules define the behavior of the robot 100 as "laughing" when the behavior of the user 10 is "laughing." Furthermore, the reaction rules define the behavior of the robot 100 as "apologizing" when the behavior of the user 10 is "angry." Furthermore, the reaction rules define the behavior of the robot 100 as "answering" when the behavior of the user 10 is "asking a question." Furthermore, the reaction rules define the behavior of the robot 100 as "calling out" when the behavior of the user 10 is "sad."
[0389] When the robot 100 recognizes that the behavior of the user 10 is "anger" based on the reaction rules, it selects the behavior of "apologizing" defined in the reaction rules as the behavior to be executed by the robot 100. For example, when the robot 100 selects the behavior of "apologizing," it performs the motion of "apologizing" and outputs a voice representing the words "apologize."
[0390] Furthermore, when the condition that the emotion of the robot 100 is "normal" (i.e., "joy" = 0, "anger" = 0, "sadness" = 0, "happiness" = 0) and the state of the user 10 is "alone and looks lonely" is satisfied, it is determined that the emotion of the robot 100 will change to "worried" and that the robot 100 will be able to perform the action of "calling out."
[0391] When the robot 100 recognizes based on the reaction rule that the current emotion of the robot 100 is "normal" and that the user 10 appears lonely, the robot 100 increases the emotion value of "sadness" of the robot 100. Furthermore, the robot 100 selects the action of "calling out" defined in the reaction rule as the action to be performed toward the user 10. For example, when the robot 100 selects the action of "calling out," the robot 100 converts the phrase "What's wrong?", which indicates concern, into a worried voice and outputs it.
[0392] The robot 100 also transmits to the server 200 user reaction information indicating that this action has elicited a positive reaction from the user 10. The user reaction information includes, for example, the user action of "getting angry," the robot 100 action of "apologizing," the fact that the user 10's reaction was positive, and the attributes of the user 10.
[0393] The server 200 stores the user reaction information received from the robot 100. The server 200 receives and stores user reaction information not only from the robot 100 but also from each of the robots 101 and 102. The server 200 then analyzes the user reaction information from the robots 100, 101, and 102 and updates the reaction rules.
[0394] The robot 100 receives the updated reaction rules from the server 200 by inquiring about the updated reaction rules from the server 200. The robot 100 incorporates the updated reaction rules into the reaction rules stored in the robot 100. This allows the robot 100 to incorporate the reaction rules acquired by the robot 101, the robot 102, etc. into its own reaction rules.
[0395] 40 shows a schematic functional configuration of the robot 100. The robot 100 has a sensor unit 400, a sensor module unit 410, a storage unit 220, a user state recognition unit 231, an emotion determination unit 232, a behavior recognition unit 234, a behavior determination unit 236, a memory control unit 238, a behavior control unit 251, a control target 252, a communication processing unit 280, and an event detection unit 290.
[0396] The control object 252 includes a display device 2521, a speaker 2522, a lamp 2523 (e.g., LEDs in the eyes), and motors 2524 for driving the arms, hands, and feet. The posture and gestures of the robot 100 are controlled by controlling the motors 2524 of the arms, hands, and feet. Some of the emotions of the robot 100 can be expressed by controlling these motors 2524. Furthermore, the facial expressions of the robot 100 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 100. The posture, gestures, and facial expressions of the robot 100 are examples of the attitude of the robot 100.
[0397] The sensor unit 400 includes a microphone 401, a 3D depth sensor 402, a 2D camera 403, a distance sensor 404, an acceleration sensor 405, a thermosensor 406, a touch sensor 407, and an odor sensor 408. In addition to these, the sensor unit 400 may also include a clock, a sensor for motor feedback, etc.
[0398] 40, the components of the robot 100 excluding the control target 252 and the sensor unit 400 are examples of components included in the behavior control system of the robot 100. The behavior control system of the robot 100 controls the control target 252.
[0399] The storage unit 220 includes reaction rules 221 and history data 222. The history data 222 includes the user 10's past emotional values and behavioral history. This emotional value and behavioral history is recorded for each user 10, for example, by being associated with the user 10's identification information. At least a portion of the storage unit 220 is implemented as a storage medium such as a memory. The storage unit 220 may include a person DB that stores the user 10's facial image, attribute information, etc. Note that the functions of the components of the robot 100 shown in FIG. 40 , excluding the control target 252, the sensor unit 400, and the storage unit 220, can be realized by a CPU operating based on a program. For example, the functions of these components can be implemented as CPU operations using operating system (OS) and programs running on the OS.
[0400] The sensor module unit 410 includes a voice emotion recognition unit 211, a speech understanding unit 212, a facial expression recognition unit 213, and a face recognition unit 214. Information detected by the sensor unit 400 is input to the sensor module unit 410. The sensor module unit 410 analyzes the information detected by the sensor unit 400 and outputs the analysis result to the user state recognition unit 231.
[0401] The voice emotion recognition unit 211 of the sensor module unit 410 analyzes the voice of the user 10 detected by the microphone 401 and recognizes the emotion of the user 10. For example, the voice emotion recognition unit 211 extracts feature quantities such as frequency components of the voice and recognizes the emotion of the user 10 based on the extracted feature quantities. The speech understanding unit 212 analyzes the voice of the user 10 detected by the microphone 401 and outputs text information representing the content of the utterance of the user 10.
[0402] The facial expression recognition unit 213 recognizes the facial expression and emotion of the user 10 from the image of the user 10 captured by the 2D camera 403. For example, the facial expression recognition unit 213 recognizes the facial expression and emotion of the user 10 based on the shapes, positional relationships, etc. of the eyes and mouth.
[0403] The face recognition unit 214 recognizes the face of the user 10. The face recognition unit 214 recognizes the user 10 by matching a face image stored in a person DB (not shown) with a face image of the user 10 captured by the 2D camera 403.
[0404] The user state recognition unit 231 recognizes the state of the user 10 based on the information analyzed by the sensor module unit 410. For example, it mainly performs processing related to perception using the analysis results of the sensor module unit 410. For example, it generates perceptual information such as "Dad is alone" or "There is a 90% chance that Dad is not smiling." It then performs processing to understand the meaning of the generated perceptual information. For example, it generates semantic information such as "Dad is alone and looks lonely."
[0405] The emotion determination unit 232 determines an emotion value indicating the emotion of the user 10 based on the information analyzed by the sensor module unit 410 and the state of the user 10 recognized by the user state recognition unit 231. For example, the information analyzed by the sensor module unit 410 and the recognized state of the user 10 are input into a pre-trained neural network to obtain an emotion value indicating the emotion of the user 10.
[0406] Here, the emotion value indicating the emotion of the user 10 is a value indicating the positive or negative emotion of the user. For example, if the user's emotion is a cheerful emotion accompanied by a sense of pleasure or comfort, such as "joy," "pleasure," "comfort," "relief," "excitement," "relief," and "fulfillment," the value is positive, and the more cheerful the emotion, the larger the value. If the user's emotion is a negative emotion, such as "anger," "sorrow," "discomfort," "anxiety," "sorrow," "worry," and "emptiness," the value is negative, and the more unpleasant the emotion, the larger the absolute value of the negative value. If the user's emotion is none of the above ("neutral"), the value is 0.
[0407] Furthermore, the emotion determination unit 232 determines an emotion value indicating the emotion of the robot 100 based on the information analyzed by the sensor module unit 410 and the state of the user 10 recognized by the user state recognition unit 231 .
[0408] The emotion value of the robot 100 includes emotion values for each of a plurality of emotion categories, and is a value (0 to 5) indicating the strength of each of the emotions, for example, "joy," "anger," "sorrow," and "happiness."
[0409] Specifically, the emotion determination unit 232 determines an emotion value indicating the emotion of the robot 100 in accordance with a rule for updating the emotion value of the robot 100, which rule is determined in association with the information analyzed by the sensor module unit 410 and the state of the user 10 recognized by the user state recognition unit 231.
[0410] For example, when the user state recognition unit 231 recognizes that the user 10 looks lonely, the emotion determination unit 232 increases the emotion value of "sadness" of the robot 100. When the user state recognition unit 231 recognizes that the user 10 is smiling, the emotion determination unit 232 increases the emotion value of "joy" of the robot 100.
[0411] The emotion determining unit 232 may determine the emotion value indicating the emotion of the robot 100 by further considering the state of the robot 100. For example, when the remaining battery power of the robot 100 is low or when the surrounding environment of the robot 100 is pitch black, the emotion value of "sadness" of the robot 100 may be increased. Furthermore, when the user 10 continues to talk to the robot 100 despite the remaining battery power being low, the emotion value of "anger" may be increased.
[0412] The behavior recognition unit 234 recognizes the behavior of the user 10 based on the information analyzed by the sensor module unit 410 and the state of the user 10 recognized by the user state recognition unit 231. For example, the information analyzed by the sensor module unit 410 and the recognized state of the user 10 are input into a pre-trained neural network, the probability of each of a plurality of predetermined behavior categories (for example, "laughing," "angry," "asking a question," and "sad") is obtained, and the behavior category with the highest probability is recognized as the behavior of the user 10.
[0413] As described above, in this embodiment, the robot 100 identifies the user 10 and then acquires the content of the user's utterance. When acquiring and using the content of the utterance, the robot 100 obtains the necessary consent in accordance with laws and regulations from the user 10, and the behavior control system of the robot 100 according to this embodiment takes into consideration the protection of the personal information and privacy of the user 10.
[0414] The behavior determination unit 236 determines a behavior corresponding to the behavior of the user 10 recognized by the behavior recognition unit 234 based on the current emotion value of the user 10 determined by the emotion determination unit 232, history data 222 of past emotion values determined by the emotion determination unit 232 before the current emotion value of the user 10 was determined, and the emotion value of the robot 100. In this embodiment, a case is described in which the behavior determination unit 236 uses one most recent emotion value included in the history data 222 as the past emotion value of the user 10, but the disclosed technology is not limited to this aspect. For example, the behavior determination unit 236 may use multiple most recent emotion values as the past emotion value of the user 10, or may use an emotion value from a unit period ago, such as one day ago. Furthermore, the behavior determination unit 236 may determine a behavior corresponding to the behavior of the user 10 by further considering not only the current emotion value of the robot 100 but also the history of the past emotion values of the robot 100. The behavior determined by the behavior determining unit 236 includes gestures made by the robot 100 or speech content of the robot 100 .
[0415] The behavior determination unit 236 according to this embodiment determines the behavior of the robot 100 as a behavior corresponding to the behavior of the user 10, based on a combination of the past and current emotional values of the user 10, the emotional value of the robot 100, the behavior of the user 10, and the reaction rules 221. For example, if the past emotional value of the user 10 is a positive value and the current emotional value is a negative value, the behavior determination unit 236 determines, as a behavior corresponding to the behavior of the user 10, an behavior that will change the emotional value of the user 10 to a positive value.
[0416] The behavior determination unit 236 may determine a behavior corresponding to the behavior of the user 10 based on the emotion of the robot 100. For example, if the robot 100 is verbally abused by the user 10 or is arrogant to the user 10 (i.e., if the user's reaction is poor), if the surrounding noise is loud and the robot 100 cannot detect the voice of the user 10, or if the remaining battery charge of the robot 100 is low, and the emotional value of "anger" or "sadness" of the robot 100 increases, the behavior determination unit 236 may determine a behavior corresponding to the behavior of the user 10, which corresponds to the behavior of the user 10. Furthermore, if the user's reaction is good or the remaining battery charge of the robot 100 is high, and the emotional value of "joy" or "happiness" of the robot 100 increases, the behavior determination unit 236 may determine a behavior corresponding to the behavior of the user 10, which corresponds to the behavior of the user 10. Furthermore, the behavior determination unit 236 may determine a behavior for the user 10 who has increased the emotional value of "joy" or "happiness" of the robot 100 that is different from the behavior for the user 10 who has increased the emotional value of "anger" or "sadness" of the robot 100. In this way, the behavior determination unit 236 may determine a different behavior depending on the emotion of the robot 100 itself or how the user 10 has changed the emotion of the robot 100 through the action of the user 10.
[0417] The reaction rule 221 defines the behavior of the robot 100 according to a combination of the past emotional value and the current emotional value of the user 10, the emotional value of the robot 100, and the behavior of the user 10. For example, if the past emotional value of the user 10 is a positive value and the current emotional value is a negative value, and the user 10 is behaving sadly, a combination of gestures and speech content when asking a question to encourage the user 10 with gestures is defined as the behavior of the robot 100.
[0418] For example, the reaction rule 221 defines behaviors of the robot 100 for all combinations of patterns of the robot 100's emotional values (1296 patterns, which are the fourth power of six values from "0" to "5" for "joy," "anger," "sadness," and "happiness"), patterns of combinations of the user 10's past emotional values and current emotional values, and behavioral patterns of the user 10. That is, for each pattern of the robot 100's emotional values, behaviors of the robot 100 are defined in accordance with the behavioral patterns of the user 10 for each of a plurality of combinations of the user 10's past emotional values and current emotional values, such as negative and negative values, negative and positive values, positive and negative values, positive and positive values, negative and normal values, and normal and normal values. Note that, for example, when the user 10 makes an utterance intending to continue a conversation from a past topic, such as "I want to talk about that topic we talked about last time," the behavior determination unit 236 may transition to an operation mode that determines the behavior of the robot 100 using the history data 222.
[0419] The reaction rule 221 may define at least one of a gesture and a statement as the behavior of the robot 100 for each of the patterns (1296 patterns) of the emotional value of the robot 100. Alternatively, the reaction rule 221 may define at least one of a gesture and a statement as the behavior of the robot 100 for each group of patterns of the emotional value of the robot 100.
[0420] The strength of each gesture included in the behavior of the robot 100 defined in the reaction rule 221 is predetermined. The strength of each utterance included in the behavior of the robot 100 defined in the reaction rule 221 is predetermined.
[0421] The memory control unit 238 determines whether or not to store data including the behavior of the user 10 in the history data 222 based on the predetermined behavior intensity for the behavior determined by the behavior determination unit 236 and the emotion value of the robot 100 determined by the emotion determination unit 232.
[0422] Specifically, if the total intensity value, which is the sum of the sum of the emotion values for each of the multiple emotion classifications of the robot 100, the predetermined intensity for the gestures included in the behavior determined by the behavior determination unit 236, and the predetermined intensity for the speech content included in the behavior determined by the behavior determination unit 236, is equal to or greater than a threshold value, it is determined that data including the behavior of the user 10 is to be stored in the history data 222.
[0423] When the memory control unit 238 decides to store data including the behavior of the user 10 in the history data 222, it stores in the history data 222 the behavior determined by the behavior determination unit 236, information analyzed by the sensor module unit 410 from the present time up to a certain period of time ago (for example, all surrounding information such as data on the sound, images, smells, etc. of the scene), and the state of the user 10 recognized by the user state recognition unit 231 (for example, the facial expression, emotions, etc. of the user 10).
[0424] Furthermore, in this embodiment, the storage control unit 238 switches the information to be recorded depending on the intensity of the emotion of the user 10 determined by the emotion determination unit 232. Specifically, when the intensity of the emotion of the user 10 exceeds a threshold, the storage control unit 238 generates information with a reduced amount of information compared to when the intensity of the emotion does not exceed the threshold, and stores the information in the history data 222. In other words, when the emotion of the user 10 exceeds a predetermined intensity, the storage control unit 238 limits the types and capacity of information to be stored compared to normal times.
[0425] For example, in a case where the user 10 is feeling very sad, the storage control unit 238 performs control so that only points, not detailed information, are recorded in the history data 222. In other words, in this case, the storage control unit 238 records only important information in the history data 222.
[0426] Furthermore, if the intensity of the emotion of the user 10 exceeds a threshold, the storage control unit 238 excludes information about events that occurred within a predetermined period from the records. For example, if the emotion of the user 10 is very happy, the storage control unit 238 deletes from the history data 222 the most recent event that resulted in a negative emotion, and if the emotion of the user 10 is very sad, the storage control unit 238 deletes from the history data 222 the most recent event that resulted in a positive emotion.
[0427] Through these processes, the memory control unit 238 can record only information that is memorable to the user 10 in the history data 222, and exclude information that is less memorable to the user 10 from the history data 222.
[0428] In this way, the memory control unit 238 can limit the recording of events and information that are memorable to the user 10 in the history data 222, and can therefore appropriately record data according to the increasing intensity of the user's emotions.
[0429] Note that when the intensity of the user 10's emotion is below a threshold, the storage control unit 238 may record more information about the pressed information volume in the history data 222 than when the intensity of the emotion exceeds the threshold. In this case, when the intensity of the user 10's emotion exceeds the threshold, the storage control unit 238 will record more detailed data in the history data 222. That is, in this case, it is possible to record more detailed data when the user 10's emotion is heightened.
[0430] Furthermore, the storage control unit 238 may temporarily store information in the history data 222, and then delete (forget) the information after adjusting the retention period depending on subsequent emotional changes of the user 10. For example, when an event occurs that heightens the emotions of the user 10, the storage control unit 238 may record information immediately before the event in the history data 222, and delete the information after a certain period of time has passed.
[0431] The behavior control unit 251 controls the control target 252 based on the behavior determined by the behavior determination unit 236. For example, when the behavior determination unit 236 determines an behavior including speaking, the behavior control unit 251 outputs a sound from a speaker included in the control target 252. At this time, the behavior control unit 251 may determine the speaking rate of the sound based on the emotional value of the robot 100. For example, the behavior control unit 251 determines a faster speaking rate as the emotional value of the robot 100 increases. In this way, the behavior control unit 251 determines the execution form of the behavior determined by the behavior determination unit 236 based on the emotional value determined by the emotional determination unit 232.
[0432] The behavior control unit 251 may recognize a change in the user 10's emotion in response to the execution of the behavior determined by the behavior determination unit 236. For example, the change in emotion may be recognized based on the voice or facial expression of the user 10. Alternatively, the change in emotion of the user 10 may be recognized based on the detection of an impact by a touch sensor included in the sensor unit 400. If an impact is detected by the touch sensor included in the sensor unit 400, the behavior control unit 251 may recognize that the user 10's emotion has worsened. Alternatively, if the detection result of the touch sensor included in the sensor unit 400 indicates that the user 10 is smiling, happy, or the like, the behavior control unit 251 may recognize that the user 10's emotion has improved. Information indicating the user 10's reaction is output to the communication processing unit 280.
[0433] Furthermore, after the behavior control unit 251 executes the behavior determined by the behavior determination unit 236 in the execution mode determined according to the emotion of the robot 100, the emotion determination unit 232 further changes the emotion value of the robot 100 based on the user's reaction to the execution of the behavior. Specifically, the emotion determination unit 232 increases the emotion value of "joy" of the robot 100 when the user's reaction to the behavior determined by the behavior determination unit 236 being performed in the execution mode determined by the behavior control unit 251 is not negative. Furthermore, the emotion determination unit 232 increases the emotion value of "sad" of the robot 100 when the user's reaction to the behavior determined by the behavior determination unit 236 being performed in the execution mode determined by the behavior control unit 251 is negative.
[0434] Furthermore, the behavior control unit 251 expresses the emotion of the robot 100 based on the determined emotion value of the robot 100. For example, when the emotion value of "happiness" of the robot 100 is increased, the behavior control unit 251 controls the control object 252 to make the robot 100 perform a happy gesture. When the emotion value of "sadness" of the robot 100 is increased, the behavior control unit 251 controls the control object 252 to make the robot 100 assume a droopy posture.
[0435] The communication processing unit 280 is responsible for communication with the server 200. As described above, the communication processing unit 280 transmits user reaction information to the server 200. The communication processing unit 280 also receives updated reaction rules from the server 200. When the communication processing unit 280 receives the updated reaction rules from the server 200, it updates the reaction rules 221.
[0436] The event detection unit 290 realizes the output function described above. Details of the event detection unit 290 will be described later.
[0437] The server 200 communicates between the robots 100, 101, and 102 and the server 200, receives user reaction information sent from the robot 100, and updates the reaction rules based on reaction rules that include actions that have received positive reactions.
[0438] Fig. 41 shows an example of an operation flow for determining information to be recorded in the robot 100. The operation flow shown in Fig. 41 is repeatedly executed. At this time, it is assumed that information analyzed by the sensor module unit 410 has been input.
[0439] First, in step S101 , the user state recognition unit 231 recognizes the state of the user 10 based on the information analyzed by the sensor module unit 410 .
[0440] In step S102 , the emotion determination unit 232 determines an emotion value indicating the emotion of the user 10 based on the information analyzed by the sensor module unit 410 and the state of the user 10 recognized by the user state recognition unit 231 .
[0441] In step S103 , the emotion determination unit 232 determines an emotion value indicating the emotion of the robot 100 based on the information analyzed by the sensor module unit 410 and the state of the user 10 recognized by the user state recognition unit 231 .
[0442] In step S104 , the storage control unit 238 selects information to be collected (stored) according to the emotion value of the user 10 determined by the emotion determination unit 232 .
[0443] In step S105 , the storage control unit 238 stores the selected information in the history data 222 .
[0444] As described above, the robot 100 determines an emotion value indicating the emotion of the robot 100 based on the state of the user, and determines whether to store data including the behavior of the user 10 in the history data 222 based on the emotion value of the robot 100. This makes it possible to reduce the capacity of the history data 222 that stores data including the behavior of the user 10. For example, when the robot 100 determines that the state of the user 10 in 10 years will be the same as that of 10 years ago, the robot 100 can read the history data 222 from 10 years ago and present to the user 10 all kinds of peripheral information, such as the state of the user 10 from 10 years ago (e.g., the facial expression, emotions, etc. of the user 10), as well as data on the sounds, images, smells, etc. of the situation.
[0445] Furthermore, the robot 100 can be made to perform an appropriate action in response to the user's 10 action. Conventionally, the user's actions are classified and the robot's actions, including facial expressions and posture, are determined. In contrast, the robot 100 determines the user's current emotional value and performs an action on the user 10 based on the past and current emotional values. Therefore, for example, if the user 10 was cheerful yesterday but is depressed today, the robot 100 can utter, "You were cheerful yesterday, but what's up with you today?" The robot 100 can also utter with gestures. For example, if the user 10 was depressed yesterday but is cheerful today, the robot 100 can utter, "You were depressed yesterday, but you seem cheerful today, don't you?" For example, if the user 10 was cheerful yesterday but is more cheerful today than yesterday, the robot 100 can utter, "You're more cheerful today than yesterday. Has anything better happened than yesterday?" Furthermore, for example, the robot 100 can say to a user 10 whose emotional value is equal to or greater than 0 and whose emotional value fluctuation range continues to be within a certain range, "You've been feeling stable lately, which is good."
[0446] Furthermore, for example, the robot 100 can ask the user 10, "Did you finish the homework you told me about yesterday?" and, if the user 10 replies, "Yes, I did," make a positive utterance such as "Great!" and perform a positive gesture such as clapping or a thumbs-up. Furthermore, for example, if the user 10 utters, "The presentation you gave the day before yesterday went well," the robot 100 can make a positive utterance such as "Good job!" and perform the above-mentioned positive gesture. In this way, the robot 100 can be expected to make the user 10 feel a sense of affinity with the robot 100 by performing an action based on the state history of the user 10.
[0447] In the above embodiment, the robot 100 recognizes the user 10 using a facial image of the user 10. However, the disclosed technology is not limited to this. For example, the robot 100 may recognize the user 10 using a voice uttered by the user 10, an email address of the user 10, an SNS ID of the user 10, an ID card with a built-in wireless IC tag that the user 10 has, or the like.
[0448] The robot 100 is an example of an electronic device equipped with a behavior control system. The application of the behavior control system is not limited to the robot 100, and the behavior control system can be applied to various electronic devices. Furthermore, the functions of the server 200 may be implemented by one or more computers. At least some of the functions of the server 200 may be implemented by a virtual machine. Furthermore, at least some of the functions of the server 200 may be implemented in the cloud.
[0449] (Other Embodiments) The robot 100 described above may be mounted on a stuffed toy, or may be applied to a control device connected wirelessly or by wire to a control target device (speaker or camera) mounted on the stuffed toy.
[0450] The emotion determining unit 232 may determine the emotion of the user in accordance with a specific mapping. Specifically, the emotion determining unit 232 may determine the emotion of the user in accordance with an emotion map (see FIG. 42 ), which is a specific mapping.
[0451] FIG. 42 is a diagram showing an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental state. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.
[0452] (1) For example, if the emotion engine, which is the emotion determination unit 232 of the robot 100, detects emotions at approximately 100 msec, the frequency of determining the reaction action of the robot 100 (for example, a backchannel) may be set at a timing at least equal to the detection frequency (100 msec) of the emotion engine, or may be set at a timing earlier than this. The detection frequency of the emotion engine may be interpreted as a sampling rate.
[0453] By detecting emotions within about 100 msec and immediately performing a corresponding reaction (e.g., nodding), unnatural reactions are eliminated, enabling a natural, well-read dialogue. The reaction (e.g., nodding) is performed according to the directionality and degree (strength) of the mandala in the emotion map 400 of the robot 100. Note that the detection frequency (sampling rate) of the emotion engine is not limited to 100 ms and may be changed depending on the situation (e.g., when playing sports), the user's age, etc.
[0454] (2) The directionality and intensity of emotions may be set in advance in reference to the emotion map 400, and the movement of the interjections and the strength of the interjections may be set. For example, if the robot 100 feels a sense of stability or security, the robot 100 may nod and continue listening. If the robot 100 feels anxious, confused, or suspicious, the robot 100 may tilt its head or stop shaking its head.
[0455] These emotions are distributed in the three o'clock direction of emotion map 400, and usually fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.
[0456] (3) If the robot 100 receives a compliment and feels good, the filler "ah" may be inserted before the line, and if the robot 100 receives harsh words and feels pain, the filler "ugh!" may be inserted before the line. A physical reaction, such as the robot 100 crouching down while saying "ugh!", may also be included. These emotions are distributed around the 9 o'clock position on the emotion map 400.
[0457] (4) In the left half of the emotion map 400, internal sensations (reactions) are more important than situational awareness. This can give the impression of an unconscious reaction.
[0458] When the robot 100 feels a positive feeling in its situational awareness while experiencing an internal sensation (reaction) of understanding, the robot 100 may nod deeply while looking at the other person, or may say "uh-huh." In this way, the robot 100 may generate a behavior that shows a balanced positive feeling toward the other person, that is, tolerance and tolerance toward the other person. Such emotions are distributed around 12 o'clock on the emotion map 400.
[0459] Conversely, even when the robot 100 is aware of an internal sensation (reaction) of discomfort, it may shake its head when it feels disgust, or turn its eye LEDs red and glare at the other person when it feels hatred. These emotions are distributed around the 6 o'clock position on the emotion map 400.
[0460] (5) The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior. Therefore, the further outside emotion map 400 you go, the more visible the emotion becomes (the more it is expressed in behavior).
[0461] (6) When listening to someone while feeling a sense of security, which is distributed around 3 o'clock on the emotion map 400, the robot 100 may nod its head lightly and say "hmm." However, when listening to someone while feeling a sense of love, which is distributed around 12 o'clock, the robot 100 may nod its head strongly.
[0462] The emotion determination unit 232 inputs the information analyzed by the sensor module unit 410 and the recognized state of the user 10 into a pre-trained neural network, obtains emotion values indicating each emotion shown in the emotion map 400, and determines the emotion of the user 10. This neural network is pre-trained based on multiple pieces of training data that are combinations of the information analyzed by the sensor module unit 410, the recognized state of the user 10, and emotion values indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 43. FIG. 43 is a diagram showing another example of an emotion map. FIG. 43 shows an example in which multiple emotions, such as "relieved," "calm," and "reassuring," have similar emotion values.
[0463] Furthermore, the emotion determination unit 232 may determine the emotion of the robot 100 according to a specific mapping. Specifically, the emotion determination unit 232 inputs the information analyzed by the sensor module unit 410, the state of the user 10 recognized by the user state recognition unit 231, and the state of the robot 100 into a pre-trained neural network, obtains emotion values indicating each emotion shown in the emotion map 400, and determines the emotion of the robot 100. This neural network is pre-trained based on a plurality of training data that are combinations of the information analyzed by the sensor module unit 410, the recognized state of the user 10, the state of the robot 100, and emotion values indicating each emotion shown in the emotion map 400. For example, the neural network is trained based on training data indicating that the emotion value of "happy" is "3" when it is recognized from the output of the touch sensor 407 that the robot 100 is being stroked by the user 10, and training data indicating that the emotion value of "anger" is "3" when it is recognized from the output of the acceleration sensor 405 that the robot 100 is being hit by the user 10. Furthermore, this neural network is trained so that emotions that are placed close to each other have similar values, as in the emotion map 900 shown in FIG.
[0464] Furthermore, the emotion determination unit 232 may determine the emotion of the robot 100 based on the behavioral content of the robot 100 generated by the sentence generation model. Specifically, the emotion determination unit 232 inputs the behavioral content of the robot 100 generated by the sentence generation model into a pre-trained neural network, obtains emotion values indicating each emotion shown in the emotion map 400, and integrates the obtained emotion values indicating each emotion with emotion values indicating each current emotion of the robot 100 to update the emotion of the robot 100. For example, the emotion values indicating each obtained emotion and emotion values indicating each current emotion of the robot 100 are averaged and integrated. This neural network is trained in advance based on multiple pieces of training data that are combinations of text indicating the behavioral content of the robot 100 generated by the sentence generation model and emotion values indicating each emotion shown in the emotion map 400.
[0465] For example, if the utterance content of the robot 100, "That's great. You're lucky," is obtained as the behavior content of the robot 100 generated by the sentence generation model, when the text representing this utterance content is input into the neural network, a high value is obtained as the emotion value of the emotion "happy," and the emotion of the robot 100 is updated so that the emotion value of the emotion "happy" becomes high.
[0466] The behavior determination unit 236 generates the robot's behavior content by adding fixed sentences to the text representing the user's behavior, the user's emotions, and the robot's emotions to ask about the robot's behavior content corresponding to the user's behavior and inputting the result into a sentence generation model with an interactive function.
[0467] For example, the behavior determining unit 236 obtains text representing the state of the robot 100 from the emotion of the robot 100 determined by the emotion determining unit 232, using an emotion table such as that shown in Fig. 44. Fig. 44 is a diagram showing an example of an emotion table. In the emotion table, an index number is assigned to each emotion value for each type of emotion, and text representing the state of the robot 100 is stored for each index number.
[0468] When the emotion of the robot 100 determined by the emotion determination unit 232 corresponds to the index number "2", the text "very happy state" is obtained. When the emotions of the robot 100 correspond to multiple index numbers, multiple texts representing the state of the robot 100 are obtained.
[0469] An emotion table such as that shown in FIG. 45 is also prepared for the emotions of the user 10. FIG. 45 is a diagram showing an example of an emotion table. Here, if the user's behavior is speaking "It's AAA," the emotion of the robot 100 is index number "2," and the emotion of the user 10 is index number "3," then the following is input into the sentence generation model, and the content of the robot's behavior is obtained: "The robot is in a very happy state. The user is in a normal happy state. The user spoke to you saying, 'It's AAA.' How would you respond as the robot?" The behavior determination unit 236 determines the robot's behavior from this content of the behavior.
[0470] In this way, the robot 100 can change its behavior according to the index number corresponding to the robot's emotion, so the user gets the impression that the robot 100 has a heart, and is encouraged to take actions such as talking to the robot.
[0471] Furthermore, the behavior determination unit 236 may generate the robot's behavior content by adding not only text representing the user's behavior, the user's emotions, and the robot's emotions, but also text representing the contents of the history data 222, and then adding a fixed sentence for asking about the robot's behavior content corresponding to the user's behavior, and inputting the result into the dialogue function. This allows the robot 100 to change its behavior in accordance with the history data representing the user's emotions and behavior, so that the user has the impression that the robot has individuality and is encouraged to take actions such as talking to the robot. Furthermore, the history data may further include the robot's emotions and actions.
[0472] Specifically, the stuffed toy is configured as follows. For example, the robot 100 may be used as a cohabitant (specifically, a stuffed toy 100N shown in FIGS. 46 and 47 ) that spends daily life with a user 10, and that engages in dialogue with the user 10 based on information about the user's daily life, or that provides information tailored to the user's hobbies and preferences. In this embodiment (and other embodiments), an example will be described in which the control unit of the robot 100 is applied to a smartphone 50.
[0473] The stuffed toy 100N, which is equipped with the function of an input / output device for the robot 100, has a detachable smartphone 50 that functions as the control part of the robot 100, and the input / output device and the housed smartphone 50 are connected inside the stuffed toy 100N.
[0474] As shown in FIG. 46(A), in this embodiment (and other embodiments), the stuffed animal 100N has the appearance of a bear covered in soft fabric. As shown in FIG. 46(B), the space 52 formed inside the stuffed animal 100N contains input / output devices, such as a microphone 401 of the sensor unit 400 located in the ears 54, a 2D camera 403 of the sensor unit 400 located in the eyes 56, and a speaker 51 constituting part of the control target 252 located in the mouth 58. The microphone 401 and speaker 51 do not necessarily need to be separate entities and may be integrated into a single unit. In this case, they should be positioned in a location where speech can be heard naturally, such as at the nose of the stuffed animal 100N. While the stuffed animal 100N has been described as being shaped like an animal, the present invention is not limited to this. The stuffed animal 100N may also be shaped like a specific character.
[0475] The smartphone 50 has the functions of a sensor module unit 410, a storage unit 220, a user state recognition unit 231, an emotion determination unit 232, a behavior recognition unit 234, a behavior determination unit 236, a memory control unit 238, a behavior control unit 251, a communication processing unit 280, and an event detection unit 290, as shown in FIG. 40 .
[0476] As shown in Figure 47, a zipper 62 is attached to a part of the stuffed animal 100N (for example, the back), and by opening the zipper 62, the outside and the space 52 are connected to each other.
[0477] Here, the smartphone 50 is housed in the space 52 from the outside and connected to each input / output device via USB via a USB hub 64 (see Figure 46 (B)), thereby giving it functionality equivalent to that of the robot 100 shown in Figure 39.
[0478] A non-contact type power receiving plate 66 is also connected to the USB hub 64. A power receiving coil 66A is built into the power receiving plate 66. The power receiving plate 66 is an example of a wireless power receiving unit that receives power wirelessly.
[0479] The power receiving plate 66 is disposed near the bases 68 of both feet of the stuffed toy 100N, and is located closest to the mounting base 70 when the stuffed toy 100N is placed on the mounting base 70. The mounting base 70 is an example of an external wireless power transmission unit.
[0480] The stuffed toy 100N placed on the mounting base 70 can be appreciated as an ornament in its natural state.
[0481] In addition, this base portion is formed thinner than the surface thickness of other portions of the stuffed toy 100N, so that it is held closer to the mounting base 70.
[0482] The mounting base 70 is equipped with a charging pad 72. The charging pad 72 incorporates a power transmitting coil 72A, which sends a signal to search for the power receiving coil 66A on the power receiving plate 66. When the power receiving coil 66A is found, a current flows through the power transmitting coil 72A, generating a magnetic field. The power receiving coil 66A reacts to the magnetic field, initiating electromagnetic induction. This causes a current to flow through the power receiving coil 66A, and power is stored in the battery (not shown) of the smartphone 50 via the USB hub 64.
[0483] That is, by placing the stuffed toy 100N on the mounting base 70 as an ornament, the smartphone 50 is automatically charged, so there is no need to remove the smartphone 50 from the space 52 of the stuffed toy 100N to charge it.
[0484] In this embodiment (and other embodiments), the smartphone 50 is housed in the space 52 of the stuffed toy 100N and connected via a wire (USB connection), but this is not limiting. For example, a control device with wireless functionality (e.g., Bluetooth (registered trademark)) may be housed in the space 52 of the stuffed toy 100N and connected to the USB hub 64. In this case, the smartphone 50 and the control device communicate wirelessly without placing the smartphone 50 in the space 52, and the external smartphone 50 connects to each input / output device via the control device, thereby providing functionality equivalent to that of the robot 100. Furthermore, the control device housed in the space 52 of the stuffed toy 100N may be connected via a wire to the external smartphone 50.
[0485] In addition, in this embodiment (and other embodiments), the teddy bear 100N is exemplified, but it may be another animal, a doll, or the shape of a specific character. It may also be dressable. Furthermore, the material of the skin is not limited to cloth, and may be other materials such as soft vinyl, but a soft material is preferable.
[0486] Furthermore, a monitor may be attached to the skin of the stuffed toy 100N to add a control target 252 that provides visual information to the user 10. For example, the eyes 56 may be used as a monitor to express joy, anger, sadness, or happiness by the image reflected in the eyes, or a window may be provided in the abdomen through which the monitor of the built-in smartphone 50 can be projected. Furthermore, the eyes 56 may be used as a projector to express joy, anger, sadness, or happiness by the image projected onto a wall.
[0487] According to another embodiment, an existing smartphone 50 is placed inside the stuffed toy 100N, and the camera 203, microphone 401, speaker 51, etc. are extended from the smartphone 50 at appropriate positions via a USB connection.
[0488] Furthermore, for wireless charging, the smartphone 50 and the power receiving...
Claims
1. A control system for controlling the behavior of an electronic device, comprising: a recording control unit that processes at least a portion of information continuously detected by a sensor to generate and record information in a first recording format or information in a second recording format having a larger amount of information than the information in the first recording format; and a switching control unit that switches the recording format used by the recording control unit from the first recording format to the second recording format when the information in the first recording format is being generated and an emotion of the electronic device or a user exceeds a threshold.
2. The control system according to claim 1, wherein the switching control unit switches the recording format from the first recording format to the second recording format when the user's emotion exceeding the threshold is transmitted to the electronic device.
3. The control system according to claim 1, wherein the switching control unit switches the recording format from the first recording format to the second recording format when an emotion of the electronic device exceeding the threshold is transmitted to the user.
4. The control system according to claim 1, wherein the recording control unit switches a trigger for switching from the first recording format to the second recording format depending on the situation.
5. The control system according to claim 1, wherein the recording control unit records information in the second recording format after tagging the information.
6. The control system according to claim 5, wherein the tagged information includes at least one of surrounding information, smell, temperature, humidity, location information, weather, the number of people in the vicinity, facial expressions, and emotions.
7. The control system of claim 1, further comprising an image generation unit that receives information in the second recording format including video data from a server, and generates an image to be presented to the user based on the video data included in the received information in the second recording format.
8. A control system for controlling the behavior of an electronic device, comprising: a recording control unit that processes at least a portion of information continuously detected by a sensor to generate and record information in a first recording format or information in a second recording format having a larger amount of information than the information in the first recording format; and a switching control unit that switches the recording format by the recording control unit from the first recording format to the second recording format when the information in the first recording format is being generated and an emotion of the electronic device or a user exceeds a threshold; and a server that receives and records the information in the first recording format or the second recording format.
9. A program for controlling the behavior of an electronic device, comprising: a recording control procedure for processing at least a portion of information continuously detected by a sensor to generate and record information in a first recording format or information in a second recording format having a larger amount of information than the information in the first recording format; and a switching control procedure for switching the recording format used by the recording control procedure from the first recording format to the second recording format when the electronic device or the emotion of the user exceeds a threshold value while the information in the first recording format is being generated.
10. A control system comprising: an emotion determination unit that determines an emotion value indicating the strength of an emotion of an electronic device or a user, or a type of emotion, based on information detected by a sensor provided in the electronic device; an identification unit that identifies a manner in which to record data indicating information about the user detected by the sensor based on the information determined by the emotion determination unit; and a recording control unit that records the data based on the manner identified by the identification unit.
11. A control system for controlling the behavior of an electronic device, comprising: an emotion determination unit that determines an emotion of one or both of a user and the electronic device based on at least a portion of information detected by a sensor that continuously detects a user; a memory unit that stores information obtained by processing at least a portion of the information detected by the sensor, in association with the determined emotion of the user and / or the electronic device; and an output control unit that, when the information stored in the memory unit is output by the electronic device, controls to change one or both of the behavior of the electronic device and the output timing of the information in accordance with the emotion associated with the information.
12. A control system for controlling the memory of an electronic device, comprising: an emotion determination unit that determines emotions of a user and the electronic device based on at least a portion of information detected by a sensor that continuously detects a user; a memory unit that stores information obtained by processing at least a portion of the information detected by the sensor, by associating the determined emotions of one or both of the user and the electronic device; and a storage control unit that controls the storage period of the information stored in the memory unit according to the emotion associated with the information.
13. A system for controlling a memory of an electronic device, comprising: an emotion determination unit that determines an emotion of the electronic device based on information detected by a sensor provided in the electronic device; a memory unit that associates and stores the information detected by the sensor provided in the electronic device with the emotion determined by the emotion determination unit; and a record control unit that updates, based on the emotion newly determined by the emotion determination unit, the emotion previously determined by the emotion determination unit, which is associated with information previously detected by a sensor provided in the electronic device and includes elements common to the information newly detected by the sensor provided in the electronic device, based on elements related to the emotion newly determined by the emotion determination unit.
14. A control system comprising: an emotion determination unit that determines the intensity of a user's emotion based on at least a portion of information detected by a sensor that continuously detects the user; and a memory control unit that records information obtained by processing at least a portion of the information detected by the sensor, and switches the information to be recorded depending on whether the intensity of the user's emotion determined by the emotion determination unit exceeds a threshold.
15. A control system comprising: an information acquisition unit that acquires event information regarding an event experienced by a target object whose memory is to be controlled; an emotion determination unit that determines the intensity of the emotion of the target object based on the event information acquired by the information acquisition unit; and a memory control unit that associates the intensity of the emotion determined by the emotion determination unit with the event information and stores it.
16. A control system for controlling the behavior of an electronic device, comprising: a recording control unit that processes at least a portion of information continuously detected by a sensor to generate and record information in a first recording format or information in a second recording format having a larger amount of information than the information in the first recording format; and a switching control unit that switches the recording format used by the recording control unit from the first recording format to the second recording format in response to frequent changes in the electronic device or the emotions of a user when the information in the first recording format is being generated.
17. A control system for controlling the behavior of an electronic device, comprising: a recording control unit that processes at least a portion of information continuously detected by a sensor to generate information in a first recording format or information in a second recording format having a larger amount of information than the information in the first recording format, and tags and records the information in the second recording format; and an output control unit that controls the output of information in the second recording format selected from the information in the second recording format recorded by the recording control unit based on the tagged information.
Citation Information
Patent Citations
Feeling generator and method therefor
JP1998254592A
Terminal and reply information creation program for pseudo conversation
JP2011253389A
Methods for memorizing spacetime patterns, systems for doing so, and computer programs for doing so.
JP2013535067A
User support system, user support program, and user support method
JP2018081444A
Recording control system, system and program
JP6058053B2
Cited By
Customer service dialogue method and device, computer equipment and storage medium
CN120692356A
A customer service dialogue method and device, computer equipment and storage medium
CN120692356B
