Recording control system and recording control program
The emotion determination system addresses the challenge of inappropriate data recording by determining emotion values and controlling the recording process, ensuring data is recorded appropriately based on user or robot emotions, enhancing interaction and management.
Patent Information
- Application Number
- JP2023083501
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2023-05-19
- Publication Date
- 2024-11-29
AI Technical Summary
Conventional techniques fail to appropriately record data based on the emotions of users or robots.
An emotion determination system that determines an emotion value for the robot or user, specifies how data is recorded based on detected information, and controls the recording process using a recording control unit.
Enables appropriate data recording based on the emotions of users or robots, allowing for enhanced interaction and improved data management.
Smart Images

Figure 2024167000000001_ABST
Abstract
Description
[Technical field]
[0001] The present invention relates to a recording control system and a recording control program. [Background technology]
[0002] A terminal is known that learns a conversation between a user and a call partner and stores the call partner's replies to questions from the user in a reply table (see, for example, Patent Document 1). Also, an emotion generation device is known that includes a neural network that inputs user information, device information, and its own current emotional state and outputs the next emotional state (see, for example, Patent Document 2). Also, a technology is known that stores a spatiotemporal pattern in an associative memory that includes multiple electronic neurons having a layer-neural network relationship with directional artificial synaptic connectivity (see, for example, Patent Document 3). [Prior art documents] [Patent documents]
[0003] [Patent Document 1] JP 2011-253389 A [Patent Document 2] Japanese Patent Application Publication No. 10-254592 [Patent Document 3] Special Publication No. 2013-535067 Summary of the Invention [Problem to be solved by the invention]
[0004] Conventional techniques may not be able to properly record data according to the emotions of a user or a robot. [Means for solving the problem]
[0005] A recording control system according to one embodiment of the present invention includes an emotion determination unit that determines an emotion value indicating the strength of an emotion of the robot or a user, or a type of emotion, based on information detected by a sensor provided in the robot, a determination unit that determines a manner in which to record data indicating information about the user detected by the sensor based on the information determined by the emotion determination unit, and a recording control unit that records the data based on the manner determined by the determination unit. Effect of the Invention
[0006] According to one aspect of the embodiment, data can be appropriately recorded depending on the emotion of the user or the robot. [Brief description of the drawings]
[0007] [Figure 1] 1 illustrates a schematic diagram of an example of a system 5 according to the present embodiment. [Diagram 2] 1 shows a schematic diagram of actions that the robot 100 performs toward the user 10. [Diagram 3] 1 shows a schematic diagram of a robot 100 performing data recording. [Figure 4] 2 shows a schematic functional configuration of the robot 100. [Diagram 5] 2 shows an outline of the functional configuration of the server 200. [Figure 6] 4 shows a schematic representation of information perceived by the perception processing section 416. [Figure 7] 4 shows an outline of information understood by the semantic understanding unit 418. [Figure 8] The information contained in the reaction rule 421 is shown in table form. [Figure 9] 1 illustrates a schematic diagram of a reaction rule-based process. [Figure 10] 1 shows an example of an operation flow of the robot 100. [Figure 11] 2 shows an example of reaction rule statistical information 522 stored in the server 200. [Figure 12]2 illustrates generally the actions taken to determine the labels to be associated with information to be recorded by server 200. [Figure 13] An example of the recording format switching rule 471 for switching the recording format is shown below. [Figure 14] 13 shows an example of an operation flow for switching the recording format. [Figure 15] 1 shows an example of a pattern table. [Figure 16] 13 shows an example of a recording mode table. [Figure 17] 13 shows an example of a process flow for recording data in a recording mode according to emotions. [Figure 18] 1 is a diagram illustrating an example of a hardware configuration of a computer 1200 that functions as a robot 100 and a server 200. FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0008] The present invention will be described below through embodiments of the invention, but the following embodiments do not limit the scope of the invention according to the claims. Furthermore, not all of the combinations of features described in the embodiments are necessarily essential to the solution of the invention.
[0009] FIG. 1 is a schematic diagram of an example of a system 5 according to the present embodiment. The system 5 includes a robot 100, a robot 101, a robot 102, and a server 200. A user 10a, a user 10b, a user 10c, and a user 10d are users of the robot 100. A user 11a, a user 11b, and a user 11c are users of the robot 101. A user 12a and a user 12b are users of the robot 102. In the description of the present embodiment, the user 10a, the user 10b, the user 10c, and the user 10d may be collectively referred to as the user 10. The user 11a, the user 11b, and the user 11c may be collectively referred to as the user 11. The user 12a and the user 12b may be collectively referred to as the user 12. The robot 101 and the robot 102 have substantially the same functions as the robot 100. Therefore, the system 5 will be described by mainly focusing on the functions of the robot 100.
[0010] The robot 100 converses with the user 10 and provides images to the user 10. At this time, the robot 100 cooperates with a server 200 or the like with which it can communicate via a communication network 20 to converse with the user 10 and provide images, etc. to the user 10. For example, the robot 100 not only learns appropriate conversation by itself, but also cooperates with the server 200 to learn so as to be able to have a more appropriate conversation with the user 10. The robot 100 also causes the server 200 to record captured image data, etc. of the user 10, and requests the server 200 for the image data, etc. as necessary, and provides the data, etc. to the user 10.
[0011] The robot 100 also has an emotion value that indicates the type of its own emotion. For example, the robot 100 has emotion values that indicate the intensity of each of the emotions of "joy", "anger", "sorrow", "pleasure", "discomfort", "relief", "anxiety", "sadness", "excitement", "concern", "relief", "fulfillment", "emptiness", and "neutral". When the robot 100 converses with the user 10 in a state where the emotion value of excitement is high, for example, the robot 100 utters sounds at a fast speed. In this way, the robot 100 can express its own emotion through actions.
[0012] The robot 100 also has a function of recognizing the emotions of the user 10. The robot 100 recognizes the emotions of the user 10 by analyzing a facial image of the user 10 acquired by a camera function and a voice of the user 10 acquired by a microphone function. The robot 100 determines the behavior to be performed by the robot 100 based on the recognized emotions of the user 10. For example, when the robot 100 recognizes that the user 10 looks lonely, it performs an action such as calling out to the user 10.
[0013] The robot 100 stores rules that define actions to be taken by the robot 100 based on the emotions of the user 10 and its own emotions, and performs various actions according to the rules. An example of an action based on a rule will be roughly described with reference to Figs. 2 and 3.
[0014] 2 is a schematic diagram showing actions that the robot 100 performs toward the user 10. The robot 100 has reaction rules for determining an action based on the emotion of the user 10. According to the reaction rules shown in the figure, when the emotion of the robot 100 is "normal" and the condition of the user's state is "alone and looks lonely", it is determined that the emotion of the robot 100 changes to "worried" and the action of "calling out" can be executed. Here, a consideration point indicating the strength of the consideration of the user 10 is associated with each action.
[0015] When the robot 100 recognizes based on the reaction rule that the current emotion of the robot 100 is "normal" and that the user 10 is alone and seems lonely, the robot 100 increases the emotion value of "worry" of the robot 100. In addition, the robot 100 selects the action of "calling out" defined in the reaction rule as the action to be performed on the user 10. For example, when the robot 100 selects the action of "calling out", the robot 100 converts the words "What's wrong?", which indicate concern, into a worried voice and outputs it.
[0016] After outputting a voice to the user 10, the robot 100 recognizes a change in the user's 10 emotions. For example, if the robot 100 outputs a voice saying "What's wrong?" and the user 10 replies "Thanks. I'm fine!", the robot 100 recognizes that the user's 10 emotions have improved by analyzing the user's 10 response. Since the user's emotions have improved as a result of the action of "calling out", the robot 100 determines that the action of "calling out" is an action with a high level of consideration, and increases the consideration point of the action of "calling out". This allows the robot 100 to learn appropriate actions for the user 10.
[0017] The robot 100 also transmits to the server 200 user reaction information indicating that this action has elicited a positive reaction from the user 10. The user reaction information includes, for example, the condition "looking lonely," the action of "calling out," the fact that the reaction of the user 10 was positive, and attributes of the user 10.
[0018] The server 200 stores the user reaction information received from the robot 100. The server 200 receives and stores user reaction information not only from the robot 100, but also from each of the robots 101 and 102. The server 200 then analyzes the user reaction information from the robots 100, 101, and 102 to generate recommended reaction rules.
[0019] The robot 100 inquires of the server 200 about the recommended reaction rules, and receives the recommended reaction rules from the server 200. The robot 100 incorporates the recommended reaction rules into the reaction rules stored in the robot 100. This allows the robot 100 to incorporate the reaction rules acquired by the robot 101, the robot 102, etc. into its own reaction rules.
[0020] 3 is a schematic diagram showing a scene in which the robot 100 switches the recording format of data to be recorded in the server 200. The robot 100 has a recording format switching rule for switching the recording format based on the emotion of the user 10. According to the illustrated recording format switching rule, when the conditions that "there is a certain number of family members or more" and "a certain number of family members' smiles are detected" are satisfied, the robot 100 performs an action of "switching the data recording format to a low compression format."
[0021] During a period in which the conditions of the recording format switching rule are not satisfied, the robot 100 continuously transmits highly compressed data such as skeletal data of the user 10 to the server 200 and causes the server 200 to record the data. The skeletal data is an example of shape data that represents the shape of the user. When the robot 100 determines that the conditions of the recording format switching rule are satisfied, it starts transmitting full HD video data and audio data and causes the server 200 to record low-compression data including the full HD video data and audio data in addition to the skeletal data.
[0022] When the robot 100 wants to provide the user 10 with some entertaining video, it requests the server 200 to transmit full HD video data and audio data, and provides the video data received from the server 200 to the user 10. In this way, when the robot 100 encounters a scene that is strongly memorable to the user 10, it can store high-quality video data of the scene in the server 200.
[0023] 4 shows a schematic functional configuration of the robot 100. The robot 100 has a sensor unit 400, a sensor module unit 410, a storage unit 420, a user state recognition unit 430, a condition determination unit 434, a user reaction recognition unit 432, a behavior determination unit 440, an emotion determination unit 442, a rule update unit 422, a communication processing unit 480, a record data generation unit 470, a switching control unit 477, a label generation unit 478, and a control target 452.
[0024] The control target 452 includes a display device, a speaker, LEDs in the eyes, and motors for driving the arms, hands, legs, etc. The posture and gestures of the robot 100 are controlled by controlling the motors of the arms, hands, legs, etc. Some of the emotions of the robot 100 can be expressed by controlling these motors. Some of the emotions of the robot 100 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 100. The posture, gestures, and facial expressions of the robot 100 are examples of the attitude of the robot 100.
[0025] The sensor unit 400 includes a microphone 401, a 3D depth sensor 402, a 2D camera 403, and a distance sensor 404. The microphone 401 continuously detects sound and outputs sound data. The microphone 401 may be provided on the head of the robot 100 and may have a function of performing binaural recording. The 3D depth sensor 402 detects the contour of an object by continuously irradiating an infrared pattern and analyzing the infrared pattern from infrared images continuously captured by the infrared camera. The 2D camera 403 is an example of an image sensor. The 2D camera 403 captures an image using visible light and generates visible light video information. The distance sensor detects the distance to an object by irradiating, for example, a laser or ultrasonic waves. The sensor unit 400 may also include a clock, a gyro sensor, a touch sensor, a sensor for motor feedback, and the like.
[0026] 4, the components other than the control target 452 and the sensor unit 400 are examples of components included in a behavior control system included in the robot 100. The behavior control system of the robot 100 controls the control target 452.
[0027] The storage unit 420 includes a reaction rule 421, a perception confirmation rule 417, a meaning understanding rule 419, a recording format switching rule 471, and a person DB 424. At least a part of the storage unit 420 is implemented by a storage medium such as a memory. The person DB 424 stores a face image of the user 10, attribute information of the user 10, and the like. Among the components of the robot 100 shown in FIG. 4, the functions of the components other than the control target 452, the sensor unit 400, and the storage unit 420 can be realized by a CPU operating based on a program. For example, the functions of these components can be implemented as the operation of a CPU by an operating system (OS) and a program operating on the OS.
[0028] The sensor module unit 410 includes a voice emotion recognition unit 411, a speech understanding unit 412, a facial expression recognition unit 413, and a face recognition unit 414. Information detected by the sensor unit 400 is input to the sensor module unit 410. The sensor module unit 410 analyzes the information detected by the sensor unit 400, and outputs the analysis result to the user state recognition unit 430.
[0029] The voice emotion recognition unit 411 of the sensor module unit 410 analyzes the voice of the user 10 detected by the microphone 401 and recognizes the emotion of the user 10. For example, the voice emotion recognition unit 411 extracts feature amounts such as frequency components of the voice and recognizes the emotion of the user 10 based on the extracted feature amounts. The speech understanding unit 412 analyzes the voice of the user 10 detected by the microphone 401 and outputs text information representing the content of the utterance of the user 10.
[0030] The facial expression recognition unit 413 recognizes the facial expression and emotion of the user 10 from the image of the user 10 captured by the 2D camera 403. For example, the facial expression recognition unit 413 recognizes the facial expression and emotion of the user 10 based on the shapes, positional relationship, etc. of the eyes and mouth.
[0031] The face recognition unit 414 recognizes the face of the user 10. The face recognition unit 414 recognizes the user 10 by matching a face image stored in the person DB 424 with a face image of the user 10 captured by the 2D camera 403.
[0032] The user state recognition unit 430 recognizes the state of the user 10. The user state recognition unit 430 includes a perception processing unit 416 and a meaning understanding unit 418. Information analyzed by the sensor module unit 410 is output to the perception processing unit 416. The perception processing unit 416 mainly performs processing related to perception using the analysis result of the sensor module unit 410 and the perception confirmation rule 417 of the storage unit 420. For example, the perception processing unit 416 generates perception information such as "Daddy is alone" and "There is a 90% chance that Daddy is not smiling."
[0033] The meaning understanding unit 418 performs a process of understanding the meaning of the perceptual information generated by the perception processing unit 416 using the meaning understanding rules 419 in the storage unit 420. For example, the meaning understanding unit 418 generates semantic information such as "Daddy is alone and looks lonely." Note that the perceptual information and semantic information are examples of information representing the state of the user 10. Here, the information representing the state of the user 10 may include information representing the emotions of the user 10.
[0034] The semantic information generated by the semantic understanding unit 418 is output to the condition decision unit 434 , the user reaction recognition unit 432 , and the label generation unit 478 .
[0035] The condition determination unit 434 determines whether the condition defined in the reaction rule 421 is satisfied based on the semantic information and the reaction rule 421 in the storage unit 420. The reaction rule 421 is an example of decision information for determining the behavior of the robot 100. As described above, the reaction rule 421 associates a plurality of behaviors with each of the plurality of behaviors and their respective consideration points. The reaction rule 421 also includes information for determining the behavior based on an emotion value indicating the emotion of the robot 100 and the state of the user 10. In this way, the storage unit 420 functions as an example of a behavior information storage unit that stores information for determining the behavior based on the state of the user.
[0036] The condition determination unit 434 determines that the condition stored in the reaction rule 421 is satisfied. The behavior determination unit 440 determines an behavior to be executed from among a plurality of behaviors stored in the reaction rule 421 in association with the condition determined to be satisfied by the condition determination unit 434. Specifically, the behavior determination unit 440 determines an behavior to be executed based on the consideration points of each of the plurality of behaviors stored in the reaction rule 421 in association with the condition determined to be satisfied by the condition determination unit 434. In this way, the behavior determination unit 440 determines an behavior to be executed based on the state of the user 10 recognized by the user state recognition unit 430 and the information of the reaction rule 421.
[0037] The emotion determination unit 442 determines an emotion value indicating the current emotion of the robot 100 based on the semantic information generated by the semantic understanding unit 418. In this way, the emotion determination unit 442 determines an emotion value indicating the current emotion of the robot 100 based on the state of the user recognized by the user state recognition unit 430.
[0038] The behavior control unit 450 controls the control target 452 based on the behavior determined by the behavior determination unit 440. For example, when the behavior determination unit 440 determines to speak, the behavior control unit 450 causes a sound to be output from a speaker included in the control target 452. At this time, the behavior control unit 450 determines the speech speed of the sound based on the emotion value updated by the emotion determination unit 442. For example, the behavior control unit 450 determines a faster generation speed as the emotion value of excitement increases. In this way, the behavior control unit 450 determines the execution form of the behavior determined by the behavior determination unit 440 based on the emotion value determined by the emotion determination unit 442.
[0039] The user reaction recognition unit 432 recognizes a change in the emotion of the user 10 in response to the execution of the action determined by the action determination unit 440. The change in the emotion of the user 10 is an example of the reaction of the user 10. That is, the user reaction recognition unit 432 recognizes the reaction of the user 10 in response to the execution of the action determined by the action determination unit 440. The user reaction recognition unit 432 may recognize the change in emotion based on the voice or facial expression of the user 10. In addition, the user reaction recognition unit 432 may recognize the change in emotion of the user 10 based on the detection of an impact by a touch sensor included in the sensor unit 400. For example, the user reaction recognition unit 432 may recognize that the emotion of the user 10 has worsened when an impact is detected by a touch sensor included in the sensor unit 400.
[0040] Information indicating the reaction of the user 10 recognized by the user reaction recognition unit 432 is output to the emotion determination unit 442, the communication processing unit 480, and the label generation unit 478. As described above, when the user state recognition unit 430 recognizes that the user 10 looks lonely, the emotion determination unit 442 may increase the sad emotion value according to the reaction rule 421. Also, when the user reaction recognition unit 432 recognizes that the user 10 has smiled, the emotion determination unit 442 may increase the happy emotion value according to the reaction rule 421. In this way, the emotion determination unit 442 updates the emotion value based on at least one of the user's state recognized by the user state recognition unit 430 and the user's reaction to the execution of the action determined by the action determination unit 440.
[0041] Furthermore, the rule update unit 422 changes the consideration points stored by the reaction rules 421 in association with the behavior determined by the behavior determining unit 440, based on the change in the emotion of the user 10 recognized by the user reaction recognition unit 432. Specifically, the rule update unit 422 changes the consideration points of the behavior determined by the behavior determining unit 440, among the consideration points of the multiple behaviors stored by the reaction rules 421 in association with the condition determined to be satisfied by the condition determination unit 434, based on the change in the emotion of the user 10 recognized by the user reaction recognition unit 432. For example, when the user reaction recognition unit 432 recognizes that the emotion of the user 10 has improved, the rule update unit 422 increases the consideration points stored by the reaction rules 421 in association with the condition determined to be satisfied by the condition determination unit 434 and the behavior determined by the behavior determining unit 440. On the other hand, when the user reaction recognition unit 432 recognizes that the user 10's feelings have worsened, the rule update unit 422 reduces the consideration points stored in the reaction rules 421 in association with the condition determined to be satisfied by the condition judgment unit 434 and the behavior determined by the behavior determination unit 440.
[0042] The communication processing unit 480 is responsible for communication with the server 200. As described above, the communication processing unit 480 transmits user reaction information to the server 200. In this way, the communication processing unit 480 transmits to the server 200 the state of the user 10 recognized by the user state recognition unit 430, the action determined by the action determination unit 440, and the reaction of the user 10 recognized by the user reaction recognition unit 432. In addition, the communication processing unit 480 receives a recommended reaction rule from the server 200 when the action determination unit 440 cannot determine the action to be performed based on the state of the user 10 and the reaction rule 421. In this way, the communication processing unit 480 receives from the server 200 an action suitable for the state of the user 10 recognized by the user state recognition unit 430 when the action determination unit 440 cannot determine the action to be performed based on the state of the user 10 and the reaction rule 421.
[0043] The rule update unit 422 updates the reaction rules 421 based on the recommended reaction rules that the communication processing unit 480 receives from the server 200. In this manner, the rule update unit 422 updates the reaction rules 421 based on the behavior that the communication processing unit 480 receives from the server 200.
[0044] In addition, when the behavior decision unit 440 is unable to decide on a behavior toward the user based on the state of the user 10 recognized by the user state recognition unit 430, the emotion decision unit 442 may change the emotion value representing the intensity of sadness to a value representing a greater intensity of sadness.
[0045] Recording data generation unit 470 generates data to be recorded in server 200 based on information detected by sensor unit 400. Communication processing unit 480 transmits the information generated by recording data generation unit 470 to an external server and causes the server to record the information.
[0046] For example, the recording data generating unit 470 continuously generates skeleton data for recording based on skeleton information continuously detected by the 3D depth sensor 402. The recording data generating unit 470 also generates video data for recording based on video information captured by the 2D camera 403. For example, the recording data generating unit 470 generates full HD video data. The recording data generating unit 470 also generates audio data for recording based on audio information continuously detected by the microphone 401.
[0047] Here, the video data is, for example, full HD video data, and has a larger amount of information than the skeleton data. Therefore, the skeleton data can be considered as highly compressed data, and the video data can be considered as low compression data. In this way, the skeleton data is an example of information in a first recording format, and the video data is an example of information in a second recording format that has a larger amount of information than the information in the first recording format. In this way, the recording data generating unit 470 continuously processes at least a portion of the information detected by a sensor that continuously detects the user 10, and generates information in the first recording format or information in the second recording format that has a larger amount of information than the information in the first recording format.
[0048] Here, the perception processing unit 416 perceives the number of recognized family members and the number of family members whose smiles are detected based on at least a part of the information detected by the sensor unit 400. As described above, the perception processing unit 416 perceives a smiling user based on at least one of the image of the user 10 and the voice of the user 10 detected by the sensor unit 400. Here, a smile is an example of information representing the intensity of the emotion of the user 10. Therefore, the number of family members whose smiles are detected is an example of the total value of the intensity of the emotions of the multiple users 10. In this way, the perception processing unit 416 detects the intensity of the emotion of the user based on at least a part of the information detected by the sensor unit 400.
[0049] Then, the perception processing unit 416 perceives whether or not a certain number of family members have been recognized and a certain number of smiles of the family members have been detected based on the recording format switching rule 471. Then, the switching control unit 477, when the recording data generating unit 470 is caused to generate skeleton data, causes the recording data generating unit 470 to start generating video data in response to the perception processing unit 416 recognizing a certain number of family members and detecting a certain number of smiles of the family members. In this way, when the switching control unit 477 is caused to generate information in the first recording format by the recording data generating unit 470, in response to the number of multiple users 10 exceeding a predetermined number of people and the total value of the strength of the detected emotions exceeding a predetermined threshold, the switching control unit 477 switches the recording format of the information to be generated by the recording data generating unit 470 from the first recording format to the second recording format having a larger amount of information than the information in the first recording format.
[0050] The label generating unit 478 generates a label to be recorded by being attached to the recording data. The label is an example of the attached information to be attached to the recording data. The label generating unit 478 generates a label including words such as "Yoshiko's birthday" and "Yoshiko's birthday party". Other examples of the label include "celebration of entering school", "Christmas" and "New Year's". The label generating unit 478 judges a situation such as a birthday or a birthday party based on the date and time information and the response content recognized by the user reaction recognition unit 432 to a question to the user 10, and generates a label based on the judged situation. In this way, the label generating unit 478 generates a label including one or more words expressing the current situation based on at least a part of the information detected by the sensor unit 400. The communication processing unit 480 associates the label generated by the label generating unit 478 with the recording data generated by the recording data generating unit 470, transmits it to the server 200, and causes the server 200 to record it.
[0051] 5 shows an outline of the functional configuration of the server 200. The server 200 includes a storage unit 520, a reaction rule generation unit 530, a reaction rule search unit 540, a statistical information control unit 550, an access control unit 560, and a communication processing unit 580.
[0052] The storage unit 520 stores reaction rules 521, reaction rule statistical information 522, and recording data 561. The reaction rules 521 include information similar to the reaction rules 421. The reaction rule statistical information 522 includes, in addition to information similar to the reaction rules 421, attribute information of the users 10, 11, and 12, and information on the ratio of positive reactions obtained. The recording data 561 is data for recording, such as the above-mentioned skeletal data, video data, and audio data.
[0053] The communication processing unit 580 is responsible for communication between the robot 100, the robot 101, and the robot 102 and the server 200. The communication processing unit 580 receives user reaction information transmitted from the robot 100. As described above, the robots 101 and 102 also have functions similar to those of the robot 100. Therefore, for example, the robot 101 transmits user reaction information including a reaction to an action performed on the user 11 to the server 200. The communication processing unit 580 receives the user reaction information transmitted from the robot 101 and the user reaction information transmitted from the robot 102. In this way, the communication processing unit 580 receives from the robot 101 the state of the user 11 recognized by the robot 101, the action determined by the robot 101 based on the state of the user 11, and the reaction of the user 11 to the execution of the action.
[0054] The statistical information control unit 550 accumulates the user reaction information received by the communication processing unit 580 as reaction rule statistical information 522 in the storage unit 520. In this manner, the storage unit 520 stores the user's state, action, and user reaction received by the communication processing unit 580 in association with each other. The user reaction information accumulated in the statistical information control unit 550 includes user reaction information regarding user 10, user reaction information regarding user 11, and user reaction information regarding user 12.
[0055] The reaction rule generation unit 530 generates recommended reaction rules based on the user reaction information accumulated in the reaction rule statistical information 522, and stores them as reaction rules 521. Based on the information included in the reaction rule statistical information 522, a reaction rule including a behavior that has received a positive reaction is generated as a recommended reaction rule.
[0056] Here, an operation will be described when the robot 100 inquires of the server 200 about recommended reaction rules. The robot 100 inquires of the server 200 about recommended reaction rules by transmitting the state of the user 10 recognized by the robot 100 to the server 200. In this case, the reaction rule search unit 540 searches at least one of the reaction rules 521 and the reaction rule statistical information 522 for a reaction rule having a condition that is satisfied by the state of the user 10 received by the communication processing unit 580. The communication processing unit 580 transmits the reaction rule searched by the reaction rule search unit 540 to the robot 100 as a recommended reaction rule.
[0057] In this way, the reaction rule search unit 540 selects an action suitable for the state of the user 10 received by the communication processing unit 580 from among the actions stored in the reaction rule statistical information 522 in association with the state matching the state of the user received by the communication processing unit 580, based on the reaction of the user 10 stored in the action result storage unit in association with the state matching the state of the user 10 received by the communication processing unit 580. The communication processing unit 580 transmits the action selected by the reaction rule search unit 540 to the robot 100.
[0058] The access control unit 560 stores the record data received by the communication processing unit 580 from the robot 100, the robot 101, and the robot 102 in the record data 561 of the storage unit 520. When the communication processing unit 580 receives a request to transmit record data, the access control unit 560 reads the requested record data from the record data 561 and causes the communication processing unit 580 to transmit it.
[0059] 5, the functions of the components other than the storage unit 420 can be realized by a CPU operating based on a program. For example, the functions of these components can be implemented as the operations of a CPU by an operating system (OS) and a program that runs on the OS.
[0060] 6 is a schematic diagram of information perceived by the perception processing unit 416. When the person detection information 601 by the 3D depth sensor 402, the face detection information 602 by the face recognition unit 414, and the user recognition information 603 are input to the perception processing unit 416, the person detection information 601, the face detection information 602, and the user recognition information 603 are integrated to generate perception information 611, perception information 612, and perception information 613. Next, when the smile detection information 604 from the facial expression recognition unit 413 is input, the smile detection information 604 is converted to generate perception information 614. Furthermore, the perception processing unit 416 perceives, "Maybe Daddy is lonely?" based on the high probability that the perception information 614 is not a smile (perception information 615).
[0061] Here, the perception confirmation rule 417 includes a perception confirmation rule 630 having a condition part of "someone is alone" and "the probability of smiling is 20% or less" and a conclusion part of "check whether there is any speech from the microphone for 30 seconds" as a perception confirmation rule when "maybe I'm lonely?" is perceived. The perception processing unit 416 determines that the condition part of the perception confirmation rule 630 is satisfied from the perception information 613 and the smile detection information 604, and executes confirmation of the detection result by the microphone 401 based on the conclusion part of the perception confirmation rule 630. Specifically, the perception processing unit 416 causes the speech understanding unit 412 to analyze the voice output from the microphone 401.
[0062] Here, when no voice is detected for 30 seconds by the microphone 401, the speech understanding unit 412 outputs timeout information indicating that no speech has been detected for 30 seconds. When the timeout information 605 from the speech understanding unit 412 is input, the perception processing unit 416 converts the timeout information 605 into perception information 616 such as "Dad isn't saying anything."
[0063] In this way, the perceptual processing unit 416 generates a perceptual history 610 including perceptual information 611, perceptual information 612, perceptual information 613, perceptual information 614, perceptual information 615, and perceptual information 616. The perceptual processing unit 416 outputs the generated perceptual information to the semantic understanding unit 418.
[0064] 7 is a schematic diagram of information understood by the semantic understanding unit 418. The semantic understanding unit 418 receives the perception history 610 generated by the perception processing unit 416. The semantic understanding unit 418 generates semantic information 711, semantic information 712, semantic information 713, semantic information 714, and semantic information 715 from the perception information 611, perception information 612, perception information 613, perception information 614, and perception information 615, respectively.
[0065] When generating the semantic information 714 from the perceptual information 614, the semantic understanding unit 418 vagues the numerical information of "90% probability" included in the perceptual information 614 to information of "not very...". In this way, the semantic understanding unit 418 generates the semantic information 714 by vaguely interpreting the perceptual information 614.
[0066] Here, the meaning understanding rule 419 includes a meaning understanding rule 730 having a condition part of "someone is alone", "not smiling much", and "not saying anything" and a conclusion part of understanding "someone is alone and looks lonely" as a meaning confirmation rule when "is he lonely?" is understood. When the meaning understanding unit 418 determines that the semantic information 713, the semantic information 714, and the perceptual information 616 satisfy the condition part of the meaning understanding rule 730 based on the meaning understanding rule 730, it generates the semantic information 717 of "Daddy is alone and looks lonely." In this way, the meaning understanding unit 418 uses the meaning understanding rule 730 to integrate the semantic information 714 and the perceptual information 616 to generate the semantic information 717.
[0067] In this way, the semantic understanding unit 418 generates a semantic history 710 including semantic information 711, semantic information 712, semantic information 713, semantic information 714, semantic information 715, and semantic information 717. The semantic information generated by the semantic understanding unit 418 is output to the condition determination unit 434, the user reaction recognition unit 432, and the label generation unit 478.
[0068] As described in relation to Figures 6 and 7, the perception processing unit 416 and the semantic understanding unit 418 vaguely store the meaning of the perception result using patterned language, even though the accuracy of the perception result is lost.
[0069] 8 shows, in table form, information included in the reaction rule 421. The reaction rule 421 includes a condition part including conditions of "own emotion" and "user state", a conclusion part including "emotion change" and "behavior", and "attachment information" including "consideration points" and "praise points".
[0070] The "Own emotion" condition is a condition related to the robot 100's own emotion. The "User state" is a condition related to the state of the user 10. The "Emotion change" indicates how the emotion of the robot 100 is changed. The "Action" represents the action that the robot 100 can perform. The "Consideration point" indicates the degree of consideration for the user 10. The "Consideration point" is an example of the degree of appropriateness.
[0071] As shown in the figure, the condition part of the reaction rule 421 includes a condition of "own emotion" of "normal" and a "user state" of "someone looks lonely". The conclusion part corresponding to this condition part includes an "emotion change" that indicates an emotion change of "worried". In addition, the conclusion part corresponding to this condition part includes an "action" that indicates an action of "calling out", "only showing worried gestures", and "greeting". In addition, the "attached information" associates 30 points, 20 points, and 10 points as "consideration points" for the actions of "calling out", "only showing worried gestures", and "greeting", respectively. In addition, the "attached information" associates 50 points, 30 points, and 30 points as "praise points" for the actions of "calling out", "only showing worried gestures", and "greeting", respectively.
[0072] In this way, the reaction rules 421 include information indicating a condition indicating a state of the user to be satisfied and an action associated with the condition. The reaction rules 421 also include information indicating a condition indicating an emotion of the robot 100 to be satisfied and an action associated with the condition. The reaction rules 421 also include emotion change information for changing the emotion value based on the emotion value of the robot 100 and the state of the user 10. The emotion determination unit 442 then updates the emotion value based on the emotion change information stored by the reaction rules 421 in association with the emotion value determined by the emotion determination unit 442 and the state of the user 10 recognized by the user state recognition unit 430.
[0073] 9 is a schematic diagram showing a process based on the reaction rule 421. When the semantic information 717 of “Daddy is alone and looks lonely” is input from the semantic understanding unit 418, the condition decision unit 434 searches the reaction rule 421 for a condition part that is satisfied by the state of the user 10 of “Daddy is alone and looks lonely” and the current emotion of the robot 100, which is “normal.”
[0074] The condition determination unit 434 determines that the state of the user 10, "Daddy is alone and looks lonely," and the emotion, "normal," satisfy the condition part of the reaction rule 810 in the reaction rule 421. As a result, the behavior determination unit 440 specifies, as the candidate behaviors 910, "call out," "only respond with worried gestures," and "greet," which are defined in the reaction rule 810. Then, the behavior determination unit 440 selects an action to be actually performed from the specified candidate behaviors 910. For example, the behavior determination unit 440 selects one action with a selection probability according to the size of the consideration point. Here, it is assumed that the behavior determination unit 440 selects the action, "call out."
[0075] Furthermore, the emotion determination unit 442 updates the emotion value based on the change in emotion of "worried" defined in the reaction rule 810. For example, the emotion determination unit 442 increases the emotion value of "worried" representing the intensity of anxiety. In this way, the emotion determination unit 442 updates the emotion value based on the state of the user 10 recognized by the user state recognition unit 430 based on the reaction rule 421. For example, when the user state recognition unit 430 recognizes that the user 10 is in a state of feeling lonely, the emotion determination unit 442 changes the emotion value representing the intensity of anxiety to a value representing a greater intensity of anxiety.
[0076] Next, the behavior control unit 450 causes a voice with the utterance content of "What's wrong?" to be output from the speaker of the control target 452. At this time, the behavior control unit 450 causes a voice with voice characteristics expressing a worried emotion to be output based on the current emotion value. For example, a voice with a relatively slow speed is generated. The behavior control unit 450 may determine the utterance content of "What's wrong?" based on the emotion of "worrying."
[0077] Next, the user reaction recognition unit 432 waits for the reaction of the user 10 to be detected. For example, when the semantic information 940 "Dad is smiling a little" is output from the semantic understanding unit 418 based on the voice "It's okay. Thank you!" and the facial expression of the user 10 at that time, the user reaction recognition unit 432 determines that the user 10 has changed from an emotion of "looking lonely" to an emotion of "looking happy". As a result, the rule update unit 422 determines that the emotion of the user 10 has improved due to the action of "calling out". As a result, the consideration points for the action of "calling out" are increased from 30 points to 40 points. Also, the rule update unit 422 increases the praise points from 50 points to 60 points.
[0078] Furthermore, the emotion determining unit 442 increases the emotion value of joy because the emotion of the user 10 has improved due to the action performed by the robot 100. Furthermore, the behavior control unit 450 expresses the joyful emotion through an action based on the increased emotion value of joy. For example, the behavior control unit 450 causes the robot 100 to make a gesture expressing joy.
[0079] In addition, the communication processing unit 480 transmits to the server 200 user reaction information 960 including information contained in the reaction rules 810 excluding the “considerate points” and the “praised points”, information indicating that the reaction of the user 10 was positive, and attributes of the user 10 recognized as “Dad”.
[0080] In this way, the behavior determining unit 440 determines the behavior to be executed based on the emotion value determined by the emotion determining unit 442, the state of the user recognized by the user state recognizing unit 430, and the reaction rule 421. Specifically, the behavior determining unit 440 determines, as the behavior to be executed, more frequently, from among the multiple behaviors stored by the reaction rule 421 in association with the condition determined to be satisfied by the condition determining unit 434. Note that the behavior determining unit 440 may select the behavior with the highest consideration point as the behavior to be actually executed. In this case, when there are multiple behaviors with the highest consideration point, the behavior determining unit 440 may randomly select one behavior to be actually executed from the multiple behaviors with the highest consideration point. In this case, the behavior determining unit 440 may select one behavior to be actually executed with a selection probability according to the praised point.
[0081] Fig. 10 shows an example of an operational flow for determining an action of the robot 100. The operational flow shown in Fig. 10 is started when, for example, semantic information including the emotion of the user 10 is input. Note that "S" in the operational flow indicates a step to be executed.
[0082] The condition determination unit 434 searches for a condition part that is satisfied by the emotion of the robot 100 and the input semantic information from among the condition parts included in the reaction rule 421 (S1002). The condition determination unit 434 judges whether or not there is a reaction rule whose condition part is satisfied (S1004). If it is judged that there is a reaction rule whose condition part is satisfied, the behavior determination unit 440 selects one behavior to be executed against the user 10 from among the behaviors defined in the conclusion parts of the searched reaction rules (S1006). In addition, the emotion determination unit 442 updates the emotion value based on the emotion change defined in the conclusion part of the searched reaction rule (S1008).
[0083] Next, the behavior control unit 450 controls the control object 452 so as to execute the behavior selected in S1006 (S1010). At this time, the behavior control unit 450 controls the control object 452 based on the emotion value updated in S1008. For example, when expressing the emotion of "worry", the behavior control unit 450 controls the control object 452 so that the posture of the robot 100 becomes a worried posture.
[0084] Next, the user reaction recognition unit 432 monitors for input of semantic information representing the reaction of the user 10 (S1012). When the semantic information representing the reaction of the user 10 is input, the user reaction recognition unit 432 determines whether or not the user's reaction is positive based on the input semantic information (S1016).
[0085] If it is determined in S1016 that the reaction is positive, the emotion determining unit 442 increases the emotion value of "happiness" (S1018). In addition, the rule updating unit 422 increases the consideration points and praise points corresponding to the performed action. On the other hand, if it is determined in S1016 that the reaction is negative, the emotion determining unit 442 increases the emotion value of "sadness" (S1022). In addition, the rule updating unit 422 decreases the consideration points and praise points corresponding to the performed action (S1024).
[0086] In this way, when the user's reaction recognized by the user reaction recognition unit 432 is not bad, the rule update unit 422 increases the consideration points stored by the reaction rules 421 in association with the condition determined to be satisfied by the condition judgment unit 434 and the behavior determined by the behavior determination unit 440. On the other hand, when the user's reaction recognized by the user reaction recognition unit 432 is bad, the rule update unit 422 decreases the consideration points stored by the reaction rules 421 in association with the condition determined to be satisfied by the condition judgment unit 434 and the behavior determined by the behavior determination unit 440.
[0087] Furthermore, after the behavior control unit 450 executes the behavior determined by the behavior determining unit 440 in the execution mode determined according to the emotion, the emotion determining unit 442 further changes the emotion value based on the user's reaction to the execution of the behavior. Specifically, when the user's reaction to the behavior determined by the behavior determining unit 440 being performed on the user in the execution mode determined by the behavior control unit 450 is not bad, the emotion determining unit 442 changes the emotion value representing the intensity of joy to a value representing a greater intensity of joy. Furthermore, when the user's reaction to the behavior determined by the behavior determining unit 440 being performed on the user in the execution mode determined by the behavior control unit 450 is bad, the emotion determining unit 442 changes the emotion value representing the intensity of sadness to a value representing a greater intensity of sadness.
[0088] The behavior control unit 450 expresses the emotion of the robot 100 based on the emotion value determined in S1018 or S1022 (S1026). For example, when the behavior control unit 450 increases the emotion value of "happiness", it controls the control object 452 to make the robot 100 perform a happy gesture. When the behavior control unit 450 increases the emotion value of "sadness", it controls the control object 452 to make the robot 100 assume a drooping posture. Next, the communication processing unit 480 transmits the user reaction information to the server 200 (S1028), and the operation related to the determination of the behavior of the robot 100 is completed.
[0089] If it is determined in the determination of S1004 that there is no reaction rule whose condition part is satisfied, the communication processing unit 480 inquires of the server 200 about a recommended reaction rule. For example, information indicating the emotion of the robot 100, information indicating the state of the user represented by the input semantic information, and information indicating the attributes of the user 10 are transmitted to the server 200 (S1030). The server 200 transmits a reaction rule suitable for the information transmitted in the inquiry to the robot 100 as a recommended reaction rule. When the communication processing unit 480 receives the recommended reaction rule from the server 200, the rule update unit 422 incorporates the recommended reaction rule into the reaction rule 421 (S1032) and proceeds to S1006. In this case, in S1006 and S1008, the action is selected and the emotion value is updated based on the reaction rule received from the server 200. In this way, when the condition judgment unit 434 determines that none of the conditions included in the reaction rule 421 are satisfied, the communication processing unit 480 receives from the server 200 a behavior appropriate to the user's state recognized by the user state recognition unit 430.
[0090] According to this operation flow, if there is no reaction rule whose condition part is satisfied, an inquiry about a recommended reaction rule is made. Even if there is a reaction rule whose condition part is satisfied, if the consideration points associated with any of the actions are smaller than a predetermined value, it may be determined that the action cannot be determined. In this case, the communication processing unit 480 may make an inquiry about a recommended reaction rule. That is, the action decision unit 440 decides, as an action to be executed, an action having a consideration point included in the decision information higher than a predetermined value among one or more actions associated with a condition determined to be satisfied by the condition determination unit 434 among the information included in the decision information. The communication processing unit 480 may receive, from the server 200, an action suitable for the state of the user 10 recognized by the user state recognition unit 430, if the consideration points of one or more actions associated with the condition determined to be satisfied by the condition determination unit 434 are all equal to or lower than a predetermined value. Then, the rule update unit 422 may add the behavior received by the communication processing unit 480 from the server 200 to the reaction rule in association with the condition determined by the condition determination unit 434 to be satisfied.
[0091] The inquiry about the recommended reaction rule may be made periodically by batch processing or the like, in addition to when the behavior decision unit 440 cannot decide the behavior. The inquiry about the recommended reaction rule may be made when it is determined that the user 10 is asleep, or when the robot 100 transitions to an idle state or a sleep state. In addition, the inquiry about the recommended reaction rule may be made at any timing when the robot 100 is about to take an action.
[0092] Furthermore, the reaction rule 421 may have a rule that defines an action to be taken when no action to be taken for the user 10 is found. For example, the reaction rule 421 may have a rule that includes a condition part that defines a condition that "no action to be taken for the user 10 is found," and a conclusion part that defines an action to "query the server 200 for recommended reaction rules." In this way, the action determination part 440 may determine to query the server 200 for recommended reaction rules in accordance with the reaction rule 421.
[0093] 11 shows an example of reaction rule statistical information 522 stored in server 200. Reaction rule statistical information 522 includes "reaction rule," "attribute," "number of people," and "reaction result." "Reaction rule" includes "condition part," "conclusion part," and "consideration point." The "condition part," "conclusion part," and "consideration point" of reaction rule statistical information 522 store information corresponding to the "condition part," "conclusion part," and "consideration point" of reaction rule 421.
[0094] The "attributes" of the reaction rule statistical information 522 stores the attributes of the user 10. The attributes of the user 10 include information such as age group, sex, role in the household, and area of residence. The "number of people" stores the number of people who contributed to the information accumulated in the reaction rule statistical information 522. The "reaction result" stores information indicating the number of times there was a positive reaction and the number of times there was a positive / negative reaction.
[0095] When the communication processing unit 580 receives user reaction information from the robot 100, the statistical information control unit 550 stores the "condition part" and the "conclusion part" of the reaction rule included in the user reaction information in the reaction rule statistical information 522 in association with an attribute that matches the attribute included in the user reaction information. In addition, the statistical information control unit 550 increments the value of the "positive number of times" or the value of the "negative number of times" based on the good or bad reaction information included in the user reaction information. In addition, the statistical information control unit 550 increments the value of the "number of people" as necessary. For example, if the currently received user reaction information is user reaction information of a user whose user reaction information has been reflected in the same record of the reaction rule statistical information 522 in the past, the value of the "number of people" is not incremented.
[0096] Furthermore, the statistical information control unit 550 calculates a recommended consideration point and stores it in the "consideration point" of the "reaction rule". For example, the statistical information control unit 550 may calculate a higher value as the recommended consideration point as the value of the "positive count" is larger. Specifically, the statistical information control unit 550 may calculate a higher value as the recommended consideration point as the ratio of the value of the "positive count" to the value of the "negative count" is larger.
[0097] When the communication processing unit 580 receives an inquiry about a recommended reaction rule from the robot 100, the reaction rule searching unit 540 searches for the recommended reaction rule from among the reaction rules stored in the reaction rules 521. For example, the reaction rule searching unit 540 searches the reaction rules 521 for a reaction rule having a "condition part" that matches the "condition part" included in the inquiry. The reaction rule searched by the reaction rule searching unit 540 is transmitted from the communication processing unit 580 to the robot 100 as a response to the inquiry.
[0098] The reaction rule search unit 540 may search for a recommended reaction rule from among the reaction rules stored in the reaction rule statistical information 522. Specifically, the reaction rule search unit 540 may search for a reaction rule that has a "condition part" that matches a "condition part" included in the query, and has an "attribute" that matches an "attribute" included in the query, from among the reaction rules stored in the reaction rule statistical information 522. This allows the server 200 to provide the robot 100 with an action that is suited to the attributes of the user 10.
[0099] 12, reaction results are tallied for each user attribute in the reaction rule statistical information 522. As another example of the reaction rule statistical information, reaction results may be tallied for each user.
[0100] 12 shows an outline of the behavior executed by the robot 100 to determine the label to be attached to the recorded data. When the behavior decision unit 440 determines that it is the birthday of the user 10 based on the date and time information and the attribute information of the user 10, it decides to ask the user 10 a question to confirm whether it is his / her birthday or not. In this case, the behavior control unit 450 causes a speaker of the control target 452 to output a voice with the utterance content, for example, "Today is your birthday, Mr. XX. Happy birthday!"
[0101] Thereafter, the user reaction recognition unit 432 recognizes whether the reaction of the user 10 was positive or negative from the content of the utterance of the user 10. For example, if the user 10 replies with "Thank you, I'm happy!", the user reaction recognition unit 432 recognizes that the reaction of the user 10 was positive. If the recognized name of the user 10 is "Yoshiko-chan", the label generation unit 478 generates the label "Yoshiko-chan's birthday".
[0102] The label generating unit 478 generates a label including information indicating the name of the recognized user 10 and information "Yoshiko's birthday" as a label to be attached to the record data. The communication processing unit 480 continuously transmits the record data in which the label generated by the label generating unit 478 is attached to the skeleton data based on the skeleton information acquired by the 3D depth sensor 402 to the server 200. The server 200 stores the record data received from the robot 100 in the record data 561 of the storage unit 520.
[0103] The meaning understanding unit 418 calculates the number of people currently recognized by the current skeleton data based on the skeleton information acquired from the 3D depth sensor 402. Also, the skeleton data acquired in the same time period in the past is acquired from the server 200, and the average number of people recognized in the same room in the same time period in the past is calculated. Here, when the meaning understanding unit 418 recognizes that the currently recognized number of people is greater than the average number of people recognized in the same room in the same time period in the past by a predetermined threshold or more, it recognizes that the situation is unusual. Also, when the meaning understanding unit 418 compares the label attached to the skeleton data acquired in the same time period in the past with the currently recognized user 10 and determines that a user 10 who is not usually at home is currently recognized, it recognizes that the situation is unusual. Note that, when the behavior control unit 450 recognizes that the situation is unusual, it may control the sensor unit 400 and the sensor module unit 410 to perform a more detailed detection operation. For example, the behavior control unit 450 may control the utterance understanding unit 412 to recognize the utterance content with higher accuracy. Also, the voice emotion recognition unit 411 may be controlled to recognize emotions with higher accuracy, and the facial expression recognition unit 413 may be controlled to recognize facial expressions with higher accuracy.
[0104] When the meaning understanding unit 418 recognizes that the situation is different from usual, the behavior decision unit 440 decides to ask a question to the user 10 to confirm the situation. For example, when the label "Yoshiko's birthday" has been determined, if it is determined that many people have been recognized, the behavior control unit 450 outputs a voice with the utterance content "Are you having a birthday party for Yoshiko?" from the speaker of the control target 452.
[0105] In this case, the user reaction recognition unit 432 recognizes whether the reaction of the user 10 was positive or negative from the content of the utterance of the user 10. For example, if the user 10 replies "That's right!", the user reaction recognition unit 432 recognizes that the reaction of the user 10 was positive. In this case, the label generation unit 478 generates a label "Yoshiko's birthday party." The labels "Yoshiko's birthday" and "Yoshiko's birthday party" are examples of labels that represent the current situation of the user 10.
[0106] In this manner, the robot 100 interacts with the user 10 to confirm the current situation based on at least a part of the information detected by the sensor unit 400. Then, the label generation unit 478 determines the current situation of the user 10 based on the result of the interaction with the user 10, and generates a label including one or more pieces of character information representing the current situation of the user 10 based on the determined information.
[0107] 13 shows an example of the recording format switching rule 471. The recording format switching rule 471 includes a rule 1300 having a condition part that defines the conditions that "a certain number or more of family members are in front" and "a certain number or more of smiling family members have been detected" and a conclusion part that defines the action of "switching to a low compression recording format." The recording format switching rule 471 also includes a rule having a condition part that defines the conditions that "the number of family members in front falls below a certain number" and "a certain number or more of anger from family members has been detected" and a conclusion part that defines the action of "switching to a high compression recording format."
[0108] The value of the "fixed number" may be larger as the number of people belonging to one family increases. For example, if the family has four members, the value of the "fixed number" may be 3. If the family has five members, the value of the "fixed number" may be 4. The value of the "fixed number" may be determined based on the history of the number of users 10 recognized by the meaning understanding unit 418. For example, even if the family has five members, if the meaning understanding unit 418 often only recognizes four members, the value of the "fixed number" may be determined to be 3.
[0109] 14 shows an example of an operation flow for switching the recording format. This operation flow starts when the recording data generating section 470 starts generating recording data in a high compression format.
[0110] In S1402, the perception processing unit 416 starts detecting the number of users 10 belonging to a specific family and the number of users 10 whose smile degree exceeds a predetermined value based on the information output by the sensor unit 400 and the sensor module unit 410. Here, the perception processing unit 416 counts the number of users 10 who are registered as family members in the person DB 424 among the users whose presence is recognized based on the information output by the sensor unit 400 and the sensor module unit 410. In addition, the perception processing unit 416 counts the number of users 10 who are registered as family members in the person DB 424 among the users whose smile degree exceeds a predetermined value.
[0111] In S1404, the switching control unit 477 judges whether or not the condition part of the rule 1300 that specifies switching the recording format to a low compression format is satisfied. If it is judged that the condition part of the rule 1300 is satisfied, the behavior control unit 450 causes the speaker of the controlled object 452 to output a voice with the utterance content "It looks like fun, so I'll take a picture!" (S1406). This notifies the user 10 that the video will be recorded.
[0112] In S1408, the switching control unit 477 judges whether or not there has been a negative reaction from the user 10. For example, the switching control unit 477 judges that there has been no negative reaction from the user 10 when no negative words or phrases are recognized from the voice of the user 10. When it is judged that there has been no negative reaction from the user 10, the switching control unit 477 controls the recording data generating unit 470 to start generating full HD video data and audio data in addition to the skeleton data (S1410). In this way, the switching control unit 477 switches the recording format from the high compression format to the low compression format. When it is judged that there has been a negative reaction from the user 10 in the judgment of S1408, the process proceeds to S1404 after a predetermined wait time has elapsed (S1420).
[0113] After switching to the low compression format in S1410, it is determined in S1412 whether or not the conditional part of rule 1310, which specifies switching to the high compression format, is satisfied. If it is determined that the conditional part of rule 1310 is satisfied, the switching control unit 477 switches the recording format from the low compression format to the high compression format (S1414). If it is determined in the determination in S1412 that the conditional part of rule 1310 is not satisfied, the determination in S1412 is repeated until the conditional part of rule 1310 is satisfied.
[0114] In addition to the judgment based on rule 1310 in S1412, the recording format may be switched from the low compression format to the high compression format when an utterance instructing to stop recording in the low compression format, such as "end recording," is recognized. Furthermore, after switching the recording format to the low compression format, if a predetermined maximum recording time has elapsed, the recording format may be switched from the low compression format to the high compression format.
[0115] Note that the switching control unit 477 may switch the recording format in response to a drastic change in the intensity of the emotion of the user 10, not limited to the number of smiles. For example, the switching control unit 477 may switch the recording format to a low compression format when a touch sensor included in the sensor unit 400 detects an impact stronger than a predetermined value.
[0116] [Record data in a manner that reflects the emotions of the user or robot] So far, it has been explained that the robot 100 switches the recording format of the data to be recorded in the server 200 based on the user's emotion. For example, the recording format may be a high compression format or a low compression format.
[0117] On the other hand, the robot 100 may change the data recording mode based on the emotion of the user or the robot 100 itself. For example, the recording mode is the type of data, such as text, still image, audio, and moving image.
[0118] As described above, the robot 100 records data (e.g., voice data, skeletal data, image data) obtained by processing information detected by a sensor that continuously detects. The robot 100 may also record text data obtained by converting information detected by a sensor that continuously detects.
[0119] As described above, the robot 100 can express its own emotions using emotion values corresponding to a number of types of emotions, such as "happiness," "anger," "sorrow," and "pleasure."
[0120] The robot 100 can also detect changes and strengths of the user's emotions. Here, the robot 100 represents the user's emotions by emotion values for each type of emotion.
[0121] That is, the robot 100 can convert emotions of both the user and the robot 100 itself into structured data according to the type and emotion value of the emotion. For example, the emotion determination unit 442 determines the emotion value indicating the strength of the emotion of the robot or the user, or the type of emotion, based on information detected by a sensor provided in the robot. In addition, the robot 100 specifies the recording mode based on the emotion (type and emotion value) converted into structured data.
[0122] As a method for converting the emotions of the user and the robot 100 into structured data, the same method as that described in this paper is disclosed in Japanese Patent No. 6058053. In addition, the robot 100 may identify the emotions of the user or the robot 100 itself by known methods such as a method using an emotion map (see, for example, Japanese Patent No. 6273313) and a method using a neural network (see, for example, Japanese Patent No. 6199927).
[0123] The data to be recorded here includes the data described above. That is, the robot 100 can record information (hereinafter, main information) such as still images (e.g., facial images) of the user, moving images, recorded voice, and text (e.g., labels of supplementary information) as data. Furthermore, the robot 100 can record peripheral information such as the smell, temperature, humidity, position, and weather of the location where the robot 100 is present, and the number and details of nearby users as data.
[0124] For example, when the emotion value of the robot 100's negative emotion (such as "anger" or "sadness") increases, there is a possibility that the robot 100 or the user is suffering from a disadvantage or danger. For example, when the user is seriously injured, the emotion value of "sadness" of the robot 100 may increase significantly. Also, when the robot 100 is physically hit, the emotion value of "anger" of the robot 100 may increase significantly.
[0125] In order to be able to later consider countermeasures and avoidance measures for such events that cause disadvantage and danger to the robot 100 or the user, it is desirable for the data to be recorded in a (rich) recording format that provides more information about the event (e.g., long video images, detailed text).
[0126] Also, for example, when the emotion value of the positive emotion (such as "happiness" or "fun") of the robot 100 increases, the user may want to later recall the situation at that time in detail.
[0127] Therefore, when the emotion value of the user's positive emotion becomes large, it is desirable to record the data in a recording format (e.g., audio, high-quality video images) that allows the user to easily recall the situation at that time. This allows the user to refer to the recorded data and recall, for example, a situation when they had fun.
[0128] The relationship between the emotions of the robot 100 and the user and the recording mode is not limited to that described here, and can be set arbitrarily in advance. Here, the relationship between the emotions and the recording mode is stored in a pattern table and a recording mode table. The pattern table and the recording mode table are stored in the storage unit 520.
[0129] Record data generation section 470 specifies a recording format for recording data indicating information about the user detected by the sensor, based on the emotion value. Record data generation section 470 is an example of a specification section.
[0130] The record data generation unit 470 refers to the pattern table and the record mode table stored in the storage unit 520, and specifies the record mode according to the emotion of the robot 100 or the user. The record data generation unit 470 generates data based on the specified record mode. Note that the record data generation unit 470 may specify the record mode based on both the emotion value and the type of emotion, or may specify the record mode based on either one of them.
[0131] Communication processing unit 480 records the generated data in server 200. That is, communication processing unit 480 records the data based on the recording mode specified by record data generating unit 470. Communication processing unit 480 is an example of a record control unit.
[0132] An example of a pattern table is shown in Fig. 15. As shown in Fig. 15, the pattern table has columns such as "subject", "type", "emotion value range", and "pattern". "Subject", "type", and "emotion value range" correspond to conditions. Recorded data generation unit 470 identifies a pattern that matches the conditions based on the emotional value of the user or robot 100 and the pattern table.
[0133] For example, when the emotion value of the emotion of "happiness" of the robot 100 is 0.7 (0.5 or more), the recorded data generating unit 470 identifies a pattern "P11" that matches the condition based on the record shown in the first row of the pattern table in Fig. 15. Note that the emotion value range is assumed to be from 0 to 1 here.
[0134] Also, for example, when the emotion value of the user's emotion of “easy” is 0.9 (0.8 or more), the recorded data generating section 470 identifies the pattern “P14” that matches the condition based on the record shown in the fourth row of the pattern table in FIG.
[0135] An example of the recording mode table is shown in Fig. 16. The recording data generating unit 470 specifies the recording mode based on the pattern specified based on the pattern table and the recording mode table.
[0136] 16, the recording mode table has a column for "pattern" and a column for indicating the recording mode. The column for indicating the recording mode includes "text", "audio", "still image", "moving image (low quality)", "moving image (high quality)", "length", "capacity", and "peripheral information".
[0137] "Text", "Audio", "Still image", "Video (low quality)", and "Video (high quality)" are columns indicating the type of data. A value of 1 in the column indicating the type of data means that the corresponding type of data is recorded. Recorded data generation unit 470 can specify, as the recording mode, whether the data includes text, audio, still image, or video.
[0138] "Length" is the maximum length of time-definable data (e.g., audio and video). Also, "capacity" is the maximum capacity of data to be recorded. Recorded data generation unit 470 can specify the maximum capacity of data as the recording mode. Also, recorded data generation unit 470 can specify the maximum length of time-definable data as the recording mode.
[0139] The "peripheral information" is information recorded as data together with the main information (text, voice, image). The robot 100 acquires peripheral information such as odor, temperature, humidity, location, weather, and the number and details of nearby users using the sensor unit 400. The details of nearby users are information equivalent to the main information (e.g., face images, etc.) related to users in the vicinity of the user whose main information is to be recorded.
[0140] The recorded data generating unit 470 can specify, as the recording mode, which of the following should be included in the data: the smell, temperature, humidity, location, weather, and the number and details of nearby users in the location where the robot is present.
[0141] For example, when the recording data generation unit 470 identifies the pattern "P11", it identifies the recording mode to record up to 500 MB of data including "text" and "still images" together with temperature, humidity, and position (position information of the robot 100).
[0142] Furthermore, for example, when the recording data generating unit 470 specifies "P12" as the pattern, it specifies the recording mode in which "text", "audio", "still image", "moving image (low quality)", and "moving image (high quality)" are recorded with no limit on length or capacity, together with smell, temperature, humidity, location, weather, and the number and details of nearby users. The recording mode of pattern "P12" is the recording mode that can include the most information (richest) among the recording modes that can be expressed in the recording mode table.
[0143] Note that the "video (low quality)" and the "video (high quality)" have different resolutions and frame rates. For example, the "video (low quality)" and the "video (high quality)" are full HD images at 30 FPS and 4K images at 60 FPS, respectively.
[0144] The recording mode may vary not only according to the strength of the emotion (the magnitude of the emotion value) but also according to the type of emotion. In other words, the recorded data may differ depending on the type of emotion felt by the robot 100 or the user. This can be achieved by setting the pattern table and the recording mode table.
[0145] For example, a setting may be made so that a moving image is not recorded for the emotion type "joy" and a still image is recorded, and a setting may be made so that a moving image is recorded for the emotion type "pleasure."
[0146] In addition, for image and audio data, targets may be set for each type of emotion. For example, for the emotion type of "happy," a paper-cut image of the face and audio of another user (other user) who is having a conversation with the target user may be set to be recorded. Also, for example, for the emotion type of "anger," a paper-cut image of the other user's face may be set to be recorded.
[0147] Furthermore, as shown in the recording mode table of Fig. 16, the peripheral information to be recorded differs depending on the type of emotion. For example, when the type of emotion of the robot 100 is "sadness" (pattern "P11"), "weather" is not recorded as peripheral information, whereas when the type of emotion of the robot 100 is "anger" (pattern "P12"), "weather" is recorded as peripheral information.
[0148] The robot 100 records the data based on the specified recording format. That is, the record data generating unit 470 generates data according to the specified recording format.
[0149] The communication processing unit 480 causes the server 200 to record the generated data.
[0150] An example of a process flow for recording data in a recording format according to emotions is shown in Fig. 17. As shown in Fig. 17, first, the robot 100 acquires the type and emotion value of the emotion of the user or the robot 100 itself (step S2101).
[0151] Next, the robot 100 identifies a pattern that matches the conditions of the type of emotion and the emotion value of the user or the robot 100 (step S2102). For example, the robot 100 identifies the pattern by referring to a pattern table.
[0152] Next, the robot 100 identifies a recording mode corresponding to the identified pattern (step S2103). For example, the robot 100 identifies the pattern by referring to a recording mode table.
[0153] Here, the robot 100 generates data based on the recording mode (step S2104), and records the generated data (step S2105). For example, the robot 100 causes the server 200 to record data.
[0154] In this way, the robot 100 records data in a recording mode according to the emotion, and thus can appropriately record data according to the emotion of the user or the robot 100. Furthermore, the robot 100 can assist in dealing with disadvantages and dangers that arise for the robot 100 or the user. In addition, the robot 100 can thereby assist the user in remembering a situation that the user wishes to remember.
[0155] The robot 100 is an example of an electronic device equipped with a behavior control system. The application of the behavior control system is not limited to the robot 100, and the behavior control system can be applied to various electronic devices. The functions of the server 200 may be implemented by one or more computers. At least a part of the functions of the server 200 may be implemented by a virtual machine. At least a part of the functions of the server 200 may be implemented in the cloud.
[0156] 18 is a diagram illustrating an example of a hardware configuration of a computer 1200 functioning as the robot 100 and the server 200. A program installed in the computer 1200 can cause the computer 1200 to function as one or more "parts" of the apparatus according to the present embodiment, or cause the computer 1200 to execute operations or one or more "parts" associated with the apparatus according to the present embodiment, and / or cause the computer 1200 to execute a process or steps of the process according to the present embodiment. Such a program can be executed by the CPU 1212 to cause the computer 1200 to execute specific operations associated with some or all of the blocks of the flowcharts and block diagrams described herein.
[0157] The computer 1200 according to this embodiment includes a CPU 1212, a RAM 1214, and a graphic controller 1216, which are connected to each other by a host controller 1210. The computer 1200 also includes input / output units such as a communication interface 1222, a storage device 1224, a DVD drive, and an IC card drive, which are connected to the host controller 1210 via an input / output controller 1220. The DVD drive may be a DVD-ROM drive, a DVD-RAM drive, etc. The storage device 1224 may be a hard disk drive, a solid state drive, etc. The computer 1200 also includes input / output units such as a ROM 1230 and a keyboard, which are connected to the input / output controller 1220 via an input / output chip 1240.
[0158] The CPU 1212 operates according to a program stored in the ROM 1230 and the RAM 1214, thereby controlling each unit. The graphic controller 1216 retrieves image data generated by the CPU 1212 into a frame buffer or the like provided in the RAM 1214 or into itself, and causes the image data to be displayed on the display device 1218.
[0159] The communication interface 1222 communicates with other electronic devices via a network. The storage device 1224 stores programs and data used by the CPU 1212 in the computer 1200. The DVD drive reads programs or data from a DVD-ROM or the like and provides them to the storage device 1224. The IC card drive reads programs and data from an IC card and / or writes programs and data to an IC card.
[0160] The ROM 1230 stores therein a boot program or the like that is executed by the computer 1200 upon activation, and / or a program that depends on the hardware of the computer 1200. The input / output chip 1240 may also connect various input / output units to the input / output controller 1220 via a USB port, a parallel port, a serial port, a keyboard port, a mouse port, and the like.
[0161] The programs are provided by a computer-readable storage medium such as a DVD-ROM or an IC card. The programs are read from the computer-readable storage medium, installed in the storage device 1224, the RAM 1214, or the ROM 1230, which are also examples of computer-readable storage media, and executed by the CPU 1212. Information processing described in these programs is read by the computer 1200, and brings about cooperation between the programs and the various types of hardware resources described above. An apparatus or method may be constructed by implementing operations or processing of information according to the use of the computer 1200.
[0162] For example, when communication is performed between the computer 1200 and an external device, the CPU 1212 may execute a communication program loaded in the RAM 1214 and instruct the communication interface 1222 to perform communication processing based on the processing described in the communication program. Under the control of the CPU 1212, the communication interface 1222 reads transmission data stored in a transmission buffer area provided in the RAM 1214, the storage device 1224, a DVD-ROM, or a recording medium such as an IC card, and transmits the read transmission data to the network, or writes reception data received from the network to a reception buffer area or the like provided on the recording medium.
[0163] Furthermore, the CPU 1212 may cause all or a necessary portion of a file or database stored in an external recording medium such as the storage device 1224, a DVD drive (DVD-ROM), an IC card, etc. to be read into the RAM 1214, and perform various types of processing on the data on the RAM 1214. The CPU 1212 may then write back the processed data to the external recording medium.
[0164] Various types of information, such as various types of programs, data, tables, and databases, may be stored in the recording medium and undergo information processing. The CPU 1212 may perform various types of processing on the data read from the RAM 1214, including various types of operations, information processing, conditional judgment, conditional branching, unconditional branching, information search / replacement, etc., as described throughout this disclosure and specified by the instruction sequence of the program, and write back the results to the RAM 1214. The CPU 1212 may also search for information in a file, database, etc. in the recording medium. For example, when a plurality of entries each having an attribute value of a first attribute associated with an attribute value of a second attribute are stored in the recording medium, the CPU 1212 may search for an entry whose attribute value of the first attribute matches a specified condition from among the plurality of entries, read the attribute value of the second attribute stored in the entry, and thereby obtain the attribute value of the second attribute associated with the first attribute that satisfies a predetermined condition.
[0165] The above-mentioned programs or software modules may be stored in a computer-readable storage medium on the computer 1200 or in the vicinity of the computer 1200. In addition, a recording medium such as a hard disk or a RAM provided in a server system connected to a dedicated communication network or the Internet can be used as a computer-readable storage medium, thereby providing the programs to the computer 1200 via the network.
[0166] The blocks in the flowcharts and block diagrams in the present embodiment may represent stages of a process in which an operation is performed or "parts" of an apparatus responsible for performing the operation. Particular stages and "parts" may be implemented by dedicated circuitry, programmable circuitry provided with computer-readable instructions stored on a computer-readable storage medium, and / or a processor provided with computer-readable instructions stored on a computer-readable storage medium. The dedicated circuitry may include digital and / or analog hardware circuits, and may include integrated circuits (ICs) and / or discrete circuits. The programmable circuitry may include reconfigurable hardware circuits, such as, for example, field programmable gate arrays (FPGAs), programmable logic arrays (PLAs), and the like, including AND, OR, XOR, NAND, NOR, and other logical operations, flip-flops, registers, and memory elements.
[0167] A computer-readable storage medium may include any tangible device capable of storing instructions that are executed by a suitable device, such that a computer-readable storage medium having instructions stored thereon comprises an article of manufacture that includes instructions that can be executed to create means for performing the operations specified in the flowcharts or block diagrams. Examples of computer-readable storage media may include electronic storage media, magnetic storage media, optical storage media, electromagnetic storage media, semiconductor storage media, and the like. More specific examples of computer-readable storage media may include floppy disks, diskettes, hard disks, random access memories (RAMs), read-only memories (ROMs), erasable programmable read-only memories (EPROMs or flash memories), electrically erasable programmable read-only memories (EEPROMs), static random access memories (SRAMs), compact disk read-only memories (CD-ROMs), digital versatile disks (DVDs), Blu-ray disks, memory sticks, integrated circuit cards, and the like.
[0168] The computer readable instructions may include either assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or either source code or object code written in any combination of one or more programming languages, including object-oriented programming languages such as Smalltalk®, JAVA®, C++, etc., and conventional procedural programming languages such as the “C” programming language or similar programming languages.
[0169] Computer readable instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus, or to a programmable circuit, either locally or over a local area network (LAN), a wide area network (WAN), such as the Internet, etc., to cause the processor of the general purpose computer, special purpose computer, or other programmable data processing apparatus, or to a programmable circuit, to execute the computer readable instructions to generate means for performing the operations specified in the flowcharts or block diagrams. Examples of processors include computer processors, processing units, microprocessors, digital signal processors, controllers, microcontrollers, etc.
[0170] Although the present invention has been described above using the embodiments, the technical scope of the present invention is not limited to the scope described in the above embodiments. It is clear to those skilled in the art that various modifications and improvements can be made to the above embodiments. It is clear from the description of the claims that such modifications and improvements can also be included in the technical scope of the present invention.
[0171] It should be noted that the order of execution of each process, such as operations, procedures, steps, and stages, in the devices, systems, programs, and methods shown in the claims, specifications, and drawings is not specifically stated as "before" or "prior to," and that the process may be performed in any order unless the output of a previous process is used in a later process. Even if the operational flow in the claims, specifications, and drawings is described using "first," "next," and the like for convenience, this does not mean that the process must be performed in this order. [Explanation of symbols]
[0172] 5. System 10, 11, 12 users 20 Communication Network 100, 101, 102 Robot 200 Servers 400 Sensor section 401 Mike 402 3D Depth Sensor 403 2D Camera 404 Distance Sensor 410 Sensor module part 411 Voice Emotion Recognition Unit 412 Speech Understanding Unit 413 Facial expression recognition unit 414 Face Recognition Department 416 Perception Processing Section 417 Perception Confirmation Rules 418 Semantic understanding part 419 Semantic Comprehension Rules 420 Storage Unit 421 Reaction Rules 422 Rules Update Department 424 Person DB 430 User state recognition unit 432 User Reaction Recognition Unit 434 Condition judgment section 440 Action Decision-Making Department 442 Emotion Determination Department 450 Behavior Control Unit 452 Controlled 470 Recording Data Generation Unit 471 Recording format switching rules 477 Switching control section 478 Label Generation Unit 480 Communication Processing Unit 520 Storage Unit 521 Reaction Rules 522 Reaction Rule Statistics 530 Reaction rule generation unit 540 Reaction rule search unit 550 Statistical Information Control Unit 560 Access Control Unit 561 Recorded Data 580 Communication Processing Unit 601 Person Detection Information 602 Face detection information 603 User Identity Information 604 Smile Detection Information 605 Timeout Information 610 Perception History 611, 612, 613, 614, 615, 616 Perceptual information 630 Perception Confirmation Rules 710 Semantic History 711, 712, 713, 714, 715, 717 Semantic information 730 Semantic Comprehension Rules 810 Reaction Rules 910 Candidates 940 Semantic Information 960 User Responses 1300, 1310 Rules
Claims
1. an emotion determining unit that determines an emotion value indicating a strength of an emotion of the robot or a user, or a type of emotion, based on information detected by a sensor provided in the robot; a determination unit that determines a manner in which data indicating information about the user detected by the sensor is to be recorded based on information determined by the emotion determination unit; a recording control unit that records the data based on the aspect identified by the identification unit; A recording control system comprising:
2. The specification unit specifies whether the data includes text, audio, a still image, or a moving image as a recording mode of the data. The recording control system according to claim 1 .
3. The specifying unit specifies the maximum capacity of the data as a mode for recording the data. The recording control system according to claim 1 .
4. The specification unit specifies a maximum length of the data that can define time as a manner of recording the data. The recording control system according to claim 1 .
5. The identification unit identifies, as a manner of recording the data, which of the odor, temperature, humidity, position, weather, and the number and details of nearby users of the location where the robot is located. The recording control system according to claim 1 .
6. an emotion determination step of determining an emotion value indicating the strength of an emotion of the robot or a user, or a type of emotion, based on information detected by a sensor provided in the robot; a determination step of determining a manner in which data indicating information about the user detected by the sensor is to be recorded based on the information determined by the emotion determination step; a recording control step of recording the data based on the aspect specified by the specifying step; A recording control program that causes a computer to execute the above.
Citation Information
Patent Citations
Feeling generator and method therefor
JP1998254592A
Terminal and reply information creation program for pseudo conversation
JP2011253389A
Methods for memorizing spacetime patterns, systems for doing so, and computer programs for doing so.
JP2013535067A
Cited By
Display device, machine learning device, inference device, information processing method, machine learning method, and inference method
JP2026149050A
Display device, machine learning device, inference device, information processing method, machine learning method, and inference method
JP7796446B1
