Robot autonomously selecting behavior according to internal state or external environment

Abstract

To provide a user with sense of security when living with robots.SOLUTION: An autonomous robot includes: an operation control unit which selects the motion of the robot; a driving mechanism which executes the motion selected by the operation control unit; a recognition unit which determines whether or not a target object satisfies a predetermined watching condition; a mode setting unit which sets a watching mode of the target object when the watching condition has been established; and a communication unit which transmits a taken image of the target object to a predetermined communication terminal in the watching mode.SELECTED DRAWING: Figure 1A

Description

Technical Field

[0001] The present invention relates to a robot that autonomously makes action selections according to an internal state or an external environment.

Background Art

[0002] People keep pets in search of healing. On the other hand, many people give up keeping pets for various reasons such as not being able to secure enough time to take care of pets, not having a living environment where they can keep pets, having allergies, and finding it painful to experience the death of a pet. If there were a robot that could serve the role of a pet, it might be able to give those who cannot keep pets the same kind of healing that a pet provides (see Patent Documents 1 and 2).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0004] In recent years, although robot technology has been rapidly advancing, it has not yet achieved the presence as a companion like a pet. This is because it is difficult to think that a robot has free will. People observe the actions of pets in such a way that they can only think that the pets have free will, feel the existence of free will in the pets, empathize with the pets, and are healed by the pets.

[0005] The present invention is an invention completed based on the recognition of the above problems, and its first object is to provide a technology for providing various senses of security to users in living with robots. Further, the second object is to provide a technology for richly expressing the love of the robot for the user. Further, the third object is to provide a technology for a plurality of robots to act in cooperation.

Means for Solving the Problems

[0006] The autonomous behavior type robot in a certain aspect of the present invention includes an operation control unit that selects the motion of the robot, a drive mechanism that executes the motion selected by the operation control unit, a recognition unit that determines whether the target person satisfies a predetermined monitoring condition, a mode setting unit that sets the target person to the monitoring mode when the monitoring condition is satisfied, and a communication unit that transmits the captured image of the target person to a predetermined communication terminal in the monitoring mode.

Effects of the Invention

[0007] According to the present invention, it becomes easier to further enhance the presence of the robot.

Brief Description of the Drawings

[0008] The above-described objects, as well as other objects, features, and advantages, will become even more apparent from the preferred embodiments described below and the accompanying drawings below.

[0009]

Figure 1A

Figure 1B

Figure 2

Figure 3

Figure 4

Figure 5A

Figure 5B

Figure 5C

Figure 5D

Figure 6A

Figure 6B

Figure 6C

Figure 6D

Figure 7A

Figure 7B

Figure 7C

Figure 7D

Figure 7E

Figure 8A

Figure 8B

Figure 8C

Figure 8D

Figure 9A

Figure 9B

Figure 9C

Figure 9D

Figure 10A

Figure 10B

Figure 10C

Figure 11A

Figure 11B

Figure 11C

Figure 12A

Figure 12B

Figure 12C

Figure 12D

Figure 13A

Figure 13B

Figure 13C

Figure 13D

Figure 14A

Figure 14B

Figure 14C

Figure 14D

Figure 15A

Figure 15B

Figure 15C

Figure 15D

Figure 16A

Figure 16B

Figure 16C

Figure 16D

Figure 17A

Figure 17B

Figure 17C

Figure 17D

Figure 18A

Figure 18B

Figure 18C

Mode for Carrying Out the Invention

[0010] The robot 100 in this embodiment shares daily life with the user, sometimes shows consideration for the user, sometimes struggles to be useful to the user, and actively seeks the user's love, thereby exerting a sense of presence as a member of the family. Hereinafter, after explaining the basic configuration of the robot 100 with reference to FIGS. 1 to 4, various behavior scenes of the robot 100 will be described.

[0011] [Basic Configuration] FIG. 1 is a diagram showing the appearance of the robot 100. FIG. 1A is a front view, and FIG. 1B is a side view. Robot 100 is an autonomous mobile robot that determines its actions based on the external environment and internal state. The external environment is recognized by various sensors such as cameras and a thermosensor 115. The internal state is quantified as various parameters representing the emotions of robot 100. Robot 100 operates within the house of the owner's family. Hereinafter, a person related to robot 100 is referred to as a "user". Among the users, the owner or administrator of robot 100 is referred to as the "owner".

[0012] The body 104 of robot 100 has an overall rounded shape and includes an outer skin 314 formed of a soft and elastic material such as urethane, rubber, resin, or fiber. Robot 100 may be dressed in clothes. The total weight of robot 100 is about 5 to 15 kilograms, and the height is about 0.5 to 1.2 meters. Due to its appropriate weight, roundness, softness, and pleasant touch, the user can easily hold robot 100 in their arms and has the desire to do so.

[0013] Robot 100 includes a pair of front wheels 102 (left wheel 102a, right wheel 102b) and one rear wheel 103. The front wheels 102 are drive wheels, and the rear wheel 103 is a driven wheel. The front wheels 102 do not have a steering mechanism, but the rotational speed and direction of the left and right wheels can be controlled individually. The rear wheel 103 is a caster and is rotatable to move robot 100 forward, backward, left, and right. The rear wheel 103 may be an omni-wheel. By increasing the rotational speed of the right wheel 102b more than that of the left wheel 102a, robot 100 can turn left or rotate counterclockwise. By increasing the rotational speed of the left wheel 102a more than that of the right wheel 102b, robot 100 can turn right or rotate clockwise.

[0014] The front wheel 102 and the rear wheel 103 can be completely housed in the body 104 by a drive mechanism (rotation mechanism, link mechanism). A pair of left and right covers 312 are provided in the lower half of the body 104. The cover 312 is made of a resin material (rubber, silicone rubber, etc.) having flexibility and elasticity, constitutes a soft body, and can house the front wheel 102. A slit 313 (opening) that opens from the side surface to the front surface is formed in the cover 312, and the front wheel 102 can be advanced through the slit 313 and exposed to the outside.

[0015] Even during traveling, most of each wheel is hidden by the body 104, but when each wheel is completely housed in the body 104, the robot 100 becomes in a non-movable state. That is, as the wheels are housed, the body 104 descends and sits on the floor surface F. In this seated state, a flat seating surface 108 (grounding bottom surface) formed at the bottom of the body 104 abuts on the floor surface F.

[0016] The robot 100 has two arms 106. There are hands at the tips of the arms 106, but they do not have the function of gripping objects. The arms 106 can perform simple operations such as raising, bending, waving, and vibrating by the drive of an actuator described later. The two arms 106 can be controlled individually.

[0017] A face area 116 is exposed on the front of the head of the robot 100. Two eyes 110 are provided in the face area 116. The eyes 110 can display images by liquid crystal elements or organic EL elements, and are devices for expressing the line of sight and expressions by moving the pupils and eyelids displayed as images. A nose 109 is provided in the center of the face area 116. An analog stick is provided on the nose 109, and in addition to all directions of up, down, left, and right, the pushing direction can also be detected. Also, a plurality of touch sensors are provided on the robot 100, and can detect the touch of a user over almost the entire area of the robot 100, such as the head, torso, hips, and arms. The robot 100 is equipped with various sensors such as a microphone array and an ultrasonic sensor that can specify the sound source direction. It also has a built-in speaker and can emit simple sounds.

[0018] The horn 112 is attached to the head of the robot 100. An omnidirectional camera 113 is attached to the horn 112, and it can image the entire upper area of the robot 100 at once. The horn 112 also incorporates a thermosensor 115 (thermal camera). Also, a plurality of modules (not shown) for infrared communication are provided on the horn 112, and these modules are installed annularly facing the surroundings. For this reason, the robot 100 can perform infrared communication while recognizing directions. Furthermore, an emergency stop switch is provided on the horn 112, and the user can emergency-stop the robot 100 by pulling out the horn 112.

[0019] Figure 2 is a cross-sectional view schematically showing the structure of the robot 100. The body 104 includes a main body frame 310, a pair of arms 106, a pair of covers 312, and an outer skin 314. The main body frame 310 includes a head frame 316 and a body frame 318. The head frame 316 has a hollow hemispherical shape and forms the head skeleton of the robot 100. The body frame 318 has a rectangular tube shape and forms the body skeleton of the robot 100. The lower end of the body frame 318 is fixed to the lower plate 334. The head frame 316 is connected to the body frame 318 via a connection mechanism 330.

[0020] The body frame 318 constitutes the axis of the body 104. The body frame 318 is configured by fixing a pair of side plates 336 to the left and right on the lower plate 334, and supports a pair of arms 106 and internal mechanisms. Inside the body frame 318, a battery 118, a control circuit 342, various actuators, etc. are accommodated. The bottom surface of the lower plate 334 forms the seating surface 108.

[0021] The body frame 318 has an upper plate 332 at its upper part. A bottomed cylindrical support portion 319 is fixed to the upper plate 332. The upper plate 332, the lower plate 334, the pair of side plates 336 and the support portion 319 constitute the body frame 318. The outer diameter of the support portion 319 is smaller than the interval between the left and right side plates 336. The pair of arms 106 are assembled integrally with an annular member 340 to constitute an arm unit 350. The annular member 340 is annular, and the pair of arms 106 are attached so as to be radially separated on its center line. The annular member 340 is coaxially inserted into the support portion 319 and is placed on the upper end surfaces of the pair of side plates 336. The arm unit 350 is supported from below by the body frame 318.

[0022] The head frame 316 has a yaw axis 321, a pitch axis 322 and a roll axis 323. The pivoting (yawing) around the yaw axis 321 of the head frame 316 realizes a head shaking operation, the pivoting (pitching) around the pitch axis 322 realizes a nodding operation, a looking up operation and a looking down operation, and the pivoting (rolling) around the roll axis 323 realizes an operation of tilting the head left and right. Each axis can change its position and angle in a three-dimensional space according to the driving mode of the connection mechanism 330. The connection mechanism 330 is composed of a link mechanism and is driven by a plurality of motors installed on the body frame 318.

[0023] The body frame 318 houses a wheel drive mechanism 370. The wheel drive mechanism 370 includes a front wheel drive mechanism and a rear wheel drive mechanism for respectively taking in and out the front wheels 102 and the rear wheels 103 from the body 104. The front wheels 102 and the rear wheels 103 function as a "moving mechanism" for moving the robot 100. The front wheel 102 has a direct drive motor at its central part. Therefore, the left wheel 102a and the right wheel 102b can be driven individually. The front wheel 102 is rotatably supported by the wheel cover 105, and the wheel cover 105 is rotatably supported by the body frame 318. The wheel cover 105 is rotatably supported by the body frame 318.

[0024] A pair of covers 312 are provided to cover the body frame 318 from the left and right, and have a smooth curved surface shape so as to give a roundness to the outline of the body 104. A closed space is formed between the body frame 318 and the cover 312, and this closed space serves as the accommodation space S for the front wheels 102. The rear wheels 103 are accommodated in an accommodation space provided at the lower rear of the body frame 318.

[0025] The outer skin 314 covers the main body frame 310 and the pair of arms 106 from the outside. The outer skin 314 has a thickness such that a person can feel elasticity, and is formed of a stretchable material such as urethane sponge. Thereby, when the user hugs the robot 100, the user feels appropriate softness and can have a natural physical contact as if the user were treating the robot as a pet. The outer skin 314 is attached to the main body frame 310 in a manner that exposes the cover 312. An opening 390 is provided at the upper end of the outer skin 314. The horn 112 is inserted through this opening 390.

[0026] Touch sensors are disposed between the main body frame 310 and the outer skin 314. Touch sensors are embedded in the cover 312. All of these touch sensors are capacitance sensors and detect touches over substantially the entire area of the robot 100. Note that the touch sensors may be embedded in the outer skin 314 or may be disposed inside the main body frame 310.

[0027] The arm 106 has a first joint 352 and a second joint 354, has an arm 356 between both joints, and has a hand 358 at the tip of the second joint 354. The first joint 352 corresponds to the shoulder joint, and the second joint 354 corresponds to the wrist joint. A motor is provided at each joint to drive the arm 356 and the hand 358 respectively. The drive mechanism for driving the arm 106 includes these motors and their drive circuits 344.

[0028] Figure 3 is a hardware configuration diagram of the robot 100. Robot 100 includes an internal sensor 128, a communication device 126, a storage device 124, a processor 122, a drive mechanism 120, and a battery 118. The drive mechanism 120 includes the connection mechanism 330 and the wheel drive mechanism 370 described above. The processor 122 and the storage device 124 are included in the control circuit 342. Each unit is connected to each other by a power line 130 and a signal line 132. The battery 118 supplies power to each unit via the power line 130. Each unit transmits and receives control signals via the signal line 132. The battery 118 is a lithium-ion secondary battery and is the power source of the robot 100.

[0029] The internal sensor 128 is an aggregate of various sensors built into the robot 100. Specifically, it includes a camera, a microphone array, a distance measuring sensor (infrared sensor), a thermosensor 115, a touch sensor, an acceleration sensor, a barometric pressure sensor, a smell sensor, etc. The touch sensor corresponds to most regions of the body 104 and detects a user's touch based on a change in capacitance. The smell sensor is a known sensor that applies the principle that the electrical resistance changes due to the adsorption of molecules that are the source of the smell.

[0030] The communication device 126 is a communication module that performs wireless communication with various external devices. The storage device 124 is composed of a non-volatile memory and a volatile memory, and stores computer programs and various setting information. The processor 122 is a means for executing a computer program. The drive mechanism 120 includes a plurality of actuators. In addition, a display, a speaker, etc. are also mounted.

[0031] The drive mechanism 120 mainly controls the wheels and the head. The drive mechanism 120 can change the moving direction and moving speed of the robot 1 00, and can also raise and lower the wheels. When the wheels rise, the wheels are completely housed in the body 104, and the robot 100 abuts on the floor surface F at the seating surface 108 and assumes a seated state. In addition, the drive mechanism 120 controls the arm 106.

[0032] Figure 4 is a functional block diagram of the robot system 300. The robot system 300 includes a robot 100, a server 200, and a plurality of external sensors 114. Each component of the robot 100 and the server 200 is realized by hardware including an arithmetic unit such as a CPU (Central Processing Unit) and various coprocessors, a storage device such as a memory and a storage, and a wired or wireless communication line connecting them, and software stored in the storage device and supplying processing instructions to the arithmetic unit. The computer program may be composed of a device driver, an operating system, various application programs located in the upper layers thereof, and a library providing common functions to these programs. Each block described below indicates a block of a functional unit, not a configuration of a hardware unit. A part of the functions of the robot 100 may be realized by the server 200, or a part or all of the functions of the server 200 may be realized by the robot 100.

[0033] A plurality of external sensors 114 are installed in the house in advance. The server 200 manages the external sensors 114 and provides the detection values acquired by the external sensors 114 to the robot 100 as necessary. The robot 100 determines basic actions based on the information obtained from the internal sensor 128 and the plurality of external sensors 114. The external sensors 114 are for reinforcing the sensory organs of the robot 100, and the server 200 is for reinforcing the processing ability of the robot 100. The communication device 126 of the robot 100 may communicate with the server 200 periodically, and the server 200 may be responsible for the process of identifying the position of the robot 100 by the external sensors 114 (see also Patent Document 2).

[0034] (Server 200) The server 200 includes a communication unit 204, a data processing unit 202, and a data storage unit 206. The communication unit 204 is responsible for communication processing with the external sensor 114 and the robot 100. The data storage unit 206 stores various types of data. The data processing unit 202 executes various processes based on the data acquired by the communication unit 204 and the data stored in the data storage unit 206. The data processing unit 202 also functions as an interface for the communication unit 204 and the data storage unit 206.

[0035] The data storage unit 206 includes a motion storage unit 232 and a personal data storage unit 218. The robot 100 has a plurality of motion patterns (motions). Various motions are defined, such as shaking the arm 106, approaching the owner while meandering, and staring at the owner while tilting the head.

[0036] The motion storage unit 232 stores "motion files" that define the control content of motions. Each motion is identified by a motion ID. The motion files are also downloaded to the motion storage unit 160 of the robot 100. Whether to execute which motion may be determined by the server 200 or by the robot 100.

[0037] Many of the motions of the robot 100 are configured as composite motions that include a plurality of unit motions. For example, when the robot 100 approaches the owner, it can be expressed as a combination of unit motions such as turning towards the owner, approaching while raising the hand, approaching while swaying the body, and sitting down while raising both hands. That is fine. By combining these four motions, a motion of "approaching the owner, raising the hand halfway, and finally swaying the body and then sitting down" is realized. In the motion file, the rotation angle and angular velocity of the actuators provided in the robot 100 are defined in association with the time axis. By controlling each actuator over time according to the motion file (actuator control information), various motions are expressed.

[0038] The transition time when changing from the previous unit motion to the next unit motion is called an "interval". The interval may be defined according to the time required for unit motion change and the content of the motion. The length of the interval is adjustable. Hereinafter, the settings related to the behavior control of the robot 100, such as when to select which motion and the output adjustment of each actuator in realizing the motion, are collectively referred to as "behavior characteristics". The behavior characteristics of the robot 100 are defined by a motion selection algorithm, a motion selection probability, a motion file, etc.

[0039] The motion storage unit 232 stores, in addition to the motion file, a motion selection table that defines the motions to be executed when various events occur. In the motion selection table, one or more motions and their selection probabilities are associated with an event.

[0040] The personal data storage unit 218 stores user information. Specifically, it stores the intimacy with the user and the master information indicating the user's physical characteristics and behavioral characteristics. Other attribute information such as age and gender may also be stored.

[0041] The robot 100 has an internal parameter called intimacy for each user. When the robot 100 recognizes an action showing goodwill towards itself, such as picking itself up or speaking to it, the intimacy with that user increases. The intimacy with users who have nothing to do with the robot 100, users who act violently, or users with a low encounter frequency decreases.

[0042] The data processing unit 202 includes a position management unit 208, a recognition unit 212, an operation control unit 222, an intimacy management unit 220, and a state management unit 244. The position management unit 208 identifies the position coordinates of the robot 100. The state management unit 244 manages various internal parameters such as various physical states such as the charge rate, internal temperature, and processing load of the processor 122. Further, the state management unit 244 manages various emotion parameters indicating the emotions (such as loneliness, curiosity, and desire for approval) of the robot 100. These emotion parameters are constantly fluctuating. The movement target point of the robot 100 changes according to the emotion parameters. For example, when loneliness is increasing, the robot 100 sets the location where the user is as the movement target point.

[0043] The emotion parameters change over time. Also, the various emotion parameters change due to the coping actions described later. For example, when the owner gives a "hug", the emotion parameter indicating loneliness decreases, and when the owner is not visually recognized for a long time, the emotion parameter indicating loneliness gradually increases.

[0044] The recognition unit 212 recognizes the external environment. The recognition of the external environment includes various recognitions such as the recognition of the weather and seasons based on temperature and humidity, and the recognition of shaded areas (safe zones) based on light quantity and temperature. The recognition unit 156 of the robot 100 acquires various environmental information by the internal sensor 128, performs primary processing on this, and then transfers it to the recognition unit 212 of the server 200.

[0045] Specifically, the recognition unit 156 of the robot 100 extracts an image area corresponding to a moving object, particularly a person or an animal from the image, and extracts a "feature vector" as a set of feature quantities indicating the physical features and behavioral features of the moving object from the extracted image area. The feature vector components (feature quantities) are numerical values obtained by quantifying various physical and behavioral features. For example, the horizontal width of a human eye is quantified in the range of 0 to 1 and forms one feature vector component. The method of extracting the feature vector from the captured image of a person is an application of known face recognition technology. The robot 100 transmits the feature vector to the server 200.

[0046] The recognition unit 212 of the server 200 determines which person the imaged user corresponds to (user identification process) by comparing the feature vector extracted from the captured image by the built-in camera of the robot 100 with the feature vector of the user (cluster) registered in advance in the personal data storage unit 218. In addition, the recognition unit 212 estimates the user's emotion by recognizing the user's facial expression in the image. The recognition unit 212 also performs user identification processing on moving objects other than people, such as cats and dogs that are pets.

[0047] The recognition unit 212 recognizes various response actions made to the robot 100 and classifies them into pleasant / unpleasant actions. The recognition unit 212 also classifies them into positive / negative reactions by recognizing the owner's response actions to the actions of the robot 100. Pleasant / unpleasant actions are discriminated based on whether the user's response action is comfortable or uncomfortable as a living being. For example, being hugged is a pleasant action for the robot 100, and being kicked is an unpleasant action for the robot 100. Positive / negative reactions are discriminated based on whether the user's response action indicates the user's pleasant emotion or unpleasant emotion. Being hugged is a positive reaction indicating the user's pleasant emotion, and being kicked is a negative reaction indicating the user's unpleasant emotion.

[0048] The motion control unit 222 of the server 200 determines the motion of the robot 100 in cooperation with the motion control unit 150 of the robot 100. The motion control unit 222 of the server 200 creates the moving target point of the robot 100 and the moving route therefor. The motion control unit 222 may create a plurality of moving routes and select any one of them.

[0049] The motion control unit 222 selects the motion of the robot 100 from a plurality of motions in the motion storage unit 232. A selection probability is associated with each motion according to the situation. For example, when a pleasant action is performed by the owner, motion A is executed with a probability of 20%, and when the temperature reaches 30 degrees or more, motion B is executed with a probability of 5%. Such a selection method is defined.

[0050] The intimacy management unit 220 manages the intimacy for each user. As described above, the intimacy is registered as part of the personal data in the personal data storage unit 218. When a positive behavior is detected, the intimacy management unit 220 increases the intimacy for its owner. When a negative behavior is detected, the intimacy decreases. Also, the intimacy of an owner who has not been viewed for a long time gradually decreases.

[0051] (Robot 100) Robot 100 includes a communication unit 142, a data processing unit 136, a data storage unit 148, an internal sensor 128, and a drive mechanism 120. The communication unit 142 corresponds to the communication device 126 (see FIG. 3) and is responsible for communication processing with the external sensor 114, the server 200, and other robots 100. The data storage unit 148 stores various data. The data storage unit 148 corresponds to the storage device 124 (see FIG. 3). The data processing unit 136 executes various processes based on the data acquired by the communication unit 142 and the data stored in the data storage unit 148. The data processing unit 136 corresponds to the processor 122 and the computer program executed by the processor 122. The data processing unit 136 also functions as an interface for the communication unit 142, the internal sensor 128, the drive mechanism 120, and the data storage unit 148.

[0052] The data storage unit 148 includes a motion storage unit 160 that defines various motions of the robot 100. Various motion files are downloaded from the motion storage unit 232 of the server 200 to the motion storage unit 160 of the robot 100. Motions are identified by motion IDs. To express various motions such as sitting while accommodating the front wheels 102, lifting the arm 106, rotating the two front wheels 102 in the reverse direction, or rotating only one of the front wheels 102 to rotate the robot 100, shaking by rotating the front wheels 102 in a state where the front wheels 102 are accommodated, and pausing and looking back once when leaving the user, the operation timing, operation time, operation direction, etc. of various actuators (drive mechanism 120) are defined in time series in the motion file. Various data may also be downloaded from the personal data storage unit 218 to the data storage unit 148.

[0053] The data processing unit 136 includes the recognition unit 156 and the motion control unit 150. The motion control unit 150 of the robot 100 determines the motion of the robot 100 in cooperation with the motion control unit 222 of the server 200. For some motions, the server 200 may make the determination, and for other motions, the robot 100 may make the determination. Also, even if the robot 100 determines the motion, the server 200 may make the motion determination when the processing load on the robot 100 is high. The server 200 may determine the base motion, and the robot 100 may determine additional motions. How to distribute the motion determination process between the server 200 and the robot 100 may be designed according to the specifications of the robot system 300.

[0054] The motion control unit 150 of the robot 100 issues an execution instruction for the selected motion to the drive mechanism 120. The drive mechanism 120 controls each actuator according to the motion file.

[0055] When a user with a high level of intimacy is nearby, the motion control unit 150 can execute a motion of raising both arms 106 as a gesture of asking for a hug. When getting tired of hugs, it can also express a motion of resisting a hug by alternately repeating reverse rotation and stopping while keeping the left and right front wheels 102 retracted. The drive mechanism 120 drives the front wheels 102, arms 106, and neck (head frame 316) according to the instructions of the motion control unit 150, causing the robot 100 to express various motions.

[0056] The recognition unit 156 of the robot 100 interprets the external information obtained from the internal sensor 128. The recognition unit 156 is capable of visual recognition (vision unit), smell recognition (olfactory unit), sound recognition (auditory unit), and tactile recognition (tactile unit).

[0057] The recognition unit 156 extracts feature vectors from the captured images of moving objects. As described above, the feature vector is a set of parameters (feature quantities) indicating the physical characteristics and behavioral characteristics of the moving object. When a moving object is detected, physical characteristics and behavioral characteristics are also extracted from the odor sensor, built-in sound-collecting microphone, temperature sensor, etc. These features are also quantified and become feature vector components. The recognition unit 156 identifies the user from the feature vector based on known techniques described in Patent Document 2 and the like.

[0058] Among a series of recognition processes including detection, analysis, and determination, the recognition unit 156 of the robot 100 selects and extracts the information necessary for recognition, and the interpretation process such as determination is performed by the recognition unit 212 of the server 200 The recognition process may be performed only by the recognition unit 212 of the server 200, only by the recognition unit 156 of the robot 100, or the above-mentioned recognition process may be performed while both parties share the roles as described above.

[0059] When a strong impact is given to the robot 100, the recognition unit 156 recognizes this by the touch sensor and the acceleration sensor, and the recognition unit 212 of the server 200 recognizes that a "violent act" has been committed by a nearby user. When a user grabs the horn 112 and lifts the robot 100, this may also be recognized as a violent act. When a user facing the robot 100 speaks in a specific volume range and a specific frequency band, the recognition unit 212 of the server 200 may recognize that a "calling act" has been performed on the robot 100. In addition, when a temperature of about body temperature is detected, it recognizes that a "contact act" has been performed by the user, and when an upward acceleration is detected in a state where contact has been recognized, it recognizes that a "hug" has been performed. Physical contact when the user lifts the body 104 may be sensed, or a hug may be recognized by a decrease in the load on the front wheels 102. In summary, the robot 100 acquires the user's actions as physical information by the internal sensor 128, and the recognition unit 212 of the server 200 judges whether the user feels comfortable or uncomfortable. The recognition unit 212 of the server 200 also performs a user identification process based on the feature vector.

[0060] The recognition unit 212 of the server 200 recognizes various responses of the user to the robot 100. Some typical responses among the various responses are associated with pleasant or unpleasant, positive or negative. Generally, most pleasant responses are positive reactions, and most unpleasant responses are negative reactions. Pleasant and unpleasant responses are related to the degree of intimacy, and positive and negative reactions affect the robot 100's selection of actions.

[0061] The intimacy management unit 220 of the server 200 changes the intimacy with the user in response to the interaction behavior recognized by the recognition unit 156. In principle, the intimacy with a user who has performed a pleasant behavior increases, and the intimacy with a user who has performed an unpleasant behavior decreases.

[0062] Each function of the server 200 is realized by loading a program for realizing the function into the memory and instantiating it. The processing capabilities of the server 200 complement various processes performed by the robot 100. The server 200 can be used as a resource of the robot 100. How to utilize the resources of the server 200 is dynamically determined according to requests from the robot 100. For example, in the robot 100, when it is necessary to continuously generate complex motions according to detection values from a large number of touch sensors, the processing of the processor 122 in the robot 100 may be preferentially allocated to the selection and generation of motions, and the processing for recognizing the surrounding situation may be performed by the recognition unit 212 of the server 200. In this way, various processes of the robot system 300 can be decentralized between the robot 100 and the server 200.

[0063] A single server 200 can also control multiple robots 100. In this case, each function of the server 200 is instantiated independently for each robot 100. For example, the server 200 may prepare a recognition unit 212 for the robot 100B separately from the recognition unit 212 (instance / object) for the robot 100A.

[0064] Based on the above basic configuration, next, regarding the implementation of the robot 100 in this embodiment, in particular, the description will focus on the differences between the features and objectives of this implementation and the basic configuration.

[0065] [SLAM] The robot 100 in this embodiment periodically images the surroundings by means of the omnidirectional camera 113 to obtain a large number of captured images (still images). The robot 100 forms a memory (hereinafter referred to as "image memory") based on the captured images.

[0066] Image memory is a collection of a plurality of key frames. A key frame is distribution information of feature points (feature quantities) in a captured image. The robot 100 of this embodiment forms key frames by a graph-based SLAM (Simultaneous Localization and Mapping) technique using image feature quantities, more specifically, a SLAM technique based on ORB (Oriented FAST and Rotated BRIEF) feature quantities (see Patent Document 3).

[0067] The robot 100 forms an image memory as a collection of key frames, in other words, as an image feature distribution, by periodically forming key frames while moving. The robot 100 estimates the current location by comparing the key frame acquired at the current location with a large number of key frames it already possesses. That is, the robot 100 compares the captured image it is actually viewing with the captured images (memories) it has viewed in the past, and performs "spatial recognition" by matching its current situation with past memories. The image memory formed as a collection of feature points becomes a so-called map. The robot 100 updates the map while moving while estimating the current location.

[0068] It is assumed that the basic configuration of the robot 100 recognizes its position by an external sensor 114 instead of by key frames. The robot 100 of this embodiment will be described as recognizing locations based only on key frames.

[0069] The robot 100 in this embodiment includes a "mode setting unit" for setting various modes.

[0070] <Baby watching> Figures 5 to 7 are schematic diagrams for explaining the action scenes when a plurality of robots 100 watch a baby. First, in the baby's room, the baby (hereinafter referred to as the "subject") who is the object of watching is sleeping in a bouncer (Figure 5A). There are two robots 100 in this house. The two robots 100 recognize the presence and position of a baby (infant) via image recognition respectively. The two robots 100 may share the position of the baby via mutual communication. The two robots 100 determine the viewing points of their respective robots from the position of the baby. The two robots 100 correct the viewing points via mutual communication so that the viewing points of each other are the same or within a predetermined range. By controlling the movement and parts of the two robots 100 so that they turn their heads towards the determined respective viewing points, an action in which the two robots 100 peek at the baby is realized (Fig. 5B). One of the two robots 100, robot 100A, continues to watch over the baby by turning the head of robot 100A in the direction of the baby's presence according to the position of the baby. Robot 100A continuously keeps the distance from the baby constant so as not to leave the baby's side. The other robot 100B executes an action such that after a while, it stops watching and starts playing by sharing the role via communication with robot 100A (Fig. 5C). The mother has entrusted the robots 100 to watch over the baby and is cooking in the kitchen (Fig. 5D).

[0071] The mother places a smartphone (communication terminal) in the kitchen (Fig. 6A). Robot 100A (while watching) captures images of the baby with the omnidirectional camera 113 and continuously sends the captured images to the smartphone as live video. On the smartphone, the area in which the baby appears in the omnidirectional image is displayed. At this time, the distorted omnidirectional image may be corrected and displayed on the smartphone so as to be flat The corrected image may be displayed on the smartphone.

[0072] Here, suddenly, the baby starts crying (Fig. 6B). When the playing 100B hears the crying sound of the baby through the microphone (collects the voice), it approaches the baby based on the position of the baby (Fig. 6C). The robot 100B executes an interference motion. The robot 100B executes an interference motion when at least one of the conditions that a voice such as a crying sound is collected and the condition that it is determined by image analysis that the baby is in a specific state such as crying is satisfied. The interference motion is a motion for soothing, such as outputting a predetermined voice to attract the baby's attention, outputting a lullaby-like voice, waving the arm 106, swaying the head or body, and swaying the bouncer. The interference motion distracts the baby. Also, by executing the interference motion, an impression that the robot 100B is struggling to do something about the crying baby can be given to a third party. Instead of or in addition to the robot B, the robot A may execute the interference motion. The live video relayed from the robot 100A projects the crying figure of the baby onto the mother's smartphone (Fig. 6D). The video before and after the baby starts crying (the video from a predetermined time before to a predetermined time after the time when the baby starts crying) may be stored in an HDD or the like for persistence.

[0073] The mother is startled and instinctively grabs the smartphone (Fig. 7A). The mother hurriedly goes to the child's room and rushes to the baby (Fig. 7B). The mother picks up the baby (Fig. 7C). Robots 100 are beside the mother and the baby, staring at the mothers. The two robots recognize the respective positions of the mother and the baby through image analysis, and set the fixation points from the respective positions. The two robots 100 share the fixation points via communication and correct them so that their respective fixation points are the same or within a predetermined range as needed. The two robots 100 perform the operation of turning their heads toward their respective fixation points. As a result, by having the two robots 100 line up and stare at the mother and the baby, it is possible to give the mother the impression that the robots 100 are worried about the baby. The baby held by the mother falls asleep again (Fig. 7D). When the robots 100 confirm that the baby has fallen asleep, they leave the side of the bouncer (Fig. 7E).

[0074] When the robot 100 is manually set via an input switch, a smartphone, etc. from the mother to the "watching mode", the robot 100 may search for the baby in order to start the above-described watching behavior. When the robot 100 detects that the baby is in the bouncer, the mode setting unit may automatically set it to the watching mode. When the robot 100 detects an infant through image recognition and there is no caregiver near the infant, the robot 100 may automatically shift to the watching mode. The case where there is no caregiver near the infant may be, for example, when an image of a person of a predetermined age or older is not detected, or when an image of a person associated with the infant is not detected. Also, when multiple robots 100 are in the same room, all the robots 100 may be set to the watching mode, or only some of the robots 100 may be set to the watching mode.

[0075] The robot 100 may shift to the watching mode on the condition that "the baby" is detected in the captured image and no other user or specific user such as the mother or father is detected.

[0076] The monitoring robot 100 may limit its movement range within a range where it can visually recognize the baby being monitored. Alternatively, the robot 100 may not leave the room where the baby is during the monitoring. At least, it is desirable that the robot 100 does not take its eyes off the baby during the monitoring (for example, always captures the baby within the camera's field of view). The robot 100 monitors the state of the target person using sensors such as an omnidirectional camera 113, a microphone, and a temperature sensor to measure the external environment. When the baby moves around, the robot 100 may use sensors such as an omnidirectional camera 113, a microphone, and a temperature sensor to measure the external environment, recognize the position of the baby, and control its own direction to face the direction of the baby. Also, when the baby enters within a predetermined distance of itself, the robot 100 may reduce the contact points with the baby by, for example, accommodating the wheels. When a predetermined warning condition is satisfied, the robot 100 executes a motion for actively interacting with the baby. Alternatively, it may transmit "warning information" to another communication terminal such as the mother's smartphone. The warning condition can be arbitrarily set. For example, when the baby (the person being monitored) cries, approaches the stairs, tries to go outside, plays with small objects, falls, etc., situations where danger threatens the baby can be considered. The motion for actively interacting may be the interference motion described above.

[0077] When the robot 100A shifts to the monitoring mode, it may notify the robot 100B that it has shifted to the monitoring mode. Upon receiving the notification, the robot 100B may also shift to the monitoring mode or approach the position of the robot 100A or the baby. By having multiple robots 100 gather around the baby, it becomes possible to exhibit behaviors as if the robots 100 are concerned about the baby.

[0078]

[0079] ​At this time, in order not to wake up the baby with the operating sound of the robot 100, the robot 100 suppresses the amount of operation of the drive mechanism 120 (especially the mechanism related to movement and posture adjustment) more than in the normal mode, and operates quietly without making more operating sounds than usual. When shifting to the watching mode when the baby is awake, the robot B may execute specific motions that attract the baby's interest on the baby's side, such as a motion of running around the bouncer or dancing. For example, it can be determined that the baby is awake when conditions such as detecting the baby's voice by voice analysis or detecting an image in which the baby has opened its eyes by image analysis are satisfied. Such motions are not executed when the baby is sleeping. The robot B executes appropriate motions according to the state of the person being watched. The baby may also feel at ease if a number of robots 100 gather around it.

[0080] The robot 100 may shift to the watching mode when receiving a specific voice command such as "Please watch the baby" from a specific user such as the mother via a microphone or the like. At the start of watching, the robot 100 may execute confirmation actions such as walking around the baby, facing the baby, or turning the hand 358 towards the baby based on the position of the baby detected by image analysis or the like. The mother (instructor) can judge whether the robot 100 has misidentified the person being watched through the confirmation action. It is considered that the mother will utter an affirmative reply such as "Leave it to you" if it is correct, or a negative reply such as "No, it's wrong" if it is incorrect. Therefore, the robot 100 may judge whether the target of watching is correct based on the mother's reply collected by the microphone after the confirmation action.

[0081] While keeping watch, the robot 100 maintains its line of sight on the baby, such as by peering into the baby. The baby feels reassured believing that it is being watched by the robot 100. The mother is inspired with feelings of affection and trust towards the robot 100 upon seeing the robot 100 diligently continuing to watch over the baby. The watching behavior also serves as an appeal to the user (witness) that "the robot 100 is working hard".

[0082] As described above, while the robot 100A is keeping watch, the robot 100B may play freely. However, the movement range of the robot 100B is restricted to a range where the baby or the robot 100A can be visually recognized. The robot 100A may send a live image (close-up image) of the baby's face to the smartphone, and the robot 100B may send a live image (wide-angle image) of the robot 100A watching over the baby to the smartphone. It is desirable for the robot 100B to further restrict its movement range so as not to interfere with the robot 100A's photographing of the baby. For example, the robot 100B may identify the positions of the baby and the robot 100A and move so as not to overlap on the straight line connecting the baby and the robot 100A. Also, when the robot 100A recognizes the image being photographed and the baby's figure becomes unrecognizable by the robot 100B, the robot 100A may notify the robot 100B of an instruction to request movement. In response to that instruction, the robot 100B moves.

[0083] When it is detected by image analysis or voice analysis that the baby has reached a predetermined state, for example, when the baby starts crying, in addition to the live video, the robot 100A notifies that the state of the baby has changed. The robot 100B may execute an interference motion. The robot 100A notifies the smartphone of a message that simply indicates a state associated with a warning condition such as "the baby is crying" or "the baby has started to be fussy". As an interference motion, the robot 100B may dance or play music such as a lullaby through a built-in audio player. In any case, the robot 100B soothes the baby by performing an action to distract the baby. When the mother returns (see Fig. 7B), the robot 100B stops the interference motion. Specifically, the robot 100B stops the interference motion when it can confirm "the mother" in the captured image. When it is recognized via a microphone or the like that the mother has pronounced a keyword (hereinafter referred to as "completion command") instructing the completion of a watching action such as "thank you" or "it's okay now", the robot 100B may stop the interference motion. At this time, it may be determined whether or not the completion command has been uttered by the person who instructed the watching action to the robot 100A, and if it is an instruction from the person, the watching action may be completed.

[0084] The robot 100 recognizes the baby from the captured image. The detection method of the "baby" may be realized by applying known face recognition technology. In addition, when the baby is moving, the robot 100 may adjust the moving direction so that the baby can always be visually recognized regardless of whether it is in the watching state.

[0085] The robots 100A and 100B may take turns watching. For example, when the robot 100A watches for 10 minutes, the robot 100B shifts to the watching mode, and the robot 100A may move freely. It is considered that having multiple robots 100 take turns watching will not bore the baby. In addition, it is easier for a third party to feel as if the robots 100A and 100B are watching in cooperation.

[0086] If the robot 100 can watch over the baby, even a mother who is busy with child-rearing can more easily concentrate on housework with peace of mind. When doing housework such as laundry in a place away from the baby, there are times when she may not notice immediately even if the baby is crying. There are times when the baby may start crying seriously before she notices that the baby is crying. Such a situation imposes a great burden on the mother. By having the robot 100 watch over the baby, it can quickly detect signs of a major cry such as the baby's "whimpering".

[0087] Not only the mother but also the robot 100 participates in child-rearing. Also, it is expected that a baby who grows up being watched over by the robot 100 will have a sense of familiarity with the robot 100 in the future. is expected.

[0088] <Baby-sitting> Figures 8 to 11 are schematic diagrams for explaining the action scenes when the robot 100 is baby-sitting. Assume a household where two robots 100A and 100B and a female owner live together. The female owner goes out to work (Fig. 8A). At this time, the female owner calls out to the nearby robot 100A, saying "Please baby-sit for me". Upon hearing this statement (collecting the voice via the microphone), the robot 100A recognizes that the female owner is going out and shifts to the "baby-sitting mode" (Fig. 8B). The female owner leaves from the entrance (Fig. 8C). The robot 100A moves to the entrance and sees off the female owner by executing a predetermined motion.

[0089] On the other hand, the robot 100B in the room chases after the robot 100A and starts playing (Fig. 8D). Note that when the female owner goes out, both the robot 100A and the robot 100B may see her off, or only the robot 100 with a predetermined value or more of intimacy with the female owner may see her off.

[0090] After a while, the front door opens while you are away (Figure 9A). When it is determined by voice analysis, image analysis, or the like that the front door is open, the two robots 100 move to the front door in "hope" that the female owner has returned home (FIG. 9B). However, the person who appeared (identified through image analysis, etc.) was not the female owner, but a stranger (suspicious person) carrying a large bag (Figure 9C). At this time, the robot 100 approaches the suspicious person sufficiently and takes a picture of the suspicious person (FIG. 9D). The robot 100 transitions to the alert mode. When the robot 100 transitions to the alert mode, the robots 100 may move close to the suspicious person and continue taking a picture of the suspicious person.

[0091] The female owner is working in the office. The robot 100 transmits a captured image of the suspicious person to the female owner's smartphone (FIG. 10A). The female owner, who is at work, finds out through her smartphone that the robot 100 has found a suspicious person (FIG. 10B). In this case, it is assumed that the person thought to be suspicious is the female owner's mother. The robots 100 do not know the female owner's mother. The woman notifies the robots 100 through her smartphone that "there is no suspicious person." The female owner may inform the robot 100 by voice that "there is no need to worry, it's your mother." The robot 100 then cancels the alert mode. The mother cooks home-cooked meals (Figure 10C).

[0092] The female owner (the daughter) returns home and chats with her mother while eating a home-cooked meal (Figure 11A). Robots 100 are playing nearby them (Figures 11B and 11C).

[0093] When the robot 100 detects a suspicious person (someone it has never seen before or someone with an intimacy level below a threshold) during the owner's absence through image analysis or voice recognition, it takes a picture of the suspicious person and sends the captured image of the suspicious person to the smartphone of the female owner (specific user). According to such a control method, the female owner can safely entrust the security of her home to the robot 100. In addition, through image analysis and voice recognition, the robot 100 can also detect various events that occur during the owner's absence, such as an earthquake occurred and things were damaged, a gas leak occurred, a visitor such as a delivery person (a call from the intercom), and notify the owner's smartphone of the content informing about the event. Also, the robot 100 records the events during the owner's absence as a life log, and the owner may check the events during the absence by viewing the life log through the smartphone after returning home.

[0094] The mode setting unit of the robot 100 may also set the standby mode based on an operation input from the user. When the robot 100 detects a specific event that the user goes out from the entrance, it may automatically change the setting to the standby mode. Alternatively, when the robot 100 cannot visually recognize the user indoors for a certain period of time or more, it may automatically change the setting to the standby mode.

[0095] In the warning mode, the robot 100 may set the action range to a position where it can photograph a suspicious person. This is to avoid missing the actions of the suspicious person. Also, to prevent violence from the suspicious person, it may move away from the suspicious person. After reporting the suspicious person, when receiving a notice from the user that "it is not a suspicious person", the robot 100 cancels the warning mode and returns to the home guard mode. After that, the robot 100 may interact with the unconfirmed person (former suspicious person) in a normal way. Also, it may remember the appearance of the unconfirmed person and manage parameters such as intimacy. It may become familiar with the unconfirmed person. In the case of the example shown in FIGS. 8 to 11, the mother is initially warned by the robot 100, but after receiving an approval notice for the robot 100 from the female owner (daughter), the robot 100 begins to cling to the mother. The mother can feel that she has been suddenly accepted and welcomed by the robot 100.

[0096] In the home guard mode, a user (owner) who is in a remote location may send an indoor confirmation instruction to the robot 100 via a smartphone. When the robot 100 receives the indoor confirmation instruction, it patrols the interior according to the map generated based on SLAM. At this time, the robot 100 may send the captured image to the user's smartphone or notify the presence or absence of an abnormal event. By sending the indoor confirmation instruction, the user can check the state of their home at any time.

[0097] <Remote operation> FIGS. 12 to 14 are schematic diagrams for explaining the action scenes when a user who is out remotely operates the robot 100. At home, two robots 100A and 100B are on home guard. There is also a cat in this house (FIG. 12A). On the other hand, the rest of the family leaves the robots 100 and the cat at home and goes out into the city (FIG. 12B). The boy has a glum face (FIG. 12C). The boy takes out his smartphone (mobile terminal). The boy starts operating the smartphone in some way (FIG. 12D).

[0098] Two robots 100 are playing in the house (Fig. 13A). When robot 100A starts to move, robot 100B follows robot 100A (Fig. 13B). Robots 100A and 100B are playing tag. The mother is concerned about the boy (son) (Fig. 13C). When the boy goes out, he is concerned that the cat was not very energetic (Fig. 13D).

[0099] The boy sends a command "Check on the cat" from the smartphone to robot 100. Robots 100A and 100B move to photograph the location or object indicated in the command and take appropriate photographs. The captured images of robots 100A and 100B are sent to the smartphone. The cat is playing lively on the cat tree (Fig. 14A). Seeing the energetic appearance of the cat, the family is relieved (Fig. 14B). Robot 100 photographs the cat playing on the cat tower. Robot 100 photographs the cat from above, and the cat looks at robot 100 (Figs. 14C and 14D).

[0100] In this way, the user can send various instructions to robot 100 from the smartphone. In particular, the user can command robot 100 to perform an indoor check. When a command such as "Check on the cat" is sent as in the above embodiment, robot 100 detects an object corresponding to "the cat" from the captured image and sends the captured image centered on the cat to the smartphone. Such a command may be a voice command or may be input from the graphical user interface provided on the smartphone. The user may be able to operate robot 100 like a radio-controlled car (hereinafter, such an operation method is referred to as "remote operation").

[0101] The captured image by the robot 100 is displayed on the smartphone, and the user may enlarge and display a particularly desired part of the captured image that is live-streamed to the smartphone. The robot 100 may transmit the omnidirectional image itself to the smartphone, and the user may check what the robot 100 "sees" from the omnidirectional image.

[0102] The robot 100 can recognize not only species such as humans and cats but also at the individual level of "who" and "which". Cats, whether black or white, big or small, are treated as different cats. Also, based on the user's call to the cat, the robot 100 learns the name of the cat. For example, a model that extracts a cat's image from the name of the cat recognized by voice analysis and the captured image when the name of the cat is recognized, and outputs the name of the cat with the image of the cat as input may be generated by machine learning. By using such a model, even when there are multiple cats, if the user designates and gives an order by the name of the cat, the robot 100 can select the designated cat as the shooting target. The user may register the name of the cat and the photo of the cat in advance via a smartphone or the like.

[0103] When the user remotely operates the robot 100A, the robot 100B may move with respect to the robot 100A. The captured image from the robot 100A and the captured image from the robot 100B are transmitted to the user's smartphone. When the robot 100A captures a cat, the robot 100B near the robot 100A also captures the cat with the omnidirectional camera 113. The user can obtain the captured image of the cat not only from the robot 100A but also from the robot 100B by only remotely operating the robot 100A. The user can indirectly remotely operate the robot 100B by only remotely operating the robot 100A. This is because the robot 100B is given a "tracking function".

[0104] The user can set the robot 100 from the smartphone to the remote control mode. Also, the user can end the remote control mode of the robot 100 from the smartphone. The robot 100 in the remote control mode may change the display of the eyes 110. For example, the robot 100 may change the eyes 110 to red eyes, or visually represent "being dominated (remotely controlled)" by displaying an icon on the eyes 110. When the remote control mode ends, the robot 100 returns the eyes 110 to the normal black eye display and returns to the location at the start of the remote control mode. When the remote control mode ends, the robot 100 may sit down, or act to show "breaking free from control and regaining self", such as shaking its head violently.

[0105] The robot 100 in the remote mode does not change the emotion parameters or intimacy.

[0106] When switching to the remote control mode, the robot 100 authenticates the person who requested the remote control. Only when the authentication is successful does the robot 100 switch to the remote control mode. The authentication may be a common authentication method using an account name and password, or an electronic certificate may be registered in advance in the device used for remote control, and only access from a device with the electronic certificate is permitted. Furthermore, by using a camera or microphone provided in a mobile terminal such as a smartphone, it may be authenticated by confirming that the person operating the mobile terminal is the owner of the robot 100. Also, when requested to switch to the remote mode, the surrounding users may be asked to approve the switch to the remote mode. In this way, by thoroughly confirming that the person who requested the remote mode is the owner of the robot 100, unauthorized remote control by a third party can be prevented.

[0107] <Monitoring of the Elderly> Figures 15 to 18 are schematic diagrams for explaining the action scenes when a plurality of robots 100 monitor the elderly. ​The elderly father lives alone. The only daughter lives away from her father. There are two robots 100 in the father's house (Figure 15A). In the home living room, the daughter is looking at her smartphone (Figure 15B). The robots 100 record the life with the father as a life log. The life log is a diary that shows what is happening around the father while taking into account the father's privacy. The daughter checks the life log on her smartphone (Figure 15C). The life log contains simple information such as what time the father got up and whether he had breakfast. On the other hand, the father is hugging the robot 100 and showing affection (Figure 15D). If the permission of the father is recognized through image analysis, voice analysis, communication, etc., the robot 100 may transmit an imaging image of the father playing with the robot 100 to the daughter's smartphone. For example, when the robot 100A is being hugged by the father, the robot 100B may act as a cameraman and image the robot 100A and the father, and transmit the imaging image to the daughter's smartphone.

[0108] The daughter feels at ease seeing the father enjoying life with the robot 100 in the living room (Figure 16A).

[0109] Next, assume a scene where the daughter is working in the office. The daughter suddenly takes out her smartphone and checks the life log of her father (Figure 16B). Suppose there is almost no record of the father in this life log. The daughter suddenly becomes worried about her father (Figure 16C). The daughter calls her father from the corridor of the office (Figure 16D).

[0110] Immediately, the father's cheerful voice is heard on the phone (Figure 17A). The father is at a hotel. It seems that the phone call came just as he was getting out of the bath (Figure 17B). The father tells the daughter that he has come to the hot spring with friends (Figure 17C). Since the daughter didn't know that her father had gone to the hot spring, she feels relieved after learning the situation (Figure 17D).

[0111] The conversation between the father and daughter continues (Figs. 18A and 18B). At the father's home, two robots 100 are on guard (Fig. 18C).

[0112] The robot 100 records various events occurring in the life with the father (the elderly person to be watched over) as a life log. This life log records routine actions in the father's daily life, such as the time of waking up and whether he did his usual gymnastics today. The daughter can check through the life log provided by the robot 100 whether the father is living his normal life.

[0113] When an event indicating the interaction between the father and the robot 100 does not occur (is not detected) for a predetermined time or more, the robot 100 may send an abnormality notification to the daughter's smartphone. For example, when the robot 100 has not been touched by the father for a while, or when the father is lying down even at noon, an abnormality notification may be sent. Whether the father is lying down can be determined by image analysis, temperature sensor analysis, etc. The robot 100 may act to increase the father's chance of visual recognition by actively moving around indoors.

[0114] With the life log in which information is abstracted, the daughter can check the father's daily life while protecting the father's privacy. When the elderly person is the subject to be watched over, the robot 100 does not always need to visually recognize the elderly person. The elderly person lives an independent life, and the robot 100 may basically act autonomously. It is considered preferable for the elderly person and the robot 100 to maintain an appropriate sense of distance. Not limited to the watching-over time, the robot 100 may always record the life log when the user desires. The robot 100 may just send an abnormality notification to the daughter only when an abnormality occurs in the life of the elderly person.

[0115] <Expression of jealousy> A person cannot be indifferent to the love directed towards themselves. When multiple people direct their love towards the same person, jealousy is likely to sprout. Therefore, when robot 100A and robot 100B live with a user, one of the robots 100 may take actions that make the user feel the jealousy of the other robot 100.

[0116] For example, when robot 100B is being hugged by the user, the state management unit 244 increases the approval desire value (the desire to be approved), which is a type of emotional parameter of robot 100A. When the approval desire value increases, the motion control unit 150 of robot 100A demands a hug from the user. Robot 100A may stare fixedly at the user, approach the user, or seek a hug by wandering around the user. When the user walks, robot 100A may follow the user and move. The increase in the approval desire value is externally expressed as the behavioral characteristics of robot 100 as if jealousy has been provoked.

[0117] When the user continues to hug robot 100B, robot 100A may actively express "strong jealousy" by clinging to the user. Alternatively, robot 100A may passively express jealousy by moving to a position away from the user and directing its gaze from afar. These ways of expressing jealousy are determined according to the personality of each robot 100 (the initially set personality or the cultivated personality). Jealousy may be expressed by "being sulky", and for a certain period after an event of jealousy occurs, the robot 100 may exhibit behavioral expressions such as approaching or leaving the user. Robot 100 may express the appearance of "being sulky" by temporarily rejecting a hug.

[0118] Robot 100 may take actions that make it feel jealous when the intimacy level is high. For example, assume that Robot 100A has a high level of intimacy with User P1 and a relatively low level of intimacy with User P2. At this time, when User P1 hugs Robot 100B, the approval desire value may be increased more than when User P2 hugs Robot 100B. According to such a control method, it is possible to exhibit behavioral expressions as if having the possessiveness of wanting to monopolize the love of a user who one particularly likes.

[0119] Robot 100 may notify other Robots 100 of its own state (emotional parameters, intimacy level, events, etc.) (hereinafter, such a notification is referred to as a "status notification"). Based on the status notification, Robots 100 may be able to understand each other's states. For example, by Robot 100A sending a status notification to Robot 100B about states such as "being hugged by the user", "being stroked by the user", "being dressed by the user", etc., Robot 100B can understand the state of Robot 100A. While the approval request value (desire to be approved) decreases due to an event such as Robot 100A being hugged, when the approval request value of Robot 100B is in a high state above the threshold, Robot 100B exhibits specific behavioral characteristics of expressing jealousy.

[0120] In this embodiment, the state management unit 244 of the server 200 collectively manages the emotional parameters of each Robot 100. In this case, the state management unit 244 may internally notify Robot 100B of the value of the emotional parameter of Robot 100A, or may change the emotional parameter of Robot 100B based on the emotional parameter of Robot 100A. In this case, it may be assumed that the emotional parameter of Robot 100B is changed on the condition that Robot 100A is in a visible position from Robot 100B. This is to express the situation where Robot 100B near Robot 100A visually senses the change in the emotion of Robot 100A and changes its own emotional parameter.

[0121] Not limited to the emotional parameters, the robot 100A may notify the robot 100B by short-range wireless communication such as infrared rays. In this case, since the robot 100B can receive the status notification of the robot 100A only when it is near the robot 100A and there is no obstacle blocking the line of sight, it can express the situation of "being able to sense the status only when being close enough to be visible". Note that the robot 100B may detect events such as the robot 100A being held or petted based on the captured image. When the robot 100B recognizes a kind act towards the robot 100A, it may change the emotional parameters.

[0122] The robot 100 may not only be jealous of another robot 100, but also be jealous of a pet or a child. For example, when the user holds a cat, the approval desire value of the robot 100 may be increased. The user may also need to take care such as loving the pet where the robot 100 cannot see, or loving the pet and the robot 100 equally, in order not to make the robot 100 jealous. By actively creating opportunities for the user to consider the feelings of the robot 100, the user's attachment to the robot 100 can be deepened.

[0123] Even without engaging in conversation, the robot 100 in this embodiment can express its feelings through its actions. The robot 100A may be able to receive the feelings (emotional parameters) of the robot 100B. The server 200 may reflect changes in the emotional parameters of the robot 100B in the actions of the robot 100A. For example, when the approval desire value of the robot 100B drops sharply (when something good is thought to have happened to the robot 100B), the robot 100A may move closer to the robot 100B. When the robot 100A is notified that the approval desire of the robot 100B is satisfied, it may increase its own approval desire value (the feeling of wanting to be recognized). According to such a control method, even when the user secretly pets the robot 100B, it is possible to realize behavioral expressions as if the robot 100A sensed something. So to speak, it is possible to realize a mysterious behavioral expression as if telepathy is passing between the robots 100.

[0124] <Multiple robots staring at the same thing> The robot 100A and the robot 100B may continue to stare at the same object. For example, when the robot 100A is staring at a user who is relaxing, the robot 100B may also stare at the same user. The robot 100B may detect that the robot 100A is staring at a user who is relaxing (the direction of the head of the robot 100A is the direction of the user's presence) through communication with the robot 100A or through image analysis. The robot B may move close to the robot A and then stare at the user. Since the user feels multiple lines of sight, the user can feel that the robots 100 have a strong interest in him / her. When the user does not pay attention to the robots 100 for a long time, the robot 100A and the robot 100B may wordlessly seek "connection" by staring at the user at the same time.

[0125] Suppose that robot 100A has a high intimacy level with user P1 that is equal to or higher than a predetermined value, and robot 100B also has a high intimacy level with user P1 that is equal to or higher than the predetermined value. Robots tend to look at users with a higher intimacy level more often. Therefore, in the above situation, there will be a chance that robots 100A and 100B will both look at user P1 without intention. Robot 100A may notify robot 100B of the state that it is looking at user P1. When robot 100B receives a state notification from robot 100A saying "(Robot 100A is also) looking at user P1" when it is looking at user P1 itself, it may create an "accidental coincidence" by performing specific motions such as a surprised motion or turning its line of sight in the direction of robot 100A. Also, when they are both looking at the same user, robots 100A and 100B may move closer to each other and perform motions so that they line up and look at user P1 together. By moving both robots 100 to a position where the user's face can be seen as large as possible and looking at the user side by side, a strong pressure can be given to the user.

[0126] Also, when an insect enters the house (when an insect is detected in the house by image analysis or voice analysis, etc.), robots 100A and 100B may share the target insect and show abnormal interest in the insect by both looking at the insect at the same time. Furthermore, by looking at each other, robots 100A and 100B can realize a behavior expression as if they are indicating something to each other among the robots 100.

[0127] When the value of the emotional parameter indicating the curiosity of the robot 100A exceeds the threshold, the robot 100A may notify the robot 100B of the state of "increasing curiosity". At this time, the robot 100B may approach the robot 100A and execute motions such as moving its hand to touch the robot 100A as if it wants to know the source of the curiosity of the robot 100A. The robot 100A may notify the robot 100B of the object of interest and its direction among the omnidirectional images. When receiving this notification, the robot 100B may look at the same object by turning its head and line of sight towards the same object as the object of the robot 100A.

[0128] <Appeal> When a predetermined appeal condition is satisfied, for example, when the approval desire value exceeds the threshold, the robot 100 executes a strong appeal action to the user. The appeal action mentioned here is an action that actively seeks interaction from the user to the robot 100, such as touch, greeting, or hugging. For example, assume that the user is doing exercises such as yoga indoors. When the user is engrossed in yoga and the appeal condition of the robot 100 is satisfied, the robot 100 may execute appeal actions such as continuously staring at the user and wandering around the user, and may seek to interrupt the yoga.

[0129] <Following behavior> As described above, when the robot 100A moves, the robot 100B may follow and move behind the robot 100A while keeping the distance from the robot 100A constant. The robot 100A may also follow and move in the same way with respect to the user or the pet. For example, when a dog is following behind the user, the robot 100A may follow the dog or the user. When the robot 100A is following a dog or the like, the robot 100B may follow the robot 100A. The distance between the robot and the following object (for example, the dog following the user) may be equal to the distance between the following object and the object being followed by the following object (for example, the user), may be shorter than the said distance by a predetermined length, or may be longer than the said distance by a predetermined length.

[0130] When the robot 100 detects that the moving objects Q1 and Q2 are moving in the same direction for a predetermined period of time or more, it determines that "following" is occurring. According to such a control method, when following is occurring, it becomes possible to perform behavioral expressions that make the user feel the instinct of the robot 100 to want to follow along. The posture of multiple robots 100 performing following behavior is considered to be effective in appealing to the user's affection for the robot 100.

[0131] The robot 100 may execute a following behavior on the condition that the value of an emotion parameter of the robot 100, for example, the emotion parameter indicating curiosity, is equal to or less than a threshold value. According to such a control method, it becomes possible to perform behavioral expressions such that when curiosity fades and boredom sets in, a following behavior towards other robots is executed, and when curiosity is increasing, the following behavior is not executed. When the following behavior is executed, the following behavior may be terminated on the condition that curiosity has increased above the threshold value due to various events. The source of the behavior of the autonomous behavior type robot is a predetermined parameter indicating the internal state. In the present embodiment, the parameter indicating curiosity contributes greatly to the source of the behavior, but if there is little change in the external environment, the curiosity parameter may approach 0. In such a case, instead of waiting for a change in its own parameters, it is possible to actively change its own parameters by hitching a ride on the behavior of other robots.

[0132] As described above, the plurality of robots 100 may perform the same actions like following actions, or may change their action characteristics while being affected by each other's actions. In order to enhance the cooperation and interlocking of the plurality of robots 100, it is desirable that the robots can grasp each other's states within the server 200 or between the robots 100. The robot 100B that has grasped the state of the robot 100A may act in synchronization with the state of the robot 100A, or may act independently without synchronization. An example of the action of the robot 100B synchronizing with the robot 100A is that the robot B executes the same category of motion for the same object as the object that the robot 100A is looking at, such as the robot 100B looking at the same thing that the robot A is looking at.

[0133] When the plurality of robots 100 perform cooperative actions, it is considered that the user will feel the robots 100 are cute. The user may want to take a photo of the robots 100 while they are performing cooperative actions. When the image recognition determines that the user is holding the camera with the omnidirectional camera 113, the robot 100 may maintain at least one of its own actions and states until the user finishes shooting. Also, in this case, instead of maintaining the action or state, the robot 100 may select a specific motion. For example, the robot 100 may turn its body in the direction of the user, or may cooperate with the user's shooting by temporarily stopping the cooperative action. In this way, the robot 100 may temporarily stop its operation when it detects the shooting action. Also, the robot 100 may pose or temporarily stop its action not only during cooperative actions but also when it detects the user's shooting action. According to such a control method, it becomes easier for the user to upload a photographed image of the cute appearance of the robot 100 to an SNS (Social Networking Service) or the like. Also, it is considered that it becomes easier to take various best-shot images of the robot 100 related to various things (pets, children, toys, furniture, etc.) in the home.

[0134] <Structure of the outer skin> The outer skin 314 of the robot 100 is configured by housing a stretchable base material in a cloth bag. The bag may be made of a soft material with a warm touch for the user. The base material is preferably a flame-retardant material, and more preferably a material that releases self-extinguishing gas when it gets hot. For example, the base material is composed of a flame-retardant sponge. Since the outer skin 314 is formed by wrapping a flame-retardant base material with a cloth bag, even if the cloth bag catches fire, self-extinguishing gas is released from the base material, so the spread of fire in the cloth bag can be prevented. When the base material generates self-extinguishing gas, the threshold temperature is preferably lower than the ignition temperature of the cloth. In this case, when the cloth gets hot, self-extinguishing gas is generated before the cloth catches fire, so ignition of the cloth can be prevented. By making the outer skin 314 a double structure of a flame-retardant base material and a soft bag, it is possible to achieve both a warm touch of the robot 100 and safety against high temperatures.

[0135] Note that the present invention is not limited to the above embodiments and modifications, and components can be modified and embodied without departing from the gist. Various inventions may be formed by appropriately combining a plurality of components disclosed in the above embodiments and modifications. Also, some components may be deleted from all the components shown in the above embodiments and modifications.

[0136] Although the robot system 300 has been described as being configured by one or more robots 100 and one server 200, part of the functions of the robot 100 may be realized by the server 200, or part or all of the functions of the server 200 may be assigned to the robot 100. One server 200 may control a plurality of robots 100, or a plurality of servers 200 may cooperate to control one or more robots 100.

[0137] A third device other than the robot 100 and the server 200 may undertake part of the functions. The aggregate of the functions of the robot 100 and the functions of the server 200 described in FIG. 4 can also be grasped as one "robot" overall. How to distribute the multiple functions necessary for implementing the present invention to one or more pieces of hardware may be determined in view of the processing capabilities of each piece of hardware and the specifications required for the robot system 300 and the like.

[0138] As described above, the "robot in the narrow sense" refers to the robot 100 that does not include the server 200, while the "robot in the broad sense" refers to the robot system 300. Many of the functions of the server 200 may be integrated into the robot 100 in the future.

Claims

1. An operation control unit that selects a motion of a robot, A drive mechanism that executes the motion selected by the operation control unit, A recognition unit that determines whether a target person satisfies a predetermined monitoring condition, A robot comprising a mode setting unit that sets the robot to a monitoring mode for the target person when the monitoring condition is satisfied.

2. Comprising a head, The robot according to claim 1, wherein the drive mechanism executes a motion of continuously turning the head in the direction in which the target person exists during the monitoring mode.

3. The robot according to claim 2, wherein in the monitoring mode, the positions of the other robot and the target person are shared, and the head is turned toward the same viewing point as the other robot determined based on the target person or a viewing point within a predetermined distance from the viewing point of the other robot.

4. The robot according to claim 2 or 3, wherein in the monitoring mode, at least one of the other robot and itself is configured to turn the head toward the target person.

5. The robot according to any one of claims 1 to 4, wherein the distance from the object is controlled within a predetermined distance during the monitoring mode.

6. The robot according to any one of claims 1 to 5, wherein during the monitoring mode, the amount of operation of the drive mechanism is reduced compared to at least one mode when the monitoring condition is not satisfied.

7. The robot according to any one of claims 1 to 6, wherein during the monitoring mode, when it is determined that the object satisfies a predetermined condition, a motion for the object is executed.

8. The robot according to any one of claims 1 to 7, wherein during the monitoring mode, when it is determined that the object satisfies a predetermined condition, a motion for the object is executed.

9. The robot according to any one of claims 1 to 9, wherein the mode setting unit is configured to set the monitoring mode when a condition that no person of a predetermined age or older is detected around the monitoring condition is satisfied.

10. The robot according to any one of claims 1 to 9, wherein the mode setting unit is configured to set the monitoring mode when a condition that no person associated with the target person is detected during the week is satisfied in addition to the monitoring condition.

11. The robot according to any one of claims 1 to 10, characterized in that in the monitoring mode, it comprises a communication unit that transmits a captured image of the subject to a predetermined communication terminal.

Images