A robot that autonomously selects actions based on its internal state or external environment.

Abstract

To provide a user with sense of security when living with robots.SOLUTION: An autonomous robot includes: an operation control unit which selects the motion of the robot; a driving mechanism which executes the motion selected by the operation control unit; a recognition unit which determines whether or not a target object satisfies a predetermined watching condition; a mode setting unit which sets a watching mode of the target object when the watching condition has been established; and a communication unit which transmits a taken image of the target object to a predetermined communication terminal in the watching mode.SELECTED DRAWING: Figure 1A

Description

Technical Field

[0001] The present invention relates to a robot that autonomously selects actions according to its internal state or external environment.

Background Art

[0002] People keep pets in search of healing. On the other hand, many people give up on keeping pets for various reasons such as not being able to secure enough time to take care of the pet, not having a living environment suitable for keeping a pet, having allergies, or finding it too painful to experience the death of a pet. If there were a robot that could perform the role of a pet, it might be able to give the kind of healing that a pet provides to those who cannot keep a pet (see Patent Documents 1 and 2).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0004] In recent years, although robot technology has been rapidly advancing, it has not yet achieved the presence as a companion like a pet. This is because it is difficult to think that a robot has free will. People observe the actions of a pet as if it has free will, feel the existence of free will in the pet, empathize with the pet, and are healed by the pet.

[0005] This invention was completed based on the above-mentioned problem recognition, and its first objective is to provide technology that provides users with various kinds of reassurance in their daily lives with robots. Its second objective is to provide technology that allows robots to express their affection for their users in a rich way. Its third objective is to provide technology that enables multiple robots to act in cooperation with each other. [Means for solving the problem]

[0006] An autonomous robot in one aspect of the present invention comprises: a motion control unit that selects the robot's motion; a drive mechanism that executes the motion selected by the motion control unit; a recognition unit that determines whether a subject meets predetermined monitoring conditions; a mode setting unit that sets the subject to a monitoring mode when the monitoring conditions are met; and a communication unit that transmits captured images of the subject to a predetermined communication terminal in the monitoring mode. [Effects of the Invention]

[0007] According to the present invention, it becomes easier to further enhance the presence of the robot. [Brief explanation of the drawing]

[0008] The aforementioned objectives, as well as other objectives, features, and advantages, will become even clearer from the preferred embodiments described below and the accompanying drawings.

[0009] [Figure 1A] This is a front view of the robot. [Figure 1B] This is a side view of the robot. [Figure 2] This is a cross-sectional view illustrating the structure of the robot. [Figure 3] This is a hardware configuration diagram of the robot in its basic configuration. [Figure 4] This is a functional block diagram of the robot system in its basic configuration. [Figure 5A] This is a schematic diagram illustrating the behavioral scenarios of multiple robots watching over a baby. [Figure 5B] It is a schematic diagram for explaining the action scene when multiple robots watch over a baby. [Figure 5C] It is a schematic diagram for explaining the action scene when multiple robots watch over a baby. [Figure 5D] It is a schematic diagram for explaining the action scene when multiple robots watch over a baby. [Figure 6A] It is a schematic diagram for explaining the action scene when multiple robots watch over a baby. [Figure 6B] It is a schematic diagram for explaining the action scene when multiple robots watch over a baby. [Figure 6C] It is a schematic diagram for explaining the action scene when multiple robots watch over a baby. [Figure 6D] It is a schematic diagram for explaining the action scene when multiple robots watch over a baby. [Figure 7A] It is a schematic diagram for explaining the action scene when multiple robots watch over a baby. [Figure 7B] It is a schematic diagram for explaining the action scene when multiple robots watch over a baby. [Figure 7C] It is a schematic diagram for explaining the action scene when multiple robots watch over a baby. [Figure 7D] It is a schematic diagram for explaining the action scene when multiple robots watch over a baby. [Figure 7E] It is a schematic diagram for explaining the action scene when multiple robots watch over a baby. [Figure 8A] It is a schematic diagram for explaining the action scene when a robot stays at home. [Figure 8B] It is a schematic diagram for explaining the action scene when a robot stays at home. [Figure 8C] It is a schematic diagram for explaining the action scene when a robot stays at home. [Figure 8D]It is a schematic diagram for explaining the action scene when the robot is on guard. [Figure 9A] It is a schematic diagram for explaining the action scene when the robot is on guard. [Figure 9B] It is a schematic diagram for explaining the action scene when the robot is on guard. [Figure 9C] It is a schematic diagram for explaining the action scene when the robot is on guard. [Figure 9D] It is a schematic diagram for explaining the action scene when the robot is on guard. [Figure 10A] It is a schematic diagram for explaining the action scene when the robot is on guard. [Figure 10B] It is a schematic diagram for explaining the action scene when the robot is on guard. [Figure 10C] It is a schematic diagram for explaining the action scene when the robot is on guard. [ [Figure 11A] It is a schematic diagram for explaining the action scene when the robot is on guard. [Figure 11B] It is a schematic diagram for explaining the action scene when the robot is on guard. [Figure 11C] It is a schematic diagram for explaining the action scene when the robot is on guard. [Figure 12A] It is a schematic diagram for explaining the action scene when a user who is out remotely operates the robot. [Figure 12B] It is a schematic diagram for explaining the action scene when a user who is out remotely operates the robot. [Figure 12C] It is a schematic diagram for explaining the action scene when a user who is out remotely operates the robot. [Figure 12D] It is a schematic diagram for explaining the action scene when a user who is out remotely operates the robot. [Figure 13A] It is a schematic diagram for explaining the action scene when a user who is out remotely operates the robot. [Figure 13B] This is a schematic diagram illustrating the actions of a user who remotely controls a robot while away from home. [Figure 13C] This is a schematic diagram illustrating the actions of a user who remotely controls a robot while away from home. [Figure 13D] This is a schematic diagram illustrating the actions of a user who remotely controls a robot while away from home. [Figure 14A] This is a schematic diagram illustrating the actions of a user who remotely controls a robot while away from home. [Figure 14B] This is a schematic diagram illustrating the actions of a user who remotely controls a robot while away from home. [Figure 14C] This is a schematic diagram illustrating the actions of a user who remotely controls a robot while away from home. [Figure 14D] This is a schematic diagram illustrating the actions of a user who remotely controls a robot while away from home. [Figure 15A] This is a schematic diagram illustrating the behavioral scenarios of multiple robots monitoring an elderly person. [Figure 15B] This is a schematic diagram illustrating the behavioral scenarios of multiple robots monitoring an elderly person. [Figure 15C] This is a schematic diagram illustrating the behavioral scenarios of multiple robots monitoring an elderly person. [Figure 15D] This is a schematic diagram illustrating the behavioral scenarios of multiple robots monitoring an elderly person. [Figure 16A] This is a schematic diagram illustrating the behavioral scenarios of multiple robots monitoring an elderly person. [Figure 16B] This is a schematic diagram illustrating the behavioral scenarios of multiple robots monitoring an elderly person. [Figure 16C] This is a schematic diagram illustrating the behavioral scenarios of multiple robots monitoring an elderly person. [Figure 16D] This is a schematic diagram illustrating the behavioral scenarios of multiple robots monitoring an elderly person. [Figure 17A]This is a schematic diagram illustrating the behavioral scenarios of multiple robots monitoring an elderly person. [Figure 17B] This is a schematic diagram illustrating the behavioral scenarios of multiple robots monitoring an elderly person. [Figure 17C] This is a schematic diagram illustrating the behavioral scenarios of multiple robots monitoring an elderly person. [Figure 17D] This is a schematic diagram illustrating the behavioral scenarios of multiple robots monitoring an elderly person. [Figure 18A] This is a schematic diagram illustrating the behavioral scenarios of multiple robots monitoring an elderly person. [Figure 18B] This is a schematic diagram illustrating the behavioral scenarios of multiple robots monitoring an elderly person. [Figure 18C] This is a schematic diagram illustrating the behavioral scenarios of multiple robots monitoring an elderly person. [Modes for carrying out the invention]

[0010] In this embodiment, the robot 100 shares daily life with the user, sometimes shows consideration for the user, sometimes strives to be helpful to the user, and actively seeks the user's affection, thereby demonstrating its presence as a member of the family. The following describes the basic configuration of robot 100 in relation to Figures 1 to 4, and then explains various behavioral scenarios of robot 100.

[0011] [Basic configuration] Figure 1 shows the external appearance of robot 100. Figure 1A is a front view, and Figure 1B is a side view. Robot 100 is an autonomous robot that determines its actions based on its external environment and internal state. The external environment is perceived by various sensors, such as a camera and a thermal sensor 115. The internal state is quantified as various parameters that express the emotions of Robot 100. Robot 100 operates within the confines of the owner's home. Hereinafter, humans involved with Robot 100 will be referred to as "users." Among the users, the owner or manager of Robot 100 will be referred to as the "owner."

[0012] The body 104 of robot 100 has an overall rounded shape and includes an outer skin 314 made of a soft, elastic material such as urethane, rubber, resin, or fiber. Robot 100 may be dressed in clothes. The total weight of robot 100 is approximately 5 to 15 kilograms, and its height is approximately 0.5 to 1.2 meters. The combination of moderate weight, roundness, softness, and pleasant texture makes it easy for users to pick up and want to pick up robot 100.

[0013] The robot 100 includes a pair of front wheels 102 (left wheel 102a, right wheel 102b) and one rear wheel 103. The front wheels 102 are the drive wheels, and the rear wheel 103 is the driven wheel. The front wheels 102 do not have a steering mechanism, but the rotation speed and direction of the left and right wheels can be controlled individually. The rear wheel 103 is a caster and is rotatable to move the robot 100 forward, backward, left, and right. The rear wheel 103 may also be an omni-wheel. By increasing the rotation speed of the right wheel 102b compared to the left wheel 102a, the robot 100 can turn left or rotate counterclockwise. By increasing the rotation speed of the left wheel 102a compared to the right wheel 102b, the robot 100 can turn right or rotate clockwise.

[0014] The front wheels 102 and rear wheels 103 can be fully retracted into the body 104 by a drive mechanism (rotation mechanism, linkage mechanism). The lower half of the body 104 is provided with a pair of left and right covers 312. The covers 312 are made of a flexible and elastic resin material (rubber, silicone rubber, etc.) and form a soft body while also housing the front wheels 102. The covers 312 have slits 313 (openings) that open from the sides to the front, and the front wheels 102 can be extended through these slits 313 and exposed to the outside.

[0015] Even when the robot is moving, most of the wheels are hidden within the body 104, but when the wheels are fully retracted into the body 104, the robot 100 becomes immobile. That is, as the wheels are retracted, the body 104 lowers and sits on the floor surface F. In this seated state, the flat seating surface 108 (ground contact bottom surface) formed on the bottom of the body 104 comes into contact with the floor surface F.

[0016] Robot 100 has two arms 106. Each arm 106 has a hand at its end, but it does not have the function of grasping objects. The arms 106 can perform simple movements such as raising, bending, waving, and vibrating, driven by actuators described later. Each of the two arms 106 can be controlled individually.

[0017] A face region 116 is exposed on the front of the robot 100's head. Two eyes 110 are provided in the face region 116. The eyes 110 are capable of displaying images using liquid crystal or organic EL elements, and are devices that express gaze and facial expressions by moving the pupils and eyelids displayed as images. A nose 109 is provided in the center of the face region 116. An analog stick is provided in the nose 109, which can detect movement in all directions (up, down, left, and right) as well as in the direction of pressing. In addition, the robot 100 is equipped with multiple touch sensors on the head, torso, and The robot 100 can detect user touches on almost its entire surface, including its buttocks and arms. It is equipped with various sensors, including a microphone array capable of identifying the direction of sound sources and ultrasonic sensors. It also has a built-in speaker and can emit simple voice commands.

[0018] A horn 112 is attached to the head of robot 100. A panoramic camera 113 is attached to the horn 112, allowing for simultaneous imaging of the entire upper area of ​​robot 100. A thermal sensor 115 (thermal camera) is also built into the horn 112. In addition, multiple modules (not shown) for infrared communication are provided on the horn 112, and these modules are arranged in a ring facing outwards. As a result, robot 100 can communicate using infrared while recognizing direction. Furthermore, an emergency stop switch is provided on the horn 112, allowing the user to emergency stop robot 100 by pulling out the horn 112.

[0019] Figure 2 is a schematic cross-sectional view showing the structure of robot 100. The body 104 includes a main frame 310, a pair of arms 106, a pair of covers 312, and an outer skin 314. The main frame 310 includes a head frame 316 and a torso frame 318. The head frame 316 is hollow and hemispherical, forming the head skeleton of the robot 100. The torso frame 318 is rectangular and forms the torso skeleton of the robot 100. The lower end of the torso frame 318 is fixed to a lower plate 334. The head frame 316 is connected to the torso frame 318 via a connecting mechanism 330.

[0020] The torso frame 318 forms the axis of the body 104. The torso frame 318 is constructed by fixing a pair of left and right side plates 336 to a lower plate 334, and supports a pair of arms 106 and internal mechanisms. Inside the torso frame 318 are the battery 118, control circuit 342, and various actuators. The bottom surface of the lower plate 334 forms the seating surface 108.

[0021] The body frame 318 has an upper plate 332 on its upper part. A bottomed cylindrical support part 319 is fixed to the upper plate 332. The upper plate 332, lower plate 334, a pair of side plates 336, and support part 319 constitute the body frame 318. The outer diameter of the support part 319 is smaller than the distance between the left and right side plates 336. A pair of arms 106 are assembled integrally with an annular member 340 to constitute an arm unit 350. The annular member 340 is ring-shaped, and the pair of arms 106 are attached so as to be radially separated along its centerline. The annular member 340 is inserted coaxially through the support part 319 and rests on the upper end surfaces of the pair of side plates 336. The arm unit 350 is supported from below by the body frame 318.

[0022] The head frame 316 has a yaw axis 321, a pitch axis 322, and a roll axis 323. Rotation of the head frame 316 around the yaw axis 321 (yawing) enables head-turning motion, rotation around the pitch axis 322 (pitching) enables nodding, looking up, and looking down motion, and rotation around the roll axis 323 (rolling) enables tilting the head from side to side. The position and angle of each axis can change in three-dimensional space depending on the driving mode of the connection mechanism 330. The connection mechanism 330 consists of a link mechanism and is driven by multiple motors installed on the body frame 318.

[0023] The torso frame 318 houses the wheel drive mechanism 370. The wheel drive mechanism 370 includes a front-wheel drive mechanism and a rear-wheel drive mechanism that extend and retract the front wheels 102 and rear wheels 103 from the body 104, respectively. The front wheels 102 and rear wheels 103 function as a "movement mechanism" to move the robot 100. The front wheels 102 have a direct-drive motor in their center. Therefore, the left wheel 102a and the right wheel 102b can be driven individually. The front wheels 102 are wheels The wheel cover 105 is rotatably supported by the body frame 318.

[0024] A pair of covers 312 are provided to cover the fuselage frame 318 from the left and right sides, and have a smooth curved shape to give the outline of the body 104 a rounded shape. A closed space is formed between the fuselage frame 318 and the covers 312, and this closed space serves as the housing space S for the front wheels 102. The rear wheels 103 are housed in a housing space provided at the lower rear of the fuselage frame 318.

[0025] The outer skin 314 covers the main frame 310 and a pair of arms 106 from the outside. The outer skin 314 has a thickness that a person can feel as elastic and is made of an elastic material such as urethane sponge. This allows a user to feel a moderate softness when hugging the robot 100, enabling natural physical contact similar to that with a pet. The outer skin 314 is attached to the main frame 310 in such a manner that the cover 312 is exposed. An opening 390 is provided at the upper end of the outer skin 314. The horns 112 are inserted through this opening 390.

[0026] A touch sensor is positioned between the main frame 310 and the outer casing 314. A touch sensor is embedded in the cover 312. All of these touch sensors are capacitive sensors and detect touches across almost the entire surface of the robot 100. The touch sensors may be embedded in the outer casing 314 or positioned inside the main frame 310.

[0027] The arm 106 has a first joint 352 and a second joint 354, with the arm 356 between the two joints and a hand 358 at the end of the second joint 354. The first joint 352 corresponds to the shoulder joint, and the second joint 354 corresponds to the wrist joint. A motor is provided at each joint to drive the arm 356 and the hand 358, respectively. The drive mechanism for driving the arm 106 includes these motors and their drive circuit 344.

[0028] Figure 3 is a hardware configuration diagram of robot 100. Robot 100 includes an internal sensor 128, a communication device 126, a storage device 124, a processor 122, a drive mechanism 120, and a battery 118. The drive mechanism 120 includes the aforementioned connection mechanism 330 and wheel drive mechanism 370. The processor 122 and storage device 124 are included in the control circuit 342. Each unit is connected to the others by power lines 130 and signal lines 132. Battery 118 supplies power to each unit via power lines 130. Each unit sends and receives control signals via signal lines 132. Battery 118 is a lithium-ion secondary battery and is the power source for robot 100.

[0029] The internal sensor 128 is a collection of various sensors built into the robot 100. Specifically, it includes a camera, microphone array, distance sensor (infrared sensor), thermosensor 115, touch sensor, acceleration sensor, barometric pressure sensor, and odor sensor. The touch sensor covers most of the body 104 and detects user touch based on changes in capacitance. The odor sensor is a known sensor that applies the principle that electrical resistance changes due to the adsorption of odor-causing molecules.

[0030] The communication device 126 is a communication module that performs wireless communication with various external devices. The storage device 124 consists of non-volatile memory and volatile memory and stores computer programs and various setting information. The processor 122 is a means for executing computer programs. The drive mechanism 120 includes multiple actuators. In addition, a display and a speaker are also installed.

[0031] The drive mechanism 120 primarily controls the wheels and the head. The drive mechanism 120 controls robot 1 In addition to changing the direction and speed of movement of 00, the wheels can also be raised and lowered. When the wheels are raised, they are completely retracted into the body 104, and the robot 100 comes into contact with the floor surface F at the seating surface 108, entering a seated position. The drive mechanism 120 also controls the arms 106.

[0032] Figure 4 is a functional block diagram of the robot system 300. The robot system 300 includes a robot 100, a server 200, and a number of external sensors 114. Each component of the robot 100 and server 200 is realized by hardware including arithmetic units such as a CPU (Central Processing Unit) and various coprocessors, storage devices such as memory and storage, and wired or wireless communication lines connecting them, and software stored in the storage devices that supplies processing instructions to the arithmetic units. The computer program may consist of device drivers, an operating system, various application programs located at a higher layer, and libraries that provide common functions to these programs. The blocks described below represent functional units, not hardware units. Some of the functions of the robot 100 may be implemented by the server 200, and some or all of the functions of the server 200 may be implemented by the robot 100.

[0033] Multiple external sensors 114 are pre-installed inside the house. The server 200 manages the external sensors 114 and provides the robot 100 with detection values ​​acquired by the external sensors 114 as needed. The robot 100 determines its basic actions based on information obtained from the internal sensor 128 and the multiple external sensors 114. The external sensors 114 are for augmenting the sensory organs of the robot 100, and the server 200 is for augmenting the processing capabilities of the robot 100. The robot 100's communication device 126 may periodically communicate with the server 200, and the server 200 may be responsible for determining the robot 100's position using the external sensors 114 (see also Patent Document 2).

[0034] (Server 200) The server 200 includes a communication unit 204, a data processing unit 202, and a data storage unit 206. The communication unit 204 is responsible for communication processing with the external sensor 114 and the robot 100. The data storage unit 206 stores various types of data. The data processing unit 202 performs various processes based on the data acquired by the communication unit 204 and the data stored in the data storage unit 206. The data processing unit 202 also functions as an interface between the communication unit 204 and the data storage unit 206.

[0035] The data storage unit 206 includes a motion storage unit 232 and a personal data storage unit 218. Robot 100 has multiple motion patterns. Various motions are defined, such as shaking its arm 106, approaching the owner in a meandering motion, and gazing at the owner while tilting its head.

[0036] The motion storage unit 232 stores "motion files" that define the control content of the motions. Each motion is identified by a motion ID. The motion files are also downloaded to the motion storage unit 160 of the robot 100. Which motion to execute may be determined by the server 200 or by the robot 100.

[0037] Many of the robot 100's motions are composed of complex motions that include multiple unit motions. For example, when robot 100 approaches its owner, it can be expressed as a combination of unit motions: turning to face the owner, raising its arms as it approaches, swaying its body as it approaches, and sitting down with both arms raised. This is also acceptable. By combining these four motions, the motion of "approaching the owner, raising a hand midway, and finally shaking the body before sitting down" is realized. The motion file defines the rotation angle and angular velocity of the actuators installed on the robot 100, relating them to the time axis. Various motions can be expressed by controlling each actuator as time progresses according to the motion file (actuator control information).

[0038] The transition time between one unit motion and the next is called the "interval." The interval should be defined according to the time required for the unit motion change and the nature of the motion. The length of the interval is adjustable. Hereinafter, the settings related to the control of robot 100's actions, such as when and which motion to select, and the output adjustment of each actuator in realizing the motion, will be collectively referred to as "action characteristics." The action characteristics of robot 100 are defined by the motion selection algorithm, motion selection probability, motion file, etc.

[0039] The motion storage unit 232 stores motion files as well as a motion selection table that defines the motions to be executed when various events occur. In the motion selection table, one or more motions and their selection probabilities are associated with each event.

[0040] The personal data storage unit 218 stores user information. Specifically, it stores master information indicating the level of familiarity with the user and the user's physical and behavioral characteristics. Other attribute information such as age and gender may also be stored.

[0041] Robot 100 has an internal parameter called "affection level" for each user. When Robot 100 recognizes actions that show it is fond of the user, such as picking it up or talking to it, its affection level with that user increases. The affection level will be lower for users who do not interact with Robot 100, users who act rudely, or users it does not encounter often.

[0042] The data processing unit 202 includes a location management unit 208, a recognition unit 212, an operation control unit 222, a closeness management unit 220, and a state management unit 244. The position management unit 208 identifies the position coordinates of the robot 100. The state management unit 244 manages various internal parameters such as the charge level, internal temperature, and processing load of the processor 122. The state management unit 244 also manages various emotional parameters that indicate the robot 100's emotions (loneliness, curiosity, need for recognition, etc.). These emotional parameters are constantly fluctuating. The robot 100's target destination changes according to the emotional parameters. For example, when loneliness is high, the robot 100 sets the user's location as its target destination.

[0043] Emotional parameters change over time. Furthermore, various emotional parameters also change depending on the interaction described later. For example, when the owner holds the dog, the emotional parameter indicating loneliness decreases, and when the dog does not see the owner for a long period of time, the emotional parameter indicating loneliness gradually increases.

[0044] The recognition unit 212 recognizes the external environment. Recognition of the external environment includes various types of recognition, such as recognition of weather and season based on temperature and humidity, and recognition of shaded areas (safe zones) based on light intensity and temperature. The recognition unit 156 of the robot 100 acquires various environmental information using the internal sensor 128, processes it, and then transfers it to the recognition unit 212 of the server 200.

[0045] Specifically, the recognition unit 156 of the robot 100 identifies moving objects from an image, particularly people and animals. The system extracts the corresponding image region and, from the extracted image region, extracts a "feature vector" as a set of feature quantities that represent the physical and behavioral characteristics of the moving object. Feature vector components (feature quantities) are numerical values ​​that quantify various physical and behavioral characteristics. For example, the width of a human eye is quantified in the range of 0 to 1, forming one feature vector component. The method for extracting feature vectors from captured images of people is an application of known face recognition technology. Robot 100 transmits the feature vector to server 200.

[0046] The recognition unit 212 of the server 200 determines which person the captured user belongs to by comparing the feature vector extracted from the image captured by the robot 100's built-in camera with the feature vector of a user (cluster) pre-registered in the personal data storage unit 218 (user identification processing). The recognition unit 212 also estimates the user's emotions by performing image recognition on the user's facial expressions. The recognition unit 212 also performs user identification processing on moving objects other than people, such as pets like cats and dogs.

[0047] The recognition unit 212 recognizes various responses made to the robot 100 and classifies them as pleasant or unpleasant. The recognition unit 212 also recognizes the owner's responses to the robot 100's actions and classifies them as positive or negative responses. Pleasant and unpleasant actions are determined by whether the user's response is biologically pleasant or unpleasant. For example, being held is a pleasant action for robot 100, while being kicked is an unpleasant action. Positive and negative responses are determined by whether the user's response indicates a pleasant or unpleasant emotion for the user. Being held is a positive response indicating a pleasant emotion for the user, while being kicked is a negative response indicating an unpleasant emotion for the user.

[0048] The motion control unit 222 of the server 200 works in cooperation with the motion control unit 150 of the robot 100 to determine the motion of the robot 100. The motion control unit 222 of the server 200 creates the target location for the robot 100's movement and the movement route for that location. The motion control unit 222 may create multiple movement routes and then select one of them.

[0049] The motion control unit 222 selects a motion for the robot 100 from a plurality of motions stored in the motion storage unit 232. Each motion is associated with a selection probability for each situation. For example, a selection method is defined such as executing motion A with a 20% probability when the owner performs a pleasant action, or executing motion B with a 5% probability when the temperature exceeds 30 degrees Celsius.

[0050] The intimacy management unit 220 manages the intimacy level for each user. As mentioned above, the intimacy level is registered as part of the personal data in the personal data storage unit 218. When a pleasant action is detected, the intimacy management unit 220 increases the intimacy level with that owner. When an unpleasant action is detected, the intimacy level decreases. In addition, the intimacy level of owners who have not been viewed for a long period of time gradually decreases.

[0051] (Robot 100) The robot 100 includes a communication unit 142, a data processing unit 136, a data storage unit 148, an internal sensor 128, and a drive mechanism 120. The communication unit 142 corresponds to the communication device 126 (see Figure 3) and is responsible for communication processing with the external sensor 114, the server 200, and other robots 100. The data storage unit 148 stores various types of data. The data storage unit 148 corresponds to the storage device 124 (see Figure 3). The data processing unit 136 performs various processes based on the data acquired by the communication unit 142 and the data stored in the data storage unit 148. The data processing unit 136 corresponds to the processor 122 and the computer program executed by the processor 122. The processing unit 136 also functions as an interface for the communication unit 142, the internal sensor 128, the drive mechanism 120, and the data storage unit 148.

[0052] The data storage unit 148 includes a motion storage unit 160 that defines various motions of the robot 100. Various motion files are downloaded from the motion storage unit 232 of the server 200 to the motion storage unit 160 of the robot 100. Motions are identified by motion IDs. In order to express various motions such as retracting the front wheels 102 and sitting down, raising the arms 106, making the robot 100 rotate by rotating the two front wheels 102 in opposite directions or by rotating only one of the front wheels 102, shaking by rotating the front wheels 102 while they are retracted, and stopping briefly and looking back when moving away from the user, the timing of operation, duration, and direction of operation of various actuators (drive mechanisms 120) are defined chronologically in the motion files. Various data may be downloaded to the data storage unit 148 from the personal data storage unit 218.

[0053] The data processing unit 136 includes a recognition unit 156 and an operation control unit 150. The motion control unit 150 of the robot 100 works in cooperation with the motion control unit 222 of the server 200 to determine the motion of the robot 100. Some motions may be determined by the server 200, while others may be determined by the robot 100. Alternatively, the robot 100 may determine the motion, but if the processing load on the robot 100 is high, the server 200 may determine the motion. The server 200 may determine the base motion, and the robot 100 may determine additional motions. How the motion determination process is divided between the server 200 and the robot 100 should be designed according to the specifications of the robot system 300.

[0054] The motion control unit 150 of the robot 100 instructs the drive mechanism 120 to execute the selected motion. The drive mechanism 120 controls each actuator according to the motion file.

[0055] The motion control unit 150 can perform a motion of raising both arms 106 as a gesture of asking to be held when a user with whom it has a close relationship is nearby, or it can express a motion of not wanting to be held by repeatedly rotating in the opposite direction and stopping while keeping the left and right front wheels 102 retracted. The drive mechanism 120 drives the front wheels 102, arms 106, and neck (head frame 316) according to the instructions of the motion control unit 150, thereby causing the robot 100 to express various motions.

[0056] The recognition unit 156 of the robot 100 interprets external information obtained from the internal sensor 128. The recognition unit 156 is capable of visual recognition (visual unit), smell recognition (olfactory unit), sound recognition (auditory unit), and tactile recognition (tactile unit).

[0057] The recognition unit 156 extracts feature vectors from the captured image of the moving object. As described above, the feature vector is a set of parameters (feature quantities) that indicate the physical and behavioral characteristics of the moving object. When a moving object is detected, physical and behavioral characteristics are also extracted from the odor sensor, built-in sound-collecting microphone, temperature sensor, etc. These features are also quantified and become feature vector components. Based on known technologies described in Patent Document 2, etc., the recognition unit 156 identifies the user from the feature vector.

[0058] In the series of recognition processes including detection, analysis, and judgment, the recognition unit 156 of the robot 100 selects and extracts the information necessary for recognition, while the interpretation process such as judgment is performed by the recognition unit 212 of the server 200. This is performed by the following. The recognition process may be performed by the recognition unit 212 of the server 200 alone, or by the recognition unit 156 of the robot 100 alone, or as described above, both may perform the recognition process while sharing responsibilities.

[0059] When a strong impact is applied to the robot 100, the recognition unit 156 recognizes this using the touch sensor and acceleration sensor, and the recognition unit 212 of the server 200 recognizes that a "violent act" has been committed by a nearby user. A violent act may also be recognized when a user grabs the horns 112 and lifts the robot 100. When a user facing the robot 100 speaks in a specific volume range and frequency band, the recognition unit 212 of the server 200 may recognize that a "verbal interaction" has been performed on it. Furthermore, when a temperature of approximately body temperature is detected, it is recognized that a "contact action" has been performed by a user, and when upward acceleration is detected while contact has been recognized, it is recognized that the robot has been "picked up". Physical contact when a user lifts the body 104 may be sensed, or the robot may be picked up when the load on the front wheels 102 decreases. In summary, the robot 100 acquires the user's actions as physical information using its internal sensor 128, and the recognition unit 212 of the server 200 determines whether the user is pleased or displeased. The recognition unit 212 of the server 200 also performs user identification processing based on feature vectors.

[0060] The recognition unit 212 of the server 200 recognizes various user responses to the robot 100. Some typical responses among the various responses are associated with pleasant or unpleasant, and positive or negative. Generally, most pleasant responses are positive responses, and most unpleasant responses are negative responses. Pleasant and unpleasant responses are related to the level of familiarity, and positive and negative responses influence the robot 100's behavioral choices.

[0061] In response to the interaction recognized by the recognition unit 156, the intimacy management unit 220 of the server 200 changes the intimacy level with the user. In principle, the intimacy level increases for users who perform pleasant actions and decreases for users who perform unpleasant actions.

[0062] Each function of the server 200 is realized by loading the program necessary to implement that function into memory and instantiating it. The processing power of the server 200 supplements the various processes performed by the robot 100. The server 200 can be used as a resource for the robot 100. How the resources of the server 200 are used is dynamically determined in response to requests from the robot 100. For example, if the robot 100 needs to continuously generate complex motions in response to detection values ​​from numerous touch sensors, the processing of the processor 122 in the robot 100 may be preferentially allocated to motion selection and generation, while the processing for image recognition of the surrounding environment may be performed by the recognition unit 212 of the server 200. In this way, various processes of the robot system 300 can be distributed between the robot 100 and the server 200.

[0063] Multiple robots 100 can also be controlled by a single server 200. In this case, each function of the server 200 is implemented independently for each robot 100. For example, the server 200 may have a recognition unit 212 for robot 100B in addition to the recognition unit 212 (instance object) for robot 100A.

[0064] Based on the above basic configuration, the implementation of the robot 100 in this embodiment will now be explained, focusing particularly on the features and purpose of this implementation and the differences from the basic configuration.

[0065] [SLAM] The robot 100 in this embodiment acquires a large number of captured images (still images) by periodically imaging the surrounding area with the panoramic camera 113. The robot 100 uses the captured images to... It forms memories (hereinafter referred to as "pictorial memories").

[0066] Image memory is a collection of multiple keyframes. A keyframe is distribution information of feature points (feature quantities) in an captured image. The robot 100 of this embodiment forms keyframes using graph-based SLAM (Simultaneous Localization and Mapping) technology that uses image features, more specifically, SLAM technology based on ORB (Oriented FAST and Rotated BRIEF) features (see Patent Document 3).

[0067] Robot 100 periodically forms keyframes while moving, creating an image memory as a collection of keyframes, or in other words, an image feature distribution. Robot 100 estimates its current location by comparing the keyframe acquired at its current location with a large number of keyframes it already possesses. That is, Robot 100 performs "spatial recognition" by comparing the captured image it is currently viewing with previously viewed captured images (memories) and aligning its current situation with past memories. The image memory, formed as a collection of feature points, becomes a map. Robot 100 updates the map while moving and estimating its current location.

[0068] The basic configuration of robot 100 is based on the premise that it recognizes its position using an external sensor 114 rather than keyframes. In this embodiment, robot 100 will be described as recognizing its location based solely on keyframes.

[0069] In this embodiment, the robot 100 includes a "mode setting unit" for setting various modes.

[0070] <Baby monitoring> Figures 5 to 7 are schematic diagrams illustrating the behavioral scenarios when multiple robots 100 watch over a baby. First, in the child's room, the baby being monitored (hereinafter referred to as "the subject") is sleeping in a bouncer (Figure 5A). There are two robots 100 in this house. The two robots 100 each recognize the presence and location of the baby (infant) through image recognition. The two robots 100 may share the baby's location through mutual communication. The two robots 100 determine their respective fixation points based on the baby's location. The two robots 100 correct their fixation points through mutual communication so that their fixation points are the same or within a predetermined range. By controlling the movement and parts of the two robots 100 to orient their heads towards their respective determined fixation points, the two robots 100 can perform an action as if they are peering at the baby (Figure 5B). One of the two robots 100, robot 100A, continuously watches over the baby by orienting its head towards the baby's location according to the baby's position. Robot 100A maintains a constant distance from the baby so as not to leave its side. The other robot, 100B, communicates with robot 100A to share the role and, after a while, performs actions that make it appear as if it has stopped watching and started playing with the baby (Figure 5C). The mother is cooking in the kitchen, leaving the 100 robots to watch over the baby (Figure 5D).

[0071] The mother has placed her smartphone (communication device) in the kitchen (Figure 6A). Robot 100A (monitoring) captures images of the baby using the panoramic camera 113 and continuously sends the captured images as live video to the smartphone. The smartphone displays the area of ​​the panoramic image in which the baby is visible. At this time, the distortion of the panoramic image is displayed on a flat surface. The corrected image may be displayed on the smartphone.

[0072] Suddenly, the baby starts crying (Figure 6B). When robot 100B, which was playing, hears a baby crying through the microphone (collects the sound), it approaches the baby based on the baby's location (Figure 6C). Robot 100B performs an interference motion. Robot 100B performs an interference motion if at least one of the following conditions is met: it has collected a sound like crying, and image analysis has determined that the baby is in a specific state, such as crying. The interference motion is a soothing motion such as outputting a predetermined sound that will attract the baby's attention, outputting a lullaby-like sound, waving its arm 106, shaking its head and body, or rocking the bouncer. The interference motion distracts the baby. Also, by performing the interference motion, it is possible to give a third party the impression that robot 100B is struggling to do something about the crying baby. Robot A may perform the interference motion instead of or in addition to robot B. Live video feeds from robot 100A show the crying baby on the mother's smartphone (Figure 6D). Videos of the baby before and after it starts crying (videos from a predetermined time before to a predetermined time after the baby starts crying) may be stored on an HDD or similar device for persistence.

[0073] The mother is surprised and instinctively picks up her smartphone (Figure 7A). The mother rushes to the nursery and runs to the baby (Figure 7B). The mother holds the baby (Figure 7C). The robots 100 stand beside the mother and baby and gaze upon them. The two robots recognize the relative positions of the mother and baby through image analysis and set a point of focus from each of their respective positions. The two robots 100 share their points of focus via communication and, if necessary, adjust them so that they are identical or within a predetermined range. The two robots 100 then perform the action of turning their heads toward their respective points of focus. As a result, the two robots 100, standing side by side and gazing at the mother and baby, can give the mother the impression that the robots 100 are concerned about her baby. The baby, held in its mother's arms, falls asleep again (Figure 7D). Once the robots confirm that the baby has fallen asleep, they move away from the bouncer (Figure 7E).

[0074] When the mother manually sets the robot 100 to "monitoring mode" via an input switch or smartphone, the robot 100 may search for the baby to initiate the monitoring actions described above. When the robot 100 detects that the baby is in the bouncer, the mode setting unit may automatically set it to monitoring mode. The robot 100 may also automatically switch to monitoring mode when it detects an infant through image recognition and there is no caregiver nearby. The absence of a caregiver nearby may include, for example, when no image of a person above a certain age is detected, or when no image of a person associated with the infant is detected. Furthermore, when multiple robots 100 are in the same room, all robots 100 may be set to monitoring mode, or only some of the robots 100 may be set to monitoring mode.

[0075] The robot 100 may switch to monitoring mode if it detects a "baby" in the captured image and does not detect any other users or specific users such as the mother or father.

[0076] The monitoring robot 100 will keep the baby it is monitoring within its line of sight. The area may be limited. Alternatively, the robot 100 may not leave the room where the baby is while monitoring. At a minimum, it is desirable that the robot 100 does not take its eyes off the baby while monitoring (for example, always keeping the baby in the camera's field of view). The robot 100 monitors the subject's condition using a panoramic camera 113, a microphone, and sensors that measure the external environment, such as a temperature sensor. If the baby moves around, the robot 100 may use the panoramic camera 113, a microphone, and sensors that measure the external environment, such as a temperature sensor, to recognize the baby's position and control its own direction to face the baby. In addition, if the baby enters a predetermined distance from the robot 100, the robot 100 may reduce the points of contact with the baby by retracting its wheels, etc.

[0077] When predetermined alert conditions are met, robot 100 will perform motions to actively engage with the baby. Alternatively, it may send "alert information" to another communication device, such as the mother's smartphone. Alert conditions can be set arbitrarily, but examples of situations where the baby (the person being monitored) is in danger include when the baby cries, approaches the stairs, tries to go outside, plays with small objects, or falls down. The motions to actively engage may be the interference motions mentioned above.

[0078] When robot 100A switches to monitoring mode, it may notify robot 100B that it has switched to monitoring mode. Upon receiving the notification, robot 100B may also switch to monitoring mode, or it may approach robot 100A or the baby. By having multiple robots 100 gather near the baby, it becomes possible to express behavior that suggests the robots 100 are concerned about the baby.

[0079] At this time, to avoid waking the baby with the sound of the robot 100's operation, the robot 100 suppresses the amount of movement of the drive mechanism 120 (especially the mechanism related to movement and posture adjustment) more than in normal mode, and operates quietly without making any noise. If the system switches to monitoring mode when the baby is awake, robot B may perform specific motions to attract the baby's attention, such as running around the bouncer or dancing. For example, the baby may be determined to be awake if certain conditions are met, such as detecting the baby's voice through voice analysis or detecting an image of the baby with their eyes open through image analysis. Such motions will not be performed when the baby is asleep. Robot B will perform appropriate motions according to the state of the person being monitored. It is thought that the baby will also feel more secure if many robots 100 gather around them.

[0080] Robot 100 may switch to monitoring mode when it receives a specific voice command, such as "Watch over the baby," from a specific user, such as a mother, via a microphone or the like. When monitoring begins, Robot 100 may perform confirmation actions, such as circling the baby, facing the baby, or pointing its hand 358 towards the baby, based on the baby's position detected by image analysis or the like. The mother (instructor) can determine from these confirmation actions whether Robot 100 is monitoring the correct person. The mother is expected to give a positive response such as "I'll leave it to you" if it is correct, and a negative response such as "That's not right" if it is incorrect. Therefore, Robot 100 may determine whether the person being monitored is correct based on the mother's response collected via the microphone after the confirmation actions.

[0081] During monitoring, Robot 100 maintains its gaze on the baby, such as by peering at the baby. The baby feels safe and secure, believing that Robot 100 is watching over it. The mother is moved to feel affection and trust for Robot 100 as she sees it diligently watching over the baby. The monitoring behavior also serves as an appeal to the user (witness), conveying that "Robot 100 is working hard."

[0082] As described above, robot 100B may play freely while robot 100A is monitoring. However, robot 100B's range of movement is limited to within a range where it can see the baby or robot 100A. Robot 100A may send a close-up live image (macro image) of the baby to the smartphone, and robot 100B may send a wide-angle live image (wide-angle image) of robot 100A watching over the baby to the smartphone. It is also desirable that robot 100B further restrict its range of movement so as not to interfere with robot 100A's filming of the baby. For example, robot 100B should identify the positions of the baby and robot 100A and act in a way that does not overlap the straight line connecting the baby and robot 100A. In addition, if robot 100A uses image recognition to determine that the baby can no longer be recognized by robot 100B, robot 100A may notify robot 100B of a command to move. Robot 100B will move in response to that command.

[0083] When image or sound analysis detects that the baby has entered a predetermined state, for example, when the baby starts crying, robot 100A notifies the user that the baby's state has changed, in addition to providing live video. Robot 100B may perform an interference motion. Robot 100A sends a message to the smartphone that succinctly indicates the state associated with the alert condition, such as "the baby is crying" or "the baby has started to fuss." As an interference motion, robot 100B may dance or play music such as lullabies using its built-in audio player. In any case, robot 100B soothes the baby by performing an action that distracts the baby. When the mother returns (see Figure 7B), robot 100B stops the interference motion. Specifically, robot 100B stops the interference motion when it can confirm "the mother" in the captured image. When the robot 100B recognizes, via a microphone or similar device, that the mother has pronounced a keyword indicating the completion of the monitoring action, such as "thank you" or "it's okay now" (hereinafter referred to as a "completion command"), the robot 100B may stop its interference motion. At this time, the robot may determine whether the completion command was spoken by the person who instructed the robot 100A to perform the monitoring action, and if the instruction came from that person, it may complete the monitoring action.

[0084] Robot 100 performs image recognition on the captured image to identify the baby. The method for detecting the "baby" can be implemented by applying known facial recognition technology. Furthermore, when the baby is moving, robot 100 may adjust its direction of movement so that it can always see the baby, regardless of whether it is monitoring the baby or not.

[0085] Robot 100A and Robot 100B may take turns performing the monitoring. For example, after Robot 100A has monitored for 10 minutes, Robot 100B may switch to monitoring mode, and Robot 100A may be allowed to act freely. It is thought that having multiple Robot 100s take turns monitoring will prevent the baby from getting bored. It also makes it easier for a third party to feel as if Robot 100A and Robot 100B are cooperating in monitoring the baby.

[0086] With Robot 100 watching over the baby, busy mothers can focus on household chores with peace of mind. When doing chores like laundry in a separate area from the baby, it's easy to miss the baby crying. Before the mother even realizes the baby is crying, the baby may start crying intensely. This situation places a heavy burden on the mother. By having Robot 100 watch over the baby, it can quickly detect signs of fussiness or other signs that indicate an impending crying fit.

[0087] Not only the mother, but also Robot 100 will participate in childcare. Furthermore, babies raised under the watchful eye of Robot 100 may develop a sense of closeness to Robot 100 in the future. It is hoped that this will happen.

[0088] <Staying home alone> Figures 8 to 11 are schematic diagrams illustrating the behavioral scenarios of robot 100 when it is left alone at home. Let's consider a household with two robots, 100A and 100B, and a female owner. The female owner leaves for work (Figure 8A). At this time, the female owner says to robot 100A, who is nearby, "Please stay home and watch the house." Upon hearing these words (collecting the voice via a microphone), robot 100A recognizes that the female owner is leaving and switches to "home mode" (Figure 8B). The female owner leaves through the front door (Figure 8C). Robot 100A moves to the front door and sees the female owner off by performing a predetermined motion.

[0089] Meanwhile, robot 100B, which was in the room, starts chasing robot 100A and playing (Figure 8D). When the female owner goes out, both robot 100A and robot 100B may see her off, or only robot 100 whose intimacy level with the female owner is above a predetermined value may see her off.

[0090] After a while, the front door opens while the house is out (Figure 9A). If the two robots 100 detect that the front door has opened through voice analysis or image analysis, they move to the front door in anticipation of the female owner's return (Figure 9B). However, the person who appeared (identified through image analysis, etc.) was not the female owner, but an unknown person (suspicious person) carrying a large bag (Figure 9C). At this time, the robot 100 approaches the suspicious person sufficiently and photographs the suspicious person (Figure 9D). The robot 100 switches to alert mode. Once in alert mode, the robots 100 may move closer to the suspicious person and continue photographing them.

[0091] The female owner is working in her office. Robot 100 sends the image of the suspicious person to the female owner's smartphone (Figure 10A). The female owner, while at work, learns via her smartphone that Robot 100 has detected a suspicious person (Figure 10B). In this case, let's assume the person she thought was suspicious was actually her mother. The robots 100 do not know the owner's mother. The woman notifies the robots 100 via her smartphone that "it's not a suspicious person." The owner could also tell the robots 100 by voice, "It's my mother, so there's nothing to worry about." The robots 100 then deactivate their alert mode. The mother cooks homemade meals (Figure 10C).

[0092] The female owner (daughter) returns home and chats with her mother while eating a home-cooked meal (Figure 11A). Robot 100 is playing beside the two (Figures 11B and 11C).

[0093] Robot 100, through image analysis and voice recognition, detects suspicious individuals (people it has never seen before or people with a level of familiarity below a certain threshold) while the homeowner is away. It then photographs the suspicious individuals and sends the images to the female owner's (specific user's) smartphone. This control method allows the female owner to confidently entrust the security of her home to Robot 100. In addition, through image analysis and voice recognition, Robot 100 can detect various events that occur while the homeowner is away, such as earthquakes causing damage, gas leaks, or visits from delivery personnel (calls via the intercom), and notifies the owner's smartphone of these events. Robot 100 also records events that occur while the homeowner is away as a life log, and the owner can check the life log on their smartphone after returning home to see what happened while they were away.

[0094] The mode setting unit of robot 100 may also set the home-guarding mode based on user input. Robot 100 may automatically change to home-guarding mode when it detects a specific event, such as the user leaving the house through the front door. Alternatively, robot 100 may automatically change to home-guarding mode if it has not been able to see the user in the room for a certain period of time or longer.

[0095] In alert mode, robot 100 may set its range of action to a position where it can photograph suspicious individuals, in order to avoid missing any suspicious activity. It may also act away from suspicious individuals to prevent violence from them. After reporting a suspicious individual, if robot 100 receives notification from the user that the individual is not suspicious, it will deactivate alert mode and return to home-sitting mode. After this, robot 100 may interact with the unidentified person (former suspicious individual) in a normal manner. It may also remember the appearance of the unidentified person and manage parameters such as familiarity. It may even become attached to the unidentified person. In the example shown in Figures 8 to 11, the mother is initially wary of robot 100, but after the female owner (daughter) gives robot 100 an approval notification, robot 100 begins to cling to the mother. The mother can then feel that she has been accepted and welcomed by robot 100.

[0096] In home security mode, a user (owner) in a remote location may send an indoor check command to the robot 100 via their smartphone. When the robot 100 receives an indoor check command, it patrols the house according to the map generated based on SLAM. At this time, the robot 100 may send captured images to the user's smartphone or notify the user of any abnormal events. The user can check the status of their home at any time by sending an indoor check command.

[0097] <Remote control> Figures 12 to 14 are schematic diagrams illustrating the actions of a user who remotely controls robot 100 while away from home. Two robots, 100A and 100B, are left to look after the house while the residents are away. There is also a cat in the house (Figure 12A). Meanwhile, the rest of the family leaves the robots 100 and the cat behind and heads out into the city (Figure 12B). The boy looks dejected (Figure 12C). The boy takes out his smartphone (mobile device). The boy starts operating the smartphone in some way (Figure 12D).

[0098] Two Robot 100s are playing in the empty house (Figure 13A). When robot 100A starts moving, robot 100B follows robot 100A (Figure 13B). Robots 100A and 100B are playing a game of chase. The mother is worried about her son (Figure 13C). The boy is worried because the cat seemed less energetic when he leaves the house (Figure 13D).

[0099] The boy sends a command to robot 100 from his smartphone, saying, "Check on the cat." Robots 100A and 100B move to the location or object indicated in the command and take pictures as appropriate. The images captured by robots 100A and 100B are sent to the smartphone. The cat is playing happily on the cat tree (Figure 14A). Seeing the cat looking healthy brings relief to the family (Figure 14B). Robot 100 films a cat playing on top of a cat tower. Robot 100 takes a close-up shot of the cat, and the cat stares back at Robot 100 (Figures 14C and 14D).

[0100] In this way, the user can send various instructions to the robot 100 from their smartphone. In particular, the user can command the robot 100 to check the room. If the user sends a command such as "check on the cat" as in the above embodiment, the robot 100 will detect an object corresponding to the "cat" from the captured image and send the captured image centered on the cat to the smartphone. Such commands may be voice commands or may be entered through the graphical user interface of the smartphone. The user may also operate the robot 100 like a radio-controlled car (hereinafter, this method of operation will be referred to as "remote control").

[0101] Images captured by robot 100 are displayed on a smartphone, and the user may enlarge and view parts of the captured images that are being live-streamed to the smartphone that they particularly want to see. Robot 100 may also transmit the entire panoramic image to the smartphone, and the user may confirm what robot 100 "saw" through the panoramic image.

[0102] Robot 100 can recognize not only species such as humans and cats, but also individual animals, distinguishing between "who" and "which." Even cats are treated differently; black cats and white cats, large cats and small cats, are treated as separate animals. Furthermore, Robot 100 learns cat names based on user interactions with the cats. For example, a machine learning model could be generated that extracts a cat's image from the cat's name recognized through voice analysis and the image captured at the time the name was recognized, and then outputs the cat's name using the image as input. Using such a model, even when there are multiple cats, the user can specify the cat's name and command Robot 100 to select it as the target for photography. Users may also pre-register cat names and photos via a smartphone or other device.

[0103] When a user remotely controls robot 100A, robot 100B may move along with robot 100A. Images captured by robot 100A and images captured by robot 100B are transmitted to the user's smartphone. When robot 100A captures an image of a cat, robot 100B, which is near robot 100A, also captures an image of the cat using its panoramic camera 113. By remotely controlling robot 100A, the user can obtain images of the cat not only from robot 100A but also from robot 100B. By remotely controlling only robot 100A, the user can also indirectly remotely control robot 100B. This is because robot 100B has a "follow function".

[0104] The user can set robot 100 to remote control mode from their smartphone. The user can also exit remote control mode from their smartphone. While in remote control mode, robot 100 may change the display of its eyes 110. For example, robot 100 may change its eyes 110 to red, or it may visually represent being "controlled (remotely controlled)" by displaying an icon on its eyes 110. When remote control mode is exited, robot 100 returns its eyes 110 to their normal black display and returns to the location where it was located when remote control mode was started. When remote control mode is exited, robot 100 may sit down, or it may shake its head violently to express that it has "escaped control and regained its self-awareness."

[0105] Robot 100 in remote mode does not change emotional parameters or affinity.

[0106] When switching to remote control mode, robot 100 authenticates the person requesting remote control. Only if authentication is successful will robot 100 switch to remote control mode. Authentication can be done using a common method such as an account name and password, or by pre-registering an electronic certificate on the device used for remote control and only allowing access from devices with the electronic certificate. Furthermore, by using the camera and microphone on a mobile device such as a smartphone, it can be verified that the person operating the mobile device is the owner of robot 100. Authentication may be performed by confirming that the user is the owner. Alternatively, when remote mode is requested, users nearby may be required to authorize the switch to remote mode. In this way, by thoroughly confirming that the person requesting remote mode is the owner of robot 100, unintended remote control by a third party can be prevented.

[0107] <Monitoring the elderly> Figures 15 to 18 are schematic diagrams illustrating the behavioral scenarios when multiple robots 100 monitor an elderly person. An elderly father lives alone. His only daughter lives separately from him. There are two robots, 100, in the father's house (Figure 15A). In the living room at home, the daughter is looking at her smartphone (Figure 15B). The 100 robots record their life with their father as a life log. The life log is a diary that shows what is happening around the father in a way that respects his privacy. The daughter checks her life log on her smartphone (Figure 15C). The life log includes simple information such as what time her father woke up and whether he ate breakfast. Meanwhile, the father is holding and doting on robot 100 (Figure 15D). If the father's permission is recognized through image analysis, voice analysis, or communication, robot 100 may send images of itself playing with the father to the daughter's smartphone. For example, when robot 100A is being held by the father, robot 100B may act as a cameraman, taking images of robot 100A and the father, and sending those images to the daughter's smartphone.

[0108] The daughter is relieved to see her father enjoying life with Robot 100 in the living room (Figure 16A).

[0109] Next, let's imagine a scene where the daughter is working in the office. She suddenly takes out her smartphone and checks her father's life log (Figure 16B). Let's say this life log contains very little information about the father. The daughter suddenly becomes worried about her father (Figure 16C). The daughter calls her father from the office hallway (Figure 16D).

[0110] The father's cheerful voice could be heard immediately over the phone (Figure 17A). My father is at the inn. Apparently, he received a phone call just as he was getting out of the bath (Figure 17B). The father tells his daughter that he was at a hot spring resort with his friends (Figure 17C). The daughter was unaware that her father was going to the hot springs, so she was relieved to learn the reason (Figure 17D).

[0111] The conversation between father and daughter continues (Figures 18A and 18B). Two robots, number 100, are left at home while the father is out (Figure 18C).

[0112] Robot 100 records various events that occur in daily life with the father (the elderly person being monitored) as a life log. This life log records the father's routine daily activities, such as the time he woke up and whether he did his usual exercises. The daughter can check through the life log provided by Robot 100 whether her father is living his life as usual.

[0113] If no events indicating interaction between the father and robot 100 occur (are not detected) for a predetermined period of time, robot 100 may send an anomaly notification to the daughter's smartphone. For example, robot 100 may send an anomaly notification if it hasn't been touched by the father for a while, or if the father is still lying down at lunchtime. Whether the father is lying down can be determined by image analysis or temperature sensor analysis. Robot 100 may also act to increase opportunities for the father to see it by actively moving around the room.

[0114] By using an abstracted life log, the daughter can monitor her father's daily life while protecting his privacy. When monitoring elderly individuals, Robot 100 does not need to constantly see them. Elderly individuals live independent lives, and Robot 100 should basically act autonomously. It is preferable for the elderly and Robot 100 to maintain an appropriate distance. Robot 100 may record life logs at any time the user wishes, not just during monitoring. Robot 100 only needs to notify the daughter of any changes in the elderly person's life.

[0115] <Expressions of jealousy> People cannot remain indifferent to the affection directed towards them. When multiple people show affection towards the same person, jealousy is likely to arise. Therefore, when robots 100A and 100B live with the user, one robot 100 may take actions that make the user feel jealous of the other robot 100.

[0116] For example, when robot 100B is being held by the user, the state management unit 244 increases the approval need value (desire to be recognized), which is a type of emotional parameter of robot 100A. When the approval need value of robot 100A increases, the motion control unit 150 of robot 100A begs the user to hold it. Robot 100A may stare at the user, approach the user, or wander around the user to ask to be held. When the user walks, robot 100A may follow the user. The increase in the approval need value is externally expressed as behavioral characteristics of robot 100 that appear as if it has been aroused by jealousy.

[0117] When the user continues to hold robot 100B, robot 100A may actively express "strong jealousy" by clinging to the user. Alternatively, robot 100A may passively express jealousy by moving to a position away from the user and looking at them from a distance. The manner in which jealousy is expressed is determined according to the individuality of each robot 100 (initialized personality or cultivated personality). Jealousy may also be expressed by "sulking," and for a certain period after an event that causes jealousy occurs, robot 100 may exhibit behaviors such as moving away when the user approaches. Robot 100 may also express "sulking" by temporarily refusing to be held.

[0118] Robot 100 may exhibit jealousy-like behavior the higher its level of intimacy with the user. For example, suppose robot 100A has a high level of intimacy with user P1 and a relatively low level of intimacy with user P2. In this case, when user P1 holds robot 100B, robot 100A may exhibit a higher level of need for approval than when user P2 holds robot 100B. This control method allows for behavioral expressions that suggest possessiveness, such as wanting to monopolize the affection of a user it particularly likes.

[0119] Robot 100 may notify other robots 100 of its own state (emotional parameters, intimacy level, events, etc.) (hereinafter, such notifications will be referred to as "state notifications"). Based on state notifications, robots 100 may be able to understand each other's states. For example, by notifying robot 100A of its state, such as "being held by the user," "being petted by the user," or "being dressed by the user," robot 100B can understand robot 100A's state. When robot 100A's need for approval (desire to be recognized) decreases due to an event such as being held, while robot 100B's need for approval remains high above a threshold, robot 100B will exhibit specific behavioral characteristics that express jealousy.

[0120] In this embodiment, the state management unit 244 of the server 200 manages the emotional parameters of each robot 100 collectively. In this case, the state management unit 244 may internally notify robot 100B of the value of robot 100A's emotional parameters, or it may change robot 100B's emotional parameters based on robot 100A's emotional parameters. In this case, robot 100B's emotional parameters may be changed on the condition that robot 100A is in a position visible to robot 100B. This is to represent robot 100B, which is near robot 100A, visually sensing the change in robot 100A's emotions and changing its own emotional parameters.

[0121] In addition to emotion parameters, robot 100A may also notify robot 100B via short-range wireless communication such as infrared. In this case, robot 100B can only receive status notifications from robot 100A when it is near robot 100A and there are no obstacles obstructing its line of sight, thus representing the idea that "its state can only be perceived when it is close enough to see." Furthermore, robot 100B may detect events such as robot 100A being held or petted based on captured images. When robot 100B recognizes a pleasant action towards robot 100A via image recognition, it may change its emotion parameters.

[0122] Robot 100 may not only be jealous of other Robot 100s, but also of pets and children. For example, when a user holds a cat, Robot 100's need for approval may increase. The user may also need to be considerate, such as showing affection to their pet when Robot 100 is not looking, or showing affection equally to both the pet and Robot 100, in order to avoid making Robot 100 jealous. By actively creating opportunities for the user to consider Robot 100's feelings, the user's attachment to Robot 100 can be deepened.

[0123] In this embodiment, robot 100 can express its feelings through its actions without engaging in conversation. Robot 100A may receive the feelings (emotional parameters) of robot 100B. Server 200 may reflect changes in robot 100B's emotional parameters in robot 100A's actions. For example, when robot 100B's need for approval drops sharply (when it is thought that something good has happened to robot 100B), robot 100A may move closer to robot 100B. When robot 100A is notified that robot 100B's need for approval has been met, it may increase its own need for approval (the desire to be recognized itself). With this control method, even when a user secretly shows affection to robot 100B, robot 100A can exhibit behavioral expressions as if it has sensed something. In other words, it can achieve mysterious behavioral expressions as if robots 100A are communicating telepathically with each other.

[0124] <Multiple robots stare at the same thing> Robots 100A and 100B may continue to gaze at the same object. For example, when robot 100A gazes at a user who is relaxing, robot 100B may also gaze at the same user. Robot 100B may detect that robot 100A is gazing at the user who is relaxing (i.e., robot 100A's head is facing the direction of the user) through communication with robot 100A or through image analysis. Robot B may move closer to robot A before gazing at the user. Because the user perceives multiple gazes, they can feel that the robots 100s are very interested in them. If the user has not interacted with the robots 100s for a long period of time, robots 100A and 100B may silently seek "interaction" by gazing at the user simultaneously.

[0125] Assume that robot 100A has a high level of intimacy with user P1, and robot 100B also has a high level of intimacy with user P1, above a predetermined value. Robot 100 has more opportunities to gaze at users with higher levels of intimacy. Therefore, in the above situation, there will be an opportunity for robot 100A and robot 100B to gaze at user P1 at the same time by chance. Robot 100A may notify robot 100B that it is gazing at user P1. When robot 100B receives a status notification from robot 100A that "(robot 100A is also) gazing at user P1" while it is gazing at user P1, it may perform a specific motion, such as a surprised motion or turning its gaze in the direction of robot 100A, to create the illusion of a "coincidence." Also, if both are gazing at the same user, robot 100A and robot 100B may move closer to each other and perform a motion so that they gaze at user P1 side by side. By having both robots 100 move to a position where the user's face is visible as clearly as possible and gazing at the user side by side, a strong sense of pressure can be exerted on the user.

[0126] Furthermore, when an insect enters the house (when the insect is detected inside the house through image analysis or sound analysis, etc.), robots 100A and 100B may share the insect and simultaneously gaze at it, expressing an unusual interest in the insect through their actions. In addition, by gazing at each other, robots 100A and 100B can express actions that suggest they are communicating something to each other.

[0127] When the value of the emotion parameter indicating robot 100A's curiosity exceeds a threshold, robot 100A may notify robot 100B that it is "highly curious." At this time, robot 100B may approach robot 100A and perform motions that suggest it wants to know the source of robot 100A's curiosity, such as moving its hands to touch robot 100A. Robot 100A may also notify robot 100B of the object of interest and its direction in the panoramic image. Upon receiving this notification, robot 100B may turn its head and gaze towards the same object as robot 100A and look at the same object.

[0128] <Appeal> When a predetermined appeal condition is met, for example, when the need for recognition exceeds a threshold, robot 100 will perform a strong appeal action towards the user. Appeal actions here refer to actions that actively seek interaction from the user to robot 100, such as touching, talking to, or hugging. For example, suppose the user is doing exercise such as yoga indoors. When the robot 100's appeal condition is met while the user is engrossed in yoga, robot 100 may perform appeal actions such as staring at the user or wandering around the user to ask them to stop yoga.

[0129] <Following behavior> As described above, when robot 100A moves, robot 100B may follow behind robot 100A, maintaining a constant distance from robot 100A. Robot 100A may similarly follow a user or a pet. For example, when a dog is following a user, robot 100A may follow the dog or the user. Robot 100B may follow robot 100A when robot 100A is following a dog or the like. The distance between the robot and the object being followed (for example, a dog following a user) may be equal to the distance between the object being followed and the object being followed by the object being followed (for example, the user), or it may be shorter by a predetermined length than that distance, or it may be longer by a predetermined length than that distance.

[0130] Robot 100 determines that "following" has occurred when it detects that moving object Q1 and moving object Q2 are moving in the same direction for a predetermined period of time or longer. With this control method, when following occurs, it becomes possible to express behavior that evokes the robot 100's instinct to want to follow. The sight of multiple robots 100 following each other is considered effective in appealing to users with the cuteness of robot 100.

[0131] Robot 100 may perform follow-up behavior on the condition that the value of its emotional parameter, for example, the emotional parameter indicating curiosity, is below a threshold. With such a control method, it is possible to express behavior in which the robot performs follow-up behavior towards other robots when its curiosity has waned and it is bored, and does not perform follow-up behavior when its curiosity has increased. When follow-up behavior is performed, the follow-up behavior may be terminated on the condition that curiosity has risen above a threshold due to various events. The source of the behavior of an autonomous robot is predetermined parameters that indicate its internal state. In this embodiment, the parameter indicating curiosity contributes greatly to the source of behavior, but if there is little change in the external environment, the curiosity parameter may approach 0. In such cases, instead of waiting for its own parameters to change, it can actively change its own parameters by piggybacking on the behavior of other robots.

[0132] As described above, multiple robots 100 may perform the same actions, such as following each other, or their behavioral characteristics may change as they are influenced by each other's actions. To enhance the cooperation and coordination of multiple robots 100, it is desirable that they be able to understand each other's states within the server 200 or among themselves. Robot 100B, having understood the state of robot 100A, may act in synchronization with robot 100A's state, or it may act independently without synchronization. An example of robot 100B acting in synchronization with robot 100A is when robot 100B looks at the same object that robot 100A is looking at, or when robot 100B performs a motion of the same category on the same object as robot 100A.

[0133] When multiple robots 100 perform coordinated actions, users are likely to find the robots 100 endearing. Users may want to take pictures of the robots 100 performing these coordinated actions. When the panoramic camera 113 recognizes that a user is holding the camera, robot 100 may maintain at least one of its actions and / or state until the user finishes taking the picture. Alternatively, instead of maintaining an action or state, robot 100 may select a specific motion. For example, robot 100 may turn its body towards the user, or it may temporarily suspend its coordinated actions to assist the user in taking the picture. Thus, robot 100 may temporarily stop its actions when it detects a user taking a picture. Furthermore, robot 100 may strike a pose or temporarily stop its actions not only during coordinated actions, but also when it detects a user taking a picture. This control method makes it easier for users to upload pictures of the robots 100 in their cute state to social networking services (SNS), etc. Furthermore, it is expected that this will make it easier to capture various best shots of the Robot 100 interacting with various things in the home (pets, children, toys, furniture, etc.).

[0134] <Structure of the outer layer> The outer shell 314 of the robot 100 is constructed by housing a stretchable base material in a cloth bag. The bag may be made of a flexible material that is warm and pleasant to the touch for the user. The base material is preferably a flame-retardant material, and more preferably a material that releases self-extinguishing gas when heated to high temperatures. For example, the base material is made of flame-retardant sponge. Since the outer shell 314 is formed by enclosing the flame-retardant base material in a cloth bag, even if the cloth bag ignites, the base material releases self-extinguishing gas, thus preventing the cloth bag from burning. The threshold temperature for generating the self-extinguishing gas is preferably lower than the ignition temperature of the fabric. In this case, when the fabric becomes hot, the self-extinguishing gas is generated before the fabric ignites, thus preventing the fabric from igniting. By making the outer shell 314 a double structure of a flame-retardant base material and a flexible bag, it is possible to achieve both a warm feel for the robot 100 and safety against high temperatures.

[0135] It should be noted that the present invention is not limited to the embodiments and modifications described above, and the components can be modified and implemented without departing from the spirit of the invention. Various inventions may be formed by appropriately combining the multiple components disclosed in the embodiments and modifications described above. In addition, some components may be deleted from all the components shown in the embodiments and modifications described above.

[0136] Although the robot system 300 is described as being composed of one or more robots 100 and one server 200, some of the functions of the robots 100 may be implemented by the server 200, or some or all of the functions of the server 200 may be assigned to the robots 100. One server 200 may control multiple robots 100, or multiple servers 200 may cooperate to control one or more robots 100.

[0137] A third device other than robot 100 or server 200 may perform some of the functions. The collection of functions of robot 100 and server 200 described in Figure 4 can also be viewed as a single "robot" in a broader sense. How to distribute the multiple functions necessary to realize the present invention to one or more hardware devices should be determined in consideration of the processing capacity of each hardware device and the specifications required for the robot system 300.

[0138] As mentioned above, "robot in the narrow sense" refers to robot 100 excluding server 200, while "robot in the broad sense" refers to robot system 300. Many of the functions of server 200 may be integrated into robot 100 in the future.

Claims

1. A motion control unit that selects the robot's motion, A drive mechanism that executes the motion selected by the motion control unit, A recognition unit that determines whether the subject meets the predetermined monitoring conditions, When the aforementioned monitoring conditions are met, the mode setting unit sets the subject to the monitoring mode. Equipped with, In addition to the monitoring conditions, the mode setting unit also considers the presence of a person associated with the subject in the vicinity. The system is configured to set to the monitoring mode when the condition of not being detected is met. A robot characterized by having [this feature].

2. In the aforementioned monitoring mode, the robot shares the location of the subject with other robots and based on the subject The same gaze point as the other robot or the gaze point of the other robot determined accordingly The robot according to claim 1, which turns its head toward a point of focus within a fixed distance.

3. In the aforementioned monitoring mode, at least one of the robots and itself does not touch the target's head. The robot according to claim 1 or 2, configured to face a certain direction.

4. During the monitoring mode, the distance to the subject is controlled to be within a predetermined distance, any of claims 1 to 3 Any robot as described in item 1.

5. During the aforementioned monitoring mode, at least one mode is used when the aforementioned monitoring conditions are not met. A robot according to any one of claims 1 to 4, which also reduces the amount of operation of the drive mechanism.

6. During the monitoring mode, when it is determined that the subject has met the predetermined conditions, the subject The robot according to any one of items 1 to 5, characterized in that it performs motion toward a person .

7. In the monitoring mode, a communication unit transmits the captured image of the subject to a predetermined communication terminal. A robot according to any one of claims 1 to 6, characterized by being equipped with the following:

8. The mode setting unit, if no specific user, including the parent of the target person, is detected, The robot according to any one of claims 1 to 7, which is configured to be set to the aforementioned monitoring mode.

Images