robot

Abstract

Providing users with a sense of security when living with a robot. The autonomously acting robot includes an operation control unit that selects the motion of the robot, a drive mechanism that executes the motion selected by the operation control unit, a recognition unit that determines whether the target object satisfies predetermined monitoring conditions, a mode setting unit that sets the target object to a monitoring mode when the monitoring conditions are met, and a communication unit that, in the monitoring mode, transmits a captured image of the target object to a predetermined communication terminal.

Description

[Technical field]

[0001] The present invention relates to a robot that autonomously selects an action depending on an internal state or an external environment. [Background technology]

[0002] Humans keep pets to find comfort. However, many people give up on having pets for various reasons, such as not having enough time to look after them, not having a living environment that allows them to keep a pet, allergies, or the pain of losing a pet. If there were a robot that could play the role of a pet, it might be able to provide the same comfort that pets provide to people who cannot keep pets (see Patent Documents 1 and 2). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2000-323219 A [Patent Document 2] International Publication No. 2017 / 169826 Summary of the Invention [Problem to be solved by the invention]

[0004] In recent years, robot technology has been rapidly advancing, but we have not yet achieved the ability to create a robot that can act as a companion like a pet. This is because we do not believe that a robot has free will. By observing the behavior of our pets, which we can only assume to have free will, we can sense the presence of free will in our pets, empathize with them, and feel comforted by them.

[0005] The present invention was completed based on the above problem recognition, and its first objective is to provide technology that provides various feelings of security to users living with robots. The second objective is to provide technology that enables a robot to express affection for its user in a rich way. The third objective is to provide technology that enables multiple robots to act in cooperation with each other. [Means for solving the problem]

[0006] An autonomously acting robot in one aspect of the present invention includes a motion control unit that selects a motion of the robot, a drive mechanism that executes the motion selected by the motion control unit, a recognition unit that determines whether a subject satisfies specified monitoring conditions, a mode setting unit that sets the subject to a monitoring mode when the monitoring conditions are met, and a communication unit that, in the monitoring mode, transmits a captured image of the subject to a specified communication terminal. Effect of the Invention

[0007] According to the present invention, it becomes easier to further enhance the presence of the robot. [Brief description of the drawings]

[0008] The above objects, as well as other objects, features and advantages, will become more apparent from the following preferred embodiments and the accompanying drawings.

[0009] [Figure 1A] FIG. [Figure 1B] FIG. [Diagram 2] FIG. 2 is a cross-sectional view that illustrates a schematic structure of the robot. [Diagram 3] FIG. 2 is a diagram showing the hardware configuration of the robot in the basic configuration. [Figure 4] FIG. 2 is a functional block diagram of the robot system in a basic configuration. [Figure 5A] FIG. 13 is a schematic diagram for explaining a behavioral scene when a plurality of robots are watching over a baby. [Figure 5B] FIG. 13 is a schematic diagram for explaining a behavioral scene when a plurality of robots are watching over a baby. [Figure 5C] FIG. 13 is a schematic diagram for explaining a behavioral scene when a plurality of robots are watching over a baby. [Figure 5D] FIG. 13 is a schematic diagram for explaining a behavioral scene when a plurality of robots are watching over a baby. [Figure 6A] FIG. 13 is a schematic diagram for explaining a behavioral scene when a plurality of robots are watching over a baby. [Figure 6B] FIG. 13 is a schematic diagram for explaining a behavioral scene when a plurality of robots are watching over a baby. [Figure 6C] FIG. 13 is a schematic diagram for explaining a behavioral scene when a plurality of robots are watching over a baby. [Figure 6D] FIG. 13 is a schematic diagram for explaining a behavioral scene when a plurality of robots are watching over a baby. [Figure 7A] FIG. 13 is a schematic diagram for explaining a behavioral scene when a plurality of robots are watching over a baby. [Figure 7B] FIG. 13 is a schematic diagram for explaining a behavioral scene when a plurality of robots are watching over a baby. [Figure 7C] FIG. 13 is a schematic diagram for explaining a behavioral scene when a plurality of robots are watching over a baby. [Figure 7D] FIG. 13 is a schematic diagram for explaining a behavioral scene when a plurality of robots are watching over a baby. [Figure 7E] FIG. 13 is a schematic diagram for explaining a behavioral scene when a plurality of robots are watching over a baby. [Figure 8A] 1A and 1B are schematic diagrams for explaining behavioral scenes of a robot when the robot is left at home alone. [Figure 8B] 1A and 1B are schematic diagrams for explaining behavioral scenes of a robot when the robot is left at home alone. [Figure 8C] 1A and 1B are schematic diagrams for explaining behavioral scenes of a robot when the robot is left at home alone. [Figure 8D]1A and 1B are schematic diagrams for explaining behavioral scenes of a robot when the robot is left at home alone. [Figure 9A] 1A and 1B are schematic diagrams for explaining behavioral scenes of a robot when the robot is left at home alone. [Figure 9B] 1A and 1B are schematic diagrams for explaining behavioral scenes of a robot when the robot is left at home alone. [Figure 9C] 1A and 1B are schematic diagrams for explaining behavioral scenes of a robot when the robot is left at home alone. [Figure 9D] 1A and 1B are schematic diagrams for explaining behavioral scenes of a robot when the robot is left at home alone. [Figure 10A] 1A and 1B are schematic diagrams for explaining behavioral scenes of a robot when the robot is left at home alone. [Figure 10B] 1A and 1B are schematic diagrams for explaining behavioral scenes of a robot when the robot is left at home alone. [Figure 10C] 1A and 1B are schematic diagrams for explaining behavioral scenes of a robot when the robot is left at home alone. [Figure 11A] 1A and 1B are schematic diagrams for explaining behavioral scenes of a robot when the robot is left at home alone. [Figure 11B] 1A and 1B are schematic diagrams for explaining behavioral scenes of a robot when the robot is left at home alone. [Figure 11C] 1A and 1B are schematic diagrams for explaining behavioral scenes of a robot when the robot is left at home alone. [Figure 12A] FIG. 1 is a schematic diagram for explaining a behavior scene when a user who is out remotely controls a robot. [Figure 12B] FIG. 1 is a schematic diagram for explaining a behavior scene when a user who is out remotely controls a robot. [Figure 12C] FIG. 1 is a schematic diagram for explaining a behavior scene when a user who is out remotely controls a robot. [Figure 12D] FIG. 1 is a schematic diagram for explaining a behavior scene when a user who is out remotely controls a robot. [Figure 13A] FIG. 1 is a schematic diagram for explaining a behavior scene when a user who is out remotely controls a robot. [Figure 13B] FIG. 1 is a schematic diagram for explaining a behavior scene when a user who is out remotely controls a robot. [Figure 13C] FIG. 1 is a schematic diagram for explaining a behavior scene when a user who is out remotely controls a robot. [Figure 13D] FIG. 1 is a schematic diagram for explaining a behavior scene when a user who is out remotely controls a robot. [Figure 14A] FIG. 1 is a schematic diagram for explaining a behavior scene when a user who is out remotely controls a robot. [Figure 14B] FIG. 1 is a schematic diagram for explaining a behavior scene when a user who is out remotely controls a robot. [Figure 14C] FIG. 1 is a schematic diagram for explaining a behavior scene when a user who is out remotely controls a robot. [Figure 14D] FIG. 1 is a schematic diagram for explaining a behavior scene when a user who is out remotely controls a robot. [Figure 15A] FIG. 13 is a schematic diagram for explaining a behavioral scene when a plurality of robots are watching over an elderly person. [Figure 15B] FIG. 13 is a schematic diagram for explaining a behavioral scene when a plurality of robots are watching over an elderly person. [Figure 15C] FIG. 13 is a schematic diagram for explaining a behavioral scene when a plurality of robots are watching over an elderly person. [Figure 15D] FIG. 13 is a schematic diagram for explaining a behavioral scene when a plurality of robots are watching over an elderly person. [Figure 16A] FIG. 13 is a schematic diagram for explaining a behavioral scene when a plurality of robots are watching over an elderly person. [Figure 16B] FIG. 13 is a schematic diagram for explaining a behavioral scene when a plurality of robots are watching over an elderly person. [Figure 16C] FIG. 13 is a schematic diagram for explaining a behavioral scene when a plurality of robots are watching over an elderly person. [Figure 16D] FIG. 13 is a schematic diagram for explaining a behavioral scene when a plurality of robots are watching over an elderly person. [Figure 17A]FIG. 13 is a schematic diagram for explaining a behavioral scene when a plurality of robots are watching over an elderly person. [Figure 17B] FIG. 13 is a schematic diagram for explaining a behavioral scene when a plurality of robots are watching over an elderly person. [Figure 17C] FIG. 13 is a schematic diagram for explaining a behavioral scene when a plurality of robots are watching over an elderly person. [Figure 17D] FIG. 13 is a schematic diagram for explaining a behavioral scene when a plurality of robots are watching over an elderly person. [Figure 18A] FIG. 13 is a schematic diagram for explaining a behavioral scene when a plurality of robots are watching over an elderly person. [Figure 18B] FIG. 13 is a schematic diagram for explaining a behavioral scene when a plurality of robots are watching over an elderly person. [Figure 18C] FIG. 13 is a schematic diagram for explaining a behavioral scene when a plurality of robots are watching over an elderly person. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] The robot 100 in this embodiment shares daily life with the user, sometimes being considerate of the user, sometimes striving to be helpful to the user, and actively seeking the user's affection, thereby exerting its presence as a member of the family. Below, the basic configuration of the robot 100 will be described with reference to Figs. 1 to 4, and then various behavioral scenes of the robot 100 will be described.

[0011] [Basic configuration] Fig. 1 is a diagram showing the external appearance of a robot 100. Fig. 1A is a front view, and Fig. 1B is a side view. The robot 100 is an autonomous robot that determines its behavior based on the external environment and its internal state. The external environment is recognized by various sensors such as a camera and a thermosensor 115. The internal state is quantified as various parameters that express the emotions of the robot 100. The robot 100's range of movement is within the interior of the owner's home. Hereinafter, a human being related to the robot 100 is referred to as a "user." Among the users, the owner or manager of the robot 100 is referred to as the "owner."

[0012] The body 104 of the robot 100 has an overall rounded shape, and includes an outer skin 314 formed of a soft and elastic material such as urethane, rubber, resin, or fiber. The robot 100 may be dressed in clothing. The total weight of the robot 100 is about 5 to 15 kilograms, and the height is about 0.5 to 1.2 meters. Due to various attributes such as appropriate weight, roundness, softness, and pleasant feel, the effect of making the user easy to hold the robot 100 and wanting to hold it is realized.

[0013] The robot 100 includes a pair of front wheels 102 (left wheel 102a, right wheel 102b) and one rear wheel 103. The front wheels 102 are driving wheels, and the rear wheels 103 are driven wheels. The front wheels 102 do not have a steering mechanism, but the rotation speed and rotation direction of the left and right wheels can be controlled individually. The rear wheels 103 are casters and can rotate freely to move the robot 100 forward, backward, left and right. The rear wheels 103 may be omniwheels. By making the rotation speed of the right wheel 102b higher than that of the left wheel 102a, the robot 100 can turn left or rotate counterclockwise. By making the rotation speed of the left wheel 102a higher than that of the right wheel 102b, the robot 100 can turn right or rotate clockwise.

[0014] The front wheel 102 and the rear wheel 103 can be completely stored in the body 104 by a drive mechanism (rotation mechanism, link mechanism). A pair of left and right covers 312 are provided on the lower half of the body 104. The covers 312 are made of a flexible and elastic resin material (rubber, silicone rubber, etc.), which forms a soft body and can store the front wheel 102. A slit 313 (opening) that opens from the side to the front is formed in the cover 312, and the front wheel 102 can be advanced through the slit 313 and exposed to the outside.

[0015] Even when the robot 100 is moving, most of the wheels are hidden by the body 104, but when the wheels are completely retracted into the body 104, the robot 100 is unable to move. That is, as the wheels are retracted, the body 104 descends and sits on the floor F. In this seated state, a flat seating surface 108 (ground contact surface) formed on the bottom of the body 104 comes into contact with the floor F.

[0016] The robot 100 has two arms 106. There are hands at the ends of the arms 106, but they do not have the function of grasping objects. The arms 106 can perform simple movements such as lifting, bending, waving, and vibrating when driven by actuators described below. The two arms 106 can be controlled individually.

[0017] A face region 116 is exposed on the front of the head of the robot 100. Two eyes 110 are provided in the face region 116. The eyes 110 are capable of displaying images using liquid crystal elements or organic EL elements, and are devices for expressing gaze and facial expressions by moving the pupils and eyelids displayed as images. A nose 109 is provided in the center of the face region 116. An analog stick is provided on the nose 109, and it can detect not only all directions, up, down, left, and right, but also the direction of pressing. In addition, the robot 100 is provided with multiple touch sensors, and can detect the touch of the user on almost the entire area of ​​the robot 100, such as the head, torso, buttocks, and arms. The robot 100 is equipped with various sensors, such as a microphone array and an ultrasonic sensor, which can identify the direction of a sound source. In addition, it has a built-in speaker and can emit simple sounds.

[0018] A horn 112 is attached to the head of the robot 100. A panoramic camera 113 is attached to the horn 112, and the entire upper area of ​​the robot 100 can be imaged at once. The horn 112 also has a built-in thermosensor 115 (thermocamera). The horn 112 is also provided with a plurality of modules (not shown) for communication using infrared rays, and these modules are installed in a ring shape facing the surroundings. This allows the robot 100 to communicate by infrared rays while recognizing its direction. Furthermore, the horn 112 is provided with an emergency stop switch, and the user can bring the robot 100 to an emergency stop by pulling out the horn 112.

[0019] FIG. 2 is a cross-sectional view that illustrates a schematic structure of the robot 100. As shown in FIG. The body 104 includes a main body frame 310, a pair of arms 106, a pair of covers 312, and an outer skin 314. The main body frame 310 includes a head frame 316 and a torso frame 318. The head frame 316 is hollow and hemispherical, and forms the head skeleton of the robot 100. The torso frame 318 is rectangular and tubular, and forms the torso skeleton of the robot 100. The lower end of the torso frame 318 is fixed to a lower plate 334. The head frame 316 is connected to the torso frame 318 via a connection mechanism 330.

[0020] The trunk frame 318 forms the axis of the body 104. The trunk frame 318 is configured by fixing a pair of left and right side plates 336 to a lower plate 334, and supports the pair of arms 106 and the internal mechanisms. The battery 118, a control circuit 342, various actuators, etc. are housed inside the trunk frame 318. The bottom surface of the lower plate 334 forms the seating surface 108.

[0021] The trunk frame 318 has an upper plate 332 at its upper portion. A cylindrical support portion 319 with a bottom is fixed to the upper plate 332. The upper plate 332, the lower plate 334, the pair of side plates 336, and the support portion 319 constitute the trunk frame 318. The outer diameter of the support portion 319 is smaller than the distance between the left and right side plates 336. The pair of arms 106 are integrally assembled with the ring-shaped member 340 to constitute the arm unit 350. The ring-shaped member 340 has a circular ring shape, and the pair of arms 106 are attached to the center line of the ring-shaped member 340 so as to be spaced apart in the radial direction. The ring-shaped member 340 is inserted coaxially through the support portion 319 and placed on the upper end surfaces of the pair of side plates 336. The arm unit 350 is supported from below by the trunk frame 318.

[0022] The head frame 316 has a yaw axis 321, a pitch axis 322, and a roll axis 323. The head frame 316 rotates (yawing) about the yaw axis 321 to realize a swinging motion, rotates (pitching) about the pitch axis 322 to realize a nodding motion, a motion to look up, and a motion to look down, and rotates (rolling) about the roll axis 323 to realize a motion to tilt the head left and right. The position and angle of each axis can change in three-dimensional space according to the driving mode of the connection mechanism 330. The connection mechanism 330 is composed of a link mechanism, and is driven by multiple motors installed in the body frame 318.

[0023] The body frame 318 houses a wheel drive mechanism 370. The wheel drive mechanism 370 includes a front wheel drive mechanism and a rear wheel drive mechanism that respectively move the front wheels 102 and the rear wheels 103 in and out of the body 104. The front wheels 102 and the rear wheels 103 function as a "movement mechanism" that moves the robot 100. The front wheel 102 has a direct drive motor at its center. This allows the left wheel 102a and the right wheel 102b to be driven separately. The front wheel 102 is rotatably supported by a wheel cover 105, which is rotatably supported by the body frame 318.

[0024] The pair of covers 312 are provided to cover the left and right sides of the trunk frame 318, and have a smoothly curved shape to round the outline of the body 104. A closed space is formed between the trunk frame 318 and the cover 312, and this closed space serves as a storage space S for the front wheel 102. The rear wheel 103 is stored in a storage space provided at the lower rear of the trunk frame 318.

[0025] The outer skin 314 covers the main body frame 310 and the pair of arms 106 from the outside. The outer skin 314 has a thickness that allows a person to feel its elasticity, and is formed of a stretchy material such as urethane sponge. As a result, when the user hugs the robot 100, the user feels an appropriate softness, and can have natural skinship with the robot, as with a pet. The outer skin 314 is attached to the main body frame 310 in a manner that exposes the cover 312. An opening 390 is provided at the upper end of the outer skin 314. The horn 112 passes through this opening 390.

[0026] A touch sensor is disposed between the main body frame 310 and the outer skin 314. A touch sensor is embedded in the cover 312. These touch sensors are all capacitance sensors, and detect touches over almost the entire area of ​​the robot 100. The touch sensor may be embedded in the outer skin 314, or may be disposed inside the main body frame 310.

[0027] Arm 106 has a first joint 352 and a second joint 354, with arm 356 between the two joints and hand 358 at the end of second joint 354. First joint 352 corresponds to a shoulder joint, and second joint 354 corresponds to a wrist joint. A motor is provided at each joint to drive arm 356 and hand 358, respectively. The drive mechanism for driving arm 106 includes these motors and their drive circuits 344.

[0028] FIG. 3 is a diagram showing the hardware configuration of the robot 100. The robot 100 includes an internal sensor 128, a communication device 126, a memory device 124, a processor 122, a drive mechanism 120, and a battery 118. The drive mechanism 120 includes the above-mentioned connection mechanism 330 and wheel drive mechanism 370. The processor 122 and the memory device 124 are included in a control circuit 342. Each unit is connected to each other by a power line 130 and a signal line 132. The battery 118 supplies power to each unit via the power line 130. Each unit transmits and receives control signals via the signal line 132. The battery 118 is a lithium-ion secondary battery, and is the power source of the robot 100.

[0029] The internal sensor 128 is a collection of various sensors built into the robot 100. Specifically, the internal sensors 128 include a camera, a microphone array, a distance sensor (infrared sensor), a thermosensor 115, a touch sensor, an acceleration sensor, an air pressure sensor, and an odor sensor. The touch sensor corresponds to most of the area of ​​the body 104, and detects the user's touch based on a change in capacitance. The odor sensor is a known sensor that applies the principle that electrical resistance changes due to the adsorption of odor-causing molecules.

[0030] The communicator 126 is a communication module that performs wireless communication with various external devices. The storage device 124 is composed of a non-volatile memory and a volatile memory, and stores computer programs and various setting information. The processor 122 is a means for executing computer programs. The drive mechanism 120 includes a plurality of actuators. In addition, a display, a speaker, etc. are also mounted.

[0031] The driving mechanism 120 mainly controls the wheels and the head. The driving mechanism 120 can change the direction and speed of movement of the robot 100, as well as raise and lower the wheels. When the wheels are raised, they are completely stored in the body 104, and the robot 100 comes into contact with the floor surface F at the seating surface 108, and assumes a seated state. The driving mechanism 120 also controls the arms 106.

[0032] FIG. 4 is a functional block diagram of the robot system 300. The robot system 300 includes a robot 100, a server 200, and a plurality of external sensors 114. Each component of the robot 100 and the server 200 is realized by hardware including computing units such as a CPU (Central Processing Unit) and various coprocessors, storage devices such as memory and storage, and wired or wireless communication lines connecting them, and software stored in the storage devices and supplying processing instructions to the computing units. The computer program may be composed of a device driver, an operating system, various application programs located at higher layers than those, and a library that provides common functions to these programs. Each block described below shows a functional block rather than a hardware-based configuration. A part of the functions of the robot 100 may be realized by the server 200, and a part or all of the functions of the server 200 may be realized by the robot 100.

[0033] A plurality of external sensors 114 are installed in the house beforehand. The server 200 manages the external sensors 114 and provides the robot 100 with detection values ​​acquired by the external sensors 114 as necessary. The robot 100 determines basic behavior based on information obtained from the internal sensor 128 and the plurality of external sensors 114. The external sensors 114 are intended to reinforce the sensory organs of the robot 100, and the server 200 is intended to reinforce the processing ability of the robot 100. The communicator 126 of the robot 100 may periodically communicate with the server 200, and the server 200 may be responsible for processing to identify the position of the robot 100 using the external sensors 114 (see also Patent Document 2).

[0034] (Server 200) The server 200 includes a communication unit 204 , a data processing unit 202 , and a data storage unit 206 . The communication unit 204 is responsible for communication processing with the external sensor 114 and the robot 100. The data storage unit 206 stores various data. The data processing unit 202 executes various processes based on the data acquired by the communication unit 204 and the data stored in the data storage unit 206. The data processing unit 202 also functions as an interface between the communication unit 204 and the data storage unit 206.

[0035] The data store 206 includes a motion store 232 and a personal data store 218 . The robot 100 has a plurality of movement patterns (motions). Various motions are defined, such as shaking the arms 106, approaching the owner while meandering, and gazing at the owner with the head tilted.

[0036] The motion storage unit 232 stores "motion files" that define the control contents of the motion. Each motion is identified by a motion ID. The motion files are also downloaded to the motion storage unit 160 of the robot 100. Which motion is to be executed may be determined by the server 200 or by the robot 100.

[0037] Many of the motions of the robot 100 are configured as composite motions including a plurality of unit motions. For example, when the robot 100 approaches the owner, it may be expressed as a combination of a unit motion of turning toward the owner, a unit motion of approaching while raising hands, a unit motion of approaching while shaking the body, and a unit motion of sitting down while raising both hands. A combination of these four motions realizes a motion of "approaching the owner, raising hands halfway, and finally sitting down after shaking the body." In the motion file, the rotational angles and angular velocities of the actuators provided in the robot 100 are defined in association with a time axis. Various motions are expressed by controlling each actuator over time according to the motion file (actuator control information).

[0038] The transition time from one unit motion to the next is called an "interval." The interval can be defined according to the time required to change the unit motion and the content of the motion. The length of the interval can be adjusted. Hereinafter, settings related to behavior control of the robot 100, such as when and which motion to select, and adjustment of the output of each actuator to realize the motion, are collectively referred to as "behavior characteristics." The behavior characteristics of the robot 100 are defined by a motion selection algorithm, a motion selection probability, a motion file, etc.

[0039] The motion storage unit 232 stores motion files as well as a motion selection table that defines the motions to be executed when various events occur. In the motion selection table, one or more motions and their selection probabilities are associated with each event.

[0040] The personal data storage unit 218 stores information about the user. Specifically, it stores master information indicating the degree of intimacy with the user and the physical and behavioral characteristics of the user. It may also store other attribute information such as age and gender.

[0041] The robot 100 has an internal parameter called intimacy for each user. When the robot 100 recognizes an action that shows affection toward the user, such as picking up the robot 100 or talking to the user, the intimacy with the user becomes high. The intimacy with users who do not interact with the robot 100, users who are violent, and users who the robot 100 encounters infrequently becomes low.

[0042] The data processing unit 202 includes a position management unit 208 , a recognition unit 212 , a motion control unit 222 , an intimacy management unit 220 , and a state management unit 244 . The position management unit 208 specifies the position coordinates of the robot 100. The state management unit 244 manages various internal parameters such as various physical states such as the charging rate, internal temperature, and the processing load of the processor 122. The state management unit 244 also manages various emotion parameters indicating the emotions of the robot 100 (loneliness, curiosity, desire for recognition, etc.). These emotion parameters are constantly fluctuating. The movement destination of the robot 100 changes depending on the emotion parameters. For example, when the robot 100 feels increasingly lonely, it sets the location of the user as the movement destination.

[0043] Emotion parameters change over time. In addition, various emotion parameters also change depending on the interaction, which will be described later. For example, if the owner "holds" the robot, the emotion parameter indicating loneliness will decrease, and if the robot does not see the owner for a long period of time, the emotion parameter indicating loneliness will increase little by little.

[0044] The recognition unit 212 recognizes the external environment. The recognition of the external environment includes various recognitions such as recognition of weather and season based on temperature and humidity, recognition of shade (safety zone) based on the amount of light and temperature, etc. The recognition unit 156 of the robot 100 acquires various environmental information by the internal sensor 128, performs primary processing on the information, and transfers it to the recognition unit 212 of the server 200.

[0045] Specifically, the recognition unit 156 of the robot 100 extracts an image area corresponding to a moving object, particularly a person or an animal, from an image, and extracts a "feature vector" as a set of feature amounts indicating the physical and behavioral characteristics of the moving object from the extracted image area. A feature vector component (feature amount) is a numerical value that quantifies various physical and behavioral characteristics. For example, the width of a human eye is quantified in the range of 0 to 1 to form one feature vector component. A method for extracting a feature vector from a captured image of a person is an application of a known face recognition technology. The robot 100 transmits the feature vector to the server 200.

[0046] The recognition unit 212 of the server 200 compares the feature vector extracted from the image captured by the built-in camera of the robot 100 with the feature vectors of users (clusters) preregistered in the personal data storage unit 218 to determine which person the captured user corresponds to (user identification process). The recognition unit 212 also estimates the user's emotions by performing image recognition of the user's facial expression. The recognition unit 212 also performs user identification process on moving objects other than people, such as pet cats and dogs.

[0047] The recognition unit 212 recognizes various behaviors given to the robot 100 and classifies them into pleasant and unpleasant behaviors. The recognition unit 212 also recognizes the owner's behavior in response to the behavior of the robot 100 and classifies it into positive and negative reactions. A pleasant or unpleasant behavior is determined based on whether the user's behavior is pleasant or unpleasant for the living organism. For example, being held is a pleasant behavior for the robot 100, and being kicked is an unpleasant behavior for the robot 100. A positive or negative reaction is determined based on whether the user's behavior indicates a pleasant emotion or an unpleasant emotion. Being held is a positive reaction that indicates a pleasant emotion for the user, and being kicked is a negative reaction that indicates an unpleasant emotion for the user.

[0048] The motion control unit 222 of the server 200 cooperates with the motion control unit 150 of the robot 100 to determine the motion of the robot 100. The motion control unit 222 of the server 200 creates a movement target point and a movement route for the robot 100. The motion control unit 222 may create multiple movement routes and then select one of the movement routes.

[0049] The motion control unit 222 selects a motion of the robot 100 from a plurality of motions in the motion storage unit 232. A selection probability is associated with each motion for each situation. For example, a selection method is defined such that when the owner performs a pleasant action, motion A is executed with a probability of 20%, and when the temperature is 30 degrees or higher, motion B is executed with a probability of 5%.

[0050] The intimacy management unit 220 manages the intimacy for each user. As described above, the intimacy is registered as part of the personal data in the personal data storage unit 218. When a pleasant behavior is detected, the intimacy management unit 220 increases the intimacy with the owner. When an unpleasant behavior is detected, the intimacy decreases. In addition, the intimacy of an owner who has not been viewed for a long period of time gradually decreases.

[0051] (Robot 100) The robot 100 includes a communication unit 142 , a data processing unit 136 , a data storage unit 148 , an internal sensor 128 , and a drive mechanism 120 . The communication unit 142 corresponds to the communicator 126 (see FIG. 3) and is responsible for communication processing with the external sensor 114, the server 200, and other robots 100. The data storage unit 148 stores various data. The data storage unit 148 corresponds to the memory device 124 (see FIG. 3). The data processing unit 136 executes various processes based on the data acquired by the communication unit 142 and the data stored in the data storage unit 148. The data processing unit 136 corresponds to the processor 122 and the computer program executed by the processor 122. The data processing unit 136 also functions as an interface between the communication unit 142, the internal sensor 128, the drive mechanism 120, and the data storage unit 148.

[0052] The data store 148 includes a motion store 160 that defines various motions of the robot 100 . Various motion files are downloaded from the motion storage unit 232 of the server 200 to the motion storage unit 160 of the robot 100. The motions are identified by motion IDs. In order to express various motions such as sitting down with the front wheels 102 folded, lifting the arms 106, rotating the two front wheels 102 in reverse or by rotating only one of the front wheels 102 to make the robot 100 rotate, trembling by rotating the front wheels 102 with the front wheels 102 folded, stopping once and looking back when moving away from the user, etc., the operation timing, operation time, operation direction, etc. of various actuators (drive mechanisms 120) are defined in a time series in the motion files. Various data may also be downloaded to the data storage unit 148 from the personal data storage unit 218 .

[0053] The data processing unit 136 includes a recognition unit 156 and an operation control unit 150 . The motion control unit 150 of the robot 100 decides the motions of the robot 100 in cooperation with the motion control unit 222 of the server 200. Some motions may be decided by the server 200, and other motions may be decided by the robot 100. Alternatively, the robot 100 may decide the motions, but when the processing load of the robot 100 is high, the server 200 may decide the motions. The server 200 may decide base motions, and the robot 100 may decide additional motions. How the motion decision process is shared between the server 200 and the robot 100 may be designed according to the specifications of the robot system 300.

[0054] The motion control unit 150 of the robot 100 instructs the driving mechanism 120 to execute the selected motion. The driving mechanism 120 controls each actuator in accordance with the motion file.

[0055] The motion control unit 150 can execute a motion of lifting both arms 106 as a gesture of asking to be "held" when a user with high intimacy is nearby, and can also express a motion of refusing to be held by alternately rotating in the opposite directions and stopping while keeping the left and right front wheels 102 retracted when the user gets tired of being "held." The drive mechanism 120 drives the front wheels 102, arms 106, and neck (head frame 316) according to instructions from the motion control unit 150, causing the robot 100 to express various motions.

[0056] The recognition unit 156 of the robot 100 interprets external information obtained from the internal sensor 128. The recognition unit 156 is capable of visual recognition (visual unit), smell recognition (olfactory unit), sound recognition (auditory unit), and tactile recognition (tactile unit).

[0057] The recognition unit 156 extracts a feature vector from a captured image of a moving object. As described above, the feature vector is a set of parameters (feature amounts) indicating the physical and behavioral features of a moving object. When a moving object is detected, the physical and behavioral features are also extracted from an odor sensor, a built-in sound collecting microphone, a temperature sensor, and the like. These features are also quantified to become feature vector components. The recognition unit 156 identifies the user from the feature vector based on known technology described in Patent Document 2 and the like.

[0058] Of the series of recognition processes including detection, analysis, and judgment, the recognition unit 156 of the robot 100 selects and extracts information necessary for recognition, and interpretation processes such as judgment are executed by the recognition unit 212 of the server 200. The recognition process may be executed by only the recognition unit 212 of the server 200, or by only the recognition unit 156 of the robot 100, or the above recognition process may be executed by both of them with their roles shared as described above.

[0059] When a strong impact is given to the robot 100, the recognition unit 156 recognizes this by the touch sensor and the acceleration sensor, and the recognition unit 212 of the server 200 recognizes that a "violent act" has been committed by a nearby user. When a user grabs the horn 112 and lifts the robot 100, this may also be recognized as a violent act. When a user facing the robot 100 speaks in a specific volume range and a specific frequency band, the recognition unit 212 of the server 200 may recognize that a "calling act" has been performed on the robot 100. In addition, when a temperature of about body temperature is detected, it recognizes that a "contact act" has been performed by the user, and when an upward acceleration is detected in a state where contact has been recognized, it recognizes that a "hug" has been performed. Physical contact when the user lifts the body 104 may be sensed, or a hug may be recognized by a decrease in the load on the front wheels 102. In summary, the robot 100 acquires the user's actions as physical information by the internal sensor 128, and the recognition unit 212 of the server 200 judges whether the user feels comfortable or uncomfortable. The recognition unit 212 of the server 200 also performs a user identification process based on the feature vector.

[0060] The recognition unit 212 of the server 200 recognizes various responses of the user to the robot 100. Some typical responses among the various responses are associated with pleasant or unpleasant, positive or negative. Generally, most pleasant responses are positive reactions, and most unpleasant responses are negative reactions. Pleasant and unpleasant responses are related to the degree of intimacy, and positive and negative reactions affect the robot 100's selection of actions.

[0061] The intimacy management unit 220 of the server 200 changes the intimacy with the user in response to the interaction behavior recognized by the recognition unit 156. In principle, the intimacy with a user who has performed a pleasant behavior increases, and the intimacy with a user who has performed an unpleasant behavior decreases.

[0062] Each function of the server 200 is realized by loading a program for implementing the function into memory and instantiating it. The processing capacity of the server 200 supplements various processes performed by the robot 100. The server 200 can be used as a resource for the robot 100. How to use the resources of the server 200 is dynamically determined according to a request from the robot 100. For example, when the robot 100 needs to continuously generate complex motions according to detection values ​​from a large number of touch sensors, the processing of the processor 122 in the robot 100 may be preferentially assigned to selecting and generating motions, and the processing for image recognition of the surrounding situation may be performed by the recognition unit 212 of the server 200. In this way, various processes of the robot system 300 can be distributed between the robot 100 and the server 200.

[0063] A single server 200 can also control multiple robots 100. In this case, each function of the server 200 is realized independently for each robot 100. For example, the server 200 may prepare a recognition unit 212 for robot 100B in addition to a recognition unit 212 (instance object) for robot 100A.

[0064] Based on the above basic configuration, next, implementation of the robot 100 in this embodiment will be described, focusing in particular on the features and purpose of this implementation and differences from the basic configuration.

[0065] [SLAM] The robot 100 of this embodiment acquires a large number of captured images (still images) by periodically capturing images of the surroundings using the omnidirectional camera 113. The robot 100 forms a memory based on the captured images (hereinafter, referred to as an "image memory").

[0066] An image memory is a collection of multiple key frames. A key frame is distribution information of feature points (feature amounts) in a captured image. The robot 100 of this embodiment forms key frames by a graph-based SLAM (Simultaneous Localization and Mapping) technique using image features, more specifically, a SLAM technique based on ORB (Oriented FAST and Rotated BRIEF) features (see Patent Document 3).

[0067] The robot 100 periodically forms key frames while moving, and forms an image memory as a collection of key frames, in other words, an image feature distribution. The robot 100 estimates its current location by comparing a key frame acquired at its current location with a large number of key frames it already possesses. In other words, the robot 100 performs "spatial recognition" by comparing the captured image it is actually viewing with captured images (memory) it has previously viewed, and matching its current situation with its past memory. The image memory formed as a collection of feature points becomes a so-called map. The robot 100 updates the map while moving and estimating its current location.

[0068] The basic configuration of the robot 100 is based on the premise that it recognizes its location not based on key frames but on the external sensor 114. The robot 100 of this embodiment will be described as recognizing its location based only on key frames.

[0069] The robot 100 in this embodiment includes a "mode setting unit" for setting various modes.

[0070] <Baby monitoring> 5 to 7 are schematic diagrams for explaining behavioral scenes when a plurality of robots 100 are watching over a baby. First, in the child's room, the baby to be watched over (hereafter referred to as the "subject") is sleeping in a bouncer (Figure 5A). There are two robots 100 in this house. Each of the two robots 100 recognizes the presence and position of a baby (infant) through image recognition. The two robots 100 may share the position of the baby through mutual communication. The two robots 100 determine the gaze points of each robot from the position of the baby. The two robots 100 correct the gaze points through mutual communication so that the gaze points of each robot are the same or within a predetermined range. The two robots 100 control the movements and parts of the robots 100 so that their heads face the determined gaze points, thereby realizing an action of the two robots 100 peering at the baby (FIG. 5B). One of the two robots 100, robot 100A, continues to watch over the baby by turning its head in the direction of the baby according to the baby's position. Robot 100A keeps a constant distance from the baby so as not to leave the baby's side. The other robot 100B, by sharing roles through communication with robot 100A, stops watching over the baby after a while and starts playing (Fig. 5C). The mother is cooking in the kitchen, leaving the robots 100 to watch over the baby (Figure 5D).

[0071] The mother places a smartphone (communication terminal) in the kitchen (FIG. 6A). Robot 100A (watching over) captures images of the baby using omnidirectional camera 113 and continues to send the captured images to the smartphone as live video. The smartphone displays the area of ​​the omnidirectional image in which the baby appears. At this time, the smartphone may display an image that has been corrected so that the distortion of the omnidirectional image is displayed on a flat surface.

[0072] At this point, the baby suddenly begins crying (Figure 6B). When the robot 100B, which was playing, hears the baby's cry through the microphone (collects the sound), it approaches the baby based on the baby's location (FIG. 6C). The robot 100B executes an interference motion. The robot 100B executes an interference motion when at least one of the following conditions is met: that a sound like a cry has been collected; and that the baby is determined to be in a specific state such as crying through image analysis. The interference motion is a motion for soothing the baby, such as outputting a predetermined sound that attracts the baby's attention, outputting a sound like a lullaby, waving the arms 106, rocking the head or body, or rocking the bouncer. The interference motion distracts the baby. Also, by executing the interference motion, a third party can have the impression that the robot 100B is struggling to do something about the crying baby. Instead of or in addition to the robot B, the robot A may execute the interference motion. A live video feed from robot 100A shows the baby crying on the mother's smartphone (Figure 6D). The video before and after the baby starts crying (the video from a predetermined time before to a predetermined time after the baby starts crying) may be made permanent by being stored on a HDD or the like.

[0073] The mother was so surprised that she instinctively picked up her smartphone (Figure 7A). The mother rushes to the nursery and rushes to her baby (Figure 7B). The mother holds the baby in her arms (Figure 7C). The robots 100 stand beside the mother and baby and look at them. The two robots recognize the respective positions of the mother and baby through image analysis, and set their gaze points from the respective positions. The two robots 100 share their gaze points through communication, and correct them as necessary so that their gaze points are the same or within a predetermined range. The two robots 100 perform an action of directing their heads to their respective gaze points. As a result, the two robots 100 stand side by side and look at the mother and baby, giving the mother the impression that the robots 100 are concerned about the baby. The baby falls asleep again in the mother's arms (Figure 7D). Once the robots 100 confirm that the baby has fallen asleep, they leave the bouncer (Figure 7E).

[0074] When the robot 100 is manually set to the "monitoring mode" by the mother via an input switch, a smartphone, or the like, the robot 100 may search for the baby to start the above-mentioned monitoring action. When the robot 100 detects that the baby is in a bouncer, the mode setting unit may automatically set the robot 100 to the monitoring mode. When the robot 100 detects an infant by image recognition and there is no guardian near the infant, the robot 100 may automatically switch to the monitoring mode. When there is no guardian near the infant, for example, there may be a case where an image of a person of a certain age or older is not detected, or an image of a person associated with the infant is not detected. In addition, when multiple robots 100 are in the same room, all the robots 100 may be set to the monitoring mode, or only some of the robots 100 may be set to the monitoring mode.

[0075] The robot 100 may transition to the watching mode when the watching condition is that a "baby" is detected in the captured image, but no other users or a specific user such as a mother or father is detected.

[0076] The robot 100 may limit its range of movement to within a range where it can visually recognize the baby being watched. Alternatively, the robot 100 may not leave the room where the baby is while watching over the baby. It is desirable that the robot 100 does not take its eyes off the baby while watching over the baby (for example, always captures the baby in the camera's field of view). The robot 100 pays close attention to the condition of the target person using the omnidirectional camera 113, a microphone, a temperature sensor, and other sensors that measure the external environment. When the baby moves around, the robot 100 may recognize the baby's position using the omnidirectional camera 113, a microphone, a temperature sensor, and other sensors that measure the external environment, and control its own direction to face the baby. In addition, when the baby comes within a predetermined distance of the robot 100, the robot 100 may reduce the points of contact with the baby by, for example, storing its wheels.

[0077] When a predetermined alert condition is met, the robot 100 executes a motion for actively interacting with the baby. Alternatively, the robot may transmit "alert information" to another communication terminal such as the mother's smartphone. The alert condition can be set arbitrarily, and may include, for example, situations in which the baby (the person being watched over) is in danger, such as when the baby cries, when approaching stairs, when trying to go outside, when playing with small objects, or when the baby falls. The motion for actively interacting with the baby may be the interference motion described above.

[0078] When the robot 100A switches to the watch mode, it may notify the robot 100B of the switch to the watch mode. Upon receiving the notification, the robot 100B may also switch to the watch mode, or may approach the robot 100A or the baby. By gathering multiple robots 100 near the baby, the robots 100 can act as if they are concerned about the baby.

[0079] At this time, in order not to wake the baby with the operation sound of the robot 100, the robot 100 operates quietly by suppressing the amount of operation of the drive mechanism 120 (particularly the mechanism related to movement and posture adjustment) compared to the normal mode, so as to make less operation sound than normal. When the mode is switched to the watch mode while the baby is awake, the robot B may perform a specific motion that attracts the baby's interest, such as a motion of running around the bouncer or a dance, beside the baby. For example, it may be determined that the baby is awake when a condition is satisfied, such as a condition that the baby's voice is detected by sound analysis or an image that the baby has its eyes open is detected by image analysis. When the baby is asleep, such a motion is not performed. The robot B performs an appropriate motion depending on the state of the person being watched over. It is considered that the baby will also feel safe if many robots 100 gather around him.

[0080] The robot 100 may enter the watching mode when it receives a specific voice command such as "Watch the baby" from a specific user such as the mother via a microphone. When the robot 100 starts watching, it may execute a confirmation action such as going around the baby, turning toward the baby, or pointing a hand 358 at the baby based on the position of the baby detected by image analysis or the like. The mother (instructor) can determine whether the robot 100 is watching over the wrong person by the confirmation action. It is considered that the mother will utter a positive response such as "I'll leave it to you" if it is correct, and a negative response such as "No, it's not" if it is incorrect. Therefore, the robot 100 may determine whether the target of the watch is correct or not based on the mother's response collected by the microphone after the confirmation action.

[0081] During the watching over, the robot 100 keeps its gaze on the baby, for example by peering at the baby. The baby feels reassured, believing that the robot 100 is watching over it. The mother, seeing the robot 100 diligently watching over the baby, is aroused by affection and trust for the robot 100. The watching over behavior also serves as an appeal to the user (witness) that "the robot 100 is working hard."

[0082] As described above, the robot 100B may play freely while the robot 100A is being watched. However, the action range of the robot 100B is limited to the range where the baby or the robot 100A can be visually recognized. The robot 100A may transmit a live image (close-up image) of the baby to the smartphone, and the robot 100B may transmit a live image (wide-angle image) of the robot 100A watching the baby to the smartphone. It is preferable that the robot 100B further restricts the action range so as not to interfere with the robot 100A taking a picture of the baby. For example, the robot 100B may identify the positions of the baby and the robot 100A, and act so as not to overlap on the straight line connecting the baby and the robot 100A. In addition, when the robot 100A recognizes the image being taken and the robot 100B cannot recognize the figure of the baby, the robot 100A may notify the robot 100B of a command requesting it to move. The robot 100B moves in response to the command.

[0083] When the robot 100A detects that the baby has entered a predetermined state by image analysis or audio analysis, for example, when the baby starts to cry, the robot 100A notifies the user of the change in the baby's state in addition to the live video. The robot 100B may execute an interference motion. The robot 100A notifies the smartphone of a message that clearly indicates a state associated with an alert condition, such as "the baby is crying" or "the baby has started to fuss." The robot 100B may perform a dance as an interference motion, or may play music such as a lullaby using a built-in audio player. In either case, the robot 100B comforts the baby by executing an action to distract the baby. When the mother returns (see FIG. 7B), the robot 100B stops the interference motion. Specifically, the robot 100B stops the interference motion when the "mother" is confirmed in the captured image. The robot 100B may stop the interference motion when it recognizes through a microphone or the like that the mother has uttered a keyword (hereinafter referred to as a "completion command") instructing the completion of the watching action, such as "Thank you" or "It's okay now." At this time, it may be determined whether or not the completion command has been uttered by the person who instructed the watching action to the robot 100A, and if so, the watching action may be completed.

[0084] The robot 100 performs image recognition of the baby from the captured image. The method of detecting the "baby" may be realized by applying known face recognition technology. In addition, when the baby is moving, the robot 100 may adjust the direction of movement so that the baby is always visible regardless of whether the robot 100 is being watched over.

[0085] The robot 100A and the robot 100B may take turns watching over the baby. For example, when the robot 100A has watched over the baby for 10 minutes, the robot 100B may switch to a watching mode and move freely. It is considered that the baby will not get bored if multiple robots 100 take turns watching over the baby. In addition, it is easier for a third party to have the impression that the robots 100A and 100B are watching over the baby in cooperation with each other.

[0086] If Robot 100 watches over the baby, even mothers who are busy raising children can concentrate on housework with peace of mind. When doing housework such as laundry away from the baby, you may not immediately notice the baby crying. The baby may start crying in earnest before you realize it. Such situations place a great burden on the mother. By having Robot 100 watch over the baby, it is possible to quickly detect the baby's "fussiness" or other signs of heavy crying.

[0087] Not only the mother but also Robot 100 will participate in child rearing. It is also expected that babies who grow up under the watchful eye of Robot 100 will develop a sense of closeness to Robot 100 in the future.

[0088] <Home alone> 8 to 11 are schematic diagrams for explaining behavioral scenes of the robot 100 when the robot 100 is left at home alone. Consider a home with two robots, 100A and 100B, and a female owner. The female owner goes out to work (Fig. 8A). At this time, the female owner calls out to the nearby robot 100A, saying, "Please stay at home." When the robot 100A hears these words (collects audio via a microphone), it recognizes that the female owner is leaving and switches to "home alone mode" (Figure 8B). The female owner leaves through the entrance (FIG. 8C). Robot 100A moves to the entrance and sees the female owner off by executing a predetermined motion.

[0089] Meanwhile, the robot 100B in the room chases the robot 100A and starts playing with it (FIG. 8D). When the female owner leaves the room, both the robots 100A and 100B may see her off, or only the robot 100 whose intimacy with the female owner is equal to or greater than a predetermined value may see her off.

[0090] After a while, the front door opens while you are away (Figure 9A). When it is determined by voice analysis, image analysis, or the like that the front door is open, the two robots 100 move to the front door in "hope" that the female owner has returned home (FIG. 9B). However, the person who appeared (identified through image analysis, etc.) was not the female owner, but a stranger (suspicious person) carrying a large bag (Figure 9C). At this time, the robot 100 approaches the suspicious person sufficiently and takes a picture of the suspicious person (FIG. 9D). The robot 100 transitions to the alert mode. When the robot 100 transitions to the alert mode, the robots 100 may move close to the suspicious person and continue taking a picture of the suspicious person.

[0091] The female owner is working in the office. The robot 100 transmits a captured image of the suspicious person to the female owner's smartphone (FIG. 10A). The female owner, who is at work, finds out through her smartphone that the robot 100 has found a suspicious person (FIG. 10B). In this case, it is assumed that the person thought to be suspicious is the female owner's mother. The robots 100 do not know the female owner's mother. The woman notifies the robots 100 through her smartphone that "there is no suspicious person." The female owner may inform the robot 100 by voice that "there is no need to worry, it's your mother." The robot 100 then cancels the alert mode. The mother cooks home-cooked meals (Figure 10C).

[0092] The female owner (the daughter) returns home and chats with her mother while eating a home-cooked meal (Figure 11A). Robots 100 are playing nearby them (Figures 11B and 11C).

[0093] When the robot 100 finds a suspicious person (a person who has never been seen before or whose intimacy level is below a threshold) while the owner is away from home, the robot 100 takes a picture of the suspicious person and transmits the captured image of the suspicious person to the smartphone of the female owner (specific user). With such a control method, the female owner can safely leave the security of the house to the robot 100. In addition, the robot 100 detects various events that occur while the owner is away from home, such as an earthquake that breaks things, a gas leak, or a visitor such as a delivery person (a call from an intercom), through image analysis and voice recognition, and notifies the owner's smartphone of the contents of the event. In addition, the robot 100 may record events that occurred while the owner was away from home as a life log, and the owner may check the events that occurred while the owner was away from home by checking the life log via the smartphone after returning home.

[0094] The mode setting unit of the robot 100 may set the answering mode based on an operation input from a user. The robot 100 may automatically change the setting to the answering mode when a specific event such as the user leaving the house through the front door is detected. Alternatively, the robot 100 may automatically change the setting to the answering mode when the user is not visible in the room for a certain period of time or more.

[0095] In the alert mode, the robot 100 may set a range of movement to a position where it can photograph a suspicious person. This is to ensure that the suspicious person's behavior is not overlooked. Also, in order to prevent violence from the suspicious person, the robot 100 may move away from the suspicious person. After reporting the suspicious person, when the robot 100 receives a notification from the user that the person is not suspicious, the robot 100 cancels the alert mode and returns to the answering machine mode. After this, the robot 100 may interact with the unidentified person (former suspicious person) in the usual manner. Also, the robot 100 may memorize the appearance of the unidentified person and manage parameters such as the degree of intimacy. The robot 100 may become attached to the unidentified person. In the example shown in FIG. 8 to FIG. 11, the mother is initially wary of the robot 100, but after the female owner (daughter) sends an acceptance notice to the robot 100, the robot 100 starts to cling to the mother. The mother can suddenly feel that she has been accepted and welcomed by the robot 100.

[0096] In the absence mode, a user (owner) in a remote location may send an instruction to check the interior of the house to the robot 100 via a smartphone. When the robot 100 receives the instruction, it patrols the interior of the house according to a map generated based on SLAM. At this time, the robot 100 may send captured images to the user's smartphone, or may notify the user of the presence or absence of an abnormal event. By sending an instruction to check the interior of the house, the user can check the state of their home at any time.

[0097] <Remote Control> 12 to 14 are schematic diagrams for explaining behavioral scenes when a user who is out remotely controls the robot 100. FIG. Two robots, 100A and 100B, are watching the house while the person is away. There is also a cat in the house (Fig. 12A). Meanwhile, the rest of the family goes out into the city, leaving the 100 robots and the cat behind (Figure 12B). The boy looks gloomy (Figure 12C). The boy takes out his smartphone (mobile device) and begins to operate it (Figure 12D).

[0098] Two robots 100 are playing in the house while no one is there (FIG. 13A). When the robot 100A starts moving, the robot 100B follows the robot 100A (FIG. 13B). The robots 100A and 100B play a game of tag. The mother is concerned about her son's condition (Figure 13C). When the boy leaves, he is concerned that his cat does not seem very energetic (Figure 13D).

[0099] The boy sends a command to the robot 100 from his smartphone saying, "Check on the cat." The robots 100A and 100B move to photograph the location or object indicated in the command, and take pictures as appropriate. The captured images of the robots 100A and 100B are sent to the smartphone. The cat is playing happily in the cat tree (FIG. 14A). The family feels relieved to see the cat looking healthy (Figure 14B). The robot 100 takes a picture of a cat playing on a cat tower. The robot 100 takes a close-up picture of the cat, and the cat looks at the robot 100 (FIGS. 14C and 14D).

[0100] In this way, the user can send various instructions to the robot 100 from the smartphone. In particular, the user can command the robot 100 to check the room. When the command "Check on the cat" is sent as in the above embodiment, the robot 100 detects an object corresponding to the "cat" from the captured image and transmits the captured image with the cat at the center to the smartphone. Such commands may be voice commands or may be input from a graphical user interface provided in the smartphone. The user may be able to operate the robot 100 like a radio-controlled car (hereinafter, such an operation method is referred to as "remote operation").

[0101] The image captured by the robot 100 is displayed on a smartphone, and the user may enlarge and display a part of the captured image that is live-broadcasted to the smartphone that the user particularly wants to see. The robot 100 may transmit the panoramic image itself to the smartphone, and the user may confirm what the robot 100 "saw" through the panoramic image.

[0102] The robot 100 can recognize not only species such as humans and cats, but also individual levels such as "who" and "which." Black cats and white cats, and big cats and small cats are treated as different cats. The robot 100 also learns the names of cats based on the user's calls to the cats. For example, a model may be generated by machine learning that extracts an image of a cat from the name of the cat recognized by voice analysis and the captured image when the name of the cat was recognized, and outputs the name of the cat using the image of the cat as input. By using such a model, even if there are multiple cats, if the user issues a command specifying the name of a cat, the robot 100 can select the specified cat as the subject of the photograph. The user may register the name of the cat and a photo of the cat in advance via a smartphone or the like.

[0103] When a user remotely controls the robot 100A, the robot 100B may move along with the robot 100A. A captured image from the robot 100A and a captured image from the robot 100B are transmitted to the user's smartphone. When the robot 100A captures an image of the cat, the robot 100B, which is close to the robot 100A, also captures an image of the cat using the omnidirectional camera 113. By simply remotely controlling the robot 100A, the user can obtain captured images of the cat not only from the robot 100A but also from the robot 100B. By remotely controlling only the robot 100A, the user can indirectly remotely control the robot 100B as well. This is because the robot 100B has a "following function."

[0104] A user can set the robot 100 to a remote control mode from a smartphone. Also, a user can end the remote control mode of the robot 100 from the smartphone. The robot 100 in the remote control mode may change the display of the eyes 110. For example, the robot 100 may change the eyes 110 to red eyes, or may display an icon on the eyes 110 to visually express that it is "controlled (remotely controlled)". When the remote control mode ends, the robot 100 returns the eyes 110 to the normal black eye display and returns to the location at the start of the remote control mode. When the remote control mode ends, the robot 100 may sit down, or may vigorously shake its head to show that it has "escaped control and regained its sense of self".

[0105] The robot 100 in remote mode does not change emotion parameters or intimacy level.

[0106] When switching to the remote operation mode, the robot 100 authenticates the person who has requested the remote operation. Only when the authentication is successful, the robot 100 switches to the remote operation mode. The authentication may be a general authentication method using an account name and a password, or an electronic certificate may be registered in advance in a device used for remote operation, and only access from a device having the electronic certificate may be permitted. Furthermore, authentication may be performed by confirming that the person operating the mobile terminal is the owner of the robot 100 using a camera or a microphone provided in a mobile terminal such as a smartphone. Also, when the remote mode is requested, a user in the vicinity may be asked to approve the switch to the remote mode. In this way, by sufficiently confirming that the person who has requested the remote mode is the owner of the robot 100, unintended remote operation by a third party can be prevented.

[0107] <Watching over the elderly> 15 to 18 are schematic diagrams for explaining behavioral scenes when a plurality of robots 100 are watching over an elderly person. An elderly father lives alone. His only daughter lives far away from him. There are two robots 100 in the father's house (Figure 15A). In the living room at home, the daughter is looking at her smartphone (Figure 15B). The robots 100 record their life with their father as a life log. The life log is a diary that shows what is happening around the father in a way that respects his privacy. The daughter checks the life log on her smartphone (Figure 15C), which contains simple information such as what time her father woke up and whether he had breakfast. Meanwhile, the father is holding the robot 100 and loving it (FIG. 15D). If the father's permission is recognized through image analysis, audio analysis, communication, or the like, the robot 100 may transmit a captured image of the robot 100 playing with the father to the daughter's smartphone. For example, when the robot 100A is being held by the father, the robot 100B may act as a cameraman and capture an image of the robot 100A and the father, and transmit the captured image to the daughter's smartphone.

[0108] The daughter feels reassured when she sees her father enjoying life with Robot 100 in the living room (Figure 16A).

[0109] Next, imagine a scenario in which the daughter is working in the office. She suddenly takes out her smartphone and checks her father's life log (Figure 16B). Let us suppose that there were almost no records about the father in this life log. The daughter suddenly becomes worried about her father (Figure 16C). The daughter calls her father from the office hallway (Figure 16D).

[0110] The father's cheerful voice soon comes over the phone (Figure 17A). The father was at the inn and had just gotten out of the bath when the phone rang (Figure 17B). The father tells his daughter that he was at the hot springs with friends (Figure 17C). The daughter was relieved to learn the situation, as she had no idea that her father was going to the hot springs (Figure 17D).

[0111] The conversation between father and daughter continues (FIG. 18A, FIG. 18B). At the father's house, two robots 100 are staying at home (FIG. 18C).

[0112] The robot 100 records various events that occur in the life of the father (an elderly person being monitored) as a life log. This life log records the father's regular daily actions, such as the time he woke up and whether he did his usual exercises today. Through the life log provided by the robot 100, the daughter can check whether her father is living his life as usual.

[0113] When an event indicating the relationship between the father and the robot 100 does not occur (is not detected) for a predetermined period of time or more, the robot 100 may transmit an abnormality notification to the daughter's smartphone. For example, when the robot 100 has not been touched by the father for a while, or when the father is lying down even at noon, the abnormality notification may be transmitted. Whether the father is lying down can be determined by image analysis, temperature sensor analysis, or the like. The robot 100 may behave so as to increase the opportunities for the father to be seen by actively moving around the room.

[0114] A life log that abstracts information allows a daughter to check on her father's daily life while protecting his privacy. When watching over an elderly person, the robot 100 does not need to constantly visually confirm the elderly person. Elderly people live independent lives, and the robot 100 should basically behave autonomously. It is considered preferable for an elderly person and the robot 100 to maintain an appropriate distance. The robot 100 may record a life log at all times, not just when watching over the elderly person, if the user so desires. The robot 100 should only notify the daughter of an abnormality in the elderly person's life when it occurs.

[0115] <Expression of jealousy> People cannot remain indifferent to the affection shown to them. When multiple people show affection to the same person, jealousy is likely to arise. Therefore, when robot 100A and robot 100B live with a user, one robot 100 may take a behavior that makes the user feel jealous of the other robot 100.

[0116] For example, when the robot 100B is being held by the user, the state management unit 244 increases the recognition desire value (desire to be recognized), which is a type of emotion parameter of the robot 100A. When the recognition desire value increases, the operation control unit 150 of the robot 100A begs the user to hold it. The robot 100A may stare at the user, may approach the user, or may wander around the user's vicinity to ask to be held. When the user walks, the robot 100A may move following the user. The increase in the recognition desire value is externally expressed as a behavioral characteristic of the robot 100 as if jealousy was aroused.

[0117] When the user continues to hold the robot 100B, the robot 100A may actively express "strong jealousy" by clinging to the user. Alternatively, the robot 100A may passively express jealousy by moving away from the user and looking at the user from afar. Such manner of expressing jealousy is determined according to the personality of each robot 100 (initial personality or cultivated personality). Jealousy may be expressed by "sulking," or may be expressed in a behavioral manner such as moving away even if the user approaches for a certain period of time after the occurrence of an event that makes the robot jealous. The robot 100 may express "sulking" in a behavioral manner by temporarily refusing to be held.

[0118] The robot 100 may behave in a manner that makes the user feel jealous as the intimacy level increases. For example, assume that the robot 100A has a high intimacy level with the user P1 and a relatively low intimacy level with the user P2. In this case, when the user P1 holds the robot 100B, the robot 100B may have a higher approval desire value than when the user P2 holds the robot 100B. According to such a control method, the robot 100 can behave as if it has a possessive desire to monopolize the affection of a user that the robot particularly likes.

[0119] The robot 100 may notify other robots 100 of its own state (emotion parameters, intimacy, events, etc.) (hereinafter, such notification is referred to as "state notification"). Based on the state notification, the robots 100 may be able to grasp each other's state. For example, the robot 100A notifies the robot 100B of a state such as "being held by the user," "being petted by the user," or "having been changed by the user," so that the robot 100B can grasp the state of the robot 100A. While the robot 100A's recognition demand value (desire to be recognized) decreases due to an event such as being held, when the recognition demand value of the robot 100B is in a high state equal to or higher than a threshold, the robot 100B shows a unique behavioral characteristic expressing jealousy.

[0120] In this embodiment, the state management unit 244 of the server 200 collectively manages the emotion parameters of each robot 100. In this case, the state management unit 244 may internally notify the robot 100B of the value of the emotion parameter of the robot 100A, or may change the emotion parameter of the robot 100B based on the emotion parameter of the robot 100A. In this case, the emotion parameter of the robot 100B may be changed on the condition that the robot 100A is in a position visible from the robot 100B. This is to allow the robot 100B, which is close to the robot 100A, to visually sense the change in emotion of the robot 100A and express the state of changing its own emotion parameter.

[0121] In addition to the emotion parameters, the robot 100A may notify the robot 100B by short-distance wireless communication such as infrared. In this case, the robot 100B can receive the state notification of the robot 100A only when it is near the robot 100A and there is no obstacle blocking the line of sight, so that it is possible to express a state in which the robot 100B can "sense the state only when it is close enough to be visible." The robot 100B may detect events such as the robot 100A being held or stroked by the captured image. The robot 100B may change the emotion parameters when it recognizes a pleasant behavior toward the robot 100A through an image.

[0122] The robot 100 may not only be jealous of other robots 100, but also of pets or children. For example, the robot 100's need for approval value may be increased when the user holds a cat. The user may also need to be considerate, such as loving the pet when the robot 100 is not looking, or loving both the pet and the robot 100 equally, in order to avoid making the robot 100 jealous. By actively creating opportunities for the user to consider the feelings of the robot 100, the user's attachment to the robot 100 can be deepened.

[0123] The robot 100 of this embodiment can express the feelings of the robot 100 by its behavior without conversation. The robot 100A may receive the feelings (emotion parameters) of the robot 100B. The server 200 may reflect the change in the emotion parameters of the robot 100B in the behavior of the robot 100A. For example, when the recognition desire value of the robot 100B suddenly drops (when it is considered that something good has happened to the robot 100B), the robot 100A may move closer to the robot 100B. When the robot 100A is notified that the recognition desire of the robot 100B has been satisfied, the robot 100A may increase its own recognition desire value (a desire to be recognized by the robot 100A). According to this control method, even when the user secretly loves the robot 100B, the robot 100A can realize a behavioral expression as if it has sensed something. In other words, a mysterious behavioral expression can be realized as if the robots 100 are communicating with each other telepathically.

[0124] <Multiple robots staring at the same thing> The robots 100A and 100B may continue to gaze at the same object. For example, when the robot 100A gazes at a relaxing user, the robot 100B may also gaze at the same user. The robot 100B may detect that the robot 100A gazes at the relaxing user (the head of the robot 100A faces the direction of the user) through communication with the robot 100A or through image analysis. The robot B may move close to the robot A and then gaze at the user. The user can sense multiple gazes and therefore feel that the robots 100 are very interested in him / her. When the user does not pay attention to the robots 100 for a long time, the robots 100A and 100B may silently ask for "interaction" by gazing at the user at the same time.

[0125] Assume that the robot 100A has a high intimacy level with the user P1 that is equal to or higher than a predetermined value, and the robot 100B also has a high intimacy level with the user P1 that is equal to or higher than a predetermined value. The robot 100 has more opportunities to gaze at a user with a higher intimacy level. Therefore, in the above situation, the robot 100A and the robot 100B may have an opportunity to gaze at the user P1 at the same time by accident. The robot 100A may notify the robot 100B of its state that it is gazing at the user P1. When the robot 100B receives a state notification from the robot 100A that "(the robot 100A) is also gazing at the user P1" while gazing at the user P1, the robot 100B may perform a "coincidence" by executing a specific motion such as a surprised motion or turning its gaze in the direction of the robot 100A. In addition, when both robots are gazing at the same user, the robots 100A and 100B may approach each other and execute a motion to gaze at the user P1 side by side. By moving both robots 100 to positions where the user's face is visible as large as possible and gazing at the user side by side, it is possible to exert strong pressure on the user.

[0126] Also, when an insect gets into the house (when an insect is detected in the house by image analysis or sound analysis, etc.), the robots 100A and 100B may share the target insect, and the robots 100A and 100B may simultaneously stare at the insect, thereby expressing an extraordinary interest in the insect. Furthermore, by staring at each other, the robots 100A and 100B can realize a behavioral expression as if they are showing each other something.

[0127] When the value of the emotion parameter indicating the curiosity of the robot 100A exceeds a threshold, the robot 100A may notify the robot 100B of the state of "increased curiosity". At this time, the robot 100B may execute a motion as if it wants to know the source of the curiosity of the robot 100A, such as approaching the robot 100A and moving its hands to touch the robot 100A. The robot 100A may notify the robot 100B of an object of interest in the panoramic image and its direction. When the robot 100B receives this notification, the robot 100B may gaze at the same object as the object of the robot 100A by directing its head or gaze toward the same object.

[0128] <Appeal> When a predetermined appeal condition is met, for example, when the recognition desire value exceeds a threshold, the robot 100 performs a strong appeal behavior toward the user. The appeal behavior here refers to an action that actively seeks the user's interaction with the robot 100, such as touching, calling out to the user, or holding the user. For example, assume that the user is doing exercise such as yoga indoors. When the user is engrossed in yoga, the robot 100 may perform an appeal behavior such as continuing to stare at the user or wandering around the user, to request the user to stop yoga, if the appeal condition of the robot 100 is met.

[0129] <Follow-up behavior> As described above, when the robot 100A moves, the robot 100B may move behind the robot 100A while following it and keeping a constant distance from the robot 100A. The robot 100A may similarly move following a user or a pet. For example, when a dog is following a user, the robot 100A may follow the dog or the user. When the robot 100A is following a dog, the robot 100B may follow the robot 100A. The distance between the robot and a target to be followed (e.g., a dog following a user) may be equal to the distance between the target to be followed and a target to be followed by the target to be followed (e.g., a user), or may be shorter than the distance by a predetermined length, or may be longer than the distance by a predetermined length.

[0130] When the robot 100 detects that the moving objects Q1 and Q2 are moving in the same direction for a predetermined time or more, the robot 100 determines that "following" is occurring. According to such a control method, when following is occurring, it is possible for the robot 100 to express a behavior that makes the robot 100 instinctively want to follow. It is considered that the appearance of multiple robots 100 performing following behavior is effective in appealing to the user how cute the robots 100 are.

[0131] The robot 100 may execute the following behavior on the condition that the emotional parameter of the robot 100, for example, an emotional parameter indicating curiosity, is equal to or lower than a threshold value. According to such a control method, it is possible to express the behavior of executing the following behavior with respect to another robot when the curiosity is waning and the robot is bored, and not executing the following behavior when the curiosity is increasing. When the robot executes the following behavior, the robot may end the following behavior on the condition that the curiosity is increased to or higher than a threshold value due to various events. The source of the behavior of the autonomous robot is a predetermined parameter indicating the internal state. In this embodiment, the parameter indicating curiosity contributes greatly to the source of the behavior, but if there is little change in the external environment, there is a possibility that the curiosity parameter approaches 0. In such a case, instead of waiting for a change in its own parameters, the robot can actively change its own parameters by piggybacking on the behavior of the other robot.

[0132] As described above, the multiple robots 100 may take the same action, such as following action, or may change their behavioral characteristics while being influenced by each other's actions. In order to increase the cooperation and interlocking of the multiple robots 100, it is desirable for the robots 100 to be able to grasp each other's states within the server 200 or among themselves. Having grasped the state of the robot 100A, the robot 100B may act in synchronization with the state of the robot 100A, or may act independently without being synchronized. An example of the behavior of the robot 100B in synchronization with the robot 100A is when the robot 100B stares at the same thing that the robot A is staring at, and the robot B executes a motion of the same category for the same object as the object of the robot 100A.

[0133] When a plurality of robots 100 behave cooperatively, the user may feel that the robots 100 are cute. The user may want to take a picture of the robots 100 behaving cooperatively. When the robot 100 recognizes that the user is holding the camera by the omnidirectional camera 113, the robot 100 may maintain at least one of its own behavior and state until the user finishes taking pictures. In this case, instead of maintaining the behavior or state, the robot 100 may select a specific motion. For example, the robot 100 may turn its body toward the user, or may cooperate with the user in taking pictures by temporarily stopping the cooperative behavior. In this way, the robot 100 may temporarily stop its operation when it detects a photographing behavior. The robot 100 may pose or temporarily stop its behavior when it detects a photographing behavior of the user, not only when it is acting cooperatively. According to such a control method, the user can easily upload a photographed image of the robot 100 in a cute pose to a social networking service (SNS) or the like. It is also expected that it will become easier to take various best shot images of the robot 100 interacting with various things in the home (pets, children, toys, furniture, etc.).

[0134] <Structure of the outer skin> The outer skin 314 of the robot 100 is constructed by housing an elastic base material in a cloth bag. The bag may be made of a soft material that is warm and pleasant to the touch for the user. The base material is preferably a flame-retardant material, and more preferably a material that emits a self-extinguishing gas when heated. For example, the base material is made of a flame-retardant sponge. The outer skin 314 is formed so that the flame-retardant base material is wrapped in a cloth bag, so that even if the cloth bag ignites, the base material emits a self-extinguishing gas, preventing the spread of fire in the cloth bag. The threshold temperature at which the base material generates the self-extinguishing gas is preferably lower than the ignition temperature of the cloth. In this case, when the cloth becomes hot, the self-extinguishing gas is generated before the cloth ignites, so that the cloth can be prevented from catching fire. By making the outer skin 314 a double structure of a flame-retardant base material and a soft bag, the warmth of the robot 100 and safety against high temperatures can be achieved at the same time.

[0135] The present invention is not limited to the above-mentioned embodiment and modified examples, and the components can be modified without departing from the gist of the present invention. Various inventions can be formed by appropriately combining multiple components disclosed in the above-mentioned embodiment and modified examples. In addition, some components can be deleted from all the components shown in the above-mentioned embodiment and modified examples.

[0136] Although the robot system 300 has been described as being configured by one or more robots 100 and one server 200, some of the functions of the robot 100 may be realized by the server 200, or some or all of the functions of the server 200 may be assigned to the robot 100. One server 200 may control multiple robots 100, or multiple servers 200 may cooperate to control one or more robots 100.

[0137] A third device other than the robot 100 and the server 200 may take on part of the functions. The collection of the functions of the robot 100 and the functions of the server 200 described in Fig. 4 can be understood as one "robot" in general terms. How to allocate the multiple functions required to realize the present invention to one or more pieces of hardware may be determined in consideration of the processing capacity of each piece of hardware, the specifications required for the robot system 300, etc.

[0138] As mentioned above, the "robot in the narrow sense" refers to the robot 100 not including the server 200, while the "robot in the broad sense" refers to the robot system 300. It is possible that many of the functions of the server 200 will be integrated into the robot 100 in the future.

[0139] This application claims priority based on Japanese Patent Application No. XXX-XXX, filed on XXX year, XXX month, the disclosure of which is incorporated herein by reference in its entirety.

Claims

1. A motion control unit that selects the motion of the robot; A drive mechanism that executes the motion selected by the motion control unit; A recognition unit that determines whether a target person satisfies a predetermined monitoring condition; a mode setting unit that sets a monitoring mode for the target person when the monitoring condition is satisfied; a head provided with a facial region; The head is provided with an all-sky camera on an upper portion of the head, The driving mechanism rotates the face area of ​​the head in a direction in which the subject is present during the monitoring mode. A robot characterized by performing a pointing motion.

2. In the monitoring mode, the position of the target person is shared with other robots, and the target person is monitored based on the position of the target person. The gaze point is determined based on the same gaze point as the other robot or the gaze point of the other robot. The robot of claim 1 , wherein the face region of the head is directed toward a gaze point within a certain distance.

3. In the watching mode, at least one of the other robot and the robot itself is in front of the subject.

3. The robot of claim 1 or 2, configured to orient a facial region of the head.

4. The method according to any one of claims 1 to 3, wherein the distance to the subject is controlled to within a predetermined distance during the monitoring mode. Item 1. The robot according to item 1.

5. During the monitoring mode, at least one mode when the monitoring condition is not satisfied is selected.

5. The robot according to claim 1, wherein the amount of movement of the drive mechanism is reduced relative to the amount of movement of the drive mechanism.

6. When it is determined that the target person satisfies a predetermined condition during the monitoring mode, The robot according to any one of claims 1 to 5, characterized in that it executes a motion for a person. Yat.

7. When it is determined that the target person satisfies a predetermined condition during the monitoring mode, The robot according to any one of claims 1 to 6, characterized in that it executes a motion for a person. Yat.

8. a communication unit that transmits a captured image of the subject to a predetermined communication terminal in the monitoring mode; The robot according to any one of claims 1 to 7, characterized in that it comprises:

Images