Robots, programs and methods

Abstract

This paper proposes an optimal control method for robots when they are looking at a user. [Solution] A motion control unit that selects the robot's motion, A drive mechanism that executes the motion selected by the motion control unit, The robot comprises an eye control unit that displays an eye image on the robot's display device, The present invention provides a robot that vibrates the pupil region in the aforementioned eye image.

Description

Technical Field

[0001] The present invention relates to a robot that autonomously makes action selections according to its internal state or external environment.

Background Art

[0002] Humans keep pets in search of healing. On the other hand, there are many people who give up on keeping pets for various reasons, such as not being able to secure enough time to take care of the pet, not having a living environment suitable for keeping a pet, having allergies, or finding it too painful to experience the death of a pet. If there were a robot that could serve the role of a pet, it might be able to give the kind of healing that a pet gives to those who cannot keep a pet (see Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In recent years, although robot technology has been rapidly advancing, it has not yet achieved the presence as a companion like a pet. This is because it is difficult to think that a robot has free will. Humans observe the actions of pets as if they have free will, feel the existence of free will in pets, empathize with pets, and are healed by pets.

[0005] Since pets cannot speak, they try to convey their feelings with their eyes. Eyes are also called "the windows of the heart". Looking at someone means having an interest in that person. When humans make eye contact with a pet, they feel that their hearts are connected, and their affection for the pet is stirred up. The inventors of the present invention consider that in order to make a robot exhibit "a sense of presence as a living being", the expressiveness of the eyes in the robot, particularly the control of "looking", is important.

[0006] This invention was completed based on the above understanding, and its main purpose is to propose a suitable control method for when a robot looks at a user. [Means for solving the problem]

[0007] An autonomous robot in one aspect of the present invention comprises a motion control unit that selects the robot's motion, a drive mechanism that executes the motion selected by the motion control unit, an eye control unit that causes the robot's display device to display an eye image, and a recognition unit that detects a user. The eye control unit changes the pupil region included in the eye image according to the relative position between the user and the robot.

[0008] An autonomous robot in another aspect of the present invention comprises a camera, a temperature detection sensor, a motion control unit that selects the robot's motion, a drive mechanism that executes the motion selected by the motion control unit, an eye control unit that causes the robot's display device to display an eye image, and a recognition unit that detects a user based on an image captured by the camera or a heat distribution image acquired by the temperature detection sensor. The recognition unit selects either an image capture or a thermal distribution image depending on the relative positions of the robot and the user, and identifies the user's face region from the selected image. The eye control unit sets a specific facial region as the fixation point and moves the pupil region included in the eye image according to the fixation point.

[0009] An autonomous robot in another aspect of the present invention comprises a motion control unit that selects the robot's motion, a drive mechanism that executes the motion selected by the motion control unit, an eye control unit that causes the robot's display device to display an eye image, and a recognition unit that detects the user's eyes. The eye control unit sets the user's eyes as the fixation point and moves the pupil region included in the eye image according to the fixation point to direct the robot's gaze towards the user's eyes. Even if the relative positions of the robot and the user change, the unit maintains the robot's gaze directed towards the user's eyes by setting the changed position of the user's eyes as the new fixation point.

[0010] An autonomous robot in another aspect of the present invention comprises a motion control unit that selects the robot's motion, a drive mechanism that executes the motion selected by the motion control unit, an eye control unit that causes the robot to display an eye image on its display device, and a recognition unit that detects a user. The recognition unit further detects the user's gaze direction. The eye control unit sets the user's eye as the fixation point and moves the pupil region included in the eye image according to the fixation point to direct the robot's gaze towards the user's eye. When the user's gaze moves away from the robot, the fixation point is shifted away from the user's eye. [Effects of the Invention]

[0011] According to the present invention, it becomes easier to give the observer of a robot the feeling that the robot is looking at them. [Brief explanation of the drawing]

[0012] [Figure 1] (a) is a front view of the robot, and (b) is a side view of the robot. [Figure 2] This is a cross-sectional view illustrating the structure of the robot. [Figure 3] This is a diagram illustrating the configuration of a robot system. [Figure 4] This is a conceptual diagram of an emotion map. [Figure 5] This is a hardware configuration diagram of the robot. [Figure 6] This is a functional block diagram of a robot system. [Figure 7] This is an external view of an eye image. [Figure 8] This is a magnified view of an eye image. [Figure 9] It is a schematic diagram showing a method for generating an eye image. [Figure 10] It is a schematic diagram for explaining the relationship between the line-of-sight direction and the target direction. [Figure 11] It is an external view of the robot when the observed person is located on the left of the robot. [Figure 12] (a) is a diagram showing the display mode of the eye image in the normal state, (b) is a diagram showing the display mode of the eye image in the flattened state, and (c) is a diagram showing the viewing mode of the eye image shown in (b) from an oblique direction. [Figure 13] It is a schematic diagram showing the target direction, the monitor, and the appearance of the pupil image. [Figure 14] It is a schematic diagram for explaining microsaccades. [Figure 15] It is an external view of the robot when the observed person is located above the robot. [Figure 16] It is a schematic diagram for explaining the fixation point. [Figure 17] It is a schematic diagram of an eye image when the pupil image in the right-eye monitor is in the central position. [Figure 18] It is a schematic diagram of an eye image when the pupil image in the right-eye monitor deviates from the central position. [Figure 19] It is a schematic diagram for explaining the interlocking of the eyelid image and the pupil image. [Figure 20] (a) is a schematic diagram showing the miosis state before opening the eyes, (b) is a schematic diagram showing the normal state during closing the eyes, (c) is a schematic diagram showing the normal state immediately after opening the eyes, and (d) is a schematic diagram showing the miosis state after opening the eyes. [Figure 21] (a) is a schematic diagram showing the mydriasis state before opening the eyes, (b) is a schematic diagram showing the normal state during closing the eyes, (c) is a schematic diagram showing the normal state immediately after opening the eyes, and (d) is a schematic diagram showing the mydriasis state after opening the eyes.

Embodiments for Carrying Out the Invention

[0013] Figure 1(a) is a front view of robot 100. Figure 1(b) is a side view of robot 100. In this embodiment, robot 100 is an autonomous robot that determines its actions and gestures based on the external environment and its internal state. The external environment is recognized by various sensors such as cameras and thermal sensors. The internal state is quantified as various parameters that express the emotions of robot 100. These will be described later.

[0014] Robot 100 will, in principle, operate only within the premises of the owner's home. Hereafter, any person involved with Robot 100 will be referred to as a "user," and in particular, a user who is a member of the household to which Robot 100 belongs will be referred to as the "owner."

[0015] The body 104 of robot 100 has an overall rounded shape and is made of urethane and rubber. This includes an outer shell formed from soft, elastic materials such as resin and fiber. The robot 100 may be dressed in clothes. By making the body 104 round, soft, and pleasant to the touch, the robot 100 provides the user with a sense of security and a comfortable tactile sensation.

[0016] Robot 100 has a total weight of 15 kilograms or less, preferably 10 kilograms or less, and more preferably 5 kilograms or less. By 13 months of age, the majority of babies begin to walk on their own. The average weight of a 13-month-old baby is slightly over 9 kilograms for boys and slightly under 9 kilograms for girls. Therefore, if the total weight of Robot 100 is 10 kilograms or less, the user can carry Robot 100 with roughly the same effort as carrying a baby who cannot yet walk. The average weight of a baby under 2 months old is less than 5 kilograms for both boys and girls. Therefore, if the total weight of Robot 100 is 5 kilograms or less, the user can carry Robot 100 with roughly the same effort as carrying an infant.

[0017] The robot 100 possesses attributes such as moderate weight, roundness, softness, and a pleasant feel, which make it easy for users to pick up and want to pick up. For the same reason, it is desirable that the height of the robot 100 be 1.2 meters or less, preferably 0.7 meters or less. For the robot 100 in this embodiment, the ability to be picked up is an important concept.

[0018] Robot 100 is equipped with three wheels for three-wheeled movement. As shown in the figure, it includes a pair of front wheels 102 (left wheel 102a, right wheel 102b) and one rear wheel 103. The front wheels 102 are the drive wheels, and the rear wheel 103 is the driven wheel. The front wheels 102 do not have a steering mechanism, but their rotation speed and direction can be controlled individually. The rear wheel 103 is a so-called omni-wheel and is rotatable to move robot 100 forward, backward, left, and right. By increasing the rotation speed of the right wheel 102b compared to the left wheel 102a, robot 100 can turn left or rotate counterclockwise. By increasing the rotation speed of the left wheel 102a compared to the right wheel 102b, robot 100 can turn right or rotate clockwise.

[0019] The front wheels 102 and rear wheels 103 can be completely retracted into the body 104 by a drive mechanism (rotation mechanism, linkage mechanism). Even when the robot is moving, most of each wheel is hidden within the body 104, but when each wheel is completely retracted into the body 104, the robot 100 becomes immobile. That is, as the wheels are retracted, the body 104 lowers and sits on the floor surface F. In this seated state, the flat seating surface 108 (ground contact bottom surface) formed on the bottom of the body 104 comes into contact with the floor surface F.

[0020] Robot 100 has two hands 106. Hands 106 do not have the function of grasping objects. Hands 106 can perform simple movements such as lifting, shaking, and vibrating. The two hands 106 can also be controlled individually.

[0021] The eye 110 is capable of displaying images using liquid crystal elements or organic EL elements. In this embodiment, the eye 110 is treated to prevent reflection by attaching an anti-reflective film to a flat monitor on which organic EL elements are arranged. A convex lens with anti-reflective treatment may also be attached to the monitor. The robot 100 is equipped with various sensors, such as a microphone array capable of identifying the direction of a sound source and an ultrasonic sensor. It also has a built-in speaker and can emit simple sounds.

[0022] A horn 112 is attached to the head of robot 100. As mentioned above, robot 100 is lightweight, so users can lift robot 100 by grasping the horn 112. A panoramic camera is attached to the horn 112, which can capture images of the entire upper part of robot 100 at once.

[0023] Figure 2 is a schematic cross-sectional view showing the structure of robot 100. As shown in Figure 2, the body 104 of the robot 100 includes a base frame 308, a main frame 310, a pair of resin wheel covers 312, and an outer shell 314. The base frame 308 is made of metal and forms the axis of the body 104 and supports the internal mechanism. The base frame 308 is constructed by connecting an upper plate 332 and a lower plate 334 vertically with a plurality of side plates 336. Sufficient spacing is provided between the plurality of side plates 336 to allow for ventilation. Inside the base frame 308 are the battery 118, the control circuit 342, and various actuators.

[0024] The main frame 310 is made of resin and includes a head frame 316 and a torso frame 318. The head frame 316 is hollow and hemispherical and forms the head skeleton of the robot 100. The torso frame 318 is stepped and cylindrical and forms the torso skeleton of the robot 100. The torso frame 318 is fixed integrally with the base frame 308. The head frame 316 is assembled to the upper end of the torso frame 318 so as to be displaceable relative to it.

[0025] The head frame 316 is provided with three axes: a yaw axis 320, a pitch axis 322, and a roll axis 324, and actuators 326 for rotationally driving each axis. The actuators 326 include multiple servo motors for individually driving each axis. The yaw axis 320 is driven for head-turning motion, the pitch axis 322 is driven for nodding motion, and the roll axis 324 is driven for head tilting motion.

[0026] A plate 325 supporting the yaw axis 320 is fixed to the upper part of the head frame 316. Multiple ventilation holes 327 are formed in the plate 325 to ensure ventilation between the upper and lower parts.

[0027] A metal base plate 328 is provided to support the head frame 316 and its internal mechanism from below. The base plate 328 is connected to plate 325 via a cross-link mechanism 329 (pantograph mechanism), while also being connected to upper plate 332 (base frame 308) via a joint 330.

[0028] The fuselage frame 318 houses the base frame 308 and the wheel drive mechanism 370. The wheel drive mechanism 370 includes a pivot shaft 378 and an actuator 379. The lower half of the fuselage frame 318 is narrowed to form a storage space S for the front wheels 102 between it and the wheel cover 312.

[0029] The outer shell 314 is made of urethane rubber and covers the main frame 310 and wheel cover 312 from the outside. The handle 106 is integrally molded with the outer shell 314. An opening 390 for introducing outside air is provided at the upper end of the outer shell 314.

[0030] Figure 3 is a diagram showing the configuration of the robot system 300. The robot system 300 includes a robot 100, a server 200, and a number of external sensors 114. A number of external sensors 114 (external sensors 114a, 114b, ..., 114n) are pre-installed inside the house. The external sensors 114 may be fixed to the walls of the house or placed on the floor. The server 200 registers the position coordinates of the external sensors 114. The position coordinates are defined as x,y coordinates within the house, which is assumed to be the robot 100's operating range.

[0031] The server 200 is installed inside the house. In this embodiment, the server 200 and the robot 100 typically correspond one-to-one. The robot 100 has built-in sensors and multiple external sensors. Based on the information obtained from the signal 114, the server 200 determines the basic actions of the robot 100. The external sensor 114 is for augmenting the sensory organs of the robot 100, and the server 200 is for augmenting the brain of the robot 100.

[0032] External sensor 114 periodically transmits a wireless signal (hereinafter referred to as "robot search signal") that includes the ID of external sensor 114 (hereinafter referred to as "beacon ID"). When robot 100 receives the robot search signal, it replies with a wireless signal (hereinafter referred to as "robot response signal") that includes the beacon ID. Server 200 measures the time from when external sensor 114 transmits the robot search signal until it receives the robot response signal, and measures the distance from external sensor 114 to robot 100. By measuring the distance between multiple external sensors 114 and robot 100, the position coordinates of robot 100 are determined. Of course, it is also possible for the robot 100 to periodically transmit its own position coordinates to the server 200.

[0033] Figure 4 is a conceptual diagram of the emotion map 116. The emotion map 116 is a data table stored in the server 200. The robot 100 selects actions according to the emotion map 116. The emotion map 116 shown in Figure 4 shows the degree of the robot 100's like or dislike towards a place. The x and y axes of the emotion map 116 represent two-dimensional spatial coordinates. The z axis represents the degree of like or dislike. A positive z value indicates a high degree of like towards that place, and a negative z value indicates a dislike towards that place.

[0034] In the emotion map 116 of Figure 4, coordinate P1 is a point within the indoor space managed by the server 200 as part of the robot 100's range of movement where positive emotions are high (hereinafter referred to as a "favorable point"). A favorable point may be a "safe place" such as behind a sofa or under a table, or it may be a place where people tend to gather, such as a living room, or a lively place. It may also be a place where the robot has been gently stroked or touched in the past. While the definition of what kind of places Robot 100 prefers is arbitrary, it is generally desirable to set its preferred locations as places that small children or small animals such as dogs and cats enjoy.

[0035] Coordinate P2 is a location associated with high levels of negative emotions (hereinafter referred to as the "aversion location"). Aversion locations may include places with loud noises, such as near a television; places prone to getting wet, such as bathrooms or washrooms; enclosed or dark spaces; or places associated with unpleasant memories of being treated roughly by the user. While the definition of what places Robot 100 dislikes is arbitrary, it is generally desirable to set the disliked locations as places that small children or small animals such as dogs and cats are afraid of.

[0036] Coordinate Q indicates the current position of robot 100. The server 200 determines the position coordinates of robot 100 based on robot search signals and robot response signals periodically transmitted by multiple external sensors 114. For example, when external sensor 114 with beacon ID=1 and external sensor 114 with beacon ID=2 each detect robot 100, the server 200 calculates the distance from the two external sensors 114 to robot 100 and then determines the position coordinates of robot 100 from that distance.

[0037] Given the emotion map 116 shown in Figure 4, the robot 100 will move in the direction that attracts it to favorable points (coordinates P1) and away from disliked points (coordinates P2).

[0038] The emotion map 116 changes dynamically. When robot 100 reaches coordinate P1, the z value (positive emotion) at coordinate P1 decreases over time. As a result, robot 100 is favored. Upon reaching a location (coordinate P1), the robot can emulate the biological behavior of "emotional satisfaction" followed by "boredom" with that location. Similarly, negative emotions at coordinate P2 also lessen over time. As time passes, new locations of favor and aversion emerge, prompting robot 100 to make new behavioral choices. Robot 100 becomes "interested" in new locations of favor and continuously makes behavioral choices.

[0039] The emotion map 116 represents the fluctuations of emotions as the internal state of robot 100. Robot 100 aims for favorable points, avoids unfavorable points, stays in favorable points for a while, and then takes the next action. This type of control makes robot 100's behavioral choices more human-like and biological.

[0040] It should be noted that the maps influencing the behavior of robot 100 (hereinafter collectively referred to as "behavioral maps") are not limited to the type of emotion map 116 shown in Figure 4. For example, various behavioral maps can be defined, such as curiosity, fear avoidance, the desire for security, and the desire for physical comfort such as quietness, dimness, coolness, or warmth. The destination of robot 100 may then be determined by weighting and averaging the z-values ​​of multiple behavioral maps.

[0041] Robot 100 has parameters that indicate the magnitude of various emotions and sensations, separate from its behavior map. For example, when the value of the emotion parameter for loneliness is high, the weighting coefficient of the behavior map that evaluates safe places is set higher, and reaching the target location reduces the value of this emotion parameter. Similarly, when the value of the parameter indicating boredom is high, the weighting coefficient of the behavior map that evaluates places that satisfy curiosity should be set higher.

[0042] Figure 5 is a hardware configuration diagram of robot 100. The robot 100 includes an internal sensor 128, a communication device 126, a storage device 124, a processor 122, a drive mechanism 120, and a battery 118. The drive mechanism 120 includes the wheel drive mechanism 370 described above. The processor 122 and the storage device 124 are included in the control circuit 342. Each unit is connected to the others by power lines 130 and signal lines 132. The battery 118 supplies power to each unit via the power lines 130. Each unit sends and receives control signals via the signal lines 132. The battery 118 is a lithium-ion secondary battery and is the power source for the robot 100.

[0043] The internal sensor 128 is a collection of various sensors built into the robot 100. Specifically, it includes a camera (360-degree camera), a microphone array, a distance measuring sensor (infrared sensor), a thermal sensor, a touch sensor, an acceleration sensor, and an odor sensor. The touch sensor is installed between the outer shell 314 and the main frame 310 to detect user touch. The odor sensor is a known sensor that applies the principle that electrical resistance changes due to the adsorption of odor-causing molecules.

[0044] The communication device 126 is a communication module that performs wireless communication with various external devices such as the server 200, the external sensor 114, and the user's portable device. The storage device 124 consists of non-volatile memory and volatile memory and stores computer programs and various setting information. The processor 122 is a means for executing computer programs. The drive mechanism 120 is an actuator that controls the internal mechanism. Other components such as a display and speaker are also installed.

[0045] The processor 122 selects actions for the robot 100 while communicating with the server 200 and external sensors 114 via the communication device 126. Various information obtained from the internal sensor 128 External information also influences the selection of actions. The drive mechanism 120 primarily controls the wheels (front wheels 102) and the head (head frame 316). The drive mechanism 120 changes the direction and speed of movement of the robot 100 by changing the rotation speed and direction of rotation of each of the two front wheels 102. The drive mechanism 120 can also raise and lower the wheels (front wheels 102 and rear wheels 103). When the wheels are raised, they are completely retracted into the body 104, and the robot 100 comes into contact with the floor surface F at the seating surface 108, entering a seated state. The drive mechanism 120 also controls the hands 106 via wires 134.

[0046] Figure 6 is a functional block diagram of the robot system 300. As described above, the robot system 300 includes a robot 100, a server 200, and a number of external sensors 114. Each component of the robot 100 and the server 200 is realized by hardware including arithmetic units such as a CPU (Central Processing Unit) and various coprocessors, storage devices such as memory and storage, and wired or wireless communication lines connecting them, and software stored in the storage devices that supplies processing instructions to the arithmetic units. The computer program may consist of device drivers, an operating system, various application programs located at a higher layer, and libraries that provide common functions to these programs. The blocks described below represent functional units, not hardware units. Some of the functions of the robot 100 may be implemented by the server 200, and some or all of the functions of the server 200 may be implemented by the robot 100.

[0047] (Server 200) The server 200 includes a communication unit 204, a data processing unit 202, and a data storage unit 206. The communication unit 204 is responsible for communication processing with the external sensor 114 and the robot 100. The data storage unit 206 stores various types of data. The data processing unit 202 performs various processes based on the data acquired by the communication unit 204 and the data stored in the data storage unit 206. The data processing unit 202 also functions as an interface between the communication unit 204 and the data storage unit 206.

[0048] The data storage unit 206 includes a motion storage unit 232, a map storage unit 216, and a personal data storage unit 218. Robot 100 has multiple motion patterns. Various motions are defined, such as shaking its hand 106, approaching the owner in a meandering motion, and gazing at the owner while tilting its head.

[0049] The motion storage unit 232 stores "motion files" that define the control content of the motions. Each motion is identified by a motion ID. The motion files are also downloaded to the motion storage unit 160 of the robot 100. Which motion to execute may be determined by the server 200 or by the robot 100.

[0050] Many of the robot 100's motions are composed of complex motions that include multiple unit motions. For example, when robot 100 approaches its owner, it may be represented as a combination of unit motions: turning to face the owner, approaching while raising its arms, approaching while swaying its body, and sitting down while raising both arms. This combination of four motions realizes the motion of "approaching the owner, raising its arms along the way, and finally sitting down after swaying its body." The motion file defines the rotation angles and angular velocities of the actuators installed on robot 100, relating them to the time axis. (Motion file - actuator control information) Accordingly, various motions can be expressed by controlling each actuator as time progresses.

[0051] The transition time between one unit motion and the next is called the "interval." The interval should be defined according to the time required for the unit motion change and the nature of the motion. The length of the interval is adjustable. Hereinafter, the settings related to the control of robot 100's actions, such as when and which motion to select, and the output adjustment of each actuator in realizing the motion, will be collectively referred to as "action characteristics." The action characteristics of robot 100 are defined by the motion selection algorithm, motion selection probability, motion file, etc.

[0052] The motion storage unit 232 stores motion files as well as a motion selection table that defines the motions to be executed when various events occur. In the motion selection table, one or more motions and their selection probabilities are associated with each event.

[0053] The map storage unit 216 stores multiple behavioral maps, as well as maps showing the placement of obstacles such as chairs and tables. The personal data storage unit 218 stores user, particularly owner, information. Specifically, it stores master information indicating the level of familiarity with the user and the user's physical and behavioral characteristics. Other attribute information such as age and gender may also be stored.

[0054] Robot 100 has an internal parameter called "affection level" for each user. When Robot 100 recognizes actions that show it is fond of the user, such as picking it up or talking to it, its affection level with that user increases. The affection level will be lower for users who do not interact with Robot 100, users who act rudely, or users it does not encounter often.

[0055] The data processing unit 202 includes a location management unit 208, a map management unit 210, a recognition unit 212, an operation control unit 222, a closeness management unit 220, and a state management unit 244. The position management unit 208 determines the position coordinates of the robot 100 using the method described with reference to Figure 3. The position management unit 208 may also track the user's position coordinates in real time.

[0056] The state management unit 244 manages various internal parameters, such as the charge level, internal temperature, and processing load of the processor 122. The state management unit 244 includes the emotion management unit 234. The emotion management unit 234 manages various emotional parameters that indicate the emotions of robot 100 (loneliness, curiosity, need for recognition, etc.). These emotional parameters are constantly fluctuating. The importance of multiple behavioral maps changes according to the emotional parameters, the target locations of robot 100's movement change according to the behavioral maps, and the emotional parameters change as robot 100 moves and time passes.

[0057] For example, when the emotion parameter indicating loneliness is high, the emotion management unit 234 sets a larger weighting coefficient for the behavioral map that evaluates safe places. When the robot 100 reaches a point on this behavioral map where loneliness can be alleviated, the emotion management unit 234 lowers the emotion parameter indicating loneliness. In addition, various emotion parameters change depending on the interaction actions described later. For example, when the owner "holds" the robot, the emotion parameter indicating loneliness decreases, and when the robot does not see the owner for a long period of time, the emotion parameter indicating loneliness gradually increases.

[0058] The map management unit 210 changes the parameters of each coordinate for multiple action maps in the manner described in relation to Figure 4. The map management unit 210 then selects one of the multiple action maps. You can choose one z-value, or you can take a weighted average of the z-values ​​of multiple action maps. For example, suppose action map A has z-values ​​of 4 and 3 at coordinates R1 and R2, and action map B has z-values ​​of -1 and 3 at coordinates R1 and R2. In the case of a simple average, the total z-value at coordinate R1 is 4-1=3, and the total z-value at coordinate R2 is 3+3=6, so robot 100 will move in the direction of coordinate R2, not R1. When action map A is given five times more importance than action map B, the total z-value at coordinate R1 is 4 × 5 - 1 = 19, and the total z-value at coordinate R2 is 3 × 5 + 3 = 18. Therefore, robot 100 will move in the direction of coordinate R1.

[0059] The recognition unit 212 recognizes the external environment. Recognition of the external environment includes various types of recognition, such as recognition of weather and season based on temperature and humidity, and recognition of shaded areas (safe zones) based on light intensity and temperature. The recognition unit 156 of the robot 100 acquires various environmental information using the internal sensor 128, processes it, and then transfers it to the recognition unit 212 of the server 200.

[0060] Specifically, the recognition unit 156 of the robot 100 extracts image regions corresponding to moving objects, particularly people and animals, from the image region, and extracts a "feature vector" from the extracted image region as a set of feature quantities that indicate the physical and behavioral characteristics of the moving object. Feature vector components (feature quantities) are numerical values ​​that quantify various physical and behavioral characteristics. For example, the width of a human eye is quantified in the range of 0 to 1, forming one feature vector component. The method for extracting feature vectors from captured images of people is an application of known face recognition technology. The robot 100 transmits the feature vector to the server 200.

[0061] The recognition unit 212 of the server 200 further includes a person recognition unit 214 and a response recognition unit 228. The person recognition unit 214 determines which person the captured user belongs to by comparing the feature vector extracted from the image captured by the robot 100's built-in camera with the feature vector of a user (cluster) pre-registered in the personal data storage unit 218 (user identification process). The person recognition unit 214 includes an expression recognition unit 230. The expression recognition unit 230 estimates the user's emotions by recognizing the user's facial expressions from the image. Furthermore, the person recognition unit 214 also performs user identification processing for moving objects other than people, such as pets like cats and dogs.

[0062] The response recognition unit 228 recognizes various responses made to the robot 100 and classifies them as pleasant or unpleasant. The response recognition unit 228 also recognizes the owner's responses to the robot 100's actions and classifies them as positive or negative responses. Pleasant and unpleasant actions are determined by whether the user's response is biologically pleasant or unpleasant. For example, being held is a pleasant action for robot 100, while being kicked is an unpleasant action. Positive and negative responses are determined by whether the user's response indicates a pleasant or unpleasant emotion for the user. For example, being held is a positive response indicating a pleasant emotion for the user, while being kicked is a negative response indicating an unpleasant emotion for the user.

[0063] The motion control unit 222 of the server 200 works in cooperation with the motion control unit 150 of the robot 100 to determine the motion of the robot 100. Based on the action map selection by the map management unit 210, the motion control unit 222 of the server 200 creates the target locations for the robot 100's movement and the corresponding movement routes. The motion control unit 222 may create multiple movement routes and then select one of them.

[0064] The motion control unit 222 selects a motion for the robot 100 from a plurality of motions in the motion storage unit 232. Each motion is associated with a selection probability for each situation. For example, when the owner performs a pleasant action, motion A is executed with a 20% probability. A selection method is defined, such as executing motion B with a 5% probability when the temperature exceeds 30 degrees Celsius. The movement target and route are determined on the action map, and motions are selected through various events described later.

[0065] The intimacy management unit 220 manages the intimacy level for each user. As mentioned above, the intimacy level is registered as part of the personal data in the personal data storage unit 218. When a pleasant action is detected, the intimacy management unit 220 increases the intimacy level with that owner. When an unpleasant action is detected, the intimacy level decreases. In addition, the intimacy level of owners who have not been viewed for a long period of time gradually decreases.

[0066] (Robot 100) The robot 100 includes a communication unit 142, a data processing unit 136, a data storage unit 148, an internal sensor 128, and a drive mechanism 120. The communication unit 142 corresponds to the communication device 126 (see Figure 5) and is responsible for communication processing with the external sensor 114, the server 200, and other robots 100. The data storage unit 148 stores various types of data. The data storage unit 148 corresponds to the storage device 124 (see Figure 5). The data processing unit 136 performs various processes based on the data acquired by the communication unit 142 and the data stored in the data storage unit 148. The data processing unit 136 corresponds to the processor 122 and the computer program executed by the processor 122. The data processing unit 136 also functions as an interface to the communication unit 142, the internal sensor 128, the drive mechanism 120, and the data storage unit 148.

[0067] The data storage unit 148 includes a motion storage unit 160 that defines various motions of the robot 100 and an eye image storage unit 172. Various motion files are downloaded from the motion storage unit 232 of the server 200 to the motion storage unit 160 of the robot 100. Motions are identified by motion IDs. In order to express various motions such as sitting down with the front wheels 102 retracted, lifting the hand 106, rotating the robot 100 by rotating the two front wheels 102 in opposite directions or by rotating only one front wheel 102, shaking by rotating the front wheels 102 while they are retracted, and stopping briefly and looking back when moving away from the user, the timing of operation, duration, and direction of operation of various actuators (drive mechanisms 120) are defined chronologically in the motion file.

[0068] The data storage unit 148 may also receive various data downloaded from the map storage unit 216 and the personal data storage unit 218. The eye image storage unit 172 stores the data of the eye image (described later) displayed in the eye 110.

[0069] The internal sensor 128 includes a touch sensor 154, a thermosensor 138, a gaze detection unit 140, a camera 144, and a distance measuring sensor 180. In this embodiment, the camera 144 is a panoramic camera (omnidirectional camera) attached to the horn 112. The camera 144 continuously captures images of the area around the robot 100. The thermosensor 138 periodically detects the ambient temperature distribution around the robot 100. The robot 100 detects whether or not a user is present in its vicinity using the camera 144 and the thermosensor 138. The gaze detection unit 140 is a known sensor that detects the user's eye movements from the images captured by the camera 144. The gaze detection unit 140 detects the direction of the user's gaze toward the robot 100.

[0070] The touch sensor 154 detects user touch on the body 104. The distance measuring sensor 180 is a known sensor that measures the distance to an object.

[0071] The data processing unit 136 includes a recognition unit 156, an operation control unit 150, and an eye control unit 152. The motion control unit 150 of the robot 100 works in cooperation with the motion control unit 222 of the server 200 to determine the motion of the robot 100. Some motions may be determined by the server 200, while others may be determined by the robot 100. Alternatively, the robot 100 may determine the motion, but if the processing load on the robot 100 is high, the server 200 may determine the motion. The server 200 may determine the base motion, and the robot 100 may determine additional motions. How the motion determination process is divided between the server 200 and the robot 100 should be designed according to the specifications of the robot system 300.

[0072] The motion control unit 150 of the robot 100, together with the motion control unit 222 of the server 200, determines the direction of movement for the robot 100. The server 200 may determine movement based on an action map, while the motion control unit 150 of the robot 100 may determine immediate movements such as avoiding obstacles. The drive mechanism 120 drives the front wheels 102 according to the instructions of the motion control unit 150, thereby directing the robot 100 toward the target location.

[0073] The motion control unit 150 of the robot 100 instructs the drive mechanism 120 to execute the selected motion. The drive mechanism 120 controls each actuator according to the motion file.

[0074] The motion control unit 150 can perform a motion of raising both hands 106 as a gesture of asking to be held when a user with whom it has a close relationship is nearby, or it can express a motion of not wanting to be held by repeatedly rotating in the opposite direction and stopping while keeping the left and right front wheels 102 retracted. The drive mechanism 120 drives the front wheels 102, hands 106, and neck (head frame 316) according to the instructions of the motion control unit 150, causing the robot 100 to express various motions.

[0075] The eye control unit 152 generates an eye image and displays it on a monitor installed in the eye 110. Details of the eye image will be described later.

[0076] The recognition unit 156 of the robot 100 interprets external information obtained from the internal sensor 128. The recognition unit 156 is capable of visual recognition (visual unit), smell recognition (olfactory unit), sound recognition (auditory unit), and tactile recognition (tactile unit). The relative position determination unit 182 included in the recognition unit 156 determines the relative positional relationship between the robot 100 and the person being observed.

[0077] The recognition unit 156 detects moving objects such as people and pets based on images (spherical images) captured by the camera 144. The recognition unit 156 includes a feature extraction unit 146. The feature extraction unit 146 extracts feature vectors from the captured images of the moving objects. As described above, a feature vector is a set of parameters (feature quantities) that indicate the physical and behavioral characteristics of the moving object. When a moving object is detected, physical and behavioral characteristics are also extracted from odor sensors, built-in sound-collecting microphones, temperature sensors, etc. These features are also quantified and become feature vector components.

[0078] The robot system 300 clusters users who appear frequently as "owners" based on physical and behavioral characteristics obtained from a large amount of image information and other sensing information. For example, if a mobile object (user) with a beard is often active in the early morning (early riser) and rarely wears red clothing, then a first profile can be created: a cluster (user) that is early riser, has a beard, and rarely wears red clothing. On the other hand, if a person wears glasses... While the moving object is often wearing a skirt, if this moving object does not have a beard, a second profile can be created: a cluster (user) that wears glasses and a skirt but definitely does not have a beard. The above is a simple example, but using the method described above, a first profile corresponding to the father and a second profile corresponding to the mother are formed, and the robot 100 recognizes that there are at least two users (owners) in this house.

[0079] However, robot 100 does not need to recognize that the first profile is "father." It only needs to recognize the person's characteristics as "a cluster of people who have beards, often get up early, and rarely wear red clothes." For each profile, a feature vector is defined that characterizes the profile.

[0080] Assuming that this cluster analysis has been completed, the robot 100 has now recognized a new moving object (user). At this time, the person recognition unit 214 of the server 200 performs user identification processing based on the feature vector of the new moving object and determines which profile (cluster) the moving object belongs to. For example, when a moving object with a beard is detected, there is a high probability that this moving object is a father. If this moving object is active in the early morning, it is even more certain that it is a father. On the other hand, when a moving object wearing glasses is detected, there is also a possibility that this moving object is a mother. If this moving object has a beard, it is determined that it is neither a mother nor a father, and is a new person that has not been clustered.

[0081] The formation of clusters (profiles) through feature extraction (cluster analysis) and the application of the extracted features to the clusters may be performed simultaneously. The characteristics of the owners may be registered in advance. When an unknown user is detected, the feature extraction unit 146 may extract feature vectors from the user, and the person recognition unit 214 may identify the person by determining which of the known owners they correspond to.

[0082] In a series of recognition processes including detection, analysis, and judgment, the recognition unit 156 of the robot 100 selects and extracts the information necessary for recognition, while the interpretation process, such as judgment, is performed by the recognition unit 212 of the server 200. The recognition process may be performed solely by the recognition unit 212 of the server 200, solely by the recognition unit 156 of the robot 100, or both may perform the recognition process while sharing responsibilities as described above.

[0083] When a strong impact is applied to the robot 100, the recognition unit 156 recognizes this using its built-in acceleration sensor, and the server 200's response recognition unit 228 recognizes that a "violent act" has been committed by a nearby user. A violent act may also be recognized when a user grabs the horns 112 and lifts the robot 100. When a user facing the robot 100 speaks in a specific volume range and frequency band, the server 200's response recognition unit 228 may recognize that a "verbal interaction" has been made towards it. Furthermore, when a temperature of approximately body temperature is detected, it is recognized that a "contact action" has been made by a user, and when upward acceleration is detected while contact has been recognized, it is recognized that the robot has been "picked up". Physical contact when a user lifts the body 104 may be sensed, or the robot may be picked up when the load on the front wheels 102 decreases. In summary, the robot 100 acquires the user's actions as physical information using its internal sensor 128, the server 200's response recognition unit 228 determines whether the user is pleased or unhappy, and the server 200's recognition unit 212 performs user identification processing based on feature vectors.

[0084] The server 200's response recognition unit 228 recognizes various user responses to the robot 100. Some typical responses among these responses include positive or negative, affirmative or negative. These are associated with each other. Generally, most pleasant responses are positive, and most unpleasant responses are negative. Pleasant and unpleasant responses are related to intimacy, and positive and negative responses influence the behavioral choices of robot 100.

[0085] In response to the interaction recognized by the recognition unit 156, the intimacy management unit 220 of the server 200 changes the intimacy level with the user. In principle, the intimacy level increases for users who perform pleasant actions and decreases for users who perform unpleasant actions.

[0086] The recognition unit 212 of the server 200 determines whether the response is pleasant or unpleasant, and the map management unit 210 may change the z-value of the location where the pleasant or unpleasant action occurred in the behavior map that expresses "attachment to a place". For example, when a pleasant action occurs in the living room, the map management unit 210 may set the living room as a favored location with a high probability. In this case, a positive feedback effect is achieved in which the robot 100 likes the living room, and by receiving pleasant actions in the living room, it likes the living room even more.

[0087] The level of intimacy with a moving object (user) changes depending on the actions taken by that user.

[0088] Robot 100 sets a high level of familiarity with people it frequently encounters, touches often, and speaks to often. Conversely, it sets a low level of familiarity with people it rarely sees, doesn't touch often, is rough, or scolds loudly. Robot 100 changes its level of familiarity with each user based on various external information detected by its sensors (sight, touch, and hearing).

[0089] The actual robot 100 autonomously makes complex behavioral choices according to a behavior map. Robot 100 acts while being influenced by multiple behavior maps based on various parameters such as loneliness, boredom, and curiosity. If the influence of the behavior maps is excluded, or when the internal state is such that the influence of the behavior maps is small, robot 100 will, in principle, try to approach people with whom it has a high level of intimacy and try to move away from people with whom it has a low level of intimacy.

[0090] Robot 100's behavior can be categorized as follows, depending on its level of familiarity with the user. (1) Users with a very high level of intimacy Robot 100 expresses affection strongly by approaching the user (hereinafter referred to as "proximity behavior") and performing affectionate gestures that are predetermined as gestures that show goodwill towards a person. (2) Users with a relatively high level of intimacy Robot 100 will only perform close-range actions. (3) Users with a relatively low level of intimacy Robot 100 does not perform any special actions. (4) Users with particularly low intimacy levels Robot 100 performs a detachment action.

[0091] According to the control method described above, when robot 100 finds a user with whom it has a high level of familiarity, it approaches that user, and conversely, when it finds a user with whom it has a low level of familiarity, it moves away from that user. This control method allows robot 100 to express what is known as "shyness" through its behavior. Also, when a visitor (user A with a low level of familiarity) appears, robot 100 may move away from the visitor and towards a family member (user B with a high level of familiarity). In this case, user B can sense that robot 100 is feeling shy and anxious, and that it is relying on them. Through such behavioral expressions, user B is moved by the joy of being chosen and relied upon, and the resulting feelings of attachment.

[0092] On the other hand, when User A, a visitor, frequently visits, speaks to, and touches Robot 100 The robot's affinity for user A gradually increases, and it stops exhibiting shy behavior (avoidance behavior) towards user A. User A also senses that robot 100 has become more comfortable around them, and thus develops an attachment to robot 100.

[0093] It should be noted that the above behavioral choices are not always performed. For example, when the internal parameter indicating robot 100's curiosity is high, the behavioral map that seeks places to satisfy that curiosity will be prioritized, and robot 100 may not choose behaviors influenced by familiarity. Also, if the external sensor 114 installed at the entrance detects the user's return home, robot 100 may prioritize greeting the user.

[0094] The recognition unit 156 detects the position of the user's face using the camera 144 and the thermal sensor 138. The relative positioning unit 182 of the recognition unit 156 measures the distance from the robot 100 to the user (hereinafter referred to as the "target distance") using the distance measuring sensor 180. The relative positioning unit 182 may also measure the target distance based on the size of the user's face identified from the captured image or thermal distribution image.

[0095] [Ocular Image Control] Figure 7 is an external view of eye image 174. The eye control unit 152 generates an eye image 174, which includes a pupil image 164 (pupil region) and a peripheral image 168, and displays the eye image 174 as a video on a monitor 170 embedded in the position of the eye 110. The robot 100's "gaze" is represented by moving the pupil image 164. It can also perform blinking actions at predetermined timings.

[0096] The pupil image 164 includes the pupil region 158 and the iris region 162. The pupil image 164 also displays a catchlight 166 to represent the reflection of ambient light. The catchlight 166 in the eye image 174 is not illuminated by the reflection of ambient light, but is an image region represented as a high-luminosity area by the eye control unit 152.

[0097] The eye control unit 152 moves the pupil image 164 up, down, left, and right. When the robot 100's recognition unit 156 recognizes a user, the eye control unit 152 directs the pupil image 164 towards the user's location. The eye control unit 152 expresses the robot 100's "gaze" by changing the eye image 174. Details of the control of the eye image 174 will be described later in relation to Figure 9.

[0098] The eye control unit 152 may change the shape of the pupil image 164. For example, when the pupil image 164 is in the center of the monitor 170, it may be made into a perfect circle, and when it is in the peripheral area, it may be changed to an elliptical shape. By changing the shape of the pupil image 164 according to its position within the monitor 170, the flat monitor 170 can be made to appear as if it has a curved shape like an actual eyeball.

[0099] The eye control unit 152 changes the position of the catchlight 166 in accordance with the direction of the external light source. Figure 7 shows the display position of the catchlight 166 when the external light source is located to the upper left as viewed from the robot 100. By linking the position of the catchlight 166 to the external light source, a more realistic eye image 174 can be represented. The eye control unit 152 may determine the direction of the external light source from the captured image by image recognition, or from detection data of a light sensor (not shown).

[0100] Figure 8 is a magnified view of eye image 174. In the eye image 174, the pupil image 164 and the peripheral image 168 are superimposed with an eyelid image 176 showing the eyelid. The eyelid image 176 includes the eyelashes 178. The peripheral image 168 corresponds to the conjunctiva in humans. The iris region 162 included in the pupil image 164 corresponds to the cornea in humans. This is the part.

[0101] The eye control unit 152 modifies the eyelid image 176, pupil region 158, iris region 162, and catchlight 166 of the eye image 174. When the light intensity is high, the eye control unit 152 reduces the diameter of the pupil region 158. The eye control unit 152 may also expand or contract the entire pupil image 164 instead of just the pupil region 158. When the light intensity is particularly high, the eye control unit 152 may lower the eyelid image 176 to represent a "blinded appearance."

[0102] Figure 9 is a schematic diagram showing the method for generating eye images 174. The eyeball model 250 is a three-dimensional computer graphic modeled after the eyeball of robot 100. The eye control unit 152 first forms a three-dimensional sphere using polygons and then applies a texture (hereinafter referred to as "eyeball texture") to it to form the eyeball model 250. The eyeball texture is an image that includes the pupil image 164. The eye image storage unit 172 stores multiple types of eyeball textures.

[0103] A first surface 252 and a second surface 254 are positioned in front of the eyeball model 250. The first surface 252 and the second surface 254 are virtual planes corresponding to the display surface of the monitor 170 of the eye 110. The eye control unit 152 generates a two-dimensional eyeball projection image 256 from the three-dimensional eyeball model 250 by projecting the eyeball model 250 onto the first surface 252.

[0104] The eye control unit 152 displays the eyelid image 176 on the second surface 254. By superimposing the eyeball projection image 256 on the first surface 252 and the eyelid image 176 on the second surface 254, the eye image 174 shown in Figure 8, etc., is generated. The eye control unit 152 generates two eyeball models 250, one for the right eye and one for the left eye, and generates an eye image 174 for each. Hereinafter, the first surface 252 and the second surface 254 will be referred to collectively as the "eyeball surface 258".

[0105] The eye control unit 152 changes the eyeball projection image 256 by rotating the eyeball model 250. Because this method generates a three-dimensional eyeball model 250 and projects it onto the first surface 252 while rotating it, it can express the movement of the robot's gaze more smoothly than directly drawing the eye image 174 onto the first surface 252. Even though the eye image 174 is two-dimensional, it is generated and controlled based on the three-dimensional eyeball model 250, so this method can easily express the complex movements unique to the eyeballs of living organisms, such as the fixation tremors described later.

[0106] The eye control unit 152 displays the eyelid image 176 on a second surface 254, which is different from the first surface 252, thereby superimposing the eyelid image 176 onto the eyeball projection image 256. When a person's hands are clapped in front of their eyes, they reflexively close their eyes. In order to implement such a conditioned reflex of the eye in the robot 100, it is necessary to change the eyelid image 176 at high speed. In this embodiment, the image processing of the first surface 252 and the image processing of the second surface 254 are independent. When the eye control unit 152 is expressing closing its eyes, it only needs to control the image of the second surface 254. When blinking, the eye control unit 152 also only needs to perform image processing on the second surface 254. Because the eyeball model 250 (eyeball projection image 256) and the eyelid image 176 can be controlled separately, the eyelid image 176 can be controlled at high speed.

[0107] Figure 10 is a schematic diagram illustrating the relationship between the line of sight and the direction of object. Living organisms have spherical eyeballs and change the direction of their gaze by rotating their eyeballs. For example, even when a human is on the left side of a dog, they can sense the dog's gaze because the dog can rotate its eyeballs and direct its pupils towards the human on its left. The direction directly opposite the dog's pupil, in other words, the direction of the center line of the pupil on the surface of the dog's eyeball (normal direction), represents the direction of the dog's gaze. In this case, the dog's pupil appears as a perfect circle to a human. Without moving their body or neck, dogs can change the direction of the pupil's normal direction (direction of gaze) by moving their eyeballs, thus gazing at humans with what is known as a "sidelong glance." can.

[0108] As described above, in order for the robot 100 to represent gazing, the normal direction of the pupil image 164 must be directed toward the person being gazed at (hereinafter referred to as the "observed person"). In this embodiment, the eyes 110 of the robot 100 are generated by displaying the eye image 174 on the monitor 170. The monitor 170 is a planar display device fixed to the body 104. Therefore, the line of sight direction (normal direction of the pupil) of the eyes 110 of the robot 100 is fixed.

[0109] Figure 10 is a top-down view of robot 100. The right eye normal NR represents the normal direction (viewing direction) of the right eye monitor 170R, and the left eye normal NL represents the normal direction (viewing direction) of the left eye monitor 170L. Robot 100 cannot move the right eye normal NR (right eye line of sight) or the left eye normal NL (left eye line of sight) unless the head frame 316 or the torso frame 318 is rotated.

[0110] Hereafter, the direction from which the robot 100 is moving toward the observed person will be referred to as the "target direction." The relative position identification unit 182 identifies not only the target distance but also the target direction. When the observed person is to the right of the robot 100, the right eye normal NR and the target direction DR are close, so the observed person can feel that the robot 100 is looking at them with its right eye. On the other hand, the left eye normal NL and the target direction DR are quite different, so the observed person is less likely to feel that the robot 100 is spotting them with its left eye. Similarly, when the observed person is to the left of the robot 100, the observed person can feel that the robot 100 is looking at them with its left eye, but they are less likely to feel a gaze from the right eye.

[0111] When the observer is in front of robot 100, neither the right eye normal NR nor the left eye normal NL coincides with the target direction DF. The observer will have difficulty perceiving a line of sight from either the left or right eye of robot 100.

[0112] Figure 11 is an external view of robot 100 when the observer is positioned to the left of robot 100. In Figure 11, the observer is positioned to the left of the robot 100. Since the left eye normal NL and the target direction DL are almost the same, if the pupil image 164 of the left eye monitor 170L (hereinafter referred to as "left pupil image 164L") is moved slightly to the left, the observer can feel that the robot 100's left eye is looking at them. On the other hand, the right eye normal NR and the target direction DL are significantly different. Even if the eye control unit 152 moves the pupil image of the right eye monitor 170R (hereinafter referred to as "right pupil image 164R") to the left, the observer will have difficulty feeling the gaze of the robot 100's right eye. This is because the right eye normal NR (direction of the right eye's gaze) is not directed towards the observer. When the target distance is close, the feeling of being looked at by the left eye but not by the right eye tends to become stronger.

[0113] If the motion control unit 150 moves the head frame 316 to the left, the discrepancy between the right eye normal NR and the target direction DL can be reduced. However, there is a limit to the amount of rotation of the head frame 316. Also, there are cases where it is desired to achieve the representation of directing the gaze towards the observer without moving the neck.

[0114] When one perceives a gaze, the eyes of the observer and the observer are directly facing each other. When the pupil image 164 of robot 100 appears as a perfect circle to the observer, the observer can perceive that robot 100 is looking at them. In this embodiment, the monitor 170 is fixed to the main body and cannot be rotated to face the observer like an eyeball. Also, since the monitor 170 is a flat display, the direction of gaze does not change according to the position of the pupil image 164. Therefore, in this embodiment, when the right eye normal NR and the target direction DL are misaligned, the pupil image 164 is deformed to change the direction of gaze of the right eye image 174R. To simulate a change in direction.

[0115] Figure 12(a) shows the display of the eye image 174 under normal conditions. Figure 12(b) shows the display of the eye image 174 when flattened. Figure 12(c) shows the viewing of the eye image 174 shown in Figure 12(b) from an oblique direction. In this embodiment, in order to suppress the sense of unnaturalness caused by the discrepancy between the line of sight direction (normal direction) and the target direction, the eye control unit 152 simulates moving the line of sight direction of the pupil image 164 by flattening the pupil image 164 (hereinafter referred to as "pupil flattening processing"). Pupil flattening processing is a process that compresses the pupil image 164 in the vertical direction and deforms the pupil image 164 into a horizontally elongated ellipse shape.

[0116] When the observer is on the left side of the robot 100, the observer faces the display surface of the left eye monitor 170L directly. At this time, the eye control unit 152 does not flatten the pupil image 164 of the left eye. Since the observer sees the perfectly circular left eye image 174L and the perfectly circular pupil image 164, the observer can sense the gaze of the robot 100's left eye.

[0117] On the other hand, since the observer will be viewing the right eye monitor 170R from a diagonal left direction (see Figure 11), the right eye monitor 170R will appear as a vertically elongated ellipse. This is the same as how a round coin appears vertically elongated when viewed from an angle. The eye control unit 152 flattens the pupil image 164 of the right eye as shown in Figure 12(b). When the horizontally flattened pupil image 164 is viewed from a diagonal left direction, the pupil image 164 appears as a perfect circle as shown in Figure 12(c). Because the monitor 170 is flattened vertically when viewed from an angle, flattening the pupil image 164 horizontally will result in a display that is close to a perfect circle.

[0118] Figure 13 is a schematic diagram showing the direction of view and how the monitor 170 and pupil image 164 appear. In Figure 13, the pupil image 164 is assumed to be flattened as shown in Figure 12(b). When the observer is facing directly towards the monitor 170 (front direction D1), the monitor 170 appears as a perfect circle, but the pupil image 164 is significantly flattened horizontally. When the observer is to the left of the monitor 170 (D2), the observer views the monitor 170 from an oblique angle, so the monitor 170 appears as a vertically flattened shape. The vertical flattening of the monitor 170 and the horizontal flattening of the pupil image 164 cancel each other out, and the pupil image 164 appears as a horizontally flattened shape that is close to a perfect circle. When the observer is positioned further to the left (D3), the monitor 170 appears even more vertically flattened, and the pupil image 164 appears as a perfect circle. Thus, when the horizontally elliptical pupil image 164 is viewed from an oblique angle, the pupil image 164 appears as a perfect circle to the observer.

[0119] If the flat monitor 170 does not move, the normal direction of the monitor 170 does not change, so it is not possible to actually change the direction of the robot 100's line of sight. In this embodiment, by deforming the pupil image 164, a perfectly circular pupil image 164 is shown to the observer even when viewing the monitor 170 from an oblique direction. By flattening the pupil image 164 horizontally, a pseudo-perfect circle of the pupil image 164 can be shown to an observer who is not directly facing the monitor 170. With this control method, an observer on the left side of the robot 100 can feel as if they are being gazed upon not only by the left eye but also by the right eye. Through pupil flattening, even with a stationary and flat monitor 170, it is possible to express "gazing with a sidelong glance". However, when pupil flattening is performed in response to an observer in the left direction D3, another user in the front direction D1 will see the flattened pupil image 164.

[0120] In this embodiment, two patterns of pupil images 164 are provided: a perfectly round pupil image 164 and a flattened pupil image 164. The eye control unit 152 switches between the two types of pupil images 164 by changing the eyeball texture. Not limited to two patterns, multiple types of eyeball textures may be provided depending on the degree of flatness. Since it only involves changing the eyeball texture, the eye control unit 152 can easily and quickly change the degree of flatness of the pupil image 164.

[0121] When the observer is in front of the robot 100 (see target direction DF in Figure 10), the eye control unit 152 flattens both the pupil image 164 of the right eye and the pupil image 164 of the left eye. When the observer is to the left of the robot 100 (see target direction DL in Figure 10), the eye control unit 152 flattens the pupil image 164 of the right eye but not the pupil image 164 of the left eye. When the observer is to the right of the robot 100 (see target direction DR in Figure 10), the eye control unit 152 flattens the pupil image 164 of the left eye but not the pupil image 164 of the right eye.

[0122] When the robot 100 detects that a user is holding it, it identifies the user holding it as the subject of observation. When the robot is being held, the eye control unit 152 flattens the pupil images 164 of one or both of the robots 100 and the subject of observation according to their relative positions. The "relative position" identified by the relative position identification unit 182 may be the direction of the object, or it may be defined based on both the direction of the object and the distance to the object.

[0123] When not being held, the robot 100 uses the camera 144 to photograph its surroundings, and the recognition unit 156 recognizes the user through image processing. When there is only one user, the recognition unit 156 identifies that single user as the observation target. The eye control unit 152 changes the pupil image 164 (pupil region) according to the relative position of the robot 100 and the observation target. The relative position identification unit 182 identifies the target direction by recognizing the user's face from the image captured by the camera 144. When there are two or more users, the recognition unit 156 selects one of the users as the observation target. The observation target may be the user closest in distance, or the user with the highest level of familiarity.

[0124] Figure 14 is a schematic diagram illustrating fixation micro-movements. The human eye constantly makes small movements to prevent light from being burned onto the retina and to prevent fatigue of the optic nerve. This is called "involuntary eye movement." Involuntary eye movement can be broken down into drift, tremor, and microsaccade. Drift represents the wave-like movement of the pupil's center point, tremor is the zigzag movement superimposed on the drift, and microsaccade represents the linear movement of the pupil. Involuntary eye movement is also said to reflect a person's psychological state.

[0125] In this embodiment, the eye image 174 of the robot 100 simulates this fixation micro-movement. Specifically, the eye control unit 152 slowly moves the axis point of the pupil image 164 in a wave-like manner, and vibrates the pupil image 164 around the axis point to represent tremors and drift. Microsaccades are also represented by periodically or randomly moving the axis point of the pupil image 164 in a jumping motion. The axis point can be any point included in the pupil image 164. In this embodiment, the axis point is defined as the center point of the pupil image 164.

[0126] By vibrating the pupil image 164 displayed on the monitor 170, particularly by making it move in a manner similar to fixation tremors, the lifelike appearance of the robot 100 can be further enhanced. The eye control unit 152 causes the pupil image 164 to perform trema motion while simultaneously performing microsaccade motion at random intervals. After the robot 100 is powered on, the eye control unit 152 continuously causes the pupil image 164 to perform drift motion while continuing both trema motion and microsaccade motion.

[0127] The robot 100 and the robot system 300 including the robot 100 have been described above based on the embodiments. According to the control method shown in this embodiment, even when the user is not directly facing the robot 100, the user can feel as if the robot 100 is looking at them. Being looked at means feeling that the gaze, or in other words, the normal of the pupil of the eyeball, is directed towards the user. And so it is. At this time, the pupils that are looking at you become perfect circles. In this embodiment, the robot 100 has a three-dimensional eyeball model 250, but since it projects it onto a two-dimensional monitor 170 (eyeball surface 258), it is not possible to actually move the normal of the pupil. Therefore, in this embodiment, the pupil image 164 is flattened so that even users who are not directly facing the monitor 170 can see a pupil image 164 that is close to a perfect circle, thereby giving the user the feeling that they are being "looked at".

[0128] Robot 100 has a round head and displays eye images 174 on two monitors 170 with different normal directions. Essentially, the left and right pupil images 164 are facing in different directions (right eye normal NR and left eye normal NL in Figure 10). The inventors noticed that when facing Robot 100, it is difficult to feel that Robot 100 is looking at the user, and they analyzed the cause. In this embodiment, pupil flattening processing is applied according to the relative position of the user (observed person) and Robot 100, so that the user is shown a perfectly round pupil image 164. Pupil flattening processing allows a pseudo-viewing direction to be set without being fixed to the normal direction of the monitor 170.

[0129] In this embodiment, the eye control unit 152 generates a three-dimensional eyeball model 250 and projects it onto the first surface 252 to generate a two-dimensional eye image 174. By rotating the eyeball model 250, movement of the pupil image 164 and fixation tremors can be represented. Furthermore, the eye control unit 152 overlays the second surface 254, onto which the eyelid image 176 is projected, onto the first surface 252. Since the eyelid image 176 can be controlled independently of the eyeball model 250 (projected eyeball image 256), the eye control unit 152 can quickly change the eyelid image 176.

[0130] The eye control unit 152 causes the pupil image 164 to subtly move in a manner that mimics fixation tremors. When a user looks at the robot 100's eye image 174 at close range, the fixation tremors, in addition to the gaze, make it easier for the user to feel that they are being "looked at." Furthermore, by expressing fixation tremors, the user can more easily perceive the robot 100 as a living being.

[0131] It should be noted that the present invention is not limited to the embodiments and modifications described above, and the components can be modified and implemented without departing from the spirit of the invention. Various inventions may be formed by appropriately combining the multiple components disclosed in the embodiments and modifications described above. In addition, some components may be deleted from all the components shown in the embodiments and modifications described above.

[0132] Although the robot system 300 is described as consisting of one robot 100, one server 200, and multiple external sensors 114, some of the functions of the robot 100 may be implemented by the server 200, or some or all of the functions of the server 200 may be assigned to the robot 100. One server 200 may control multiple robots 100, or multiple servers 200 may cooperate to control one or more robots 100.

[0133] A third device other than robot 100 or server 200 may perform some of the functions. The collection of functions of robot 100 and server 200 described in Figure 6 can also be viewed as a single "robot" in a broader sense. How to distribute the multiple functions necessary to realize the present invention to one or more hardware devices should be determined in consideration of the processing capacity of each hardware device and the specifications required for the robot system 300.

[0134] As mentioned above, "robot in the narrow sense" refers to robot 100 excluding server 200, while "robot in the broad sense" refers to robot system 300. Many of the functions of server 200 may be integrated into robot 100 in the future.

[0135] The robot 100's behavior control program may be provided via the Internet from a designated server, or it may be provided on a fixed recording medium such as a CD-ROM. In any case, the robot 100's behavior control program may be provided from a different recording medium (server, CD-ROM, etc.) than the robot 100 and installed on the robot 100.

[0136] [Differentiation] In this embodiment, the robot 100 was described as having two eyes 110, but the present invention can also be applied to robots with three or more eyes 110, or to robots with only one eye 110.

[0137] In this embodiment, the robot 100 was described as having two eyes 110 with different normal directions (the orientation of the monitor 170), but the present invention can also be applied when two or more eyes 110 have the same normal direction. For example, consider a robot having a planar face with two eyes 110 set on the planar face. When the observer is in front of this robot, pupil flattening is unnecessary. When the observer is diagonally in front of the robot, both pupil images 164 should be flattened to the relative position between the observer and the robot 100.

[0138] When User A hands Robot 100 to User B, if User A's level of familiarity with User B is high, Robot 100 may choose User A as its observation subject. In this case, it becomes possible to express emotions to User A through eye contact, conveying the anxiety caused by being handed from its preferred User A to User B.

[0139] The eye control unit 152 may choose not to perform pupil flattening processing, or to suppress the degree of flattening, when multiple users are detected around the robot 100. When user A is the subject of observation, the robot 100 flattens the pupil image 164 so that user A can see a perfectly round pupil image 164. In this case, user B (a user who is not the subject of observation), who is located in a different direction from user A, will be able to see the flattened pupil image 164. When multiple users are present, or when multiple users are located in directions that are more than a predetermined angle apart from the robot 100, the eye control unit 152 may stop or suppress the flattening of the pupil image 164.

[0140] The eye control unit 152 may perform pupil flattening processing only when the person being observed is within a predetermined range (hereinafter referred to as the "pupil control range") from the robot 100, for example, within 3 meters from the robot 100 and within a 150-degree range in front of the robot 100. When the person being observed and the robot 100 are far apart, the person being observed is less likely to perceive the robot 100's gaze and is less likely to experience discomfort due to a shift in the direction of gaze. Also, when the person being observed is behind the robot 100, the person being observed cannot see the robot 100's eyes 110, so there is little need to perform pupil flattening processing.

[0141] The eye control unit 152 may choose not to perform pupil flattening processing, or to suppress it, when multiple users are within the pupil control range. This is because if pupil flattening processing is performed for one user when multiple users are within the pupil control range, the flattened pupil image 164 will be visible to the other users. On the other hand, even when multiple users are present, if only one user is within the pupil control range, the eye control unit 152 may perform pupil flattening processing for that user only.

[0142] The eye control unit 152 may expand or contract the pupil image 164. The eye control unit 152 may expand or contract the pupil region 158 while maintaining the size of the peripheral image 168, or it may expand or contract the iris region 162 itself. The eye control unit 152 may respond to a predetermined event (hereinafter referred to as the "expansion / contraction event"). The eye control unit 152 expands or contracts the pupil image 164 when a loud noise or person is detected. The eye control unit 152 may also expand the pupil image 164 to express surprise or interest in the robot 100 when a user with a familiarity level above a predetermined threshold is detected, or when the user's gaze is detected.

[0143] The recognition unit 156 may detect the brightness of the external environment using a light sensor or the like. The eye control unit 152 may reduce the pupil image 164 when the brightness of the external environment is above a predetermined value, and enlarge the pupil image 164 when the brightness is below a predetermined value. Thus, the enlargement / reduction event can be defined as an event that indicates a change in the external environment, including interest or surprise.

[0144] The eye control unit 152 may display a predetermined effect on the eye image 174 when a predetermined event (hereinafter referred to as an "effect event") occurs. The eye control unit 152 may wink with a predetermined probability when it detects a gaze directed at the robot 100 from a user within a predetermined range. Specifically, a wink may be expressed by momentarily opening and closing the eyelid image 176. In addition to winking, other effect displays such as averting the pupils or blurring the pupils are also possible. For example, when a user with a familiarity level below a predetermined value (a disliked user) is detected, the eye control unit 152 may move the pupil image 164 so as not to direct its gaze towards the user. In other words, the eye control unit 152 may exclude "disliked users" from the subjects of observation. The eye control unit 152 may set a longer gazing time for users with higher familiarity levels. When a user speaks to the eye control unit 152 may show "interest" by enlarging the pupil image 164 or by displaying moist eyes as if the pupils were moist. Specific methods for displaying moist eyes include enlarging the size of the catchlight 166 reflected in the pupil image 164, making the catchlight 166 vibrate, changing the number, shape, and position of the catchlight 166, enlarging the pupil image 164 or pupil region 158, and vibrating the pupil image 164.

[0145] In this embodiment, the monitor 170 was described as a planar display using organic EL elements. Since organic EL can be curved, it is thought that if the monitor 170 itself could be curved, even more natural eye-tracking control would be possible. In addition, although the monitor 170 in this embodiment is fixed to the body 104, the monitor 170 may be movable on the body 104. It is thought that by making the monitor 170 movable and curved, even more natural changes in eye-tracking can be expressed. Even when there are limitations on the curvature and mobility of the monitor 170, natural eye-tracking expression can be achieved by using it in combination with the pupil flattening process shown in this embodiment.

[0146] In this embodiment, the eye control unit 152 was described as flattening the pupil image 164 in the lateral direction. The eye control unit 152 may deform the pupil image 164 in any direction depending on the direction of observation. As shown in Figure 15, when the face of the person being observed is positioned above the robot 100, the eye control unit 152 may flatten the pupil image 164 in the vertical direction. Not limited to the vertical and horizontal directions, the pupil image 164 may also be deformed diagonally to match the direction of observation. When the robot 100 is being held by the person being observed, the relative position determination unit 182 detects the tilt of the hold using a gyro sensor, touch sensor, captured image, etc., and determines the relative positional relationship between the robot 100 and the person being observed based on the tilt of the robot 100 and the position of the person being observed's face. Based on the relative position, the eye control unit 152 selects the direction of flattening of the iris region 162 when performing pupil flattening processing. This control method allows for the representation of robot 100 gazing intently at the observer with an upward gaze, even when viewed from above. The principle is the same as for lateral flattening. When a small animal looks up with an upward gaze, the observer is more likely to perceive the animal as cute. Similarly, robot 100 can achieve an upward gaze similar to that of a small animal by vertically flattening the pupil image 164.

[0147] In this embodiment, the eye control unit 152 processes the eyeball texture of the perfectly circular pupil image 164 and flattened The explanation described how to change the pupil image 164 in two stages by replacing the eyeball texture of the pupil image 164. The eye control unit 152 may change the pupil image 164 in three or more stages. In addition, the eye control unit 152 may continuously change the flattening of the pupil image 164, not limited to replacing the texture. For example, the eye control unit 152 may express continuous flattening of the pupil image 164 by continuously flattening the pupil image 164 in the eyeball texture applied to the eyeball model 250. The eye control unit 152 may set the flattening of the pupil image 164 to be greater the larger the angle between the normal direction and the symmetry direction of the monitor 170.

[0148] An upper limit may be set for the flattening of the pupil image 164. By suppressing excessive flattening, it is possible to prevent users other than the observed person from recognizing the pupil image 164 in an unnatural shape. Also, when the flattening of the pupil image 164 exceeds the first threshold, the recognition unit 156 may rotate the head frame 316 toward the observed person. By slightly moving the head frame 316, it is possible to represent gazing toward the observed person without excessively increasing the flattening of the pupil image 164. Similarly, when the flattening exceeds the second threshold, which is greater than the first threshold, the recognition unit 156 may rotate the torso frame 318 (the entire body) toward the observed person.

[0149] The eye control unit 152 may select one of several types of eyeball models 250 (eye images 174). For example, the eye control unit 152 may select an eyeball texture set from several eyeball texture sets to be applied to the eyeball model 250. Multiple eyeball texture sets (eye images 174) with different pupil colors may be prepared. Alternatively, an eyeball texture set with cartoon-like pupils and an eyeball texture set with photographic pupils may be prepared. Preparing several types of eyeball texture sets in advance and selecting the eyeball texture set to be controlled from them is called "first selection". Multiple eyeball textures are associated with the eyeball texture set according to the degree of flatness. Selecting one of the eyeball textures included in the eyeball texture set according to the degree of flatness is called "second selection". For example, when the blue eyeball texture set is first selected, the eye control unit 152 second selects one of the multiple blue eyeball textures included in the first selected eyeball texture set according to the degree of flatness. Furthermore, the eye control unit 152 may first select one of several types of eyeball textures and change the flatness of the pupil image 164 in the first selected eyeball texture.

[0150] The user may switch the eyeball model 250 by giving selection instructions to the eye control unit 152. The user may input selection instructions for the eyeball texture set by voice, or the robot 100 may be provided with selection buttons not shown. The eyeball model 250 (eyeball texture set) may be selected according to the region. For example, the blue-eyed eyeball model 250 (eyeball texture set) may be set for robots 100 shipped to North America, and the black-eyed eyeball model 250 may be set for robots 100 shipped to Asia. The eye control unit 152 may determine the location of the robot 100 using GPS (Global Positioning System) and change the eyeball model 250 according to the location. The eye control unit 152 may first select the eyeball texture set that is closest to the color of the owner's eyes when first seen after power-on.

[0151] The eye control unit 152 may individually change the brightness of the two monitors 170. Generally, the monitor 170 appears bright when viewed directly, but appears dark when viewed from an oblique angle. As shown in Figure 11, the right eye monitor 170R, where the deviation between the normal direction and the direction of target (line of sight) is greater for the left eye monitor 170L, appears darker than the left eye monitor 170L. In this case, the eye control unit 152 can improve the visibility of the pupil image 164 in the right eye monitor 170R, which is farther from the observer, by increasing the brightness of the right eye monitor 170R compared to the left eye monitor 170L. In other words, it is possible to balance the brightness of the left and right eyes 110. The greater the deviation between the normal direction and the direction of target of the monitor 170, the greater the visibility of the pupil image 164 in the right eye monitor 170R. You can simply set the brightness of 70 higher. Alternatively, the eye control unit 152 may set the brightness of the monitor 170 higher the greater the flatness of the pupil image 164.

[0152] The main functions of the eye control unit 152 are: a first function to select an eyeball texture set; a second function to select one of the eyeball textures included in the first selected eyeball texture set according to its flatness; a third function to move the eyeball model 250 to which the eyeball texture is attached in three dimensions and project it onto the first surface 252; and a fourth function to display the eyelid image 176. The eye control unit 152 may also include a "first eye selection unit," a "second eye control unit," a "third eye control unit," and a "fourth eye control unit," each responsible for the first to fourth functions, respectively.

[0153] In this embodiment, the explanation was based on the premise of pupil flattening, but various modifications without pupil flattening are also conceivable.

[0154] <Point of focus S> Figure 16 is a schematic diagram illustrating the point of focus S. The eye control unit 152 may set a point of focus S at any point in the space where the robot 100 is located. The eye control unit 152 rotates the eyeball model 250R (right eye) and the eyeball model 250L (left eye) to direct the two pupil images 164 towards the point of focus S. In other words, the point of focus S is positioned on the normal of the pupil image 164 in the eyeball model 250. By concentrating the two lines of sight of the robot 100's two pupil images 164 on the point of focus S, the "attention" of the point of focus S by the robot 100 can be represented.

[0155] Camera 144, left eye monitor 170L, and right eye monitor 170R are mounted on the head frame 316 of robot 100, and their relative positions are fixed. First, a three-dimensional coordinate system (hereinafter referred to as the "gaze coordinate system") is defined with the forehead of robot 100 (a predetermined area on the head frame 316) as the origin. In the gaze coordinate system, the position coordinates of camera 144, left eye monitor 170L, and right eye monitor 170R are fixed. By positioning the gaze point S in the gaze coordinate system, the display position of the pupil image 164 on the monitor 170 for representing the gaze can be calculated. The eye control unit 152 determines the coordinates of the gaze point S in this gaze coordinate system. The distance from robot 100 (left eye monitor 170L and right eye monitor 170R) to the gaze point S may be measured using a distance measuring sensor such as a depth camera or stereo camera. When setting the gaze point S to a human face, the distance may be determined based on the size of the face image included in the captured image. The fixation point S is not limited to a human face; it can be any object that the robot 100 is interested in at that moment. For example, the object that the robot 100 is performing image recognition on could be set as the fixation point S, and the fixation point S could be switched sequentially as the image recognition progresses.

[0156] When the user is holding or touching the robot 100, the approximate distance between the user and the robot 100 is determined according to the manner in which the user is interacting with the robot. The eye control unit 152 may identify the manner in which the robot 100 is interacting with the user and estimate the distance to the gaze point S according to that manner. In this way, the coordinate value of the gaze point S in the line-of-sight coordinate system (space) is calculated, and the position of the pupil image 164 is calculated so that the gaze point S is located on the normals of the left and right eyeball models 250.

[0157] The eye control unit 152 can set the user's face region, preferably the user's eyes, as the fixation point S. This allows the robot 100 to express attention (stare) towards the user. In addition, the eye control unit 152 may set the fixation point S to something newly discovered by the robot 100 or a user that it has newly seen. Setting the fixation point S to an accessory worn by the user, such as a ring, can express the robot 100's interest and curiosity towards the ring. The fixation point S may also be set to another robot 100. By moving the fixation point S at a high frequency, distraction can be expressed.

[0158] The eye control unit 152 may set a fixation point S on the user's face area or eyes when the gaze detection unit 140 detects the user's gaze. This control method makes it possible to express behavior in which the robot gazes back at the user when the user gazes at it, creating a "moment of gazing" between the user and the robot 100.

[0159] It is said that humans have an instinct to keep things they like in their field of vision. When a user gazes at Robot 100, if Robot 100 also gazes back at the user, the user can feel that "their affection for Robot 100 is being acknowledged" and "not only do I like Robot 100, but Robot 100 likes me too." This "time spent gazing at each other" is considered an important factor in deepening the interaction between the user and Robot 100.

[0160] <Face recognition using thermal sensor 138> As mentioned above, the act of "gazing" signifies "having an interest in the object." In particular, when two people gaze into each other's eyes, they experience an inexpressible sense of "unity" and "connection." In order to achieve such a strong bond between the user and Robot 100, Robot 100 needs to recognize the position of the user's face, especially the position of the user's eyes.

[0161] The recognition unit 156 of the robot 100 recognizes the user's face region from the captured image (spherical image) acquired by the camera 144. The recognition unit 156 further identifies the position of the user's eyes from the face region in the captured image. Specifically, the recognition unit 156 (relative position identification unit 182) identifies the direction and distance of the user's eyes from the robot 100's current position. The eye control unit 152 sets a fixation point S at the position of the user's eyes.

[0162] On the other hand, when a user is holding the robot 100, especially when the user is cradling the robot 100 on their lap, it can be difficult for the robot 100 to recognize the position of the user's face. For example, the captured image of the user may be dark due to backlighting, or the camera 144 (horn 112) of the robot 100 may be located directly below the user's face, causing the user's face area to appear flattened from the robot 100's perspective. If a suitable image cannot be obtained, the recognition unit 156 identifies the position of the user's face based not on the captured image, but on the thermal distribution image (ambient temperature distribution) obtained from the thermosensor 138.

[0163] While it can be difficult to pinpoint the user's eye position in thermal distribution images, the user's face region is a heat source, making it easier to identify the user's face region with high accuracy regardless of the relative positions of the user and the robot 100. The eye control unit 152 sets a fixation point S in the user's face region identified by the thermal distribution image. With this control method, the robot 100 can maintain contact with the user's face and gaze at it even when being held.

[0164] If it is difficult to detect the face region even in the thermal distribution image, the motion control unit 150 may move the head frame 316 to acquire a thermal distribution image from a different angle. Even when being held, the robot can move its neck and torso to find a position that makes it easier to detect the face, allowing it to look at the user's face more accurately. The user can feel a strong interest in the robot 100 as it moves its neck and torso to look at them. Of course, even when being held, if a suitable image can be obtained, the user's face or eyes may be identified based on the image.

[0165] As mentioned above, when being held, robot 100 does not capture images but rather heat The robot 100 may recognize the user's face region based on a cloth image. Alternatively, when the robot 100 detects a touch, it is assumed that the user is in close proximity to the robot 100. Therefore, even when the robot 100 detects a touch, it may recognize the user's face region based on a thermal distribution image rather than a captured image. On the other hand, when the robot 100 is standing upright, or when it has not detected a touch, it may recognize the user's face region based on a captured image. By using two types of images—a captured image (visible light image) and a thermal distribution image—the robot 100 can more reliably recognize the position of the user's face.

[0166] The recognition unit 156 of the robot 100 may search for heat sources from the heat distribution image even when the robot 100 is operating upright. The heat source may be a face. While a face can be a heat source, a heat source is not necessarily a face. Home appliances such as refrigerators can also be heat sources. Therefore, the location of the heat source is identified by the thermosensor 138 and designated as a "face region candidate". When the recognition unit 156 identifies a face region from the captured image (spherical image), it preferentially performs image recognition from the location of the face region candidate identified by the thermosensor 138. By narrowing down the locations of the face region candidate in advance using the thermosensor 138, face recognition processing based on the captured image (spherical image) can be performed efficiently. Thus, the recognition unit 156 may detect the user's face region or the user's eyes based on both the heat distribution image and the captured image, rather than just one of them.

[0167] <Movement of pupil image 164> Figure 17 is a schematic diagram of the eye image 174 when the pupil image 164 is in the central position on the right eye monitor 170R. Figure 18 is a schematic diagram of the eye image 174 when the pupil image 164 is out of the central position on the right eye monitor 170R. When a person is looking straight ahead, they can easily move their pupils. However, when looking to the side, or in what is called a "side glance," the extraocular muscles (a collective term for various muscles such as the superior rectus muscle) that move the eyeball are strained, making it more difficult to move the pupils. For example, when you are already looking to the left and try to look further to the left, or when you are looking to the left and try to shift your gaze to the upper left, the extraocular muscles are strained. Similarly, when you are looking upwards, it is more difficult to move your pupils further upwards than when you are looking straight ahead.

[0168] In light of these physical characteristics of the human eyeball, it is thought that applying similar control to the eye image 174 of robot 100 could further enhance the lifelike quality of robot 100. In Figure 17, point R indicates the center of the right eye monitor 170R (perfect circle), and axis point V indicates the center of the pupil image 164 (perfect circle). A home region 400 is pre-set around the periphery of the center point R. The shape of the home region 400 is arbitrary. In the right eye monitor 170R shown in Figure 17, the home region 400 has a slightly wider oval shape towards the inside (towards the center of robot 100's face).

[0169] When the axis point V of the pupil image 164 is within the home region 400 (Figure 17), the eye control unit 152 can move the pupil image 164 at a speed of less than or equal to the first upper limit speed T1. On the other hand, when the axis point V moves out of the home region 400 (Figure 18), the iris region 162 moves the pupil image 164 at a speed of less than or equal to the second upper limit speed T2 (low speed), which is less than the first upper limit speed T1. The pupil image 164 moves quickly in the central part of the eye image 174, but moves slowly in the peripheral part. In other words, when the robot 100 is facing forward, its gaze moves easily, and when it is looking around, its gaze movement becomes sluggish. With this control method, the characteristics of the robot 100's eyes can be made even closer to those of the human eye.

[0170] As described above, the eye control unit 152 may set a fixation point S for any object to be fixed on. When a predetermined displacement condition is met, the motion control unit 150 controls the neck or... The robot 100 may also move its torso to orient its face towards the gaze point S. Specifically, the position of the robot 100's neck and torso (body 104) may be physically adjusted so that the pupil image 164 is oriented towards the gaze point S when the axis point V of the pupil image 164 is within the home region 400 (hereinafter referred to as the "central position").

[0171] The displacement condition may be met, for example, when the robot 100 sees a user whose level of familiarity is above a predetermined threshold (a favorite user), or when the robot 100 is picked up by a user. In addition, the displacement condition can be arbitrarily set, such as when the robot 100 detects something moving, something flashing, or a predetermined object such as a red garment.

[0172] When held, Robot 100 directs its gaze towards the user's face. If it cannot maintain eye contact with the user's face in the central position (if Robot 100 cannot look directly at the user), it changes its body orientation so that it can maintain eye contact with the user's face in the central position. When the user is holding Robot 100, they can perceive that Robot 100 is moving its body to face them directly. Moving not only the pupil image 164 but also, in some cases, the body to gaze at the user is thought to further evoke the user's affection for Robot 100.

[0173] It is not limited to just two areas, such as the home area 400 and the area other than the home area 400; three or more areas may be set. Also, it is not necessary to explicitly set an area like the home area 400. At a minimum, the eye control unit 152 should move the pupil image 164 more quickly in the central part of the monitor 170 and more slowly in the peripheral part.

[0174] <Linkage between eyelid image 176 and pupil image 164> Figure 19 is a schematic diagram illustrating the relationship between the eyelid image 176 and the pupil image 164. When a person rotates their eyeballs to look upwards, or in other words, when they look upwards, their eyelids also move upwards (towards opening the eyes) in accordance with the movement of the eyeballs. Conversely, when a person rotates their eyeballs to look downwards, their eyelids also move downwards (towards closing the eyes). If similar control is implemented in robot 100, it is possible to express the physical characteristics of the human eye, and thus further enhance the lifelike quality of robot 100.

[0175] In Figure 19, the monitor 170 has an upper eye area 402 and a lower eye area 404 pre-set. The upper eye area 402 corresponds to the entire area above the home area 400, and the lower eye area 404 corresponds to the entire area below the home area 400. The upper eye area 402 only needs to be set at least above the center line K of the monitor 170. Similarly, the lower eye area 404 only needs to be set at least below the center line K.

[0176] When the axis point V of the pupil image 164 enters the upper eye region 402 (hereinafter referred to as the "upper position"), the eye control unit 152 moves the eyelid image 176 upward (in the direction of eye opening). Alternatively, when the pupil image 164 moves further upward in the upper position, the eye control unit 152 moves the eyelid image 176 upward in conjunction with the pupil image 164. The amount of movement of the pupil image 164 and the amount of movement of the eyelid image 176 do not need to be the same.

[0177] Similarly, when the axis point V of the pupil image 164 enters the lower eye region 404 (hereinafter referred to as the "lower position"), the eye control unit 152 moves the eyelid image 176 downward (towards the closed eye direction). Alternatively, when the pupil image 164 moves further downward in the lower position, the eye control unit 152 moves the eyelid image 176 downward in conjunction with the pupil image 164.

[0178] In this embodiment, the robot 100 is shorter than an adult, so it looks up at the user. The opportunities to look at it increase. When robot 100 looks up at a nearby user, the eyelid image 176 also rises in conjunction, making it appear to the user that "robot 100 is opening its eyes to look at the user." Also, when robot 100 sets a point of focus S on an object on the floor, it lowers its gaze. At this time, the eyelid image 176 also lowers. By lowering not only the gaze but also the eyelid image 176, it appears to the user that "robot 100 is concentrating on something." In this way, not only the pupil image 164 of robot 100, but also the eyelid image 176 can be effectively used to make robot 100 express attention and interest.

[0179] <Pupillary constriction and pupillary dilation> Figures 20(a) to 20(d) are schematic diagrams illustrating pupillary constriction when the eyes are open. Figures 21(a) to 21(d) are schematic diagrams illustrating pupillary dilation when the eyes are open. Robot 100 closes its eyes by closing its eyelids as needed for blinking or sleeping (charging). In this modified version, when the eyes are open, the pupil area 158 is reduced or expanded to actively create opportunities for the user to focus on the eye image 164 of Robot 100. Hereafter, the state in which the pupil area 158 is reduced will be called "miosis," and the state in which the pupil area 158 is expanded will be called "mydriasis." Here, the pupil area 158 is described as transitioning between three states: "miosis (small size)," "normal state (basic size)," and "mydriasis (large size)."

[0180] In Figure 20(a), the pupil region 158 is in a constricted state. As described above, the eye control unit 152 may constrict the pupil region 158 when the brightness of the external environment is above a predetermined value, and dilate the pupil region 158 when the brightness is below a predetermined value. In Figure 20(a), the pupil region 158 is constricted because the area around the robot 100 is bright. Now, let's assume that the robot 100 has closed its eyes to blink.

[0181] Figure 20(b) shows the state of the pupil region 158 when the eyes are closed. When the eyes are closed, the eye control unit 152 changes the pupil region 158 from a constricted state to a normal state (basic size). Since the eyes are closed, the user cannot see this change in the pupil region 158. Next, let's assume that the robot 100 opens its eyes. Figure 20(c) shows the state of the pupil region 158 immediately after opening the eyes. Immediately after opening the eyes, the pupil region 158 is in a normal state.

[0182] After opening the eyes, the eye control unit 152 gradually changes the pupil region 158 from its normal state to a constricted state. It returns to its original constricted state in a short time of about 0.5 to 3.0 seconds. Due to this control, the pupil region 158 of the robot 100 briefly enlarges immediately after blinking (Figure 20(c)), but then returns to its original size (constricted state) (Figure 20(d)). Immediately after blinking, the user will see the pupil region 158 shrinking.

[0183] On the other hand, in Figure 21(a), the pupil region 158 is dilated because the area around the robot 100 is dark. Now, let's assume the robot 100 closes its eyes to blink. Figure 21(b) shows the state of the pupil region 158 when the eyes are closed. When the eyes are closed, the eye control unit 152 changes the pupil region 158 from a dilated state to a normal state (basic size).

[0184] Next, the robot 100 opens its eyes. Figure 21(c) shows the state of the pupil region 158 immediately after opening its eyes. Immediately after opening its eyes, the pupil region 158 is temporarily in a normal state. After opening its eyes, the eye control unit 152 gradually changes the pupil region 158 from a normal state to a dilated state. Through this control, the pupil region 158 of the robot 100 becomes smaller after blinking (Figure 21(c)), but then returns to its original size (dilated state) (Figure 21(d)). Immediately after blinking, the user will see the pupil region 158 expanding.

[0185] As mentioned above, when robot 100 opens its eyes, the pupil area 158 changes from a normal state to a constricted state, or from a normal state to a dilated state. Because humans have an instinct to follow moving objects with their eyes, changing the pupil area 158 can create opportunities for users to focus on robot 100's eyes.

[0186] When the robot 100 has its eyes closed, in other words, when the user is not looking at it, the pupil region 158 is changed to a normal state, and after the eyes are opened, the pupil region 158 is returned to its original state (constricted or dilated pupil), so that the user is less likely to feel any discomfort. For example, in the series of controls shown in Figures 20(a) to 20(d), the pupil region 158 is basically in a constricted pupil state, and is in a normal state only immediately after the eyes are opened. When blinking occurs in a constricted pupil state, the pupil region 158 appears to expand for just a moment, but quickly returns to a constricted pupil state. Due to the slight image change from the normal state to the constricted pupil state after the eyes are opened, humans are more likely to unconsciously focus on the robot 100's eye image 174.

[0187] Robot 100 will focus on the user when the user looks at it. Therefore, the more opportunities the user has to look at Robot 100, the easier it becomes to create "moments of mutual gaze." Pupillary constriction and dilation not only enhance the lifelike appearance of Robot 100, but are also effective in actively creating opportunities for the user and Robot 100 to make eye contact.

[0188] The robot 100 may measure the brightness of the external environment using an illuminance sensor or the like. Alternatively, the robot 100 may measure the brightness from the captured image. The eye control unit 152 may constrict or dilate the pupil region 158 depending on the brightness. For example, immediately after opening the eyes as shown in Figure 21(c), if the brightness of the external environment is above a predetermined threshold, the eye control unit 152 may change the pupil region 158 from a normal state to a constricted state.

[0189] The motion control unit 150 directs the robot 100 to a charging station (not shown) when the battery 118's charge level decreases. The robot 100 closes its eyes when connecting to the charging station. The charging state may be associated with a sleep state by having the robot 100 close its eyes during charging. When the battery 118's charge level exceeds a predetermined threshold, the motion control unit 150 detaches the robot 100 from the charging station. At this time, the eye control unit 152 expresses "waking up" by opening the eye image 174.

[0190] Even when opening the eyes upon waking, the eye control unit 152 may change the size of the pupil area 158. The eye control unit 152 may also express waking by slowly moving the eyelid image 176.

[0191] The eye control unit 152 may constrict or dilate the pupil region 158 while the robot 100 is moving. Furthermore, the eye control unit 152 may also constrict or dilate the pupil region 158 when the robot 100's gaze point S does not change for a predetermined period of time or longer.

[0192] Regardless of brightness, the eye control unit 152 may constrict or dilate the pupil region 158 in response to various expansion and contraction events occurring when the eyes are open. For example, when the robot 100 opens its eyes and sees a user whose familiarity level is above a predetermined threshold, the eye control unit 152 may dilate the pupil region 158 to express the robot 100's "pleasant surprise." Alternatively, when the robot 100 opens its eyes and sees a user whose familiarity level is below a predetermined threshold (a user it does not like), or an unregistered user, the eye control unit 152 may constrict the pupil region 158 or move its gaze point S away from the user to express "awkwardness." With such a control method, the user can sense how the robot 100 perceives them.

[0193] The user may give voice commands to the robot 100, such as "close your eyes" or "open your eyes." The robot 100 detects the voice using a microphone array, and the recognition unit 156 interprets the voice command. For example, the user can keep the robot 100's eyes closed and place a gift, such as new clothes, in front of it. When the robot 100 opens its eyes, the eye control unit 152 dilates the pupil area 158 in response to the event of detecting something new. This control method allows for interactions with the robot 100, such as surprising it by showing it something new.

[0194] In Figure 20(b) (while the eyes are closed), when the eye control unit 152 returns the pupil region 158 from the constricted state to the normal state, it may do so gradually rather than instantly. In this case, the size of the pupil region 158 immediately after opening the eyes can be changed depending on the length of time the eyes are closed.

[0195] <Expressing emotions through the eyes> The eye control unit 152 may change the eye image 174 according to the emotion parameter. The eye control unit 152 may change the drift speed, amount of drift, trema amplitude, frequency, microsaccade frequency, amount of drift, and speed of drift according to the emotion parameter. The eye control unit 152 may change the shape and size of the pupil region 158 according to the emotion parameter.

[0196] The way in which the eye image 174 changes according to the emotional parameters can be arbitrarily set. For example, when the emotional parameter representing the need for approval is above a predetermined value, the eye control unit 152 may express the robot 100's anxiety or agitation by increasing the frequency of microsaccades. The extent to which the frequency of microsaccades is increased when the emotional parameter indicating the need for approval is high, or whether the frequency is increased at all, may be determined randomly.

[0197] <Continued staring> Robot 100 sets a fixation point S on the user's face, particularly the eyes. When robot 100 moves, or when the user moves, the relative positions of the user and robot 100 change. The recognition unit 156 re-detects the position of the user's face when the relative positions of the user and robot 100 change, and the eye control unit 152 resets the fixation point S to the re-detected point. In other words, the recognition unit 156 continuously tracks the fixation point S, and the eye control unit 152 continuously moves the pupil image 154 to express smooth eye movement. With this control method, robot 100 can continue to gaze at the user even when the relative positions of the user and robot 100 change.

[0198] When the relative position changes, the eye control unit 152 first moves the pupil image 164 to maintain the robot 100's gaze on the user. When the pupil image 164 moves away from the central position, the motion control unit 150 maintains the gaze by moving the robot 100's neck or entire body. By maintaining the robot 100's gaze on the user, the user can sense the robot 100's strong interest in and expectations of them.

[0199] <Release the gaze> The user may also notice that robot 100 was looking at them when it averts its gaze. The following conditions can be considered to determine when robot 100 averts its gaze from the object it is fixating on (such as the user). (C1) After the robot 100 sets the fixation point S on the user's face (object of gaze), a predetermined time When the first gaze maintenance time (hereinafter referred to as the "first gaze maintenance time") has elapsed, the robot 100 shifts its gaze away from the user by resetting the gaze point S to something other than the user's face. (C2) The robot 100 does not take its eyes off the user while the user is speaking to it. After a predetermined time has elapsed since the user finished speaking to the robot 100 (hereinafter referred to as the "second gaze maintenance time"), the robot 100 takes its eyes off the user. (C3) The robot does not avert its gaze when the user is looking at the robot 100. After a predetermined time has elapsed since the user took their eyes off the robot 100 (hereinafter referred to as the "third gaze maintenance time"), the robot 100 averts its gaze from the user. (C4) When a predetermined event occurs, the robot 100 looks away from the user. For example, this could occur when it detects a loud noise or when it sees another user. (C5) When the emotional parameters reach a predetermined state, the robot 100 averts its gaze from the user. For example, the emotional parameter indicating the need for approval may continue to rise while the robot detects the user's gaze, and when this emotional parameter exceeds a predetermined threshold, the robot 100 averts its gaze from the user.

[0200] The various gaze maintenance times described above may be variable values. The eye control unit 152 may set the gaze maintenance time to be longer the higher the user's level of familiarity with the object being gazed at. Alternatively, the eye control unit 152 may determine the length of the gaze maintenance time based on emotional parameters. For example, the eye control unit 152 may set the gaze maintenance time to be longer the higher the emotional parameter indicating the need for approval is.

[0201] Furthermore, the object being watched does not necessarily have to be the user; it could be another robot 100. Multiple robots 100 can also stare at each other.

[0202] <Formation of eye image 174 using multiple images> In this embodiment, a two-dimensional eye image 174 is generated by projecting the eyeball model 250 onto the first surface 252. As a modification, the eye image 174 may be formed by superimposing multiple images.

[0203] In this modified example, the eye image 174 is generated by superimposing five images: a white of the eye image, an iris image, a pupil image, a highlight image, and a reflection image. Furthermore, a second surface 254 (eyelid image 176) is superimposed on these five images. (L1) White of the eye image The white of the eye image is the bottommost image and corresponds to the white portion of the eye (peripheral image 168) of the eye image 174. It is a fixed image that does not track eye movement. (L2) Iris image It is superimposed on the white of the eye image. It corresponds to the iris region 162 of the eye image 174. It moves up, down, left, and right in the eye image 174 in accordance with the movement of the gaze. (L3) Pupil image It is superimposed on the iris image. It corresponds to the pupil region 158 of the eye image 174. It moves up, down, left, and right in the eye image 174 in accordance with the movement of the gaze. (L4) Highlight image It is superimposed on the pupil image. It corresponds to the catchlight 166 of the eye image 174. It does not follow the movement of the gaze. When the orientation of the robot 100 changes, the eye control unit 152 adjusts the position of the catchlight 166 by moving the highlight image in accordance with the direction of light incidence. (L5) Reflected image The highlight image is superimposed on top of the highlight image. The highlight image is the reflection in the pupil. The eye control unit 152 distorts the image captured from the front of the robot 100 like a fisheye lens and then superimposes it on top of the highlight image. The reflected image does not follow the movement of the gaze. Robot 100 When the orientation changes, the eye control unit 152 changes the reflected image to match the captured image.

[0204] In summary, the white of the eye image is fixed as the "background" for the eye image 174. The pupil and iris images move in accordance with the direction of gaze. The highlight and reflection images move according to the orientation of the robot 100, regardless of the direction of gaze. Furthermore, although the pupil and iris images move in the same direction, the amount of movement does not need to be the same. In this modified example, when the iris image is moved by a first amount in a first direction, the pupil image is moved by a second amount in the first direction. In this case, the second amount of movement is a larger value than the first amount of movement. By creating a slight discrepancy between the movement of the pupil and the movement of the iris, the lifelike and realistic feel of the eye image 174 can be further enhanced.

Claims

1. A motion control unit that selects the robot's motion, A drive mechanism that executes the motion selected by the motion control unit, The robot includes an eye control unit that displays an eye image on a display device, It comprises a recognition unit that detects a user, The aforementioned eye control unit is characterized by changing the pupil region included in the eye image according to the relative position between the user and the robot, thereby enabling autonomous behavior.

2. The autonomous robot according to claim 1, characterized in that the eye control unit changes the pupil area on the condition that the user is within a predetermined range.

3. The autonomous robot according to claim 1, characterized in that the eye control unit generates an eyeball model, which is a three-dimensional model of the eyeball, and generates the eye image by projecting the eyeball model onto a first surface.

4. The autonomous robot according to claim 3, characterized in that the eye control unit displays an eyelid image on a second surface and superimposes the eye image onto the eye image projected onto the first surface and displays it on the display device.

5. The autonomous robot according to claim 1, characterized in that the eye control unit vibrates the pupil region in the eye image.

6. The autonomous robot according to claim 5, characterized in that the eye control unit vibrates the pupil region around the axis point while moving the axis point in the eye image.

7. The autonomous robot according to claim 1, characterized in that the eye control unit expands and contracts the pupil region.

8. The recognition unit further detects the user's gaze direction, The autonomous robot according to claim 1, characterized in that the eye control unit changes the eye image when it detects the user's gaze toward the robot.

9. The autonomous robot according to claim 1, characterized in that the eye control unit causes the eye image to be displayed on the first and second display devices, respectively, and deforms the pupil region of the eye image displayed on the display device further from the user to be larger than the pupil region of the eye image displayed on the display device closer to the user.

10. The autonomous robot according to claim 1, characterized in that the motion control unit selects a motion to orient the robot's head or torso toward the user when the amount of change in the pupil area exceeds a predetermined value.

11. The autonomous robot according to claim 1, characterized in that the eye control unit selects one of several types of eye images as the control target and changes the pupil region included in the selected eye image.

12. A function to select the motion of an autonomous robot, The autonomous robot has a function to display eye images on a display device, A function to detect users, The pupil region included in the eye image is changed according to the relative position between the user and the robot. A robot control program characterized by enabling a computer to perform certain functions.

13. Camera and, A temperature detection sensor and A motion control unit that selects the robot's motion, A drive mechanism that executes the motion selected by the motion control unit, The robot includes an eye control unit that displays an eye image on a display device, The system includes a recognition unit that detects a user based on an image captured by the camera or a heat distribution image acquired by the temperature detection sensor, The recognition unit selects either the captured image or the heat distribution image according to the positional relationship between the robot and the user, and identifies the user's face region from the selected image. The aforementioned eye control unit sets the identified facial region as a fixation point and moves the pupil region included in the eye image according to the fixation point, thereby enabling autonomous robotic behavior.

14. A motion control unit that selects the robot's motion, A drive mechanism that executes the motion selected by the motion control unit, The robot includes an eye control unit that displays an eye image on a display device, It includes a recognition unit that detects the user's eyes, An autonomous robot characterized in that the eye control unit sets the user's eyes as a fixation point and moves the pupil region included in the eye image according to the fixation point to direct the robot's gaze toward the user's eyes, and maintains the robot's gaze toward the user's eyes by setting the changed position of the user's eyes as a new fixation point when the relative positions of the robot and the user change.

15. A motion control unit that selects the robot's motion, A drive mechanism that executes the motion selected by the motion control unit, The robot includes an eye control unit that displays an eye image on a display device, It comprises a recognition unit that detects a user, The recognition unit further detects the user's gaze direction, The eye control unit is characterized by setting the user's eye as the fixation point, moving the pupil region included in the eye image according to the fixation point to direct the robot's gaze toward the user's eye, and moving the fixation point away from the user's eye when the user's gaze is moved away from the robot.

Images