Robot, method of controlling the same, and program

Abstract

To provide a robot, a method of controlling the same, and a program which allow a user to control movement of the robot.SOLUTION: A robot according to the present invention has a movement mechanism, an imaging unit for imaging a peripheral space, a marker recognition unit for recognizing a predetermined marker included in a captured image obtained by the imaging unit, and a movement control unit for controlling movement by the movement mechanism, based on the recognized marker.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a robot, a control method thereof, and a program. [Background technology]

[0002] Conventionally, there have been robots that autonomously move around a house, capture images with a camera, recognize the indoor space from the captured images, and set a movement path based on the recognized space to move around the house. The robot's movement path is set by a user creating in advance a map that defines the path the robot will follow. The robot can move along the path that is set based on the created map (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2016-103277 Summary of the Invention [Problem to be solved by the invention]

[0004] However, since an autonomous robot can move autonomously through space, it may enter areas where the user does not want the robot to enter or areas where it is dangerous for the robot to move.

[0005] In addition to autonomous movement, there are also cases where it is desired to move a robot to a predetermined location and have the robot perform a predetermined action.

[0006] The present invention has been made in consideration of the above circumstances, and in one embodiment, an object is to provide a robot, a control method thereof, and a program that allow a user to control the movement of the robot. [Means for solving the problem]

[0007] (1) In order to solve the above problems, the robot of the embodiment includes a moving mechanism, a photographing unit that photographs the surrounding space, a marker recognition unit that recognizes a predetermined marker included in the photographed image photographed by the photographing unit, and a movement control unit that controls the movement of the moving mechanism based on the recognized marker.

[0008] (2) In the robot of the embodiment, the movement control unit prohibits entry by movement based on the recognized marker.

[0009] (3) In the robot of the embodiment, the movement control unit limits the speed of the movement based on the recognized marker.

[0010] (4) In the robot of the embodiment, the movement control unit controls the movement based on the recognized installation position of the marker.

[0011] (5) In the robot of the embodiment, the movement control unit sets a limit range based on the installation position and limits the movement within the limit range.

[0012] (6) In the robot of the embodiment, the movement control unit sets the limit range to a predetermined range behind the installation position or around the installation position.

[0013] (7) In the robot of the embodiment, when a plurality of the markers are recognized, the movement control unit restricts the movement based on the recognized installation positions of the plurality of markers.

[0014] (8) In the robot of the embodiment, the movement control unit limits the movement based on a line segment connecting the recognized installation position of the first marker and the recognized installation position of the second marker.

[0015] (9) In the robot of the embodiment, the movement control unit controls the movement based on the type of the recognized marker.

[0016] (10) In the robot of the embodiment, the movement control unit controls the movement based on the recorded markers.

[0017] (11) In addition, in the robot of the embodiment, when the marker is not recognized in the captured image, the movement control unit controls the movement based on the recorded marker.

[0018] (12) In addition, the robot of the embodiment further includes a spatial data generation unit that generates spatial data that recognizes the space based on the captured image taken by the photographing unit, a visualization data generation unit that generates visualization data that visualizes spatial elements included in the space based on the generated spatial data, and a visualization data provision unit that provides the generated visualization data to a user terminal.

[0019] (13) In addition, in the robot of the embodiment, a designation acquisition unit is further provided that acquires from the user terminal a designation of an area included in the provided visualization data, and the spatial data generation unit re-recognizes the space based on the captured image re-photographed in the area related to the acquired designation.

[0020] (14) In addition, the robot of the embodiment further includes a state information acquisition unit that acquires state information indicating the state of the destination of the movement, and the movement control unit controls the movement further based on the state information.

[0021] (15) In addition, the robot of the embodiment further includes a marker information storage unit that stores the position of the marker, a first event detection unit that detects a first event, a second event detection unit that detects a second event, and an action execution unit that executes an action, and when the first event is detected, the robot moves to the vicinity of the position of the marker, and when the second event is detected, the robot executes the action corresponding to at least one of the marker, the first event, and the second event.

[0022] (16) In order to solve the above problem, the robot control method of the embodiment includes an imaging step of imaging the surrounding space, a marker recognition step of recognizing a predetermined marker included in the image captured in the imaging step, and a movement control step of controlling movement by a moving mechanism based on the recognized marker.

[0023] (17) In order to solve the above problem, the robot control program of the embodiment causes a computer to realize a photographing function for photographing the surrounding space, a marker recognition function for recognizing a specified marker contained in the photographed image by the photographing function, and a movement control function for controlling movement by a moving mechanism based on the recognized marker. [Effects of the Invention]

[0024] According to one embodiment, it is possible to provide a robot, a control method thereof, and a program that allow a user to control the movement of the robot. [Brief explanation of the drawings]

[0025] [Figure 1] FIG. 2 is a block diagram showing an example of the software configuration of the autonomously acting robot according to the first embodiment. [Figure 2] 1 is a block diagram showing an example of a hardware configuration of an autonomously acting robot according to a first embodiment. FIG. [Figure 3] 4 is a flowchart showing an example of the operation of the autonomously acting robot control program in the first embodiment. [Figure 4] 10 is a flowchart showing another example of the operation of the autonomously acting robot control program in the first embodiment. [Figure 5] FIG. 4 is a diagram showing a method for setting no-entry lines in the first embodiment. [Figure 6] FIG. 2 is a diagram showing an example of a display on a user terminal in the first embodiment. [Figure 7] FIG. 2 is a diagram showing an example of a display on a user terminal in the first embodiment. [Figure 8]FIG. 10 is a block diagram showing an example of a module configuration of a robot according to a second embodiment. [Figure 9] FIG. 10 is a block diagram showing an example of a module configuration of a data providing device according to a second embodiment. [Figure 10] FIG. 10 is a block diagram showing an example of a data configuration of an event information storage unit in the second embodiment. [Figure 11] FIG. 10 is a block diagram showing an example of a data configuration of a marker information storage unit in the second embodiment. [Figure 12] 12A is a flowchart showing a processing procedure in a marker registration phase according to the second embodiment. FIG. 12B is a flowchart showing a processing procedure in an action phase according to the second embodiment. [Figure 13] 13A is a flowchart showing a processing procedure in a marker registration phase according to the first embodiment, and FIG. 13B is a flowchart showing a processing procedure in an action phase according to the first embodiment. [Figure 14] 14A is a flowchart showing a processing procedure in a marker registration phase according to the second embodiment, and FIG. 14B is a flowchart showing a processing procedure in an action phase according to the second embodiment. [Figure 15] 15A is a flowchart showing a processing procedure in a marker registration phase according to the third embodiment, and FIG. 15B is a flowchart showing a processing procedure in an action phase according to the third embodiment. [Figure 16] 16A is a flowchart showing a processing procedure in a marker registration phase according to the fourth embodiment, and FIG. 16B is a flowchart showing a processing procedure in an action phase according to the fourth embodiment. [Figure 17] 17A is a flowchart showing a processing procedure in a marker registration phase according to the fifth embodiment, and FIG. 17B is a flowchart showing a processing procedure in an action phase according to the fifth embodiment. [Figure 18]Fig. 18(A) is a flowchart showing a processing procedure in a marker registration phase according to the sixth embodiment. Fig. 18(B) is a flowchart showing a processing procedure in an action phase according to the sixth embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0026] Hereinafter, an autonomously acting robot, a data providing device, and a data providing program according to an embodiment of the present invention will be described in detail with reference to the drawings.

[0027] [Embodiment 1] First, the software configuration of the autonomously acting robot 1 will be described with reference to Fig. 1. Fig. 1 is a block diagram showing an example of the software configuration of the autonomously acting robot 1 in the embodiment.

[0028] In FIG. 1, the autonomously acting robot 1 includes a data providing device 10 and a robot 2. The data providing device 10 and the robot 2 are connected via communication and function as the autonomously acting robot 1. The robot 2 is a mobile robot having the following functional units: a photographing unit 21, a marker recognition unit 22, a movement control unit 23, a state information acquisition unit 24, and a movement mechanism 29. The data providing device 10 has the following functional units: a first communication control unit 11, a point cloud data generation unit 12, a spatial data generation unit 13, a visualization data generation unit 14, a photographed target recognition unit 15, and a second communication control unit 16. The first communication control unit 11 has the following functional units: a photographed image acquisition unit 111, a spatial data providing unit 112, and an instruction unit 113. The second communication control unit 16 has the following functional units: a visualization data providing unit 161 and a specification acquisition unit 162. In this embodiment, the above-mentioned functional units of the data providing device 10 of the autonomously acting robot 1 will be described as functional modules implemented by a data provision program (software) that controls the data providing device 10. Furthermore, the functional units of the robot 2, ie, the marker recognition unit 22, the movement control unit 23, and the state information acquisition unit 24, will be described as functional modules realized by a program that controls the robot 2 in the autonomously acting robot 1.

[0029] The data providing device 10 is a device that can execute some of the functions of the autonomously acting robot 1, and is, for example, an edge server that is installed in a location physically close to the robot 2, communicates with the robot 2, and distributes the processing load of the robot 2. In this embodiment, the autonomously acting robot 1 is described as being configured by the data providing device 10 and the robot 2. However, the functions of the data providing device 10 may be included in the functions of the robot 2. The robot 2 is a robot that can move based on spatial data, and is one type of robot whose movement range is determined based on the spatial data. The data providing device 10 may be configured in one housing or multiple housings.

[0030] The first communication control unit 11 controls the communication function with the robot 2. Any communication method with the robot 2 may be used, and for example, short-range wireless communication such as wireless LAN (Local Area Network), Bluetooth (registered trademark), or infrared communication, or wired communication may be used. The functions of the captured image acquisition unit 111, the spatial data provision unit 112, and the instruction unit 113 of the first communication control unit 11 communicate with the robot 2 using the communication functions controlled by the first communication control unit 11.

[0031] The photographed image acquisition unit 111 acquires photographed images taken by the photographing unit 21 of the robot 2. The photographing unit 21 is provided in the robot 2 and can change the photographing range in accordance with the movement of the robot 2. Here, the photographing unit 21, marker recognition unit 22, movement control unit 23, state information acquisition unit 24, and movement mechanism 29 of the robot 2 will be described.

[0032] The photographing unit 21 may be configured with one or more cameras. For example, if the photographing unit 21 is a stereo camera configured with two cameras, the photographing unit 21 can photograph the spatial element, which is the photographing target, in three dimensions from different photographing angles. The photographing unit 21 is, for example, a video camera using an imaging element such as a CCD (Charge-Coupled Device) sensor or a CMOS (Complementary Metal Oxide Semiconductor) sensor. By photographing the spatial element with two cameras (stereo camera), the shape of the spatial element can be measured. The photographing unit 21 may also be a camera using ToF (Time of Flight) technology. In a ToF camera, the shape of the spatial element can be measured by irradiating the spatial element with modulated infrared light and measuring the distance to the spatial element. The photographing unit 21 may also be a camera using structured light. Structured light is a light that projects light in a striped or grid-like pattern onto the spatial element. By photographing the spatial element from a different angle from the structured light, the photographing unit 21 can measure the shape of the spatial element from the distortion of the projected pattern. The photographing unit 21 may be any one of these cameras, or a combination of two or more of them.

[0033] The photographing unit 21 is attached to the robot 2 and moves in accordance with the movement of the robot 2. However, the photographing unit 21 may be installed separately from the robot 2.

[0034] The photographed images taken by the photographing unit 21 are provided to the photographed image acquiring unit 111 in a communication method corresponding to the first communication control unit 11. The photographed images are temporarily stored in a memory unit of the robot 2, and the photographed image acquiring unit 111 acquires the temporarily stored photographed images in real time or at predetermined communication intervals.

[0035] The marker recognition unit 22 recognizes a predetermined marker included in the captured image taken by the photographing unit 21. A marker is a spatial element that indicates a restriction on the movement of the robot 2. A marker is the shape, pattern, or color of an object that can be recognized from a photographed image, or a character or figure attached to an object, or a combination of these. By placing a marker in a position that restricts the movement of the robot 2, the marker is photographed together with furniture, etc. when the robot 2 photographs the space with the photographing unit 21. The marker may be a flat or three-dimensional object. For example, a marker may be a sticker or a label printed with a two-dimensional code or a specific color combination or shape. The marker may be a piece of paper, etc. The marker may also be an ornament or rug of a specific color or shape. By using printed materials or everyday objects as markers in this way, the user does not need to secure a power source for the marker or a place to install it. Furthermore, the user can restrict the robot's movement at will without spoiling the atmosphere of the room. Furthermore, since the user can see the marker, the user can intuitively understand the movement restriction range and can easily change the restriction range. The marker is installed by the user, for example, by sticking it on a wall or furniture, or by placing it on the floor. The marker recognition unit 17 can recognize that the movement of the robot 2 is restricted by recognizing the image of the marker included in the captured image.

[0036] Here, if the marker is planar, it can be attached to a wall or furniture, allowing for space-saving installation. If the marker is planar, capturing an image of the marker's plane from a horizontal direction (when the shooting angle is small) distorts the marker in the captured image, making it difficult to recognize. On the other hand, capturing an image of the marker's plane from a vertical direction (when the shooting angle is large) makes it easier to recognize the marker. Therefore, for example, if a marker is attached in a hallway, the shooting angle is small at positions far from the marker, so robot 2 can prevent the marker from being recognized. As the robot moves down the hallway and approaches the marker, the shooting angle becomes larger, allowing the marker to be recognized. Therefore, with a planar marker, the marker's installation position (described below) can be made closer to the position at which the robot can recognize the marker, allowing the robot to accurately determine the installation position of the marker. Furthermore, if the marker is three-dimensional, it can be easily installed in the center of a room, for example. If the marker is three-dimensional, the marker can be recognized from various shooting angles. Therefore, by installing a three-dimensional marker, it becomes possible for a robot 2 located far from the installation position of the marker to recognize the marker.

[0037] The marker recognition unit 22 stores the visual characteristics of the marker in advance. For example, the marker recognition unit 22 stores in advance two-dimensional codes or three-dimensional objects to be recognized as markers. The marker recognition unit 22 may recognize an object registered in advance by a user as a marker. For example, if a user registers a flowerpot photographed by the camera of the user terminal 3 as a marker, the flowerpot installed in a hallway or the like can be recognized as a marker. This allows the user to install an object that does not look out of place in the location where the marker is to be installed. The marker recognition unit 22 may also recognize spatial elements other than objects as markers. For example, the marker recognition unit 22 may recognize a user gesture, such as the user crossing their arms in front of their body, as a marker. The marker recognition unit 22 recognizes the position where the user makes the gesture as the installation position of the marker.

[0038] The marker recognition unit 22 recognizes the position where a marker is affixed or installed (hereinafter referred to as the "installation position"). The installation position is the position in space where the marker is installed in the spatial data. The installation position can be recognized, for example, based on the distance between the current position of the robot 2 and the captured marker, based on the spatial data recognized by the robot 2. For example, if the size of the marker is known in advance, the marker recognition unit 22 can calculate the distance between the robot 2 and the marker from the size of the marker image included in the captured image, and recognize the installation position of the marker based on the current position of the robot 2 and the capture direction (for example, the direction measured by a compass, not shown). The installation position may also be recognized from the relative position of the marker from a spatial element whose position in space is already known. For example, if the position of a door is already known, the marker recognition unit 22 may recognize the installation position from the relative position of the marker and the door. Furthermore, if the captured image is captured by a depth camera, the installation position can be recognized based on the imaging depth of the marker captured by the depth camera.

[0039] The marker recognition unit 22 may also recognize a plurality of markers included in the captured image. For example, when a user wants to set a range in which movement is restricted as a straight line, the user may use a first marker and a second marker. A marker consisting of a pair of two markers can be placed. The marker recognition unit 22 may recognize the position of a line segment (straight or curved) connecting the start point and end point by recognizing the placement position (start point) of the first marker and the placement position (end point) of the second marker. The marker recognition unit 22 can recognize the position of the line segment in the spatial data by mapping the positions of the first marker and the second marker to the spatial data. A user can easily set a line segment that restricts movement by placing a marker at a predetermined position. Three or more markers may be placed. For example, if there are three or more markers, the marker recognition unit 22 can recognize a broken line or a polygon (area) based on the placement positions of each marker.

[0040] The movement control unit 23 restricts movement based on the installation positions of the markers recognized by the marker recognition unit 22. The movement control unit 23 includes a restricted range setting unit 231 that sets a restricted range for restricting movement according to the installation positions of the recognized markers. The movement control unit 23 restricts movement of the robot 2 within the restricted range set by the restricted range setting unit 231. The installation positions of the markers are points, lines, surfaces, or spaces that are set based on the installation positions of one or more markers. The restricted range setting unit 231 can set the restricted range by recognizing the installation positions of the markers as, for example, coordinate data in spatial data. The restricted range setting unit 231 may set a restricted range based on the installation positions and restrict movement within the restricted range. For example, the restricted range setting unit 231 can set a line segment that separates spatial elements such as a hallway, or a circular or spherical area in space centered on the marker, as the restricted range for restricting movement, based on the installation position of a single marker. That is, the limited range setting unit 231 sets the limited range by arranging a geometrically determined range, such as a rectangle, a circle, or a line, in space based on the installation position of the marker. For example, if a circular range is set, the limited range setting unit 231 may set the limited range as a circular range with a predetermined radius centered on the installation position of the marker. Also, if a rectangular range is set, the limited range setting unit 231 may determine the rectangular limited range by arranging the installation position of the marker so that it is at the center of one side of the rectangle. The limited range is, for example, approximately 1 to 3 m from the marker, which is narrower than the range in which the marker recognition unit 22 can recognize the marker. The limited range may be predetermined for each marker, or may be arbitrarily adjusted by the user using an application described below.

[0041] Furthermore, the limited range setting unit 231 may set a line, a plane, or a space defined by a plurality of markers as the limited range. For example, the limited range setting unit 231 may set a predetermined range behind the installation position of the marker or around the installation position as the limited range, based on the position of the robot 2 when the marker recognition unit 22 recognized the marker. When the limited range setting unit 231 sets the limited range in a line, the movement control unit 23 limits the movement of the robot 2 so as not to go beyond that line. In this way, the limited range setting unit 231 may set the limited range based on a predetermined rule, based on the installation position of the marker.

[0042] Furthermore, the restricted range setting unit 231 may recognize spatial characteristics around the marker and set the restricted range according to the spatial characteristics. In other words, the restricted range setting unit 231 may recognize the floor plan and set the restricted range according to the floor plan. For example, if the marker is located near the entrance to a passage (within a predetermined range), the restricted range setting unit 231 may set the passage as the restricted range. Furthermore, if the marker is placed in the center of the room (a predetermined distance from the wall), the restricted range setting unit 231 may set a circular range centered on the marker as the restricted range. Furthermore, if the marker is attached to a wall and there is no door nearby, the restricted range setting unit 231 may set a predetermined range from the wall as the restricted range.

[0043] A type may be set for the marker. For example, a marker that restricts movement only when the marker is visible (called a "temporary marker") and a marker that stores the position of the marker and permanently restricts movement even when the marker is not visible (called a "permanent marker") may be set as the type of marker. When the robot 2 visually recognizes a permanent marker, the robot 2 stores the position of the marker in a storage unit (not shown), and even if the marker disappears from that location, its movement is restricted based on the stored position of the marker. Furthermore, when the robot 2 visually recognizes a temporary marker, the robot 2 does not store the position of the temporary marker, and therefore the restricted range is lifted when the temporary marker is removed.

[0044] The marker recognition unit 22 recognizes the type of the set marker. The type of marker can be pre-classified by, for example, the shape, pattern, color, text, or graphic of the marker, or a combination of these. The type of marker may also be classified by the number of markers installed or the installation method (e.g., installing the markers upside down). When the marker includes a two-dimensional code, information identifying the type of marker is written in the two-dimensional code. In this case, the marker recognition unit 22 can identify whether the marker is temporary or permanent by reading the two-dimensional code. Furthermore, identification information identifying the marker (referred to as "marker identification information") may be written in the two-dimensional code. In this case, the marker recognition unit 22 reads the marker identification information from the two-dimensional code and identifies the type of marker associated with the marker identification information by referring to a pre-prepared table.

[0045] That is, the marker recognition unit 22 may be configured to read the accompanying information from the marker itself when the marker itself includes accompanying information, such as a two-dimensional code. Alternatively, the marker recognition unit 22 may be configured to read marker identification information from the marker and read the accompanying information by referencing a table using the marker identification information as a key. In this embodiment, the marker recognition unit 22 has a marker information storage unit (not shown) that stores accompanying information of the marker in association with the marker identification information, and is configured to acquire the accompanying information for each marker by referencing the marker information storage unit. The marker recognition unit 22 may read the marker identification information from the two-dimensional code, or may acquire the marker identification information by identifying the marker using general object recognition.

[0046] In this way, by configuring the system to manage information associated with a marker, it is possible not only to set a restricted area around the marker and restrict the movement of the robot 2, but also to restrict the behavior of the robot 2 under various conditions (referred to as "behavior restriction") according to the user's wishes. For example, if you do not want the robot 2 to enter the dressing room when the bathroom is in use, you can associate a time period during which entry is prohibited with the marker. Also, if you do not want the robot 2 to enter the kitchen when the kitchen is in use, you can associate a condition that prohibits entry when a person is present (when a person is detected) with the marker. Furthermore, even if the robot is allowed to enter, rather than prohibiting its entry, instructions that restrict the robot's behavior, such as requiring it to be quiet and restrained within the restricted area, not making noise, or moving slowly, may be associated as the associated information. In other words, the associated information may include information that identifies the type of marker or information that specifies the behavior of the robot within the restricted area. The information that specifies the behavior is information for restricting the robot's behavior, and if movement within the restricted area is prohibited, information specifying the prohibited time period may be included in addition to the prohibition. If movement within a restricted area is permitted under certain conditions, the accompanying information may include information specifying the conditions (referred to as "action conditions") in addition to the permission.

[0047] If the marker is not recognized by the marker recognition unit 22, the movement control unit 23 may restrict movement based on the stored installation position. For example, if the command by the marker is to set a permanent marker that sets a permanent restriction, the movement control unit 23 permanently restricts movement based on the marker even if the marker has been removed and cannot be recognized from the captured image. Note that the marker set by the restriction range setting unit 231 may be edited, for example, by instructions from the user terminal 3, such as erasing the marker, changing its position, or changing the command. For example, the user terminal 3 may use an application (not shown) that can edit the marker. The robot 2 may have an application program (hereinafter referred to as "app"). For example, the app may display selectable markers on the display screen of the user terminal 3 and allow the user to edit the selected marker. The app may also change a temporary marker to a permanent marker. This allows the user to cancel the restricted range by removing the installed temporary marker, and to maintain the restricted range even after the installed marker is removed by changing the temporary marker to a permanent marker using the app. The app may also have a registration function for registering spatial elements captured by the camera of the user terminal 3 as markers. The app may also have a function for adjusting the restricted range or setting or changing the content of the above-mentioned behavioral restrictions. The app may also have a function for connecting to a marker information storage unit of the robot 2 and referring to and updating the behavioral restrictions and behavioral conditions for each marker.

[0048] The restriction on the movement of the robot 2 set by the markers can coexist with the setting of a restricted range by the status information, which will be described later. For example, a no-entry area in a hallway can be set by installing markers, and entry to a changing room can be prohibited by the status information. Furthermore, the area where movement is restricted can be set by the markers, and the content of the restriction in that area (for example, conditions such as the time during which entry is restricted) can be set by the status information.

[0049] The status information acquisition unit 24 acquires status information indicating the status of a destination during movement. The status information is information for restricting the movement of the robot 2 according to the status of the destination detected by the robot 2. The status of the destination may include, for example, the presence or absence of people or pets in the movement range, the temperature or humidity of the room, the locked status of the door, or the on / off status of the lights, and may also include the time, day of the week, weather, and other conditions. The status information is, for example, information for restricting the movement speed in an area (range) when a person is detected in the movement range. The status information may also prohibit entry into the area on a specific day or time, prohibit movement through a locked door, or prohibit photography in an area where the lights are on. The status information can be provided together with spatial data.

[0050] The data providing device 10 will be described again. The spatial data providing unit 112 provides the robot 2 with spatial data generated by the spatial data generating unit 13. The spatial data is a digital representation of spatial elements recognized by the robot 2 in the space in which the robot 2 exists. The robot 2 can move within a range defined in the spatial data. In other words, the spatial data functions as a map for determining the movable range of the robot 2. The robot 2 is provided with spatial data from the spatial data providing unit 112. For example, the spatial data may include position data of spatial elements such as walls, furniture, electrical appliances, and steps that the robot 2 cannot move through. Based on the provided spatial data, the robot 2 can determine whether it is within a location where it can move. The robot 2 may also be able to recognize whether the spatial data includes an ungenerated area. Whether an ungenerated area is included can be determined, for example, by whether part of the spatial data includes a space without spatial elements.

[0051] The instruction unit 113 instructs the robot 2 to take a photograph based on the spatial data generated by the spatial data generation unit 13. The spatial data generation unit 13 creates spatial data based on the captured image acquired by the captured image acquisition unit 111. For example, when creating spatial data for an indoor space, the spatial data may include a portion for which no image has been captured. Furthermore, if the captured image is unclear, noise may be included in the created spatial data, resulting in an inaccurate portion of the spatial data. If there is a portion in the spatial data that has not been generated, the instruction unit 113 may issue an instruction to capture the ungenerated portion. Furthermore, if the spatial data contains inaccurate parts, the instruction unit 113 may instruct the robot 2 to capture the inaccurate parts. The instruction unit 113 may also instruct the robot 2 to capture the images voluntarily based on the spatial data. The instruction unit 113 may also instruct the robot 2 to capture the images based on an explicit instruction from a user who has checked visualized data (described later) generated based on the spatial data. The user can specify an area included in the visualized data and instruct the robot 2 to capture the images, thereby recognizing the space and generating spatial data.

[0052] The point cloud data generation unit 12 generates three-dimensional point cloud data of spatial elements based on the captured images acquired by the captured image acquisition unit 111. The point cloud data generation unit 12 generates point cloud data by converting the spatial elements included in the captured images into a set of three-dimensional points in a predetermined space. As described above, the spatial elements include the walls, steps, doors, furniture, home appliances, luggage, and houseplants in a room. Since the point cloud data generation unit 12 generates point cloud data based on the captured images of the spatial elements, the point cloud data represents the surface shapes of the captured spatial elements. The captured images are generated by the imaging unit 21 of the robot 2 capturing images at a predetermined imaging angle from a predetermined imaging position. Therefore, if the robot 2 captures images of spatial elements such as furniture from a frontal position, point cloud data cannot be generated for the shape of the back side of the furniture that is not captured in the image. Therefore, even if there is a space behind the furniture where the robot 2 can move, the robot 2 cannot recognize it. On the other hand, if the robot 2 moves and photographs the furniture from a side photographing position, point cloud data can be generated for the shape of the back side of the spatial element such as furniture, making it possible to correctly grasp the space.

[0053] The spatial data generation unit 13 generates spatial data that defines the movable range of the robot 2 based on the point cloud data of the spatial elements generated by the point cloud data generation unit 12. Because the spatial data is generated based on the point cloud data in space, the spatial elements included in the spatial data also have three-dimensional coordinate information. The coordinate information may include information on the position, length (including height), area, or volume of the points. The robot 2 can determine the movable range based on the position information of the spatial elements included in the generated spatial data. For example, if the robot 2 has a movement mechanism 29 that moves horizontally on the floor, the robot 2 can determine that it cannot move if the difference in height from the floor, which is a spatial element in the spatial data, is equal to or greater than a predetermined height (e.g., 1 cm or more). On the other hand, if the spatial element in the spatial data, such as a tabletop or bed, is at a predetermined height from the floor, the robot 2 determines that a range equal to or greater than a predetermined height from the floor (e.g., 60 cm or more) is a movable range, taking into account the clearance from its own height. Furthermore, the robot 2 determines in the spatial data that a range where the gap between a wall and furniture, which is a spatial element, is equal to or greater than a predetermined width (for example, equal to or greater than 40 cm) is a movable range, taking into account the clearance with respect to its own width.

[0054] The space data generation unit 13 may set attribute information for a predetermined area in space. The attribute information is information that defines the movement conditions of the robot 2 for the predetermined area. The movement conditions are, for example, conditions that define the clearance between the robot 2 and spatial elements within which the robot 2 can move. For example, if the normal movement condition within which the robot 2 can move is a clearance of 30 cm or more, attribute information can be set that defines a clearance of 5 cm or more for the predetermined area. The movement conditions set in the attribute information may also include information that restricts the movement of the robot. Restrictions on movement include, for example, a restriction on movement speed or prohibition of entry. For example, attribute information that reduces the movement speed of the robot 2 in areas with small clearance or areas where people are present may be set. The movement conditions set in the attribute information may also be determined by the floor material of the area. For example, the attribute information may set changes to the operation (travel speed, travel means, etc.) of the movement mechanism 29 when the floor is cushion flooring, flowing long, tatami, or carpet. The attribute information may also include charging spots where the robot 2 can move and charge, and areas where the robot 2's posture is unstable. It may be possible to set a limiting condition for steps or carpet edges where movement is restricted due to the obstacles. The areas for which attribute information has been set may be made visible to the user by changing the display method in visualized data, which will be described later, for example.

[0055] The spatial data generation unit 13 performs, for example, a Hough transform on the point cloud data generated by the point cloud data generation unit 12 to extract figures such as lines and curves common to the point cloud data, and generates spatial data based on the contours of spatial elements represented by the extracted figures. The Hough transform is a coordinate transformation method that extracts a figure that passes through the most feature points when the point cloud data are feature points. Because point cloud data represents the shapes of spatial elements such as furniture placed in a room in a point cloud, users may have difficulty identifying the spatial elements represented by the point cloud data (e.g., recognizing a table, chair, wall, etc.). By performing a Hough transform on the point cloud data, the spatial data generation unit 13 can represent the contours of furniture, etc., making it easier for users to identify the spatial elements. Note that the spatial data generation unit 13 may generate spatial data by converting the point cloud data generated by the point cloud data generation unit 12 into basic shapes of spatial elements (e.g., a table, chair, wall, etc.) recognized by image recognition. By recognizing a spatial element such as a table as a table through image recognition, the shape of the table can be accurately predicted from point cloud data of a portion of the spatial element (for example, point cloud data of the table viewed from the front). By combining point cloud data with image recognition, the spatial data generator 13 can generate spatial data that accurately captures the spatial element.

[0056] The spatial data generation unit 13 generates spatial data based on point cloud data included within a predetermined range from the position to which the robot 2 has moved. The predetermined range from the position to which the robot 2 has moved includes the actual position to which the robot 2 has moved, and may be, for example, a range 30 cm from the position to which the robot 2 has moved. Since the point cloud data is generated based on images captured by the camera unit 21 of the robot 2, the captured images may include spatial elements located far from the robot 2. If the spatial elements are far from the camera unit 21, there may be areas not captured in the image, or there may be areas where the robot 2 cannot actually move due to the presence of obstacles not captured in the image. Furthermore, if a spatial element located far from the camera unit 21 is included in the captured image, such as a hallway, the spatial element extracted at the feature point may be distorted. Furthermore, if the shooting distance is long, the spatial element included in the captured image may be small, which may reduce the accuracy of the point cloud data. The spatial data generation unit 13 may generate spatial data that does not include low-accuracy or distorted spatial elements by ignoring feature points that are far away. The spatial data generation unit 13 generates spatial data by deleting point cloud data outside a predetermined range from the position to which the robot 2 has moved, thereby preventing the occurrence of outlying areas where no data actually exists, and making it possible to generate spatial data with high data accuracy that does not include areas where the robot 2 cannot move.In addition, it is possible to prevent outlying areas from being drawn in the visualization data generated from the spatial data, thereby improving visibility.

[0057] When a marker is recognized by the marker recognition unit 22, the spatial data generation unit 13 sets a limit range for the generated spatial data. By setting a limit range for the spatial data, it becomes possible to visualize the limit range as part of the visualization data. Furthermore, when state information is acquired by the state information acquisition unit 24, the spatial data generation unit 13 sets state information for the spatial data. By setting state information for the spatial data, it becomes possible to make the state information part of the visualization data.

[0058] The visualization data generation unit 14 generates visualization data based on the spatial data generated by the spatial data generation unit 13, which visualizes spatial elements included in the space so that people can intuitively distinguish them.

[0059] Generally, robots have various sensors, such as cameras and microphones, and recognize their surroundings by comprehensively assessing information obtained from these sensors. For a robot to move, it needs to recognize various objects in space and determine a movement route based on spatial data. However, a robot's movement route may be inappropriate due to an inability to correctly recognize objects. Misrecognition can lead to, for example, a robot perceiving an area as narrow due to obstacles, even when a human believes there is a sufficiently large space. When this type of misrecognition occurs between a human and a robot, the robot behaves in a way that contradicts the human's expectations, causing stress for the human. To reduce the misrecognition between humans and robots, the autonomous robot of this embodiment visualizes spatial data representing its own recognition state and provides it to the human, and can re-perform recognition processing on areas pointed out by the human.

[0060] The spatial data is data including spatial elements recognized by the autonomously acting robot 1, whereas the visualized data is data that allows a user to visually recognize the spatial elements recognized by the autonomously acting robot 1. The spatial data may include spatial elements that have been misrecognized. By visualizing the spatial data, it becomes easier for a person to check the recognition state of the spatial elements in the autonomously acting robot 1 (presence or absence of misrecognition, etc.).

[0061] The visualization data is data that can be displayed on a display device. The visualization data is a so-called floor plan, and spatial elements recognized as tables, chairs, sofas, etc. are included in an area surrounded by spatial elements recognized as walls. The visualization data generation unit 14 generates visualization data expressed, for example, as RGB data, of the shapes of furniture and other objects formed in figures extracted by the Hough transform. The spatial data generation unit 13 generates visualization data in which the plane rendering method is changed based on the three-dimensional plane direction of the spatial element. The three-dimensional plane direction of the spatial element is, for example, the normal direction of the plane formed by the figure generated in the point cloud data generated by the point cloud data generation unit 12 through a Hough transform. The visualization data generation unit 14 generates visualization data in which the plane rendering method is changed depending on the normal direction. The rendering method may be, for example, color attributes such as hue, brightness, or saturation to be applied to the plane, or a pattern or texture to be applied to the plane. For example, if the normal of a plane is vertical (the plane is horizontal), the visualization data generation unit 14 increases the brightness of the plane and draws it in a bright color. On the other hand, if the normal of the plane is horizontal (the plane is vertical), the visualization data generation unit 14 decreases the brightness of the plane and draws it in a dark color. By changing the plane drawing method, it is possible to represent the shape of furniture, etc. in three dimensions, making it easier for users to recognize the shape of furniture, etc. Furthermore, the visualization data may include coordinate information in the visualization data (referred to as "visualization coordinate information") associated with the coordinate information of each spatial element included in the spatial data. Because the visualization coordinate information is associated with the coordinate information, points in the visualization coordinate information correspond to points in the actual space, and surfaces in the visualization coordinate information correspond to surfaces in the actual space. Therefore, when a user identifies the position of a point in the visualization data, the user can identify the corresponding position of a point in the actual room. Furthermore, a conversion function for converting coordinate systems may be provided to enable conversion between the coordinate system in the visualization data and the coordinate system in the spatial data. Of course, it may be possible to convert between the coordinate system in the visualization data and the coordinate system in the actual space.

[0062] The visualization data generation unit 14 generates the visualization data as three-dimensional (3D (Dimensions)) data. Alternatively, the visualization data generation unit 14 may generate the visualization data as planar (2D) data. Generating the visualization data in 3D makes it easier for the user to check the shape of furniture, etc. The visualization data generation unit 14 may generate the visualization data in 3D when the spatial data generation unit 13 has generated enough data for generating the visualization data in 3D. The visualization data generation unit 14 generates the visualization data in 3D when the spatial data generation unit 13 has generated enough data for generating the visualization data in 3D. The visualization data may be generated in 3D based on a specified 3D viewpoint position (such as viewpoint height and viewpoint elevation / depression angle). By allowing the viewpoint position to be specified, the user can more easily confirm the shape of furniture, etc. Furthermore, the visualization data generation unit 14 may generate visualization data in which only the far wall of the room is colored and the near wall or ceiling is transparent (uncolored). By making the near wall transparent, the user can more easily confirm the shape of furniture, etc. placed beyond the near wall (in the room).

[0063] The visualized data generation unit 14 generates visualized data to which a color attribute corresponding to the captured image acquired by the captured image acquisition unit 111 has been assigned. For example, if the captured image includes wood-grain furniture and the visualized data generation unit 14 detects the color of the wood grain (e.g., brown), the visualized data generation unit 14 generates visualized data to which a color similar to the detected color has been assigned to the extracted furniture figure. By assigning a color attribute corresponding to the captured image, it becomes easier for the user to identify the type of furniture, etc.

[0064] The visualization data generator 14 generates visualization data in which the rendering method for fixed objects and moving objects is changed. Fixed objects include, for example, room walls, steps, and fixed furniture. Moving objects include, for example, chairs, trash cans, and furniture with casters. Moving objects may also include temporary objects placed on the floor, such as luggage or bags. The rendering method includes, for example, color attributes such as hue, brightness, or saturation applied to a plane, patterns or textures applied to a plane, etc.

[0065] The classification of a spatial element as a fixed, moving, or temporary object can be identified based on the period of time it has existed at that location. For example, the spatial data generation unit 13 generates spatial data by classifying the spatial element as a fixed, moving, or temporary object based on changes over time in the point cloud data generated by the point cloud data generation unit 12. For example, the spatial data generation unit 13 determines that a spatial element is a fixed element if the spatial element has not changed based on the difference between the spatial data generated at a first time and the spatial data generated at a second time. The spatial data generation unit 13 may also determine that a spatial element is a moving element if the position of the spatial element has changed based on the difference in the spatial data. The spatial data generation unit 13 may also determine that a spatial element is a temporary element if the spatial element has disappeared or appeared based on the difference in the spatial data. The visualization data generation unit 14 changes the rendering method based on the classification identified by the spatial data generation unit 13. Changing the rendering method may include, for example, color coding, adding hatching, or adding a predetermined mark. For example, the spatial data generation unit 13 may display fixed objects in black, moving objects in blue, or temporary objects in yellow. The spatial data generation unit 13 generates spatial data by identifying the classification of fixed objects, moving objects, or temporary objects. The visualization data generation unit 14 may generate visualization data in which the drawing method is changed based on the classification identified by the spatial data generation unit 13. Furthermore, the spatial data generation unit 13 may generate visualization data in which the drawing method of spatial elements recognized by image recognition is changed.

[0066] The visualization data generation unit 14 can generate visualization data for multiple divided areas. For example, the visualization data generation unit 14 generates visualization data for each of walled spaces such as a living room, bedroom, dining room, and hallway, treating them as a single room. Generating visualization data for each room makes it possible to, for example, generate spatial data or visualization data separately for each room, facilitating the generation of spatial data, etc. Furthermore, it becomes possible to create spatial data, etc. only for areas where the robot 2 may move. The visualization data providing unit 161 provides visualization data that allows the user to select an area. For example, the visualization data providing unit 161 may enlarge the visualization data for the area selected by the user or provide detailed visualization data for the area selected by the user.

[0067] The photographic target recognition unit 15 performs image recognition of spatial elements based on the photographic images acquired by the photographic image acquisition unit. Recognition of spatial elements can be performed, for example, by using an image recognition engine that determines what the spatial elements are based on image recognition results accumulated through machine learning. Image recognition of spatial elements can be performed, for example, by recognizing the shape, color, pattern, or text or graphics attached to the spatial elements. The photographic target recognition unit 15 may perform image recognition of spatial elements by using an image recognition service provided by a cloud server (not shown). The visualization data generation unit 14 generates visualization data in which the rendering method is changed depending on the spatial element image-recognized by the photographic target recognition unit 15. For example, if the spatial element image-recognized is a sofa, the visualization data generation unit 14 generates visualization data in which a texture having a fabric texture is added to the spatial element. Furthermore, if the spatial element image-recognized is a wall, the visualization data generation unit 14 may generate visualization data in which the color attribute of the wallpaper (e.g., white) is added. By performing such visualization processing, the user can intuitively grasp the state of the space recognized by the robot 2.

[0068] The second communication control unit 16 controls communication with the user terminal 3 owned by the user. The user terminal 3 is, for example, a smartphone, a tablet PC, a notebook PC, a desktop PC, etc. Any communication method with the user terminal 3 can be used, and for example, short-range wireless communication such as wireless LAN, Bluetooth (registered trademark), or infrared communication, or wired communication can be used. The functions of the visualization data providing unit 161 and the specification acquisition unit 162 of the second communication control unit 16 communicate with the user terminal 3 using communication functions controlled by the second communication control unit 16.

[0069] The visualization data providing unit 161 provides the visualization data generated by the visualization data generating unit 14 to the user terminal 3. The visualization data providing unit 161 is, for example, a web server, and provides the visualization data to the browser of the user terminal 3 as a web page. The visualization data providing unit 161 may be configured to provide the visualization data to multiple user terminals 3. By visually checking the visualization data displayed on the user terminal 3, the user can confirm the range in which the robot 2 can move as a 2D or 3D display. The visualization data depicts the shapes of furniture and the like using a predetermined drawing method. By operating the user terminal 3, the user can, for example, switch between 2D and 3D display, zoom in or out on the visualization data, or move the viewpoint in the 3D display.

[0070] A user can visually check the visualization data displayed on the user terminal 3 and confirm the spatial data generation status and area attribute information. The user can specify an area in the visualization data for which spatial data has not been generated and instruct the creation of spatial data. Furthermore, the user can visually check the visualization data displayed on the user terminal 3 and, if there is an area where the spatial data appears to be inaccurate, such as an unnatural shape of a spatial element such as furniture, specify that area and instruct the regeneration of the spatial data. As described above, the visualization coordinate information in the visualization data is associated with the coordinate information of the spatial data. Therefore, an area in the visualization data for which the user has specified regeneration can be uniquely identified as an area in the spatial data. The regenerated spatial data is regenerated by the visualization data generation unit 14 and provided by the visualization data provision unit 161. Note that there may be cases in which the generation status of the spatial data remains unchanged, such as when a spatial element is misrecognized even in the regenerated visualization data. In such cases, the user may instruct the generation of spatial data by changing the operation parameters of the robot 2. The operation parameters are, for example, the shooting conditions (exposure amount, shutter speed, etc.) of the shooting unit 21 of the robot 2, the sensitivity of a sensor (not shown), and clearance conditions when allowing movement of the robot 2. The operation parameters may be included in the spatial data as, for example, attribute information of the area.

[0071] The visualization data generation unit 14 generates visualization data that includes, for example, a button display that instructs creation of spatial data (including "re-creation"). When the user operates the displayed button, the user terminal 3 can transmit an instruction to create spatial data to the autonomously acting robot 1. The instruction to create spatial data transmitted from the user terminal 3 is acquired by the designation acquisition unit 162.

[0072] The designation acquisition unit 162 acquires an instruction to create spatial data for an area designated by a user based on the visualization data provided by the visualization data providing unit 161. The designation acquisition unit 162 may acquire an instruction to set (including change) attribute information for the area. The designation acquisition unit 162 also acquires the location of the area and the direction in which the robot will approach the area, i.e., the direction in which the image should be captured. The creation instruction can be acquired, for example, by operating a web page provided by the visualization data providing unit 161. This allows the user to understand how the robot 2 perceives the space and instruct the robot 2 to redo the recognition process depending on the recognition state.

[0073] The instruction unit 113 instructs the robot 2 to take a photograph of the area for which spatial data creation has been instructed. The instruction unit 113 may instruct the robot 2 to take a photograph of a marker installed in the area. The photographing of the area for which spatial data creation has been instructed may include photographing conditions such as the coordinate position of the robot 2 (photographing unit 21), the photographing direction of the photographing unit 21, and resolution. If the spatial data instructed to be created relates to an area for which no data has been created, the spatial data generation unit 13 adds the newly created spatial data to the existing spatial data. If the spatial data instructed to be created relates to re-creation, the spatial data generation unit 13 generates spatial data by updating the existing spatial data. Furthermore, if a marker is included in the photographed image, spatial data including the recognized marker may be generated.

[0074] As described above, in FIG. 1, the autonomously acting robot 1 is described as being composed of the data providing device 10 and the robot 2, but the functions of the data providing device 10 may be included in the functions of the robot 2. For example, the robot 2 may include all of the functions of the data providing device 10. The data providing device 10 may temporarily take over the functions of the robot 2 when, for example, the processing power of the robot 2 is insufficient.

[0075] Furthermore, in this embodiment, "acquisition" may refer to active acquisition by the acquiring entity, or passive acquisition by the acquiring entity. For example, the designated acquisition unit 162 may acquire the spatial data by receiving an instruction to create spatial data transmitted by the user from the user terminal 3, or by reading from a storage area (not shown) an instruction to create spatial data that the user has stored in the storage area.

[0076] Furthermore, the functional units of the data providing device 10, namely, the first communication control unit 11, the point cloud data generation unit 12, the spatial data generation unit 13, the visualization data generation unit 14, the imaging target recognition unit 15, the second communication control unit 16, the captured image acquisition unit 111, the spatial data providing unit 112, the instruction unit 113, the visualization data providing unit 161, and the designation acquisition unit 162, are merely examples of the functions of the autonomously acting robot 1 in this embodiment and do not limit the functions of the autonomously acting robot 1. For example, the autonomously acting robot 1 does not need to have all the functional units of the data providing device 10, but may have only some of the functional units. The autonomously acting robot 1 may also have functional units other than those described above. Furthermore, the functional units of the robot 2, namely, the marker recognition unit 22, the movement control unit 23, the limited range setting unit 231, and the state information acquisition unit 24, are merely examples of the functions of the autonomously acting robot 1 in this embodiment and do not limit the functions of the autonomously acting robot 1. For example, the autonomously acting robot 1 does not need to have all the functional units that the robot 2 has, but may have some of the functional units. It may also be something.

[0077] As described above, the above-mentioned functional units of the autonomously acting robot 1 have been described as being realized by software. However, at least one of the above-mentioned functions of the autonomously acting robot 1 may be realized by hardware.

[0078] Furthermore, any of the above functions possessed by the autonomously acting robot 1 may be implemented by dividing one function into multiple functions. Furthermore, any two or more of the above functions possessed by the autonomously acting robot 1 may be implemented by integrating them into one function. In other words, FIG. 1 shows the functions possessed by the autonomously acting robot 1 as functional blocks, and does not indicate, for example, that each function is configured as a separate program file.

[0079] Furthermore, the autonomously acting robot 1 may be a device realized by a single housing, or may be a system realized by multiple devices connected via a network or the like. For example, the autonomously acting robot 1 may have some or all of its functions realized by a virtual device such as a cloud service provided by a cloud computing system. In other words, the autonomously acting robot 1 may have at least one or more of the above functions realized by another device. Furthermore, the autonomously acting robot 1 may be a general-purpose computer such as a tablet PC, or may be a dedicated device with limited functions.

[0080] Furthermore, the autonomously acting robot 1 may have some or all of its functions realized by the robot 2 or the user terminal 3.

[0081] Next, the hardware configuration of the autonomously acting robot 1 (control unit of the robot 2) will be described with reference to Fig. 2. Fig. 2 is a block diagram showing an example of the hardware configuration of the autonomously acting robot 1 in the embodiment.

[0082] The autonomously acting robot 1 has a CPU (Central Processing Unit) 101, RAM (Random Access Memory) 102, ROM (Read Only Memory) 103, a touch panel 104, a communication I / F (Interface) 105, a sensor 106, and a clock 107. The autonomously acting robot 1 is a device that executes the autonomously acting robot control program described in FIG.

[0083] The CPU 101 controls the autonomously acting robot 1 by executing an autonomously acting robot control program stored in the RAM 102 or the ROM 103. The autonomously acting robot control program is obtained, for example, from a recording medium on which the autonomously acting robot control program is recorded or from a program distribution server via a network, and is installed in the ROM 103, and is read out and executed by the CPU 101.

[0084] The touch panel 104 has an operation input function and a display function (operation display function). The touch panel 104 enables the user of the autonomously acting robot 1 to input operations using a fingertip, a touch pen, or the like. In this embodiment, the autonomously acting robot 1 is described as using the touch panel 104 with an operation display function, but the autonomously acting robot 1 may have a display device with a display function and an operation input device with an operation input function, separately. In this case, the display screen of the touch panel 104 can be the display screen of the display device, and the operation of the touch panel 104 can be implemented as the operation of the operation input device. The touch panel 104 may be realized in various forms, such as a head-mounted, eyeglass-type, or wristwatch-type display.

[0085] The communication I / F 105 is an I / F for communication. The communication I / F 105 executes short-range wireless communication such as wireless LAN, wired LAN, infrared, etc. In FIG. 2, only the communication I / F 105 is illustrated as the communication I / F, but the autonomously acting robot 1 may have communication I / Fs for multiple communication methods. The communication I / F 105 may communicate with a control unit that controls the image capture unit 21 or a control unit that controls the movement mechanism 29, both of which are not shown.

[0086] The sensor 106 is hardware such as the camera of the imaging unit 21, a TOF or thermal camera, a microphone, a thermometer, an illuminance meter, or a proximity sensor. Data acquired by these hardware devices is stored in the RAM 102 and processed by the CPU 101.

[0087] The clock 107 is an internal clock for acquiring time information. The time information acquired by the clock 107 is used, for example, to confirm the time period during which entry is prohibited.

[0088] Next, the operation of the robot control program for providing visualization data will be described with reference to Fig. 3. Fig. 3 is a flowchart showing an example of the operation of the robot control program in the embodiment. In the following description of the flowchart, the autonomously acting robot 1 will be described as the entity that executes the operations, but each operation is executed by each function of the autonomously acting robot 1 described above.

[0089] In FIG. 3, the autonomously acting robot 1 determines whether or not a photographed image has been acquired (step S11). Whether or not a photographed image has been acquired can be determined based on whether or not the photographed image acquisition unit 111 has acquired the photographed image from the robot 2. Whether or not a photographed image has been acquired is determined for each processed photographed image. For example, if the photographed image is a video, the video is continuously transmitted from the robot 2, and therefore whether or not a photographed image has been acquired can be determined based on whether or not the number of frames or data amount of the acquired video has reached a predetermined value. The photographed images may be transmitted by the mobile robot itself, or may be received by the photographed image acquisition unit 111 from the mobile robot itself. If it is determined that a photographed image has not been acquired (step S11: NO), the autonomously acting robot 1 repeats the process of step S11 and waits for a photographed image to be acquired.

[0090] On the other hand, if it is determined that a photographed image has been acquired (step S12: NO), the autonomously acting robot 1 generates point cloud data (step S12). The point cloud data can be generated by the point cloud data generator 12, for example, detecting points in the photographed image where there is a large change in brightness as feature points and assigning three-dimensional coordinates to the detected feature points. The feature points may be detected, for example, by performing a differential process on the photographed image to detect areas where there is a large change in gradation. Furthermore, the assignment of coordinates to feature points may be performed by detecting the same feature points photographed from different shooting angles. The determination of whether a photographed image has been acquired in step S11 can be made based on whether photographed images photographed from multiple directions have been acquired.

[0091] After executing the process of step S12, the autonomously acting robot 1 generates spatial data (step S13). The spatial data can be generated by the spatial data generator 13, for example, by performing a Hough transform on the point cloud data. Details of step S13 will be described with reference to FIG. 4.

[0092] After executing the process of step S13, the autonomously acting robot 1 provides the generated spatial data to the robot 2 (step S14). The spatial data may be provided to the robot 2 sequentially each time the spatial data is generated, as shown in FIG. 3, or may be provided sequentially every time the spatial data is generated. The spatial data may be provided asynchronously with the processes shown in steps S11 to S 18. The robot 2 provided with the spatial data can grasp the movable range based on the spatial data.

[0093] After executing the process of step S14, the autonomously acting robot 1 determines whether or not to recognize the spatial element (step S15). The determination of whether or not to recognize the spatial element can be executed, for example, by setting whether or not to recognize the spatial element in the photographing target recognition unit 15. Note that even if it is determined that the spatial element is recognized, if the recognition fails, it may be determined that the spatial element is not recognized.

[0094] If it is determined that the spatial element is recognized (step S15: YES), the autonomously acting robot 1 generates first visualized data (step S16). The generation of the first visualized data can be executed by the visualized data generation unit 14. The first visualized data is visualized data generated after the imaging target recognition unit 15 recognizes the spatial element. For example, if the imaging target recognition unit 15 determines that the spatial element is a table, the visualized data generation unit 14 can generate visualized data by assuming that the top surface of the table is flat, even if the top surface of the table has not been photographed and no point cloud data is available. Furthermore, if it is determined that the spatial element is a wall, the visualized data generation unit 14 can generate visualized data by assuming that the unphotographed portion is also flat.

[0095] If it is determined that the spatial element is not recognized (step S15: NO), the autonomously acting robot 1 generates second visualized data (step S17). The generation of the second visualized data can be executed by the visualized data generator 14. The second visualized data is visualized data that is generated without the photographic target recognition unit 15 recognizing the spatial element, that is, based on the point cloud data and spatial data generated from the photographed image. By not performing the spatial element recognition process, the autonomously acting robot 1 can reduce the processing load.

[0096] After executing the process of step S16 or step S17, the autonomously acting robot 1 provides visualization data (step S18). The visualization data is provided by the visualization data providing unit 161 providing the visualization data generated in the visualization data generation unit 14 to the user terminal 3. The autonomously acting robot 1 may generate and provide visualization data in response to a request from the user terminal 3, for example. After executing the process of step S18, the autonomously acting robot 1 ends the operation shown in the flowchart.

[0097] Next, the operation of the robot control program relating to the generation of space data will be described with reference to Fig. 4. Fig. 4 is a flowchart showing another example of the operation of the robot control program in the embodiment.

[0098] In FIG. 4, the autonomously acting robot 1 generates spatial data (step S121). The spatial data can be generated by the spatial data generation unit 13, for example, by performing a Hough transform on point cloud data. After executing step S131, the autonomously acting robot 1 determines whether or not it has recognized a marker (step S122). Whether or not it has recognized a marker can be determined by whether or not the marker recognition unit 22 recognizes an image of the marker in an image captured by the image capture unit 21. The robot 2 can notify the data providing device 10 of the marker recognition result.

[0099] If it is determined that the marker has been recognized (step S122: YES), the autonomously acting robot 1 sets a restricted range in which movement is restricted in the spatial data generated in step S121 (step S123).

[0100] If it is determined that the marker is not recognized (step S122: NO), the autonomously acting robot 1 determines whether or not state information has been acquired (step S124). Whether or not state information has been acquired can be determined by whether or not state information has been acquired by the state information acquisition unit 24. If it is determined that state information has been acquired (step S124: YES), the autonomously acting robot 1 sets the acquired state information in association with the spatial data (step S125). Note that the set state information is provided by the visualization data provision unit 161 in association with the visualization data.

[0101] On the other hand, if it is determined that the status information has not been acquired (step S124: NO), after executing the processing of step S123 or the processing of step S125, the autonomously acting robot 1 ends the operation of generating the data to be provided of step S12 shown in the flowchart.

[0102] The order of execution of the processes in each step in the operation of the robot control program (robot control method) described in this embodiment is not limited.

[0103] Next, a method for setting no-entry lines by installing paired markers will be described with reference to Fig. 5. Fig. 5 is a diagram showing a method for setting no-entry lines in an embodiment.

[0104] <Installation of no-entry lines in aisles using paired markers> In FIG. 5, it is assumed that there are passages (entrances and exits) between wall 1 and wall 2, and between wall 1 and wall 3, through which robot 2 can move. Between wall 1 and wall 2, the user places marker 1a as a first marker on the wall 1 side, and places marker 1b as a second marker on the wall 2 side. Marker 1a and marker 1b are recognized as a pair of markers by marker recognition unit 22. Restriction range setting unit 231 sets a straight line connecting marker 1a and marker 1b as no-entry line 1. By setting no-entry line 1, the user can restrict robot 2 from moving beyond no-entry line 1.

[0105] <Installation of no-entry lines in passageways using individual markers> A user places marker 2 on the wall 1 side near the passage between walls 1 and 3. Marker 2 is recognized as a single marker by marker recognition unit 22. Restricted range setting unit 231 checks whether there is a passage near marker 2. If there is a passage, restricted range setting unit 231 sets a straight line on the passage as no-entry line 2 based on the installation position of marker 2 and the position of the passage. Because restricted range setting unit 231 can check whether there is a passage near marker 2, the user can prohibit a robot from entering the passage even with a single marker.

[0106] <Establishment of the first no-entry area using a single marker> A user attaches a marker 3 to a wall 3. The marker 3 is recognized as a single marker by the marker recognition unit 22. The restricted range setting unit 231 checks whether there is an aisle near the marker 3. If there is no aisle, the restricted range setting unit 231 sets a predetermined range from the installation position of the marker 3 (for example, a semicircle centered on the installation position of the marker 3) as a first no-entry area.

[0107] <Establishment of a secondary no-entry area using a single marker> The user places marker 4 in the center of the room. Marker 4 is, for example, a three-dimensional marker. Marker 4 is recognized as a single marker by marker recognition unit 22. Restricted range setting unit 231 sets a predetermined range around marker 4 (for example, a circle centered on the installation position of marker 4) as a second no-entry area.

[0108] Next, the setting of a restricted range in the user terminal 3 provided by the autonomously acting robot 1 will be described with reference to Fig. 6 to Fig. 7. Fig. 6 to Fig. 7 are diagrams showing an example of the display of the user terminal 3 in the embodiment. Fig. 6 to Fig. 7 are examples of the display of a web page provided as visualization data from the visualization data providing unit 161 on the touch panel of a smartphone, which is exemplified as the user terminal 3.

[0109] In FIG. 6, the user terminal 3 displays 2D display data generated by the visualization data generation unit 14 based on an image of the living room captured by the robot 2. The visualization data generation unit 14 rasterizes line segments (straight lines or curves) extracted by applying a Hough transform to the point cloud data, and renders the boundaries of spatial elements such as walls and furniture in 2D. The visualization data generation unit 14 displays a no-entry line 36 based on paired marker images recognized by the marker recognition unit 22 or status information acquired by the status information acquisition unit 24. The no-entry line 36 can be set by specifying a start point and an end point. The start point and end point can be set by installing a paired marker or by setting it from the user terminal 3. A no-entry mark is displayed in the center of the no-entry line 36. Pressing the no-entry mark displays a delete button 37. Pressing the delete button 37 erases the no-entry line 36 that was previously set. The house-shaped icon h shown in the figure represents the home position to which the robot 2 returns for charging.

[0110] In FIG. 6 , the area to the right of the no-entry line 36 is an area for which spatial data has not yet been created. The user can set, confirm, or delete the no-entry line 36 from the visualized data displayed on the user terminal 3. That is, the autonomously acting robot 1 can generate a visualized map defining the range within which the robot 2 can move from images captured by the image capture unit 21, and also enables the user terminal 3 to set no-entry areas into which the robot 2 cannot enter. The user terminal 3 may also be configured to allow the user to set conditions for restricting the movement of the robot 2. For example, the user terminal 3 may allow the user to set the time periods during which the robot 2 is prohibited from entering, the restrictions on movement based on the presence or absence of people, the lighting conditions when restricting movement, and the like. For example, the user terminal 3 may allow the user to set conditions such as prohibiting entry into the kitchen in the morning and evening when people are preparing meals, or prohibiting entry into a study when the lights are off.

[0111] In FIG. 7 , the user terminal 3 displays 2D visualization data generated by the visualization data generator 14 based on the captured image of the living room taken by the robot 2, as in FIG. 6 . The visualization data of the Western-style room is displayed to the right of the no-entry line 36, and it can be seen that the no-entry line 36 prohibits the robot 2 from entering the Western-style room 38. For example, after spatial data of the Western-style room 38 has been generated based on the captured image of the Western-style room 38, the user may wish to set the Western-style room 38 as a no-entry area. The user can set the Western-style room 38 as a no-entry area by placing a pair of no-entry markers at the entrance to the Western-style room 38 or by setting the no-entry line 36 from the user terminal 3. Furthermore, because spatial data for the Western-style room 38 has already been generated, the user can see that the robot 2 can move into the Western-style room 38 by deleting the no-entry line 36. The user can zoom in and out of the display by pinching in or out on the touch panel of the user terminal 3.

[0112] As a modification of the first embodiment, for example, the visualization data providing unit 161 in FIG. 1 may provide the user terminal 3 with the original image of the visualization data together with the visualization data. For example, when the user specifies a part of the floor plan displayed on the user terminal 3, an image of that part may be displayed. That is, the image used to identify each spatial element is stored, and when the user specifies a spatial element, an image associated with that spatial element is provided. This allows the user to determine the recognition state using the image when the user cannot determine the recognition state from the visualization data. You can judge the state of mind.

[0113] As yet another modified example, the method of specifying the restricted area for no entry described with reference to Fig. 6 can be achieved by sliding a fingertip in a circular motion across the screen of the touch panel of the user terminal 3, thereby enclosing the restricted area. In conjunction with this operation, the trajectory of the fingertip is drawn on the screen, and the trajectory of the fingertip is visualized as if it were superimposed on the visualization data.

[0114] Furthermore, if the marker is planar, as described above, it may not be possible to recognize the marker depending on the shooting angle of the marker. For example, road signs are installed approximately perpendicular to the direction of travel to make them easier for drivers to see. However, markers may be attached to walls along the path along which the robot 2 travels, and depending on the shooting direction of the camera, the marker may be overlooked. For example, since the marker 2 shown in FIG. 5 is attached to wall 1, if the robot 2 moves toward marker 3 along wall 3, it may enter the passage before confirming the marker. Therefore, as a modified example of this embodiment, when the robot 2 discovers a space it can enter (e.g., a passageway in a wall), it actively checks whether a marker is installed near the entrance of the space. For example, if the entrance to the space is a passageway between walls, the marker may be attached to a wall near the entrance of the passageway. The robot 2 moves to a position where it can easily see a marker that may be installed on the wall at the entrance of the passageway, for example, a position directly in front of the passageway, and actively checks by photographing the wall, thereby preventing the marker from being overlooked. The marker may be actively checked when an area with different environmental conditions is discovered or when the robot 2 enters that area. The environmental conditions may be, for example, the illuminance, temperature, humidity, or noise level of the space in which the robot 2 moves, and changes in the environmental conditions may include changes in spatial elements such as the color of the walls.

[0115] Furthermore, the restricted range setting unit 231 may set the restricted range based on the feature points around the marker instead of the spatial data of the installation position where the marker is installed. The surrounding feature points are, for example, spatial elements such as cables placed on the floor or steps. By learning the feature points where the marker is installed, it is possible to obtain a learning effect of restricting movement even in places with similar feature points where no marker is installed.

[0116] Furthermore, if there are multiple robots, different movement restrictions may be set for each robot. For example, when the same marker is recognized, different no-entry areas may be set for each robot. By setting different restrictions for each robot, it is possible to set restrictions according to the robot's purpose (for example, a cleaning robot, an alarm robot, etc.).

[0117] The robot may also be configured to learn the content of the restrictions once set by the marker. For example, the robot may learn that a no-entry area has been set by a temporary marker, and subsequently reduce the frequency of entry into the area even after the marker is removed. To achieve this, the marker recognition unit 22 stores the positions of the temporary marker and the permanent marker in association with information that identifies the type of marker, respectively. In this way, by storing the positions of the temporary marker, the robot can perform actions such as behaving in a way that makes it hesitant to enter an area where a temporary marker has been set, or reducing the frequency of entry into the area.

[0118] [Embodiment 2] In the first embodiment, an example is shown in which a marker is used to make the autonomously acting robot 1 recognize a no-entry place, but a marker may be used to make the autonomously acting robot 1 recognize an arbitrary place. In other words, a marker may be installed in an arbitrary place in a house or facility, and the autonomously acting robot may recognize the no-entry place. The location where the marker is installed may be recognized by the remote control 1.

[0119] A house may include any area, such as an entrance, a children's room, or a bedroom. Facilities may include any area, such as a reception counter, a rest area, or an emergency exit. Markers are used to identify area types associated with these areas (e.g., an entrance type or a reception counter type, as described below). Markers may have any characteristics that allow the area type to be identified by image recognition, such as shape, pattern, color, text or graphics attached to the marker, or a combination of these. For example, a graphic code obtained by converting an area type code into a graphic using a general-purpose conversion method (such as a barcode or a two-dimensional barcode) may be used as the marker. In this case, the marker recognition unit 22 can read the area type code from the graphic code using the conversion method. Alternatively, a uniquely designed graphic may be used as the marker. In this case, the marker recognition unit 22 may identify the type of graphic when the shape of the graphic included in the captured image is a predetermined shape, and then identify the area type corresponding to the identified type of graphic. Data associating graphic types with area types is stored in the data providing device 10 or the robot 2. That is, the marker in the second embodiment can identify any area type.

[0120] If such a marker is placed in an area identified by the area type, the autonomously acting robot 1 can recognize that the location where the marker is placed corresponds to the area identified by the area type. For example, if a front door type marker is placed at the front door, the autonomously acting robot 1 can recognize that the location where the front door type marker is placed is the front door.

[0121] In the second embodiment, the autonomously acting robot 1 stores marker information that associates a predetermined event with an area type. When the autonomously acting robot 1 detects a predetermined event, the robot 2 moves to a location where a marker of the area type corresponding to the predetermined event is installed (referred to as a marker installation location). The predetermined event that triggers the robot 2 to move to the marker installation location is referred to as a first event. The first event may be detected by the robot 2 or by the data providing device 10.

[0122] Furthermore, if the autonomously acting robot 1 detects another predetermined event after the robot 2 moves to the marker installation location, the robot 2 executes a predetermined action. The event that triggers the robot 2 to execute a predetermined action is called a second event. The second event may be detected by the robot 2 or by the data providing device 10. The action to be executed corresponds to at least one of the first event, the area type, and the second event. In the second embodiment, the autonomously acting robot 1 stores event information that associates at least one of the first event, the area type, and the second event with an action. The event information may be stored in the robot 2.

[0123] Fig. 8 is a block diagram showing an example of a module configuration of the robot 2 in the second embodiment. Fig. 8 also shows functional units related to Examples 1 to 6, which will be described later. The robot 2 has the following functional units: a marker recognition unit 22, a position measurement unit 25, a movement control unit 23, a communication control unit 26, a first event detection unit 210, a second event detection unit 220, and an action execution unit 230. The above-mentioned functional units of the robot 2 in the second embodiment will be described as functional modules realized by a program that controls the robot 2.

[0124] The marker recognition unit 22 recognizes markers included in the captured image and identifies the area type indicated by the marker. The position measurement unit 25 measures the current position and direction of the robot 2. The position measurement unit 25 may measure the current position and direction based on the captured image, or may measure the current position based on radio waves received from a wireless communication device installed at a predetermined position. The method for measuring the current position may be a conventional technique. The position measurement unit 25 estimates its own position and the environmental location. Simultaneous Localization and Mapping (SLAM) technology may be used to simultaneously create a map. The movement control unit 23 controls the movement of the robot 2. The movement control unit 23 sets a route to a destination and drives the movement mechanism 29 to move the robot 2 along the route to the destination. The communication control unit 26 communicates with the data providing device 10.

[0125] The first event detection unit 210 detects a first event. The first event detection unit 210 may detect a first event based on the results of recognition processing such as voice recognition or image recognition. That is, the first event detection unit 210 may detect a first event when it determines that a sound input through a microphone provided in the robot 2 includes characteristics that suggest that the sound corresponds to a predetermined sound. For example, the first event detection unit 210 may record a sample of the predetermined sound in advance, analyze the sound, and extract feature data such as a frequency distribution, intonation, and a period in which the volume increases. The first event detection unit 210 may then perform a similar analysis on the sound input through the microphone, and infer that the sound input through the microphone corresponds to the predetermined sound if the feature data such as the frequency distribution, intonation, and a period in which the volume increases matches or is similar to that of the sample.

[0126] Furthermore, the first event detection unit 210 may detect a first event when it is estimated that an arbitrary person, a predetermined person, or a predetermined object is captured in an image captured by the image capture unit 21. For example, the first event detection unit 210 may capture an arbitrary person, a predetermined person, or a predetermined object as a sample in advance using the image capture unit 21, analyze the captured image, and extract feature data such as the size, shape, and part arrangement of the object. Then, the first event detection unit 210 may analyze the image captured by the image capture unit 21, and when it is determined that the sample includes an object having the same or similar feature data such as the size, shape, and part arrangement, it may estimate that an arbitrary person, a predetermined person, or a predetermined object is captured in the captured image.

[0127] The first event detection unit 210 may detect the first event based on measurement results from various sensors, such as a temperature sensor, a contact sensor, or an acceleration sensor. The first event detection unit 210 may detect the first event when the sensor measurement value falls below a predetermined lower limit, falls within a predetermined range, falls outside the predetermined range, or exceeds a predetermined upper limit. The first event detection unit 210 may detect the first event, for example, when a temperature equivalent to a human body temperature is measured by a temperature sensor. The first event detection unit 210 may detect the first event when a contact equivalent to a human touch is measured by a contact sensor. The first event detection unit 210 may detect the first event when a change in acceleration equivalent to the impact of a traffic accident or large shaking such as an earthquake is measured by an acceleration sensor.

[0128] The first event detection unit 210 may detect a first event based on a communication state in a communication process. For example, the first event detection unit 210 may detect a first event when a characteristic value indicating a communication state, such as radio wave intensity in wireless communication, a communication time with a predetermined communication partner, or a transmission amount per predetermined time, falls below a lower limit, falls within a predetermined range, falls outside a predetermined range, or exceeds a predetermined upper limit.

[0129] The first event detection unit 210 may detect the first event based on data received from the data providing device 10, the user terminal 3, or another external device. The first event detection unit 210 may detect the first event, for example, when a predetermined notification is received from the data providing device 10 or another external device. The first event detection unit 210 may detect the first event when a predetermined request from a user is accepted from the user terminal 3.

[0130] The voice recognition unit 211, the radio wave condition detection unit 213, the patrol event detection unit 215, and the wake-up event detection unit 217 are examples of the first event detection unit 210. will be described in Example 2 (application example of babysitting). The radio wave condition detection unit 213 will be described in Example 4 (application example of call support). The patrol event detection unit 215 will be described in Example 5 (application example of security). The wake-up event detection unit 217 will be described in Example 6 (application example of alarm clock).

[0131] The second event detection unit 220 detects a second event. As in the case of the first event detection unit 210, the second event detection unit 220 may detect the second event based on the results of recognition processing such as voice recognition or image recognition. As in the case of the first event detection unit 210, the second event detection unit 220 may detect the second event based on the measurement results of various sensors such as a temperature sensor, a contact sensor, or an acceleration sensor. As in the case of the first event detection unit 210, the second event detection unit 220 may detect the second event based on the communication state in the communication processing. As in the case of the first event detection unit 210, the second event detection unit 220 may detect the second event based on data received from the data providing device 10, the user terminal 3, or another external device.

[0132] The user recognition unit 221, the call request acceptance unit 223, the person recognition unit 225, and the body posture recognition unit 227 are examples of the second event detection unit 220. The user recognition unit 221 will be described in Example 1 (application example of pickup). The call request acceptance unit 223 will be described in Example 4 (application example of call support). The person recognition unit 225 will be described in Example 5 (application example of security). The body posture recognition unit 227 will be described in Example 6 (application example of alarm clock).

[0133] The action executing unit 230 executes an action triggered by the second event. The action executing unit 230 may execute an action involving movement of the robot 2 itself. The action executing unit 230 may execute an action involving input processing such as image, sound, or communication in the robot 2. The action executing unit 230 may execute an action involving output processing such as image, sound, or communication in the robot 2.

[0134] The posture control unit 231, the audio output unit 232, the remote control unit 233, the message output unit 235, and the telephone communication unit 237 are examples of the action execution unit 230. The movement control unit 23 may also function as the action execution unit 230. When the movement control unit 23 functions as the action execution unit 230, the movement control unit 23 controls the movement mechanism 29 to perform an action related to the movement of the robot 2. The posture control unit 231 performs an action related to the posture of the robot 2. If the robot 2 has a shape imitating a human or a virtual character and can move its neck and arms with actuators, the posture control unit 231 may drive the actuators to make the robot 2 assume various poses or perform various gestures. If the robot 2 has a shape imitating a four-legged animal and can move each leg with actuators provided at the joints, the posture control unit 231 may drive the actuators to make the robot 2 assume various poses or perform various gestures.

[0135] The remote control unit 233 will be described in Example 2 (application example of babysitting) and Example 6 (application example of alarm clock). The message output unit 235 will be described in Example 3 (application example of customer service). The telephone communication unit 237 will be described in Example 4 (application example of call support).

[0136] FIG. 9 is a block diagram showing an example of the module configuration of the data providing device 10 in the second embodiment. FIG. 9 also shows functional units related to Examples 1 to 6, which will be described later. The robot 2 has the following functional units: a first communication control unit 11, a second communication control unit 16, a marker registration unit 110, a first event detection unit 120, a second event detection unit 130, and an action selection unit 140. The above-mentioned functional units of the data providing device 10 in the second embodiment are The description will be given assuming that the function modules are realized by a control program.

[0137] The first communication control unit 11 controls wireless communication with the robot 2. The second communication control unit 16 controls wireless or wired communication with the user terminal 3 or other external devices. The marker registration unit 110 registers marker information including the position and orientation of the marker in the marker information storage unit 153 described later.

[0138] The first event detection unit 120 detects a first event. As with the first event detection unit 210 of the robot 2, the first event detection unit 120 may detect the first event based on the results of recognition processing such as voice recognition or image recognition. As with the first event detection unit 210 of the robot 2, the first event detection unit 120 may detect the first event based on measurement results from various sensors of the robot 2, such as a temperature sensor, a contact sensor, or an acceleration sensor. As with the first event detection unit 210 of the robot 2, the first event detection unit 120 may detect the first event based on the communication state in the communication processing. As with the first event detection unit 210 of the robot 2, the first event detection unit 120 may detect the first event based on data received from the user terminal 3 or another external device. The first event detection unit 120 may detect the first event based on data received from the robot 2.

[0139] The coming-home event detection unit 121 and the visitor event detection unit 123 are examples of the first event detection unit 120. The coming-home event detection unit 121 will be described in Example 1 (application example of welcoming guests). The visitor event detection unit 123 will be described in Example 3 (application example of customer service).

[0140] The second event detection unit 130 detects the second event. As in the case of the first event detection unit 210 of the robot 2, the second event detection unit 130 may detect the second event based on the results of recognition processing such as voice recognition or image recognition. As in the case of the first event detection unit 210 of the robot 2, the second event detection unit 130 may detect the second event based on the measurement results of various sensors of the robot 2, such as a temperature sensor, a contact sensor, or an acceleration sensor. As in the case of the first event detection unit 210 of the robot 2, the second event detection unit 130 may detect the second event based on the communication state in the communication processing. As in the case of the first event detection unit 210 of the robot 2, the second event detection unit 130 may detect the second event based on data received from the user terminal 3 or another external device. The second event detection unit 130 may detect the second event based on data received from the robot 2.

[0141] The vacant room event detection unit 131 is an example of the second event detection unit 130. The vacant room event detection unit 131 will be described in Example 3 (application example of customer service). The action selection unit 140 selects an action corresponding to at least one of the first event, the area type, and the second event. The following mainly shows an example of selecting an action corresponding to a combination of the first event and the second event.

[0142] The robot 2 further has an event information storage unit 151 and a marker information storage unit 153. The event information storage unit 151 stores event information including an area type corresponding to a first event, a second event, and an action. The event information storage unit 151 will be described later with reference to FIG. 10. The marker information storage unit 153 stores marker information that associates an area type with the position and orientation of a marker. The marker information storage unit 153 will be described later with reference to FIG. 11.

[0143] FIG. 10 is a block diagram showing an example of the data configuration of the event information storage unit 151 in the second embodiment. Each record in FIG. 10 is stored in the event information storage unit 151 when the first event is detected. 10 specifies that the robot 2 will move to a location where a marker of an area type corresponding to the first event is installed. Furthermore, each record in FIG. 10 specifies that when a second event is detected, the robot 2 will execute an action corresponding to, for example, a combination of the first event and the second event. Details of each record will be described in Examples 1 to 6. The event information may be set as a default, or may be set by the user using an application on the user terminal 3.

[0144] Fig. 11 is a block diagram showing an example of the data configuration of the marker information storage unit 153 in embodiment 2. In each record in Fig. 11, the position and orientation of a marker for an area type are set in association with the area type.

[0145] The user attaches a marker of an area type that identifies the area near a desired location in a house or facility in advance. Then, in the marker registration phase, when the robot 1 detects an unknown marker, the data providing device 10 registers the marker information.

[0146] 12(A) is a flowchart showing the processing procedure in the marker registration phase of embodiment 2. For example, the robot 2 may detect an unknown marker while moving autonomously (S21). Specifically, the marker recognition unit 22 of the robot 2 recognizes a marker included in an image captured by the image capture unit 21 and identifies the area type indicated by the marker. The marker recognition unit 22 stores markers that have been detected in the past, and can identify undetected markers by comparing the stored markers with the detected marker.

[0147] When the marker recognition unit 22 determines that it has recognized an undetected marker, it identifies the relative positional relationship and orientation between the robot 2 and the marker through image recognition. The marker recognition unit 22 determines the distance between the robot 2 and the marker based on the size of the marker included in the captured image. The marker recognition unit 22 also determines the orientation of the marker relative to the robot 2 based on the distortion of the marker included in the captured image. The marker recognition unit 22 then determines the position and orientation of the marker based on the current position and orientation of the robot 2 measured by the position measurement unit 25. The communication control unit 26 transmits marker information including the area type and the position and orientation of the marker in that area type to the data providing device 10.

[0148] When the first communication control unit 11 of the data providing device 10 receives the marker information, the marker registration unit 110 registers the position and orientation of the marker for the area type in association with the area type in the marker information storage unit 153 (step S22).

[0149] After the marker registration phase, in the action phase, the robot 2 moves to the marker installation location in response to a first event, and the robot 2 performs a predetermined action in response to a second event.

[0150] 12(B) is a flowchart showing the processing procedure in the action phase of embodiment 2. When the first event detection unit 210 of the robot 2 or the first event detection unit 120 of the data provision device 10 detects a first event (S23), the action selection unit 140 of the data provision device 10 identifies the mark type corresponding to the first event and instructs the robot 2 to move to the position of the mark of that mark type. The movement control unit 23 of the robot 2 sets a route to the mark position in accordance with the movement instruction and controls the movement mechanism 29. The movement mechanism 29 operates to move the robot 2 to the vicinity of the mark (S24).

[0151] When the second event detection unit 220 of the robot 2 or the second event detection unit 130 of the data providing device 10 detects the second event, the action selection unit 140 of the data providing device 10 selects an action corresponding to, for example, a combination of the first event and the second event, and The robot 2 is instructed to take the action instructed above (S26). Then, the robot 2 executes the instructed action (S26). Examples 1 to 6 relating to the second embodiment will be described below.

[0152] [Example 1] An application example of welcoming the user by the autonomously acting robot 1 will be described. In the application example of welcoming the user, when the user returns home, the robot 2 goes to the entrance and greets the user. If a marker of an area type that identifies the entrance as an area is placed at the entrance, the autonomously acting robot 1 can recognize that the location where the marker is placed is the entrance. The area type that identifies the entrance is called the entrance type.

[0153] The timing when the user will return home can be determined by the fact that the position measured by the GPS (Global Positioning System) device of the user terminal 3 carried by the user approaches the user's home. For example, when the distance between the position of the user terminal 3 and the entrance (or the data providing device 10) becomes shorter than a reference distance, it is determined that the user will return home. An event for determining the timing when the user will return home is called a user return home event. In other words, the user return home event corresponds to the first event in the application example of picking up someone.

[0154] When the robot 2 arrives at the entrance and recognizes the user from the image captured by the image capturing unit 21, it performs an action in response to the user's return home. The event in which the robot 2 recognizes the user is called a user recognition event. The action in response to the user's return home is called a welcoming action. The user recognition event is the trigger for performing the welcoming action. The user recognition event corresponds to the second event in the application example of welcoming.

[0155] In the welcoming action, for example, the audio output unit 232 outputs audio from a speaker provided in the robot 2. The output audio may be a natural language such as "Welcome home" or a non-verbal voice such as a cheer. The movement control unit 23 may control the movement mechanism 29 to perform an action such as moving the robot 2 back and forth or rotating as a welcoming action. If the robot 2 has a shape resembling a human or a virtual character and its arms can be moved by actuators, the posture control unit 231 may drive the actuators to raise and lower its arms. If the robot 2 has an actuator that can move its neck, the posture control unit 231 may drive the actuator to shake its head. If the robot 2 has a shape resembling a four-legged animal and its legs can be moved by actuators provided at its joints, the posture control unit 231 may cause the robot 2 to pose by standing up on its hind legs as a welcoming action. This allows the robot 2 to appear happy about the user's return home. The user will feel a sense of closeness to the robot 2 that is waiting for them, and will develop a deeper attachment to it.

[0156] The coming-home event detection unit 121 of the data providing device 10 shown in FIG. 9 detects a user coming-home event when it is determined that the user is approaching home based on the location information of the user terminal 3 as described above. The coming-home event detection unit 121 may detect a user coming-home event when it receives a notification from the user terminal 3 indicating that the user terminal 3 has communicated with a beacon transmitter installed at the entrance. The coming-home event detection unit 121 may detect a user coming-home event when it recognizes the user based on an image captured by an intercom with a camera or an input voice. The coming-home event detection unit 121 may also detect a user coming-home event when it receives an email from the user terminal 3 notifying the user of their arrival.

[0157] The user recognition unit 221 of the robot 2 shown in Fig. 8 detects the user by, for example, recognizing the face part included in the captured image or by recognizing the input voice. For example, the user recognition unit 221 captures the user's face as a sample in advance using the image capture unit 21, analyzes the captured image, and extracts feature data such as the size, shape, and arrangement of features (eyes, nose, mouth, etc.). The person recognition unit 225 may analyze an image captured by the image capture unit 21, and when it determines that the captured image contains a subject whose feature data, such as size, shape, and part arrangement, is the same as or similar to that of the sample, infer that the captured image contains the user's face. The user recognition unit 221 may also record a sample user's voice in advance, analyze the user's voice, and extract feature data, such as frequency distribution and intonation. The user recognition unit 221 may then perform a similar analysis on a sound input via a microphone, and infer that the sound input via the microphone corresponds to the user's voice if the feature data, such as frequency distribution and intonation, matches or is similar to that of the user's voice sample.

[0158] 10, in the application example of welcoming, when a user returning home event is detected as a first event, the robot 2 moves to a location where a marker whose area type is an entrance type is installed (i.e., the entrance). The first record further specifies that when a user recognition event is detected as a second event, the robot 2 performs a welcoming action. A marker whose area type is an entrance type is called an entrance marker.

[0159] In the first record in the data configuration of the marker information storage unit 153 shown in FIG. 11, the position and orientation of the entrance marker are set in association with the entrance type of the area type for the application example of meeting someone.

[0160] 13(A) is a flowchart showing the processing procedure in the marker registration phase of Example 1. For example, when the marker recognition unit 22 detects an entrance marker while the robot 2 is moving autonomously (step S31), the communication control unit 26 transmits entrance marker information including the entrance type of the area type and the position and orientation of the entrance marker to the data providing device 10.

[0161] When the first communication control unit 11 of the data providing device 10 receives the entrance marker information, the marker registration unit 110 registers the position and orientation of the entrance marker in the marker information storage unit 153 in association with the entrance type of the area type (step S32).

[0162] 13(B) is a flowchart showing the processing procedure in the action phase of the first embodiment. When the coming-home event detection unit 121 of the data providing device 10 detects a user coming-home event (step S33), it refers to the event information storage unit 151 to identify the entrance type of the area type corresponding to the user coming-home event. The coming-home event detection unit 121 further refers to the marker information storage unit 153 to identify the position and orientation of the entrance marker corresponding to the entrance type of the area type. The first communication control unit 11 transmits an instruction to move to the entrance, including the position and orientation of the entrance marker, to the robot 2. At this time, the coming-home event detection unit 121 notifies the action selection unit 140 of the user coming-home event.

[0163] When the communication control unit 26 of the robot 2 receives the instruction to move to the entrance, including the position and orientation of the entrance marker, the movement control unit 23 of the robot 2 controls the movement mechanism 29, and the robot 2 moves to the entrance (step S34). The robot 2 stays in front of the entrance marker for at least a first predetermined time. The first predetermined time corresponds to the upper limit of the expected interval from the detection of a user return event to the detection of a user recognition event. If a user recognition event has not been detected after the first predetermined time has elapsed, the processing in the action phase of the first embodiment may be interrupted.

[0164] When the user opens the door and enters, the robot 2 recognizes the user. Specifically, the user recognition unit 221 recognizes the user's face included in the image captured by the image capture unit 21, or performs voice recognition on the sound input by the microphone, to detect a user recognition event (step S35). When the user recognition unit 221 of the robot 2 detects a user recognition event, the communication control unit 26 transmits a user recognition event to the data providing device 10.

[0165] When the first communication control unit 11 of the data providing device 10 receives the user recognition event, the action selection unit 140 refers to the event information storage unit 151 to identify a user returning home event of the first event that corresponds to the user recognition event of the second event. The action selection unit 140 refers to the event information storage unit 151 to select a meeting action that corresponds to the combination of the user returning home event of the first event and the user recognition event of the second event. The first communication control unit 11 transmits an instruction for the selected meeting action to the robot 2.

[0166] When the communication control unit 26 of the robot 2 receives the instruction to perform the welcoming action, the action execution unit 230 of the robot 2 executes the welcoming action (step S36).

[0167] [Example 2] An example of a babysitting application using the autonomous robot 1 will be described. In this example of a babysitting application, when an infant starts crying in a child's room, the robot 2 goes to the child's room and comforts the infant. If a marker of an area type that identifies the child's room as an area is placed in the child's room, the autonomous robot 1 can recognize that the location where the marker is placed is a child's room. The area type that identifies the child's room is called a child's room type.

[0168] In the nanny application example, when an infant cries, Robot 2 moves to the child's room and acts concerned about the infant. The infant's crying is detected by analyzing the sound input to the microphone equipped in Robot 2. In other words, the detection of the infant's crying corresponds to the first event in the nanny application example. This event is called the infant crying event.

[0169] If there is no adult in the child's room, the robot 2 behaves as if it is soothing an infant. The behavior of soothing an infant is called a soothing action. The absence of an adult is detected by recognizing the image captured by the image capture unit 21. The detection of the absence of an adult corresponds to the second event in the babysitting application example. This event is called an adult-absent event.

[0170] In the soothing action, for example, a sound effective for soothing a crying infant (e.g., soothing voice, laughter, or the sound of rustling a paper bag) is output from a speaker provided in the robot 2. In addition to outputting sound, the soothing action may be a behavior that moves a component of the robot 2, such as tilting its head or raising and lowering its arms, or an action that displays a predetermined image on a display, such as an image that is effective for soothing a crying infant. The soothing action may also be a remote control that controls a device different from the robot 2, such as turning on a nearby television or outputting music from an audio device. If the robot 2 calms the infant through a soothing action, the parent user will feel reassured.

[0171] The voice recognition unit 211 of the robot 2 shown in Fig. 8 detects an infant crying event when it determines that the sound input to the microphone is an infant crying sound. For example, the voice recognition unit 211 may record a sample infant crying sound in advance, analyze the infant crying sound, and extract feature data such as frequency distribution and volume increase period. The voice recognition unit 211 may then perform a similar analysis on the sound input to the microphone, and infer that the sound input to the microphone corresponds to an infant crying sound when the feature data such as frequency distribution and volume increase period match or are similar to those of the sample infant crying sound.

[0172] The person recognition unit 225 of the robot 2 shown in Fig. 8 detects an adult absence event when it recognizes that no adult appears in the image captured by the image capture unit 21. For example, the person recognition unit 225 captures images of a plurality of adults of different genders and body types as samples in advance using the image capture unit 21, and The image is analyzed to extract feature data such as size and shape. The person recognition unit 225 may then analyze the image captured by the image capture unit 21 and infer that an adult is in the captured image when it is determined that the image contains a subject whose feature data such as size and shape is the same as or similar to that of the sample. The remote control unit 233 of the robot 2 shown in FIG. 8 transmits a remote control wireless signal to remotely control devices other than the robot 2, such as turning on a nearby television or outputting music from an audio device.

[0173] 10, in the application example of babysitting, the second record stipulates that when a baby crying event is detected as a first event, the robot 2 moves to a location where a marker whose area type is a child's room type is installed (i.e., a child's room). The second record further stipulates that when an adult absent event is detected as a second event, the robot 2 performs a soothing action. A marker whose area type is a child's room type is called a child's room marker.

[0174] In the second record in the data configuration of marker information storage unit 153 shown in FIG. 11, for the application example of babysitting, the position and orientation of the child's room marker are set in association with the child's room type of area type.

[0175] 14(A) is a flowchart showing the processing procedure in the marker registration phase of Example 2. For example, when the marker recognition unit 22 detects a child's room marker while the robot 2 is moving autonomously (step S41), the communication control unit 26 transmits child's room marker information including the child's room type of area type and the position and orientation of the child's room marker to the data providing device 10.

[0176] When the first communication control unit 11 of the data providing device 10 receives the children's room marker information, the marker registration unit 110 registers the position and orientation of the entrance marker in association with the entrance type of the area type in the marker information storage unit 153 (step S42).

[0177] 14(B) is a flowchart showing the processing procedure in the action phase of Example 2. When the voice recognition unit 211 of the robot 2 detects an infant crying event (step S43), the communication control unit 26 notifies the data providing device 10 of the infant crying event.

[0178] When the first communication control unit 11 of the data providing device 10 receives a notification of an infant crying event, the action selection unit 140 refers to the event information storage unit 151 to identify the children's room type of area type associated with the infant crying event. The action selection unit 140 further refers to the marker information storage unit 153 to identify the position and orientation of a children's room marker corresponding to the children's room type of area type. The first communication control unit 11 transmits an instruction to the robot 2 to move to the children's room, including the position and orientation of the children's room marker.

[0179] When the communication control unit 26 of the robot 2 receives the instruction to move to the child's room, including the position and orientation of the child's room marker, the movement control unit 23 controls the movement mechanism 29, and the robot 2 moves to the child's room (step S44). The robot 2 stays in front of the position of the child's room marker for at least a second predetermined time. The second predetermined time corresponds to the upper limit of the estimated time required for the robot 2 to recognize the absence of an adult after entering the child's room. If an adult absence event has not been detected after the second predetermined time has elapsed, the processing in the action phase of the second embodiment may be interrupted.

[0180] When the person recognition unit 225 detects an adult-absent event (step S45), the communication control unit 26 transmits the adult-absent event to the data providing device 10.

[0181] When the first communication control unit 11 of the data providing device 10 receives the adult-absent event, the action selection unit 140 refers to the event information storage unit 151 to identify the baby crying event of the first event that corresponds to the adult-absent event of the second event. The action selection unit 140 refers to the event information storage unit 151 to select a soothing action that corresponds to the combination of the baby crying event of the first event and the adult-absent event of the second event. The first communication control unit 11 transmits an instruction for the selected soothing action to the robot 2.

[0182] When the communication control unit 26 of the robot 2 receives the instruction to perform the soothing action, the action execution unit 230 of the robot 2 executes the soothing action (step S46).

[0183] [Example 3] An example of an application of the autonomously acting robot 1 to customer service will be described. An example of an application of customer service is assumed to be a store that provides food and drink service in guest rooms. When a customer enters the store, the robot 2 heads to the reception counter and guides them to their guest room. If an area type marker that identifies the reception counter as an area is placed on the reception counter, the autonomously acting robot 1 can recognize that the location where the marker is placed is the reception counter. The area type that identifies the reception counter is called the reception counter type.

[0184] In the customer service application example, when a visitor is detected, the robot 2 moves to the reception counter and responds to the customer. A visitor is detected when the customer entering the store is captured in an image taken by a camera installed at the store entrance. In other words, the detection of a visitor corresponds to the first event in the customer service application example. This event is called a visitor event.

[0185] If there is a vacant room, Robot 2 behaves as if it is guiding the customer to the vacant room. The behavior of guiding the customer to the vacant room is called a guide action. Information about vacant rooms is obtained from a guest room management system (not shown). The detection of a vacant room corresponds to the second event in the customer service application example. This event is called a vacant room event.

[0186] In the guidance action, for example, the robot 2 leads the customer to an available room. Alternatively, the message output unit 235 may output a voice message such as "Please enter room number X," or the message may be displayed on a display device provided in the robot 2. When the robot 2 serves customers, the customer feels a sense of convenience that cannot be achieved by human service. Unmanned operation also contributes to cost reduction.

[0187] The visitor event detection unit 123 of the data providing device 10 shown in FIG. 9 detects a visitor event when it recognizes a customer entering a store in an image captured by a camera installed at the store entrance, for example. The vacant room event detection unit 131 of the data providing device 10 shown in FIG. 9 inquires about the status of guest rooms from a guest room management system (not shown) and detects a vacant room event if there are any vacant rooms. The movement control unit 23 of the robot 2 shown in FIG. 8 drives the movement mechanism 29 to move slowly to the guest room. The robot 2 may control its movement speed to maintain a constant distance from the customer while gauging the distance from the customer based on the appearance of the customer captured by the camera unit 21 of the robot 2. Furthermore, the message output unit 235 of the data providing device 10 shown in FIG. 9 outputs a guidance message such as "Please enter room number X" as a voice message, or outputs the guidance message on an output device so as to display it on a display device provided in the robot 2.

[0188] The third record in the data configuration of the event information storage unit 151 shown in FIG. 10 specifies that, in the application example of customer service, when a visitor event is detected as the first event, the robot 2 moves to a location where a marker whose area type is a reception counter type is installed (i.e., the reception counter). The third record further specifies that, as the second event, an empty It is stipulated that when a reception room event is detected, the robot 2 will perform a guidance action. A marker whose area type is a reception counter type is called a reception counter marker.

[0189] In the third record in the data configuration of the marker information storage unit 153 shown in FIG. 11, the position and orientation of the reception counter marker are set in association with the reception counter type of the area type for the application example of customer service.

[0190] 15(A) is a flowchart showing the processing procedure in the marker registration phase of Example 3. For example, when the marker recognition unit 22 detects a reception counter marker while the robot 2 is moving autonomously (step S51), the communication control unit 26 transmits reception counter marker information including the reception counter type of the area type and the position and orientation of the reception counter marker to the data providing device 10.

[0191] When the first communication control unit 11 of the data providing device 10 receives the reception counter marker information, the marker registration unit 110 registers the position and orientation of the reception counter marker in the marker information storage unit 153 in association with the reception counter type of the area type (step S52).

[0192] 15(B) is a flowchart showing the processing steps in the action phase of Example 3. When the visitor event detection unit 123 of the data providing device 10 detects a visitor event (step S53), it refers to the event information storage unit 151 to identify the reception counter type of the area type associated with the visitor event. The visitor event detection unit 123 further refers to the marker information storage unit 153 to identify the position and orientation of the reception counter marker corresponding to the reception counter type of the area type. The first communication control unit 11 transmits to the robot 2 an instruction to move to the reception counter, including the position and orientation of the reception counter marker. At this time, the visitor event detection unit 123 notifies the action selection unit 140 of the visitor event.

[0193] When the communication control unit 26 of the robot 2 receives an instruction to move to the reception counter, including the position and orientation of the reception counter marker, the movement control unit 23 controls the movement mechanism 29, and the robot 2 moves to the reception counter (step S54). The robot 2 stays in front of the reception counter marker for at least a third predetermined time. The third predetermined time corresponds to the upper limit of the switching time until a store staff member takes over from the robot 2 and serves the customer. If a vacant room event has not been detected after the third predetermined time has elapsed, the processing in the action phase of the third embodiment may be interrupted.

[0194] When the vacant room event detection unit 131 of the data providing device 10 detects a vacant room event (step S55), the action selection unit 140 refers to the event information storage unit 151 to identify a visitor event of the first event that corresponds to a vacant room event of the second event. The action selection unit 140 refers to the event information storage unit 151 to select a customer service action that corresponds to the combination of the visitor event of the first event and the vacant room event of the second event. The first communication control unit 11 transmits an instruction for the selected customer service action to the robot 2.

[0195] When the communication control unit 26 of the robot 2 receives the instruction to perform the customer service action, the action execution unit 230 of the robot 2 executes the customer service action (step S56).

[0196] [Example 4] An application example of telephone support by an autonomous robot 1 will be explained. In this example, it is assumed that the robot 2 has a telephone function. It is also assumed that in the home or facility where the robot 2 is placed, there are some places where the radio waves used for the telephone function can easily reach and some places where they cannot. The robot 2 can move from a place where the radio waves are poor to a place where the radio waves are good (hereinafter referred to as The autonomous robot 1 moves to a location with good reception (called a location with good reception) and starts a call in response to a call request. If a marker of an area type that identifies a location with good reception as an area is placed in a location with good reception, the autonomous robot 1 can recognize that the location where the marker is placed is a location with good reception. An area type that identifies a location with good reception is called a good reception type.

[0197] In the application example of call support, when the signal strength deteriorates, the robot 2 moves to a location with good signal strength to prevent any interference with wireless communication. In other words, the detection of signal strength deterioration corresponds to the first event in the application example of call support. This event is called a signal strength deterioration event.

[0198] When the robot 2 receives a call request from a user, it starts the call. The processing action to start a call is called a call start action. The reception of a call request is detected by recognition processing, such as recognizing a voice such as "Please make a call" or recognizing a gesture or pose to make a call from a captured image. The reception of a call request corresponds to the second event in the application example of call support. This event is called a call request event.

[0199] The radio wave condition detection unit 213 of the robot 2 shown in FIG. 8 monitors the condition of radio waves for wireless communication used in the telephone function, and detects a radio wave deterioration event when the radio wave strength falls below an allowable standard. The call request acceptance unit 223 of the robot 2 shown in FIG. 8 accepts a call request from a user by voice recognition of the utterance "Please make a call" or by recognizing a gesture or pose for making a call from a captured image. The call request acceptance unit 223 may also identify a telephone number or the other party through voice recognition. The telephone communication unit 237 of the robot 2 shown in FIG. 8 controls telephone communication and processes calls.

[0200] 10, in the application example of telephone call support, when a signal deterioration event is detected as a first event, the robot 2 moves to a location where a marker whose area type is good signal is installed (i.e., a location with good signal). The fourth record further stipulates that when a call request event is detected as a second event, the robot 2 starts a call. A marker whose area type is good signal is called a good signal marker.

[0201] In the fourth record in the data configuration of the marker information storage unit 153 shown in FIG. 11, the position and orientation of a good signal marker are set in association with the good signal type of the area type for the application example of call support.

[0202] 16(A) is a flowchart showing the processing procedure in the marker registration phase of Example 4. For example, when the marker recognition unit 22 detects a good signal marker while the robot 2 is moving autonomously (step S61), the communication control unit 26 transmits good signal marker information including the good signal type of the area type and the position and orientation of the good signal marker to the data providing device 10.

[0203] When the first communication control unit 11 of the data providing device 10 receives the good signal marker information, the marker registration unit 110 registers the position and orientation of the good signal marker in the marker information storage unit 153 in association with the good signal type of the area type (step S62).

[0204] 16(B) is a flowchart showing the processing procedure in the action phase of Example 4. When the radio wave condition detection unit 213 of the robot 2 detects a radio wave deterioration event (step S63), the communication control unit 26 notifies the data providing device 10 of the radio wave deterioration event.

[0205] When the first communication control unit 11 of the data providing device 10 receives a notification of a signal deterioration event, the action selection unit 140 refers to the event information storage unit 151 and selects a response to the signal deterioration event. The action selection unit 140 further refers to the marker information storage unit 153 to identify the position and orientation of the good signal marker corresponding to the good signal type of the area type. The first communication control unit 11 transmits to the robot 2 an instruction to move to a place with good signal, including the position and orientation of the good signal marker.

[0206] When the communication control unit 26 of the robot 2 receives an instruction to move to a location with good signal strength, including the position and orientation of the poor signal strength marker, the movement control unit 23 controls the movement mechanism 29 to move the robot 2 to the location with good signal strength (step S64). The robot 2 stays in front of the location of the good signal strength marker for at least a fourth predetermined time. The fourth predetermined time corresponds to the maximum length of the period during which a call request is expected from the user. If a call request event has not been detected after the fourth predetermined time has elapsed, the processing in the action phase of the fourth embodiment may be interrupted.

[0207] When the call request accepting unit 223 detects a call request event (step S65), the communication control unit 26 transmits the call request event to the data providing device 10.

[0208] When the first communication control unit 11 of the data providing device 10 receives a call request event, the action selection unit 140 refers to the event information storage unit 151 and selects a call start action corresponding to the call request event of the second event. The first communication control unit 11 transmits an instruction for the selected call start action to the robot 2. Therefore, the call start action is transmitted regardless of whether a signal deterioration event has been notified previously.

[0209] When the communication control unit 26 of the robot 2 receives the instruction for the call start action, the telephone communication 237 of the robot 2 executes the call start action (step S66). When the telephone communication unit 237 makes a call and enters into a call state, a signal obtained by converting the input voice of the microphone provided in the robot 2 is transmitted, and the voice of the other party converted from the received signal is output from the microphone.

[0210] [Example 5] An example of a security application using the autonomous robot 1 will be described. In this example of a security application, the robot 2 patrols a residence or facility where a safe is kept, to guard against safe break-ins. If a marker of an area type that identifies the safe storage area as an area is placed in the safe storage area, the autonomous robot 1 can recognize that the location where the marker is placed is a safe storage area. The area type that identifies the safe storage area is called a safe type.

[0211] In a security application example, when the robot 2 is instructed to patrol, it moves to the vicinity of the safe and behaves as if to grasp the situation. The patrol instruction may be received by a recognition process, such as recognizing a voice such as "Go and check the safe" or recognizing a predetermined pose or gesture from a captured image. The data providing device 10 may receive a patrol instruction from an application on the user terminal 3 and transfer it to the robot 2, and the robot 2 may receive the patrol instruction. At a predetermined time, the data providing device 10 may automatically transmit a patrol instruction to the robot 2, and the robot 2 may receive the patrol instruction. In a security application example, the patrol instruction given to the robot 2 corresponds to the first event. This event is called a patrol event.

[0212] When the robot 2 detects a person near the safe through recognition of the captured image, it executes a security action. That is, in a security application example, the recognition of a person near the safe corresponds to the second event. This event is called a person recognition event. It is assumed that a person near the safe may be a suspicious person. As a security action, the communication control unit 26 of the robot 2 transmits the video (video or still image) captured by the image capturing unit 21 to the data providing device 10. With regard to this process, the communication control unit 26 is an example of the action execution unit 230. The data providing device 10 may record the received video as evidence. Furthermore, the data providing device 10 may send a warning message to an application on the user terminal 3 or record the video received from the robot 2. The data may be transferred to an application in the user terminal 3. As a precautionary action, the audio output unit 232 of the robot 2 may emit an alarm sound.

[0213] The patrol event detection unit 215 of the robot 2 shown in FIG. 8 detects a patrol event when it receives a patrol instruction in the above-described recognition process or when it receives a patrol instruction from the data providing device 10. The person recognition unit 225 of the robot 2 shown in FIG. 8 recognizes the figure of a person included in a captured image. The presence of a person may be recognized by performing voice recognition on a person's speaking voice. The person recognition unit 225 may determine that the sound input to the microphone is a human speaking voice when a frequency corresponding to a standard human voice is extracted from the sound.

[0214] 10 specifies that, in a security application example, when a patrol event is detected as a first event, the robot 2 moves to a location where a marker whose area type is a safe type is installed (i.e., a safe storage area). The fifth record further specifies that when a human recognition event is detected as a second event, the robot 2 performs a guard action. A marker whose area type is a safe type is called a safe marker.

[0215] In the fifth record in the data configuration of the marker information storage unit 153 shown in FIG. 11, the position and orientation of the safe marker are set in association with the safe type of the area type for the security application example.

[0216] 17(A) is a flowchart showing the processing procedure in the marker registration phase of Example 5. For example, when the marker recognition unit 22 detects a safe marker while the robot 2 is moving autonomously (step S71), the communication control unit 26 transmits safe marker information including the safe type of the area type and the position and orientation of the safe marker to the data providing device 10.

[0217] When the first communication control unit 11 of the data providing device 10 receives the safe marker information, the marker registration unit 110 registers the position and orientation of the safe marker in the marker information storage unit 153 in association with the safe type of the area type (step S72).

[0218] 17(B) is a flowchart showing the processing procedure in the action phase of Example 5. When the patrol event detection unit 215 of the robot 2 detects a patrol event (step S73), the communication control unit 26 notifies the data providing device 10 of the patrol event.

[0219] When the first communication control unit 11 of the data providing device 10 receives a notification of a patrol event, the action selection unit 140 refers to the event information storage unit 151 to identify the safe type of the area type associated with the patrol event. The action selection unit 140 further refers to the marker information storage unit 153 to identify the position and orientation of a safe marker corresponding to the safe type of the area type. The first communication control unit 11 transmits to the robot 2 an instruction to move to the safe area, including the position and orientation of the safe marker.

[0220] When the communication control unit 26 of the robot 2 receives an instruction to move to the safe storage area, including the position and orientation of the safe marker, the movement control unit 23 controls the movement mechanism 29, and the robot 2 moves to the vicinity of the safe (step S74). The robot 2 stays in front of the position of the safe marker for at least a fifth predetermined time. The fifth predetermined time corresponds to the upper limit of the estimated time required for the robot 2 to recognize a person after moving near the safe. If a human recognition event has not been detected after the fifth predetermined time has elapsed, the processing in the action phase of the fifth embodiment may be interrupted.

[0221] When the person recognition unit 225 detects a person recognition event (step S75), the communication control unit 26 transmits the person recognition event to the data providing device 10.

[0222] When the first communication control unit 11 of the data providing device 10 receives the human recognition event, the action selection unit 140 refers to the event information storage unit 151 to identify a patrol event of the first event that corresponds to the human recognition event of the second event. The action selection unit 140 refers to the event information storage unit 151 to select an alert action that corresponds to the combination of the patrol event of the first event and the human recognition event of the second event. The first communication control unit 11 transmits an instruction for the selected alert action to the robot 2.

[0223] When the communication control unit 26 of the robot 2 receives the instruction to perform the guard action, the action execution unit 230 of the robot 2 executes the guard action (step S76).

[0224] [Example 6] An example of an application of an alarm clock using the autonomous robot 1 will be described. For example, in the morning, the robot 2 wakes up a user who is sleeping in a bedroom. If a marker of an area type that identifies the bedroom as an area is placed in the bedroom, the autonomous robot 1 can recognize that the location where the marker is placed is a bedroom. The area type that identifies the bedroom is called a bedroom type.

[0225] In the alarm clock application example, the robot 2 moves to the bedroom and performs an alarm action when it is time to wake up the user. The timing to wake up the user is determined, for example, based on the scheduled wake-up time. Alternatively, the timing to wake up the user may be determined by recognition processing such as recognizing a voice uttered by the user's family member, such as "Please wake up Dad (the user)," or recognizing a predetermined pose or gesture from a captured image. In other words, the determination of the timing to wake up the user corresponds to the first event in the alarm clock application example. This event is called a wake-up event.

[0226] When the robot 2 recognizes that the user is lying on a bed in the bedroom (supine position), it executes an alarm action. When the user is not in a supine position, such as when the user is not on the bed or when the user is already awake, the robot 2 does not execute the alarm action. In other words, in the alarm application example, detecting the user in a supine position on the bed corresponds to a second event. This event is called a user-supine position event. As an alarm action, for example, the audio output unit 232 outputs audio such as an alarm sound or a call from a speaker. As an alarm action, the remote control unit 233 may output audio from an external device, such as starting a television or outputting music from an audio device. The remote control unit 233 may also turn on a lighting device. Alternatively, the movement control unit 23 may control the movement mechanism 29 to make the robot 2 move vigorously around the bed.

[0227] The wake-up event detection unit 217 of the robot 2 shown in FIG. 8 detects a wake-up event when the scheduled wake-up time arrives or through the recognition process described above. The body position recognition unit 227 of the robot 2 shown in FIG. 8 recognizes the body position of a person (actually considered to be the user) lying on a bed in a bedroom. If the person lying on the bed is in a lying position, the body position recognition unit 227 detects a user lying position event. For example, the body position recognition unit 227 may capture a sample image of the user in a lying position in advance using the imaging unit 21 and analyze the captured image to extract feature data such as the size, shape, and location of body parts (head, hands, feet, etc.) of the user in the lying position. The body position recognition unit 227 may then analyze the image captured by the imaging unit 21 and determine that the captured image contains a subject whose feature data, such as size, shape, and location of body parts, is the same as or similar to that of the sample.

[0228] The sixth record in the data configuration of the event information storage unit 151 shown in FIG. 10 is an area type when a wake-up event is detected as the first event in the alarm application example. The sixth record specifies that the robot 2 will move to a location where a bedroom-type marker is installed (i.e., a bedroom). The sixth record further specifies that the robot 2 will perform an alarm action when a user lying down event is detected as a second event. A bedroom-type marker is called a bedroom marker.

[0229] In the sixth record in the data configuration of marker information storage unit 153 shown in FIG. 11, the position and orientation of the bedroom marker are set in association with the bedroom type of area type for the alarm application example.

[0230] 18(A) is a flowchart showing the processing procedure in the marker registration phase of Example 6. For example, when the marker recognition unit 22 detects a bedroom marker while the robot 2 is moving autonomously (step S81), the communication control unit 26 transmits bedroom marker information including the bedroom type of the area type and the position and orientation of the bedroom marker to the data providing device 10.

[0231] When the first communication control unit 11 of the data providing device 10 receives the bedroom marker information, the marker registration unit 110 registers the position and orientation of the bedroom marker in the marker information storage unit 153 in association with the bedroom type of the area type (step S82).

[0232] 18(B) is a flowchart showing the processing procedure in the action phase of Example 6. When the wake-up event detection unit 217 of the robot 2 detects a wake-up event (step S83), the communication control unit 26 notifies the data providing device 10 of the wake-up event.

[0233] When the first communication control unit 11 of the data providing device 10 receives a notification of a wake-up event, the action selection unit 140 refers to the event information storage unit 151 to identify the bedroom type of area type associated with the wake-up event. The action selection unit 140 further refers to the marker information storage unit 153 to identify the position and orientation of a bedroom marker corresponding to the bedroom type of area type. The first communication control unit 11 transmits an instruction to move to the bedroom, including the position and orientation of the bedroom marker, to the robot 2.

[0234] When the communication control unit 26 of the robot 2 receives the instruction to move to the bedroom, including the position and orientation of the bedroom marker, the movement control unit 23 controls the movement mechanism 29, and the robot 2 moves to the bedroom (step S84). The robot 2 stays in front of the position of the bedroom marker for at least a sixth predetermined time. The sixth predetermined time corresponds to the upper limit of the estimated time required for the robot 2 to detect a user lying down event after entering the bedroom. If a user lying down event has not been detected after the sixth predetermined time has elapsed, the processing in the action phase of the sixth embodiment may be interrupted.

[0235] When the body position recognition unit 227 detects a user lying down event (step S85), the communication control unit 26 transmits the user lying down event to the data providing device 10.

[0236] When the first communication control unit 11 of the data providing device 10 receives the user lying down event, the action selection unit 140 refers to the event information storage unit 151 to identify the first event, a wake-up event, that corresponds to the second event, a user lying down event. The action selection unit 140 refers to the event information storage unit 151 to select an alarm action that corresponds to the combination of the first event, a wake-up event, and the second event, a user lying down event. The first communication control unit 11 transmits an instruction for the selected alarm action to the robot 2.

[0237] When the communication control unit 26 of the robot 2 receives the instruction for the wake-up action, the action execution unit 230 of the robot 2 executes the wake-up action (step S86).

[0238] In the above-described first to sixth embodiments, the processing described as being performed by the first event detection unit 210 of the robot 2 may be performed by the first event detection unit 120 of the data providing device 10. The processing described as being performed by the second event detection unit 220 of the robot 2 may be performed by the second event detection unit 130 of the data providing device 10. The processing described as being performed by the first event detection unit 120 of the data providing device 10 may be performed by the first event detection unit 210 of the robot 2. The processing described as being performed by the second event detection unit 130 of the data providing device 10 may be performed by the second event detection unit 220 of the robot 2.

[0239] In the second embodiment described above, the robot 2 is moved to a predetermined location at a predetermined timing, and a predetermined action is executed when a predetermined condition is met. By placing a marker at a predetermined location, such a series of actions can be easily realized, which is convenient.

[0240] [Embodiment 3] When the robot 2 recognizes a marker, the robot 2 may execute an action instructed by the marker. To this end, a marker representing a graphic representation of an action identifier may be used. For example, a graphic code representing an action identifier represented by a general-purpose conversion method (such as a barcode or two-dimensional barcode) may be used as the marker. In this case, the marker recognition unit 22 can read the action identifier from the graphic code using the conversion method. Alternatively, a uniquely designed graphic may be used as the marker. In this case, the marker recognition unit 22 may identify the type of graphic when the shape of the graphic included in the captured image is a predetermined shape, and identify the action identifier corresponding to the identified type of graphic. The marker recognition unit 22 stores data associating the type of graphic with the action identifier. In other words, the marker in the third embodiment can identify the identifier of any action.

[0241] For example, if a marker is placed at the entrance to a child's room that instructs the robot to move to the living room as an action, when Robot 2 recognizes the marker in front of the child's room, it will immediately move to the living room without entering the child's room.

[0242] The action instructed by the marker may be a search for another specified marker. If the action instructed by marker A is to search for marker B, the robot 2 will start moving in search of marker B when it recognizes marker A. Furthermore, if the action instructed by marker B is to search for marker C, the robot 2 will start moving in search of marker C when it recognizes marker B. By placing a series of markers at points on a route that instruct the robot 2 to search for the markers in sequence, the robot 2 will search for a route that goes around the series of markers in order. In this way, it is possible to have a child compete with the robot 2 in an indoor game that imitates orienteering. It is also possible to have multiple robots 2 run in an indoor race.

[0243] When playing indoor games such as the indoor games or indoor races described above, the user may instruct the autonomously acting robot 1 via an application on the user terminal 3 to have the robot 2 first search for marker A. The robot 2 may perform voice recognition of a call from the user, such as "Start" or "Search," and start searching for the marker when the user's call is detected. The robot 2 may perform image recognition of a pose or gesture from a captured image indicating a start, and start searching for the marker when the pose or gesture is detected. Alternatively, the user may set a start time for the autonomously acting robot 1 via an application on the user terminal 3, and the robot 2 may start searching for the starting marker when the start time is reached.

[0244] In the third embodiment described above, by placing a marker at a predetermined location, You can easily give instructions to the robot 2's actions.

[0245] [Embodiment 4] Markers indicating no approach may be used to allow the robot 2 to recognize items that should not be approached. For example, a marker depicting a no-approach code may be placed on fragile items such as ornaments or precision equipment. As described above, a graphic code depicting a no-approach code using a general-purpose conversion method (such as a barcode or two-dimensional barcode) may be used as the marker. Alternatively, a uniquely designed graphic may be used as the marker. In this case, the marker recognition unit 22 stores types of graphics corresponding to no-approach indications and determines that the type of graphic identified from the captured image corresponds to no-approach indications. The marker recognition unit 22 of the robot 2 measures the distance from the recognized marker. If the movement control unit 23 determines that the distance from the marker has become shorter than a first reference distance, it controls the movement mechanism 29 to move the robot 2 away from the marker. The first reference distance is the distance required to control the robot 2 to change direction and prevent the robot 2 from reaching the position of the marker. This reduces the risk of the robot 2 colliding with a fragile item.

[0246] Furthermore, a marker indicating that approach is prohibited may be placed on an object that is likely to move, such as a balance ball or a vacuum cleaner body, to allow the robot 2 to recognize the object that is likely to move. When the movement control unit 23 determines that the distance to the marker has become shorter than the second reference distance, it controls the movement mechanism 29 to move the robot 2 in a direction away from the marker. The second reference distance is a distance necessary to control the robot 2 to increase the distance from the object before the object moves and reaches the position of the robot 2. In this way, the risk of the object moving and colliding with the robot 2 can be reduced.

[0247] The type of item may be identified by a marker, and the approach of the robot 2 may be restricted depending on the type of item. A marker indicating the type of item may be used, or item management data associating the marker ID with the type of item may be held in the data providing device 10. When a marker indicating the type of item is used, the type of item is also detected when the marker recognition unit 22 recognizes the marker. The marker recognition unit 22 stores item management data associating the marker ID with the type of item. The marker recognition unit 22 detects the marker ID from the marker and identifies the type of item corresponding to the marker ID by referring to the item management data. The item management data may be set by the user using an application on the user terminal 3.

[0248] Furthermore, the marker recognition unit 22 may store approach control data that specifies whether the robot 2 can approach each type of item, and determine whether the robot 2 can approach each type of item identified by the marker ID. In the approach control data, for example, access permission is set for items that are not easily broken and are unlikely to move, such as tables and chairs, while access prohibition is set for items that are easily broken, such as ornaments and precision instruments, and items that are likely to move, such as balance balls and vacuum cleaner bodies. Because the vacuum cleaner body always moves forward and never backward, approach to the rear of the vacuum cleaner body may be permitted. In other words, it is sufficient to prohibit only approach to the front of the vacuum cleaner body. If the front and rear of the vacuum cleaner body can be recognized based on the orientation of the marker, only approach to the front of the vacuum cleaner body may be prohibited. If the marker includes an arrow and is attached so that the tip of the arrow points to the front of the vacuum cleaner, the robot 2 can determine the range within which approach is prohibited based on the marker. If a rule is established that approach is prohibited in the direction of the arrow, there is no operational problem as long as attention is paid to the orientation of the marker when attaching it. If a no-approach range is to be set that is not limited to the destination indicated by the arrow, the no-approach range based on the arrow may be set in the access control data, or a similar no-approach range may be set from an application on the user terminal 3.

[0249] In the fourth embodiment described above, by placing marks on items that are at risk of colliding with the robot 2, such as fragile items or items that may move, it becomes easier to avoid collisions between the robot 2 and the items.

[0250] [Other variations]

[0251] In the second embodiment, the marker recognition unit 22 may store marker definition information that associates a marker ID with an area type, detect the marker ID from the marker, and identify the area type corresponding to the marker ID. The association between the marker ID and the area type may be set by the user through an application on the user terminal 3.

[0252] Regarding the second embodiment, the first event is not limited to the above-described example. The first event is arbitrary. The first event detection unit 210 of the robot 2 and the first event detection unit 120 of the data providing device 10 may detect the first event when they recognize a person other than the user. The first event detection unit 210 of the robot 2 and the first event detection unit 120 of the data providing device 10 may detect the first event when they recognize an unknown person they have not recognized before for the first time. The first event detection unit 210 of the robot 2 and the first event detection unit 120 of the data providing device 10 may detect the first event when they recognize a known person they have recognized in the past. The first event detection unit 210 of the robot 2 and the first event detection unit 120 of the data providing device 10 may detect the first event when they repeatedly recognize the same person within a predetermined time. The first event detection unit 210 of the robot 2 and the first event detection unit 120 of the data providing device 10 may detect the first event when they recognize a person they have not recognized within a predetermined time. Furthermore, the first event detection unit 210 of the robot 2 and the first event detection unit 120 of the data providing device 10 may detect the first event when it is determined that the positional relationship and orientation between the person and the marker recognized in the captured image satisfy predetermined conditions.

[0253] Regarding the second embodiment, the second event is not limited to the above-described example. The second event is arbitrary. The second event detection unit 220 of the robot 2 and the second event detection unit 130 of the data providing device 10 may detect the second event when they recognize a person other than the user. The second event detection unit 220 of the robot 2 and the second event detection unit 130 of the data providing device 10 may detect the second event when they recognize an unknown person they have not recognized before for the first time. The second event detection unit 220 of the robot 2 and the second event detection unit 130 of the data providing device 10 may detect the second event when they recognize a known person they have recognized in the past. The second event detection unit 220 of the robot 2 and the second event detection unit 130 of the data providing device 10 may detect the second event when they repeatedly recognize the same person within a predetermined time. The second event detection unit 220 of the robot 2 and the second event detection unit 130 of the data providing device 10 may detect the second event when they recognize a person they have not recognized within a predetermined time. Furthermore, the second event detection unit 220 of the robot 2 and the second event detection unit 130 of the data providing device 10 may detect the second event when it determines that the positional relationship and orientation between the person and the marker recognized in the captured image satisfy predetermined conditions.

[0254] Regarding the second embodiment, the first event and the second event may be a combination of events of multiple stages. For example, in the application example of the welcoming event described in the first embodiment, the coming-home event detection unit 121 of the data providing device 10 may detect a first-stage event when the position of the user terminal 3 approaches home, and may detect a second-stage event when the user terminal 3 communicates with a beacon transmitter installed at the entrance. Then, the coming-home event detection unit 121 may determine that the first event has been detected when both the first-stage event and the second-stage event are detected.

[0255] Regarding the second embodiment, the action selection unit 140 selects a combination of a first event and a second event. Although the example of selecting an action corresponding to a combination of an area type and a second event has been described, the action selection unit 140 may select an action corresponding to a second event. The action selection unit 140 may select an action corresponding to an area type. The action selection unit 140 may select an action corresponding to a first event. The action selection unit 140 may select an action corresponding to a combination of an area type and a second event. The action selection unit 140 may select an action corresponding to a combination of a first event and an area type. The action selection unit 140 may select an action corresponding to a combination of a first event, an area type, and a second event.

[0256] In the second embodiment, the action execution unit 230 may perform an action targeted at a specific person. For example, in the application example of customer service described in the third embodiment, the action execution unit 230 may guide a specific customer to an available room. Furthermore, the content of the action to be executed by the action execution unit 230 may be set by the user through an application on the user terminal 3.

[0257] In the second embodiment, the first event detection unit 210 of the robot 2 may detect a first event when the communication control unit 26 communicates with another robot 2. The second event detection unit 220 of the robot 2 may detect a second event when the communication control unit 26 communicates with another robot 2. The first event detection unit 210 of the robot 2 and the first event detection unit 120 of the data providing device 10 may detect a first event when a predetermined instruction or data is received from the user terminal 3 or another external device. The second event detection unit 220 of the robot 2 and the second event detection unit 130 of the data providing device 10 may detect a second event when a predetermined instruction or data is received from the user terminal 3 or another external device. For example, in the application example of customer service described in the third embodiment, the visitor event detection unit 123 of the data providing device 10 may detect a first event when the reception tablet terminal notifies the reception tablet terminal that it has received information about the number of visitors, room conditions such as non-smoking, etc.

[0258] In the second embodiment, the detection conditions for the first event and the second event may be different for each of the multiple robots 2. For example, for a specific robot 2 only, the first event or the second event may be detected later than the timing at which the first event or the second event is normally detected. By delaying the timing at which the first event or the second event is detected, it is possible to portray the robot 2 as having a passive personality. Conversely, for a specific robot 2 only, the first event or the second event may be detected earlier than the timing at which the first event or the second event is normally detected. By earlier detecting the first event or the second event, it is possible to portray the robot 2 as having a proactive personality. The content of the first event and the second event may be set by the user through an application on the user terminal 3.

[0259] Regarding the second embodiment, different event information may be set for each robot 2 in the event information storage unit 151. In other words, event information may be set that applies only to a specific robot 2. For example, in a household where two robots 2 are operated, the event information may be set so that only one robot performs a welcoming action and only the other robot performs a babysitting action.

[0260] When multiple robots 2 are operated, marker information recognized by one robot 2 may be notified to the other robots 2. In this way, the area type, marker position, and orientation can be quickly shared.

[0261] The robot 2 may regard the feature points or feature shapes detected by the SLAM technology as markers. Alternatively, a light-emitting device may be used as the marker. The marker may be installed in a charging station for supplying power to a storage battery provided in the robot 2, and the robot 2 may move based on the marker. When the robot 2 approaches the charging station for automatic coupling, the robot 2 may use the positional relationship and orientation of the charging station detected by the marker.

[0262] The marker recognition unit 22 may measure the position of the same marker multiple times and calculate the average value of those positions. If the marker recognition unit 22 uses the average value of the marker's position, the influence of errors in measuring the marker's position can be reduced. Similarly, the marker recognition unit 22 may measure the orientation of the same marker multiple times and calculate the average value of those orientations. If the marker recognition unit 22 uses the average value of the marker's orientation, the influence of errors in measuring the marker's position can be reduced.

[0263] The application of the user terminal 3 may display the position and orientation of the marker on an output device of the user terminal 3. The application of the user terminal 3 may allow the user to set or modify the content of the marker (no entry, area type, action identifier, no approach, etc.), the position and orientation of the marker, via an input device of the user terminal 3.

[0264] The robot 2 may be equipped with a beacon receiver, which receives a beacon signal transmitted by a beacon transmitter installed at a predetermined position and identifies the ID of the beacon transmitter. The robot 2 may further be equipped with a beacon analysis unit, which analyzes the radio wave intensity of the beacon signal and identifies the position of the beacon transmitter. Therefore, the autonomously acting robot 1 may regard the ID of the beacon transmitter as a marker ID and the position of the beacon transmitter as the position of the marker, and may be applied to the above-mentioned embodiment.

[0265] The various processes described above may be performed by recording a program for implementing the functions of the device described in this embodiment on a computer-readable recording medium, and then loading and executing the program recorded on the recording medium into a computer system. The term "computer system" as used herein may also include hardware such as an OS and peripheral devices. Furthermore, if a WWW system is used, the term "computer system" also includes the homepage provision environment (or display environment). Furthermore, the term "computer-readable recording medium" refers to a storage device such as a flexible disk, a magneto-optical disk, a ROM, a writable nonvolatile memory such as a flash memory, a portable medium such as a CD-ROM, or a hard disk built into a computer system.

[0266] Furthermore, the term "computer-readable recording medium" also includes a storage medium that stores a program for a certain period of time, such as a volatile memory (e.g., DRAM (Dynamic Random Access Memory)) within a computer system that serves as a server or client when the program is transmitted via a network such as the Internet or a communication line such as a telephone line. The program may also be transmitted from a computer system that stores the program in a storage device or the like to another computer system via a transmission medium or by transmission waves within the transmission medium. Here, the "transmission medium" that transmits the program refers to a medium that has the function of transmitting information, such as a network (communication network) such as the Internet or a communication line (communication line) such as a telephone line. The program may also be a program that realizes part of the aforementioned functions. Furthermore, the program may be a so-called differential file (differential program) that realizes the aforementioned functions in combination with a program already stored in the computer system.

[0267] The above describes an embodiment of the present invention with reference to the drawings, but the specific configuration is not limited to this embodiment, and various modifications are also included within the scope that does not deviate from the spirit of the present invention.

Claims

1. A robot that, when it detects an enterable space, determines whether a marker is installed near the entrance to that space.

2. A robot as described in claim 1, which moves to a position where it is easy to detect markers that may be installed.

Images