Robot, its control method, and control program

Abstract

We provide a robot that can deliver information to specific target individuals. [Solution] The robot comprises an execution information acquisition unit that acquires execution information for performing an action to notify a person to be notified of information that should be performed; a search unit that searches for a person to be notified based on the execution information acquired by the execution information acquisition unit; and a notification action execution unit that performs a notification action based on the execution information for the person to be notified found by the search unit.

Description

Technical Field

[0001] The present invention relates to a robot, its control method, and a control program.

Background Art

[0002] Conventionally, there is a robot that autonomously moves indoors, takes images with a camera, recognizes the indoor space from the captured images, and sets a movement route based on the recognized space to move indoors. The setting of the robot movement route is performed by the user creating in advance a map that defines the route along which the robot moves. The robot can move along the route defined based on the created map (for example, refer to Patent Document 1).

[0003] Also, there is a robot that can output an alarm at a predetermined time (for example, refer to Patent Document 2).

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, when a specific target person to whom information such as an alarm is to be notified is not present near the robot, there are cases where the information cannot be notified to the target person in conventional robots.

[0006] The present invention has been made in view of the above circumstances, and in one embodiment, an object is to provide a robot, its control method, and a control program that can notify information to a specific notification target person.

Means for Solving the Problems

[0007] (1) In order to solve the above problems, the robot of the embodiment includes an execution information acquisition unit that acquires execution information for performing an execution operation to notify a person to be notified of information to be performed; a search unit that searches for a person to be notified based on the execution information acquired by the execution information acquisition unit; and a notification operation execution unit that performs a notification operation based on the execution information for the person to be notified found by the search unit.

[0008] (2) In addition, in the robot of the embodiment, the execution information acquisition unit acquires location information relating to a location specified by the user as execution information.

[0009] (3) In addition, in the robot of the embodiment, the execution information acquisition unit acquires location information specified by the user when the user operates the map displayed on the user terminal operated by the user.

[0010] (4) In addition, in the robot of the embodiment, the search unit searches for the person to be notified based on the captured images of the surrounding space.

[0011] (5) In addition, in the robot of the embodiment, the search unit searches for the person to be notified by recognizing the person included in the captured image.

[0012] (6) The robot of the embodiment is further provided with a movement control unit that controls the movement mechanism, the search unit calculates a movement path by the movement mechanism based on execution information, and the movement control unit controls the movement mechanism based on the movement path calculated by the search unit.

[0013] (7) In addition, in the robot of the embodiment, the search unit calculates a movement path based on restriction information for restricting movement by the movement mechanism.

[0014] (8) The robot of the embodiment further includes a marker recognition unit that recognizes predetermined markers included in captured images of the surrounding space, and the movement control unit controls the movement mechanism based on the markers recognized by the marker recognition unit.

[0015] (9) The robot of the embodiment further includes a state information acquisition unit that acquires state information relating to the state of the person to be notified that has been searched for by the search unit, and the notification operation execution unit changes the notification operation according to the state acquired by the state information acquisition unit.

[0016] (10) In addition, in the robot of the embodiment, the state information acquisition unit is when the person to be notified is asleep. The notification action unit obtains whether the person is asleep or awake, and as a notification action, if the person is asleep, it performs an awakening action to wake the person being notified, and if the person is awake, it performs a greeting action to the person being notified.

[0017] (11) In addition, in the robot of the embodiment, the execution information acquisition unit acquires execution information associated with the person to be notified, and the notification action execution unit executes a notification action associated with the found person to be notified.

[0018] (12) In addition, in the robot of the embodiment, the execution information acquisition unit acquires intimacy information as execution information that indicates the degree of intimacy between the person to be notified and the robot, and the notification operation execution unit performs a notification operation in cooperation with other robots when the intimacy in the intimacy information acquired by the execution information acquisition unit satisfies predetermined intimacy conditions.

[0019] (13) In addition, in the robot of the embodiment, the execution information acquisition unit acquires execution information associated with multiple notification targets, and the notification action execution unit executes the notification actions associated with each of the multiple notification targets in parallel.

[0020] (14) Also, in the robot of the embodiment, the execution information acquisition unit acquires time information related to the time for executing the notification operation as execution information, and the notification operation execution unit executes the notification operation based on the time information.

[0021] (15) Also, the robot of the embodiment further includes a voice recognition unit that recognizes the voice input to the microphone and converts it into language data, and the execution information acquisition unit specifies the execution information based on the converted language data.

[0022] (16) Also, in the robot of the embodiment, when the acquired execution information does not include the time information related to the time for executing the notification operation, the notification operation execution unit executes the notification operation at the time when a predetermined time has elapsed since the time when the execution information was acquired.

[0023] (17) Also, in the robot of the embodiment, when the execution information does not include location information, the execution information acquisition unit uses the location where the microphone that input the voice as the location information.

[0024] (18) Also, after executing the notification operation, the robot of the embodiment further includes a photographing unit that photographs the notification target person, and a transmission unit that transmits the photographed image data of the notification target person to the user terminal.

[0025] (19) Also, in the robot of the embodiment, the photographing unit changes the conditions for photographing the notification target person according to the intimacy between the notification target person and the robot.

[0026] (20) Also, after executing the notification operation, the robot of the embodiment further includes a state information acquisition unit that acquires state information related to the state of the notification target person, and a transmission unit that transmits the acquired state information to the user terminal.

[0027] (21) In order to solve the above problems, the robot comprises a message receiving unit that receives messages addressed to the user of the robot, a notification recipient identification unit that identifies a person to be notified based on the recipient of the received message, a search unit that searches for a person to be notified at a location specified by the message or at a location identified in correspondence with the person to be notified, and a notification unit that reads out the message or performs a notification action instructed by the message to the person to be notified found by the search unit.

[0028] (22) In order to solve the above problems, the robot control method of the embodiment includes an execution information acquisition step of acquiring execution information for performing an announcement action of information to be performed for a person to be notified by the robot; a search step of searching for a person to be notified based on the execution information acquired in the execution information acquisition step; and an announcement action execution step of performing an announcement action based on the execution information for the person to be notified found in the search step.

[0029] (23) In order to solve the above problems, the robot control program of the embodiment causes the robot to perform an execution information acquisition process to acquire execution information for performing an announcement action for information to be announced to the person to be announced; a search process to search for the person to be announced based on the execution information acquired in the execution information acquisition process; and an announcement action execution process to perform an announcement action based on the execution information for the person to be announced found in the search process.

[0030] (24) In order to solve the above problems, a robot control method includes a message receiving step in which the robot receives a message addressed to the user of the robot; a notification recipient identification step in which the notification recipient is identified by the destination of the received message; a search step in which the notification recipient is searched for at a location specified by the message or at a location identified in correspondence with the notification recipient; and a notification step in which the message is read aloud or a notification action instructed by the message is performed for the notification recipient found in the search step.

[0031] (25) In order to solve the above problems, the robot control program causes the robot to perform a message receiving process to receive a message addressed to the robot's user, a notification recipient identification process to identify a person to be notified based on the recipient of the received message, a search process to search for a person to be notified at a location specified by the message or at a location identified in correspondence with the person to be notified, and a notification process to read the message aloud or perform a notification action instructed by the message to the person to be notified found in the search process. [Effects of the Invention]

[0032] According to one embodiment, a robot, its control method, and control program acquire execution information for performing an information notification action to be performed for a person to be notified, search for a person to be notified based on the acquired execution information, and then perform a notification action for the searched person to be notified based on the execution information, thereby enabling the notification of information to a specific person to be notified. [Brief explanation of the drawing]

[0033] [Figure 1] This is a block diagram showing an example of the software configuration of an autonomous robot in an embodiment. [Figure 2] A block diagram showing an example of the hardware configuration of an autonomous robot in an embodiment. [Figure 3] This flowchart shows a first example of the operation of an autonomous robot control program in an embodiment. [Figure 4] This flowchart shows a second example of the operation of an autonomous robot control program in an embodiment. [Figure 5] This is a flowchart showing how an autonomous robot control program in an embodiment performs an alarm function as a notification action. [Figure 6] This figure shows an example of execution information in an embodiment. [Figure 7] This figure shows an example of how to set execution information in an embodiment. [Figure 8] This figure shows an example of a module configuration for an autonomous robot that acquires execution information via voice input. [Figure 9] This figure shows an example of a module configuration for a data provision device related to identifying the mounting location of a mountable device. [Figure 10] This figure shows an example of a module configuration for a data provision device related to identifying the location of a user terminal. [Figure 11] This figure shows an example of a module configuration for a data provision device related to identifying the location of a robot. [Figure 12] This figure shows an example of a module configuration for an autonomous robot that provides captured images and status information after performing a notification operation. [Figure 13] This figure shows the first specific example of a notification operation performed collaboratively by two robots. [Figure 14] This figure shows a second specific example of a notification operation performed collaboratively by two robots. [Figure 15] This figure shows a third specific example of a notification operation performed collaboratively by two robots. [Figure 16] This flowchart shows the notification process upon receiving a message. [Modes for carrying out the invention]

[0034] Hereinafter, with reference to the drawings, an autonomous robot, its control method, and control program according to one embodiment of the present invention will be described in detail.

[0035] First, the software configuration of the autonomous robot 1 will be explained using Figure 1. Figure 1 is a block diagram showing an example of the software configuration of the autonomous robot 1 in an embodiment.

[0036] In Figure 1, the autonomous robot 1 comprises a data provision device 10 and a robot 2. The data provision device 10 and the robot 2 are connected by communication and function as the autonomous robot 1. The robot 2 is a mobile robot having the following functional units: an imaging unit 21, a marker recognition unit 22, a movement control unit 23, a state information acquisition unit 24, a search unit 25, a notification action execution unit 26, a notification unit 27, and a movement mechanism 29. The data provision device 10 comprises the following functional units: a first communication control unit 11, a point cloud data generation unit 12, a spatial data generation unit 13, a visualization data generation unit 14, a target recognition unit 15, and a second communication control unit 16. The first communication control unit 11 comprises the following functional units: an image acquisition unit 111, a spatial data provision unit 112, and an instruction unit 113. The second communication control unit 16 comprises the following functional units: a visualization data provision unit 161, a specified acquisition unit 162, and an execution information acquisition unit 163. In this embodiment, each of the functional units of the data provision device 10 of the autonomous robot 1 is described as a functional module realized by a data provision program (software) that controls the data provision device 10. Furthermore, each of the functional units of the robot 2—the marker recognition unit 22, the movement control unit 23, the state information acquisition unit 24, the search unit 25, and the notification operation execution unit 26—is described as a functional module realized by a program that controls the robot 2 in the autonomous robot 1.

[0037] The functions of the autonomous robot 1 can be added, deleted, or modified (added, etc.) by adding, deleting, or changing (adding, etc.) function modules. The basic functions of the autonomous robot 1 will be described as "basic functions," and the additional functions of the autonomous robot 1 will be described as "additional functions." In the following description, the first communication control unit 11, point cloud data generation unit 12, spatial data generation unit 13, visualization data generation unit 14, target recognition unit 15, and second communication control unit 16 (visualization data provision unit 161, designated acquisition unit 162) of the data provision device 10, as well as the shooting unit 21, marker recognition unit 22, and movement control unit 23 of the robot 2 will be described as "basic functions." In addition, the execution information acquisition unit 163 of the data provision device 10, and the state information acquisition unit 24, search unit 25, and notification operation execution unit 26 of the robot 2 will be described as "additional functions."

[0038] [Basic Functions] The data provision device 10 is a device that can perform some of the functions of the autonomous robot 1, and is, for example, an edge server installed in a location physically close to the robot 2, communicating with the robot 2 and distributing the processing load of the robot 2. The description of the behavioral robot 1 will be based on the case where it consists of a data provisioning device 10 and a robot 2, but the functions of the data provisioning device 10 may also be included in the functions of the robot 2. Furthermore, the robot 2 is a robot that can move based on spatial data, and is one embodiment of a robot in which the range of movement is determined based on spatial data. The data provisioning device 10 may consist of one housing or multiple housings.

[0039] The first communication control unit 11 controls the communication function with the robot 2. The communication method with the robot 2 is arbitrary and can be any short-range wireless communication such as wireless LAN (Local Area Network), Bluetooth (registered trademark), or infrared communication, or wired communication. The functions of the first communication control unit 11, namely the image acquisition unit 111, the spatial data provision unit 112, and the instruction unit 113, communicate with the robot 2 using the communication function controlled by the first communication control unit 11.

[0040] The image acquisition unit 111 acquires images captured by the imaging unit 21 of the robot 2. The imaging unit 21 is installed on the robot 2 and can change its imaging range as the robot 2 moves. The imaging unit 21, marker recognition unit 22, movement control unit 23, state information acquisition unit 24, search unit 25, notification operation execution unit 26, notification unit 27, and movement mechanism 29 of the robot 2 will now be described.

[0041] The imaging unit 21 captures images of the space surrounding the robot 2 and generates images that include spatial elements. Spatial elements are elements that exist in the space surrounding the robot 2 and constitute that space, such as room walls, steps, doors, furniture, appliances, luggage, and houseplants. The imaging unit 21 can consist of one or more cameras. For example, if the imaging unit 21 is a stereo camera consisting of two cameras, the imaging unit 21 can capture spatial elements that are the target of the imaging from different shooting angles in three dimensions. If the imaging unit 21 consists of multiple cameras, the captured image may be multiple image data captured by each camera, or a single image data combined from multiple image data. The imaging unit 21 is, for example, a video camera using an image sensor such as a CCD (Charge-Coupled Device) sensor or a CMOS (Complementary Metal Oxide Semiconductor) sensor. By capturing spatial elements with two cameras (stereo camera), the shape of the spatial elements can be measured. The imaging unit 21 may also be a camera using ToF (Time of Flight) technology. In a ToF camera, the shape of a spatial element can be measured by irradiating it with modulated infrared light and measuring the distance to the spatial element. Alternatively, the imaging unit 21 may be a camera that uses structured light. Structured light is a light that projects stripe or grid-like patterned light onto a spatial element. By imaging the spatial element from a different angle than the structured light, the imaging unit 21 can measure the shape of the spatial element from the distortion of the projected pattern. The imaging unit 21 may be one or more of these cameras.

[0042] Furthermore, the imaging unit 21 is attached to the robot 2 and moves in accordance with the robot 2's movement. However, the imaging unit 21 may also be installed separately from the robot 2.

[0043] The captured images taken by the imaging unit 21 are provided to the image acquisition unit 111 using a communication method corresponding to the first communication control unit 11. The captured images are temporarily stored in the memory unit of the robot 2, and the image acquisition unit 111 acquires the temporarily stored images in real time or at predetermined communication intervals.

[0044] The marker recognition unit 22 recognizes a predetermined marker included in the captured image taken by the imaging unit 21. A marker is a spatial element that indicates a restriction on the movement of the robot 2. Restricting movement means restricting the actions that accompany the movement of the robot 2, for example, limiting the movement speed of the robot 2, prohibiting the robot 2 from entering a certain area, or prohibiting a predetermined action of the robot while it is moving (for example, the generation of sound from the robot 2). A marker is the shape, pattern, or color of an object that can be recognized from the captured image, or a combination thereof. By placing a marker at a location that restricts the movement of the robot 2, the marker will be captured along with furniture, etc., when the robot 2 captures the space with the imaging unit 21. A marker may be a flat object or a three-dimensional object. A marker may be, for example, a two-dimensional code or a sticker or paper printed with a specific color combination or shape. Alternatively, a marker may be an ornament or rug of a specific color or shape. In this way, by using printed materials or everyday objects as markers, the user does not need to secure a power source or installation location for the marker. Furthermore, the robot's movement can be restricted at the user's discretion without disrupting the room's atmosphere. Since the user can also see the markers, they can intuitively understand the restricted movement range and easily change it. The markers are installed by the user, for example, by being attached to walls or furniture, or placed on the floor. The marker recognition unit 17 recognizes that the robot 2's movement is restricted by recognizing the marker image included in the captured image.

[0045] The marker recognition unit 22 stores the visual characteristics of markers in advance. For example, the marker recognition unit 22 stores two-dimensional codes or three-dimensional objects that should be recognized as markers in advance. The marker recognition unit 22 may also recognize objects that have been registered in advance by the user as markers. For example, if a user registers a flowerpot photographed with the camera of the user terminal 3 as a marker, the marker recognition unit 22 can recognize flowerpots placed in corridors, etc., as markers. Therefore, users can place objects that do not look out of place in the location where they are to be placed as markers. The marker recognition unit 22 may also recognize spatial elements other than objects as markers. For example, the marker recognition unit 22 may recognize a user's gesture, such as crossing their arms in front of their body, as a marker. The marker recognition unit 22 recognizes the position where the user made the gesture as the marker placement position.

[0046] The marker recognition unit 22 recognizes the position where the marker is attached or installed (hereinafter referred to as the "installation position"). The installation position is the position in the spatial data where the marker is installed. The installation position can be recognized, for example, based on the spatial data recognized by the robot 2, by determining the distance between the robot 2's current position and the captured marker. For example, if the size of the marker is known in advance, the marker recognition unit 22 can calculate the distance between the robot 2 and the marker from the size of the marker image included in the captured image, and recognize the installation position of the marker based on the robot 2's current position and the shooting direction (for example, the direction from a compass not shown). Alternatively, the installation position may be recognized from the relative position from a spatial element whose position in space is already known to the marker. For example, if the position of a door is already known, the marker recognition unit 22 may recognize the installation position from the relative position between the marker and the door. Furthermore, if the captured image was taken with a depth camera, the installation position can be recognized based on the depth of field of the marker captured by the depth camera.

[0047] The movement control unit 23 controls the movement mechanism 29. The movement control unit 23 can control the movement mechanism 29 based on the movement path (described later) calculated by the search unit 25. By controlling the movement mechanism 29, the movement control unit 23 controls the direction and speed of movement of the robot 2. The movement control unit 23 can recognize the current position of the robot 2 based on the spatial data recognized by the robot 2. The movement control unit 23 can move the robot 2 by controlling the movement mechanism 29 from the current position on the movement path. The movement control unit 23, for example, For example, the current position may be adjusted as needed based on spatial elements such as walls and corridors, and the movement path may be modified accordingly.

[0048] Furthermore, the movement control unit 23 may control the movement mechanism 29 based on the markers recognized by the marker recognition unit 22. For example, the movement control unit 23 may restrict movement by the movement mechanism 29 based on the installation position of the markers recognized by the marker recognition unit 22. The installation position of a marker includes a point, line, plane, or space set based on the installation position of one or more markers.

[0049] The spatial data provision unit 112 provides the robot 2 with spatial data generated by the spatial data generation unit 13. The spatial data is a digital representation of the spatial elements recognized by the robot in the space where the robot 2 is located. The robot 2 can move within the range defined in the spatial data. In other words, the spatial data functions as a map for defining the range of movement for the robot 2. The robot 2 is provided with spatial data from the spatial data provision unit 112. For example, the spatial data may include positional data of spatial elements that the robot 2 cannot move, such as walls, furniture, electrical appliances, and steps. Based on the provided spatial data, the robot 2 can determine whether or not it is in a place where it can move. The robot 2 may also be able to recognize whether or not the spatial data includes areas that have not yet been generated. Whether or not an area that has not yet been generated is included can be determined, for example, by whether or not a part of the spatial data includes space that does not contain any spatial elements.

[0050] The instruction unit 113 instructs the robot 2 to take photographs based on the spatial data generated by the spatial data generation unit 13. Since the spatial data generation unit 13 creates spatial data based on the captured images acquired by the captured image acquisition unit 111, for example, when creating spatial data for a room, parts that have not been photographed may include parts for which spatial data has not yet been created. Also, if the captured images are unclear, noise may be included in the created spatial data, resulting in inaccurate parts of the spatial data. The instruction unit 113 may also instruct the robot 2 to take photographs for parts that have not yet been generated in the spatial data. Furthermore, the instruction unit 113 may also instruct the robot 2 to take photographs for inaccurate parts of the spatial data if such inaccuracies exist. The instruction unit 113 may also spontaneously instruct the robot to take photographs based on the spatial data. The instruction unit 113 may also instruct the robot 2 to take photographs based on explicit instructions from the user who has confirmed the visualization data (described later) generated based on the spatial data. The user can specify the area included in the visualization data and instruct the robot 2 to take photographs, thereby allowing the robot 2 to recognize the space and generate spatial data.

[0051] The instruction unit 113 may instruct the robot to photograph markers placed in a region. The photography in a region for which spatial data creation is instructed may include, for example, the coordinate position of the robot 2 (photography unit 21), the shooting direction of the photography unit 21, and other shooting conditions such as resolution. If the spatial data generation unit 13 is instructed to create spatial data relating to a region that has not yet been created, it adds the newly created spatial data to the existing spatial data. If the spatial data generation unit 13 is instructed to create spatial data relating to a re-creation, it generates spatial data that updates the existing spatial data. Furthermore, if the captured image contains markers, the system may generate spatial data that includes the recognized markers.

[0052] The point cloud data generation unit 12 generates three-dimensional point cloud data of spatial elements based on the captured image acquired by the captured image acquisition unit 111. The point cloud data generation unit 12 generates point cloud data by converting the spatial elements contained in the captured image into a set of three-dimensional points in a predetermined space. As mentioned above, spatial elements include walls, steps, doors, furniture, appliances, luggage, and houseplants in a room. Since the point cloud data generation unit 12 generates point cloud data based on captured images of spatial elements, the point cloud data will represent the shape of the surface of the captured spatial elements. Captured image The point cloud data is generated when the robot 2's imaging unit 21 takes a picture at a predetermined position and at a predetermined angle. Therefore, if the robot 2 photographs a spatial element such as furniture from a frontal position, it cannot generate point cloud data for the shape of the back of the furniture or other parts that are not photographed. Even if there is space behind the furniture that the robot 2 can move to, the robot 2 cannot recognize it. On the other hand, if the robot 2 moves and photographs the furniture from a side position, it can generate point cloud data for the shape of the back of the spatial element such as furniture, making it possible to correctly grasp the space.

[0053] The spatial data generation unit 13 generates spatial data that defines the range of movement for the robot 2 based on the point cloud data of spatial elements generated by the point cloud data generation unit 12. Since the spatial data is generated based on point cloud data in space, the spatial elements included in the spatial data also have three-dimensional coordinate information. The coordinate information may include information on the position, length (including height), area, or volume of a point. The robot 2 can determine the range of movement based on the position information of the spatial elements included in the generated spatial data. For example, if the robot 2 has a movement mechanism 29 that moves horizontally on the floor, the robot 2 can determine that movement is impossible if the step from the floor, which is a spatial element in the spatial data, is above a predetermined height (e.g., 1 cm or more). On the other hand, if the tabletop or bed, which is a spatial element in the spatial data, has a predetermined height from the floor, the robot 2 determines that the range in which the height from the floor is above the predetermined height (e.g., 60 cm or more) is the range of movement possible, taking into account the clearance with its own height. Furthermore, robot 2 determines, based on the spatial data, that the area where the gap between spatial elements such as walls and furniture is greater than a predetermined width (for example, 40 cm or more) is within the range in which it can move, taking into account the clearance with its own width.

[0054] The spatial data generation unit 13 may set attribute information for a predetermined area in space. Attribute information is information that defines the movement conditions for the robot 2 in the predetermined area. Movement conditions are, for example, conditions that define the clearance between the robot 2 and spatial elements in which it can move. For example, if the normal movement conditions for the robot 2 are a clearance of 30 cm or more, attribute information can be set to exceptionally set the clearance for the predetermined area to 5 cm or more. In addition, the movement conditions set in the attribute information may include information that restricts the robot's movement. Restrictions on movement include, for example, restrictions on movement speed or prohibitions on entry. For example, attribute information may be set to reduce the movement speed of the robot 2 in areas with small clearance or areas where people are present. Furthermore, the movement conditions set in the attribute information may be determined by the flooring material of the area. For example, the attribute information may set changes to the operation of the movement mechanism 29 (travel speed or means of travel, etc.) depending on whether the floor is cushion flooring, hardwood flooring, tatami mats, or carpet. Furthermore, the attribute information may include settings for charging spots where robot 2 can move to charge, and movement conditions such as steps or carpet edges where robot 2's movement is restricted due to unstable posture. The areas where attribute information is set may be made recognizable to users by changing the display method in the visualization data described later.

[0055] The spatial data generation unit 13 takes the point cloud data generated by the point cloud data generation unit 12, for example, and applies a Hough transform to extract common shapes such as straight lines and curves from the point cloud data. It then generates spatial data using the contours of the spatial elements represented by the extracted shapes. The Hough transform is a coordinate transformation method that, when the point cloud data is treated as feature points, extracts the shape that passes through the feature points most frequently. Since point cloud data represents the shapes of spatial elements such as furniture in a room, it can be difficult for users to distinguish what the spatial elements represented by the point cloud data are (for example, recognizing tables, chairs, walls, etc.). The spatial data generation unit 13 can represent the contours of furniture, etc., by applying a Hough transform to the point cloud data, making it easier for users to distinguish spatial elements. The point cloud data generated in the group data generation unit 12 may be converted into the basic shape of the spatial element recognized by image recognition (e.g., a table, chair, wall, etc.) to generate spatial data. When a spatial element such as a table is recognized as a table by image recognition, the shape of the table can be accurately predicted from the point cloud data of a part of the spatial element (e.g., point cloud data when the table is viewed from the front). By combining the point cloud data and image recognition, the spatial data generation unit 13 can generate spatial data that accurately captures the spatial element.

[0056] The spatial data generation unit 13 generates spatial data based on point cloud data included within a predetermined range from the position where the robot 2 moved. The predetermined range from the position where the robot 2 moved includes the position where the robot 2 actually moved, and may be, for example, a range at a distance of 30 cm from the position where the robot 2 actually moved. Since the point cloud data is generated based on images captured by the robot 2's imaging unit 21, the captured images may include spatial elements located far from the robot 2. If the spatial elements are far from the imaging unit 21, there may be parts that are not captured, or there may be areas where the robot 2 cannot actually move due to the presence of obstacles that are not captured. Also, if spatial elements located far from the imaging unit 21, such as corridors, are included in the captured images, the spatial elements extracted at feature points may be distorted. Furthermore, if the shooting distance is large, the spatial elements included in the captured images become smaller, which may result in lower accuracy of the point cloud data. The spatial data generation unit 13 may generate spatial data that does not include low-accuracy or distorted spatial elements by ignoring feature points that are far away. The spatial data generation unit 13 generates spatial data by deleting point cloud data outside a predetermined range from the robot 2's position. This prevents the occurrence of disconnected areas where data does not actually exist, and enables the generation of highly accurate spatial data that does not include areas where the robot 2 cannot move. Furthermore, it prevents the rendering of disconnected areas in the visualization data generated from the spatial data, thereby improving visibility.

[0057] When a marker is recognized by the marker recognition unit 22, the spatial data generation unit 13 sets a limit range for the generated spatial data. By setting a limit range for the spatial data, it becomes possible to visualize the limit range as part of the visualization data. Furthermore, when state information is acquired by the state information acquisition unit 24, the spatial data generation unit 13 sets state information for the spatial data. By setting state information for the spatial data, it becomes possible to include the state information as part of the visualization data.

[0058] The visualization data generation unit 14 generates visualization data based on the spatial data generated by the spatial data generation unit 13, which visualizes the spatial elements contained in the space in a way that allows a person to intuitively identify them.

[0059] Generally, robots have various sensors such as cameras and microphones, and they perceive their surroundings by comprehensively judging the information obtained from these sensors. In order for a robot to move, it needs to recognize various objects in space and determine a movement route based on spatial data, but the movement route may be inappropriate if objects are not correctly recognized. Due to misrecognition, for example, even if a person thinks there is a sufficiently large space, the robot may perceive that there are obstacles and that it can only move in a narrow area. When such a discrepancy in perception occurs between humans and robots, the robot may act contrary to the person's expectations, causing the person stress. In this embodiment, the autonomous behavioral robot 1 can reduce the discrepancy in perception between humans and robots by visualizing its spatial data, which represents its perception state, and providing it to the person, and can also perform recognition processing again on the areas pointed out by the person.

[0060] The spatial data is data that includes spatial elements recognized by the autonomous robot 1. In contrast, the visualization data is data that allows users to visually confirm the spatial elements recognized by the autonomous robot 1. The spatial data may contain misrecognized spatial elements. Visualizing the spatial data makes it easier for people to check the recognition status of spatial elements by the autonomous robot 1 (whether or not there are misrecognitions, etc.).

[0061] Visualization data is data that can be displayed on a display device. Visualization data is a so-called floor plan, and within a region surrounded by spatial elements recognized as walls, spatial elements recognized as tables, chairs, sofas, etc., are included. The visualization data generation unit 14 generates visualization data, for example, represented as RGB data, of the shapes of furniture etc. formed in the figures extracted by the Hough transform. The spatial data generation unit 13 generates visualization data with a modified plane drawing method based on the direction of the three-dimensional plane of the spatial elements. The direction of the three-dimensional plane of the spatial elements is, for example, the normal direction of the plane formed by the figures generated in the point cloud data generated by the point cloud data generation unit 12 after applying the Hough transform to the point cloud data. The visualization data generation unit 14 generates visualization data with a modified plane drawing method according to the normal direction. The drawing method is, for example, a color attribute such as hue, lightness or saturation applied to the plane, a pattern applied to the plane, or a texture. For example, the visualization data generation unit 14 increases the brightness of the plane and draws it in a bright color when the normal of the plane is vertical (the plane is horizontal). On the other hand, the visualization data generation unit 14 decreases the brightness of the plane and draws it in a dark color when the normal of the plane is horizontal (the plane is vertical). By changing the method of drawing the plane, it becomes possible to represent the shape of furniture, etc., in three dimensions, making it easier for users to confirm the shape of furniture, etc. Furthermore, the visualization data may also include coordinate information in the visualization data ("visualization coordinate information") that is associated with the coordinate information of each spatial element included in the spatial data. Since the visualization coordinate information is associated with the coordinate information, points in the visualization coordinate information correspond to points in actual space, and surfaces in the visualization coordinate information correspond to surfaces in actual space. Therefore, when a user identifies the position of a point in the visualization data, the position of the corresponding point in the actual room can be identified. Furthermore, a conversion function for converting coordinate systems may be provided so that the coordinate system in the visualization data and the coordinate system in the spatial data can be converted to each other. Of course, it would be possible to enable conversion between the coordinate system in the visualization data and the coordinate system in the actual space.

[0062] The visualization data generation unit 14 generates visualization data in three dimensions. Alternatively, the visualization data generation unit 14 may generate visualization data in two dimensions. Generating visualization data in 3D makes it easier for users to confirm the shapes of furniture, etc. The visualization data generation unit 14 may generate visualization data in 3D when sufficient data has been generated in the spatial data generation unit 13. The visualization data generation unit 14 may also generate visualization data in 3D based on a 3D viewpoint position (viewpoint height, viewpoint elevation angle, etc.) specified by the user. By allowing the viewpoint position to be specified, it makes it easier for users to confirm the shapes of furniture, etc. Furthermore, the visualization data generation unit 14 may generate visualization data in which only the back wall of a room is colored, and the front wall or ceiling is made transparent (uncolored). Making the front wall transparent makes it easier for users to confirm the shapes of furniture, etc. placed beyond the front wall (inside the room).

[0063] The visualization data generation unit 14 generates visualization data to which color attributes corresponding to the captured image acquired by the captured image acquisition unit 111 are assigned. For example, if the visualization data generation unit 14 detects that the captured image includes wood-grain furniture and detects the color of the wood grain (for example, brown), it generates visualization data to which a color similar to the detected color is assigned to the extracted furniture shape. By assigning color attributes corresponding to the captured image, it becomes easier for users to identify the type of furniture, etc.

[0064] The visualization data generation unit 14 generates visualization data with modified rendering methods for fixed objects and moving objects. Fixed objects include, for example, walls of a room, steps, and fixed furniture. Moving objects include, for example, chairs, trash cans, and furniture with casters. Moving objects may also include temporary objects placed on the floor, such as luggage or bags. Rendering methods include, for example, color attributes such as hue, lightness, or saturation applied to a plane, patterns applied to a plane, or textures.

[0065] The distinction between a fixed object, a moving object, or a temporary object can be identified by the duration of its presence in that location. For example, the spatial data generation unit 13 identifies whether a spatial element is a fixed object, a moving object, or a temporary object based on the temporal changes in the point cloud data generated by the point cloud data generation unit 12, and generates spatial data. For example, the spatial data generation unit 13 determines that a spatial element is a fixed object if it has not changed, based on the difference between the spatial data generated at a first time and the spatial data generated at a second time. The spatial data generation unit 13 may also determine that a spatial element is a moving object if its position has changed, based on the difference in spatial data. Furthermore, the spatial data generation unit 13 may determine that a spatial element is a temporary object if it has disappeared or appeared, based on the difference in spatial data. The visualization data generation unit 14 changes the drawing method based on the distinction identified by the spatial data generation unit 13. Changing the drawing method may include, for example, color coding, adding hatching, or adding a predetermined mark. For example, the spatial data generation unit 13 may display fixed objects in black, moving objects in blue, or temporary objects in yellow. The spatial data generation unit 13 identifies the categories of fixed objects, moving objects, or temporary objects and generates spatial data. The visualization data generation unit 14 may generate visualization data with a modified drawing method based on the categories identified by the spatial data generation unit 13. The spatial data generation unit 13 may also generate visualization data with a modified drawing method for spatial elements recognized by image recognition.

[0066] The visualization data generation unit 14 can generate visualization data for multiple divided areas. For example, the visualization data generation unit 14 generates visualization data for each of the spaces separated by walls, such as a living room, bedroom, dining room, and corridor, treating each as a separate room. By generating visualization data for each room, it becomes possible to generate spatial data or visualization data separately for each room, making the generation of spatial data easier. It also becomes possible to create spatial data only for areas where the robot 2 may move. The visualization data provision unit 161 provides visualization data for which the user can select an area. The visualization data provision unit 161 may, for example, enlarge the visualization data of the area selected by the user, or provide detailed visualization data of the area selected by the user.

[0067] The target recognition unit 15 performs image recognition of spatial elements based on the captured image acquired by the image acquisition unit 111. Spatial element recognition can be performed, for example, by using an image recognition engine that determines what a spatial element is based on image recognition results accumulated in machine learning. Image recognition of spatial elements can be performed, for example, by recognizing the shape, color, pattern, etc. The target recognition unit 15 may also perform image recognition of spatial elements by using an image recognition service provided on a cloud server (not shown), for example. The visualization data generation unit 14 generates visualization data with a drawing method changed according to the spatial elements recognized by the target recognition unit 15. For example, if the spatial element recognized is a sofa, the visualization data generation unit 14 generates visualization data with a texture that has the feel of cloth attached to the spatial element. Also, if the spatial element recognized is a wall, the visualization data generation unit 14 may generate visualization data with the color attribute of wallpaper (for example, white) attached. By performing such visualization processing, the user can intuitively grasp the spatial recognition state of the robot 2.

[0068] The second communication control unit 16 controls communication with the user terminal 3 owned by the user. The user terminal 3 is, for example, a smartphone, tablet PC, notebook PC, or desktop PC. The communication method with the user terminal 3 is arbitrary, and for example, short-range wireless communication such as wireless LAN, Bluetooth (registered trademark), or infrared communication, or wired communication can be used. The visualization data provision unit 161, the designation acquisition unit 162, and the execution information acquisition unit 163 functions of the second communication control unit 16 communicate with the user terminal 3 using the communication functions controlled by the second communication control unit 16.

[0069] The visualization data provision unit 161 provides the visualization data generated by the visualization data generation unit 14 to the user terminal 3. The visualization data provision unit 161 is, for example, a web server and provides the visualization data as a web page to the browser of the user terminal 3. The visualization data provision unit 161 may also provide visualization data to multiple user terminals 3. By viewing the visualization data displayed on the user terminal 3, the user can confirm the range in which the robot 2 can move as a 2D or 3D display. The visualization data includes shapes of furniture, etc., drawn using a predetermined drawing method. By operating the user terminal 3, the user can, for example, switch between 2D and 3D displays, zoom in or zoom out of the visualization data, or move the viewpoint in the 3D display.

[0070] The user can view the visualization data displayed on the user terminal 3 and check the generation status of spatial data and the attribute information of the area. The user can specify an area from the visualization data where spatial data has not been generated and instruct the creation of spatial data. In addition, if the user views the visualization data displayed on the user terminal 3 and finds an area where the spatial data appears inaccurate, such as when the shape of spatial elements like furniture is unnatural, they can specify that area and instruct the regeneration of the spatial data. As described above, the visualization coordinate information in the visualization data is associated with the coordinate information of the spatial data, so an area in the visualization data that has been specified for regeneration by the user can be uniquely identified as an area in the spatial data. Based on the regenerated spatial data, the visualization data generation unit 14 regenerates the visualization data and provides it from the visualization data provision unit 161. However, even in the regenerated visualization data, there may be cases where the generation status of the spatial data does not change, such as when spatial elements are misrecognized. In such cases, the user may instruct the generation of spatial data by changing the operation parameters of the robot 2. Operating parameters include, for example, the shooting conditions (exposure amount or shutter speed, etc.) of the imaging unit 21 in robot 2, the sensitivity of sensors (not shown), and the clearance conditions for allowing robot 2 to move. Operating parameters may also be included in spatial data as, for example, area attribute information.

[0071] The visualization data generation unit 14 generates visualization data that includes, for example, the display of a button that instructs the creation of spatial data (including "recreation"). The user terminal 3 can send an instruction to the autonomous robot 1 to create spatial data by operating the displayed button. The instruction to create spatial data sent from the user terminal 3 is acquired by the designation acquisition unit 162.

[0072] The designated acquisition unit 162 acquires instructions for creating spatial data of a region specified by the user based on the visualization data provided by the visualization data provision unit 161. The designated acquisition unit 162 may also acquire instructions for setting (including changing) the attribute information of the area. The designated acquisition unit 162 also acquires the location of the region and the direction in which the robot approaches that region, i.e., the direction in which it should be photographed. The acquisition of creation instructions can be performed, for example, by operating a web page provided by the visualization data provision unit 161. This allows the user to understand how the robot 2 perceives space and, according to the recognition state, perform recognition processing. You can instruct robot 2 to redo the task.

[0073] [Additional Features] The status information acquisition unit 24 acquires status information relating to the status of the person to be notified, which has been searched for by the search unit 25. The status of the person to be notified refers to the state of the person's actions, such as being asleep, awake, sitting, standing, walking, or performing tasks such as cooking or cleaning. The health status or mental state of the person to be notified may also be acquired as part of the status of the person to be notified. The status information acquisition unit 24 determines the status of the person to be notified, for example, based on the image of the person to be notified taken by the shooting unit 21. The status information acquisition unit 24 may also determine the status of the person to be notified based on the voice emitted by the person to be notified, collected by a microphone (not shown), the heat distribution of the person to be notified measured by a radiation thermometer, the illuminance of the room measured by a light meter, the movement of the person to be notified detected by a proximity sensor, the heart rate acquired from a heart rate monitor worn by the person to be notified, etc. For example, the status information acquisition unit 24 may determine the sleep state of the person to be notified from this information. Sleep states refer to conditions such as REM sleep, non-REM sleep, and sleep depth, which can be measured from factors such as a person's heart rate and body movements.

[0074] The state information acquisition unit 24 determines that the person being notified is asleep if, for example, the captured image of the person being notified by the imaging unit 21, the heat distribution of the person being notified, or the movement of the person being notified detected by the proximity sensor is small within a predetermined time and operating range. The state information acquisition unit 24 may also determine that the person being notified is awake if the amount of movement of the person being notified (for example, the cumulative or average value of the amount of movement of the person being notified per unit of time) is greater than a predetermined amount of movement within a predetermined time and operating range. The predetermined amount of movement may be a value determined, for example, through experiments. The state information acquisition unit 24 may also determine that the person being notified is sitting based on the posture of the person being notified, or the positional relationship between the person being notified and spatial elements such as a chair. The state information acquisition unit 24 may also determine that the person being notified is standing based on the posture of the person being notified. Furthermore, the status information acquisition unit 24 may determine that the person being notified is walking if their position in space is moving. Alternatively, the status information acquisition unit 24 may determine that the person is working by comparing the person's movements in the captured image with pre-learned movement patterns.

[0075] Furthermore, the status information acquisition unit 24 may determine the status of the person to be notified based on the lighting status and the door open / closed status. For example, if the status information acquisition unit 24 determines from the room illuminance measured by the illuminance meter that the lights are off, it may determine that the person to be notified is asleep.

[0076] The search unit 25 searches for notification targets based on the execution information acquired by the execution information acquisition unit 163. The notification operation execution unit 26 executes a notification operation based on the execution information for the notification targets found by the search unit 25.

[0077] Execution information refers to information necessary for performing a notification action to deliver information to a target person, and is information set by the user for the autonomous robot 1. Execution information includes, for example, information about the target person to identify the target person for whom the notification action should be performed, information about the location to search for the target person, information about the notification action, and information about the time to perform the notification action. Execution information is pre-set by the user and provided to the execution information acquisition unit 163.

[0078] <Information about those who will be reported> Information about the person to be notified is information used to identify the person to be notified, such as the person's name, ID (Identification), information indicating physical characteristics, belongings, and clothing. This includes information such as the person's attire or their level of familiarity with robot 2 (described later). Information indicating the physical characteristics of the person to be notified includes, for example, information for recognizing the person's face (face recognition), information for recognizing the person's fingerprints (fingerprint recognition), and information for recognizing the person's build (shape recognition). Information regarding the person's possessions or clothing includes, for example, information about a wireless tag owned by the person to be notified, or information indicating the characteristics of the person's clothing. The search unit 25 can identify the person to be notified by determining whether or not they are the person to be notified based on the person's information. For example, if multiple people are in one room, the search unit 25 can identify the person to be notified based on the person's information. The person to be notified may, for example, be stored in advance in the autonomous robot 1 along with an ID that identifies the person to be notified, and only the ID may be specified in the execution information.

[0079] Furthermore, the execution information may include information on one or more notification targets. By including information on multiple notification targets in the execution information, robot 2 can sequentially perform notification actions for multiple notification targets. If notification targets are included in the notification target information, a pre-configured notification action will be performed for each notification target. In addition, the execution order (priority) of notification actions for multiple notification targets may be set in the execution information.

[0080] <Information about the exploration location> Information about the location where the person to be notified is searched (search location) is information about the place where the person to be notified is expected to be located (location information), for example, the destination to which robot 2 moves to search for the person to be notified. Location information is, for example, position information indicating a point, line or range in space, or information about a room in which position information in space is pre-registered. The search unit 25 calculates a movement path by the movement mechanism 29 based on the location information included in the execution information. The movement path can be calculated from the current position of robot 2 and the search location. For example, the search unit 25 can store in advance the range that the movement mechanism 29 can move in, and calculate a movement path that allows the robot to move from the current position to the search location in the shortest distance within the range that can be moved in. The search unit 25 may also include the movement speed along the movement path in the calculation of the movement path. For example, the search unit 25 can calculate a movement path so that the movement speed in the corridor and the movement speed in the room are different. The movement control unit 23 controls the movement mechanism 29 to move robot 2 based on the movement path calculated by the search unit 25.

[0081] Furthermore, the search unit 25 may calculate a movement path based on the movement restriction range where movement is restricted at the marker recognized by the marker recognition unit 22. For example, if a no-entry zone is set based on the marker, the search unit 25 will calculate a movement path that avoids the no-entry zone. Also, if the movement speed is restricted based on the marker, the search unit 25 may calculate a movement path that takes the shortest time considering the restricted speed.

[0082] Location information may be specified by the user. For example, the user specifies location information by specifying the search location on the map displayed on the user terminal 3 operated by the user. The execution information acquisition unit 163 can acquire the location information specified by the user from the user terminal 3. The search unit 25 calculates a movement path by the movement mechanism 29 based on the location information specified by the user from the user terminal 3. For example, the search unit 25 calculates a movement path from the home position to which the robot 2 returns for charging to the search location. The search unit 25 may also display an alert on the user terminal 3 if there are markers, stairs, etc. that restrict movement along the calculated movement path. That is, the search unit 25 may calculate the movement path when the robot 2 moves to the search location (immediately before or during movement), or it may calculate the movement path when the user sets the execution information.

[0083] The execution information may include information on one or more search locations. By including information on multiple search locations in the execution information, robot 2 can, for example, search for the person to be notified at other designated search locations if it cannot find them at the initially designated search location.

[0084] <Information about notification operation> The notification action information is information indicating the content of the notification action to be performed on the person to be notified. The notification action execution unit 26 performs the notification action based on the notification action information for the person to be notified that has been found by the search unit 25. A notification action is an action that notifies the person to be notified of notification information using the notification unit 27. In other words, the notification action execution unit 26 can perform a notification action via the notification unit 27. The notification action information includes time information for when the notification action will be performed. The search unit 25 searches for the person to be notified according to the time information.

[0085] The notification unit 27 is, for example, an output device such as a speaker, display, or actuator. The speaker notifies the auditory information of the person being notified by sound (including voice). The display is, for example, a display or light, and notifies the visual information of the person being notified by the information (characters, images, or light, etc.) displayed on the display. The actuator is, for example, a movable part such as a robot hand, a vibration generator, or a compressed air output valve, and notifies the tactile information of the person being notified. The notification unit 27 may also notify the olfactory or gustatory information of the person being notified. The notification action is, for example, outputting sound from the speaker to the person being notified, outputting display information from the display, or making contact with the person being notified by the robot hand. The notification action may be a combination of these notification actions. The notification action can be specified by the user. The notification action may specify the purpose of the notification, for example, "alarm action" or "time notification action". Furthermore, the notification action execution unit 26 may perform a notification action according to information about the person to be notified. For example, if the person's ability to wake up is stored as information about the person to be notified, the notification action execution unit 26 will perform a notification action according to the person's ability to wake up. The ability to wake up can be evaluated based on the time from when the notification action is performed until the person's state changes from sleep to something other than sleep. For example, if the person's information is stored as indicating that the person has difficulty waking up (for example, the average time from when the notification action is performed until the person's state changes from sleep to something other than sleep is greater than the wake-up threshold), the notification action execution unit 26 may perform a notification action with a sound at a predetermined volume or higher. Alternatively, the notification action execution unit 26 may control the duration of sound output to be longer, instead of increasing the volume. Alternatively, the notification action execution unit 26 may control the type of sound to be a first type of sound, instead of increasing the volume. In addition, the notification operation execution unit 26 may control the system so that the increase in volume per unit of time is relatively large, either instead of or in addition to these controls.Furthermore, if it is remembered that the person to be notified wakes up easily (for example, the average time from the execution of the notification action until the person to be in a state other than sleep is less than the wake-up threshold), the notification action execution unit 26 may execute the notification action with a sound below a predetermined volume so as not to startle the person to be notified. Alternatively, the notification action execution unit 26 may control the duration of sound output to be shorter instead of increasing the volume. Alternatively, the notification action execution unit 26 may control the sound to be a second type of sound different from the first type of sound. Alternatively, the notification action execution unit 26 may control the volume to increase by a relatively small amount per unit of time. Furthermore, the notification action execution unit 26 may change the notification action according to the level of intimacy with the person to be notified (described later), or it may change the notification action based on past data.

[0086] [Alarm clock operation] An "alarm clock operation" is a notification operation that wakes up a person who is sleeping based on time information. In the "alarm clock" notification operation, the notification operation execution unit 26 controls the speed of the notification unit 27. The system wakes the person to be notified by outputting an alarm sound from the device, displaying the current time on the display unit, or by making contact with the person to be notified using the robotic arm. The notification action execution unit 26 executes the alarm action if the person to be notified, as detected by the search unit 25, is asleep. Whether or not the person to be notified is asleep can be determined by the status information acquisition unit 24. The notification action execution unit 26 may repeat the alarm action until the person to be notified wakes up. Whether or not the person to be notified has woken up can be determined by the status information acquisition unit 24. Note that the person to be notified waking up may mean that the person does not fall back asleep (fall asleep again after waking up). The notification action execution unit 26 may repeat the alarm action if the status information acquisition unit 24 determines that the person has fallen back asleep.

[0087] [Time notification operation] A "time notification operation" is an operation that notifies a person to be notified of a predetermined time. A time notification operation to a person to be notified is, for example, an operation that notifies the start time of a television broadcast. The notification operation execution unit 26 may output an audio prompt to the person to watch television when the predetermined time arrives. The notification operation execution unit 26 may also turn on the television when the predetermined time arrives. If the person to be notified is not watching television, the notification operation execution unit 26 may record the television program. Whether or not the person to be notified is watching television can be determined by the status information acquisition unit 24 determining whether or not the person to be notified is sitting in a place where they can watch television.

[0088] Furthermore, the time notification operation for the person to be notified may be an operation to notify the person of the time they leave the house. The notification operation execution unit 26 may output a voice prompting the person to leave the house when a preset time has arrived. The notification operation execution unit 26 may also execute locking the doors if the person to be notified leaves the house after a preset time. Whether or not the person to be notified has left the house can be determined by the status information acquisition unit 24 determining that the person has left the front door.

[0089] <Conditions for completion of notification operation> The notification operation information may include the completion conditions for the notification operation. The completion conditions for the notification operation are the conditions under which the notification operation for the notification target is considered to have been completed. The completion conditions for the notification operation may also include actions to be taken when the completion conditions are not met. For example, if the completion conditions are not met, the notification operation execution unit 26 may continue or repeat the notification operation until the completion conditions are met. By including the completion conditions in the notification operation information, it becomes possible to perform notification operations according to the situation of the notification target. The completion conditions for the notification operation may be provided to the autonomous robot 1 as part of the notification operation information, or they may be pre-set in the autonomous robot 1. An example of the completion conditions when the notification operation is an alarm operation is given below.

[0090] For example, the completion condition may be "when it is determined that the person to be notified has woken up at the search location." If it is determined that the person to be notified has woken up at the search location, the purpose of the notification operation has been achieved, and the notification operation can be completed. Whether or not the person to be notified has woken up can be determined by the status information acquisition unit 24 as described above.

[0091] Alternatively, the completion condition may be set as "when the person to be notified cannot be found at the search location." If the person to be notified cannot be found at the search location, it can be assumed that the person to be notified is already awake, and therefore the notification operation can be completed. Whether or not the person to be notified was found at the search location can be determined by the status information acquisition unit 24.

[0092] Alternatively, the completion condition may be defined as "when the person to be notified is found in a location other than the search area." A location other than the search area is, for example, a location along the movement path of robot 2. If the person to be notified is found in a location other than the search area, the person to be notified can be assumed to be already awake, and the notification process can be completed. The discovery of the person to be notified can be performed, for example, by the state information acquisition unit 24 performing image recognition of the person to be notified in the captured image, or by voice recognition of the voice spoken by the person to the robot 2.

[0093] Alternatively, the completion condition may be set as "when a pleasant action is detected from the person being notified." A pleasant action is a predetermined type of action by the person being notified, such as the person "stroking" robot 2, or the person "saying thank you" or "greeting" robot 2. When a pleasant action is detected from the person being notified, the person being notified can be considered to have already woken up, and the notification operation can be completed. Detection of pleasant actions from the person being notified can be performed, for example, by the status information acquisition unit 24.

[0094] <Level of responsibility for notification actions> The notification action information may include the responsibility level of the notification action. The responsibility level of the notification action is information indicating the importance of the notification action to the person being notified. For example, if the responsibility level is high, the notification action execution unit 26 will repeatedly execute the notification action until the state of the person being notified reaches a predetermined state. For example, if the responsibility level is high for an alarm clock action, the notification action execution unit 26 will continue the alarm clock action until the person being notified wakes up. If the responsibility level is high, the robot 2 may start searching for the person being notified earlier to ensure that the alarm clock action is executed reliably at the set time. Also, if the responsibility level is lower than the high level, the notification action execution unit 26 may execute the alarm clock action a predetermined number of times. If the responsibility level is medium, the robot 2 may start searching for the person being notified a predetermined time before the set time. Also, if the responsibility level is lower than the medium level, the notification action execution unit 26 may execute the alarm clock action only once. If the level of responsibility is low, robot 2 may begin searching for the person to be notified only after it has finished performing other notification actions (for example, an alarm action for other people to be notified).

[0095] <Relationship level between the person being reported and the robot> The level of intimacy between the person being notified and robot 2 is an index of the subjective feelings the person being notified has towards robot 2. For example, the person being notified may feel affection for robot 2 based on its shape (e.g., humanoid or animal-like shape), the voice it emits, or its movements. The person being notified may also feel affection based on past interactions with robot 2. The level of intimacy is expressed based on the person being notified's feelings, for example, as a percentage (0-100%) or in multiple levels (S, A, B, and C levels). The autonomous robot 1 may store the level of intimacy and update it according to the person being notified's past actions and opinions. For example, the autonomous robot 1 may update the level of intimacy to be higher if the person being notified performs a pleasant action towards robot 2.

[0096] If there are multiple notification recipients, the affinity level may be set for each recipient. Similarly, if there are multiple robots 2, the affinity level may be set for each robot 2. For example, if there is one notification recipient A and there are two robots 2a and 2b, robot 2a and robot 2b can each set their affinity level with notification recipient A.

[0097] Notification actions based on execution information may be performed according to the level of intimacy between the person being notified and robot 2. For example, in the notification action for person A as described above, if the level of intimacy is high Robot 2a may perform notification actions that include physical contact with the person to be notified, while robot 2b, which has a lower affinity level, may only perform notification actions that involve audio output to the person to be notified. Furthermore, if notification actions are set for robot 2b, which has a lower affinity level, robot 2a may perform the notification actions jointly with robot 2b or on behalf of robot 2. Also, if both robot 2a and robot 2b have high affinity levels, robots 2a and 2b may perform notification actions jointly or competitively. By performing notification actions according to affinity levels, it becomes possible to perform notification actions tailored to the person to be notified.

[0098] The execution information acquisition unit 163 acquires execution information for performing the notification action of information to be performed for the person to be notified. The execution information acquisition unit 163 can acquire execution information from the user terminal 3. The execution information acquisition unit 163 may also acquire execution information by receiving execution information transmitted from the user terminal 3 through the operation of the user operating the user terminal 3. Alternatively, the execution information acquisition unit 163 may acquire execution information by downloading execution information stored in the memory of the user terminal 3. Furthermore, the execution information acquisition unit 163 may acquire execution information from a data server (not shown).

[0099] Furthermore, the execution information acquisition unit 163 can acquire specified location information from the user terminal 3 by the user operating the map displayed on the user terminal 3, based on the visualization data provided to the user by

[0100] As mentioned above, Figure 1 illustrates a case where the autonomous robot 1 is configured with a data provisioning device 10 and a robot 2 separated. However, the functions of the data provisioning device 10 may also be included in the functions of the robot 2. For example, the robot 2 may include all the functions of the data provisioning device 10. The data provisioning device 10 may also temporarily substitute for functions in cases where the processing capacity of the robot 2 is insufficient.

[0101] Furthermore, in this embodiment, "acquisition" may refer to an active acquisition by the acquiring entity, or a passive acquisition by the acquiring entity. For example, the designated acquisition unit 162 may acquire by receiving an instruction to create spatial data transmitted by the user from the user terminal 3, or by reading an instruction to create spatial data stored by the user in a storage area (not shown) from the storage area.

[0102] Furthermore, the functions of the data provision device 10, namely the first communication control unit 11, point cloud data generation unit 12, spatial data generation unit 13, visualization data generation unit 14, target recognition unit 15, second communication control unit 16, captured image acquisition unit 111, spatial data provision unit 112, instruction unit 113, visualization data provision unit 161, designated acquisition unit 162, and execution information acquisition unit 163, are examples of the functions of the autonomous robot 1 in this embodiment and do not limit the functions of the autonomous robot 1. For example, the autonomous robot 1 does not need to have all the functions of the data provision device 10, and may have only some of them. Also, the autonomous robot 1 may have other functions other than those mentioned above. Furthermore, the functions of the robot 2, namely the marker recognition unit 22, movement control unit 23, state information acquisition unit 24, search unit 25, and notification operation execution unit 26, are examples of the functions of the autonomous robot 1 in this embodiment and do not limit the functions of the autonomous robot 1. For example, autonomous robot 1 does not need to have all the functional parts of robot 2; it may have only some of the functional parts.

[0103] Furthermore, as described above, each of the functions of the autonomous robot 1 is implemented by software. However, among the above functions of the autonomous robot 1 And at least one of those functions may be implemented by hardware.

[0104] Furthermore, any of the above functions possessed by the autonomous robot 1 may be implemented by dividing one function into multiple functions. Alternatively, any two or more of the above functions possessed by the autonomous robot 1 may be combined into a single function. In other words, Figure 1 represents the functions possessed by the autonomous robot 1 using function blocks, and does not indicate, for example, that each function is composed of a separate program file.

[0105] Furthermore, the autonomous robot 1 may be a device realized by a single housing, or it may be a system realized by multiple devices connected via a network or the like. For example, the autonomous robot 1 may realize some or all of its functions by a virtual device such as a cloud service provided by a cloud computing system. In other words, the autonomous robot 1 may have at least one of the above functions realized by other devices. Also, the autonomous robot 1 may be a general-purpose computer such as a tablet PC, or it may be a dedicated device with limited functions.

[0106] Furthermore, the autonomous robot 1 may have some or all of its functions implemented by robot 2 or user terminal 3.

[0107] Next, the hardware configuration of the autonomous robot 1 (control unit of robot 2) will be explained using Figure 2. Figure 2 is a block diagram showing an example of the hardware configuration of the autonomous robot 1 in the embodiment.

[0108] The autonomous robot 1 has a CPU (Central Processing Unit) 101, RAM (Random Access Memory) 102, ROM (Read Only Memory) 103, a touch panel 104, a communication interface 105, a sensor 106, and a clock 107. The autonomous robot 1 is a device that executes the autonomous robot control program described in Figure 1.

[0109] The CPU 101 controls the autonomous robot 1 by executing an autonomous robot control program stored in the RAM 102 or ROM 103. The autonomous robot control program is obtained, for example, from a recording medium containing the autonomous robot control program or from a program distribution server via a network, installed in the ROM 103, read by the CPU 101, and executed.

[0110] The touch panel 104 has an operation input function and a display function (operation display function). The touch panel 104 enables the user of the autonomous robot 1 to perform operation input using their fingertip or a stylus. In this embodiment, the autonomous robot 1 is described as using a touch panel 104 with an operation display function, but the autonomous robot 1 may also have a display device with a display function and an operation input device with an operation input function separately. In that case, the display screen of the touch panel 104 can be the display screen of the display device, and the operation of the touch panel 104 can be performed as the operation of the operation input device. The touch panel 104 may be implemented in various forms such as a head-mounted type, glasses type, or wristwatch type display.

[0111] Communication I / F 105 is a communication interface. Communication I / F 105 performs short-range wireless communication such as wireless LAN, wired LAN, and infrared. In Figure 2, only Communication I / F 105 is shown among the communication interfaces; however, the autonomous robot 1 may have multiple communication interfaces for each communication method. Communication I / F 105 is not shown. It may also communicate with a control unit that controls the imaging unit 21 or a control unit that controls the moving mechanism 29.

[0112] Sensor 106 includes hardware such as the camera, TOF or thermal camera, microphone, thermometer, illuminometer, or proximity sensor of the imaging unit 21. The data acquired by this hardware is stored in RAM 102 and processed by CPU 101.

[0113] Clock 107 is an internal clock for acquiring time information. The time information acquired by clock 107 is used, for example, to confirm the time to execute a notification operation. Microphone 108 collects ambient sounds. Microphone 108 collects, for example, the voice of the person to be notified.

[0114] The speaker 109a, display unit 109b, and actuator 109c are specific hardware examples of the notification unit 27 described in Figure 1. The speaker 109a outputs sound, the display unit 109b outputs display data, and the actuator 109c is a movable part. The notification unit 27 may also have hardware other than the speaker 109a, display unit 109b, and actuator 109c.

[0115] Next, Figure 3 will be used to explain the operation related to the provision of visualization data for the robot control program. Figure 3 is a flowchart showing a first example of the operation of the robot control program in the embodiment. In the following flowchart explanation, the entity executing the operations is assumed to be the autonomous robot 1, but each operation is performed by the respective function of the autonomous robot 1 described above.

[0116] In Figure 3, the autonomous robot 1 determines whether or not it has acquired a captured image (step S11). The determination of whether or not it has acquired a captured image can be made by determining whether or not the image acquisition unit 111 has acquired the captured image from the robot 2. The determination of whether or not it has acquired a captured image is made in units of image processing. For example, if the captured image is a video, the video is transmitted continuously from the robot 2, so the determination of whether or not it has acquired a captured image can be made by determining whether or not the number of frames or the amount of data of the acquired video has reached a predetermined value. The acquisition of captured images may be mainly carried out by the mobile robot transmitting the captured images, or it may be mainly carried out by the image acquisition unit 111 retrieving the captured images from the mobile robot. If it is determined that no captured images have been acquired (step S11: NO), the autonomous robot 1 repeats the process in step S11 and waits for the captured images to be acquired.

[0117] On the other hand, if it is determined that an image has been captured (step S11: YES), the autonomous robot 1 generates point cloud data (step S12). Point cloud data generation can be performed by the point cloud data generation unit 12, for example, detecting points with large changes in brightness in the captured image as feature points and assigning three-dimensional coordinates to the detected feature points. Feature point detection may be performed, for example, by performing differential processing on the captured image to detect changes in the gradation of the captured image and detecting parts with large changes in gradation. Assigning coordinates to feature points may also be performed by detecting the same feature points captured from different shooting angles. The determination of whether or not an image has been captured in step S11 can be made by determining whether or not images captured from multiple directions have been acquired.

[0118] After executing the process in step S12, the autonomous robot 1 generates spatial data and recognizes markers (step S13). The spatial data generation can be performed by the spatial data generation unit 13, for example, by performing a Hough transform on point cloud data. Details of step S13 are explained in Figure 4.

[0119] After executing the process in step S13, the autonomous robot 1 provides the generated spatial data to robot 2 (step S14). The spatial data may be provided to robot 2 sequentially each time spatial data is generated, as shown in Figure 3, or it may be provided asynchronously with the processes shown in steps S11 to S18. The robot 2, having received the spatial data, will be able to understand its movable range based on the spatial data.

[0120] After executing the process in step S14, the autonomous robot 1 determines whether or not to recognize a spatial element (step S15). This determination can be made, for example, by setting whether or not to recognize a spatial element in the target recognition unit 15. Even if it is determined that a spatial element should be recognized, if recognition fails, it may be configured to determine that the spatial element was not recognized.

[0121] If the autonomous robot 1 determines that it recognizes a spatial element (step S15: YES), it generates first visualization data (step S16). The generation of the first visualization data can be performed by the visualization data generation unit 14. The first visualization data is the visualization data generated after the object recognition unit 15 recognizes a spatial element. For example, if the object recognition unit 15 determines that the spatial element is a table, the visualization data generation unit 14 can generate visualization data assuming that the top surface of the table is flat, even if the top surface of the table has not been photographed and therefore does not have point cloud data. Also, if the spatial element is determined to be a wall, the visualization data generation unit 14 can generate visualization data assuming that the unphotographed portion is also a plane.

[0122] If the autonomous robot 1 determines that it does not recognize spatial elements (step S15: NO), it generates second visualization data (step S17). The generation of the second visualization data can be performed by the visualization data generation unit 14. The second visualization data is visualization data generated based on point cloud data and spatial data generated from the captured image, without the target recognition unit 15 recognizing spatial elements. By not performing spatial element recognition processing, the autonomous robot 1 can reduce its processing load.

[0123] After performing the processing in step S16 or step S17, the autonomous robot 1 provides visualization data (step S18). The provision of visualization data is performed by the visualization data provision unit 161 providing the visualization data generated in the visualization data generation unit 14 to the user terminal 3. The autonomous robot 1 may, for example, generate and provide visualization data in response to a request from the user terminal 3. After performing the processing in step S18, the autonomous robot 1 terminates the operation shown in the flowchart.

[0124] Next, we will explain the operation related to spatial data generation of the robot control program using Figure 4. Figure 4 is a flowchart showing a second example of the operation of the robot control program in the embodiment.

[0125] In Figure 4, the autonomous robot 1 generates spatial data (step S131). The spatial data generation can be performed by the spatial data generation unit 13, for example, by performing a Hough transform on point cloud data. After performing step S131, the autonomous robot 1 determines whether or not it has recognized the marker (step S132). Whether or not the marker has been recognized can be determined by the marker recognition unit 22, which determines whether or not it has recognized an image of the marker in the captured image taken by the imaging unit 21. The robot 2 can notify the data provision device 10 of the marker recognition result.

[0126] If the marker is recognized (Step S132: YES), the autonomous robot Step 1 sets a limit range that restricts movement in the spatial data generated in step S121 (step S133).

[0127] After executing the process in step S133, or if it determines that it has not recognized the marker (step S132: NO), the autonomous robot 1 terminates the data generation operation in step S13 as shown in the flowchart.

[0128] Next, Figure 5 will be used to explain the notification operation of the robot control program. Figure 5 is a flowchart showing how the autonomous robot control program in the embodiment performs an alarm operation as a notification operation.

[0129] In Figure 5, the autonomous robot 1 determines whether or not it has acquired execution information (step S21). This determination can be made by checking whether the execution information acquisition unit 163 has acquired execution information from the user terminal 3. If it is determined that execution information has not been acquired (step S21: NO), the autonomous robot 1 repeats the process in step S21 and waits for execution information to be acquired.

[0130] On the other hand, if it determines that execution information has been acquired (Step S21: YES), the autonomous robot 1 calculates a movement path (Step S22). The movement path can be calculated by the search unit 25 based on the location information included in the execution information.

[0131] After executing the process in step S22, the autonomous robot 1 begins searching for the person to be notified and also begins moving along the calculated movement path (step S23). The search for the person to be notified can be performed by the search unit 25. Movement can be performed by the movement control unit 23 controlling the movement mechanism 29. The start of the search in step S23 is performed based on time information, and the autonomous robot 1 begins moving before the specified execution time. For example, if the execution time is set to 6:00 AM, the autonomous robot 1 will start moving before 6:00 AM, taking into account the travel time along the movement path, so that the notification operation can be performed at the execution time. The time to start moving may be automatically determined by the autonomous robot 1 based on time information, or it may be set manually by the user.

[0132] After executing the process in step S23, the autonomous robot 1 determines whether or not robot 2 has reached the location designated as the search location (step S24). Whether or not the search location has been reached can be determined, for example, by the movement control unit 23 comparing the search location with the current position (coordinate position) of robot 2. If it is determined that the search location has not been reached (step S24: NO), the autonomous robot 1 repeats the process in step S24 and waits for robot 2 to reach the search location.

[0133] On the other hand, if it is determined that it has reached the search location (Step S24: YES), the autonomous robot 1 determines whether or not it has found the person to be notified (Step S25). Whether or not the person to be notified has been found can be determined, for example, by the search unit 25 based on the information of the person to be notified included in the execution information. Alternatively, if a person is present at the search location, the robot may determine whether or not it has been found by considering that person as the person to be notified. If it is determined that the person to be notified has not been found (Step S25: NO), the autonomous robot 1 returns to the process in Step S23 and begins moving to the next search location. That is, the autonomous robot 1 can sequentially search the specified multiple search locations and find the person to be notified. If the person to be notified is not found at a specified search location, the search operation shown in the flowchart may be terminated, and the fact that the person to be notified was not found may be recorded, or a notification may be sent to the user terminal 3.

[0134] On the other hand, if it determines that the person to be notified has been found (Step S25: YES), the autonomous robot 1 acquires status information (Step S26). The acquisition of status information can be performed by the status information acquisition unit 24. In the flowchart shown, it is assumed that the status information acquired is whether the person to be notified is sleeping or awake.

[0135] After executing the process in step S26, the autonomous robot 1 determines whether or not the person to be notified is asleep (step S27). Whether or not the person to be notified is asleep can be determined by the state information acquisition unit 24. If it is determined that the person to be notified is asleep (step S27: YES), the autonomous robot 1 performs an alarm operation as a notification operation (step S28). The alarm operation can be performed by the notification operation execution unit 26. The notification operation execution unit 26 performs the alarm operation when the current time reaches a predetermined execution time. That is, if the autonomous robot 1 arrives at the search location before the execution time, it waits until the execution time. Also, if it arrives at the search location after the execution time, the autonomous robot 1 immediately performs the alarm operation. The alarm operation may be performed, for example, by determining the sleep state of the person to be notified in the state information acquisition unit 24 and executing it when the person to be notified is in a predetermined sleep state.

[0136] After performing the process in step S28, the autonomous robot 1 performs the process in step S27 again to determine whether the person to be notified is asleep or not. The re-execution of the process in step S27 may be performed after a certain period of time has elapsed, such as 5 minutes, 10 minutes, or 15 minutes.

[0137] On the other hand, if the system determines that the person being notified is not asleep (i.e., awake) (Step S27: NO), the autonomous robot 1 performs a greeting action (Step S29) and then terminates the actions shown in the flowchart. A greeting action may be, for example, a standard phrase such as "Good morning," or an audio output of the current time. Whether or not to perform a greeting action may be pre-configured in the execution information.

[0138] Furthermore, the execution order of the processing in each step of the robot control program operation (robot control method) described in this embodiment is not limited.

[0139] Next, we will explain the execution information using Figure 6. Figure 6 is a diagram showing an example of the execution information in the embodiment.

[0140] In Figure 6, the execution information 1000 has data items for "notification target" and "notification action". The execution information 1000 is set on the user terminal 3 and can be acquired by the execution information acquisition unit 163.

[0141] "Notification target" is information used to identify the notification target who will perform the notification action. In the diagram, three individuals, "A," "B," and "C," are specified in the "Notification Target" ID, but the number of notification targets can be any number of one or more. Note that physical characteristics and other information to identify each notification target may be pre-registered in the autonomous robot 1.

[0142] A "notification action" is information about a notification action to be performed on a person who is to be notified. Multiple notification actions can be set for a single person. The diagram shows that two notification actions, "Notification Action 1" and "Notification Action 2," are set for each person. Priority can be set for each notification action for each person.

[0143] The "notification action" has data items for "search location," "notification action," and "time." The "search location" is information indicating the search location where the target person is being searched. The diagram shows the case where a room such as "children's room," "bedroom," "Western-style room," or "living room" is specified as the search location. Information for each room is provided from the visualization data provision unit 161 to the user terminal 3 and can be shared between the autonomous robot 1 and the user terminal 3. The user can specify the search location by selecting a room from the map displayed on the user terminal 3.

[0144] "Notification action" is information indicating a notification action to be performed on the target person. "Notification action" can be set to include actions such as "alarm action" or "broadcast time notification." Each notification action can have a responsibility level assigned to it. The responsibility level indicates the importance of the notification action that the autonomous robot 1 will perform on the target person. The autonomous robot 1 can modify the notification action according to the assigned responsibility level. For example, the autonomous robot 1 may change the execution order (priority) of the notification actions, the volume of the voice output during the notification action, the content of the voice, the number of times the notification action is performed, or the termination conditions for the notification action, according to the responsibility level. For example, when the autonomous robot 1 performs multiple notification actions, it will prioritize the notification action with the highest responsibility level. Furthermore, by increasing the volume of the voice output or the number of times a high-responsibility notification action is performed, the autonomous robot 1 can ensure that the target person recognizes the notification action with a high probability. The diagram illustrates a case where three levels of responsibility are set: "high level," "medium level," and "low level." The autonomous robot 1 may prioritize "high level" notification actions over "medium level" notification actions, and further prioritize "medium level" notification actions over "low level" notification actions.

[0145] The "Alarm Clock" function can be set to either "with greeting" or "without greeting." Figure 5 illustrates the operation when "with greeting" is selected. "Broadcast Time Notification" is an action that notifies the start of television broadcasts, etc. "Departure Time Notification" is an action that notifies the designated person of the time they will be leaving the house.

[0146] "Time" is the time information for when the notification action is to be performed. "Time" can be set for a notification action that is performed only once. Alternatively, "Time" may be set for a notification action that is repeated daily.

[0147] Although the diagram shows two notification actions as a single execution information for each of the three notification targets, the execution information acquisition unit 163 may, for example, acquire each execution information individually and merge the execution information to schedule the notification actions while considering the priority of notification actions in multiple execution information, the feasibility of execution due to interference in execution times, etc.

[0148] Next, we will explain how to set the execution information using Figure 7. Figure 7 is a diagram showing an example of how to set the execution information in the embodiment.

[0149] In Figure 7, the execution information setting screen 30 is displayed on the user terminal 3's display screen. The execution information setting screen 30 includes a notification target setting unit 311, a time setting unit 312, a notification operation setting unit 313, and a search location setting unit 32. The execution information setting screen 30 is displayed, for example, in an application program (app) on the user terminal 3.

[0150] The notification recipient setting unit 311 is a pull-down menu for selecting the notification recipient. The figure shows that notification recipient A is selected as the notification recipient. The time setting unit 312 is a pull-down menu for selecting the time information to execute the notification operation. The notification operation setting unit 313 is a pull-down menu for selecting the notification operation. In the figure, This indicates that the alarm function has been selected as the intelligent action.

[0151] The search location setting unit 32 displays a floor plan of the layout of the rooms in the home based on visualization data provided by the visualization data provision unit 161, for example, and allows the user to set the search locations for the person to be notified from the floor plan. The figure shows that the search location setting unit 32 displays the home position 321 where the robot 2 returns for charging, the children's room 322, the bedroom 323, and the Western-style room 324. The user sets the selected location by touching at least one of the children's room 322, the bedroom 323, or the Western-style room 324 on the search location setting unit 32.

[0152] For example, if the user sets the child's room 322 as the search location, the search location setting unit 32 displays the movement path 325. The movement path 325 is calculated, for example, by the search unit 25 and provided to the user terminal 3. The figure shows that the movement path 325 from the home position 321 to the child's room 322 is displayed as a dashed line. The search location setting unit 32 may also display a message indicating that there is a problem with movement if movement is restricted along the movement path. For example, if there is a step at the entrance to the child's room 322 and the robot 2 cannot move, this may be indicated by an X mark on the movement path 325.

[0153] Furthermore, the various processes described above in this embodiment may be performed by recording a program for realizing the functions of the device described in this embodiment onto a computer-readable recording medium, loading the program recorded on the recording medium into a computer system, and executing it. The term "computer system" here may include hardware such as an operating system and peripheral devices. Also, if a WWW system is used, the "computer system" shall also include the homepage provisioning environment (or display environment). Furthermore, "computer-readable recording medium" refers to a storage device such as a flexible disk, magneto-optical disk, ROM, flash memory, portable media such as a CD-ROM, or a hard disk built into a computer system.

[0154] Furthermore, "computer-readable recording media" includes volatile memory (e.g., DRAM (Dynamic Random Access Memory)) within computer systems that act as servers or clients when a program is transmitted via a network such as the Internet or a communication line such as a telephone line, which retains the program for a certain period of time. In addition, the above program may be transmitted from the computer system that stores the program in a memory device, etc., to another computer system via a transmission medium or by transmission waves within the transmission medium. Here, the "transmission medium" for transmitting the program refers to a medium that has the function of transmitting information, such as a network such as the Internet or a communication line such as a telephone line. Furthermore, the above program may be for the purpose of realizing a part of the above-mentioned functions. In addition, it may be a so-called differential file (differential program) that realizes the above-mentioned functions in combination with a program already recorded in the computer system.

[0155] While embodiments of the present invention have been described above with reference to the drawings, the specific configuration is not limited to these embodiments, and various modifications (exceptions) are also possible without departing from the spirit of the present invention.

[0156] As an example of such variations, the location where user terminal 3 is located may be set as the search location, or the search operation may be performed using the search location as the search location. The location where user terminal 3 is located can be determined, for example, by the positional relationship between a wireless LAN communication base station (not shown) and user terminal 3. The location where user terminal 3 is located can also be determined by the relationship between robot 2 and user terminal 3. The signal strength of short-range wireless communication between the three devices can be determined. By searching for the person to be notified based on the location of the user terminal 3, the user can set the search location simply by bringing the user terminal to the search location, thus simplifying the setting of the notification operation.

[0157] Furthermore, the application on the user terminal 3 may summon robot 2 at a predetermined time to perform a notification action and cause the notification action execution unit 26 to perform the notification action. For example, the application on the user terminal 3 may, when the user sets a timer in the alarm function of the user terminal 3, summon robot 2 at the time set by the timer and cause a notification action to be performed. In other words, the application on the user terminal 3 can link the general alarm function of the user terminal 3 with robot 2. For example, the alarm function may have a "snooze" function that causes the alarm to sound again after a predetermined time has elapsed and a predetermined number of repetitions have been made, even if the alarm has been stopped once. The application on the user terminal 3 may summon robot 2 and cause a notification action when the alarm starts sounding due to the snooze function. Alternatively, the application on the user terminal 3 may summon robot 2 and cause a notification action when the number of snooze repetitions has ended.

[0158] Furthermore, the setting of execution information may be done by placing markers. For example, by placing a marker at the entrance of a room indicating the search location or the person to be notified, the room on which the marker is placed can be set as the search location or the person to be notified can be set. For example, a person who wishes to receive notification can place a marker indicating that they wish to receive notification on the doorknob of the room, so that Robot 2 recognizes the marker and receives notification from Robot 2. A person who does not wish to receive notification can place a marker indicating that they do not wish to receive notification (such as a "Don't Disturb" display) on the doorknob of the room, so that Robot 2 does not perform notification.

[0159] Furthermore, the execution information may be set based on the notification action performed. For example, the execution information may be set when the person receiving the notification tells robot 2, which performed the notification action, "Wake me up at the same time tomorrow." Setting the execution information based on the notification action performed makes it easier to set the execution information.

[0160] Furthermore, if robot 2 detects, via sensors or other means, that the person being notified has performed an unpleasant action, it may perform a motion that indicates it is reluctant to perform the next notification action. An unpleasant action is an action that is different from a pleasant action, such as hitting robot 2 or scolding robot 2. If robot 2 detects, via sensors or other means, that the person being notified has apologized or performed other actions in response to robot 2 performing a motion that indicates it is reluctant to perform the notification action, robot 2 may perform a motion that indicates it has improved its mood and willing to perform the notification action. In this way, by reacting to unpleasant actions, it becomes possible to deepen communication with robot 2.

[0161] Furthermore, it is conceivable that robot 2 may be unable to move into the room if the door to the room where the person to be notified is located is closed. In this case, robot 2 may stop in front of the door and perform the notification action when the execution time arrives. The notification action execution unit 26 may remain in place and perform the notification action when the execution time arrives, not only when the door is closed, but also when it is unable to reach the search location due to obstacles or restricted areas. The notification action execution unit 26 may perform the notification action at a volume louder than the volume emitted when the search location is reached. In other words, the notification action execution unit 26 takes into account that it is far from the designated search location and performs the notification action at a volume louder than usual so that the person to be notified will notice. The notification action does not only involve simply generating sound, but may also involve generating sound by lightly bumping robot 2's body against a door or wall. Also, robot 2 may bump against a door or person rather than opening the door. The notification action may be performed at a position where it will not cause a collision. Therefore, the notification action execution unit 26 recognizes the door's opening and closing range. The notification action execution unit 26 then moves the robot 2 outside that range and performs the notification action when the robot 2 is located outside the door's opening and closing range. This prevents the robot 2 from coming into contact with the door when the person to be notified notices the notification and opens the door to come out. In addition, the search unit 25 may, only when moving for the purpose of performing the notification action, break the no-entry rule specified by the marker, pass through the no-entry area, and move to the search location where the person to be notified is located.

[0162] <Acquiring execution information via voice input> The autonomous robot 1 may acquire execution information via voice input. The outline of the process for acquiring execution information via voice input will be described below.

[0163] Figure 8 shows an example of the module configuration of the autonomous robot 1 related to the acquisition of execution information by voice input. The autonomous robot 1 has a microphone 108, a voice recognition unit 201, an execution information acquisition unit 163, and an execution information storage unit 204. As described above, the autonomous robot 1 is configured as a system consisting of two devices: robot 2 and data provision device 10. Figure 8 shows the functions related to "acquisition of execution information by voice input" extracted and displayed. Furthermore, whether the functions of the microphone 108, voice recognition unit 201, execution information storage unit 204, and execution information acquisition unit 163 are realized by robot 2, data provision device 10, or other devices can be arbitrarily designed based on the specifications of the autonomous robot 1.

[0164] Microphone 108 inputs the user's voice. Microphone 108 may be located on either robot 2 or data provision device 10. Microphone 108 may also be located outside the autonomous robot 1. For example, a microphone installed indoors where robot 2 is operating may be used. Alternatively, the microphone on user terminal 3 may be used. When inputting execution information by voice, the user speaks a phrase or sentence to the microphone to identify the execution information. In this example, it is assumed that the user, who is the wife, speaks the command, "Wake up my husband in bed at 7 o'clock," in order to wake up the user, who is the husband. The voice recognition unit 201 recognizes the voice data input by microphone 108 and converts the voice data into language data.

[0165] The execution information acquisition unit 163 includes an execution information identification unit 202 and a conversion rule storage unit 203. The execution information identification unit 202 identifies execution information based on language data by referring to conversion rules stored in the conversion rule storage unit 203. The conversion rules associate parameters of execution information with language expressions. Specifically, the conversion rule storage unit 203 stores action expressions corresponding to the type of notification action, person expressions corresponding to the notification target ID, and location expressions corresponding to indoor locations such as a child's room or bedroom. For example, the action expression "Wake them up" is associated with the notification action "Alarm / Greeting", the person expression "Dad" is associated with the notification target ID "A", and the location expression "Bedroom" is associated with the location "Bed". The conversion rules may also associate time expressions corresponding to the time. For example, the time expression "Noon" may be associated with the time "12 o'clock".

[0166] The execution information identification unit 202 identifies the type of notification action corresponding to the action expression if the language data contains an action expression corresponding to the type of notification action. Furthermore, the execution information identification unit 202 identifies the notification target ID corresponding to the person expression if the language data contains a person expression corresponding to the notification target ID. In addition, the execution information identification unit 202 identifies the indoor location corresponding to the location expression if the language data contains a location expression corresponding to an indoor location. The identified indoor location corresponds to the search location. The execution information identification unit 202 identifies the time if the language data contains a time expression such as "7 o'clock," and if it contains a time expression corresponding to the time, The execution information identification unit 202 identifies the time corresponding to that time expression. The identified time corresponds to the notification time. The execution information identification unit 202 identifies execution information including the notification target ID, search location, type of notification action, and notification time. In this example, the execution information identified includes the notification target ID "A", search location "bedroom", type of notification action "alarm / greeting", and notification time "7 o'clock". The execution information storage unit 204 stores the identified execution information.

[0167] The following are eight implementation examples in which an autonomous robot 1 acquires execution information via voice input. In the first implementation example, the data provision device 10 implements the functions of all modules shown in Figure 8. That is, the microphone 108 provided in the data provision device 10 receives the user's voice as input, and the data provision device 10 performs voice recognition processing and execution information identification processing.

[0168] The data providing device 10 includes a microphone 108, a speech recognition unit 201, an execution information identification unit 202, a conversion rule storage unit 203, and an execution information storage unit 204. The speech recognition unit 201 of the data providing device 10 receives speech data from the microphone 108 of the data providing device 10 and converts it into language data. The execution information identification unit 202 of the data providing device 10 identifies execution information based on the language data by referring to the conversion rule storage unit 203 of the data providing device 10. The execution information storage unit 204 of the data providing device 10 stores the identified execution information.

[0169] If the execution information identified in the first implementation example does not include location information, the execution information identification unit 202 may use the location of the microphone 108 that inputs the voice, that is, the location where the data provision device 10 is installed, as the location information in the execution information.

[0170] In the second implementation example, the functions of the module shown in Figure 8 are realized by a mounted device 4 and a data provision device 10 installed indoors where the robot 2 operates. Specifically, the microphone 108 included in the mounted device 4 receives the user's voice, and the data provision device 10 performs voice recognition processing and execution information identification processing. For this purpose, voice data is transmitted from the mounted device 4 to the data provision device 10. One or more mounted devices 4 are identified by ID, and the data provision device 10 pre-registers the installation location (indoor position coordinates) of each mounted device 4.

[0171] Multiple mountable devices 4 may be mounted in different locations. The mountable devices 4 can transmit data to the data providing device 10 by either wired communication or wireless communication, or both. The wireless communication method may be, for example, short-range wireless communication such as wireless LAN, Bluetooth®, or infrared communication, or wired communication. The wired communication method may be, for example, wired LAN.

[0172] The mountable device 4 includes a microphone 108 and an audio data transmission unit (not shown). The audio data transmission unit transmits the audio data input by the microphone 108 of the mountable device 4 to the data providing device 10. The data providing device 10 includes an audio data receiving unit (not shown), an audio recognition unit 201, an execution information identification unit 202, a conversion rule storage unit 203, and an execution information storage unit 204. The audio data receiving unit receives the audio data sent from the mountable device 4. The audio recognition unit 201 of the data providing device 10 converts the audio data received by the audio data receiving unit into language data. The execution information identification unit 202 of the data providing device 10 identifies execution information based on the language data by referring to the conversion rule storage unit 203. Unit 204 of the data providing device 10 stores the identified execution information.

[0173] In the second implementation example, if the identified execution information does not include location information, the execution information acquisition unit 163 may use the location of the microphone 108 that inputs the voice, that is, the location where the mountable device 4 is attached, as location information in the execution information. For example, if it is attached in a bedroom When the voice input "Please wake me up in 30 minutes" is entered into the microphone 108, the bedroom may be used as the location information in the execution information. Note that this example assumes that the user is alone and the notification recipient ID can be omitted. Furthermore, the execution information identification unit 202 determines the notification time by adding 30 minutes to the current time based on the time expression "30 minutes later". In addition, the execution information identification unit 202 determines the type of notification action "Alarm / No greeting" based on the action expression "Please wake me up".

[0174] Figure 9 shows an example of the module configuration of the data provision device 10 related to the identification of the mounting location of the mounting device 4. In other words, Figure 9 shows an example of the module configuration of the data provision device 10 corresponding to the second implementation example, in particular, related to the location identification function of the mounting device 4 (microphone 108) that picked up sound. The execution information acquisition unit 163 of the data provision device 10 has a location identification unit 211 and an installation location storage unit 212. The installation location storage unit 212 stores the mounting location of each mounting device 4 that is installed in a predetermined location indoors. That is, the installation location storage unit 212 stores the installation location in association with the ID of each mounting device 4. In the second implementation example, after the execution information is identified by the execution information identification unit 202, the location identification unit 211 determines whether or not the execution information includes location information. If the execution information includes location information, the location identification unit 211 finishes processing. If the execution information does not include location information, the location identification unit 211 obtains the ID of the mounting device 4, which is the source of the audio data, from the audio data receiving unit. The location identification unit 211 then refers to the mounting location storage unit 212 to identify the mounting location corresponding to the ID of the source mounting device 4, and writes the identified mounting location to the execution information storage unit 204 as the search location. In other words, the mounting location of the mounting device 4 is stored as location information in the execution information.

[0175] In the third implementation example, the functions of the module shown in Figure 8 are realized by the user terminal 3 and the data provision device 10. The user terminal 3 here can be any computer terminal such as a smartphone or laptop PC. In the third implementation example, the microphone 108 built into the user terminal 3 is utilized. The microphone 108 of the user terminal 3 inputs the user's voice, and the data provision device 10 performs voice recognition processing and execution information identification processing. For this purpose, voice data is transmitted from the user terminal 3 to the data provision device 10.

[0176] In the fourth implementation example, the user terminal 3 has a microphone 108 and an audio data transmission unit (not shown). The audio data transmission unit transmits the audio data input by the microphone 108 of the user terminal 3 to the data provision device 10. The module configuration of the data provision device 10 is the same as in the second implementation example. In other words, the third implementation example differs from the second implementation example in that it uses a user terminal 3 as a movable general-purpose product, rather than a fixed, dedicated mounting device 4.

[0177] The fourth implementation example also realizes the functions of the module shown in Figure 8 using the user terminal 3 and the data provision device 10. The user terminal 3 inputs the user's voice and performs speech recognition processing. The data provision device 10 performs execution information identification processing. For this purpose, language data is transmitted from the user terminal 3 to the data provision device 10. The fourth implementation example differs from the third implementation example in that the user terminal 3 performs not only voice acquisition but also speech recognition.

[0178] In the fourth implementation example, the user terminal 3 has a microphone 108, a speech recognition unit 201, and a language data transmission unit (not shown). The speech recognition unit 201 of the user terminal 3 converts the speech data input by the microphone 108 of the user terminal 3 into language data. The language data transmission unit transmits the converted language data to the data providing device 10. The data providing device 10 has a language data receiving unit (not shown), an execution information identification unit 202, a conversion rule storage unit 203, and an execution information storage unit 204. The language data receiving unit receives the language data sent from the user terminal 3. The execution information identification unit 202 of the data provision device 10 identifies execution information based on the received language data by referring to the conversion rule storage unit 203 of the data provision device 10. The execution information storage unit 204 of the data provision device 10 stores the identified execution information.

[0179] The fifth implementation example also realizes the functions of the module shown in Figure 8 using the user terminal 3 and the data provision device 10. The user terminal 3 inputs the user's voice and performs voice recognition processing and execution information identification processing. Then, the execution information is transmitted from the user terminal 3 to the data provision device 10. The fifth implementation example differs from the fourth implementation example in that, in addition to voice acquisition and voice recognition, the user terminal 3 also identifies the execution information.

[0180] In the fifth implementation example, the user terminal 3 includes a microphone 108, a speech recognition unit 201, an execution information identification unit 202, a conversion rule storage unit 203, and an execution information transmission unit (not shown). The speech recognition unit 201 of the user terminal 3 converts the speech data input by the microphone 108 of the user terminal 3 into language data. The execution information identification unit 202 refers to the conversion rule storage unit 203 to identify execution information based on the converted language data. The execution information transmission unit transmits the identified execution information to the data providing device 10. The data providing device 10 includes an execution information receiving unit (not shown) and an execution information storage unit 204. The execution information receiving unit receives the execution information sent from the user terminal 3. The execution information storage unit 204 of the data providing device 10 stores the received execution information.

[0181] In the third to fifth implementation examples, if the execution information acquired by voice input does not include location information, the execution information acquisition unit 163 may use the location of the microphone 108 that input the voice, i.e., the location of the user terminal 3, as the location information in the execution information. As described above, the third to fifth implementation examples have in common that they start processing starting from the voice data acquired at the user terminal 3.

[0182] Figure 10 shows an example of the module configuration of the data provision device 10 related to the identification of the location of the user terminal 3. In other words, Figure 10 shows an example of the module configuration of the user terminal 3 and data provision device 10 corresponding to the third to sixth implementation examples, in particular the location identification function of the user terminal 3 (movable microphone 108). The user terminal 3 has a location measuring unit 221 and a terminal location transmitting unit 222. The location measuring unit 221 measures the current location of the user terminal 3 based on beacon signals transmitted from multiple beacon transmitters installed at predetermined locations indoors, for example. In this method, the user terminal 3 is equipped with a beacon receiver, which receives beacon signals transmitted from beacon transmitters installed at predetermined locations and identifies the ID of the beacon transmitter. The location measuring unit 221 identifies the current location of the user terminal 3 by analyzing the relationship between the radio wave strength of the received beacon signal and the distance between the beacon transmitter identified by the ID and the user terminal 3. The beacon transmitter may be included in the mounted device 4, or it may be provided separately from the mounted device 4.

[0183] The terminal location transmission unit 222 transmits the current location of the user terminal 3 to the data provision device 10. In the third implementation example, the terminal location transmission unit 222 transmits the current location of the user terminal 3, for example, before or after the transmission of voice data by the voice data transmission unit. In the fourth implementation example, the terminal location transmission unit 222 transmits the current location of the user terminal 3, for example, before or after the transmission of language data by the language data transmission unit. In the fifth implementation example, the terminal location transmission unit 222 transmits the current location of the user terminal 3, for example, before or after the transmission of execution information by the execution information transmission unit.

[0184] The data provision device 10 includes a terminal location receiving unit 223, a location identification unit 224, a floor plan data storage unit 225, and an execution information storage unit 204. The location identification unit 224 and the floor plan data storage unit 225 may be included in the execution information acquisition unit 163. The floor plan data storage unit 225 stores the range of each indoor location. Here, a location refers to a room such as a child's room or a bedroom. This refers to the internal area. The terminal location receiving unit 223 receives the current location of the user terminal 3 from the user terminal 3. After the execution information is identified by the execution information identification unit 202, the location identification unit 224 determines whether or not location information is included in the execution information. If location information is included in the execution information, the location identification unit 224 finishes processing. If location information is not included in the execution information, the location identification unit 224 obtains the current location of the user terminal 3 from the terminal location receiving unit 223. Then, the location identification unit 224 refers to the floor plan data storage unit 225 and writes the location including the current location of the user terminal 3 as location information in the execution information stored in the execution information storage unit 204. Since the location information in the execution information indicates the search location, it means that the search will be performed in the location including the current location of the robot 2. This concludes the explanation of Figure 10.

[0185] In the sixth implementation example, the functions of the module shown in Figure 8 are realized by the robot 2 and the data provision device 10. The user's voice is input through the microphone 108 of the robot 2, and the data provision device 10 performs voice recognition processing and execution information identification processing. For this purpose, voice data is transmitted from the robot 2 to the data provision device 10.

[0186] Robot 2 has a microphone 108 and a voice data transmission unit (not shown). The voice data transmission unit transmits the voice data input by the microphone 108 of robot 2 to the data provision device 10. The data provision device 10 is the same as in the second implementation example.

[0187] In the seventh implementation example, the robot 2 and the data provision device 10 realize the functions of the module shown in Figure 8. The robot 2 receives the user's voice as input and performs speech recognition processing. The data provision device 10 performs execution information identification processing. For this purpose, language data is transmitted from the robot 2 to the data provision device 10.

[0188] Robot 2 includes a microphone 108, a speech recognition unit 201, and a language data transmission unit (not shown). The speech recognition unit 201 of robot 2 converts the speech data input by the microphone 108 of robot 2 into language data. The language data transmission unit transmits the converted language data to the data provision device 10. The data provision device 10 is the same as in the fourth implementation example.

[0189] In the eighth implementation example, the robot 2 data provision device 10 also realizes the functions of the module shown in Figure 8. Robot 2 receives the user's voice as input and performs voice recognition processing and execution information identification processing. Then, the execution information is transmitted from robot 2 to the data provision device 10.

[0190] Robot 2 includes a microphone 108, a speech recognition unit 201, an execution information identification unit 202, a conversion rule storage unit 203, and an execution information transmission unit (not shown). The speech recognition unit 201 of Robot 2 converts the speech data input by the microphone 108 of Robot 2 into language data. The execution information identification unit 202 refers to the conversion rule storage unit 203 to identify execution information based on the converted language data. The execution information transmission unit transmits the identified execution information to the data providing device 10. The data providing device 10 is the same as in the fifth implementation example.

[0191] In the sixth to eighth implementation examples, if the execution information acquired by voice input does not include location information, the execution information acquisition unit 163 may use the location of the microphone 108 that input the voice, that is, the location of the robot 2, as the location information in the execution information. As described above, the sixth to eighth implementation examples have in common that they start processing starting from the voice data acquired by the robot 2.

[0192] Figure 11 shows an example of the module configuration of the data provision device 10 related to the identification of the location of robot 2. In other words, Figure 11 shows the robots corresponding to the 6th to 8th implementation examples. Regarding the 2 and the data provision device 10, a module configuration example is shown, particularly concerning the location identification function of the robot 2 (movable microphone 108). The robot 2 has a movement control unit 23 and a robot position transmission unit 231. The robot position transmission unit 231 obtains the current position of the robot 2 from the movement control unit 23 and transmits the current position of the robot 2 to the data provision device 10. The movement control unit 23 may measure the current position based on radio waves received from a wireless communication device installed at a predetermined location, or it may measure the current position based on captured images. For example, the movement control unit 23 measures the current position of the robot 2 based on beacon signals transmitted from a plurality of beacon transmitters installed at predetermined locations indoors. In this method, the robot 2 is equipped with a beacon receiver, which receives beacon signals transmitted by beacon transmitters installed at predetermined locations and identifies the ID of the beacon transmitter. The movement control unit 23 identifies the current position of the robot 2 by analyzing the relationship between the radio wave strength of the received beacon signal and the distance between the beacon transmitter identified by the ID and the robot 2. Alternatively, the movement control unit 23 may determine its current position using SLAM (Simultaneous Localization and Mapping) technology, which simultaneously estimates its own position and creates an environmental map.

[0193] In the sixth implementation example, the robot position transmission unit 231 transmits the current position of robot 2 before or after the transmission of voice data by the voice data transmission unit, for example. In the seventh implementation example, the robot position transmission unit 231 transmits the current position of robot 2 before or after the transmission of language data by the language data transmission unit, for example. In the eighth implementation example, the robot position transmission unit 231 transmits the current position of robot 2 before or after the transmission of execution information by the execution information transmission unit, for example.

[0194] In addition to the location identification unit 224 and floor plan data storage unit 225 described above, the data provision device 10 also has a robot position receiving unit 232. The robot position receiving unit 232 receives the current position of robot 2 from robot 2. After the execution information is identified by the execution information identification unit 202, the location identification unit 224 determines whether or not the execution information includes location information. If the execution information includes location information, the location identification unit 224 finishes processing. If the execution information does not include location information, the location identification unit 224 obtains the current position of robot 2 from the terminal position receiving unit 223. Then, the location identification unit 224 refers to the floor plan data storage unit 225 and writes the location including the current position of robot 2 as location information in the execution information stored in the execution information storage unit 204. Since the location information in the execution information indicates the search location, it means that the search will be performed in the location including the current position of robot 2.

[0195] The notification action execution unit 26 may execute the notification action immediately upon acquiring the execution information if the execution information acquired in each of the first to eighth implementation examples does not include time information relating to the time of execution of the notification action. Alternatively, the notification action execution unit 26 may execute the notification action after a predetermined time (e.g., 1 minute) has elapsed from the time the execution information was acquired, if the execution information acquired by the execution information acquisition unit 163 does not include time information relating to the time of execution of the notification action. The predetermined time is the length of time that is imagined to be required for the character of robot 2 to understand and respond to instructions from the user. This concludes the explanation regarding the acquisition of execution information by voice input.

[0196] <Information regarding those who were reported after the announcement> Next, we will explain how to provide information about the person who received the notification after the notification has been issued. After the notification operation is performed, the robot 2 may take a picture of the person who received the notification and provide the captured image from the data provision device 10 to the user terminal 3. Furthermore, after the notification operation is performed, the robot 2 may acquire status information of the person who received the notification and provide the status information from the data provision device 10 to the user terminal 3.

[0197] For example, if robot 2 performs an alarm function with the user's husband as the target of notification, robot 2 will then use its camera to take a picture of the user's husband. The robot 2 takes the main photograph after confirming that the user is visible in the preview image obtained from the camera. Therefore, the image recorded by the robot 2's camera during the main photograph will show the user (husband) after the alarm has been set off. The image may be a still image or a video. The image is sent to the data provider 10 for storage. On the other hand, when the wife (user) instructs the application on the user terminal 3 to view the image, the application on the user terminal 3 sends a request for the image taken after the alarm has been set off to the data provider 10. In response to this request, the data provider 10 sends the stored image to the application on the user terminal 3. The application on the user terminal 3 displays the received image on the display device of the user terminal 3. In this way, the wife (user) can confirm whether the husband (user) woke up after the robot 2 set off the alarm by looking at the image displayed on the display device of the user terminal 3.

[0198] Figure 12 shows an example of the module configuration of an autonomous robot 1 that provides captured images and status information after performing a notification operation. Figure 12 shows an example of the module configuration of the robot 2 and the data provision device 10, in particular the information provision function. First, the provision of captured images will be explained. The robot 2 has a shooting unit 21 and a captured image transmission unit 241. After the robot 2 performs a notification operation, the shooting unit 21 takes a photograph of the person to be notified and generates a captured image after the notification operation is performed. The shooting unit 21 performs the actual shooting after confirming, for example, that the face of the person to be notified is visible in the live view image using face recognition processing. The captured image transmission unit 241 transmits the captured image after the notification operation is performed to the data provision device 10.

[0199] The data provision device 10 includes a captured image receiving unit 242, a captured image storage unit 243, and a captured image provision unit 244. After the robot 2 performs a notification action, the captured image receiving unit 242 receives the captured image sent from the robot 2, and the captured image storage unit 243 stores the received captured image. When the application on the user terminal 3 requests a captured image after the notification action, the captured image provision unit 244 reads the captured image stored in the captured image storage unit 243 and transmits it to the application on the user terminal 3. In this way, the application on the user terminal 3 can view the captured image showing the person being notified after the notification action has been performed. Furthermore, the data provision device 10 can also store the captured image after the notification action as a record of the person being notified's reaction.

[0200] Next, I will explain how to provide status information. For example, when robot 2 performs an alarm function with the husband (user) as the target of notification, robot 2 acquires status information indicating whether the husband (user) is sleeping or awake. This status information is sent to and stored in data provision device 10. On the other hand, when the wife (user) instructs the application on user terminal 3 to view the status information, the application on user terminal 3 sends a request for the status information after the alarm function to data provision device 10. In response to this request, data provision device 10 sends the stored status information to the application on user terminal 3. Based on the received status information, the application on user terminal 3 displays a message on the display device of user terminal 3 indicating whether the husband (user) is sleeping or awake. If the status information indicates that he is sleeping, a message such as "Dad is still sleeping." is displayed. If the status information indicates that he is awake, a message such as "Dad is already awake." is displayed. In this way, the wife (the user) can confirm whether her husband (the user) woke up after the robot 2 performed the alarm function, based on the message displayed on the display device of the user terminal 3.

[0201] Robot 2 has a status information acquisition unit 24 and a status information transmission unit 251. After Robot 2 performs a notification operation, the status information acquisition unit 24 acquires status information related to the status of the person being notified. As mentioned above, the status information indicates, for example, whether the person is sleeping or awake. The status information transmission unit 251 transmits the status information after the notification operation to the data provision device 10.

[0202] The data provision device 10 includes a status information receiving unit 252, a status information storage unit 253, and a status information provision unit 254. After the robot 2 performs a notification action, the status information receiving unit 252 receives the status information sent from the robot 2, and the status information storage unit 253 stores the received status information. When the status information provision unit 254 receives a request from the application on the user terminal 3 for status information of the person being notified after the notification action has been performed, it reads the status information stored in the status information storage unit 253 and sends it to the application on the user terminal 3. In this way, the application on the user terminal 3 can view status information related to the state of the person being notified after the notification action has been performed. For example, the user of the user terminal 3, who is the wife, can check whether her husband, the person being notified, woke up after the robot 2 performed the alarm action by looking at the status information displayed on the display device of the user terminal 3. In addition, the data provision device 10 can also store the status information after the notification action has been performed as a record of the person being notified's reaction.

[0203] The image provision unit 244 and the status information provision unit 254 may transmit the captured image and status information together to the user terminal 3. For example, if the user, who is the wife, instructs the application on the user terminal 3 to view the captured image and status information, the application on the user terminal 3 sends a request for the captured image and status information after the alarm has been set to the data provision device 10. In response to this request, the data provision device 10 sends the stored captured image and status information to the application on the user terminal 3. The application on the user terminal 3 displays a message on the display device of the user terminal 3 indicating whether the user, who is the husband, is sleeping or awake, based on the received status information, along with the captured image. If the user, who is the husband, is sleeping, the captured image of the sleeping husband and the message "Dad is still sleeping." are displayed simultaneously. If the user, who is the husband, is awake, the captured image of the awake husband and the message "Dad is now awake." are displayed simultaneously.

[0204] Furthermore, a live mode may be provided in which the currently captured image and status information are automatically sent from the data provision device 10 to the application on the user terminal 3, making them immediately viewable on the user terminal 3. The live mode is set by user operation in the application on the user terminal 3. When the live mode is set in the application on the user terminal 3, a live mode setting instruction is sent to the data provision device 10. When the data provision device 10 receives the live mode setting instruction, it immediately transmits the captured image. In other words, the captured image provision unit 244 immediately transmits the captured image received from the robot 2 and stored in the captured image storage unit 243 to the application on the user terminal 3. The application on the user terminal 3 immediately displays the received captured image on the display device of the user terminal 3. In addition, in live mode, a message representing the current status information of the person being notified may be displayed in the application on the user terminal 3. For this reason, the status information provision unit 254 immediately transmits the status information received from the robot 2 and stored in the status information storage unit 253 to the application on the user terminal 3. The application on user terminal 3 may immediately display a message representing the received status information on the display device of user terminal 3.

[0205] In the live mode for alarm operation, the captured image provision unit 244 may stop transmitting the captured image when the status information of the person to be notified switches from sleeping to awake. At the same time, the status information provision unit 254 may stop transmitting the status information. This eliminates the transmission of unnecessary data.

[0206] The imaging unit 21 may change the conditions for imaging the target person after performing a notification operation, depending on the level of intimacy between the target person and the robot. A positive correlation may be established between the level of intimacy between the target person and the robot and the imaging time. In other words, the higher the level of intimacy between the target person and the robot, the longer the imaging time for the video, and similarly, the lower the level of intimacy, the shorter the imaging time. The video capture time may be set to minimize the time between the person being notified and the robot. Furthermore, a negative correlation may be established between the level of intimacy between the person being notified and the robot and the capture distance. In other words, the capture distance may be set such that the higher the level of intimacy between the person being notified and the robot, the shorter the capture distance, and similarly, the lower the level of intimacy, the longer the capture distance. The movement control unit 23 determines a position where the distance between the robot 2 and the person being notified is equal to the capture distance, and controls the movement mechanism 29 to move the robot 2 to that position.

[0207] <Supplementary information regarding the level of intimacy between the person being reported and the robot> In the section above titled "Intimacy between the notification target and the robot," it was explained that if the intimacy level satisfies predetermined intimacy conditions, the robot will cooperate with another robot (Robot 2) to perform the notification action. This point is further explained below.

[0208] The predetermined intimacy condition may be a condition that the intimacy level is above a certain threshold. The predetermined intimacy condition may also be a condition that the intimacy level is below a certain threshold. Furthermore, the condition may include the relationship between the intimacy levels of other robots 2 and the person being notified. For example, the intimacy condition may be that the intimacy level A between this robot 2a and the person being notified is greater than or equal to the intimacy level B between other robots 2b and the person being notified multiplied by a predetermined ratio. In other words, the intimacy condition may be the result of comparing the weighted intimacy levels of some or all of the multiple robots 2.

[0209] <Two robots working together to provide notification> Three specific examples of collaborative notification actions by two robots 2 are shown. In all of these examples, robots 2a and 2b operate in the same indoor space.

[0210] Figure 13 shows a first specific example of a notification operation performed collaboratively by two robots 2. In this first example, robot 2a, which has found the person to be notified, notifies robot 2b of the location where it was found, and robot 2b, upon receiving the notification, moves to the location and performs the notification operation. In other words, when robot 2a performs the notification operation, it calls robot 2b over, and the summoned robot 2b also performs the notification operation together with robot 2a.

[0211] For example, when robot 2a finds a person to be notified sleeping in the bedroom and attempts to wake them up, it notifies robot 2b that it has found the person in the bedroom. Robot 2a then begins the wake-up process for the person to be notified. Meanwhile, when robot 2b receives notification that the person to be notified has been found in the bedroom, it begins moving towards the bedroom. Upon arriving at the bedroom, robot 2b begins the wake-up process for the person to be notified together with robot 2a. As a result, the number of robots performing the wake-up process increases from one to two, and the wake-up effect on the person to be notified intensifies midway through. Therefore, even if the person to be notified is in a deep sleep and does not wake up easily, it becomes easier to wake them up.

[0212] Robot 2a has a discovery location notification transmission unit (not shown) that transmits a notification of the location where the person to be notified has been found, i.e., a notification of the discovery location, to robot 2b, and robot 2b has a discovery location notification receiving unit (not shown) that receives the discovery location notification from robot 2a.

[0213] In step S31, as explained in step S21 of Figure 5 and in the section "Acquisition of execution information by voice input", the execution information acquisition unit 163 of robot 2a acquires execution information. Then, in step S32, as explained in step S22 of Figure 5, the search unit 25 of robot 2a calculates a movement path to the search location. Then, in step S33, as explained in step S23 of Figure 5, the search unit 25 of robot 2a starts searching for the person to be notified, and the movement control unit 23 of robot 2a controls the movement mechanism 29 to start moving to the search location.

[0214] In step S34, as explained in step S24 of Figure 5, the movement control unit 23 of robot 2a determines that robot 2a has reached the search location. Furthermore, in step S35, as explained in step S25 of Figure 5, when the search unit 25 of robot 2a finds the person to be notified, the discovery location notification transmission unit of robot 2a transmits a notification of the discovery location to robot 2b (step S36). Then, in step S37, as illustrated in steps S27 to S29 of Figure 5, the notification operation execution unit 26 of robot 2a performs the notification operation.

[0215] When the robot 2b's location notification receiving unit receives a location notification from robot 2a, the robot 2b's search unit 25 calculates a movement path to the location (step S41). Then, the robot 2b's movement control unit 23 controls the movement mechanism 29 to start moving to the location (step S42). When the robot 2b's movement control unit 23 determines that robot 2b has reached the location (step S43), the robot 2b's notification action execution unit 26 performs a notification action (step S44).

[0216] In the example in Figure 13, only robot 2a performs the search, and robot 2b does not. However, robot 2b may perform the search simultaneously with robot 2a. Furthermore, if robot 2b finds the person to be notified first, contrary to the example in Figure 13, robot 2b may send a location notification to robot 2a, and upon receiving the location notification, robot 2a may move to the location and perform the notification action. For this purpose, robot 2b may further have a location notification transmission unit, and robot 2a may further have a location notification reception unit.

[0217] The discovery location notification may be transmitted via the data provision device 10. For example, the discovery location notification transmission unit of robot 2a may transmit the discovery location notification to the data provision device 10, and the discovery location notification transfer unit (not shown) of the data provision device 10 may receive the discovery location approach notification and transfer the received discovery location notification to robot 2b. The discovery location notification receiving unit of robot 2b may receive the transferred discovery location notification.

[0218] Figure 14 shows a second specific example of a notification operation performed collaboratively by two robots 2. In this second example, when both robots 2a and 2b approach a person to be notified to a distance below a predetermined standard, robots 2a and 2b perform a notification operation in sync. Hereafter, approaching a person to a distance below a predetermined standard will be referred to as "approaching the person to be notified."

[0219] In the example shown in Figure 14, robot 2a, which approaches the person to be notified first, waits until robot 2b approaches the person to be notified. When robot 2b approaches the person to be notified, robots 2a and 2b are programmed to perform notification actions in sync.

[0220] For example, when robot 2a finds a person to be notified sleeping in the bedroom, it approaches the person. Then, robot 2a notifies robot 2b that it has approached the person to be notified. Robot 2a waits without performing any alarm actions until robot 2b approaches the person to be notified. Meanwhile, robot 2b also finds a person to be notified sleeping in the bedroom and approaches the person to be notified. When both robot 2a and robot 2b are close to the person to be notified, they simultaneously begin the alarm action for the person to be notified. Therefore, the alarm effect is strong, making it easier to wake the person to be notified. It is also possible to create the illusion that robots 2a and 2b are playing a prank together.

[0221] Robot 2a has a proximity notification transmitting unit (not shown) that transmits a notification (hereinafter referred to as "proximity notification") to the other robot 2b when it approaches a person to be notified, and a proximity notification receiving unit (not shown) that receives the proximity notification from the other robot 2b. Robot 2b similarly has a proximity notification transmitting unit and a proximity notification receiving unit.

[0222] The processing of robot 2a shown in steps S51 to S54 in Figure 14 is the same as the processing of robot 2a shown in steps S31 to S34 in Figure 13. When the search unit 25 of robot 2a finds a person to be notified and determines that it is approaching the person (step S55), the proximity notification transmission unit sends a proximity notification to robot 2b (step S56). Then, robot 2a waits until it receives a proximity notification from robot 2b.

[0223] The processing of robot 2b shown in steps S61 to S63 in Figure 14 is the same as the processing of robot 2a shown in steps S31 to S33 in Figure 13. It is assumed that robot 2b receives an approach notification from robot 2a before arriving at the search location. Therefore, after the approach notification receiving unit of robot 2b receives an approach notification from robot 2a, the movement control unit 23 of robot 2b determines that robot 2a has arrived at the search location (step S64). Furthermore, when the search unit 25 of robot 2b finds the person to be notified and determines that it has approached the person to be notified (step S65), the approach notification transmitting unit of robot 2b sends an approach notification to robot 2a (step S66). The notification operation execution unit 26 of robot 2b determines that predetermined cooperation conditions have been met and performs a notification operation. The predetermined cooperation conditions are that robot 2b has received an approach notification from the other robot 2a and that its own robot 2b has approached the person to be notified.

[0224] When the proximity notification receiving unit of robot 2a receives a proximity notification from robot 2b, the notification action execution unit 26 of robot 2a determines that predetermined cooperation conditions have been met and performs a notification action. The predetermined cooperation conditions are that robot 2a has received a proximity notification from the other robot 2b and that robot 2a itself has approached the person to be notified.

[0225] The proximity notification may be transmitted via the data provision device 10. For example, the proximity notification transmission unit of robot 2a may transmit a proximity notification to the data provision device 10, and the proximity notification transfer unit (not shown) of the data provision device 10 may receive the proximity notification and transfer the received proximity notification to robot 2b. The proximity notification receiving unit of robot 2b may receive the transferred proximity notification.

[0226] Figure 15 shows a third specific example of a notification operation performed collaboratively by two robots 2. In this third example, the timing of the notification operation by robot 2a and the timing of the notification operation by robot 2b are staggered. Therefore, the timing of the notification operations by the two robots 2a and 2b do not overlap. For example, since the voice output from robot 2a and the voice output from robot 2b do not occur simultaneously, it is not noisy and the content of the voice is easy to hear.

[0227] The third specific example relates to the control of robots 2a and 2b during the notification operation. In other words, it relates to the control from the timing after the notification operation shown in step S44 of Figure 13 in the first specific example, or from the timing after the notification operation shown in step S57 and step S67 of Figure 14 in the second specific example.

[0228] Robot 2a has an action notification transmitting unit (not shown) that transmits a notification (hereinafter referred to as "action notification") to the other robot 2b that it has performed an action notification, and an action notification receiving unit (not shown) that receives an action notification from the other robot 2b. Robot 2b also has an action notification transmitting unit and an action notification receiving unit.

[0229] In the example shown in Figure 15, robot 2a performs the notification action first, and then robot 2b takes over, each performing two notification actions. First, the notification action execution unit 26 of robot 2a performs the notification action (step S71). After completing the notification action, the operation notification transmission unit of robot 2a sends an operation notification to robot 2b (step S72). Then, robot 2a The robot waits until it receives an operation notification from robot 2b.

[0230] When the motion notification receiving unit of robot 2b receives a motion notification from robot 2a (step S81), it waits for a predetermined time, and then the notification action execution unit 26 of robot 2b performs a notification action (step S82). The predetermined waiting time is at intervals (for example, 1 to 5 seconds) that give the impression that robots 2a and 2b are repeatedly sending notifications while observing the state of the person being notified, without rushing. After the notification action is completed, the motion notification transmitting unit of robot 2b sends a motion notification to robot 2a (step S83). Then, robot 2b waits until it receives a motion notification from robot 2a.

[0231] Subsequently, when the motion notification receiving unit of robot 2a receives an motion notification from robot 2b (step S73), it waits for a predetermined time, and then the notification action execution unit 26 of robot 2a performs the notification action (step S74). After the notification action is completed, the motion notification transmitting unit of robot 2a sends the motion notification to robot 2b (step S75). Then, robot 2a completes the entire process.

[0232] Robot 2b repeats the same process as in steps S81 to S83 (steps S84 to S86) to complete the entire process.

[0233] For example, if the notification action is an alarm clock action, robots 2a and 2b may both finish processing the third specific example at the moment they determine that the status information of the person being notified is awake.

[0234] For example, if the person being notified wakes up after the first alarm action by robot 2a, followed by the first alarm action by robot 2b, and then the second alarm action by robot 2a, the second alarm action by robot 2b will not be performed. In other words, unnecessary alarm actions will not be performed on the person being notified who has already woken up.

[0235] <Notification action upon receiving a message> When autonomous robot 1 receives a message addressed to a user, it may perform a notification action targeting that user. Four examples of notification actions based on message reception are shown below.

[0236] First, let's explain a concrete example of the first method. By sending an email from the parent user terminal 3 to the child user with the message "You can eat the cake in the refrigerator" in the body, the robot 2 can be made to read aloud the message to the child who is in the designated search location "child's room". In this example, it is assumed that the child's user information, including the child's email address and the child's search location "child's room", has been set in advance.

[0237] When a user terminal 3 sends an email to its child's email address, with the message "You can eat the cake in the refrigerator," the autonomous robot 1 receives this email.

[0238] Autonomous robot 1 determines from the recipient's email address that the person to be notified is a child, and robot 2 moves to the "child's room," which is the location where the child is being searched, and searches for the child. Once robot 2 finds the child, it reads aloud the body of the email address: "You can eat the cake in the refrigerator."

[0239] Thus, in the first method, the person to be notified is identified based on their email address, and the notification Based on the user information of the target individual, the location of the person to be notified is identified. Then, as a notification action, the body of the email is read aloud.

[0240] Next, a specific example of the second method will be explained. The wife's user terminal 3 can send an email to her husband, the user, with the location expression "@bedroom" set in the body of the email, which will cause robot 2 to perform a predetermined notification action, the "alarm action," for the husband who is in the bedroom. In this example, it is assumed that there is a message description rule in which a location expression specifying the search location is written after "@". In this example, it is assumed that the husband's user information, including the husband's email address and the notification action for the husband, the "alarm action," has been set in advance.

[0241] When the wife's user terminal 3 sends an email to her husband's email address, with the body containing the location expression "@bedroom", the autonomous robot 1 receives this email.

[0242] Autonomous robot 1 determines from the recipient's email address that the person to be notified is the husband, and identifies the search location "bedroom" from the location expression "@bedroom" in the body of the email address. Furthermore, autonomous robot 1 identifies the notification action "alarm action" to be performed on the husband based on the husband's user information. Then, robot 2 moves to the search location "bedroom" and searches for the husband. When robot 2 finds the husband, it performs the notification action "alarm action" on him.

[0243] In this second method, the person to be notified is identified based on their email address, the search location is identified from the body of the email, and the notification action to be taken for the person is determined based on the user information of the person to be notified.

[0244] Next, we will explain a specific example of the third method. The wife's user terminal 3 can send an email to her husband, the user, with the message "Please call me. @bedroom" in the body of the email. This allows robot 2 to read aloud the message "Please call me" directed to the husband who is in the bedroom. Similar to the second example, we assume that there is a message description rule in which a location expression specifying the search location is written after "@". Robot 2 reads aloud the text up to "@" and does not read the location expression. In this example, we assume that the husband's email address is already set as the husband's user information.

[0245] When the wife's user terminal 3 sends an email to her husband's email address, with the message "Please call me. @bedroom" in the body, the autonomous robot 1 receives this email.

[0246] Autonomous robot 1 determines from the recipient's email address that the person to be notified is the husband, and identifies the search location "bedroom" corresponding to the location expression "@bedroom" in the body of the email address. Then, robot 2 moves to the search location "bedroom" and searches for the husband. When robot 2 finds the husband, it reads aloud the words from the body of the email address, excluding the location expression "@bedroom," which read "Please call me."

[0247] In this third method, the recipient of the notification is identified based on their email address, and the search location is determined from the body of the email. The notification action then involves reading the body of the email aloud.

[0248] Finally, a concrete example of the fourth method will be explained. By sending an email from the wife's user terminal 3 to the husband user, the robot 2 can perform a predetermined notification action, an "alarm action," to the husband who is in a predetermined search location, the "bedroom." In this example, robot 2 The notification action does not refer to the email body, so the email body is optional. The email body can be empty. In this example, it is assumed that the husband's user information is pre-configured with his email address, his search location "bedroom", and the notification action for him "alarm clock action".

[0249] When the wife sends an email to her user terminal 3, with her husband's email address set as the recipient, the autonomous robot 1 receives this email.

[0250] Autonomous robot 1 determines that the recipient of the notification is the husband based on the email address. Furthermore, based on the husband's user information, autonomous robot 1 identifies the husband's search location, "bedroom," and the notification action to be performed on him, "alarm clock action." Then, robot 2 moves to the search location, "bedroom," and searches for the husband. Once robot 2 finds the husband, it performs the notification action "alarm clock action" on him.

[0251] In this fourth method, the person to be notified is identified based on their email address, and the location to search for the person and the notification action to be taken are identified based on the user information of the person to be notified.

[0252] Furthermore, the method of message transmission is not limited. Messages may be transmitted by methods other than email. For example, messages may be transmitted by messaging applications.

[0253] Next, we will explain the details of the notification operation upon message reception. Below, we will first provide a comprehensive explanation of methods 1 through 4, and then discuss the characteristics of each method.

[0254] User information is set, for example, by an application on user terminal 3. The set user information is transmitted to data provision device 10 by a user information transmission unit (not shown) in the application on user terminal 3.

[0255] The data provision device 10 includes a user information receiving unit (not shown), a user information storage unit (not shown), a message receiving unit (not shown), and a notification target identification unit (not shown). User information received by the user information receiving unit is stored in the user information storage unit. The user information storage unit stores user information such as user identification information for message communication, search location, and type of notification action, associated with the user. User identification information for message communication is, for example, an email address or a message exchange application ID. The user information storage unit may also store information about the user, such as the user's name, information indicating the user's physical characteristics, belongings, clothing, and level of familiarity with robot 2. Information indicating the user's physical characteristics is, for example, information for recognizing the face of the notification target, information for recognizing the fingerprints of the notification target, and information for recognizing the physique of the notification target. The user information receiving unit, user information storage unit, message receiving unit, and notification target identification unit may also be located in robot 2. In that case, the user information transmission unit in the application of the user terminal 3 may transmit the user information to the robot 2.

[0256] Figure 16 is a flowchart showing the notification operation upon message reception. When the message receiving unit receives a message addressed to the user of robot 2 (step S71), the notification target identification unit refers to the user information storage unit and identifies the user corresponding to the user identification information for message communication set as the destination of the message as the notification target (step S72).

[0257] The search unit 25 refers to the user information storage unit and searches for a user who is the target of notification. Identify the location (step S73). The search unit 25 may identify the search location based on the location expression contained in the received message. For example, if the message contains the location expression "@bedroom", the search unit 25 may identify "bedroom" as the search location.

[0258] In step S74, as explained in step S22 of Figure 5, the search unit 25 calculates the movement path to the search location. Then, in step S75, as explained in step S23 of Figure 5, the search unit 25 starts searching for the person to be notified, and the movement control unit 23 controls the movement mechanism 29 to start moving to the search location.

[0259] In step S76, as explained in step S24 of Figure 5, the movement control unit 23 determines that the robot 2 has reached the search location. Furthermore, in step S77, as explained in step S25 of Figure 5, the search unit 25 finds the person to be notified. In step S78, the notification action execution unit 26 performs a notification action. In this example, the notification action is, for example, reading aloud the received message. The type of notification action may be indicated by the received message. The notification action execution unit 26 may also refer to the user information storage unit to identify the type of notification action corresponding to the user who is the person to be notified.

[0260] In the first method, in step S73, the search unit 25 refers to the user information storage unit to identify the search location corresponding to the user who is the target of notification. Furthermore, in step S78, the notification operation execution unit 26 reads out the received message.

[0261] In the second method, in step S73, the search unit 25 identifies the search location based on the location representation contained in the received message. Furthermore, in step S78, the notification operation execution unit 26 refers to the user information storage unit to identify the type of notification operation corresponding to the user who is the target of the notification.

[0262] In the third method, in step S73, the search unit 25 identifies the search location based on the location representation contained in the received message. Furthermore, in step S78, the notification operation execution unit 26 reads out the received message.

[0263] In the fourth method, in step S73, the search unit 25 refers to the user information storage unit to identify a search location corresponding to the user who is the target of notification. Furthermore, in step S78, the notification operation execution unit 26 refers to the user information storage unit to identify the type of notification operation corresponding to the user who is the target of notification.

[0264] Furthermore, methods 1 through 4 may be combined in any way. For example, method 1 and method 2 may be combined. Method 1 and method 3 may be combined. Method 1 and method 4 may be combined. Method 2 and method 3 may be combined. Method 2 and method 4 may be combined. Method 3 and method 4 may be combined. Method 1, method 2, and method 3 may be combined. Method 1, method 2, and method 4 may be combined. Method 1, method 3, and method 4 may be combined. Method 2, method 3, and method 4 may be combined. Method 1, method 2, method 3, and method 4 may be combined.

[0265] In relation to the combination of the above-described methods, in step S73, the search unit 25 may, if the received message contains a location expression, identify the search location by the location expression, or, if the received message does not contain a location expression, refer to the user information storage unit to identify the search location corresponding to the user who is the target of notification. In step S78, the notification operation execution unit 26 may, if the content of the received message is empty, refer to the user information storage unit to identify the type of notification operation corresponding to the user who is the target of notification, or, if the content of the received message is not empty, read aloud the received message.

Claims

1. An execution information acquisition unit acquires execution information for performing an action to notify the person to be notified of the information to be performed, Based on the execution information acquired by the execution information acquisition unit, a search unit searches for the person to be notified, A notification operation execution unit executes the notification operation based on the execution information for the person to be notified who was found in the search unit. A robot equipped with [the following features].

2. The robot according to claim 1, wherein the execution information acquisition unit acquires location information relating to a location specified by the user as the execution information.

3. The robot according to claim 2, wherein the execution information acquisition unit acquires the location information specified by the user by the user operating a map displayed on the user terminal operated by the user.

4. The robot according to any one of claims 1 to 3, wherein the search unit further searches for the person to be notified based on captured images of the surrounding space.

5. The robot according to claim 4, wherein the search unit searches for the person to be notified by recognizing a person included in the captured image.

6. The movement control unit for controlling the movement mechanism is further provided, The search unit calculates the movement path by the movement mechanism based on the execution information, The robot according to any one of claims 1 to 5, wherein the movement control unit controls the movement mechanism based on the movement path calculated by the search unit.

7. The robot according to claim 6, wherein the search unit calculates the movement path based on restriction information for restricting movement by the movement mechanism.

8. The system further includes a marker recognition unit that recognizes predetermined markers included in captured images of the surrounding space, The robot according to claim 6 or 7, wherein the movement control unit controls the movement mechanism based on the marker recognized by the marker recognition unit.

9. The system further includes a status information acquisition unit that acquires status information relating to the status of the person to be notified, which has been searched by the search unit. The robot according to any one of claims 1 to 8, wherein the notification operation execution unit changes the notification operation according to the state acquired by the state information acquisition unit.

10. The status information acquisition unit acquires the status of whether the person to be notified is asleep or awake. The notification operation execution unit performs the notification operation as follows: If the person in the aforementioned state is asleep, an awakening action will be performed to wake the person to be notified. The robot according to claim 9, which performs a greeting action to the person to be notified if the state is while the person is awake.

11. The execution information acquisition unit acquires the execution information associated with the person to be notified, The notification operation execution unit executes the notification operation associated with the discovered notification target. A robot according to any one of claims 1 to 10, which performs the operation.

12. The execution information acquisition unit acquires intimacy information indicating the degree of intimacy between the person to be notified and the robot as the execution information. The robot according to any one of claims 1 to 11, wherein the notification operation execution unit performs the notification operation in cooperation with other robots when the intimacy level in the intimacy information acquired by the execution information acquisition unit satisfies predetermined intimacy conditions.

13. The execution information acquisition unit acquires the execution information associated with a plurality of notification recipients, The robot according to any one of claims 1 to 12, wherein the notification operation execution unit executes the notification operations associated with each of the multiple notification target persons in parallel.

14. The execution information acquisition unit acquires time information relating to the time during which the notification operation is performed as the execution information. The robot according to any one of claims 1 to 13, wherein the notification operation execution unit executes the notification operation based on the time information.

15. It further includes a speech recognition unit that recognizes the sound input to the microphone and converts it into language data. The robot according to any one of claims 1 to 14, wherein the execution information acquisition unit identifies the execution information by the converted language data.

16. The robot according to claim 15, wherein the notification operation execution unit executes the notification operation when a predetermined time has elapsed from the time the execution information was acquired, if the acquired execution information does not include time information relating to the time for executing the notification operation.

17. The robot according to claim 15 or 16, wherein, if the execution information acquisition unit does not include location information, the location of the microphone that input the voice is used as location information.

18. After performing the notification operation, the camera unit photographs the person to be notified, The robot according to any one of claims 1 to 17, further comprising a transmission unit for transmitting image data of the person to be notified, which has been photographed, to a user terminal.

19. The robot according to claim 18, wherein the imaging unit changes the conditions for imaging the person to be notified according to the level of intimacy between the person to be notified and the robot.

20. After performing the notification operation, a status information acquisition unit acquires status information relating to the status of the person to be notified, The robot according to any one of claims 1 to 19, further comprising a transmitting unit for transmitting the acquired state information to a user terminal.

21. A message receiving unit that receives messages addressed to the robot's user, A notification recipient identification unit identifies the recipient of the notification based on the recipient of the received message, A search unit searches for the person to be notified at a location specified by the message or a location identified in relation to the person to be notified, A notification unit that reads out the message or performs the notification action instructed in the message to the person to be notified who has been found by the search unit. A robot equipped with [the following features].

22. In robotics, An execution information acquisition step to acquire execution information for performing an action to notify the target person of the information to be taken, A search step to search for the person to be notified based on the execution information obtained in the execution information acquisition step, A notification action execution step in which the notification action is performed based on the execution information for the person to be notified who was found in the search step, and A robot control method, including the above.

23. To the robot, An execution information acquisition process that acquires execution information in order to perform the notification action of the information that should be performed for the person to be notified, Based on the execution information obtained in the execution information acquisition process, a search process is performed to search for the person to be notified, A notification action execution process that executes the notification action based on the execution information for the person to be notified who was found in the search process, A robot control program to execute [the command / action].

24. In robotics, A message receiving step of receiving a message addressed to the user of the robot, A step to identify the person to be notified based on the recipient of the received message, A search step of searching for the person to be notified at a location specified by the message or a location identified in relation to the person to be notified, A notification step in which the notification target person found in the search step reads the message aloud or performs the notification action instructed in the message. A robot control method, including the above.

25. To the robot, The robot receives a message, and A process for identifying the recipient of notification based on the recipient of the received message, A search process to search for the person to be notified at the location specified by the message or at a location identified in relation to the person to be notified, A notification process is performed to the person to be notified who was found in the search process, by reading out the message or performing the notification action instructed in the message. A robot control program to execute [the command / action].

Images