Information processing apparatus, remote operation control method, remote operation control system, and autonomous moving object

Abstract

The present technology pertains to an information processing device, a remote operation control method, a remote operation control system, and an autonomous moving body, whereby it is possible to efficiently perform remote operation on the autonomous moving body. An information processing apparatus according to the present technology includes: a matching unit that matches a non-autonomous task, which is not a task autonomously executed by an autonomous moving body, with a remote operator performing a remote operation of the non-autonomous task, among tasks obtained by decomposing an action of the autonomous moving body; and a remote control unit that controls remote operation of the non-autonomous task by a remote operator. This technology can be applied to, for example, a robot.

Description

Technical Field

[0001] This technology relates to information processing devices, remote operation control methods, remote operation control systems, and autonomous mobile bodies, specifically to information processing devices, remote operation control methods, remote operation control systems, and autonomous mobile bodies applicable to situations involving remote operation of autonomous mobile bodies. Background Technology

[0002] In recent years, systems for remotely operated robots have been researched and developed (see, for example, Patent Document 1).

[0003] Reference List

[0004] Patent documents

[0005] Patent Document 1: Japanese Patent Application Publication No. 2012-230506 Summary of the Invention

[0006] The problem to be solved by the present invention

[0007] However, for example, in the invention described in Patent Document 1, since a robot requires a remote operator, it is believed that the burden on the remote operator increases and the work efficiency decreases.

[0008] This technology is proposed to address this situation, with the aim of enabling efficient remote operation of autonomous moving bodies such as robots.

[0009] Solution to the problem

[0010] The information processing apparatus according to a first aspect of the present technology includes: a matching unit that matches non-autonomous tasks, which are not autonomously performed by the autonomous mobile body, with remote operators performing remote operations of the non-autonomous tasks among tasks obtained by decomposing the actions of an autonomous mobile body; and a remote control unit that controls the remote operation of the non-autonomous tasks by the remote operators.

[0011] According to the information processing method of the first aspect of the present technology, the information processing device performs: matching a non-autonomous task, which is not autonomously performed by the autonomous mobile body, with a remote operator performing the remote operation of the non-autonomous task, among tasks obtained by decomposing the actions of an autonomous mobile body; and controlling the remote operator to remotely operate the non-autonomous task.

[0012] According to a second aspect of the present technology, an information processing system includes multiple autonomous mobile bodies and an information processing device, wherein the information processing device includes: a matching unit that matches non-autonomous tasks not autonomously performed by the autonomous mobile bodies with remote operators performing remote operations of the non-autonomous tasks among tasks obtained by decomposing the actions of each autonomous mobile body; and a remote control unit that controls the remote operation of the non-autonomous tasks by the remote operators.

[0013] The autonomous mobile body according to a third aspect of the present technology includes: an action planning unit that decomposes the action to be performed into multiple tasks; and an action control unit that controls the autonomous execution of autonomous tasks within the tasks and controls the execution of non-autonomous tasks within the tasks through remote operation.

[0014] In a first aspect of this technology, among tasks obtained by decomposing the actions of an autonomous mobile body, a non-autonomous task that is not autonomously performed by the autonomous mobile body is matched with a remote operator performing the remote operation of the non-autonomous task, and the remote operator's remote operation of the non-autonomous task is controlled.

[0015] In a second aspect of this technology, among the tasks obtained by decomposing the actions of an autonomous mobile body, a non-autonomous task that is not autonomously performed by the autonomous mobile body is matched with a remote operator performing the remote operation of the non-autonomous task, and the remote operator's remote operation of the non-autonomous task is controlled.

[0016] In a third aspect of this technology, the action to be performed is decomposed into multiple tasks, and the autonomous execution of autonomous tasks within the tasks is controlled, as well as the execution of non-autonomous tasks within the tasks that are not autonomously executed, is controlled through remote operation. Attached Figure Description

[0017] Figure 1 This is a diagram used to describe an overview of this technology.

[0018] Figure 2 This is a block diagram illustrating a configuration example of a remote operation control system that utilizes this technology.

[0019] Figure 3 This is a block diagram illustrating an example of the robot's functional configuration.

[0020] Figure 4 This is a block diagram showing a configuration example of the robot's appearance.

[0021] Figure 5 This is a block diagram illustrating an example of the server's functional configuration.

[0022] Figure 6 This is a flowchart used to illustrate the processes in a robot.

[0023] Figure 7 It is a flowchart used to illustrate the details of task execution control processing.

[0024] Figure 8 This is a flowchart used to illustrate the processing in the server.

[0025] Figure 9 This is a view used to illustrate an example of a robot performing actions to measure the vital signs of a person being cared for.

[0026] Figure 10 This is a view used to illustrate an example of a robot performing actions to measure the vital signs of a person being cared for.

[0027] Figure 11 This is a view used to describe an example of a robot performing actions to measure the vital signs of a person being cared for.

[0028] Figure 12 This is a view used to describe an example of a robot performing actions to measure the vital signs of a person being cared for.

[0029] Figure 13 This is a diagram showing the calculation formula for the operational effect of a remote operation control system.

[0030] Figure 14 This is a diagram illustrating an example of the calculation of the operational effect of a remote operation control system.

[0031] Figure 15 This is a block diagram illustrating an example of a computer configuration. Detailed Implementation

[0032] In the following text, embodiments of the present technology will be described in detail with reference to the accompanying drawings. Note that the description will proceed in the following order.

[0033] 0. Background of this technology

[0034] 1. Implementation Method

[0035] 2. Modified Example

[0036] 3. Other

[0037] <<0. Background of this technology>>

[0038] There is a labor shortage in care facilities such as nursing homes, and there is a desire to introduce robots for care (hereinafter referred to as care robots) to perform various care for users of the care facilities.

[0039] However, autonomous robots capable of stably performing a variety of tasks requiring attentive service have not yet been realized in unstructured environments such as nursing facilities where humans and robots coexist.

[0040] For example, conventional autonomous robots struggle to adapt appropriately to dynamically changing environments or to interact effectively with each user who possesses different characteristics. Specifically, for instance, a conventional autonomous robot cannot continue operating if an unknown obstacle (such as a power cord, cardboard box, etc.) is present in its path. For example, in a conventional autonomous robot, when the position of objects such as tables or chairs in a room changes, the robot loses its target point and cannot perform navigation. For example, because users in facilities for the elderly often speak unclearly and at low volumes, conventional autonomous robots have a low rate of speech recognition for these users.

[0041] On the other hand, it is conceivable that a remote operator can remotely control the robot in real time, enabling the robot to diligently perform actions or interactions.

[0042] However, because conventional remotely operated robots require a remote operator, the operator's efficiency becomes very low, and necessary costs increase. Therefore, having one person perform the work can improve efficiency and significantly reduce costs compared to using a remotely operated robot.

[0043] Depending on environmental conditions, a remote operator can operate multiple remotely operated robots, but the number of remotely operated robots that can be operated is limited, and the remote operator needs to have a high level of skill.

[0044] On the other hand, in this technology, the robot's actions are subdivided into multiple micro-level tasks (hereinafter referred to as micro-tasks), the robot autonomously performs tasks within the possible scope, and tasks that the robot cannot perform autonomously are performed through remote operation.

[0045] Note that in the following text, tasks performed autonomously by the robot will be referred to as autonomous tasks, and tasks not performed autonomously by the robot will be referred to as non-autonomous tasks.

[0046] Microtasks here are, for example, tasks broken down to the following levels: these levels are independent of the context (e.g., background or surrounding environment) in which the robot performs its actions, and are tasks that do not require advanced operational skills.

[0047] Furthermore, the robot includes modules specific to a class of skills (corresponding to a single skill) (hereinafter referred to as task execution modules), and each task execution module independently performs tasks associated with each class of skills. With this arrangement, a remote operator can cause the robot to perform skill-related tasks, whereby each remote operator can handle the task by remotely manipulating the skill-specific task execution module.

[0048] Through this arrangement, for example, such as Figure 1 As shown, a remote operation control system corresponding to a skill-specific task execution module has been implemented.

[0049] exist Figure 1 In the example, the robot comprises five types of modules: action module, interaction module, observation module, work module, and movement module. Among these modules, the interaction module, observation module, work module, and movement module are skill-specific task execution modules (hereinafter referred to as task execution modules).

[0050] The motion module is the module that controls the overall movement of the robot. For example, the motion module creates the motion plan for the entire robot, breaks down each motion into tasks, and causes the corresponding task execution module to execute each task so that the robot performs the planned motion.

[0051] An interaction module is a module that performs interactive processing on the robot autonomously or through remote operation. For example, the interaction module interacts with humans and performs processes such as information communication and mutual understanding.

[0052] An observation module is a module that performs observation processing for a robot autonomously or through remote operation. For example, the observation module observes the robot's surrounding environment and performs processing to identify surrounding objects (including living organisms such as humans) or the surrounding state of the environment.

[0053] A work module is a module that performs work processes on a robot autonomously or through remote operation. For example, a work module uses a part of the robot, such as a hand, to perform work. Specifically, for example, a work module uses this part of the robot to perform manipulations, movements, creations, and processing of various objects, as well as to perform movements such as robot gestures.

[0054] A mobility module is a module that performs the robot's movement processes autonomously or through remote operation. For example, a mobility module performs the robot's movement processes by using the robot's moving parts, such as legs or wheels.

[0055] Then, for example, a dedicated remote operator for performing "interaction", a dedicated remote operator for performing "observation", a dedicated remote operator for performing "work", and a dedicated remote operator for performing "movement" perform remote operations of the robot by using the task execution module of the corresponding skill of each robot.

[0056] Note that in the following text, the interactive operator, the observation operator, the work operator, and the mobile operator will also be referred to as the interactive operator, the observation operator, the work operator, and the mobile operator, respectively.

[0057] Note that operators with multiple skills can also perform remote operations using multiple skills.

[0058] Furthermore, although not shown, a motion operator capable of remotely manipulating the entire robot can perform remote operations, such as those on the entire robot, or remote operations on tasks that are inseparable from the context or involve multiple skills, by using a motion module. For example, the motion operator designs motion rules for the robot's autonomous motion control and, when the motion module is unable to autonomously perform the robot's motion tasks, performs remote operations on the entire robot's motion by using the motion module.

[0059] An action operator is someone who can perform the highest-level actions of a robot and, for example, needs to have special permissions. Examples include employees of companies providing robot services, employees of commercial facilities that introduce and operate robots, etc. In the case of nursing robots, for instance, staff of nursing facilities that introduce and operate nursing robots, staff of home care service providers that introduce and operate nursing robots, and family members of users receiving home care from nursing robots, etc., are action operators.

[0060] Note that the action operator can also remotely perform specific skills of the robot by using the task execution module as an interactive operator, observation operator, work operator, or mobile operator.

[0061] As described above, an architecture has been implemented in which operators around the world can remotely operate robots around the world, wherein the remote operation is specific to the skills possessed by the operators.

[0062] For example, each operator can only remotely operate a portion of the task for each robot. For example, each operator can remotely operate multiple robots in parallel based on their skills. For example, multiple operators can collaboratively remotely operate a single robot. For example, each robot can receive operator support only for tasks that it cannot perform autonomously.

[0063] This arrangement allows robots around the world to effectively utilize the skills and time of operators worldwide, enabling them to perform actions efficiently. Furthermore, it improves the attentiveness of each robot; that is, each robot can reliably perform multiple tasks requiring attentive service.

[0064] <<1. Implementation Method>>

[0065] Next, we will refer to Figures 2 to 14 Describe the implementation methods of this technology.

[0066] <Configuration Example of Remote Operation Control System 1>

[0067] Figure 2 A configuration example of a remote operation control system 1 applying this technology is shown.

[0068] The remote operation control system 1 is a system for remotely controlling the operation of robots 11-1 to 11-m.

[0069] The remote operation control system 1 includes robots 11-1 to 11-m, operation terminals 12-1 to 12-n, and server 13. Robots 11-1 to 11-m, operation terminals 12-1 to 12-n, and server 13 are interconnected via network 21 and can communicate with each other.

[0070] In the following text, unless it is necessary to distinguish between robots 11-1 to 11-m, the robot will be referred to as robot 11. In the following text, unless it is necessary to distinguish between operating terminals 12-1 and 12-n, the operating terminal will be referred to as operating terminal 12.

[0071] Each robot 11 is referenced above. Figure 1 Corresponding to the aforementioned robot, robot 11 can autonomously take actions and perform tasks that are difficult to perform autonomously using an operating terminal 12 via remote control.

[0072] Note that there are no particular limitations on the form of robot 11. For example, robot 11 can be an autonomously moving robot, a manipulator that performs work by moving parts such as hands at fixed locations, etc. For example, robot 11 can be a humanoid or other living organism-shaped robot, or it can be a robot that performs a specific task. For example, robot 11 can be represented on a display including sensors or speakers using 3D computer graphics (CG).

[0073] The operating terminal 12 is a terminal (information processing device) used by each remote operator to perform remote operations on each robot 11. There are no particular limitations on the form of the operating terminal 12. For example, smartphones, personal computers (PCs), tablets, gaming terminals, dedicated operating terminals, etc., can be used as the operating terminal 12. For example, the operating terminal 12 can be not only a terminal operated by hand, but also an operating terminal operated with parts other than hands, or a terminal operated in a non-contact manner using speech recognition, gaze recognition, etc.

[0074] Server 13 controls the remote operation of each robot by each remote operator. For example, server 13 matches each robot 11 with a remote operator for non-autonomous tasks. For example, server 13 enables cooperation between the robot 11 and the operator's terminal 12 matched via network 21, and performs the necessary processing such as remote operation and data mediation between the robot 11 and the operator's terminal 12.

[0075] <Configuration Example for Robot 11>

[0076] Figure 3 An example of the functional configuration of robot 11 is shown.

[0077] The robot 11 includes an information processing unit 101, an input unit 102, a sensing unit 103, and a communication unit 104.

[0078] The information processing unit 101 includes an action module 111 and a task execution module group 112.

[0079] The motion module 111 is the module that controls the motion of the entire robot 11. The motion module 111 includes a motion planning unit 121, a motion control unit 122, and a learning unit 123.

[0080] The motion planning unit 121 creates a plan for the entire robot 11's motion based on a preset schedule, external instructions, and the state around the robot 11 identified by the state recognition unit 131 of the motion module 111. Furthermore, the motion planning unit 121 subdivides the robot 11's motion into micro-level tasks (micro-tasks).

[0081] Each task is divided into tasks related to interaction processing in robot 11 (hereinafter referred to as interaction tasks), tasks related to observation processing in robot 11 (hereinafter referred to as observation tasks), tasks related to work processing in robot 11 (hereinafter referred to as work tasks), and tasks related to movement processing in robot 11 (hereinafter referred to as movement tasks).

[0082] Interactive tasks are, for example, tasks related to the following processing: Robot 11 interacts with humans and performs information communication, mutual understanding, etc.

[0083] The observation task is, for example, a task related to the following processing: observing the environment around the robot 11 and identifying the surrounding state of objects (including living organisms such as humans) or the environment.

[0084] The work task is, for example, a task related to the following processing: performing operations, movements, creations, processing, etc. on various objects by using each part of the robot 11, as well as movements such as gestures of the robot 11.

[0085] A movement task is, for example, a task related to the following processing: moving robot 11 by using moving parts of robot 11 such as legs or wheels.

[0086] The motion control unit 122 controls the movements of the entire robot 11. The motion control unit 122 includes a status recognition unit 131, an autonomous motion control unit 132, and a remote motion control unit 133.

[0087] The state recognition unit 131 identifies the state around the robot 11 based on input data from the input unit 102, sensing data output from the sensing unit 103, and observation results from the observation module 141b. For example, the state recognition unit 131 identifies the inner state of people around the robot 11, such as emotions or intentions.

[0088] The autonomous motion control unit 132 controls the robot 11 to perform actions autonomously. For example, the autonomous motion control unit 132 controls the robot 11 to perform autonomous tasks by controlling the task execution module group 112.

[0089] The remote motion control unit 133 controls the execution of actions through remote operation of the robot 11. For example, the remote motion control unit 133 controls the task execution module group 112 to control the execution of non-autonomous tasks through remote operation of the robot 11. In addition, the remote motion control unit 133 controls the actions of the entire robot 11 through the remote operation of the aforementioned action operator based on remote operation content information indicating the content of the remote operation sent from the server 13.

[0090] The learning unit 123 learns methods for autonomously performing actions based on the execution content (e.g., operation content, execution result, etc.) of the actions of the remotely operated robot 11. Furthermore, the learning unit 123 learns methods for autonomously performing actions according to the user's intentions based on, for example, the user's intentions and reactions recognized by the state recognition unit 131.

[0091] Note that, for example, through the action module 111 (autonomous action control unit 132) which learns and grows based on the user's intentions, reactions, etc., the user can eventually become the action operator themselves. For example, through this arrangement, users who are unable to live independently, such as disabled persons or dementia patients, can become action operators, coexist with the robot 11, and live independently.

[0092] The interaction module 141a is a module that performs interactive tasks autonomously or through remote operation. The interaction module 141a includes an execution unit 151a and a learning unit 152a. The execution unit 151a includes an autonomous interaction unit 161a and a remote interaction unit 162a.

[0093] The autonomous interaction unit 161a autonomously executes interactive tasks.

[0094] The remote interaction unit 162a performs interactive tasks through remote operation based on remote operation content information sent from the server 13, which indicates the content of remote operation.

[0095] Learning Unit 152a teaches methods for autonomously performing interactive tasks based on the execution content (e.g., operation content, execution results, etc.) of interactive tasks performed by remote operations.

[0096] The observation module 141b is a module that performs observation tasks autonomously or remotely. The observation module 141b includes an execution unit 151b and a learning unit 152b. The execution unit 151b includes an autonomous observation unit 161b and a remote observation unit 162b.

[0097] The autonomous observation unit 161b autonomously performs observation tasks.

[0098] The remote observation unit 162b performs observation tasks remotely based on remote operation content information sent from the server 13.

[0099] Learning Unit 152b teaches methods for autonomously performing observation tasks based on the execution content (e.g., operation content, execution results, etc.) of remotely operated observation tasks.

[0100] Work module 141c is a module that performs work tasks autonomously or remotely. Work module 141c includes execution unit 151c and learning unit 152c. Execution unit 151c includes autonomous work unit 161c and remote work unit 162c.

[0101] Autonomous working unit 161c autonomously executes work tasks.

[0102] The remote work unit 162c performs work tasks remotely based on remote operation content information sent from the server 13.

[0103] Learning Unit 152c teaches methods for autonomously performing work tasks based on the execution content (e.g., operation content, execution results, etc.) of remotely operated work tasks.

[0104] The mobility module 141d is a module that performs mobility tasks autonomously or remotely. The mobility module 141d includes an execution unit 151d and a learning unit 152d. The execution unit 151d includes an autonomous mobility unit 161d and a remote mobility unit 162d.

[0105] The autonomous mobile unit 161d autonomously executes mobile tasks.

[0106] The remote mobile unit 162d performs a mobile task remotely based on remote operation content information sent from the server 13.

[0107] Learning Unit 152d teaches methods for autonomously executing mobile tasks based on the execution content (e.g., operation content, execution results, etc.) of remotely operated mobile tasks.

[0108] The input unit 102 includes input devices for inputting various types of input data, such as commands and data. The input unit 102 provides the input data to the information processing unit 101.

[0109] The sensing unit 103 includes various sensors, such as cameras, LiDAR, radar, and microphones, for sensing the surrounding environment of the robot 11, as well as various sensors for sensing the state of the robot 11. The sensing unit 103 provides sensor data output from each sensor to the information processing unit 101.

[0110] The communication unit 104 communicates with the operation terminal 12 and the server 13 via the network 21. The communication unit 104 provides the received data to the information processing unit 101 and obtains the data to be sent from the information processing unit 101.

[0111] In the following text, without needing to separately distinguish between the interaction module 141a and the movement module 141d, the interaction module will be referred to as the task execution module 141. In the following text, without needing to separately distinguish between the execution unit 151a of the interaction module 141a and the execution unit 151d of the movement module 141d, the execution module will be simply referred to as the execution unit 151. In the following text, without needing to separately distinguish between the learning unit 152a of the interaction module 141a and the learning unit 152d of the movement module 141d, the learning unit will be simply referred to as the learning unit 152. In the following text, without needing to separately distinguish between the autonomous interaction unit 161a, the autonomous observation unit 161b, the autonomous work unit 161c, and the autonomous movement unit 161d, the relevant unit will be referred to as the autonomous task execution unit 161. In the following text, without needing to separately distinguish between the remote interaction unit 162a, the remote observation unit 162b, the remote work unit 162c, and the remote movement unit 162d, the relevant unit will be referred to as the remote task execution unit 162.

[0112] Note that each task execution module 141 includes the necessary hardware and software and can execute tasks independently. Furthermore, the hardware and software included in each task execution module 141 can be shared with other task execution modules 141.

[0113] <Example of Robot 11's appearance configuration>

[0114] Figure 4 An example configuration of the appearance of robot 11 is shown. Figure 4 A is the front view of robot 11. Figure 4 B is the left-side view of robot 11.

[0115] Robot 11 is a humanoid mobile manipulator robot capable of performing various types of care, status observation, communication, and peripheral tasks with a quality highly acceptable to the care recipient.

[0116] Robot 11 includes a head portion 201, a chest portion 202, and a base portion 203 supporting the chest portion 202. The base portion 203 includes, for example, a bracket 208 located at the lower part of the base portion 203 that is movable in various directions. With this arrangement, robot 11 can move in various directions.

[0117] Robot 11 includes an arm portion 207L attached to the upper left portion of the chest portion 202, and an arm portion 207R attached to the upper right portion of the chest portion 202. Robot 11 includes a movable neck 204 disposed between the head portion 201 and the chest portion 202, and includes a cervical joint axis 204C. Robot 11 includes movable shoulders 205L and 205R. Movable shoulder 205L is disposed between the chest portion 202 and the arm portion 207L, and includes a shoulder joint axis 205LC. Movable shoulder 205R is disposed between the chest portion 202 and the arm portion 207R, and includes a shoulder joint axis 205RC. Furthermore, robot 11 includes a movable waist 206 disposed below the chest portion 202, and includes a waist joint axis 206C.

[0118] The head portion 201 includes an eye portion 221L and an eye portion 221R. For example, the robot 11 controls the position of the pupil portion of the eye portion 221L, the position of the pupil portion of the eye portion 221R, and the axis (roll, pitch, and yaw) of the neck 204 to gaze at the target person.

[0119] Note that in the following text, unless it is necessary to distinguish between the ocular portion 221L and the ocular portion 221R separately, the ocular portion will be referred to simply as the ocular portion 221.

[0120] The arm portion 207L includes the elbow portion 231L, the wrist portion 232L, and the hand portion 233L. The elbow portion 231L has a pitch axis. The wrist portion 232L has a yaw axis.

[0121] The arm section 207R is configured similarly to the arm section 207L, and includes the elbow section 231R, wrist 232R, and hand 233R.

[0122] In the following text, unless it is necessary to separately distinguish between arm portion 207L and arm portion 207R, the arm portion will be referred to simply as arm portion 207. In the following text, unless it is necessary to separately distinguish between elbow portion 231L and elbow portion 231R, the elbow portion will be referred to simply as elbow portion 231. In the following text, unless it is necessary to separately distinguish between wrist portion 232L and wrist portion 232R, the wrist will be referred to simply as wrist portion 232. In the following text, unless it is necessary to separately distinguish between hand portion 233L and hand portion 233R, the hand will be referred to simply as hand portion 233.

[0123] A head sensor 241 is disposed on the upper front portion of the head part 201. The head sensor 241 includes, for example, a distance image sensor, a microphone, and a LiDAR sensor. The sensing direction of the head sensor 241 is configured to be substantially consistent with the gaze direction of the robot 11. For example, the robot 11 can perform human recognition and face recognition through the head sensor 241.

[0124] A chest sensor 242 is disposed at the upper front part of the chest portion 202. The chest sensor 242 includes, for example, a non-contact vital signs sensor. Examples of non-contact vital signs sensors include, for example, a body temperature sensor, a heart rate sensor, and a respiration sensor.

[0125] A hand sensor 243L is disposed in the hand 233L. The hand sensor 243L includes, for example, a contact-type vital signs sensor. Examples of contact-type vital signs sensors include, for example, a heart rate sensor, a blood pressure sensor, and a blood oxygen saturation measurement sensor.

[0126] Furthermore, vital sign sensing can be performed by the target person placing their hand on or grasping the hand sensor 243L, rather than by the robot 11 touching the target person with the hand sensor 243L. This is an interface familiar to dementia patients and is more acceptable to the target person.

[0127] Similar to hand sensor 243L, hand sensor 243R is also located in hand 233R.

[0128] Note that in the following text, unless it is necessary to distinguish between hand sensor 243L and hand sensor 243R, the hand sensor will be referred to simply as hand sensor 243.

[0129] <Configuration Example for Server 13>

[0130] Figure 5 An example of the functional configuration of server 13 is shown.

[0131] Server 13 includes a communication unit 301, an information processing unit 302, and a storage unit 303. The information processing unit 302 includes a remote control unit 311 and a matching unit 312.

[0132] The communication unit 301 communicates with each robot 11 and each operating terminal 12 via the network 21. The communication unit 301 provides the received data to the information processing unit 302 and obtains the data to be sent from the information processing unit 302.

[0133] The remote control unit 311 controls each remote operator to remotely operate the robot 11 using the operation terminal 12.

[0134] For example, the remote control unit 311 receives task request information from the robot 11 via network 21 and communication unit 301. The task request information includes information related to the task requiring remote operation. The remote control unit 311 then sends the task request information to the operation terminal 12 of the remote operator selected by the matching unit 312 via communication unit 301 and network 21.

[0135] For example, the remote control unit 311 receives remote operation provision information from the robot 11 via network 21 and communication unit 301. This remote operation provision information includes information required for remote operation. For instance, the remote control unit 311 generates a remote operation user interface (UI) (hereinafter referred to as the remote operation UI) based on the remote operation provision information, providing a user interface for remote operation. The remote control unit 311 then sends the remote operation UI information to the operation terminal 12 of the remote operator responsible for remote operation via communication unit 301 and network 21.

[0136] For example, the remote control unit 311 receives remote operation content information from the operating terminal 12 via the network 21 and the communication unit 301, and converts the remote operation content information into information suitable for the robot 11 as the remote operation target as needed. The remote control unit 311 then sends the remote operation content information to the robot 11 as the remote operation target via the communication unit 301 and the network 21.

[0137] For example, the remote control unit 311 receives a task execution result notification signal from the robot 11 via the network 21 and the communication unit 301, which is used to notify the execution result of the task of remote operation, and sends the task execution result notification signal to the operation terminal 12 of the remote operator responsible for the remote operation. The task execution result notification signal is divided into, for example, a task completion notification signal for notifying the completion of the task and a task failure notification signal for notifying the failure of the task.

[0138] Matching unit 312 performs matching processing between multiple non-autonomous tasks of the robots 11 and multiple remote operators. For example, matching unit 312 selects a remote operator requested to perform a remote operation on a task based on task request information received from the robots 11 and a remote operator database and a task database stored in storage unit 303. Matching unit 312 updates the remote operator database and task database stored in storage unit 303 as necessary based on remote operation result notification signals.

[0139] Storage unit 303 stores various types of data required for processing in server 13. For example, storage unit 303 stores a remote operator database and a task database.

[0140] The remote operator database includes data related to each remote operator's attributes, characteristics, schedules, environment, and remote operation history.

[0141] The attributes of a remote operator include the operator's name, age, gender, nationality, occupation, etc.

[0142] Characteristics of a remote operator include, for example, the operator's skills, skill level, and the language used.

[0143] The remote operator's schedule includes, for example, information about the schedule in which the remote operator can perform remote operations on robot 11.

[0144] The remote operator's environment includes, for example, information about the environment in which the remote operator performs the remote operation. For instance, the remote operator's environment includes the form and performance of the operating terminal 12 used by the remote operator, its location coordinates on Earth, time, etc.

[0145] Remote operation history includes, for example, the date and time of past remote operations performed on the robot, the content of the task, and the execution results.

[0146] The task database includes data such as the content, difficulty level, and environment required for remote operation of each task.

[0147] <Processing of Remote Operation Control System 1>

[0148] Next, we will refer to Figures 6 to 8 Describe the processing in the remote operation control system 1.

[0149] Processing in Robots 11

[0150] First, refer to Figure 6 The flowchart in the diagram describes the processing in robot 11.

[0151] For example, the process is initiated when a trigger is generated to start a certain action for robot 11. For example, robot 11 initiates the process when a scheduled action in a preset timetable is about to begin, or when a command to perform an action is given from the outside.

[0152] In step S1, the action planning unit 121 creates an action plan. For example, the action planning unit 121 creates a plan for a series of actions required to achieve a given objective.

[0153] Note that there are no particular restrictions on the method of creating action plans. For example, action planning unit 121 can create action plans autonomously, based on information provided from external sources, or based on interactions with the user.

[0154] Furthermore, for example, the motion planning unit 121 can apply an externally provided motion plan, or, if multiple motion plans are provided, the motion planning unit 121 can select a motion plan from among the multiple motion plans. An externally provided motion plan could be, for example, a motion plan pre-designed by the operator of the remote operation control system 1, a motion plan created by the user of the robot 11, or a motion plan created by the motion operator.

[0155] Action planning unit 121 breaks down each action into multiple tasks. At this point, each action is broken down into as many tasks as possible until each task becomes a micro-level independent of the context in which the action is performed.

[0156] In step S2, robot 11 performs task execution control processing.

[0157] Here, we will refer to Figure 7 The flowchart in the document describes the details of the task execution control process.

[0158] In step S51, the autonomous motion control unit 132 determines whether it can autonomously execute the task to be performed next. If it is determined that it can autonomously execute the task to be performed next, the process proceeds to step S52.

[0159] In step S52, robot 11 autonomously performs the task.

[0160] For example, if the next task to be performed is an interactive task, the autonomous motion control unit 132 instructs the interaction module 141a to autonomously execute the task. In response, the autonomous interaction unit 161a of the interaction module 141a autonomously executes the instructed task.

[0161] For example, if the next task to be performed is an observation task, the autonomous action control unit 132 instructs the observation module 141b to autonomously perform the task. In response, the autonomous observation unit 161b of the observation module 141b autonomously performs the instructed task.

[0162] For example, if the next task to be performed is a work task, the autonomous motion control unit 132 instructs the work module 141c to autonomously execute the task. In response, the autonomous work unit 161c of the work module 141c autonomously executes the instructed task.

[0163] For example, if the next task to be performed is a movement task, the autonomous motion control unit 132 instructs the movement module 141d to autonomously perform the task. In response, the autonomous movement unit 161d of the movement module 141d autonomously performs the instructed task.

[0164] In step S53, the autonomous motion control unit 132 determines whether the task has been completed based on information from the task execution module 141, which has been instructed to perform the task. If it is determined that the task has not been completed, the process proceeds to step S54.

[0165] In step S54, the autonomous motion control unit 132 determines whether the task has failed based on information from the task execution module 141, which has been instructed to perform the task. If it is determined that the task has not failed, the process returns to step S52.

[0166] Then, the process of steps S52 to S54 is repeated until it is determined in step S53 that the task has been completed or in step S54 that the task has failed.

[0167] On the other hand, if it is determined in step S54 that the task has failed, the process proceeds to step S55.

[0168] Here, an example of determining that a task has failed will be described.

[0169] For example, if a target time is set for a task, and the task is not completed within the target time, the task is considered to have failed.

[0170] Note that with new tasks, it's difficult to set a target time, so the target time will be set longer. On the other hand, with similar tasks performed in the past, the target time is set based on the actual time taken to perform similar tasks in the past.

[0171] For example, when performing tasks such as identifying target individuals and their surroundings, the task is considered to have failed when the deterministic factor (identification accuracy) is low.

[0172] Note that the permissible range (threshold) of the deterministic factor is set, for example, based on the degree of false recognition that is allowed. For instance, in tasks such as face recognition where false recognition is not permissible, the threshold is set to a higher value (e.g., 99% or higher). On the other hand, in tasks such as dialogue content recognition where false recognition is somewhat permissible, the threshold is set to a lower value (e.g., 70% or higher).

[0173] Note that the threshold of the deterministic factor is learned and optimized each time a similar task is performed.

[0174] For example, if the robot 11 accidentally collides or comes into contact with the surrounding environment while performing a movement task, it can be determined that the task has failed. For example, the accidental collision or contact with the surrounding environment can be detected by means of sensor values ​​such as tactile sensors or force sensors, actuator current values, sound recognition of contact sounds, and switch input values.

[0175] For example, in the performance of a work task, if a work error occurs, it is determined that the task has failed. For example, if an anomaly occurs in the target object that is the objective of the task, it is determined that the task has failed. An anomaly in the target object could be, for example, the target object falling, collapsing, or colliding. For example, anomalies in the target object can be detected through image recognition, sound recognition based on contact sounds, or detection values ​​from sensors such as tactile sensors or force sensors.

[0176] On the other hand, if it is determined in step S51 that the task to be performed next cannot be performed autonomously, the processing of steps S52 to S54 is skipped, and the processing of step S55 is carried out.

[0177] In step S55, the remote motion control unit 133 requests remote operation of a task. Specifically, the remote motion control unit 133 generates task request information including information related to the task for which remote operation is requested, and sends the task request information to the server 13 via the communication unit 104 and the network 21.

[0178] In response, server 13 matches the requested task with a remote operator, as described below.

[0179] In step S56, robot 11 performs the task through remote operation.

[0180] For example, when the task to be remotely operated is an interactive task, the remote motion control unit 113 instructs the interaction module 141a to perform the task remotely. In response, the remote interaction unit 162a of the interaction module 141a performs the instructed task according to the remote operation.

[0181] For example, if the task to be remotely operated is an observation task, the remote action control unit 113 instructs the observation module 141b to perform the task remotely. In response, the remote observation unit 162b of the observation module 141b performs the instructed task according to the remote operation.

[0182] For example, when the task to be remotely operated is a work task, the remote motion control unit 113 instructs the work module 141c to perform the task remotely. In response, the remote work unit 162c of the work module 141c performs the instructed task according to the remote operation.

[0183] For example, if the task to be remotely operated is a movement task, the remote motion control unit 113 instructs the movement module 141d to perform the task remotely. In response, the remote movement unit 162d of the movement module 141d performs the instructed task according to the remote operation.

[0184] At this time, the remote task execution unit 162 (hereinafter referred to as the target remote task execution unit 162) that performs the task remotely sends the information required for the remote operation (remote operation provision information) appropriately to the server 13 via the communication unit 104 and the network 21. Furthermore, the target remote task execution unit 162 receives remote operation content information indicating the content of the remote operation from the server 13 via the network 21 and the communication unit 104. The target remote task execution unit 162 performs the task based on the remote operation content information.

[0185] In step S57, similar to the process in step S53, it is determined whether the task has been completed. If it is determined that the task has not been completed, the process proceeds to step S58.

[0186] In step S58, similar to the process in step S54, it is determined whether the task has failed. If it is determined that the task has not failed, the process returns to step S56.

[0187] Subsequently, the processing steps S56 to S58 are repeated until it is determined in step S57 that the task has been completed or in step S58 that the task has failed.

[0188] On the other hand, if it is determined in step S58 that the task has failed, the process proceeds to step S59.

[0189] In step S59, robot 11 notifies server 13 of the task failure. Specifically, remote motion control unit 133 generates a task failure notification signal to notify of the task failure and sends the task failure notification signal to server 13 via communication unit 104 and network 21.

[0190] In step S60, the remote motion control unit 133 determines whether the task should be executed again. If it is determined that the task should be executed again, the process returns to step S55.

[0191] Subsequently, steps S55 to S60 are repeated until it is determined in step S57 that the task has been completed or in step S60 that the task will no longer be executed. In other words, the execution process for failed tasks is repeatedly performed via remote operation.

[0192] On the other hand, if it is determined in step S60 that the task will no longer be executed, the task execution control process ends. This means, for example, assuming the following situations: the task fails a predetermined number of times or more, or the task is not completed within the predetermined time.

[0193] On the other hand, if it is determined in step S57 that the task has been completed, the process proceeds to step S61.

[0194] In step S61, robot 11 notifies server 13 of task completion. Specifically, remote motion control unit 133 generates a task success notification signal to notify of task completion and sends the task success notification signal to server 13 via communication unit 104 and network 21.

[0195] In step S62, robot 11 performs learning processing based on the content of the remote operation. For example, the learning unit 152 of task execution module 141, which performed the task through remote operation, learns a method for autonomously performing tasks based on the content of the remote operation.

[0196] With this arrangement, it is hoped that non-autonomous tasks will gradually be transformed into autonomous tasks in the future, and will be executed autonomously by Robot 11.

[0197] After this, the task execution control process ends.

[0198] Return to Figure 6 In step S3, the motion planning unit 121 determines whether it is necessary to change the motion plan. If it is determined that no change to the motion plan is needed, the process proceeds to step S4.

[0199] In step S4, the action planning unit 121 determines whether the action being executed has been completed based on the result of the processing in step S2. If it is determined that the action being executed has not been completed, the processing returns to step S2.

[0200] Subsequently, the process of steps S2 to S4 is repeated until it is determined in step S2 that it is necessary to change the action plan, or in step S4 that the action being executed has been completed.

[0201] On the other hand, if it is determined in step S4 that the action being executed has been completed, the process proceeds to step S6.

[0202] On the other hand, if it is determined in step S3 that it is necessary to change the action plan, the process proceeds to step S5. This means, for example, assuming the following situation: the action cannot be performed as planned due to task failure, unexpected events, interference, or other reasons.

[0203] In step S5, the action planning unit 121 modifies the action plan. For example, the action planning unit 121 reviews the action plan and modifies the plan for a series of actions required to achieve a given objective. Furthermore, the action planning unit 121 breaks down each modified action into multiple tasks.

[0204] Note that at this point, if it is determined that the given objective is difficult to achieve, the action planning unit 121 can determine to stop the action.

[0205] The process then proceeds to step S6.

[0206] In step S6, the action planning unit 121 determines whether there is an action to be performed next. If it is determined that there is an action to be performed next, the process returns to step S2.

[0207] Subsequently, in step S6, the processes of steps S2 to S6 are repeated until it is determined that there are no further operations to be performed. Under this arrangement, the actions are executed according to the action plan.

[0208] On the other hand, if it is determined in step S6 that there is no next action to be performed, the processing in robot 11 ends.

[0209] <Handling Server 13>

[0210] Next, we will refer to Figure 8 The flowchart in the document describes and Figure 6 The processing of robot 11 corresponds to the processing of server 13.

[0211] In step S101, the remote control unit 311 determines whether a remote operation for the task has been requested. The determination process in step S101 is repeated until it is determined that a remote operation for the task has been requested. Then, in Figure 7 In step S55, when the remote control unit 311 receives the task request information sent from the robot 11 via the network 21 and the communication unit 301, it determines that the remote operation of the requested task has been performed, and the process proceeds to step S102.

[0212] In step S102, the matching unit 312 performs matching for remote operators. That is, the matching unit 312 determines the remote operator responsible for the requested task by performing a matching process.

[0213] The method of matching is not specifically limited here.

[0214] For example, the matching unit 312 selects a remote operator who can be responsible for remote operation at the current time from the remote operators with the skills to perform the requested task, based on the remote operator database stored in the storage unit 303.

[0215] Alternatively, for example, matching unit 312 may solicit remote operators with skills capable of performing the requested task and select one operator from among the remote operators who have submitted applications.

[0216] Note that, for example, in cases where multiple remote operators are required, the matching unit 312 selects the required number of remote operators.

[0217] The matching unit 312 sends the task request information to the operating terminal 12 of the selected remote operator via the communication unit 301 and the network 21.

[0218] In step S103, server 13 provides a remote operation user interface. For example, remote control unit 311 receives remote operation provision information from robot 11 via network 21 and communication unit 301. Remote control unit 311 generates remote operation UI information for providing the remote operation UI based on the remote operation provision information. Remote control unit 311 sends the remote operation provision information to the operator's terminal 12 via communication unit 301 and network 21.

[0219] In response, the operating terminal 12 presents the remote operating UI to the remote operator based on the remote operating UI information.

[0220] Ideally, the remote operation UI should be one that can be actively used by operators with different skills around the world and enable them to perform remote operations simply and efficiently.

[0221] For example, the remote operation UI may include video obtained by imaging the environment in which the robot 11 actually performs the task.

[0222] Alternatively, for example, the remote operation UI may include video in a virtual space that simulates the environment in which the robot 11 performs a task by using computer graphics (CG) or similar methods.

[0223] In this context, for example, a remote operation UI can be provided in a virtual space (metaverse space) where the remote operator exists as an avatar or role and is active.

[0224] This arrangement conceals the actual environment surrounding robot 11 and protects the privacy of robot 11's users. Furthermore, it allows for the alteration of real-world information in a manner that facilitates remote operation.

[0225] Alternatively, for example, the remote operation UI may include dedicated videos for performing tasks, such as game videos for performing tasks, which are independent of the environment in which robot 11 performs the tasks.

[0226] In this context, for example, a remote operation UI could be provided within an abstract game space or the aforementioned virtual space. Furthermore, for instance, within a game, additional benefits such as level increases, reward / item acquisition, point accumulation, and storyline progression could be provided to the remote operator (actually a game player) based on skill level, success rate, number of operations processed, and processing speed. Moreover, earned points could be exchanged for money in the virtual or real world.

[0227] This arrangement conceals the actual environment surrounding robot 11 and protects the privacy of its users. Furthermore, for example, it enhances the game's operability and enjoyment, and is expected to improve the motivation and efficiency of remote operators. Additionally, for example, gamers can perform remote operation of robot 11 as a way to earn income, thus providing new employment opportunities.

[0228] In step S104, server 13 forwards remote operation content information.

[0229] For example, a remote operator performs a remote operation on a requested task based on a remote operation UI presented by the operating terminal 12. The operating terminal 12 generates remote operation content information indicating the content of the remote operation and sends the remote operation content information to the server 13 via the network 21.

[0230] In response, the remote control unit 311 of server 13 receives remote operation content information via network 21 and communication unit 301.

[0231] The remote control unit 311 converts remote operation content information into information suitable for the robot 11 as needed. For example, based on the content of the remote operation UI, the operation content performed by the remote operator needs to be converted into information (e.g., operation signals) for the actual remote operation of the robot 11. In response, the remote control unit 311 converts the remote operation content information into information suitable for the robot 11.

[0232] The remote control unit 311 sends remote operation content information to the robot 11, which is to be the target of remote operation, via the communication unit 301 and the network 21.

[0233] In step S105, the remote control unit 311 determines whether the task has been completed. If it is determined that the task has not been completed, the process proceeds to step S106.

[0234] In step S106, the remote control unit 311 determines whether the task has failed. If it is determined that the task has not failed, the process returns to step S103.

[0235] Subsequently, the processing steps S103 to S106 are repeated until it is determined in step S105 that the task has been completed or in step S106 that the task has failed.

[0236] On the other hand, in step S106, the remote control unit 311 receives... Figure 7 In step S59, if a task failure notification signal is sent from robot 11 via network 21 and communication unit 301, it is determined that the task has failed, and the process proceeds to step S107.

[0237] In step S107, the remote control unit 311 notifies the remote operator of the task failure. Specifically, the remote control unit 311 sends a task failure notification signal to the remote operator's operating terminal 12 via the communication unit 301 and the network 21.

[0238] Note that at this time, the remote control unit 311 updates the information (e.g., skill level, remote operation history, etc.) of the remote operator database stored in the storage unit 303 as needed in relation to remote operators who have failed in remote operations. Furthermore, the remote control unit 311 updates the information (e.g., content, difficulty level, etc.) of the tasks (e.g., content, difficulty level, etc.) of the task database stored in the storage unit 303 as needed in relation to tasks that have failed in remote operations.

[0239] After that, the process returns to step S101, and the processing of step S101 and subsequent steps is performed.

[0240] On the other hand, in step S106, the remote control unit 311 receives the data via network 21 and communication unit 301. Figure 7 In step S59, if a task failure notification signal is sent from robot 11, it is determined that the task has failed, and the process proceeds to step S107.

[0241] On the other hand, in step S105, the remote control unit 311 receives the data via network 21 and communication unit 301. Figure 7 In step S61, if a task completion notification signal is sent from robot 11, it is determined that the task has been completed, and the process proceeds to step S108.

[0242] In step S108, the remote control unit 311 notifies the remote operator of the task completion. Specifically, the remote control unit 311 sends a task completion notification signal to the remote operator's operating terminal 12 via the communication unit 301 and the network 21.

[0243] Note that at this time, the remote control unit 311 updates the information (e.g., skill level, remote operation history, etc.) of the remote operator database stored in the storage unit 303 as needed regarding the remote operators who have successfully completed the remote operation. Furthermore, the remote control unit 311 updates the information (e.g., content, difficulty level, etc.) of the tasks (e.g., content, difficulty level, etc.) of the task database stored in the storage unit 303 as needed.

[0244] After that, the process returns to step S101, and the processing of step S101 and subsequent steps is performed.

[0245] <Specific examples of task execution and processing for Robot 11>

[0246] Next, a specific example of task execution processing in robot 11 will be described.

[0247] <Example of the action of pouring tea into a cup and carrying the cup>

[0248] First, an example will be described of robot 11 performing the action of pouring tea into a cup and carrying the cup.

[0249] For example, the status recognition unit 131 detects a user speaking near the robot 11.

[0250] The autonomous motion control unit 132 instructs the interaction module 141a to perform tasks that interact with the user.

[0251] The autonomous interaction unit 161a performs interactions with the user and recognizes the words "I want to drink something" in the user's speech.

[0252] However, since the deterministic factor of the speech recognition of the autonomous interaction unit 161a is low, the remote action control unit 133 requests the server 13 to perform remote operation of the task.

[0253] Server 13 requests remote operation to perform the task by the selected mobile operator through the execution of matching processing.

[0254] The remote interaction unit 162a performs interaction with the user through remote operation by the interaction operator. With this arrangement, the remote interaction unit 162a identifies the user "want to drink tea" with a high degree of certainty.

[0255] In response, action planning unit 121 planned the action of "delivering a cup containing tea to the user".

[0256] Next, the autonomous motion control unit 132 instructs the motion module 141d to perform the task of moving to the front of the shelf where the cups are placed.

[0257] The autonomous mobile unit 161d moves the robot 11 to a position 1m in front of the frame.

[0258] Next, the autonomous motion control unit 132 instructs the observation module 141b to perform the task of finding a cup from the shelf.

[0259] The shelf has no doors, and the tableware inside can be seen. The autonomous observation unit 161b searches for a cup on the shelf. However, the shelf contains more items than usual, and the certainty factor for the autonomous observation unit 161b identifying the cup is low.

[0260] In response, the remote motion control unit 133 requests the server 13 to perform remote operation of the task.

[0261] Server 13 requests remote operation of the task performed by the observer selected through matching processing.

[0262] The remote observation unit 162b locates the cup by observing the operator's remote operation.

[0263] Next, the autonomous motion control unit 132 instructs the motion module 141d to perform the task of moving the robot 11 to a position 50cm in front of the cup.

[0264] The autonomous mobile unit 161d moves the robot 11 to a position 50cm in front of the cup.

[0265] Next, the autonomous motion control unit 132 instructs the working module 141c to perform the task of grasping the cup.

[0266] The autonomous work unit 161c attempts to grasp the cup but finds it difficult because of the many other objects surrounding it. Therefore, by the time the autonomous work unit 161c is searching for the optimal way to grasp the cup, the allotted time has already exceeded the specified working time.

[0267] In response, the remote motion control unit 133 requests the server 13 to perform remote operation of the task.

[0268] Server 13 requests remote operation to perform a task by the selected worker operator through the execution of matching processing.

[0269] The remote work unit 162c enables the robot 11 to grasp the cup through remote operation by a remote operator.

[0270] Next, the autonomous motion control unit 132 instructs the motion module 141d to perform the task of moving to the front of the tea machine.

[0271] The autonomous mobile unit 161d moves the robot 11 to the front of the tea machine.

[0272] Next, the autonomous motion control unit 132 instructs the working module 141c to perform the task of setting the teacup into the tea machine.

[0273] The autonomous working unit 161c sets the teacup into the tea maker.

[0274] Next, the autonomous motion control unit 132 instructs the working module 141c to perform the task of operating the tea dispenser's tea-dispensing switch.

[0275] The autonomous working unit 161c operates the tea dispensing switch of the tea machine. In this arrangement, the tea machine pours tea into cups.

[0276] The autonomous motion control unit 132 instructs the working module 141c to perform the task of grasping the cup containing tea.

[0277] Autonomous working unit 161c begins to grasp the cup.

[0278] However, while the autonomous working unit 161c was grasping the cup, the autonomous interaction unit 161a detected that a staff member, acting as a third party in the kitchen, was speaking to the robot 11. The autonomous interaction unit 161a, through speech recognition, identified the staff member's words as "Could you move aside?".

[0279] In response, the motion planning unit 121 determines that moving to a location that does not obstruct the worker takes priority over grasping the cup. The autonomous motion control unit 132 instructs the work module 141c to stop the task of grasping the cup.

[0280] The autonomous working unit 161c stops grasping the cup and restores the arm of robot 11, which was about to grasp the cup, to its original standard posture.

[0281] The autonomous motion control unit 132 instructs the movement module 141d to perform the following task: move in the opposite direction to the location of the staff member to a position 1m in front of the tea machine.

[0282] The autonomous mobile unit 161d attempted to move from the front of the tea machine. However, due to the narrow passage and the presence of numerous obstacles, including people, around the robot 11, the autonomous mobile unit 161d's planned movement route failed, and the robot 11 was unable to move.

[0283] In response, the remote motion control unit 133 requests the server 13 to perform remote operation of the task.

[0284] Server 13 requests remote operation to perform the task by the selected mobile operator through the execution of matching processing.

[0285] The remote moving unit 162d moves the robot 11 away from the front of the tea machine through remote operation by the mobile operator.

[0286] Next, the autonomous motion control unit 132 instructs the observation module 141b to perform the following task: confirm the status of obstacles (including people) in front of the tea machine in order to determine whether the robot can return to the original task (grabbing the cup).

[0287] The autonomous observation unit 161b confirmed that there were no obstacles, including people, in front of the tea machine. However, due to the presence of many objects around the tea machine, the deterministic factor of the autonomous observation unit 161b was low.

[0288] In response, the remote motion control unit 133 requests the server 13 to perform remote operation of the task.

[0289] Server 13 requests remote operation of the task performed by the observer selected through matching processing.

[0290] The remote observation unit 162b confirms that there are no obstacles in front of the tea machine by observing the operator's remote operation.

[0291] Next, the autonomous motion control unit 132 instructs the motion module 141d to perform the task of moving to the front of the tea machine.

[0292] The autonomous mobile unit 161d moves the robot 11 to the front of the tea machine.

[0293] Next, the autonomous motion control unit 132 instructs the working module 141c to perform the task of grasping the cup set in the tea machine.

[0294] The autonomous working unit 161c enables the robot 11 to grasp the cup.

[0295] Next, the autonomous motion control unit 132 instructs the motion module 141d to perform the task of moving to the front of the user's table.

[0296] The autonomous mobile unit 161d moves the robot 11 to the front of the table where the user is located.

[0297] Next, the autonomous motion control unit 132 instructs the observation module 141b to perform the task of finding the user and confirming the status.

[0298] The autonomous observation unit 161b detects the position and orientation of the user's face. However, because the user is slightly facing downwards and there are many shadows on the user's face, the deterministic factor of the autonomous observation unit 161b is low.

[0299] In response, the autonomous motion control unit 132 requests the server 13 to perform remote operation of the task.

[0300] Server 13 requests remote operation of the task performed by the observer selected through matching processing.

[0301] The remote observation unit 162b locates the user by observing the operator's remote operations and confirms that the user is not acting abnormally.

[0302] Next, the autonomous motion control unit 132 instructs the interaction module 141a to perform tasks that initiate a dialogue with the user, such as “I have prepared tea for you” or “I have put the cup here”.

[0303] Autonomous interaction unit 161a begins a dialogue with the user. In response, although there is a response from the user, the autonomous interaction unit 161a may have difficulty recognizing the user's response because the user is slightly facing downwards, the pronunciation is unclear, and there is a lot of ambient noise.

[0304] In response, the remote motion control unit 133 requests the server 13 to perform remote operation of the task.

[0305] Server 13 requests a remote operation to perform a task by the interactive operator selected through the matching process.

[0306] The remote interaction unit 162a initiates a dialogue with the user through remote operation by the interaction operator. Then, the remote interaction unit 162a (remote operator) confirms with a high degree of certainty that the user is affirmative.

[0307] Next, the autonomous motion control unit 132 instructs the working module 141c to perform the task of placing the cup on the table.

[0308] The autonomous working unit 161c places the cup on the table.

[0309] Next, the autonomous motion control unit 132 instructs the interaction module 141a to perform tasks that interact with the user.

[0310] The autonomous interaction unit 161a performs interactions with the user.

[0311] Then, the action planning unit 121 confirms that the action of "delivering the cup containing tea to the user" has been completed.

[0312] By combining the aforementioned autonomous and remote operations, robot 11 can pour tea into a cup and deliver the cup to the user while interacting with the user.

[0313] Example of performing actions to measure vital signs of a nursing target.

[0314] Next, we will refer to Figures 9 to 12 describe Figure 4 An example of robot 11 performing the action of measuring the vital signs of the nursing target 403.

[0315] Note that in the following text, robot 11 will be described in... Figure 9 The measurement of target person 403 was performed in the living room shown. Figure 10 Examples of applications of vital sign measurement, such as vital sign values.

[0316] exist Figure 9In this arrangement, the robot 11 is located at the main position P0, and a charging base (not shown) is provided at this position, for example. In addition, tables 401-1 to 401-8 and chairs 402-1 to 402-7 are provided in the living room.

[0317] Tables 401-1 to 401-8 are arranged in a U-shape. That is, a row of tables 401-1 and 401-2, a row of tables 401-3 to 401-6, and a row of tables 401-7 and 401-8 are arranged in a U-shape. Tables 401-3 to 401-6 are positioned in front of robot 11. The rows of tables 401-1 and 401-2, and the rows of tables 401-7 and 401-8, are arranged perpendicular to the direction of the rows of tables 401-3 to 401-6 and face each other.

[0318] Furthermore, chairs 402-1 to 402-7 are each configured to correspond to one of tables 401-1 to 401-7. Note that no chair is provided for table 401-8.

[0319] Note that in the following text, unless it is necessary to distinguish between tables 401-1 to 401-7, the table will be referred to as table 401. Similarly, unless it is necessary to distinguish between chairs 402-1 to 402-7, the chair will be referred to as chair 402.

[0320] In addition, Figure 9 The diagram shows the long-distance position P3, the medium-distance position P2, and the short-distance position P1 relative to each chair 402.

[0321] For example, the long-distance position P3 is set to a position about 2m away from the target person 403, with the direction being about 0 degrees relative to the front of the target person 403, where the target person 403 is the target of the application performed by the robot 11.

[0322] For example, the medium distance position P2 is set at a position about 1.2m away from the target person 403, with the direction being about 45 degrees relative to the front of the target person 403.

[0323] For example, the short-distance position P1 is set at a position about 0.6m away from the target person 403, with the direction being about 70 degrees relative to the front of the target person 403.

[0324] For example, while approaching the target person 403, the action planning unit 121 moves according to the long-distance position P3 ( Figure 10 ), medium distance position P2 ( Figure 11 ) and short distance position P1 ( Figure 12The sequence is as follows: the plan is to start a dialogue with the target 403 at each location, and finally measure the target 403's vital signs at a short distance location P1.

[0325] Next, the autonomous motion control unit 132 instructs the motion module 141d to perform the task of moving to the long-distance position P3.

[0326] The autonomous mobile unit 161d moves robot 11 to a long-distance position P3.

[0327] Here, for example, if robot 11 is unable to move to a long-distance position P3 within the target time, remote motion control unit 133 requests server 13 to perform a remote operation task.

[0328] Server 13 requests remote operation to perform the task by the selected mobile operator through the execution of matching processing.

[0329] The remote mobility unit 162d moves the robot 11 to a long-distance position P3 via remote operation by a mobile operator. Here, the task success rate is improved because the mobile operator is superior to the robot 11 in situation recognition and determination. This also applies to the following mobility tasks.

[0330] Next, the autonomous motion control unit 132 requests the interaction module 141a to perform the task of interacting with the target person 403.

[0331] The autonomous interaction unit 161a performs the task of interacting with the target person 403.

[0332] Here, for example, if the interaction with the target person 403 fails, the remote motion control unit 133 requests the server 13 to perform a remote operation of the task.

[0333] Server 13 requests a remote operation to perform a task by the interactive operator selected through the matching process.

[0334] The remote interaction unit 162a performs interaction with the target person 403 through remote operation by an interactive operator. Here, since the interactive operator is superior to the robot 11 in terms of situation recognition and determination, the task success rate is improved. This also applies to the following interactive tasks.

[0335] Next, the autonomous motion control unit 132 instructs the motion module 141d to perform the task of moving to the medium distance position P2.

[0336] The autonomous mobile unit 161d moves robot 11 to a medium distance position P2.

[0337] Here, for example, if robot 11 is unable to move to a medium distance position P2 within the target time, remote motion control unit 133 requests server 13 to perform a remote operation task.

[0338] Server 13 requests a remote operation to be performed by the mobile operator selected through the matching process. In this case, server 13 may preferably request the previously requested mobile operator to perform the remote operation.

[0339] The remote moving unit 162d moves the robot 11 to a medium-distance position P2 through remote operation by the moving operator.

[0340] Next, the autonomous motion control unit 132 requests the interaction module 141a to perform the task of interacting with the target person 403.

[0341] Autonomous interaction unit 161a performs interaction with target person 403.

[0342] Here, for example, if the interaction with the target person 403 fails, the remote motion control unit 133 requests the server 13 to perform a remote operation of the task.

[0343] Server 13 requests a remote operation to be performed by the interactive operator selected through the matching process. In this case, server 13 may preferably request the previously requested interactive operator to perform the remote operation.

[0344] The remote interaction unit 162a performs interaction with the target person 403 through remote operation by the interactive operator.

[0345] Next, the autonomous motion control unit 132 instructs the motion module 141d to perform the task of moving to the short-distance position P1.

[0346] The autonomous mobile unit 161d moves the robot 11 to a short distance position P1.

[0347] Here, for example, if robot 11 is unable to move to a short distance position P1 within the target time, remote motion control unit 133 requests server 13 to perform remote operation of the task.

[0348] Server 13 requests a remote operation to be performed by the mobile operator selected through the matching process. In this case, server 13 may preferably request the previously requested mobile operator to perform the remote operation.

[0349] The remote moving unit 162d moves the robot 11 to a short distance position P1 through remote operation by the moving operator.

[0350] Next, the autonomous motion control unit 132 requests the interaction module 141a to perform the task of interacting with the target person 403.

[0351] Autonomous interaction unit 161a performs interaction with target person 403.

[0352] Here, for example, if the interaction with the target person 403 fails, the remote motion control unit 133 requests the server 13 to perform a remote operation of the task.

[0353] Server 13 requests a remote operation to be performed by the interactive operator selected through the matching process. In this case, server 13 may preferably request the previously requested interactive operator to perform the remote operation.

[0354] The remote interaction unit 162a performs interaction with the target person 403 through remote operation by the interactive operator.

[0355] Next, the autonomous motion control unit 132 instructs the observation module 141b to perform the task of identifying the position of the face and hands of the target person 403.

[0356] The autonomous observation unit 161b attempts to identify the location of the face and hands of the target person 403.

[0357] Here, for example, if the location of the face and hands of the target person 403 fails to be identified, the remote motion control unit 133 requests the server 13 to perform a remote operation of the task.

[0358] Server 13 requests remote operation of the task performed by the observer selected through matching processing.

[0359] The remote observation unit 162b identifies the position of the face and hands of the target person 403 through remote operation by the observation operator. Here, because the observation operator is superior to the robot 11 in situation recognition and determination, the task success rate is improved. This also applies to the following observation tasks.

[0360] Next, the autonomous motion control unit 132 instructs the working module 141c to perform the task of extending the robot 11's hand 233 in front of the target person 403.

[0361] Autonomous working unit 161c attempts to extend the hand 233 of robot 11 towards the target person 403.

[0362] Here, for example, if robot 11 is unable to extend its hand 233 in front of target person 403, remote motion control unit 133 requests server 13 to perform remote operation of the task.

[0363] Server 13 requests remote operation to perform a task by the selected worker operator through the execution of matching processing.

[0364] The remote observation unit 162b enables the robot 11 to extend its hand 233 towards the target person 403 via remote operation by the operator.

[0365] At this point, for example, even if an unexpected obstacle exists on table 401, space for extending hand 233 can be easily ensured by the operator discovering and moving the obstacle. In contrast, robot 11 has difficulty autonomously moving obstacles and extending its hand 233 in front of the target person 403.

[0366] Next, for example, the autonomous interaction unit 161a detects an unexpected situation where a third-party worker is speaking to the robot 11. The autonomous interaction unit 161a attempts to recognize the content of the worker's speech.

[0367] Here, for example, if the autonomous interaction unit 161a is unable to recognize the content of the worker's speech, the remote motion control unit 133 requests the server 13 to perform a remote operation to recognize the content of the worker's speech.

[0368] Server 13 requests that the selected interactive operator perform a remote operation task by executing the matching process.

[0369] The remote interaction unit 162a interacts with the staff through remote operation by the operator and recognizes the staff's words as "Could you move aside?".

[0370] In this case, for example, even if an unexpected situation occurs where a staff member speaks to the robot, the operator can easily recognize the content of the staff member's speech.

[0371] Next, the motion planning unit 121 determines that the task of moving to enable the worker to pass has a higher priority. In response, the autonomous motion control unit 132 instructs the motion module 141d to perform the task of moving to enable the worker to pass.

[0372] The autonomous mobile unit 161d attempts to move the robot 11 so that workers can pass through.

[0373] Here, for example, if robot 11 is unable to move within the target time to allow workers to pass, remote motion control unit 133 requests server 13 to perform the remote operation of the task.

[0374] Server 13 requests a remote operation to be performed by the mobile operator selected through the matching process. In this case, server 13 may preferably request the previously requested mobile operator to perform the remote operation.

[0375] The remote mobile unit 162d enables workers to pass through by remotely operating the mobile robot 11 through a mobile operator.

[0376] At this point, for example, even if an unexpected situation occurs where the robot needs to move to an unplanned location, the operator can easily move the robot 11.

[0377] Next, the autonomous motion control unit 132 instructs the observation module 141b to perform the task of confirming the worker's passage.

[0378] The autonomous observation unit 161b attempts to confirm that the staff has passed.

[0379] Here, for example, if it is confirmed that the worker's passage has failed, the remote motion control unit 133 requests the server 13 to perform the remote operation of the task.

[0380] Server 13 requests remote operation of a task to be performed by the observer operator selected through the matching process. In this case, server 13 may preferably request the previously requested observer operator to perform the remote operation.

[0381] The remote observation unit 162b confirms the worker's passage by observing the operator's remote operation.

[0382] At this point, for example, even if an unexpected situation arises requiring confirmation of staff passage, the observer can easily confirm the staff passage.

[0383] After that, the task of moving to a short distance position P1 is carried out again, and then the task of extending the hand 233 of robot 11 towards the target person 403 is carried out again.

[0384] Next, the autonomous motion control unit 132 instructs the interaction module 141a to perform the task of requesting the target person 403 to touch the hand 233 of the robot 11.

[0385] Autonomous interaction unit 161a requests target person 403 to touch robot 11's hand 233.

[0386] Here, for example, if the interaction with the target person 403 fails, the remote motion control unit 133 requests the server 13 to perform a remote operation of the task.

[0387] Server 13 requests a remote operation to be performed by the interactive operator selected through the matching process. In this case, server 13 may preferably request the previously requested interactive operator to perform the remote operation.

[0388] The remote interaction unit 162a requests the target person 403 to touch the hand 233 of the robot 11 through the remote operation of the interactive operator.

[0389] Next, the autonomous motion control unit 132 instructs the observation module 141b to perform the task of confirming that the hand of the target person 403 is in contact with the hand 233 of the robot 11.

[0390] The autonomous observation unit 161b confirms that the hand of the target person 403 is in contact with the hand 233 of the robot 11.

[0391] Here, for example, if it is confirmed that the hand of the target person 403 fails to make contact with the hand 233 of the robot 11, the remote motion control unit 133 requests the server 13 to perform the remote operation of the task.

[0392] Server 13 requests remote operation of a task to be performed by the observer operator selected through the matching process. In this case, server 13 may preferably request the previously requested observer operator to perform the remote operation.

[0393] The remote observation unit 162b confirms that the hand of the target person 403 is in contact with the hand 233 of the robot 11 by observing the remote operation of the operator.

[0394] Here, for example, it is assumed that remote operations for the following tasks are more efficient and have a higher success rate when performed in parallel than when performed sequentially: the task of extending the hand 233 of robot 11 in front of target person 403, the task of requesting target person 403 to touch the hand 233 of robot 11, and the task of confirming that the hand of target person 403 is in contact with the hand 233 of robot 11.

[0395] In this scenario, for example, the autonomous motion control unit 132 can request the operator, the interactive operator, and the observer to perform remote operations in parallel via the server 13. Then, the operator, the interactive operator, and the observer can execute the remote operations in parallel.

[0396] In this case, since the work module 141c, which is the target of remote operation by the work operator, the interaction module 141a, which is the target of remote operation by the interaction operator, and the observation module 141b, which is the target of remote operation by the observation operator, can perform tasks independently, the parallel processing of each task through remote operation can be successfully executed.

[0397] Note that in this context, the work operator, the interaction operator, and the observation operator are not necessarily independent operators, and for example, one operator with multiple skills may act as two or more operators.

[0398] Next, the autonomous motion control unit 132 instructs the observation module 141b to perform the task of measuring the vital signs of the target person 403.

[0399] The remote observation unit 162b measures the vital signs of the target person 403.

[0400] Next, the autonomous motion control unit 132 instructs the interaction module 141a to perform the task of requesting the target person 403 to release their hand from the robot 11's hand 233.

[0401] The autonomous interaction unit 161a requests the target person 403 to release their hand from the robot 11's hand 233.

[0402] Here, for example, if the interaction with the target person 403 fails, the remote motion control unit 133 requests the server 13 to perform a remote operation of the task.

[0403] Server 13 requests a remote operation to be performed by the interactive operator selected through the matching process. In this case, server 13 may preferably request the previously requested interactive operator to perform the remote operation.

[0404] The remote interaction unit 162a requests the target person 403 to release their hand from the robot 11's hand 233 through the remote operation of the interactive operator.

[0405] Next, the autonomous motion control unit 132 instructs the observation module 141b to perform the task of confirming that the target person 403 has released their hand from the robot 11's hand 233.

[0406] The autonomous observation unit 161b confirms that the target person 403 releases their hand from the robot 11's hand 233.

[0407] Here, for example, if it is confirmed that the target person 403 has failed to release his hand from the robot 11's hand 233, the remote motion control unit 133 requests the server 13 to perform the remote operation of the task.

[0408] Server 13 requests remote operation of a task to be performed by the observer operator selected through the matching process. In this case, server 13 may preferably request the previously requested observer operator to perform the remote operation.

[0409] The remote observation unit 162b confirms that the target person 403 has released their hand from the robot 11's hand 233 by observing the operator's remote operation.

[0410] Next, the autonomous motion control unit 132 instructs the working module 141c to perform the task of restoring the robot 11's hand 233 to its original posture.

[0411] The autonomous working unit 161c attempts to restore the hand 233 of the robot 11 to its original posture.

[0412] Here, for example, if robot 11 is unable to restore its hand 233 to its original posture, remote motion control unit 133 requests server 13 to perform remote operation of the task.

[0413] Server 13 requests remote operation to perform a task by the selected worker operator through the execution of matching processing.

[0414] The remote observation unit 162b restores the robot 11's hand 233 to its original posture through remote operation by the operator.

[0415] Next, the autonomous motion control unit 132 requests the interaction module 141a to perform the task of interacting with the target person 403.

[0416] Autonomous interaction unit 161a performs interaction with target person 403.

[0417] Here, for example, if the interaction with the target person 403 fails, the remote motion control unit 133 requests the server 13 to perform a remote operation of the task.

[0418] Server 13 requests a remote operation to be performed by the interactive operator selected through the matching process. In this case, server 13 may preferably request the previously requested interactive operator to perform the remote operation.

[0419] The remote interaction unit 162a performs interaction with the target person 403 through remote operation by the interactive operator.

[0420] Next, the autonomous motion control unit 132 instructs the motion module 141d to perform the task of moving to the next target location.

[0421] The autonomous mobile unit 161d moves robot 11 to the next target position.

[0422] Here, for example, if robot 11 is unable to move to the next target location within the target time, remote motion control unit 133 requests server 13 to perform remote operation of the task.

[0423] Server 13 requests a remote operation to be performed by the mobile operator selected through the matching process. In this case, server 13 may preferably request the previously requested mobile operator to perform the remote operation.

[0424] The remote mobile unit 162d moves the robot 11 to the next target location through remote operation by the mobile operator.

[0425] By combining the above-mentioned autonomous and remote operations, robot 11 can successfully measure the vital signs of target person 403.

[0426] <Example of performing the action of opening and passing through a door>

[0427] Next, an example of robot 11 performing the action of opening and passing through a door will be described.

[0428] For example, motion planning unit 121 plans to move in front of the door, open the door, and then perform the action of opening the door.

[0429] Next, the autonomous motion control unit 132 instructs the motion module 141d to perform the task of moving to the front of the door.

[0430] The autonomous mobile unit 161d moves the robot 11 to the front of the door.

[0431] Here, for example, if robot 11 is unable to move in front of the door, remote motion control unit 133 requests server 13 to perform remote operation of the task.

[0432] Server 13 requests remote operation to perform the task by the selected mobile operator through the execution of matching processing.

[0433] The remote moving unit 162d moves the robot 11 to the front of the door through remote operation by the moving operator.

[0434] Next, the autonomous motion control unit 132 instructs the observation module 141b to perform the task of identifying the door handle.

[0435] The autonomous observation unit 161b attempts to identify door handles.

[0436] Here, for example, if the door handle fails to be identified, the remote motion control unit 133 requests the server 13 to perform a remote operation of the task.

[0437] Server 13 requests remote operation of the task performed by the observer selected through matching processing.

[0438] The remote observation unit 162b identifies the door handle by observing the operator's remote operation.

[0439] Next, the motion planning unit 121 plans the action of opening and moving the door based on the shape and position of the door and the door handle.

[0440] Next, the autonomous motion control unit 132 instructs the working module 141c to perform the task of gripping the door handle.

[0441] Autonomous work unit 161c attempts to grasp the door handle.

[0442] Here, for example, if robot 11 is unable to grasp the door handle within the target time, remote motion control unit 133 requests server 13 to perform remote operation of the task.

[0443] Server 13 requests remote operation to perform a task by the selected worker operator through the execution of matching processing.

[0444] The remote observation unit 162b grasps the door handle through remote operation by the operator. In this case, the task success rate is improved because the operator has a superior ability to recognize the shape and position of the door handle compared to the robot 11.

[0445] Next, the autonomous motion control unit 132 instructs the working module 141c to perform the task of turning the door handle.

[0446] Autonomous working unit 161c attempts to turn the door handle.

[0447] Here, for example, if robot 11 is unable to turn the door handle within the target time, remote motion control unit 133 requests server 13 to perform remote operation of the task.

[0448] Server 13 requests a remote operation to be performed by the worker selected through the matching process. In this case, server 13 may preferably request the previously requested worker to perform the remote operation.

[0449] The remote observation unit 162b rotates the door handle remotely by the operator. Since the operator has a superior ability to adjust the direction and force of movement based on the type and shape of the door handle compared to the robot 11, the task success rate is improved.

[0450] Next, the autonomous motion control unit 132 instructs the working module 141c and the moving module 141d to perform the tasks of pulling and opening the door.

[0451] While moving the robot 11 appropriately, the autonomous working unit 161c and the autonomous moving unit 161d attempt to perform door-pulling and door-opening operations.

[0452] Here, for example, if robot 11 is unable to pull and open the door within the target time, remote motion control unit 133 requests server 13 to perform the remote operation of the task.

[0453] Server 13 requests remote operations to be performed by the selected worker operator and mobile operator through the matching process. In this case, server 13 may preferably request the previously requested worker operator to perform the remote operation. Alternatively, server 13 may request a remote operator to perform the remote operation.

[0454] The remote working unit 162c and the remote moving unit 162d open and close doors remotely via the working operator and the moving operator, respectively. In this case, the task success rate is improved because the working operator and the moving operator are superior to the robot 11 in terms of situation recognition and determination.

[0455] Next, the autonomous motion control unit 132 instructs the working module 141c to perform the task of keeping the door open.

[0456] The autonomous working unit 161c attempts to keep the door open using the hand opposite the hand gripping the door handle.

[0457] Here, for example, if robot 11 is unable to keep the door open within the target time, remote motion control unit 133 requests server 13 to perform the remote operation of the task.

[0458] Server 13 requests a remote operation to be performed by the worker selected through the matching process. In this case, server 13 may preferably request the previously requested worker to perform the remote operation.

[0459] The remote work unit 162c keeps the door open via remote operation by the operator.

[0460] Next, the autonomous motion control unit 132 instructs the working module 141c and the moving module 141d to perform the task of passing through the door while keeping the door in place.

[0461] The autonomous working unit 161c and the autonomous moving unit 161d attempt to operate the door while maintaining its position.

[0462] Here, for example, if robot 11 is unable to pass through the door while holding it open within the target time, remote motion control unit 133 requests server 13 to perform the remote operation for the task.

[0463] Server 13 requests remote operations to be performed by the selected worker and mobile operators through matching processing. In this case, server 13 may preferably request previously requested worker and mobile operators to perform remote operations. Alternatively, server 13 may request a remote operator to perform a remote operation.

[0464] The remote working unit 162c and the remote moving unit 162d enable the robot to pass through the door while maintaining its position through remote operation by the working operator and the moving operator, respectively. In this case, the task success rate is improved because the working operator and the moving operator are superior to the robot 11 in terms of situation recognition and determination.

[0465] By combining the above-mentioned autonomous and remote operations, robot 11 can successfully open and pass through doors.

[0466] Note that in this operation, two or more of the work operator, the moving operator, and the observing operator can perform remote operations in parallel as needed. This arrangement can improve the efficiency of remote operations for each task in some cases. Furthermore, in this scenario, the work operator, the moving operator, and the observing operator are not necessarily separate operators, and for example, one operator with multiple skills can act as two or more operators.

[0467] <Effects of this technology>

[0468] The effects of this technique will be described below.

[0469] For example, the reliability and functionality of various actions of robot 11 are improved. That is, by enabling robot 11 to autonomously perform tasks it excels at and to perform tasks it is not adept at through remote operation, a new structure can be achieved for robot 11, which can utilize the brain or sensory organs of a remote operator as components. Through this arrangement, the reliability of various actions of robot 11 can be enhanced to a human level or higher. Furthermore, robot 11 can stably perform a variety of tasks requiring attentive service.

[0470] For example, because each action of robot 11 is divided into multiple micro-level tasks before requesting remote operation, the efficiency of the remote operator is improved. That is, because each task is simplified to a level independent of the context, remote operation becomes easy and simple, and the time required for remote operation is also reduced. Therefore, for example, each remote operator can perform remote operations on multiple robots 11 in parallel. Furthermore, for example, each remote operator can participate in the remote operation of robot 11 while performing other work or tasks, or can participate in the remote operation of robot 11 for a short period of additional time.

[0471] For example, because robot 11 is modularized by skill, skill-related remote operations can be performed in parallel. This allows multiple remote operators to remotely operate a single robot 11 in parallel.

[0472] For example, the number of skills required by remote operators is reduced, thus enabling more remote operators to perform remote operations. For example, because each task is simple, a user interface with good operability can be implemented. Therefore, for example, even disabled people can easily perform remote operations. Through this arrangement, as referenced above… Figure 1 As stated, many people around the world can participate in and play an active role in the remote operation of robots 11 worldwide. Furthermore, each remote operator finds it easier to find work, simultaneously alleviating labor shortages. Additionally, by lowering the required level for each task and increasing operability, the failure rate of remote operations is reduced, and the quality of remote operation tasks is improved.

[0473] For example, the amount and time required for remotely operating a robot 11 can be reduced. This can lower the cost of operating the robot 11.

[0474] Figure 13 An example of the formula for calculating the operating effect R of the remote operation control system 1 is shown.

[0475] The operational effectiveness R is directly proportional to the total number of robots M, the parallel subdivision coefficient P, and the autonomy coefficient Q. On the other hand, the operational effectiveness R is inversely proportional to the number of operators N, the number of operation modules K, and the required operational skill coefficient J.

[0476] The total number of robots M is the total number of robots 11 operating in the remote operation control system 1.

[0477] The parallel subdivision factor P indicates the degree of parallel subdivision of a task. For example, the more subdivided the task, the larger the parallel subdivision factor P.

[0478] The autonomy coefficient Q indicates the degree of autonomy of robot 11. As the proportion of tasks that robot 11 can perform autonomously increases, the autonomy coefficient Q also increases. Generally, the more subdivided the tasks, the greater the proportion of tasks that robot 11 can perform autonomously, and therefore the greater the autonomy coefficient Q becomes.

[0479] The number of operators N is the total number of remote operators required to successfully operate the remote operation control system 1, which consists of the total number of robots M operating in the system.

[0480] The number of operation modules K is the number of task execution modules 141 included in robot 11. For example, since the robot 11 includes four task execution modules 141, namely interaction module 141a, observation module 141b, working module 141c and movement module 141d, the number of operation modules K is 4.

[0481] The required operational skill coefficient J indicates the skill level required for remote operation of a task, and J increases as the skill level required for remote operation increases. Generally, the more subdivided the task, the lower the skill level required for remote operation, and therefore the smaller the required operational skill coefficient J. Typically, as the number of robots 11 operated by each person increases, the skill level required for remote operation of the task also increases, and therefore the required operational skill coefficient J also increases.

[0482] In the remote operation control system 1, increasing the number of operators N and the number of operation modules K reduces the operational efficiency R. On the other hand, the parallel subdivision coefficient P increases, and the required operational skill coefficient J decreases. This increases the number of robots 11 that each remote operator can operate, and allows for a significant increase in the total number of robots M. Therefore, the overall operational efficiency R can be improved.

[0483] Figure 14 An example of the effect R of the operation is shown.

[0484] Example 1 illustrates an example of a remote operator manipulating a robot 11. In this case, the operation effect R is 1, and this value is a reference value for the operation effect R.

[0485] Note that in Example 1, the number of remote operators is 1, and the task execution modules of robot 11 are not necessarily divided. Therefore, an example with the number of operation modules K is shown.

[0486] Example 2 illustrates a scenario where a remote operator controls five robots 11. In this case, to suppress the increase in the necessary operational skill coefficient J, the autonomy coefficient Q of robot 11 needs to be increased. However, even with the increased autonomy coefficient Q, a remote operator is still required to control all five robots throughout the operation; therefore, the necessary operational skill coefficient J increases. Thus, in this example, the value of the operational effect R becomes 2.

[0487] Note that in Example 2, similar to Example 1, the number of remote operators is 1, and the task execution modules of robot 11 are not necessarily divided, thus showing an example where the number of operation modules K is 1.

[0488] Example 3 illustrates an example of 10 remote operators operating 100 robots 11 in a remote operation control system 1. In this case, since each robot 11 includes 4 task execution modules 141, the number of operation modules K is 4. As the number of operation modules K increases, the degree of parallel subdivision of tasks increases, the parallel subdivision coefficient P increases, and the required operational skill coefficient J decreases. This causes the value of the operational effect R to increase from 2 to 20 compared to Example 2. In other words, a large number of robots can be operated while limiting the number of remote operators, and the operational effect is improved.

[0489] <<2. Modified Example>>

[0490] In the following text, modifications to the above-described embodiments of this technology will be described.

[0491] For example, the classification of robot 11's skills (task execution module) can be appropriately changed.

[0492] For example, after the server 13 performs matching processing and controls the connection between the robot 11, which is the target of remote operation, and the operating terminal 12 of the remote operator, the robot 11 and the operating terminal 12 can communicate directly with each other and perform remote operations without the intervention of the server 13.

[0493] For example, robot 11 can perform matching processing independently based on a remote operator database, and request the selected remote operator (operation terminal 12) to perform remote operations. In this case, for example, robot 11 can generate a remote operator database, or it can obtain a remote operator database from an external source.

[0494] This technology can be applied to remotely operated autonomous mobile bodies other than robots. For example, it can be applied to vehicles such as cars and drones that can be remotely and autonomously operated. It can also be applied to manipulators that do not move but can remotely or autonomously move parts (such as hands).

[0495] <<3. Others>>

[0496] <Example of computer configuration>

[0497] The above series of processes can be performed using either hardware or software. In the case where the processes are performed using software, a program configuring the software is installed on the computer. Examples of computers include those integrated into dedicated hardware, and those capable of performing various functions by installing various programs, such as general-purpose personal computers.

[0498] Figure 15 This is a block diagram illustrating an example configuration of computer hardware that performs the above series of processes through a program.

[0499] In computer 1000, central processing unit (CPU) 1001, read-only memory (ROM) 1002 and random access memory (RAM) 1003 are interconnected via bus 1004.

[0500] In computer 1000, central processing unit (CPU) 1001, read-only memory (ROM) 1002 and random access memory (RAM) 1003 are interconnected via bus 1004.

[0501] Input unit 1006 includes input switches, buttons, microphones, imaging elements, etc. Output unit 1007 includes displays, speakers, etc. Storage unit 1008 includes hard disks, non-volatile memory, etc. Communication unit 1009 includes network interfaces, etc. Driver 1010 drives removable media 1011 such as disks, optical disks, magneto-optical disks, or semiconductor memories.

[0502] In the computer 1000 configured as described above, for example, the CPU 1001 loads a program stored in the storage unit 1008 into the RAM 1003 via the input / output interface 1005 and the bus 1004 and executes the program to perform the series of processes described above.

[0503] The program executed by the computer 1000 (CPU 1001) can be provided, for example, by recording it in a removable medium 1011 as an encapsulation medium. Alternatively, the program can be provided via wired or wireless transmission media such as a local area network, the Internet, or digital satellite broadcasting.

[0504] In computer 1000, a program can be installed in storage unit 1008 via input / output interface 1005 by attaching removable medium 1011 to drive 1010. Alternatively, the program can be received by communication unit 1009 via wired or wireless transmission medium for installation in storage unit 1008. Alternatively, the program can be pre-installed in ROM 1002 or storage unit 1008.

[0505] Note that a program executed by a computer may be a program that performs processing in a time sequence in the order described in this specification, or it may be a program that performs processing in parallel or at necessary time intervals (e.g., when a call is made).

[0506] Furthermore, in this specification, a system refers to a collection of multiple constituent elements (devices, modules (components), etc.), regardless of whether all constituent elements are housed in the same housing. Therefore, both multiple devices housed in separate housings and interconnected via a network, and a device comprising multiple modules housed in a single housing, are systems.

[0507] Furthermore, the implementation of this technology is not limited to the above-described implementation, and various modifications can be made without departing from the essential points of this technology.

[0508] For example, this technology can be embodied in cloud computing, where multiple devices collaborate via a network to share and process a function.

[0509] Furthermore, each step described in the flowchart above can be performed by a single device, or it can be shared and performed by multiple devices.

[0510] Furthermore, in cases where a step includes multiple processes, the multiple processes included in a step can be performed by a single device, or can be performed by multiple devices in a shared manner.

[0511] <Example of configuration combinations>

[0512] This technology can also be configured as follows. (1)

[0514] An information processing apparatus, comprising:

[0515] A matching unit, which, in the tasks obtained by decomposing the actions of an autonomous mobile body, matches non-autonomous tasks not autonomously performed by the autonomous mobile body with remote operators performing the remote operations of the non-autonomous tasks; and

[0516] A remote control unit that controls the remote operation of the non-autonomous task by the remote operator. (2)

[0518] According to the information processing device described in (1) above, wherein,

[0519] The remote control unit receives remote operation content information from the operating terminal operated by the remote operator, which instructs the remote operator on the content of the remote operation, and sends the received remote operation content information to the autonomous mobile body. (3)

[0521] According to the information processing device described in (2) above, wherein,

[0522] The remote control unit provides the operating terminal with a user interface for remotely operating the non-autonomous task. (4)

[0524] According to the information processing device described in (3) above, wherein,

[0525] The remote control unit provides the operating terminal with a user interface that includes a virtual space, which simulates the environment in which the action is performed. (5)

[0527] According to the information processing device described in (3) above, wherein,

[0528] The remote control unit provides the operating terminal with a user interface that is independent of the environment in which the action is performed. (6)

[0530] According to the information processing device described in (1) above, wherein,

[0531] The remote control unit controls the connection between the autonomous mobile body and the operating terminal operated by the remote operator. (7)

[0533] The information processing apparatus according to any one of (1) to (6) above, wherein,

[0534] The matching unit matches the non-autonomous tasks of the multiple autonomous mobile entities with the multiple remote operators. (8)

[0536] The information processing apparatus according to any one of (1) to (7) above, wherein,

[0537] The non-autonomous tasks are categorized by each skill, and

[0538] The matching unit matches the non-autonomous task with a remote operator who is capable of performing remote operations with skills related to the non-autonomous task. (9)

[0540] According to the information processing device described in (8) above, wherein,

[0541] The skills mentioned include interaction, observation, work, and movement. (10)

[0543] The information processing apparatus according to any one of (1) to (9) above, wherein,

[0544] The non-autonomous task is a task that is broken down into levels that are independent of the context in which the action is performed. (11)

[0546] The information processing apparatus according to any one of (1) to (10) above, wherein,

[0547] The non-autonomous tasks include tasks that the autonomous mobile body cannot perform autonomously and tasks that the autonomous mobile body fails to perform autonomously. (12)

[0549] The information processing apparatus according to any one of (1) to (11) above, wherein,

[0550] The matching unit updates at least one of the information about the remote operator and the information about the non-autonomous task based on the execution content of the non-autonomous task through the remote operation. (13)

[0552] A remote operation and control method,

[0553] The remote operation control method causes the information processing device to perform:

[0554] In tasks obtained by decomposing the actions of an autonomous mobile body, non-autonomous tasks not performed autonomously by the autonomous mobile body are matched with remote operators performing the remote operations of the non-autonomous tasks; and

[0555] Control the remote operation of the non-autonomous task by the remote operator. (14)

[0557] A remote operation and control system, comprising:

[0558] Multiple autonomous mobile entities; and

[0559] Information processing device, wherein

[0560] The information processing device includes:

[0561] A matching unit, which, among the tasks obtained by decomposing the actions of each of the autonomous mobile bodies, matches non-autonomous tasks not autonomously performed by the autonomous mobile bodies with remote operators performing the remote operations of the non-autonomous tasks; and

[0562] A remote control unit that controls the remote operation of the non-autonomous task by the remote operator. (15)

[0564] An autonomous mobile body, comprising:

[0565] Action planning unit, which decomposes the action to be performed into multiple tasks; and

[0566] The motion control unit controls the autonomous execution of autonomous tasks within the task, and controls the execution of non-autonomous tasks within the task through remote operation. (16)

[0568] The autonomous mobile body described above (15) includes:

[0569] Multiple task execution modules, each corresponding to a different skill; among them...

[0570] Each of the task execution modules executes autonomous tasks associated with the corresponding skill, and performs non-autonomous tasks associated with the corresponding skill through remote operation. (17)

[0572] According to the autonomous mobile body described above (16), wherein,

[0573] The motion control unit categorizes the non-autonomous tasks according to each skill and requests external remote operation to perform the non-autonomous tasks. (18)

[0575] According to any one of (15) to (17) above, the autonomous mobile body, wherein,

[0576] The non-autonomous tasks include tasks that the autonomous mobile body cannot perform autonomously and tasks that the autonomous mobile body fails to perform autonomously. (19)

[0578] The autonomous mobile body according to any one of (15) to (18) above, wherein,

[0579] The action planning unit decomposes the action into tasks at the following levels: the level is independent of the background environment in which the action is performed. (20)

[0581] The autonomous mobile body according to any one of (15) to (19) above further includes:

[0582] The learning unit learns the autonomous execution method of the non-autonomous task based on the execution content of the non-autonomous task through remote operation.

[0583] Note that the effects described in this instruction are for illustrative purposes only and are not limited to other effects.

[0584] List of reference numerals

[0585] 1. Remote Operation Control System

[0586] 11-1 to 11-m robots

[0587] 12-1 to 12-m Operating Terminal

[0588] 13 servers

[0589] 21 Network

[0590] 101 Information Processing Unit

[0591] 102 input units

[0592] 103 sensing units

[0593] 104 communication units

[0594] 111 Action Module

[0595] 112 Task Execution Module Group

[0596] 121 Motion Planning Unit

[0597] 122 Motion Control Unit

[0598] Learning Unit 123

[0599] 131 Status Recognition Unit

[0600] 132 Autonomous Motion Control Unit

[0601] 133 Remote Motion Control Unit

[0602] 141a Interactive Module

[0603] 141b Observation Module

[0604] 141c Working Module

[0605] 141d mobile module

[0606] Execution units 151a to 151d

[0607] Learning Units 152a to 152d

[0608] 161a Autonomous Interactive Unit

[0609] 161b Autonomous Observation Unit

[0610] 161c Autonomous Working Unit

[0611] 161d Autonomous Mobile Unit

[0612] 162a Remote Interaction Unit

[0613] 162b Remote Observation Unit

[0614] 162c Remote Working Unit

[0615] 162d Remote Mobile Device

[0616] 302 Information Processing Device

[0617] 311 Remote Control Unit

[0618] 312 Matching Units

Claims

1. An information processing apparatus, comprising: A matching unit, which, in the tasks obtained by decomposing the actions of an autonomous mobile body, matches non-autonomous tasks not autonomously performed by the autonomous mobile body with remote operators performing the remote operations of the non-autonomous tasks; and A remote control unit that controls the remote operation of the non-autonomous task by the remote operator.

2. The information processing apparatus according to claim 1, wherein, The remote control unit receives remote operation content information from the operating terminal operated by the remote operator, which instructs the remote operator on the content of the remote operation, and sends the received remote operation content information to the autonomous mobile body.

3. The information processing apparatus according to claim 2, wherein, The remote control unit provides the operating terminal with a user interface for remotely operating the non-autonomous task.

4. The information processing apparatus according to claim 3, wherein, The remote control unit provides the operating terminal with a user interface that includes a virtual space, which simulates the environment in which the action is performed.

5. The information processing apparatus according to claim 3, wherein, The remote control unit provides the operating terminal with a user interface that is independent of the environment in which the action is performed.

6. The information processing apparatus according to claim 1, wherein, The remote control unit controls the connection between the autonomous mobile body and the operating terminal operated by the remote operator.

7. The information processing apparatus according to claim 1, wherein, The matching unit matches the non-autonomous tasks of the multiple autonomous mobile entities with the multiple remote operators.

8. The information processing apparatus according to claim 1, wherein, The non-autonomous tasks are categorized by each skill, and The matching unit matches the non-autonomous task with a remote operator who is capable of performing remote operations with skills related to the non-autonomous task.

9. The information processing apparatus according to claim 8, wherein, The skills mentioned include interaction, observation, work, and movement.

10. The information processing apparatus according to claim 1, wherein, The non-autonomous task is a task that is broken down into levels that are independent of the context in which the action is performed.

11. The information processing apparatus according to claim 1, wherein, The non-autonomous tasks include tasks that the autonomous mobile body cannot perform autonomously and tasks that the autonomous mobile body fails to perform autonomously.

12. The information processing apparatus according to claim 1, wherein, The matching unit updates at least one of the information about the remote operator and the information about the non-autonomous task based on the execution content of the non-autonomous task through the remote operation.

13. A remote operation and control method, The remote operation control method causes the information processing device to perform: In tasks obtained by decomposing the actions of an autonomous mobile body, non-autonomous tasks not performed autonomously by the autonomous mobile body are matched with remote operators performing the remote operations of the non-autonomous tasks; and Control the remote operation of the non-autonomous task by the remote operator.

14. A remote operation control system, comprising: Multiple autonomous moving bodies; as well as Information processing device, wherein The information processing device includes: A matching unit, which, in the tasks obtained by decomposing the actions of each of the autonomous mobile bodies, matches non-autonomous tasks not autonomously performed by the autonomous mobile bodies with remote operators performing the remote operations of the non-autonomous tasks; and A remote control unit that controls the remote operation of the non-autonomous task by the remote operator.

15. An autonomous mobile body, comprising: An action planning unit, which decomposes the action to be performed into multiple tasks; as well as The motion control unit controls the autonomous execution of autonomous tasks within the task, and controls the execution of non-autonomous tasks within the task through remote operation.

16. The autonomous mobile body according to claim 15, comprising: Multiple task execution modules, each corresponding to a different skill; among them... Each of the task execution modules executes autonomous tasks associated with the corresponding skill, and performs non-autonomous tasks associated with the corresponding skill through remote operation.

17. The autonomous mobile body according to claim 16, wherein, The motion control unit categorizes the non-autonomous tasks according to each skill and requests external remote operation to perform the non-autonomous tasks.

18. The autonomous mobile body according to claim 15, wherein, The non-autonomous tasks include tasks that the autonomous mobile body cannot perform autonomously and tasks that the autonomous mobile body fails to perform autonomously.

19. The autonomous mobile body according to claim 15, wherein, The action planning unit decomposes the action into tasks at the following levels: the level is independent of the background environment in which the action is performed.

20. The autonomous mobile body according to claim 15, further comprising: The learning unit learns the autonomous execution method of the non-autonomous task based on the execution content of the non-autonomous task through remote operation.

Images