ROBOT MANAGEMENT DEVICE, CONTROL METHOD, AND PROGRAM

The robot management device and method address the challenge of selecting the right operating terminal for external input by prioritizing tasks and considering terminal and operator information, improving task execution efficiency and flexibility.

JP7593481B2Active Publication Date: 2024-12-03NEC CORP
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
JP2023512634
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Filing Date
2021-04-09
Publication Date
2024-12-03
Estimated Expiration
2041-04-09

AI Technical Summary

Technical Problem

Existing robot systems lack the ability to suitably determine the appropriate operating terminal for generating external input based on the diversity of operation methods required for different tasks, leading to inefficiencies in task execution.

Method used

A robot management device and method that determines the necessity of external input control by evaluating support request information, prioritizing tasks, and selecting the most suitable operating terminal based on terminal and task information, including error handling and operator availability.

Benefits of technology

Enables the robot system to efficiently switch to external input control when necessary, ensuring smooth task execution by selecting the appropriate operating terminal, thereby enhancing operational flexibility and reliability.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007593481000001
    Figure 0007593481000001
  • Figure 0007593481000002
    Figure 0007593481000002
  • Figure 0007593481000003
    Figure 0007593481000003
Patent Text Reader

Abstract

A robot management device 3X primarily has a means 35X for assessing external input necessity and a means 36X for operation terminal determination. The means 35X for assessing external input necessity assesses the necessity of external-input-based control of a robot for executing a task. When the external-input-based control is necessary, the means 36X for operation terminal determination determines an operation terminal for generating an external input on the basis of information that pertains to the task and operation terminal information including information that pertains to the types of a plurality of operation terminals that are candidates for generating the external input.
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present disclosure relates to the technical field of controlling the movement of a robot. [Background technology]

[0002] A robot system has been proposed in which a robot executes a part of a task based on external input from a human when the robot executes the task. For example, Patent Document 1 discloses a robot system that requests remote operation from an operation terminal operated by an operator when autonomous control alone becomes difficult. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2007-190659 A Summary of the Invention [Problem to be solved by the invention]

[0004] When operating a robot based on external input, the operations required vary depending on the work content. On the other hand, the robot system described in Patent Document 1 is an interactive robot, and does not take into consideration the diversity of operation methods when selecting an operation terminal.

[0005] In view of the above-mentioned problems, one object of the present disclosure is to provide a robot management device, a control method, and a storage medium that are capable of suitably determining an operation terminal that generates an external input. [Means for solving the problem]

[0006] One aspect of the robot management device is an external input necessity determination means for determining whether or not control based on an external input of a robot executing a task is necessary; and an operation terminal determination means for determining, when control based on the external input is necessary, the operation terminal that generates the external input based on operation terminal information including information on the types of a plurality of operation terminals that are candidates for generating the external input and information on the task. death, the external input necessity determination means determines that control based on the external input is necessary when support request information including information related to the task is received from the robot or a robot controller that controls the robot; When the operation terminal determination means cannot determine the operation terminal, the operation terminal determination means accumulates the support request information for which support has not been executed, determines a priority of each piece of support request information based on an urgency of the support, and processes the support request information in descending order of priority when the support becomes executable. It is a robot management device.

[0007] One aspect of the control method includes: The computer The control requirements for a robot to execute a task based on external inputs No judgement and determining that control based on the external input is necessary when assistance request information including information related to the task is received from the robot or a robot controller that controls the robot. death, When control based on the external input is necessary, the operation terminal that generates the external input is determined based on operation terminal information including information on the types of a plurality of operation terminals that are candidates for generating the external input and information on the task. If the operation terminal cannot be determined, the support request information for which support has not been executed is accumulated, and a priority of each piece of support request information is determined based on the urgency of the support. When the support becomes executable, the support request information is processed in descending order of priority. do, It is a control method.

[0008] One aspect of the program is The control requirements for a robot to execute a task based on external inputs No judgement and determining that control based on the external input is necessary when assistance request information including information related to the task is received from the robot or a robot controller that controls the robot. death, When control based on the external input is necessary, the operation terminal that generates the external input is determined based on operation terminal information including information on the types of a plurality of operation terminals that are candidates for generating the external input and information on the task. If the operation terminal cannot be determined, the support request information for which support has not been executed is accumulated, and a priority of each piece of support request information is determined based on the urgency of the support. When the support becomes executable, the support request information is processed in descending order of priority. The process Programs to be executed by a computer in be. Effect of the Invention

[0009] It is possible to suitably determine the operation terminal that generates the external input. [Brief description of the drawings]

[0010] [Figure 1]1 shows a configuration of a robot control system in a first embodiment. [Diagram 2] (A) shows the hardware configuration of a robot controller, (B) shows the hardware configuration of an operation terminal, and (C) shows the hardware configuration of a robot management device. [Diagram 3] 13 shows an example of a data structure of application information. [Figure 4] 1A is an example of a data structure of operation terminal information, and FIG. 1B is an example of a data structure of operator information. [Diagram 5] 2 is an example of a functional block diagram illustrating an overview of processing of a robot control system. [Figure 6] 4 is an example of a functional block diagram showing a functional configuration of an operation sequence generation unit. [Figure 7] 1 is a first display example of an operation screen. [Figure 8] 13 is a second display example of the operation screen. [Figure 9] 4 is an example of a flowchart executed by the robot management device in the first embodiment. [Figure 10] FIG. 13 shows a schematic configuration diagram of a robot management device according to a second embodiment. [Figure 11] 13 is an example of a flowchart illustrating a process of a robot management device in a second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings.

[0012] First Embodiment (1) System Configuration 1 shows the configuration of a robot control system 100 according to the first embodiment. The robot control system 100 mainly includes an operation terminal 2 (2A, 2B, ...), a robot management device 3, and a plurality of task execution systems 50 (50A, 50B, ...). The operation terminal 2, the robot management device 3, and the task execution systems 50 communicate with each other via a communication network 6.

[0013] The operation terminal 2 is a terminal that accepts a support operation required for the robot 5 in the task execution system 50 to execute a task, and is used by an operator (operators a to c, ...). Specifically, when there is a task execution system 50 requesting support, the operation terminal 2 establishes a communication connection with the task execution system 50 based on the connection control by the robot management device 3, and transmits an external input signal "Se" generated by the operation (manual operation) of the operator to the task execution system 50 that is the request source. Here, the external input signal Se is an input signal of the operator that represents a command that directly or indirectly specifies the operation of the robot 5 that requires support.

[0014] In this embodiment, the operation terminal 2 includes a plurality of types of terminals with different operation methods. For example, the operation terminal 2 includes a tablet terminal, a stationary personal computer, a game controller, a virtual reality (VR) terminal, and the like. Then, as described later, when there is a task execution system 50 that requests assistance, an appropriate type of operation terminal 2 is selected as the operation terminal 2 that provides assistance depending on the content of the task to be supported, and the like.

[0015] Note that the relationship between the operation terminal 2 and the operator is not necessarily one-to-one, and there may be a situation in which a plurality of operators use one operation terminal 2, or a situation in which a single operator uses a plurality of operation terminals 2. The operation terminal 2 may further receive an input from the operator specifying a task to be executed in the task execution system 50. In this case, the operation terminal 2 transmits task specification information generated by the input to the target task execution system 50.

[0016] The robot management device 3 manages the connection between the task execution system 50 that requires support from the operation terminal 2 and the operation terminal 2 that provides the support. When the robot management device 3 receives support request information "Sr" from any of the task execution systems 50, it selects an operation terminal 2 (and an operator) suitable for supporting the requesting task execution system 50, and executes connection control for establishing a communication connection between the selected operation terminal 2 (also called the "target operation terminal 2") and the requesting task execution system 50.

[0017] The communication mode between the task execution system 50 and the target operating terminal 2 may be a mode in which data communication is performed directly by constructing a VPN (Virtual Private Network) or the like, or a mode in which data communication is performed indirectly by the robot management device 3 performing a transfer process of the data communication. In the former mode, the robot management device 3 performs a process of transmitting the communication address of the other to at least one of the task execution system 50 (more specifically, the robot controller 1) or the target operating terminal 2, for example, as a connection control, so that they can communicate directly. In the latter mode, the robot management device 3 performs a process of establishing a communication connection dedicated to transfer with each of the task execution system 50 and the target operating terminal 2, for example, as a connection control.

[0018] The task execution systems 50 are robot systems that execute designated tasks, and are installed in different environments. The task execution systems 50 may be systems that perform picking tasks in a factory (e.g., taking parts from a shelf and arranging the taken parts on a tray, etc.) as tasks, or may be any robot system outside a factory. Examples of such robot systems include robot systems that perform shelf-stocking tasks in retail (e.g., arranging products in containers on store shelves), robot systems that perform product checks (e.g., taking expired products from shelves or attaching discount labels), and the like.

[0019] Each task execution system 50 includes a robot controller 1 (1A, 1B, . . .), a robot 5 (5A, 5B, . . .), and a sensor 7 (7A, 7B, . . .).

[0020] When a task to be executed by a robot 5 belonging to the same task execution system 50 is specified, the robot controller 1 formulates an operation plan for the robot 5 and controls the robot 5 based on the operation plan. For example, the robot controller 1 converts a task expressed by temporal logic into a sequence for each time step (time interval) of the task, which is a unit that the robot 5 can accept, and controls the robot 5 based on the generated sequence. Hereinafter, a task (command) obtained by breaking down a task into units that the robot 5 can accept is also called a "subtask," and a sequence of subtasks that the robot 5 should execute to accomplish a task is called a "subtask sequence" or an "operation sequence." As will be described later, the subtasks include a task that requires assistance from the operation terminal 2 (i.e., manual control).

[0021] The robot controller 1 also has an application information storage unit 41 (41A, 41B, ...) that stores application information required to generate an operation sequence of the robot 5 from a task. The application information will be described in detail later with reference to FIG.

[0022] Furthermore, the robot controller 1 performs data communication with the robot 5 and the sensor 7 belonging to the same task execution system 50 via a communication network or by direct wireless or wired communication. For example, the robot controller 1 transmits a control signal related to the control of the robot 5 to the robot 5. In another example, the robot controller 1 receives a sensor signal generated by the sensor 7. Furthermore, the robot controller 1 performs data communication with the operation terminal 2 via the communication network 6.

[0023] One or more robots 5 are present for each task execution system 50, and perform tasks based on control signals supplied from a robot controller 1 belonging to the same task execution system 50. The robot 5 may be a vertical articulated robot, a horizontal articulated robot, or any other type of robot, and may have a plurality of control objects (manipulators, end effectors) such as robot arms that operate independently. The robot 5 may also work in cooperation with other robots, workers, or machine tools operating within the workspace. The robot controller 1 and the robot 5 may also be integrated into one unit.

[0024] Furthermore, the robot 5 may supply a status signal indicating the status of the robot 5 to the robot controller 1 belonging to the same task execution system 50. This status signal may be an output signal of a sensor that detects the status (position, angle, etc.) of the entire robot 5 or a specific part such as a joint, or may be a signal supplied to the robot 5 indicating the progress of the operation sequence.

[0025] The sensor 7 is one or more sensors, such as a camera, a range finder, a sonar, or a combination of these, that detects the state in the workspace in which the task is executed in each task execution system 50. The sensor 7 supplies the generated signal (also called a "sensor signal") to the robot controller 1 belonging to the same task execution system 50. The sensor 7 may be a self-propelled or flying sensor (including a drone) that moves in the workspace. The sensor 7 may also include a sensor provided on the robot 5 and a sensor provided on another object in the workspace. The sensor 7 may also include a sensor that detects sound in the workspace. In this way, the sensor 7 may include various sensors that detect the state in the workspace and are provided at any location.

[0026] The configuration of the robot control system 100 shown in FIG. 1 is an example, and various changes may be made to the configuration. For example, the robot control function of the robot controller 1 may be integrated into the robot management device 3. In this case, the robot management device 3 performs control required for generating an operation sequence for the robot 5 present in each task execution system 50 and for the robot 5 to execute the operation sequence. In another example, the robot management device 3 may be composed of a plurality of devices. In this case, the plurality of devices constituting the robot management device 3 transmit and receive information required for executing a process assigned in advance between these plurality of devices. In yet another example, the application information storage unit 41 may be composed of one or more external storage devices that perform data communication with the robot controller 1. In this case, the external storage device may be one or more server devices that store the application information storage unit 41 commonly referred to by each task execution system 50.

[0027] (2) Hardware Configuration 2(A) shows the hardware configuration of a robot controller 1 (1A, 1B, ...). The robot controller 1 includes, as hardware, a processor 11, a memory 12, and an interface 13. The processor 11, the memory 12, and the interface 13 are connected via a data bus 10.

[0028] The processor 11 executes a program stored in the memory 12 to function as a controller (arithmetic device) that performs overall control of the robot controller 1. The processor 11 is, for example, a processor such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or a TPU (Tensor Processing Unit). The processor 11 may be composed of multiple processors.

[0029] The memory 12 is composed of various types of volatile and non-volatile memories, such as a RAM (Random Access Memory), a ROM (Read Only Memory), and a flash memory. The memory 12 also stores programs for executing processes executed by the robot controller 1. The memory 12 also functions as an application information storage unit 41. Some of the information stored in the memory 12 may be stored in one or more external storage devices capable of communicating with the robot controller 1, or may be stored in a storage medium that is detachable from the robot controller 1.

[0030] The interface 13 is an interface for electrically connecting the robot controller 1 to other devices. These interfaces may be wireless interfaces such as a network adapter for wirelessly transmitting and receiving data to and from other devices, or may be hardware interfaces for connecting to other devices via cables or the like.

[0031] The hardware configuration of the robot controller 1 is not limited to the configuration shown in Fig. 2(A) For example, the robot controller 1 may be connected to or built-in at least one of a display device, an input device, and a sound output device.

[0032] 2(B) shows a hardware configuration of the operation terminal 2. The operation terminal 2 includes, as hardware, a processor 21, a memory 22, an interface 23, an input unit 24a, a display unit 24b, and a sound output unit 24c. The processor 21, the memory 22, and the interface 23 are connected via a data bus 20. The interface 23 is also connected to the input unit 24a, the display unit 24b, and the sound output unit 24c.

[0033] The processor 21 executes a program stored in the memory 22 to execute a predetermined process. The processor 21 is a processor such as a CPU, a GPU, or a TPU. The processor 21 also controls at least one of the display unit 24b and the sound output unit 24c via the interface 23 based on information received from the task execution system 50 via the interface 23. In this way, the processor 21 presents the operator with information to assist the operator in an operation to be performed. The processor 21 also transmits a signal generated by the input unit 24a as an external input signal Se via the interface 23 to the task execution system 50 that is the source of the support request information Sr. The processor 21 may be composed of multiple processors. The processor 21 is an example of a computer.

[0034] The memory 22 is configured with various types of volatile and non-volatile memories such as RAM, ROM, flash memory, etc. The memory 22 also stores programs for executing the processes executed by the operation terminal 2.

[0035] The interface 23 is an interface for electrically connecting the operation terminal 2 to other devices. These interfaces may be wireless interfaces such as a network adapter for wirelessly transmitting and receiving data to and from other devices, or may be hardware interfaces for connecting to other devices via cables or the like. The interface 23 also performs interface operations for the input unit 24a, the display unit 24b, and the sound output unit 24c.

[0036] The input unit 24a is an interface that accepts user input, and examples of the input include a touch panel, a button, a keyboard, and a voice input device. Depending on the type of the operation terminal 2, the input unit 24a may include an operation unit that accepts user input representing an instruction that directly specifies the operation of the robot 5. This operation unit may be, for example, a robot controller (operation panel) operated by the user in controlling the robot 5 based on an external input, a robot input system that generates an operation instruction for the robot 5 in accordance with the user's movement, or a game controller. The above-mentioned robot controller includes, for example, various buttons for specifying the part of the robot 5 to be moved, the movement, and the like, and an operation bar for specifying the movement direction. The above-mentioned robot input system includes, for example, various sensors (including, for example, a camera, a wearable sensor, and the like) used in motion capture and the like.

[0037] The display unit 24b is, for example, a display, a projector, or the like, and performs display under the control of the processor 21. The display unit 24b may also be a combination of a combiner (a plate-like member having reflectivity and transparency) for realizing virtual reality and a light source that emits display light. The sound output unit 24c is, for example, a speaker, and performs sound output under the control of the processor 21.

[0038] The hardware configuration of the operation terminal 2 is not limited to the configuration shown in Fig. 2(B). For example, at least one of the input unit 24a, the display unit 24b, and the sound output unit 24c may be configured as a separate device electrically connected to the operation terminal 2. The operation terminal 2 may be connected to various devices such as a camera, or may have these built-in.

[0039] 2(C) shows the hardware configuration of the robot management device 3. The robot management device 3 includes, as hardware, a processor 31, a memory 32, and an interface 33. The processor 31, the memory 32, and the interface 33 are connected via a data bus 30.

[0040] The processor 31 executes a program stored in the memory 32 to function as a controller (arithmetic device) that performs overall control of the robot management device 3. The processor 31 is, for example, a processor such as a CPU, a GPU, or a TPU. The processor 31 may be composed of multiple processors. The processor 31 is an example of a computer.

[0041] The memory 32 is composed of various volatile and non-volatile memories such as RAM, ROM, and flash memory. The memory 32 also stores a program for executing the processes executed by the robot controller 1. The memory 32 also stores operation terminal information 38, which is information about the operation terminal 2 capable of accepting support for the task execution system 50, and operator information 39, which is information about the operator operating the operation terminal 2. Details of the operation terminal information 38 and the operator information 39 will be described later. The operation terminal information 38 and the operator information 39 may be information generated based on an administrator's input from an input unit (not shown) connected via the interface 33 and stored in the memory 32, or may be information received from the operation terminal 2 or the like via the interface 33.

[0042] The interface 33 is an interface for electrically connecting the robot management device 3 to other devices. These interfaces may be wireless interfaces such as a network adapter for wirelessly transmitting and receiving data to and from other devices, or may be hardware interfaces for connecting to other devices via cables or the like.

[0043] The hardware configuration of the robot management device 3 is not limited to the configuration shown in Fig. 2(C) For example, the robot management device 3 may be connected to or built-in at least one of a display device, an input device, and a sound output device.

[0044] (3) Data Structure First, the data structure of the application information stored in the application information storage unit 41 will be described.

[0045] 3 shows an example of the data structure of the application information. As shown in FIG. 3, the application information includes abstract state designation information I1, constraint condition information I2, operation limit information I3, subtask information I4, abstract model information I5, and object model information I6.

[0046] The abstract state designation information I1 is information that designates an abstract state that needs to be defined when generating an action sequence. This abstract state is an abstract state of an object in the working space, and is defined as a proposition to be used in a goal formula, which will be described later. For example, the abstract state designation information I1 designates an abstract state that needs to be defined for each type of task.

[0047] The constraint information I2 is information indicating constraint conditions when executing a task. For example, when the task is pick-and-place, the constraint information I2 indicates a constraint condition that the robot 5 (robot arm) must not come into contact with an obstacle, a constraint condition that the robots 5 (robot arms) must not come into contact with each other, etc. Note that the constraint information I2 may be information in which constraint conditions appropriate for each type of task are recorded.

[0048] The motion limit information I3 indicates information related to the motion limit of the robot 5 controlled by the robot controller 1. The motion limit information I3 is, for example, information that specifies the upper limit of the speed, acceleration, or angular velocity of the robot 5. Note that the motion limit information I3 may be information that specifies the motion limit for each movable part or joint of the robot 5.

[0049] The subtask information I4 indicates information on subtasks that are components of an operation sequence. A "subtask" is a task that is broken down into units that the robot 5 can accept, and indicates the subdivided operation of the robot 5. For example, if the task is pick-and-place, the subtask information I4 specifies reaching, which is the movement of the robot arm of the robot 5, and grasping, which is the gripping by the robot arm, as subtasks. The subtask information I4 may indicate information on subtasks that can be used for each type of task.

[0050] The subtask information I4 also includes information about subtasks that require an operation command from an external input (also called an "external input type subtask"). In this case, the subtask information I4 about the external input type subtask includes, for example, identification information of the subtask, flag information for identifying that it is an external input type subtask, and information about the operation content of the robot 5 in the external input type subtask. The subtask information I4 about the external input type subtask may further include text information for requesting an external input to the operation terminal 2, information about the expected work time length, and the like.

[0051] The abstract model information I5 is information on an abstract model that abstracts dynamics in the working space. For example, the abstract model is represented by a model that abstracts real-world dynamics using a hybrid system, as described below. The abstract model information I5 includes information indicating the conditions for switching dynamics in the above-mentioned hybrid system. For example, in the case of pick-and-place, in which the robot 5 grasps an object (also called an "object") to be worked on and moves it to a predetermined position, the switching condition corresponds to a condition that the object cannot be moved unless it is grasped by the robot 5. The abstract model information I5 has information on an abstract model suitable for each type of task.

[0052] The object model information I6 is information on the object model of each object in the working space to be recognized from the signal generated by the sensor 7. The above-mentioned objects include, for example, the robot 5, obstacles, tools handled by the robot 5 and other objects, and working bodies other than the robot 5. The object model information I6 includes, for example, information required for the robot controller 1 to recognize the type, position, posture, and currently executed operation of each of the above-mentioned objects, and three-dimensional shape information such as CAD (Computer Aided Design) data for recognizing the three-dimensional shape of each object. The former information includes parameters of an inference device obtained by learning a learning model in machine learning such as a neural network. For example, when an image is input, this inference device is trained in advance to output the type, position, posture, and the like of an object that is a subject in the image.

[0053] In addition to the above-mentioned information, the application information storage unit 41 may store various information related to the generation process of the operation sequence and the display process required to accept an operation for generating the external input signal Se.

[0054] Fig. 4(A) is an example of the data structure of the operating terminal information 38. As shown in Fig. 4(A), the operating terminal information 38 exists for each operating terminal 2, and mainly includes a terminal ID 381, terminal type information 382, ​​address information 383, and a corresponding operator ID 384.

[0055] Terminal ID 381 is the terminal ID of the corresponding operating terminal 2. Terminal ID 381 may be any identification information capable of identifying the operating terminal 2. Terminal type information 382 is information indicating the type of the corresponding operating terminal 2. The types of operating terminals 2 are classified, for example, based on differences in the operation modes they accept.

[0056] The address information 383 is communication information required to communicate with the corresponding operating terminal 2, and is, for example, information related to a communication address (including an IP address, etc.) required for communication according to a predetermined communication protocol. The address information 383 is used, for example, in connection control for establishing a communication connection between the target operating terminal 2 and the task execution system 50. The corresponding operator ID 384 is identification information (operator ID) of the operator who operates the corresponding operating terminal 2. The corresponding operator ID 384 may indicate the operator IDs of a plurality of operators.

[0057] Fig. 4(B) is an example of a data structure of the operator information 39. As shown in Fig. 4(B), the operator information 39 exists for each operator registered as a person who provides support to the task execution system 50, and mainly includes an operator ID 391, skill information 392, operation record information 393, status management information 394, and an operable terminal ID 395.

[0058] The operator ID 391 is the operator ID of the corresponding operator. The skill information 392 is information indicating the operation skill (proficiency) of the corresponding operator using the operation terminal 2. The skill information 392 may indicate the operation skill level of the operator for each type of the operation terminal 2 operated. The operation record information 393 is record information of the operator regarding operations in response to a support request from the task execution system 50. The operation record information 393 may indicate the operation record of the operator for each type of the operation terminal 2 operated.

[0059] The operation record information 393 includes, for example, the number of operations performed in response to a support request from the task execution system 50, the period of registration as an operator (years of experience), the number or ratio of successes and failures in operations, etc. Here, "success or failure" is determined based on whether or not the error in the task execution system 50 that made the request has been resolved by the supply of the external input signal Se from the operation terminal 2, in the case of a support request from the task execution system 50 based on the occurrence of an error, for example. Note that in the case of a case other than the occurrence of an error, "success or failure" may be determined based on, for example, whether or not the task has been normally completed after the supply of the external input signal Se from the operation terminal 2.

[0060] The operation result information 393 may include an operation history generated for each operation in response to a support request from the task execution system 50. In this case, various information such as information on the task for which a support request was made from the task execution system 50, information on the task execution system 50 that requested the support, information on the date and time when the operation was performed, and the duration of the operation are recorded as an operation history in the operation result information 393. For example, every time a support request is made from the task execution system 50 and support in response to the request is performed, the processor 31 updates the operation result information 393 based on information received from the task execution system 50 that requested the support.

[0061] The status management information 394 is information related to status management of the corresponding operator, and may be, for example, schedule information indicating the date, time or time period when the operator is available for operation, or may be information indicating whether the operator is currently available (for example, present). The processor 31 may update the status management information 394 at a predetermined timing by, for example, receiving schedule information and presence / absence information of the operator from another system that manages the schedule of each operator, or by accepting manual input related to the operator's status. In this way, each piece of information, the operation terminal information 38 and the operator information 39, is updated at a required timing.

[0062] The operable terminal ID 395 is a terminal ID (terminal ID 381 in FIG. 4A) that can be operated by the corresponding operator. Note that the operable terminal ID 395 may be the terminal ID of one operating terminal 2 or the terminal IDs of multiple operating terminals 2.

[0063] The data structures of the operation terminal information 38 and the operator information 39 are not limited to those shown in Fig. 4(A) and Fig. 4(B). For example, the operation terminal information 38 may further include information for managing the state of the operation terminal 2, which corresponds to the state management information 394 included in the operator information 39. In addition, when the operation terminal 2 and the operator correspond one-to-one, the operation terminal information 38 and the operator information 39 may be integrated into one of them.

[0064] (4) Functional Blocks FIG. 5 is an example of a functional block showing an overview of the processing of the robot control system 100. The processor 11 of the robot controller 1 functionally has an output control unit 15, an operation sequence generation unit 16, a robot control unit 17, and a switching determination unit 18. The processor 21 of the operation terminal 2 functionally has an information presentation unit 25 and an external control unit 26. The processor 31 of the robot management device 3 functionally has an external input necessity determination unit 35, an operation terminal determination unit 36, and a connection control unit 37. In the functional block shown in FIG. 5, the blocks through which data is exchanged are connected by solid lines, but the combination of blocks through which data is exchanged and the data exchanged are not limited to FIG. 5. The same applies to other functional block diagrams described later. In FIG. 5, an example of the operation mode of the operator using the operation terminal 2 is illustrated in a balloon 60.

[0065] First, the functions of the robot controller 1 will be described. The robot controller 1 executes control of the robot 5 based on the generated operation sequence, and when it is determined that support from the operation terminal 2 is necessary, it transmits support request information Sr to the robot management device 3. As a result, even when the robot controller 1 cannot handle the task with automatic control alone, it smoothly switches the control mode of the robot 5 to control based on an external input signal Se (also called "external input control") to carry out the task. Below, the functional components of the robot controller 1 will be described.

[0066] The output control unit 15 performs processing related to the transmission of support request information Sr and the reception of an external input signal Se via the interface 13. In this case, when a switching command “Sw” to external input control is supplied from the switching determination unit 18, the output control unit 15 transmits support request information Sr requesting a necessary external input to the operation terminal 2.

[0067] Here, the support request information Sr includes information on the task (subtask) for which support is required. Specifically, the support request information Sr includes, for example, date and time information indicating the date and time when the request became necessary, type information of the robot 5 to be supported, task identification information, identification information of the subtask to be supported, the expected work time length of the subtask, the necessary operation content of the robot 5, and error information on the error when an error occurs. The error information is an error code indicating the type of error, etc. The error information may include information indicating the situation when the error occurs (for example, video information), etc. When a communication connection between the operation terminal 2 and the robot controller 1 is established based on the connection control by the robot management device 3, the output control unit 15 transmits information required for displaying the operation screen of the operator of the operation terminal 2 (also referred to as "operation screen information") to the operation terminal 2. When the output control unit 15 receives an external input signal Se from the operation terminal 2, the output control unit 15 supplies the external input signal Se to the robot control unit 17.

[0068] The movement sequence generation unit 16 generates a movement sequence "Sv" of the robot 5 required to complete a specified task, based on the signal output by the sensor 7 and the application information. The movement sequence Sv corresponds to a sequence of subtasks (subtask sequence) that the robot 5 should execute to accomplish the task, and specifies a series of movements of the robot 5. The movement sequence generation unit 16 then supplies the generated movement sequence Sv to the robot control unit 17 and the switching determination unit 18. Here, the movement sequence Sv includes information indicating the execution order and execution timing of each subtask.

[0069] The robot control unit 17 controls the operation of the robot 5 by supplying a control signal to the robot 5 via the interface 13. After receiving the operation sequence Sv from the operation sequence generation unit 16, the robot control unit 17 controls the robot 5. In this case, the robot control unit 17 executes position control or torque control of the joints of the robot 5 to realize the operation sequence Sv by transmitting a control signal to the robot 5. Then, the robot control unit 17 switches the control mode of the robot 5 to external input control based on a switching command Sw supplied from the switching determination unit 18.

[0070] In the external input control, the robot control unit 17 receives an external input signal Se generated by the operation terminal 2 via the interface 13. This external input signal Se includes, for example, information that specifies a specific operation of the robot 5 (for example, information corresponding to a control input that directly determines the operation of the robot 5). The robot control unit 17 then generates a control signal based on the received external input signal Se and transmits the generated control signal to the robot 5, thereby controlling the robot 5. The control signal generated by the robot control unit 17 in the external input control is, for example, a signal obtained by converting the external input signal Se into a data format that can be accepted by the robot 5. Note that, when such a conversion process is performed in the robot 5, the robot control unit 17 may supply the external input signal Se as it is to the robot 5 as a control signal.

[0071] Moreover, the external input signal Se may be information that assists the task execution system 50 in recognizing the working state, instead of information that specifies the specific operation of the robot 5. For example, when the task execution system 50 is unable to recognize an object to be picked and placed, the external input signal Se may be information indicating the position of the object. In this case, after establishing a communication connection with the task execution system 50, the operation terminal 2 receives an image of the working environment from the operation terminal 2, and receives an operation by the operator to specify the object from the image, thereby generating an external input signal Se that specifies the area of ​​the object. Then, the robot control unit 17 recognizes the object based on the external input signal Se, and resumes the robot control based on the interrupted operation sequence.

[0072] Note that the robot 5 may have a function equivalent to the robot control unit 17 instead of the robot controller 1. In this case, the robot 5 performs an operation based on the operation sequence Sv generated by the operation sequence generation unit 16, the switching command Sw generated by the switching determination unit 18, and the external input signal Se.

[0073] The switching determination unit 18 determines whether or not it is necessary to switch the control mode to the external input control based on the operation sequence Sv and the like. For example, the switching determination unit 18 determines that it is necessary to switch the control mode to the external input control when it is time to execute an external input type subtask incorporated in the operation sequence Sv. In another example, the switching determination unit 18 determines that some abnormality has occurred when the generated operation sequence Sv is not executed as planned, and determines that it is necessary to switch the control of the robot 5 to the external input control. In this case, for example, the switching determination unit 18 determines that some abnormality has occurred when it detects that a time or / spatial deviation from the plan based on the operation sequence Sv has occurred. Note that the switching determination unit 18 may detect the occurrence of an abnormality by receiving an error signal or the like from the robot 5, or by analyzing a sensor signal (such as an image of the working space) output by the sensor 7. Then, when the switching determination unit 18 determines that it is necessary to switch the control mode to the external input control, it supplies a switching command Sw to the output control unit 15 and the robot control unit 17 to instruct switching of the control mode to the external input control.

[0074] Next, the functional blocks of the operation terminal 2 will be described.

[0075] When a communication connection is established between the task execution system 50 that has requested assistance and the operation terminal 2 based on the connection control by the robot management device 3, the information presentation unit 25 displays an operation screen on the display unit 24b based on operation screen information and the like supplied from the task execution system 50. On the operation screen, for example, information on the operation content of the robot 5 that should be specified by external input is displayed. In this way, the information presentation unit 25 presents the information necessary for the operation to the operator. In this case, the information presentation unit 25 may output audio guidance necessary for the operation by controlling the sound output unit 24c.

[0076] The external control unit 26 acquires a signal output by the input unit 24a in response to an operation by an operator referring to the operation screen as an external input signal Se, and transmits the acquired external input signal Se to the task execution system 50 that has requested the support via the interface 23. In this case, the external control unit 27 acquires the external input signal Se in real time in response to the operation of the operator and transmits the external input signal Se, for example.

[0077] Next, the functional blocks of the robot management device 3 will be described.

[0078] The external input necessity determination unit 35 determines whether support via external input is required. In this embodiment, the external input necessity determination unit 35 determines that support via external input is required when support request information Sr is received from the task execution system 50 via the interface 33. Then, the external input necessity determination unit 35 supplies the support request information Sr to the operation terminal determination unit 36.

[0079] The operating terminal determination unit 36 ​​determines the operating terminal 2 and the operator that will support the task execution system 50 that is the sender of the support request information Sr, based on the support request information Sr and the operating terminal information 38 (and the operator information 39). A specific example of this determination method will be described later.

[0080] The connection control unit 37 performs connection control to establish a communication connection between the target operating terminal 2 determined by the operating terminal determination unit 36 ​​and the task execution system 50 that is the support request source. In this case, the connection control unit 37 transmits the communication address of the other to at least one of the target operating terminal 2 or the task execution system 50 so that the target operating terminal 2 and the task execution system 50 can directly establish a communication connection between them. In another example, the connection control unit 37 establishes a communication connection between the target operating terminal 2 and the task execution system 50 that is required for a transfer process of data exchanged between the target operating terminal 2 and the task execution system 50 during the external input control. In this case, the connection control unit 37 performs a process of receiving an external input signal Se generated by the operating terminal 2 and transferring the received external input signal Se to the task execution system 50 (more specifically, the robot controller 1) during the external input control, and a process of receiving operation screen information generated by the task execution system 50 and transferring the received operation screen information to the operating terminal 2.

[0081] Here, each of the components of the external input necessity determination unit 35, the operation terminal determination unit 36, and the connection control unit 37 can be realized, for example, by the processor 31 executing a program. Also, each component may be realized by recording a necessary program in an arbitrary non-volatile storage medium and installing it as necessary. At least a part of these components may be realized by any combination of hardware, firmware, and software, without being limited to being realized by software by a program. Also, at least a part of these components may be realized by using an integrated circuit that is programmable by a user, such as an FPGA (Field-Programmable Gate Array) or a microcontroller. In this case, the program composed of each of the above components may be realized using this integrated circuit. Also, at least a part of each component may be configured by an ASSP (Application Specific Standard Produce), an ASIC (Application Specific Integrated Circuit), or a quantum computer control chip. In this way, each component may be realized by various hardware. The above is the same in other embodiments described later. Furthermore, each of these components may be realized by the cooperation of multiple computers, for example, using cloud computing technology. The same applies to the components of the robot controller 1 and the operation terminal 2 shown in FIG.

[0082] (4) Details of the processing of the operation terminal determination part Next, a method for determining the target operating terminal 2 by the operating terminal determination unit 36 ​​will be described. In summary, the operating terminal determination unit 36 ​​refers to information on the task included in the support request information Sr and at least the terminal type information 382 of the operating terminal information 38 to determine the target operating terminal 2. A specific example of this will be described below.

[0083] In the first example, the operation terminal determination unit 36 ​​determines the operation terminal 2 to perform the assist operation based on the type of the robot 5 of the requesting task execution system 50 and the terminal type information 382 of the operation terminal information 38. In general, the user interface that is easy to operate varies depending on the type of robot. Therefore, the operation terminal determination unit 36 ​​according to the first example selects the operation terminal 2 having a user interface suitable for the assist operation of the robot 5 of the requesting task execution system 50. For example, when the target robot 5 is a robot arm, the operation terminal determination unit 36 ​​selects the operation terminal 2 having a game controller as a user interface. In another example, when the target robot 5 is a humanoid robot, the operation terminal determination unit 36 ​​selects the operation terminal 2 that can be operated by VR.

[0084] Here, a specific process of the first example will be explained in more detail. For example, the memory 32 stores information (also called "robot-operating terminal correspondence information") that associates the type of operating terminal 2 suitable for the assisting operation with each type of robot 5. The operating terminal determination unit 36 ​​then refers to the robot-operating terminal correspondence information and recognizes the type of operating terminal 2 suitable for the assisting operation from the type of robot 5 indicated by the support request information Sr. The operating terminal determination unit 36 ​​then identifies the operating terminal 2 corresponding to the type based on the terminal type information 382 included in the operating terminal information 38, and determines the identified operating terminal 2 as the target operating terminal 2.

[0085] Here, if there are multiple operating terminals 2 selected based on the first example, the operating terminal determination unit 36 ​​may determine the target operating terminal 2 based on at least one of the second to fourth examples described below, or may determine the target operating terminal 2 by random sampling.

[0086] In the second example, instead of or in addition to the first example, the operating terminal determination unit 36 ​​determines the target operating terminal 2 based on the error information included in the support request information Sr and the terminal type information 382 of the operating terminal information 38. As a result, the operating terminal determination unit 36 ​​suitably determines the operating terminal 2 that can easily handle the occurred error as the target operating terminal 2. For example, when the error information indicates that gripping has failed in pick-and-place, the operating terminal determination unit 36 ​​selects the operating terminal 2 having a game controller as a user interface. In another example, when the error information indicates that acquisition of product information has failed, the operating terminal determination unit 36 ​​selects the operating terminal 2 that is a personal computer.

[0087] Here, a specific process of the second example will be explained in more detail. For example, the memory 32 stores information (also called "error-operating terminal correspondence information") that associates each type of error that may occur with the type of operating terminal 2 suitable for the assisting operation. The operating terminal determination unit 36 ​​then refers to the error-operating terminal correspondence information and recognizes the type of operating terminal 2 suitable for the assisting operation from the type of error indicated by the support request information Sr. The operating terminal determination unit 36 ​​then identifies an operating terminal 2 that corresponds to the type based on the terminal type information 382 included in the operating terminal information 38, and determines the identified operating terminal 2 as the target operating terminal 2.

[0088] Here, if there are multiple operating terminals 2 selected based on the second example, the operating terminal determination unit 36 ​​may determine the target operating terminal 2 based on at least one of the first example and the third and fourth examples described below, or may determine the target operating terminal 2 by random sampling.

[0089] Furthermore, the operating terminal determination unit 36 ​​may perform processing to determine the target operating terminal 2 by further using information included in the operating terminal information 38 or the operator information 39 other than the terminal type information 382, ​​in addition to the terminal type information 382. Specific examples of this will be described as a third example and a fourth example. The following third example and fourth example are executed in combination with, for example, the above-mentioned first example or second example.

[0090] In the third example, the operating terminal determination unit 36 ​​determines an operator based on the type of error indicated by the support request information Sr. In this case, for example, information (also referred to as "error-operator correspondence information") that defines the conditions of the operator's track record and / or skills required for the support operation for each type of error is stored in the memory 32. Then, the operating terminal determination unit 36 ​​refers to the error-operator correspondence information and recognizes the conditions of the operator's track record and / or skills required for the support operation from the type of error indicated by the support request information Sr. Then, the operating terminal determination unit 36 ​​specifies an operator who satisfies the track record and / or skill conditions by referring to the skill information 392 and / or operation track record information 393 included in the operator information 39, and determines the operating terminal 2 used by the specified operator as the target operating terminal 2.

[0091] In a fourth example, the operating terminal determination unit 36 ​​determines, based on the state management information 394, the operating terminal 2 used by an operator who is in a state capable of executing an assist operation, as the target operating terminal 2. In this case, the operating terminal determination unit 36 ​​refers to the state management information 394, identifies an operator who is currently available, and determines the operating terminal 2 used by the identified operator as the target operating terminal 2. Thereby, for example, in a case where the operators include an overseas resident and the task execution system 50 is constantly operating, the operating terminal determination unit 36 ​​can appropriately select an available operator (i.e., an operator who is in an area during business hours) and determine the operating terminal 2 used by the operator as the target operating terminal 2.

[0092] Next, a supplementary explanation will be given for the case where the target operating terminal 2 cannot be determined and external input control cannot be started. For example, when an attempt is made to select a target operating terminal 2 based on at least one of the above-mentioned first to fourth examples, the case where the corresponding operating terminal 2 does not exist corresponds to such a case.

[0093] In this case, the operation terminal determination unit 36 ​​transmits, for example, information to the task execution system 50 of the support requester to prompt the robot to control the robot without external input control, such as autonomous recovery. In another example, the operation terminal determination unit 36 ​​accumulates support request information Sr for which support has not been executed, and places the task execution system 50 of the support requester in a standby state until support can be executed. In this case, the operation terminal determination unit 36 ​​may process the support request information Sr in order according to a FIFO (First In, First Out) method, or may determine the priority of each accumulated support request information Sr and process the support request information Sr in descending order of priority. In this case, the operation terminal determination unit 36 ​​determines the priority of each support request information Sr based on, for example, the task type or task priority information included in the support request information Sr, and / or the urgency of support identified from the work progress status in the task execution system 50 of the support requester.

[0094] (5) Details of the operation sequence generation part 6 is an example of a functional block diagram showing the functional configuration of the operation sequence generation unit 16. The operation sequence generation unit 16 functionally includes an abstract state setting unit 161, a target logical formula generation unit 162, a time step logical formula generation unit 163, an abstract model generation unit 164, a control input generation unit 165, and a subtask sequence generation unit 166.

[0095] The abstract state setting unit 161 sets an abstract state in the work space based on the sensor signal supplied from the sensor 7, the abstract state designation information I1, and the object model information I6. In this case, the abstract state setting unit 161 recognizes an object in the work space that needs to be considered when executing a task, and generates a recognition result "Im" for the object. Then, based on the recognition result Im, the abstract state setting unit 161 defines a proposition for expressing each abstract state that needs to be considered when executing a task as a logical formula. The abstract state setting unit 161 supplies information indicating the set abstract state (also referred to as "abstract state setting information IS") to the target logical formula generation unit 162.

[0096] The target logical formula generation unit 162 converts the task into a logical formula of temporal logic (also called a "target logical formula Ltag") that represents a final achieved state based on the abstract state setting information IS. In this case, the target logical formula generation unit 162 adds a constraint condition that should be satisfied in the execution of the task to the target logical formula Ltag by referring to the constraint condition information I2 from the application information storage unit 41. Then, the target logical formula generation unit 162 supplies the generated target logical formula Ltag to the time-step logical formula generation unit 163.

[0097] The time-step logical formula generation unit 163 converts the target logical formula Ltag supplied from the target logical formula generation unit 162 into a logical formula (also called a "time-step logical formula Lts") that represents a state at each time step. Then, the time-step logical formula generation unit 163 supplies the generated time-step logical formula Lts to the control input generation unit 165.

[0098] The abstract model generation unit 164 generates an abstract model "Σ" that abstracts the real dynamics in the working space, based on the abstract model information I5 stored in the application information storage unit 41 and the recognition result Im supplied from the abstract state setting unit 161. In this case, the abstract model Σ may be, for example, a hybrid system in which the target dynamics is a mixture of continuous dynamics and discrete dynamics. The abstract model generation unit 164 supplies the generated abstract model Σ to the control input generation unit 165.

[0099] The control input generation unit 165 determines a control input to the robot 5 for each time step that satisfies the time step logical formula Lts supplied from the time step logical formula generation unit 163 and the abstract model Σ supplied from the abstract model generation unit 164 and optimizes an evaluation function (for example, a function representing the amount of energy consumed by the robot). Then, the control input generation unit 165 supplies information indicating the control input to the robot 5 for each time step (also referred to as "control input information Icn") to the subtask sequence generation unit 166.

[0100] The subtask sequence generation unit 166 generates an operation sequence Sv, which is a sequence of subtasks, based on the control input information Icn supplied from the control input generation unit 165 and the subtask information I4 stored in the application information storage unit 41, and supplies the operation sequence Sv to the robot control unit 17 and the switching determination unit 18.

[0101] (6) Operation screen 7 is a first display example of an operation screen displayed by the operation terminal 2. The information presenting unit 25 of the operation terminal 2 controls to display the operation screen shown in FIG. 7 by receiving operation screen information from the robot controller 1 of the task execution system 50 that is the sender of the support request information Sr. Here, since it is time to execute an external input type subtask (i.e., a work process that requires external input), the robot controller 1 transmits the support request information Sr to the robot management device 3 in order to receive an external input signal Se required for executing the external input type subtask, and then establishes a communication connection with the operation terminal 2 based on the connection control by the robot management device 3. The operation screen shown in FIG. 7 mainly has a workspace display field 70 and an action content display area 73.

[0102] Here, the work space display field 70 displays a work space image, which is a photographed image of the current work space or a CAD image that typically represents the current work space, and the operation content display area 73 displays the contents of the operation required to cause the robot 5 to operate by external input. Note that, as an example, the target subtask here is a subtask of moving an object that is adjacent to an obstacle and cannot be directly grasped by the robot 5 to a position where it can be grasped, and grabbing it.

[0103] 7, the operation terminal 2 displays, on the operation content display area 73, a guide sentence instructing the operation content to be executed by the robot 5 (here, moving the target object to a predetermined position and grabbing it with the first arm). Furthermore, the operation terminal 2 displays, on the work space image displayed in the work space display field 70, a thick circular frame 71 surrounding the target object to be worked on, a dashed circular frame 72 indicating the destination of the target object, and the names of the arms of the robot 5 (first arm, second arm). By adding such displays to the work space display field 70, the operation terminal 2 allows the operator who refers to the text in the operation content display area 73 to easily recognize the robot arm required for the work, the target object to be worked on, and its destination.

[0104] Here, the action content of the robot 5 displayed in the action content display area 73 satisfies the condition (also called the "sequence transition condition") for transitioning to the next subtask of the target subtask. The sequence transition condition corresponds to the condition indicating the end state of the target subtask (or the start state of the next subtask) assumed in the generated action sequence Sv. The sequence transition condition in the example of FIG. 7 indicates that the first arm is in a state where it has grasped an object at a predetermined position. In this way, the operation terminal 2 can appropriately support the external input required for a smooth transition to the next subtask by displaying the guide sentence instructing the action required to satisfy the sequence transition condition in the action content display area 73.

[0105] As described above, according to the operation screen shown in FIG. 7, it is possible to accept appropriate operations by the operator when an external input type subtask that requires external input control is being executed.

[0106] Fig. 8 shows a second display example of the operation screen. The information presenter 25 of the operation terminal 2 receives operation screen information from the robot controller 1 of the task execution system 50 that has transmitted the support request information Sr, thereby controlling to display the operation screen shown in Fig. 8. The operation screen shown in Fig. 8 mainly has a work space display field 70 and an action content display area 73.

[0107] In the second display example, due to some accident, one object rolls behind an obstacle, and the object cannot be directly grasped by the robot arm. In this case, the robot controller 1 detects a time or / spatial deviation from the plan based on the operation sequence Sv, and determines that it is inappropriate to continue autonomous robot control, transmits support request information Sr to the robot management device 3, and then transmits operation screen information to the operation terminal 2 with which a communication connection has been established.

[0108] 8, the information presenter 25 displays in the operation content display area 73 that an abnormality has occurred regarding the pick-and-place of the object and that an external input is required to move the object to the goal point. In addition, the output control unit 15 displays, on the image displayed in the work space display field 70, a thick round frame 71 surrounding the object to be worked on and the names of each arm of the robot 5 (first arm, second arm).

[0109] In this way, according to the operation screen shown in Fig. 8, when external input control becomes necessary due to the occurrence of an abnormality, it is possible to accept an appropriate operation by the operator. Note that the operation terminal 2 may output a guidance voice instructing an operation for generating a necessary external input signal Se together with displaying the operation screens of Figs. 7 and 8.

[0110] (7) Processing flow FIG. 9 is an example of a flowchart showing an outline of the processing executed by the robot management device 3 in the first embodiment.

[0111] First, the external input necessity determination unit 35 of the robot management device 3 determines whether or not support request information Sr has been received (step S11). If the external input necessity determination unit 35 determines that support request information Sr has been received (step S11; Yes), the process proceeds to step S12. Then, the operation terminal determination unit 36 ​​of the robot management device 3 determines the target operation terminal 2 based on the support request information Sr and the operation terminal information 38, etc. (step S12). In this case, the operation terminal determination unit 36 ​​may further refer to the operator information 39 in addition to the operation terminal information 38, and perform a process of determining an operator suitable for the support operation.

[0112] Then, the connection control unit 37 of the robot management device 3 performs connection control to establish a communication connection between the target operating terminal 2 and the requesting task execution system 50 (step S13). As a result, the connection control unit 37 establishes a communication connection between the determined target operating terminal 2 and the requesting task execution system 50. Thereafter, the determined target operating terminal 2 and the requesting task execution system 50 exchange operation screen information, an external input signal Se, and the like.

[0113] After execution of step S13, or if support request information Sr has not been received in step S11 (step S11; No), the robot management device 3 determines whether or not to end the processing of the flowchart (step S14). For example, the robot management device 3 determines that the processing of the flowchart should be ended if the operating hours of the robot control system 100 have expired, or if other predetermined end conditions are satisfied. Then, if the processing of the flowchart should be ended (step S14; Yes), the robot management device 3 ends the processing of the flowchart. On the other hand, if the processing of the flowchart should not be ended (step S14; No), the robot management device 3 returns the processing to step S11.

[0114] (8) Variations The block configuration of the operation sequence generating unit 16 shown in FIG. 6 is an example, and various modifications may be made.

[0115] For example, information on candidates for sequences of operations to be commanded to the robot 5 may be stored in advance in the storage device 4, and the operation sequence generation unit 16 may execute an optimization process of the control input generation unit 165 based on the information. In this way, the operation sequence generation unit 16 selects the optimal candidate and determines the control input of the robot 5. In this case, the operation sequence generation unit 16 may not have functions corresponding to the abstract state setting unit 161, the target logical formula generation unit 162, and the time step logical formula generation unit 163 in generating the operation sequence Sv. In this way, information on the execution results of some of the functional blocks of the operation sequence generation unit 16 shown in FIG. 6 may be stored in advance in the application information storage unit 41.

[0116] In another example, the application information may include design information such as a flowchart for designing an operation sequence Sv corresponding to a task, and the operation sequence generating unit 16 may generate the operation sequence Sv by referring to the design information. Note that a specific example of executing a task based on a task sequence designed in advance is disclosed in, for example, JP 2017-39170 A.

[0117] <Second embodiment> Fig. 10 shows a schematic configuration diagram of a robot management device 3X in the second embodiment. The robot management device 3X mainly has an external input necessity determination means 35X and an operation terminal determination means 36X. The robot management device 3X may be composed of multiple devices. The robot management device 3X may be, for example, the robot management device 3 in the first embodiment (including the case where some of the functions of the robot controller 1 are incorporated).

[0118] The external input necessity determination means 35X determines whether or not control based on an external input of the robot executing the task (external input control) is necessary. The external input necessity determination means 35X can be, for example, the external input necessity determination unit 35 in the first embodiment.

[0119] When control based on an external input is required, the operating terminal determination means 36X determines an operating terminal that will generate the external input based on operating terminal information including information about the types of multiple operating terminals that are candidates for generating the external input, and information about the task. The "information about the task" includes various information included in the support request information Sr in the first embodiment, such as information about the type of the task, information about the robot that executes the task, and information about an error that has occurred in the task. The operating terminal determination means 36X can be, for example, the operating terminal determination unit 36 ​​in the first embodiment.

[0120] 11 is an example of a flowchart in the second embodiment. The external input necessity determination means 35X determines whether or not control based on an external input of a robot executing a task is necessary (step S21). When control based on an external input is necessary (step S22; Yes), the operation terminal determination means 36X determines an operation terminal that generates an external input based on operation terminal information including information on the types of multiple operation terminals that are candidates for generating the external input and information on the task (step S23). On the other hand, when control based on an external input is not necessary (step S22; No), the operation terminal determination means 36X does not perform the process of step S23 and ends the process of the flowchart.

[0121] According to the second embodiment, when there is a robot that needs to be controlled based on an external input, the robot management device 3X can suitably determine an operation terminal that generates the external input.

[0122] In each of the above-described embodiments, the program can be stored using various types of non-transitory computer readable media and can be supplied to a computer, such as a processor. The non-transitory computer readable medium includes various types of tangible storage media. Examples of the non-transitory computer readable medium include magnetic storage media (e.g., flexible disks, magnetic tapes, hard disk drives), magneto-optical storage media (e.g., magneto-optical disks), CD-ROM (Read Only Memory), CD-R, CD-R / W, and semiconductor memory (e.g., mask ROM, PROM (Programmable ROM), EPROM (Erasable PROM), flash ROM, and RAM (Random Access Memory)). The program may also be supplied to a computer by various types of transitory computer readable media. Examples of the transitory computer readable medium include electric signals, optical signals, and electromagnetic waves. The transitory computer readable medium can supply the program to a computer via a wired communication path such as an electric wire and an optical fiber, or via a wireless communication path.

[0123] In addition, some or all of the above embodiments may be described as, but are not limited to, the following supplementary notes.

[0124] [Appendix 1] an external input necessity determination means for determining whether or not control based on an external input of a robot executing a task is necessary; an operation terminal determination means for determining, when control based on the external input is required, the operation terminal that generates the external input based on operation terminal information including information on the types of a plurality of operation terminals that are candidates for generating the external input and information on the task; A robot management device having the above configuration. [Appendix 2] A robot management device as described in Appendix 1, further comprising a connection control means for controlling the establishment of a communication connection between the operation terminal determined by the operation terminal determination means and the robot or a robot controller that controls the robot. [Appendix 3] The information about the task includes error information about an error that occurred in the task; The robot management device according to claim 1 or 2, wherein the operation terminal determination means determines the operation terminal that generates the external input based on the operation terminal information and the error information. [Appendix 4] The information about the task includes type information of the robot, The robot management device according to any one of claims 1 to 3, wherein the operation terminal determination means determines the operation terminal that generates the external input based on the operation terminal information and the robot type information. [Appendix 5] The robot management device according to any one of appendices 1 to 4, wherein the operation terminal determination means determines the operation terminal that generates the external input based on operator information, which is information about an operator of the operation terminal, the operation terminal information, and information about the task. [Appendix 6] The operator information includes information regarding the skill or operation record of the operator, The robot management device of claim 5, wherein the operation terminal determination means determines the operation terminal used by an operator who satisfies the required skills or operation experience determined based on information about the task as the operation terminal that generates the external input. [Appendix 7] the operator information includes state management information which is information regarding state management of the operator, The robot management device according to claim 5 or 6, wherein the operation terminal determination means determines, based on the state management information, the operation terminal used by an operator that is in a state capable of executing operations related to the external input as the operation terminal that generates the external input. [Appendix 8] The robot management device according to any one of appendices 1 to 7, wherein the external input necessity determination means determines that control based on the external input is necessary when support request information including information regarding the task is received from the robot or a robot controller that controls the robot. [Appendix 9] The robot management device according to any one of appendices 1 to 8, wherein the external input necessity determination means determines that control based on the external input is necessary when an error occurs in the execution of the task by the robot, or when the external input is required for a work process. [Appendix 10] The computer Determining whether or not control is required based on an external input of the robot performing the task; When control based on the external input is necessary, the operation terminal that generates the external input is determined based on operation terminal information including information on the types of a plurality of operation terminals that are candidates for generating the external input and information on the task. Control methods. [Appendix 11] Determining whether or not control is required based on an external input of the robot performing the task; When control based on the external input is necessary, a process of determining the operation terminal that will generate the external input based on operation terminal information including information on the types of a plurality of operation terminals that are candidates for generating the external input and information on the task. A storage medium that stores a program to be executed by a computer.

[0125] Although the present invention has been described above with reference to the embodiments, the present invention is not limited to the above-mentioned embodiments. Various modifications that can be understood by a person skilled in the art can be made to the configuration and details of the present invention within the scope of the present invention. In other words, the present invention naturally includes various modifications and corrections that a person skilled in the art could make in accordance with the entire disclosure, including the claims, and the technical ideas. In addition, the disclosures of the above cited patent documents and the like are incorporated herein by reference. [Explanation of symbols]

[0126] 1, 1A, 1B Robot Controller 2, 2A, 2B Operation terminal 3. 3X Robot Management Device 5. Robot 7 Sensors 41 Application information storage unit 100 Robot Control System

Claims

1. an external input necessity determination means for determining whether or not control based on an external input of a robot executing a task is necessary; an operation terminal determination means for determining, when control based on the external input is required, the operation terminal that generates the external input based on operation terminal information including information on types of a plurality of operation terminals that are candidates for generating the external input and information on the task; the external input necessity determination means determines that control based on the external input is necessary when support request information including information related to the task is received from the robot or a robot controller that controls the robot; When the operating terminal determination means is unable to determine the operating terminal, it accumulates the support request information for which support has not yet been executed, determines the priority of each piece of support request information based on the urgency of the support, and when the support becomes executable, processes the support request information in order of highest priority.

2. 2. The robot management device according to claim 1, further comprising a connection control means for controlling the establishment of a communication connection between the operation terminal determined by the operation terminal determination means and the robot or a robot controller that controls the robot.

3. The information about the task includes error information, which is information about an error that has occurred in the task, information indicating a type of error, or information indicating a situation at the time of the error, and includes error information, The robot management device according to claim 1 , wherein the operation terminal determination means determines the operation terminal that generates the external input based on the operation terminal information and the error information.

4. The information about the task includes type information of the robot, The robot management device according to any one of claims 1 to 3, wherein the operation terminal determination means determines the operation terminal that generates the external input based on the operation terminal information and the robot type information.

5. A robot management device according to any one of claims 1 to 4, wherein the operation terminal determination means determines the operation terminal that generates the external input based on operator information, which is information about the operator of the operation terminal, the operation terminal information, and information about the task.

6. The operator information includes information regarding the skill or operation record of the operator, The robot management device according to claim 5 , wherein the operation terminal determination means determines the operation terminal used by an operator who satisfies the required skills or operation experience determined based on information about the task as the operation terminal that generates the external input.

7. the operator information includes state management information which is information regarding state management of the operator, The robot management device according to claim 5 or 6, wherein the operation terminal determination means determines, based on the state management information, the operation terminal used by an operator that is in a state capable of executing operations related to the external input as the operation terminal that generates the external input.

8. The computer determining whether or not control based on an external input of a robot executing a task is required, and when assistance request information including information related to the task is received from the robot or a robot controller controlling the robot, determining that control based on the external input is required; When control based on the external input is necessary, the operation terminal that will generate the external input is determined based on operation terminal information including information on the types of a plurality of operation terminals that are candidates for generating the external input and information on the task, and when the operation terminal cannot be determined, the support request information for which support has not yet been executed is accumulated, a priority of each piece of support request information is determined based on the urgency of the support, and when the support becomes executable, the support request information is processed in descending order of priority. Control methods.

9. determining whether or not control based on an external input of a robot executing a task is required, and when assistance request information including information related to the task is received from the robot or a robot controller controlling the robot, determining that control based on the external input is required; when control based on the external input is necessary, determining an operation terminal that will generate the external input based on operation terminal information including information on the types of a plurality of operation terminals that are candidates for generating the external input and information on the task, and when the operation terminal cannot be determined, accumulating the support request information for which support has not yet been executed, determining a priority of each piece of support request information based on an urgency of support, and when the support becomes executable, processing the support request information in descending order of priority; A program that a computer runs.

Citation Information

Patent Citations

  • Robot remote operation system

    JP2007190659A

  • Evaluation device, evaluation method, service provision system, and computer program

    JP2012230506A

  • Robot control method

    JP2016068161A

  • Systems and methods for assisting robotic devices

    JP2020507164A

  • Remote operation system, server, remotely operated device, remote operation service providing method

    WO2008140011A1