Server, its control method, program, and system

The server-based system addresses the challenge of managing multiple support requests in semi-autonomous robots by prioritizing tasks based on completion times and operator authority, enhancing operational efficiency and reducing downtime.

JP7595914B2Active Publication Date: 2024-12-09TOKYO ROBOTICS INC
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
JP2020172092
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Filing Date
2020-10-12
Publication Date
2024-12-09
Estimated Expiration
2040-10-12

AI Technical Summary

Technical Problem

Conventional semi-autonomous robot systems face challenges in handling multiple support requests simultaneously due to a lack of efficient prioritization and resource allocation mechanisms, leading to potential operational inefficiencies and downtime.

Method used

A server-based system that receives support requests from multiple robot systems, determines priority based on task completion times and operator authority, and provides operators with optimized task lists to manage support requests effectively.

Benefits of technology

Enables operators to efficiently handle multiple support requests by prioritizing tasks that have the greatest impact on the overall process, minimizing downtime and optimizing system performance.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007595914000001
    Figure 0007595914000001
  • Figure 0007595914000002
    Figure 0007595914000002
  • Figure 0007595914000003
    Figure 0007595914000003
Patent Text Reader

Abstract

To provide a technique for enabling an operator to cope with support requirements when the plurality of support requirements are performed from a robot in an automation technique using the robot in a production field and a physical distribution warehouse.SOLUTION: A server 3 that is connected to a plurality of robot systems 1 for semi-autonomously executing a predetermined task, and one or a plurality of operator terminals includes: a support requirement reception part for receiving support requirements from the robot systems; a priority determination part for determining a priority of support requirements when receiving a plurality of support requirements; and an information providing part for providing information on the support requirements and information on the priority to the operator terminal.SELECTED DRAWING: Figure 1
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to a semi-autonomous robot system that is remotely controlled by an operator. [Background technology]

[0002] Automation using robots is being promoted in production sites, logistics warehouses, and other places.

[0003] In this type of automation system, it is ideal for all robots to operate autonomously, but in reality, it is difficult to make all robots operate autonomously due to hardware limitations and the need to respond to unexpected situations.

[0004] In recent years, therefore, semi-autonomous systems have been considered in which, while most of the robot's movements are automated, an operator can temporarily intervene remotely to support the robot's operations only in situations that the robot cannot handle.

[0005] For example, Literature 1 discloses a robot control method for switching the control of the robot from autonomous control to manual control when task execution fails. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] JP 2016-068161 A Summary of the Invention [Problem to be solved by the invention]

[0007] However, in this type of conventional technology, when an operator provides support, it is assumed that there is a one-to-one relationship between the operator and the robot, and there has been no sufficient consideration given to cases where the relationship between the operator and the robot is 1:N or M:N (N and M are natural numbers equal to or greater than 2). Therefore, in cases where multiple support requests are made simultaneously from multiple robots, there was a risk that it would be difficult for the operator to support the robot.

[0008] The present invention has been made in consideration of the above-mentioned technical background, and has an object to provide a technique that enables an operator to easily respond to support requests even when multiple support requests are made from a robot. [Means for solving the problem]

[0009] The above-mentioned technical problem can be solved by a server or the like having the following configuration.

[0010] In other words, the server of the present invention is equipped with a plurality of robot systems that semi-autonomously execute predetermined tasks, one or more operator terminals, and a connected server, the server comprising: a support request receiving unit that receives support requests from the robot systems; a priority determination unit that, when a plurality of support requests are received, determines the priority of the support requests; and an information providing unit that provides information regarding the support requests and information regarding the priority to the operator terminal.

[0011] According to this configuration, even if multiple support requests are received from a robot system that executes a task semi-autonomously, the operator can easily deal with the multiple support requests because the priority order can be determined for the support requests. Note that, here, support means performing operations or instructions for the semi-autonomous robot to assist it in executing a target action. Also, a support request means information requesting such assistance from the operator.

[0012] The priority order determination unit may further include a required time calculation unit that calculates, for each of the robot systems that has made the support request, a first sum value that is a sum value of required times of tasks subsequent to a task to be executed by the robot system, and a required time based priority order determination unit that determines the priority order such that the priority is higher as the first sum value is larger.

[0013] With this configuration, the operator can deal with tasks that have the greatest impact on the entire process first.

[0014] The priority order determination unit may further include a second required time calculation unit that calculates a second sum by adding a remaining required time of the task being executed to a total value of required times of tasks subsequent to the task to be executed by the robot system that has made the support request, and a second required time based priority order determination unit that determines the priority order such that the priority order is higher as the second sum value is larger.

[0015] With this configuration, the operator can deal with tasks that have a large impact on the entire process first. In addition, since the remaining required time of the task being executed is also taken into account, if there is a task that takes a long time to execute, the priority order can be determined more precisely.

[0016] The priority order determination unit may further include a third required time calculation unit that calculates, for each of the robot systems that has made the support request, a third total value that is calculated by adding a remaining required time for the task being executed and a required time for any necessary support action to a total value of required times for tasks subsequent to the task to be executed by the robot system, and a third required time based priority order determination unit that determines the priority order such that the priority order is higher as the third total value is larger.

[0017] With this configuration, the operator can deal with tasks that have the greatest impact on the entire process first. In addition, since the remaining time required for the task being executed and the time required for the necessary support operations are also taken into account, when there are tasks that take a long time to execute or tasks that require time for support, the priority order can be determined more precisely.

[0018] Any of the required times may be determined based on actual values ​​measured by operating the robot system.

[0019] According to this configuration, the required time can be accurately estimated based on the actual measured value.

[0020] The server may further include a support authority memory unit that stores, for each operator corresponding to each of the operator terminals, support authority for the robot system, the task, and / or some of the operations that constitute the task, and the information providing unit may provide information regarding the support request to the operator terminal based on the support authority.

[0021] According to this configuration, support requests can be efficiently assigned to operator terminals based on the presence or absence of support authority.

[0022] The server may further include a sum calculation unit that calculates a sum of the first sum, the second sum, or the third sum for each combination of the support request and the operator corresponding to the operator terminal, and the information providing unit may further provide information regarding the support request to the operator terminal based on the combination that results in the smallest sum.

[0023] With such a configuration, the entire system can be optimized, and downtime of the entire process can be minimized.

[0024] The server may further include a support authority memory unit that stores, for each operator corresponding to each of the operator terminals, support authority for the robot system, the task, and / or some of the operations that constitute the task, and a sum calculation unit that calculates a sum of the first sum, the second sum, or the third sum for each combination of the support request and the operator corresponding to the operator terminal who has the support authority for the support request, and the information providing unit may further provide information regarding the support request to the operator terminal based on the combination that minimizes the sum.

[0025] With this configuration, even when the number of robot systems or operators is large, it is possible to prevent combination explosion while optimizing the system as a whole, thereby minimizing downtime of the entire process.

[0026] The priority order may be determined such that the older the time at which the support request was received from the robot system, the higher the priority order.

[0027] According to such a configuration, in a system where chronological order is appropriate, it is possible to appropriately determine the priority order.

[0028] The server may include a priority information storage unit that stores predetermined priority information for the robot system, the tasks, and / or operations that constitute part of the tasks, and the priority order may be determined based on the priority information.

[0029] According to such a configuration, it is possible to set in advance tasks that are to be emphasized or not to be emphasized during operation.

[0030] The operator terminal may include a list display unit that displays a list of the connected robot systems, and a highlighting unit that highlights the robot system that is making the support request based on information related to the support request.

[0031] With this configuration, it is possible to immediately identify the robot system that is making the support request.

[0032] The operator terminal may further include a sorting display unit that sorts and displays the robot systems that have made the support requests in order of priority based on information regarding the priority.

[0033] According to this configuration, the sorted display allows the user to intuitively grasp the robot system to be supported.

[0034] The server may further include an account information management unit that manages account information including support authority for the robot system, the task, and / or some of the operations that constitute the task, which is set for each operator of the operator terminal, and the operator terminal may further include a login processing unit that provides a login process that provides a match with the account information to an operator using the operator terminal.

[0035] According to this configuration, different operators can use the same operator terminal with their own accounts.

[0036] The operator terminal may further include an authority-specific display unit that displays, among the robot systems for which the support request is being made, the robot system for which the operator who performed the login process has the support authority in a manner that distinguishes it from other robot systems.

[0037] According to this configuration, the logged-in operator can immediately grasp the robot systems that he or she can support based on the support authority.

[0038] The robotic system may include a mobile manipulator.

[0039] With this configuration, even if multiple support requests are received from a mobile manipulator that executes a task semi-autonomously, it is possible to determine a priority order for the support requests, so that the operator who operates the operator terminal can easily deal with the multiple support requests.

[0040] The present invention can also be conceived as a system. That is, the system according to the present invention is a system including a plurality of robot systems that execute predetermined tasks semi-autonomously, one or a plurality of operator terminals, and a server connected to the robot systems and the operator terminal, the server including a support request receiving unit that receives a support request from the robot systems, a priority order determining unit that determines the priority order of the support requests when the server receives a plurality of the support requests, and an information providing unit that provides information related to the support requests and information related to the priority order to the operator terminal.

[0041] The present invention can also be conceived as a method. That is, the method according to the present invention is a control method for a plurality of robot systems that execute predetermined tasks semi-autonomously, one or a plurality of operator terminals, and a connected server, comprising: a support request receiving step of receiving a support request from the robot systems; a priority order determining step of determining the priority order of the support requests when a plurality of the support requests are received; and an information providing step of providing information on the support requests and information on the priority order to the operator terminal. It is equipped with:

[0042] The present invention can also be conceived as a computer program. That is, the program according to the present invention is a control program for a server connected to a plurality of robot systems that semi-autonomously execute a predetermined task, and one or a plurality of operator terminals, and includes a support request receiving step of receiving a support request from the robot systems, a priority order determining step of determining the priority order of the support requests when a plurality of the support requests are received, and an information providing step of providing information on the support requests and information on the priority order to the operator terminal. Effect of the Invention

[0043] According to the present invention, it is possible to provide a system etc. that enables an operator to easily respond to multiple support requests even when multiple support requests are received from a robot system that executes tasks semi-autonomously. [Brief description of the drawings]

[0044] [Figure 1] FIG. 1 is a diagram showing the overall configuration of the system. [Diagram 2] FIG. 2 is a functional block diagram of the system. [Diagram 3] FIG. 3 is a table showing the correspondence between the robot system ID and the task. [Figure 4] FIG. 4 is a table showing the correspondence between tasks, actions that are unit operations constituting the tasks, and the required time for each action. [Diagram 5] FIG. 5 is a table showing the correspondence between operator IDs and the authorities granted to the operators. [Figure 6] FIG. 6 is a flowchart illustrating the operation of the robot system. [Figure 7] FIG. 7 is a flowchart showing the information collection operation of the management server. [Figure 8] FIG. 8 is a flowchart showing a process for providing information to the operator system. [Figure 9] FIG. 9 is a flowchart showing the detailed operation of the priority order determination process. [Figure 10] FIG. 10 is an arrow diagram of a series of tasks. [Figure 11] FIG. 11 is an example of the operation status list screen. [Figure 12] FIG. 12 is an operational flowchart relating to the support process. [Figure 13] FIG. 13 is an example of a support screen. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0045] Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

[0046] 1. First Embodiment As a first embodiment, an example will be described in which the present invention is applied to a system in a logistics warehouse including a plurality of robot systems each performing a series of tasks, i.e., tasks such as sorting items and then packing them into boxes and sealing them with tape.

[0047] In this embodiment, the robot system is applied to a logistics warehouse system, but is not limited to such an application example, and can be applied to various other situations, for example, factories, etc. In this embodiment, the robot system is described as having a robot arm, but various hardware other than the robot arm can be adopted. The robot system does not need to be fixed, and for example, a mobile manipulator can be adopted. The tasks of the robot system are not limited to those described above, and can be applied to various other tasks.

[0048] (1.1 System Configuration) 1 is an overall configuration diagram of a system 200 according to this embodiment. As is clear from the diagram, the system 200 includes a plurality of robot systems 1, a management server 3, and an operator system 5, which are connected to each other via a LAN (local area network) by wire or wirelessly. Note that, in this embodiment, the devices are described as being connected to each other via a LAN, but they may also be connected via the Internet.

[0049] Each robot system 1 is composed of a robot-side information processing device 11 that performs various information processing described below, a sensor 15 connected to the robot-side information processing device 11, and a multi-joint robot arm 17 connected to the robot-side information processing device 11.

[0050] In this embodiment, the robot side information processing device 11, the sensor 15, and the robot arm 17 are described as constituting a single system while being physically arranged separately, but some or all of them may be configured as an integrated system.

[0051] The robot side information processing device 11 includes a control unit that is made up of a CPU and the like and executes various programs, and a storage unit that is made up of a ROM, a RAM, a hard disk, and the like.

[0052] The sensor 15 is a three-dimensional camera fixed to the environment to obtain image information that is the basis of recognition, and for example, when performing an item sorting task, it takes an image of the inside of a box containing a large number of target products or other workpieces, and when performing a boxing task, it takes an image of the workpieces on the conveyor. The information obtained from the sensor is transmitted to the robot side information processing device 11.

[0053] In this way, the 3D camera serving as the sensor 15 is fixed in the environment, and as described later, by using the same viewpoint for both recognition and remote control, the configuration is simple and has cost advantages. Note that the method of mounting the sensor 15 is not limited to this example of fixing it in the environment, and it is also possible to use a hand-held camera mounted on the robot arm 17 or the like, or to use an existing surveillance camera.

[0054] The articulated robot arm 17 has a gripper at its tip and can move freely. For example, when performing an item sorting task, it grips a workpiece and places it on a belt conveyor or the like. Also, when performing a boxing and tape closing task, it grips and moves the workpiece placed on the belt conveyor and tapes it down.

[0055] As described later, the management server 3 appropriately collects information on the robot system 1 and information on the operator system 5. In addition, the management server 3 provides various information to the robot system 1 and the operator system 5 as necessary.

[0056] The operator system 5 includes an operator-side information processing device 51, a display device 52, and a haptic input device 53. The operator-side information processing device 51 is an information processing device that performs overall control of the operator system 5, and is, for example, a PC (personal computer). The display device 52 is a display device, and displays a list of robot systems 1 that have issued support requests, as described later.

[0057] The haptic input device 53 is provided with an angle sensor for each joint, and can control the six-axis posture of the hand of the robot arm 17 and the opening and closing of the gripper. As will be described later, the haptic input device 53 is operated by an operator and supports the robot arm 17 in executing a task in response to a support request. The haptic input device 53 may provide force feedback based on a sensor provided in the robot arm 17.

[0058] The display device and the input device are not limited to the examples of the present embodiment, and various known devices can be used. For example, the display device may be a head-mounted display, and the input device may be an input device using a stick lever, a keyboard, or the like.

[0059] In addition, in this embodiment, the operator side information processing device 51, the display device 52, and the haptic input device 53 are described as constituting a single system while being physically arranged separately, but some or all of them may be configured as an integrated system.

[0060] 2 is a functional block diagram of the system 200. As is clear from the diagram, the robot system 1 transmits and receives information to and from the management server 3 and the operator system 5 via their respective transmitting and receiving units.

[0061] The robot system 1 comprises a robot-side information processing device 11 , a sensor 15 , and a robot arm 17 .

[0062] The robot-side information processing device 11 includes a sensor information acquisition unit 111 that acquires detection information from the sensor 15, and a recognition processing unit 112 that performs a predetermined recognition process based on the sensor information. The recognition result by the recognition processing unit 112 is provided to a determination unit 114. The determination unit 114 determines whether or not a support request, which will be described later, should be made based on the recognition result, and provides the determination result to a motion generation unit 118. The motion generation unit 118 and the motion command unit 119 issue motion commands to the robot arm 17.

[0063] When the support information is received via the transmission / reception unit 127, the support process execution unit 121 executes various support processes according to the support information. For example, the support process execution unit 121 controls the operation of the robot arm 17.

[0064] The management server 3 includes a support request detection unit 32 and an information acquisition processing unit 33. The support request detection unit 32 detects a support request transmitted from the robot system 1. Furthermore, the priority determination processing unit 35 determines the priority of the support requests when multiple support requests are received. The information provision processing unit 36 ​​transmits information related to the support request and information related to the priority to the operator system 5 via the transmitting / receiving unit.

[0065] The information acquisition processing unit 33 acquires status information and the like of the robot system 1 and the operator system 5. The acquired information is stored in a storage unit (not shown) by a storage processing unit 38. In addition, the list control unit 37 manages predetermined list information (described later) based on the status information and the like.

[0066] Next, a description will be given of table information that is stored in advance as setting information in the management server 3. Note that, although the following description assumes that the table information is stored in advance, the present invention is not limited to such a configuration, and the table information may be set via the operator system 5 or the like during operation.

[0067] 3 is a table showing the correspondence between robot system IDs and tasks, which is stored in advance in the storage unit of the management server 3. As is clear from the figure, each robot system 1 is given identification information such as "robot01" and "robot02", and identification information of the task to be executed corresponding to the identification information is given, for example, "TASK_A" (task A) and "TASK_B" (task B). In other words, based on such a table, "robot01" is set to execute "TASK_A".

[0068] 4 is a table showing the correspondence between tasks, actions that are unit operations constituting the tasks, and the required time ([s]) of each action, which are stored in advance in the storage unit of the management server 3. As is clear from the figure, "TASK_A" is composed of seven actions, from "ACTION_01" to "ACTION_07," and each action is associated with the required time shown in the figure.

[0069] 5 is a table showing the correspondence between operator IDs, which are identification information of operators, and the authorities granted to the operators, which are stored in advance in the storage unit of the management server 3. As is clear from the figure, the operator ID "Sakamoto" is granted support authority for two actions, "TASK_A" and "TASK_B". Furthermore, the operator ID "Kawanishi" is granted support authority only for "TASK_B". However, "Sakamoto" is not granted support authority for "robot03".

[0070] In this embodiment, the support authority is defined in units of a task and a robot ID, but it may be defined in other ways. For example, the support authority may be defined by other means, such as by action.

[0071] Furthermore, the system configuration in this embodiment is merely an example, and the number and configuration of each device connected to the network can be freely changed.

[0072] (1.2 System Operation) The operation of the system 200 will now be described.

[0073] 6 is a flowchart explaining the operation of the robot system 1. Generally speaking, the robot system 1 semi-autonomously executes a target motion or a task, and transmits a support request to the management server 3 when it is unable to properly recognize or fails to execute the target motion.

[0074] As is clear from the figure, when processing starts, the sensor information acquisition unit 111 acquires sensor detection information from the sensor 15, which in this embodiment is image information obtained by photographing the inside of a box or on a conveyor (S1). The recognition processing unit 112 performs processing to recognize a workpiece or the like to be grasped by the robot arm 17 based on the sensor information (S2). Note that, in this embodiment, recognition is described as being performed based on image information, but the present invention is not limited to such a configuration. Therefore, for example, a configuration may be used in which three-dimensional information acquired by a three-dimensional camera is used instead of or together with an image.

[0075] After this recognition process, the determination unit 114 performs a process of determining whether support is necessary (S4). More specifically, when the recognition unit 112 has performed the recognition normally, for example, when the workpiece to be grasped can be recognized with sufficient recognition accuracy, the determination unit 114 determines that support is unnecessary, whereas when the recognition has not been performed normally, the determination unit 114 determines that support is necessary.

[0076] In this embodiment, the necessity of support is judged based on whether or not normal recognition is possible, but the present invention is not limited to such a configuration, and various judgment methods can be adopted. For example, the judgment process may be performed based on a criterion such as whether or not a grasping solution for the object to be grasped can be calculated.

[0077] If it is determined as a result of the determination process that the recognition process has been performed normally and that support is not required (S4 NO), the motion generation unit 118 generates a target motion (S6). The target motion is a motion that the robot system 1 is expected to execute, such as a task motion set for the robot system 1. Thereafter, the motion command unit 119 issues a motion command to the robot arm 17 to execute the generated target motion, and the target motion is executed (S7).

[0078] After this action, a process is performed to evaluate whether the target action was successful or not based on the sensor 15 information (S8). If it is determined that the target action was successful as a result of this evaluation process (S8 YES), an end determination is performed (S9), and if a predetermined end signal or the like is detected (S9 YES), the process ends. On the other hand, if the end signal is not detected, the series of processes are repeated again from the sensor information acquisition process (S1) (S9 NO). If it is determined that the target action was unsuccessful (S8 NO), a support request transmission process (described below) is performed (S11).

[0079] The term "support" refers to performing operations or instructions for the semi-autonomous robot in order to assist it in executing a target action. The term "support request" refers to information requesting such assistance from an operator. In this embodiment, the support request from the robot system 1 may be referred to as an "alert."

[0080] If it is determined that support is necessary (S4 YES) or if the target action has failed (S8 NO), a support request is sent to the management server 3 (S11). After that, the robot-side information processing device 11 goes into a standby state until it receives support information (S12 NO).

[0081] When support information is received in this state (S12 YES), the support process execution unit 121 executes support execution processing according to the support information (S14). After that, a predetermined end determination is performed, and if an end signal is not detected, the reception standby state (S15 NO) is entered again, and a series of processes are repeated. On the other hand, if an end signal is detected (S15 YES), the above-mentioned end determination process (S9) is performed.

[0082] 7 is a flowchart regarding the information collection operation of the management server 3 that acquires various information from the robot system 1 and the operator system 5. As is clear from the figure, when processing starts, the information acquisition processing unit 33 executes in parallel processes for acquiring four pieces of information, namely, status information (S21) of the robot system 1, a support request (S31), a support completion signal (S41), and operator information (S51), and each process is in a standby state until the corresponding information is acquired (S22NO, S32NO, S42NO, S52NO).

[0083] The status information of the robot system 1 is information that indicates the current operating state of the robot system 1, and includes, for example, information such as the task being executed, the action, the elapsed time since the start of the operation, the joint angle information of the robot arm 17, and the sensor information detected by the sensor 15. The support request is information that is sent from the robot system 1 by the above-mentioned process and requests support from an operator, and includes, for example, information such as the ID of the robot system 1 that made the support request, and the time when the support request was generated.

[0084] The support completion signal is information transmitted from the operator system 5 when the support by the operator is completed, and includes, for example, the support completion time. The operator information is information about the operator associated by logging in to the operator side information processing device 51, and includes, for example, information such as the operator ID. The login process is performed by checking against the account information stored in the management server 3.

[0085] In this manner, in this embodiment, the operator system 5 is associated with the operator by logging in, so that the same operator system 5 can be shared by different operators.

[0086] When the storage processor 38 receives the status information from the robot system 1 (S22 YES), it performs a process of storing the status information (S23) and goes into a standby state again.

[0087] When the list control unit 37 receives a support request from the robot system 1 (S32 YES), it performs a process of adding the ID information of the robot system 1 that has made the support request to a predetermined support request list (S33). At this time, the ID information of the robot system 1 that has made the support request is stored in the storage unit by the storage processing unit 38. After that, it goes into a standby state again.

[0088] Furthermore, when the list control unit 37 receives a support completion signal (S42 YES), it performs processing to delete the ID information of the robot system 1 for which support has been completed from a predetermined support request list (S43). At this time, the storage processing unit 38 also erases the ID information of the robot system 1 that has made the support request from the storage unit. Thereafter, the system goes into standby state again.

[0089] Furthermore, when the storage processor 38 receives the operator information from the operator system 5 (S52 YES), it performs processing to store the operator information (S53) and goes into standby mode again.

[0090] That is, the management server 3 collects, acquires, stores, and manages various pieces of information transmitted from the robot system 1 and the operator system 5.

[0091] 8 is a flowchart relating to a process of providing information to the operator system 5. The flowchart on the left side of the figure shows the operation of the operator system 5, and the flowchart on the right side of the figure shows the operation of the management server 3.

[0092] As is clear from the figure, when processing starts, the display processing unit 57 of the operator-side information processing device 51 makes a display request to the management server 3 (S61). After this display request, the operator-side information processing device 51 enters a standby state for predetermined information (S62 NO). When the predetermined information is received (S62 YES), display processing according to the information is performed on the display unit 52 (S64). After that, a series of processes are repeated again (S61 to S64). That is, the operator-side information processing device 51 is constantly performing update display processing.

[0093] The support request detection unit 32 of the management server 3, which was in a waiting state (S71 NO) until receiving a display request, performs a process of detecting, from the support request list, the robot system 1 for which a support request is being made at the time of receiving the display request (S72), when it receives the display request from the operator side information processing device 51 (S71 YES).

[0094] As a result of this detection process, if there is one support request (S73 NO), the information provision processing unit 36 ​​performs a process of providing information for presenting the support request to the operator system 5 to the operator system 5 (S75). Note that this provided information may include any information collected by the management server 3.

[0095] On the other hand, if there are multiple support requests (S73 YES), the priority determination processing unit 35 performs a process of determining the priority among the support requests (S76), and after this process, performs a predetermined information providing process together with the priority (S78). Note that this provided information may include all information collected by the management server 3.

[0096] 9 is a flowchart showing the detailed operation of the priority order determination process. As is clear from the diagram, when the process starts, variables are initialized and an initialization process is performed by referring to a specific robot system 1 (S81).

[0097] Then, a process is performed to calculate the total required time of the tasks subsequent to the task performed by the referred robot system 1 (S83). More specifically, the total required time is calculated based on the above-mentioned table showing the correspondence between the robot system ID and the task and the table showing the correspondence between the task and the required time ([s]).

[0098] After calculating the total time required for the subsequent tasks, it is determined whether the calculation process for this total has been completed for all robot systems 1 (S84). If the calculation process has not yet been completed for all robot systems 1, the robot system 1 to be referenced is changed and the calculation process (S83) is performed again.

[0099] On the other hand, when the calculation process is completed for all the robot systems 1 (S84 YES), a process is performed to determine the priority of the robot system 1 that has requested support based on the total value (S86). In this embodiment, the higher the total value of the required time of the subsequent tasks, the higher the priority of the response is determined to be. Then, the priority determination process ends.

[0100] A specific example of determining priority will be described with reference to FIG. 10. FIG. 10 is an arrow diagram of a series of tasks according to this embodiment. As is clear from the diagram, when "robot01" executes a task of sorting items (task A), the sorted objects are placed on different conveyors and transported as three parallel processes. The items transported by the conveyors are packed into boxes by the following "robot02," "robot03," and "robot04," and the boxes are taped shut. After that, each robot system 1 places the taped-sealed boxes on the conveyor and transports them (task B).

[0101] In this case, it takes 11.6 seconds to execute task A, 30 seconds to execute task B, and 10 seconds for conveyor transport during those times. These times are measured by actually operating the robot system 1. With this configuration, the required time can be accurately estimated based on the actual measured values.

[0102] In this state, for example, when calculating the total required time for subsequent tasks for "robot01" in the robot system 1, the total required time is calculated as 150 seconds (=50×3) since one series of processes among the three subsequent parallel processes takes a total of 50 seconds (=10+30+10). Similarly, when calculating the total required time for subsequent tasks for "robot02," "robot03," and "robot04," each of them is calculated as 10 seconds.

[0103] That is, in the example of the figure, it is understood that among the robot systems 1, "robot01" is the robot system 1 whose stoppage or the like has a greater impact on the entire process than "robot02," "robot03," and "robot04." Therefore, the support request from "robot01," which has a longer total required time for the subsequent tasks, should be given priority.

[0104] 11 is a display example of an operation status list screen displayed on the display unit 52 of the operator system 5 as a result of the display process (S64). As is clear from the figure, an operation status list display unit 80 is provided in the center of the screen, and the ID information "Sakamoto" of the operator currently using the operator system 5 and the current time are displayed in the upper right corner of the screen.

[0105] In addition, the operation status list screen displays, from left to right, the columns "Operation status," "Time of trouble occurrence," "Robot ID," "Task," "Action," "Your assistance authority," "Assistant," and "Assistance status."

[0106] The "operation status" indicates whether the robot system 1 is operating normally or in an alert state. The alert state is a state in which the robot system 1 is requesting support from an operator because the robot system 1 has failed to execute a target operation or is unlikely to execute a target operation.

[0107] "Trouble occurrence time" is information regarding the date and time when the support request occurred. "Robot ID" is ID information that is identification information of the robot system 1 that responds to the support request.

[0108] A "task" is a task assigned to each robot system 1. In the example shown in the figure, the robot is assigned either an "item sorting" task (task A) or a "boxing and taping" task (task B).

[0109] "Action" indicates which action is being performed among the actions that make up the task. From the example in the figure, it can be seen that "robot01" requested support while performing the "ACTION_02" "Recognize grip position of object in case" action. It can also be seen that "robot02" is normally performing the "Gripper release / store" action of "ACTION_9".

[0110] "Your Assistance Authority" indicates whether the logged-in operator has support authority for each robot system 1 that is making a support request. From the figure, it can be seen that operator "Sakamoto" has support authority for task A and task B, except for "robot03."

[0111] "Assistant" displays the ID of the operator providing support. For example, in the example shown in the figure, it can be seen that the operator "Kawanishi" is providing support for "robot07."

[0112] "Assistance status" indicates whether support is being provided in response to a support request. For example, in the example shown in the figure, it is indicated that the operator "Kawanishi" is currently responding to a support request from "robot07."

[0113] In the example shown in the figure, the support requests are displayed in order of priority from highest to lowest based on the calculated priority, i.e. "robot01", "robot04", "robot07", and "robot03". In other words, the robot systems 1 that have made support requests are sorted and displayed in a manner that makes them distinguishable from the robot systems 1 that have not made support requests.

[0114] The method of distinguishably displaying the robot system 1 making the support request is not limited to this example, and various other display methods can be adopted. For example, only the robot system 1 making the support request may be displayed, or the display related to the robot system 1 making the support request may be highlighted by coloring, etc.

[0115] Next, the support process performed by the operator via the operator system 5 will be described.

[0116] 12 is an operational flowchart relating to the support processing performed by an operator. The flowchart on the left side of the figure is an operational flowchart of the operator system 5, and the flowchart on the right side of the figure is an operational flowchart of the management server 3.

[0117] In the figure, when the process starts, the operator-side information processing device 51 performs a process of detecting whether an input has been made to select one of the support requests displayed in the operation status list (S101). If no input is detected as a result of this detection process (S102 NO), the input detection process is repeated (S101, S102 NO).

[0118] As a result of the detection process, if an input is detected (S102 YES), a process is performed in which the operating operator transmits a support request requesting that the selected support request be dealt with (S104). After transmitting this support request, the operator side information processing device 51 enters a predetermined reception standby state (S105 NO).

[0119] On the other hand, when the management server 3, which was in a standby state (S111 NO) until receiving a support request, receives the support request (S111 YES), it performs a process of setting the operator who sent the support request as the operator in charge in association with the support request (S112). As a result, for example, the corresponding operator ID and the message "In Service" are displayed in the "Assistant" and "Assistance Status" columns in Fig. 11. After this, the management server 3 sends a notification to the operator system 5 indicating that a support request has been set (S114).

[0120] When the operator system 5 receives the notification (S105 YES), it performs processing to display a support screen.

[0121] Fig. 13 is an example of a support screen that allows an operator to respond to a support request. As is clear from the figure, a status display section 83 for the selected support request being processed is provided at the top center of the screen, and the ID information "Sakamoto" of the operator currently using the operator system 5 and the current time are displayed at the top right of the screen.

[0122] A camera view display area 83 is provided on the lower left side of the figure, which displays a camera view captured by the camera 15 of the robot system 1. In the example of the figure, an image of workpieces 92 piled up in a box 91, which are the targets of an item sorting task, is displayed in the camera view display area 83. The workpieces 92 are cylindrical, and are each grasped by the robot arm 17 and sorted according to a predetermined criterion.

[0123] In this embodiment, the camera view display area 83 is described as displaying a camera image, but the present invention is not limited to this configuration. Therefore, for example, a configuration may be adopted in which three-dimensional information acquired by a three-dimensional camera is presented in place of or together with the image.

[0124] At the center of the bottom of the figure, there is provided a CG image display unit 84 that displays a CG (Computer Graphics) image 841 generated based on posture information obtained from a joint angle sensor (not shown) provided on the robot arm 17 in its current posture. The image display unit allows the operator to instantly grasp in what posture the robot arm 17 is currently in.

[0125] On the lower right side of the figure, a work start button 85 for starting support processing is arranged.

[0126] Returning to FIG. 12, when the work start button 85 is selected, the operator system 5, the management server 3, and the robot system 1 appropriately communicate with each other to perform support processing (S108).

[0127] In this embodiment, the content of the support process is an operation of remotely controlling the robot arm 17 using the haptic input device 53 while checking the camera view display area 83, and changing the execution conditions of the target action. The operation of changing the execution conditions of the target action is, for example, an operation of changing the scene or environmental conditions, and more specifically, an operation of changing the position and posture of the workpiece 92 by, for example, shaking a box using the robot arm 17 by remotely controlling the operator, or by gripping and moving the workpiece 92 inside the box.

[0128] In the present embodiment, the content of the support process is an action of changing the execution conditions of the target action, but is not limited to this. Therefore, for example, the target action itself may be alternatively executed by remote control.

[0129] After that, when the support is completed, a support completion signal is sent to the management server 3, and then the series of processes are repeated again from the input detection process (S101). As described above, the support completion signal is acquired by the management server 3, and the corresponding support request is deleted from the list (S41 to S43).

[0130] With this configuration, even if multiple support requests are received from a robot system that executes tasks semi-autonomously, the priority order for the support requests can be determined, so the operator does not have to consider which task should be given priority, and multiple support requests can be easily handled.

[0131] Furthermore, with this configuration, the operator can deal with tasks that have a large impact on the entire process, based on the required time, and therefore the entire system can be operated efficiently.

[0132] (2. Modifications) In the above embodiment, the total required time of subsequent tasks is calculated to determine the priority order, but the present invention is not limited to such a configuration.

[0133] Therefore, for example, the priority order may be determined based on the total required time plus the required time of the remaining actions of the referenced robot system 1. With such a configuration, the required time of the remaining actions of the task being executed is also taken into consideration, so that even if there is an action that takes a long time to execute, the priority order can be determined more precisely.

[0134] Also, for example, the priority order may be determined based on a value obtained by further adding to the total required time an expected required time for supporting the robot system 1. With such a configuration, even if support requires a certain amount of time, the priority order can be determined more precisely.

[0135] In the above embodiment, all robot systems 1 that have requested support are displayed to the operator, but the present invention is not limited to such a configuration. Therefore, for example, the combination of the robot system 1 that has requested support and the operator system 5 may be optimized.

[0136] For example, the time required for the entire process may be calculated for all combinations of the robot system 1 that has requested support and the operator system 5, and the combination that has the shortest time required may be recommended or provided to the operator. With such a configuration, the entire system can be optimized, and downtime of the entire process can be minimized.

[0137] Also, for example, the time required for the entire process may be calculated for all combinations of the robot system 1 that has made the support request and the operator system 5 that has the authority to provide support for the support request, and the combination that has the shortest time required may be recommended or provided to the operator. With such a configuration, the entire system can be optimized, and downtime of the entire process can be minimized. Even if the system 200 becomes large-scale and the number of robot systems 1 and operator systems 5 increases, an explosion of combinations can be prevented, and the calculation load can be optimized and processing speed can be increased.

[0138] In the above embodiment, the priority order is determined based on the required time of the subsequent task, but the support requests may simply be displayed in chronological order on the display terminal 52. With this configuration, it is possible to provide optimal support for a system where processing in the order of occurrence is desirable.

[0139] Although the embodiments of the present invention have been described above, the above embodiments merely show some of the application examples of the present invention, and the technical scope of the present invention is not limited to the specific configurations of the above embodiments. Moreover, the above embodiments can be appropriately combined within a range where no contradiction occurs. [Industrial Applicability]

[0140] The present invention can be used at least in industries that manufacture robot systems and the like. [Explanation of symbols]

[0141] 1. Robot System 11 Robot side information processing device 15 Sensors 17 Robot Arm 3 Server 5 Operator Systems 51 Operator side information processing device 52 Display device 53 Haptic Input Device

Claims

1. A server connected to a plurality of robot systems that semi-autonomously execute a predetermined task and one or a plurality of operator terminals, a support request receiving unit that receives a support request from the robot system; a priority order determination unit that determines a priority order of the support requests when a plurality of the support requests are received; an information providing unit that provides information regarding the support request and information regarding the priority order to the operator terminal; The priority order determination unit further a required time calculation unit that calculates, for each of the robot systems that have made the support request, a first sum value that is a sum value of required times of tasks subsequent to a task to be executed by the robot system; The server further comprises: a sum calculation unit that calculates a sum of the first sum values ​​for each combination of the support request and the operator corresponding to the operator terminal, The information providing unit further provides information regarding the support request to the operator terminal based on the combination that minimizes the sum.

2. A server connected to a plurality of robot systems that semi-autonomously execute a predetermined task and one or a plurality of operator terminals, a support request receiving unit that receives a support request from the robot system; a priority order determination unit that determines a priority order of the support requests when a plurality of the support requests are received; an information providing unit that provides information regarding the support request and information regarding the priority order to the operator terminal; The priority order determination unit further a required time calculation unit that calculates, for each of the robot systems that have made the support request, a first sum value that is a sum value of required times of tasks subsequent to a task to be executed by the robot system; The server further comprises: a support authority storage unit that stores, for each operator corresponding to each of the operator terminals, support authority for the robot system, the task, and / or a part of operations constituting the task; a sum calculation unit that calculates a sum of the first sum values ​​for each combination of the support request and an operator corresponding to the operator terminal having the support authority for the support request, The information providing unit further provides information regarding the support request to the operator terminal based on the combination that minimizes the sum.

3. The server according to claim 1 or 2, wherein each of the required times is determined based on an actual measurement value measured by operating the robot system.

4. The server further comprises: a support authority storage unit configured to store, for each operator corresponding to each of the operator terminals, support authority for the robot system, the task, and / or a part of operations constituting the task; The server according to claim 1 , wherein the information providing unit provides information regarding the support request to the operator terminal based on the support authority.

5. The operator terminal includes: a list display unit that displays a list of the connected robot systems; The server according to claim 1 , further comprising: a highlighting unit configured to highlight the robot system making the support request based on information regarding the support request.

6. The operator terminal further includes: The server according to claim 5 , further comprising a sorting and displaying unit that sorts and displays the robot systems making the support requests in order of priority based on information regarding the priority.

7. The server further comprises: an account information management unit that manages account information including a support authority for the robot system, the task, and / or a part of operations constituting the task, which is set for each operator of the operator terminal; The operator terminal further includes: The server according to claim 1 , further comprising: a login processing unit that provides a login process for providing a matching with the account information to an operator using the operator terminal.

8. The operator terminal further includes: The server according to claim 7, further comprising an authority-based display unit that displays, among the robot systems making the support request, the robot systems for which the operator who performed the login process has the support authority in a manner that makes them distinct from other robot systems.

9. The server of any one of claims 1 to 8, wherein the robotic system includes a mobile manipulator.

10. A system including a plurality of robot systems that execute predetermined tasks semi-autonomously, one or more operator terminals, and a server connected to the robot systems and the operator terminals, The server, a support request receiving unit that receives a support request from the robot system; a priority order determination unit that determines a priority order of the support requests when a plurality of the support requests are received; an information providing unit that provides information regarding the support request and information regarding the priority order to the operator terminal; The priority order determination unit further a required time calculation unit that calculates, for each of the robot systems that have made the support request, a first sum value that is a sum value of required times of tasks subsequent to a task to be executed by the robot system; The server further comprises: a sum calculation unit that calculates a sum of the first sum values ​​for each combination of the support request and the operator corresponding to the operator terminal, The information providing unit further provides information regarding the support request to the operator terminal based on the combination that minimizes the sum.

11. A system including a plurality of robot systems that execute predetermined tasks semi-autonomously, one or more operator terminals, and a server connected to the robot systems and the operator terminals, a support request receiving unit that receives a support request from the robot system; a priority order determination unit that determines a priority order of the support requests when a plurality of the support requests are received; an information providing unit that provides information regarding the support request and information regarding the priority order to the operator terminal; The priority order determination unit further a required time calculation unit that calculates, for each of the robot systems that have made the support request, a first sum value that is a sum value of required times of tasks subsequent to a task to be executed by the robot system; The server further comprises: a support authority storage unit that stores, for each operator corresponding to each of the operator terminals, support authority for the robot system, the task, and / or a part of operations constituting the task; a sum calculation unit that calculates a sum of the first sum values ​​for each combination of the support request and an operator corresponding to the operator terminal having the support authority for the support request, The information providing unit further provides information regarding the support request to the operator terminal based on the combination that minimizes the sum.

12. A method for controlling a plurality of robot systems that semi-autonomously execute a predetermined task, one or more operator terminals, and a server connected thereto, comprising the steps of: a support request receiving step of receiving a support request from the robot system; a priority order determination step of determining a priority order of the support requests when a plurality of the support requests are received; an information providing step of providing information regarding the support request and information regarding the priority to the operator terminal; The priority determination step further comprises: a required time calculation step of calculating, for each of the robot systems that have made the support request, a first sum value that is a sum value of required times of tasks subsequent to a task to be executed by the robot system; The control method further comprises: a sum calculation step of calculating a sum of the first sum values ​​for each combination of the support request and the operator corresponding to the operator terminal; The information providing step further comprises providing information regarding the support request to the operator terminal based on the combination that minimizes the sum.

13. A method for controlling a plurality of robot systems that semi-autonomously execute a predetermined task, one or more operator terminals, and a server connected thereto, comprising the steps of: a support request receiving step of receiving a support request from the robot system; a priority order determination step of determining a priority order of the support requests when a plurality of the support requests are received; an information providing step of providing information regarding the support request and information regarding the priority to the operator terminal; The priority determination step further comprises: a required time calculation step of calculating, for each of the robot systems that have made the support request, a first sum value that is a sum value of required times of tasks subsequent to a task to be executed by the robot system; The control method further comprises: a support authority storing step of storing, for each operator corresponding to each of the operator terminals, support authority for the robot system, the task, and / or a part of operations constituting the task; a sum calculation step of calculating a sum of the first sum values ​​for each combination of the support request and an operator corresponding to the operator terminal having the support authority for the support request, The information providing step further comprises providing information regarding the support request to the operator terminal based on the combination that minimizes the sum.

14. A control program for a server connected to a plurality of robot systems that semi-autonomously execute a predetermined task, and one or more operator terminals, a support request receiving step of receiving a support request from the robot system; a priority order determination step of determining a priority order of the support requests when a plurality of the support requests are received; an information providing step of providing information regarding the support request and information regarding the priority to the operator terminal; The priority determination step further comprises: a required time calculation step of calculating, for each of the robot systems that have made the support request, a first sum value that is a sum value of required times of tasks subsequent to a task to be executed by the robot system; The control program further comprises: a sum calculation step of calculating a sum of the first sum values ​​for each combination of the support request and the operator corresponding to the operator terminal; a control program for providing information to the operator terminal based on the combination that minimizes the sum in the information providing step;

15. A control program for a server connected to a plurality of robot systems that semi-autonomously execute a predetermined task, and one or more operator terminals, a support request receiving step of receiving a support request from the robot system; a priority order determination step of determining a priority order of the support requests when a plurality of the support requests are received; an information providing step of providing information regarding the support request and information regarding the priority to the operator terminal; The priority determination step further comprises: a required time calculation step of calculating, for each of the robot systems that have made the support request, a first sum value that is a sum value of required times of tasks subsequent to a task to be executed by the robot system; The control program further comprises: a support authority storing step of storing, for each operator corresponding to each of the operator terminals, support authority for the robot system, the task, and / or a part of operations constituting the task; a sum calculation step of calculating a sum of the first sum values ​​for each combination of the support request and an operator corresponding to the operator terminal having the support authority for the support request, The information providing step further includes providing information regarding the support request to the operator terminal based on the combination that minimizes the sum.

Citation Information

Patent Citations

  • Method and alarm system for maintenance and / Or removal of trouble

    JP2003193342A

  • Automatic production system

    JP2014048685A

  • Robot control method

    JP2016068161A

  • Production support system and production support method

    JP2019016134A

  • Production management device and production management method

    JP2020095296A