Robot system and control method for robot system

Abstract

To provide a robot system and the like that allow an operator to more readily take actions in response to an unintended operation performed by a robot.SOLUTION: A robot system comprises: a robot; a first processing circuit for controlling the robot; a second processing circuit for collecting and accumulating log data related to a state of the robot; a first memory for storing a virtual space model for reproducing the robot and a surrounding environment; a third processing circuit for outputting data representing the virtual space model to a display device; and a fourth processing circuit for detecting an abnormality in operation of the robot. If the abnormality is detected, the third processing circuit applies the log data to the virtual space model to reproduce, in the virtual space model, the operation of the robot which is in accordance with the log data, and outputs, to the display device, data representing the reproduced virtual space model.SELECTED DRAWING: Figure 4

Description

Technical Field

[0001] The present disclosure relates to a robot system and a method for controlling the robot system.

Background Art

[0002] Patent Document 1 discloses an interference determination device for a robot. Based on an operator's operation input, the interference determination device determines whether interference occurs in a virtual space among a 3D model of the robot, a workpiece, and surrounding fixtures during the movement of the robot, using 3D CAD representing approximate shapes of the robot, the workpiece, and the surrounding fixtures.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] Patent Document 1 discloses a technique for preventing interference of a robot or the like in advance. For example, in an environment where an operator cannot directly visually recognize the robot, when the robot performs an unintended operation such as interfering with another object and stopping, it is difficult for the operator to eliminate such a situation by operating the robot.

[0005] An object of the present disclosure is to provide a robot system and a method for controlling the robot system that facilitate an operator's response when the robot performs an unintended operation.

Means for Solving the Problems

[0006] A robot system according to an aspect of the present disclosure includes a robot, a first processing circuit that controls the robot, a second processing circuit that collects and stores log data including information related to the state of the robot in the process of operating the robot according to an operation program, a first storage that stores a virtual space model including a virtual robot model and a virtual surrounding environment model that reproduce the robot and the surrounding environment of the robot, a third processing circuit that outputs data representing the virtual space model to a display device, and a fourth processing circuit that monitors the robot and detects an abnormality in the operation of the robot. When an abnormality is detected by the fourth processing circuit, the third processing circuit applies the log data accumulated by the second processing circuit to the virtual space model, reproduces the operation of the robot according to the log data in the virtual space model, and outputs data representing the virtual space model that has received the reproduction according to the log data to the display device.

Brief Description of the Drawings

[0007]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Mode for Carrying Out the Invention

[0008] Hereinafter, exemplary embodiments of the present disclosure will be described with reference to the drawings. Each of the embodiments described below shows either a comprehensive or a specific example. Among the components in the following embodiments, components not described in the independent claims indicating the highest-level concept are described as optional components. Each figure in the accompanying drawings is a schematic diagram and is not necessarily drawn precisely. In each figure, substantially the same components are denoted by the same reference numerals, and redundant descriptions may be omitted or simplified. In this specification and the claims, the term "apparatus" can mean not only a single apparatus but also a system including a plurality of apparatuses. Hereinafter, a robot will be exemplified as a mechanical apparatus that operates based on an operation program.

[0009] Referring to FIGS. 1 and 2, a robot system 100 according to an exemplary embodiment will be described. FIG. 1 is a diagram showing an example of a robot system 100 according to an exemplary embodiment. FIG. 2 is a diagram showing an extraction of one second server 20, one first processing terminal 30, one robot 40, one remote control device 50, and one second processing terminal 60 in the robot system 100 of FIG. 1. The robot system 100 includes a first server 10, a second server 20, a first processing terminal 30, a robot 40, and a remote control device 50. The first server 10 is communicably connected to the second server 20 via a first communication network NA. The first server 10 may be communicably connected to the first processing terminal 30 via the first communication network NA. The second server 20 is communicably connected to the first processing terminal 30 via the first communication network NA.

[0010] There are a plurality of robot operation areas A, and in each of the plurality of robot operation areas A, one or more first processing terminals 30 and one or more robots 40 are arranged. The robot operation area A is an area where one or more robots 40 provide services or perform operations. Examples of the robot operation area A include an area set within a building or facility, and an area set based on a map.

[0011] The first processing terminal 30 controls the operation of the robot 40. The first processing terminal 30 may target one robot 40 for control, or two or more robots 40 for control. The first processing terminal 30 may be mounted on the robot 40, or may be arranged at a position separated from the robot 40 and connected to the robot 40 via wired communication, wireless communication, or a combination thereof. The first processing terminal 30 may be communicably connected to the robot 40 via a communication network. Any wired communication, wireless communication, and communication network may be used.

[0012] The first processing terminal 30 is configured to autonomously cause the robot 40 to perform a predetermined task according to a predetermined computer program. That is, the first processing terminal 30 is configured to cause the robot 40 to perform autonomous operation. The robot 40 operates according to a command generated by the first processing terminal 30 using a predetermined computer program. The first processing terminal 30 may also be configured to cause the robot 40 to operate according to a manual operation manually input by an operator to a control device arranged away from the robot 40 in order to cause the robot 40 to perform an intended operation. That is, the first processing terminal 30 may be configured to cause the robot 40 to perform manual operation. In the present embodiment, the first processing terminal 30 can cause the robot 40 to perform autonomous operation and manual operation.

[0013] The robot 40 may be a robot having a structure suitable for any application such as work, handling, transportation, caregiving, medical treatment, cleaning, security, guidance, rescue, cooking, and product provision in a factory or a warehouse. The robot 40 may have a structure that is used in a fixed state, a structure that moves when an external force is applied, or a structure including moving means that moves autonomously. Examples of the moving means may include wheels, crawlers, propelling rotors, flying rotors, jet propulsion devices, and movable legs. The robot 40 may include one or more articulated robot arms. The robot 40 may include one or more articulated robot legs. The robot 40 may have a form such as an industrial robot, a humanoid robot, or an animal-shaped robot.

[0014] There are a plurality of operating area groups AG. The second server 20 is arranged in each of the operating area groups AG. The operating area group AG is set for a specific area including a plurality of robot operating areas A. The second server 20 manages all the robot operating areas A within the operating area group AG to which the second server 20 is associated. The second server 20 is communicably connected to a plurality of first processing terminals 30 within the plurality of robot operating areas A to be managed via the second communication network NB.

[0015] The second server 20 may be communicably connected to a second processing terminal 60 operated by a user who manages the robot operating area A to be managed via the second communication network NB. The above user may be a user who manages the robot 40 and the first processing terminal 30 in one or more robot operating areas A. The second server 20 is communicably connected to a remote control device 50 via the second communication network NB or another communication network. The second server 20 is also called an edge server.

[0016] The second server 20 includes a processing circuit 20a and has the functions of a computer. The processing circuit 20a includes a processor and a memory. The second server 20 includes storage either within the processing circuit 20a or separately from the processing circuit 20a. The memory and the storage are collectively referred to as a storage device 20b. The processing circuit 20a has the functions of a second processing circuit, a third processing circuit, a fifth processing circuit, and a sixth processing circuit, and may further have the function of a fourth processing circuit. The storage device 20b may have the functions of a first storage device, a second storage device, and a third storage device.

[0017] The processor executes functions, methods, or combinations thereof realized by code or instructions included in a program stored in the storage. Examples of the processor may include a central processing unit (CPU), a micro processing unit (MPU), a graphics processing unit (GPU), a microprocessor, a processor core, a multiprocessor, an application-specific integrated circuit (ASIC), and a field programmable gate array (FPGA), etc. The processor may realize each process of the present disclosure by a logic circuit formed in an integrated circuit (IC (Integrated Circuit) chip, LSI (Large Scale Integration)), or a dedicated circuit. These circuits may be realized by one or more integrated circuits. A plurality of processes may be realized by one integrated circuit.

[0018] The memory temporarily stores a program loaded from the storage and provides a working area for the processor. Various data generated while the processor is executing the program are also temporarily stored in the memory. Examples of the memory include semiconductor memories such as random access memory (RAM) and read only memory (ROM).

[0019] The storage stores programs and various data. Examples of storage include HDD (Hard Disk Drive), SSD (Solid State Drive), and flash memory.

[0020] For example, the storage stores information of a plurality of robot operation areas A. Each of the robot operation areas A is a robot operation area A under the management of the second server 20 and is registered with the second server 20. The storage stores the information of the first processing terminal 30 and the robot 40 arranged in each of the plurality of robot operation areas A in association with the robot operation area A including the first processing terminal 30 and the robot 40. Each of the first processing terminal 30 and the robot 40 is the first processing terminal 30 and the robot 40 arranged in each of the robot operation areas A registered with the second server 20 and is registered with the second server 20.

[0021] The storage may store the information of the second processing terminal 60 in association with the robot operation area A related to the second processing terminal 60. The second processing terminal 60 is the second processing terminal 60 of the user who manages the robot operation area A registered with the second server 20 and is registered with the second server 20. The second server 20 may be communicably connected to the first processing terminal 30 and the second processing terminal 60 registered with the second server 20 via the second communication network NB.

[0022] The storage stores the information of the remote control device 50. The remote control device 50 is one or more remote control devices 50 set for the operation area group AG under the management of the second server 20 and is registered with the second server 20. The second server 20 may be communicably connected to the remote control device 50 registered with the second server 20 via the second communication network NB.

[0023] The storage stores a virtual space model VAM that forms a virtual space of the robot operation area A and a virtual space program that operates the virtual space model VAM. For example, the virtual space model VAM may be a 3DCG (3 Dimensional Computer Graphics) model. The virtual space model VAM is a model that faithfully reproduces the real-world robot operation area A and includes virtual components corresponding to various components included in the real-world robot operation area A. For example, when information or data regarding real components acquired from the real-world robot operation area A is applied to the virtual space model VAM, the virtual components corresponding to the real components in the virtual space model VAM can be made to behave in the same manner as the real components.

[0024] By executing the virtual space program, the processing circuit 20a can apply various data to the virtual space model VAM, add and delete components to and from the virtual space model VAM, and operate on the components within the virtual space model VAM. The virtual space model VAM includes a virtual robot model of the robot 40 within the robot operation area A and a peripheral environment model of elements other than the robot 40. In the present embodiment, the virtual space model VAM is a virtual space model formed using digital twin technology to faithfully reproduce the robot operation area A for each of the robot operation areas A registered in the second server 20.

[0025] The virtual space model VAM may be generated by the processing circuit 20a or acquired by the processing circuit 20a from outside the second server 20. The update of the virtual space model VAM by making changes to the virtual space model VAM may be performed by the processing circuit 20a or by an external entity outside the second server 20. The processing circuit 20a may be configured to receive the updated virtual space model VAM from outside the second server 20.

[0026] The storage may store a language model LM, which is a machine learning model that outputs text data indicating the state represented by the image data, using the image data as input data. In the present embodiment, the language model LM is a large language model (LLM: Large Language Models). The language model LM may be generated by the processing circuit 20a or acquired by the processing circuit 20a from outside the second server 20. The machine learning of the language model LM may be performed by the processing circuit 20a or by an entity outside the second server 20. The processing circuit 20a may be configured to receive the language model LM after machine learning from outside the second server 20.

[0027] The storage stores various robot programs RP for autonomously executing tasks by the robot 40. The robot programs RP are generated for each task corresponding to various tasks. The robot programs RP can be generated corresponding to various robots 40 within the robot operation area A registered in the second server 20. When robots 40 with different structures perform the same task, the robot programs RP may or may not be the same. The robot programs RP generated for each task corresponding to various robots 40 are stored in the storage.

[0028] Examples of the second communication network NB may include a LAN (Local Area Network), local 5G, or other access networks. The communication distance between the second server 20 and the first processing terminal 30 via the second communication network NB is extremely smaller than the communication distance between the first server 10 and the first processing terminal 30 via the first communication network NA and the communication distance between the first server 10 and the second server 20 via the first communication network NA. That is, the communication distance via the second communication network NB is extremely smaller than the communication distance via the first communication network NA. Therefore, communication via the second communication network NB enables high-speed transmission and reception of a large amount of data.

[0029] The first server 10 is communicably connected to a plurality of second servers 20 via a first communication network NA. The first server 10 is also called a cloud server. The first server 10 may be communicably connected to a plurality of first processing terminals 30 via the first communication network NA. The first server 10 may be communicably connected to a plurality of second processing terminals 60 via the first communication network NA.

[0030] The first server 10 includes a processing circuit 10a and has the functions of a computer. The processing circuit 10a includes a processor and a memory. The first server 10 includes storage, either within the processing circuit 10a or separate from the processing circuit 10a. The memory and the storage are collectively referred to as a storage device 10b.

[0031] Examples of the processor, memory, and storage of the first server 10 are similar to those of the processor, memory, and storage of the second server 20. The functions of the processor, memory, and storage of the first server 10 are similar to those of the processor, memory, and storage of the second server 20.

[0032] The storage of the first server 10 stores user information regarding the users of the first processing terminals 30 and the users of the second processing terminals 60. The storage target of the user information is the first processing terminals 30 and the second processing terminals 60 related to all the robot operation areas A managed by all the second servers 20 connected to the first server 10. That is, the storage target of the user information is all of the first processing terminals 30 and the second processing terminals 60 registered in all the second servers 20. The user information may include one or more of the user ID, which is the identification information of the user, the login password to the system with the user ID, the information of the first processing terminal 30 registered for the user ID, the information of the robot 40 controlled by the first processing terminal 30, the information of the second processing terminal 60 registered for the user ID, and the information of the robot operation area A registered for the user ID.

[0033] The storage of the first server 10 stores area information regarding the robot operation area A. The robot operation area A to be stored is all the robot operation areas A that all the second servers 20 connected to the first server 10 manage. The area information of each robot operation area A may include one or more of the information of the second server 20 that manages the robot operation area A, the information of the first processing terminal 30 arranged in the robot operation area A, the information of the second processing terminal 60 arranged in the robot operation area A, and the information of the robot 40 arranged in the robot operation area A.

[0034] The storage of the first server 10 stores information on the program type of the robot program RP. The information on the program type is information representing the type of the robot program RP classified according to one or more of the content of the task executed by the robot program RP and the information of the robot 40 capable of executing the robot program RP. For example, the program type can be represented as the type of the object transportation task by a bipedal robot, the type of the bipedal robot, or the type of the object transportation task, etc.

[0035] The first communication network NA is a network that communicates with an area wider than the second communication network NB. Examples of the first communication network NA may include open networks such as the Internet and Internet VPN (Virtual Private Network), closed networks such as IP-VPN (Internet Protocol Virtual Private Network) and dedicated lines, or combinations thereof.

[0036] The first processing terminal 30 includes a processing circuit 30a and has the functions of a computer. The processing circuit 30a includes a processor and a memory. The first processing terminal 30 may include storage within the processing circuit 30a or separately from the processing circuit 30a. The memory and the storage are collectively referred to as a storage device 30b. The processing circuit 30a has the functions of the first processing circuit and may further have the functions of the fourth processing circuit.

[0037] Examples of the processor, memory, and storage of the first processing terminal 30 are the same as those of the processor, memory, and storage of the second server 20. The functions of the processor, memory, and storage of the first processing terminal 30 are the same as those of the processor, memory, and storage of the second server 20.

[0038] Examples of the first processing terminal 30 may include smart devices such as an electronic circuit board, an electronic control unit, a microcomputer, a desktop computer, a laptop computer, and a tablet. The first processing terminal 30 may be any of a processing terminal dedicated to the robot system 100, a processing terminal dedicated to the robot 40, a processing terminal dedicated to a system other than the robot system 100 such as a system used in the robot operation area A, and a general-purpose processing terminal. For example, the first processing terminal 30 may be a robot controller or a part thereof as a processing terminal dedicated to the robot 40. The first processing terminal 30 is communicably connected to the robot 40 via wired communication, wireless communication, or a combination thereof. The first processing terminal 30 may be disposed away from the robot 40 or mounted on the robot 40.

[0039] The first processing terminal 30 is communicably connected to the second communication network NB via wired communication, wireless communication, or a combination thereof, and is communicably connected to the second server 20 via the second communication network NB. The first processing terminal 30 may be communicably connected to the first communication network NA via wired communication, wireless communication, or a combination thereof, and may be communicably connected to the first server 10 via the first communication network NA. Any wired communication and wireless communication may be used. The first processing terminal 30 may be connected to either the first communication network NA or the second communication network NB via one or more of a regional IP network, a mobile phone network such as 4G or 5G, a private network, a wired LAN, and a wireless LAN such as wireless WiFi.

[0040] The memory 30b stores an operation system program OS1 for operating the first processing terminal 30 and an application program AP1 (hereinafter also referred to as "app AP1") that operates on the operation system program OS1. The operation system program OS1 may be either a program dedicated to the robot system 100 or a program not dedicated to the robot system 100. The app AP1 is a program dedicated to the robot system 100. The app AP1 may be constructed as an operation system program and incorporated into the first processing terminal 30.

[0041] The app AP1 is incorporated into the first processing terminal 30 by being installed. The app AP1 has a function of generating a command for operating the robot 40 connected to the first processing terminal 30. The app AP1 has a function of operating the robot 40 in an automatic control mode and a manual control mode. In the automatic control mode, the app AP1 functions using the robot program RP.

[0042] The processing circuit 30a captures the robot program RP sent from the second server 20 by executing the automatic control mode by the app AP1, and executes the robot program RP on the app AP1.

[0043] The robot program RP includes operation commands that command various target operations to be performed by the robot 40 in the process of executing the tasks specified in the robot program RP, in accordance with the execution order. For example, the operation commands may include commands for the target positions and target postures of each part of the robot 40. The processing circuit 30a generates drive commands that command the target operations of the drive part of the robot 40 in accordance with the execution order, in order to sequentially cause the robot 40 to perform operations according to the operation commands of the robot program RP, and transmits them to the robot 40. The drive commands are commands corresponding to the structure of the robot 40, and may include, for example, commands for the target operations of the joints included in the robot 40.

[0044] The processing circuit 30a causes the robot 40 to perform operations according to commands for manually operating the robot 40, which are output from a controller of the robot 40 such as the remote control device 50, by executing the manual operation mode by the application AP1. For example, the operator moves an operator of the controller in order to cause the robot 40 to perform an intended operation. The controller outputs a command for causing the robot 40 to perform an operation corresponding to the movement of the operator. For example, the controller outputs a command for causing the robot 40 to move in the direction and at the speed indicated by the movement of the operator. The processing circuit 30a generates drive commands for causing the robot 40 to perform the operations of the commands received from the controller, and transmits them to the robot 40.

[0045] The processing circuit 30a acquires from the robot 40 the operation results of the drive part of the robot 40 and the detection results of the sensors provided in the robot 40, and transmits the operation results and the detection results as log data to the second server 20. Further, the processing circuit 30a may transmit the execution results of the robot program RP as log data to the second server 20 in association with the operation results and the detection results.

[0046] For example, the processing circuit 30a may acquire the operation result and the detection result at a predetermined time interval. Examples of the operation result of the drive part may include the output value or current value of the drive device that drives the drive part, and the detection value of the sensor that detects the drive amount of the drive part. Examples of the sensors included in the robot 40 may include a camera, an acceleration sensor, a gyro sensor, a position sensor, a speed sensor, a distance measuring sensor, a tactile sensor, a force sensor, a force-tactile sensor, a vibration sensor, and a temperature sensor.

[0047] The first processing terminal 30 may include an input device 31 that receives an input from a user or the like. Examples of the input device 31 may include a button, a lever, a dial, a joystick, a mouse, a key, a touch panel, a microphone, and a motion capture. For example, the input device 31 receives a command requesting a task for the robot 40 from a user or the like, and outputs the command to the processing circuit 30a. The processing circuit 30a may be configured to send the content of the commanded task to the first server 10 via the first communication network NA. The content of the task may include, as task elements, a task subject, a task type, a task object, and a task target to be achieved by the task.

[0048] For example, the task subject is the robot that executes the task. The task type corresponds to the type of operation to be executed by the robot 40, such as a transportation operation and an assembly operation. The task object is the object on which the robot 40 acts by the task. The task target corresponds to the target state of the object to be reached by the execution of the task.

[0049] The input device 31 may receive the input of the task elements by selection by the user's hand or voice or language input. The processing circuit 30a may send the information of the selected task elements or the character string data representing the input language to the first server 10. The processing circuit 30a may send either or both of the identification information of the first processing terminal 30 and the information of the robot 40 that executes the task to the first server 10 together with the above information or data.

[0050] The processing circuit 10a of the first server 10 determines the program type to be executed by the robot 40 using the information received from the first processing terminal 30, and transmits information such as the determined program type to the second server 20 in which the first processing terminal 30 is registered. The processing circuit 20a of the second server 20 extracts the robot program RP corresponding to the program type from the storage 20b and transmits it to the first processing terminal 30. The processing circuit 30a of the first processing terminal 30 operates the robot 40 according to the received robot program RP.

[0051] The second processing terminal 60 includes a processing circuit 60a and has the functions of a computer. The processing circuit 60a includes a processor and a memory. The second processing terminal 60 may include storage within the processing circuit 60a or separately from the processing circuit 60a. The memory and storage are collectively referred to as a storage 60b.

[0052] Examples of the processor, memory, and storage of the second processing terminal 60 are the same as those of the processor, memory, and storage of the second server 20. The functions of the processor, memory, and storage of the second processing terminal 60 are the same as those of the processor, memory, and storage of the second server 20.

[0053] Examples of the second processing terminal 60 may include smart devices such as smartphones, smartwatches, and tablets, desktop computers, and laptop computers. The second processing terminal 60 may be either a processing terminal dedicated to the robot system 100, a processing terminal dedicated to the robot 40, a processing terminal dedicated to a system other than the robot system 100 such as a system used in the robot operation area A, or a general-purpose processing terminal. For example, the second processing terminal 60 may be a processing terminal owned by a user as a general-purpose processing terminal. The second processing terminal 60 is arranged away from the robot 40.

[0054] The second processing terminal 60 is communicably connected to the first communication network NA via wired communication, wireless communication, or a combination thereof, and is communicably connected to the first server 10 via the first communication network NA. The second processing terminal 60 may be communicably connected to the second communication network NB via wired communication, wireless communication, or a combination thereof, and may be communicably connected to the second server 20 via the second communication network NB. Any wired communication and wireless communication may be used. The second processing terminal 60 may be connected to either the first communication network NA or the second communication network NB via one or more of a regional IP network, a mobile phone network such as 4G or 5G, a private network, and a wireless LAN.

[0055] The second processing terminal 60 is communicably connected to the first processing terminal 30 via wired communication, wireless communication, or a combination thereof. The second processing terminal 60 may be configured to be communicably connected to one first processing terminal 30, or may be configured to be communicably connected to two or more first processing terminals 30.

[0056] The second processing terminal 60 includes an input device 61 that receives input from a user or the like. Examples of the input device 61 may include buttons, levers, dials, joysticks, mice, keys, touch panels, microphones, and motion capture. For example, the input device 61 receives a command from a user or the like to request a task from the robot 40 and outputs the command to the processing circuit 60a. The processing circuit 60a sends the content of the commanded task to the first server 10 via the first communication network NA. The content of the task may include the task subject, task type, task object, and task target as task elements.

[0057] The input device 61 may receive the input of task elements by selection by the user's hand or voice or by language input. The processing circuit 60a may transmit information on the selected task element or character string data representing the input language to the first server 10. The processing circuit 60a may transmit either or both of the identification information of the second processing terminal 60 and the information of the robot 40 for executing the task to the first server 10 together with the above information or data.

[0058] The processing circuit 10a of the first server 10 determines the program type to be executed by the robot 40 using the information received from the second processing terminal 60, and transmits information on the determined program type, etc. to the second server 20 in which the robot 40 is registered. The processing circuit 20a of the second server 20 extracts the robot program RP corresponding to the information on the program type from the storage device 20b and transmits it to the first processing terminal 30. The processing circuit 30a of the first processing terminal 30 operates the robot 40 according to the received robot program RP.

[0059] The storage device 60b stores an operation system program OS2 for operating the second processing terminal 60 and an application program AP2 (hereinafter also referred to as "app AP2") that operates on the operation system program OS2. The operation system program OS2 may be either a program dedicated to the robot system 100 or a program not dedicated to the robot system 100. The app AP2 is a program dedicated to the robot system 100. The app AP2 may be constructed as an operation system program and incorporated into the second processing terminal 60.

[0060] App AP2 is incorporated into the second processing terminal 60 by being installed. App AP2 has a function of mutually transmitting and receiving commands, information, and data with the first server 10 and the first processing terminal 30. App AP2 may have a function of mutually transmitting and receiving commands, information, and data with the second server 20. App AP2 has a function of receiving a task subject, a task type, a task object, and a task target, which are task elements of a task to be executed by the robot 40, and transmitting information on the received task elements to the first server 10. App AP2 may have a function of transmitting a command for causing the robot 40 to execute an operation not based on the robot program RP, such as starting, stopping, emergency stopping, and moving to a standby position of the robot 40, to the first processing terminal 30.

[0061] The second processing terminal 60 may be configured to access a website constructed for the robot system 100. The second processing terminal 60 may be configured to mutually transmit and receive commands, information, and data with the first server 10 via the website. The second processing terminal 60 may receive an input of task elements of a task to be executed by the robot 40 on the website, and the first server 10 may be configured to receive information on the task elements via the website. In this case, App AP2 may not be incorporated into the second processing terminal 60.

[0062] The remote control device 50 includes a control processing terminal 51, an input device 52, a display 53, a camera 54, and an acoustic device 55. The control processing terminal 51 includes a processing circuit 51a and has a function of a computer. The processing circuit 51a includes a processor and a memory. The control processing terminal 51 may include a storage inside the processing circuit 51a or separately from the processing circuit 51a. The memory and the storage are collectively referred to as a storage device 51b. The remote control device 50 is an example of a display device.

[0063] Examples of the processor, memory, and storage of the operation processing terminal 51 are the same as those of the processor, memory, and storage of the second server 20. The functions of the processor, memory, and storage of the operation processing terminal 51 are the same as those of the processor, memory, and storage of the second server 20.

[0064] Examples of the operation processing terminal 51 may include smart devices such as an electronic circuit board, an electronic control unit, a microphone computer, a desktop computer, a laptop computer, and a tablet. The operation processing terminal 51 may be either a processing terminal dedicated to the robot system 100 or a general-purpose processing terminal.

[0065] The storage 51b stores an operation system program OS3 for operating the operation processing terminal 51 and an application program AP3 (hereinafter also referred to as "app AP3") that operates on the operation system program OS3. The operation system program OS3 may be either a program dedicated to the robot system 100 or a program not dedicated to the robot system 100. The app AP3 is a program dedicated to the robot system 100. The app AP3 may be constructed as an operation system program and incorporated into the operation processing terminal 51. The app AP3 has a function of mutually transmitting and receiving commands, information, and data with the second server 20.

[0066] The remote control device 50 is communicably connected to the second communication network NB via wired communication, wireless communication, or a combination thereof, and is communicably connected to the second server 20 via the second communication network NB. In the present embodiment, the remote control device 50 can be connected to the second server 20 of one operation area group AG, but may also be connectable to the second servers 20 of two or more operation area groups AG. Any wired communication and wireless communication may be used. The remote control device 50 may be connected to the second communication network NB via one or more of a regional IP network, a mobile phone network such as 4G or 5G, a private network, a wired LAN, and a wireless LAN.

[0067] The remote control device 50 is arranged at a position away from the robot operation area A registered in the second server 20 connected to the remote control device 50, for example, at a position where the operator of the remote control device 50 cannot directly view the robot operation area A. The remote control device 50 can operate the robot 40 permitted by the second server 20 in response to a request from the second server 20. The remote control device 50 is connected to the first processing terminal 30 that controls the permitted robot 40 by the second server 20.

[0068] The input device 52 receives an input from the operator and outputs a command received from the operator to the operation processing terminal 51. Examples of the input device 52 may include buttons, levers, dials, joysticks, mice, keys, touch panels, microphones, and motion captures.

[0069] For example, the input device 52 receives a manual operation manually input by the operator to make the robot 40 perform an intended operation. The input device 52 outputs a command indicating the content of the manual operation to the operation processing terminal 51. For example, when the operator manually operates the joystick of the input device 52, the input device 52 may output a command indicating the tilting direction and tilting angle of the joystick to the operation processing terminal 51.

[0070] The processing circuit 51a of the operation processing terminal 51 processes the command received from the input device 52, generates a motion command for commanding the target motion to be executed by the robot 40, and transmits it to the second server 20. When the command received from the input device 52 is a command indicating the tilting direction and tilting angle of the joystick, the processing circuit 51a may generate a motion command for commanding the target direction and target speed of the motion of the robot 40. The tilting direction of the joystick corresponds to the target direction of the motion of the robot 40, and the tilting angle of the joystick may represent the target speed of the motion of the robot 40 in that direction.

[0071] The second server 20 transmits a movement command to the first processing terminal 30 that controls the robot 40 targeted by the movement command. The first processing terminal 30 generates a drive command for commanding the target movement of the drive part of the robot 40 in order to cause the robot 40 to execute an operation according to the movement command, and transmits it to the robot 40.

[0072] The display 53 converts the image signal sent from the operation processing terminal 51 into an image and displays it. In this specification and the claims, the image includes still images and moving images. For example, the processing circuit 51a of the operation processing terminal 51 transmits a signal representing the operation screen of the robot system 100 and an image signal received from the second server 20 to the display 53. For example, as the above signal, the processing circuit 51a receives from the second server 20 a signal representing one or more of an image of a map of the robot operation area A where the first processing terminal 30 connected to the operation processing terminal 51 is arranged, an image captured by a camera mounted on the robot 40 controlled by the first processing terminal 30, and an image captured by a camera arranged in the robot operation area A. The remote control device 50 may include a projector that displays an image in the same manner as the display 53 instead of or in addition to the display 53.

[0073] The camera 54 captures an image of the operator of the remote control device 50 and outputs a signal of the captured image to the processing circuit 51a of the operation processing terminal 51. The processing circuit 51a transmits the signal to the second server 20. The second server 20 transmits the signal to the first processing terminal 30 connected to the operation processing terminal 51, the second processing terminal 60 in the same robot operation area A as the first processing terminal 30, or both. The first processing terminal 30 and the second processing terminal 60 cause the image represented by the received signal to be displayed on a display or a projector that they may each include.

[0074] The audio device 55 includes one or more of a sound collector such as a microphone and a voice output device such as a speaker, earphone, or headphones. The sound collector collects the voice of the operator of the remote control device 50, converts it into a voice signal, and outputs it to the processing circuit 51a of the control processing terminal 51. The processing circuit 51a transmits the voice signal to the second server 20. The second server 20 transmits the voice signal to the first processing terminal 30 connected to the control processing terminal 51, the second processing terminal 60 in the same robot operation area A as the first processing terminal 30, or both of them. The first processing terminal 30 and the second processing terminal 60 cause the voice output devices they may each have to output the received voice signal as an audible sound.

[0075] The voice output device converts the voice signal sent from the control processing terminal 51 into an audible sound and outputs it. For example, the processing circuit 51a of the control processing terminal 51 transmits the voice signal received from the second server 20 to the voice output device. For example, as the above signal, the processing circuit 51a receives from the second server 20 a voice signal representing one or more of the voice collected by the sound collector provided in the first processing terminal 30 connected to the control processing terminal 51, the voice collected by the sound collector provided in the second processing terminal 60 in the same robot operation area A as the first processing terminal 30, and the voice collected by the sound collector arranged in the robot operation area A.

[0076] Therefore, the operator of the remote control device 50 can operate the robot 40 while communicating with the user who operates the first processing terminal 30 connected to the remote control device 50 or the second processing terminal 60 in the same robot operation area A as the first processing terminal 30.

[0077] With reference to FIG. 3, an example of the operation of the robot system 100 will be described. FIG. 3 is a flowchart showing an example of the operation of the robot system 100 according to the embodiment.

[0078] In step S101, the user in the robot operation area A1 inputs a command to make the robot 40A execute the task TA to the second processing terminal 60. Although not limited, in this example, the robot operation area A1 is one of the robot operation areas A and is a store that provides food and drinks to users. The user is the service provider in the store. The robot 40A is one of the robots 40 existing in the robot operation area A1. The task TA is to make the robot 40A carry a container filled with a beverage from the supply location to the table. The task elements of the task TA include the task subject TA1 which is the robot 40A, the task type TA2 including the receiving operation, transporting operation, and serving operation of the object, the task object TA3 which is the container filled with the beverage, and the task target TA4 including the starting position which is the supply location and the target position which is the table.

[0079] In step S102, the second processing terminal 60 transmits the input command and the user information including the user ID, etc. to the first server 10. The user ID can be set for the robot operation area A1, the second processing terminal 60, the robot 40A, or the first processing terminal 30A that controls the robot 40A. The first processing terminal 30A is one of the first processing terminals 30.

[0080] In step S103, the first server 10 authenticates the user using the received user ID, etc. If the first server 10 can authenticate the user, based on the received command and the user information stored in the memory 10b, it identifies the first processing terminal 30A that controls the robot 40A, the model of the robot 40A, and the robot operation area A1 where the robot 40A is arranged. Furthermore, the first server 10 identifies the second server 20A that manages the robot operation area A1 based on the area information stored in the memory 10b. The second server 20A is one of the second servers 20. If the first server 10 cannot authenticate the user, it ends a series of processes.

[0081] Furthermore, the first server 10 determines the program type of the robot program RP to be executed by the robot 40A based on the information on the program type of the robot program RP stored in the memory 10b. That is, the first server 10 determines the program type to be executed by the robot 40A and the first processing terminal 30A that executes the robot program RP, and determines the second server 20A as the transmission destination of this information.

[0082] In step S104, the first server 10 transmits a command to cause the first processing terminal 30A to execute the robot program RP to the second server 20A together with the information on the program type of the robot program RPA, the information on the first processing terminal 30A, and the information including the task target object TA3 and the task target TA4.

[0083] In step S105, the second server 20A extracts a robot program RPA that matches the program type and is executable by the robot 40A from the robot programs RP stored in the memory 20b. Furthermore, the second server 20A sets the task target object and the task target included in the robot program RPA to the task target object TA3 and the task target TA4. That is, the second server 20A sets task elements for the robot program RPA.

[0084] In step S106, the second server 20A transmits the robot program RPA with the task target object TA3 and the task target TA4 set thereto to the first processing terminal 30A designated by the first server 10.

[0085] In step S107, the first processing terminal 30A sequentially generates drive commands by executing the robot program RPA in which the task target object TA3 and the task target TA4 are set, and outputs them to the robot 40A. For example, the first processing terminal 30A generates drive commands at points where the direction or speed of the movement of the robot 40A changes on the movement trajectory of the robot 40A, and at points at a predetermined interval on the trajectory. The predetermined interval may be a time interval or a distance interval. The robot 40A sequentially performs the task TA according to the task elements included in the task TA by operating according to the drive commands.

[0086] In step S108, each time the robot 40A operates according to the drive command, the first processing terminal 30A acquires, as log data, the operation result of the drive part of the robot 40A and the detection result of the sensor included in the robot 40A from the robot 40A, and transmits it to the second server 20. The second server 20 accumulates the received log data in the storage 20b.

[0087] In step S109, when the first processing terminal 30A determines that the robot program RPA has ended (Yes in step S109), it ends the series of processes. The first processing terminal 30A may transmit information on the completion of the task TA to the second processing terminal 60. When the first processing terminal 30A determines that the robot program RPA is being executed (No in step S109), it returns to step S107.

[0088] In this way, even if the first processing terminal 30 does not have a robot program RP capable of executing a requested task, it can cause the robot 40 to execute the task by obtaining the robot program RP from the second server 20. Since the second communication network NB connecting the first processing terminal 30 and the second server 20 can transmit and receive a large amount of data at high speed, it is possible to reduce the communication delay of a large amount of data such as the robot program RP. Since the management of information related to the user and the processing using the information are performed by the first server 10, confidentiality can be ensured. Since the processing amount at the first server 10 and the amount of data transmitted and received to and from the first server 10 are small, the communication delay can be reduced.

[0089] Also, in the present embodiment, when an abnormality occurs in the operation of the robot 40 according to the robot program RP, the first processing terminal 30 that controls the robot 40 switches the control mode from the automatic operation mode to the manual operation mode. Detection of an abnormality in the operation of the robot 40 is performed by the first processing terminal 30, the second server 20, or both of them.

[0090] For example, the processing circuit 30a of the first processing terminal 30 and the processing circuit 20a of the second server 20 may determine the presence or absence of an abnormality in the operation of the robot 40 based on one or more of the comparison result between the robot program RP and the log data of the robot 40, the detection result of the sensor provided in the robot 40, the detection result of the sensor arranged in the surrounding environment of the robot 40, and the voltage value or current value of a motor or the like which is a driving device provided in the robot 40.

[0091] For example, when the processing circuits 20a and 30a determine that the position or operation of the robot 40 indicated by the log data deviates from the target position or target operation of the robot 40 according to the robot program RP, they may determine that there is an abnormality in the operation of the robot 40. When the processing circuits 20a and 30a determine that the detected value of the sensor or its behavior indicates an abnormal state, they may determine that there is an abnormality in the operation of the robot 40. When the processing circuits 20a and 30a determine that the voltage value, current value, or behavior of the motor driving the robot 40 indicates an abnormal state, they may determine that there is an abnormality in the operation of the robot 40. When the processing circuit 30a of the first processing terminal 30 determines that there is an abnormality in the operation of the robot 40, it transmits the determination result to the second server 20.

[0092] Referring to FIG. 4, the operation of the robot system 100 according to an embodiment when an abnormality occurs in the operation of the robot 40 will be described. FIG. 4 is a flowchart showing an example of the operation of the robot system 100 according to an embodiment when an abnormality occurs in the operation of the robot 40. FIG. 4 shows an example in which the first processing terminal 30 detects an abnormality in the operation of the robot 40.

[0093] During the process from the following step S201 to S213, while the robot 40 is executing a task according to the robot program RP, the first processing terminal 30 acquires, at a predetermined timing, the operation results of the driving part such as the current value of the motor of the robot 40 and the detection results of the sensors provided in the robot 40 from the robot 40 as log data and accumulates them in the storage 30b. For example, the first processing terminal 30 acquires log data at a predetermined time interval. Further, the first processing terminal 30 transmits the log data to the second server 20, and the second server 20 stores the log data in the storage 20b.

[0094] In step S201, the first processing terminal 30 autonomously causes the robot 40 to execute a task by operating the robot 40 according to the robot program RP.

[0095] In step S202, the first processing terminal 30 determines whether there is an abnormality in the operation of the robot 40, that is, whether there is an abnormality in the robot 40. If the first processing terminal 30 determines that there is an abnormality (Yes in step S202), it proceeds to step S203. If it determines that there is no abnormality (No in step S202), it proceeds to step S204. The first processing terminal 30 determines the presence or absence of an abnormality using the information acquired as log data. The first processing terminal 30 may execute step S202 at a predetermined time interval.

[0096] In step S204, the first processing terminal 30 determines whether the robot program RP has been completed. If the first processing terminal 30 determines that the robot program RP has been completed (Yes in step S204), it ends a series of processes related to the task. If it determines that the robot program RP has not been completed (No in step S204), it returns to step S201.

[0097] In step S203, the first processing terminal 30 stops the operation of the robot 40 according to the robot program RP and transmits a report of the occurrence of an abnormality in the robot 40 to the second server 20.

[0098] In step S205, the second server 20 determines a remote control device 50 for dealing with the abnormality of the robot 40. For example, the second server 20 selects, from among the remote control devices 50 connected to the second server 20, a remote control device 50 suitable for the task of the robot program RP being executed by the first processing terminal 30 based on the information of the remote control device 50 stored in the storage 20b. Examples of the information of the remote control device 50 may include one or more of the type of the input device 52 of the remote control device 50, the task suitable for manual operation with the remote control device 50, the operator of the remote control device 50, the proficiency of the operator, the task that the operator is good at manual operation, and the operating state of the remote control device 50.

[0099] In step S206, the second server 20 transmits a command to deal with the abnormality of the robot 40 to the determined remote control device 50 and connects the remote control device 50 and the first processing terminal 30 to be communicable.

[0100] In step S207, the second server 20 instructs the first processing terminal 30 to switch the control from the automatic operation mode to the manual operation mode, and the first processing terminal 30 executes the control in the manual operation mode.

[0101] In step S208, the second server 20 reproduces the normal operation of the robot 40 according to the robot program RP in the virtual space model VAM2 of the robot operation area A2 stored in the memory 20b. The robot operation area A2 is the robot operation area A where the abnormal robot 40 is arranged. The second server 20 applies the information of the task element, the robot program RP, and the robot information stored in the memory 20b to the virtual space model VAM2. The second server 20 may use the virtual space model of the entire robot operation area A2 as the virtual space model VAM2, or may use the virtual space model of the operation area of the robot 40 according to the robot program RP.

[0102] The information of the task element includes information on the task subject of the robot 40, the task type, the task object, and the task target. The information of the task object may include one or more of the specifications information such as the shape, dimensions, weight, and material, the state information of the object, and the 3D model data of the object.

[0103] The robot information may include the body information of the robot 40, the information of the sensors mounted on the robot 40, and the environmental information. The body information of the robot 40 is information for identifying the robot 40 and may include identification information or model information. The environmental information may include one or more of the detailed information of the task object, the information of the obstacles, and the environmental condition information such as the constraints on the robot 40. The obstacles may be obstacles added after the construction of the virtual space model VAM2. The second server 20 may identify the obstacles using the detection results of the sensors arranged in the robot operation area A2, or may identify the obstacles using the information of the obstacles notified by the user via the second processing terminal 60.

[0104] The virtual space model VAM includes virtual object models of various movable objects, immovable objects, and living organisms existing within the robot operation area A, and further includes a virtual controller of the robot 40. The virtual controller is an engine that operates the virtual robot model by means of a robot program RP and a controller such as the remote control device 50, and can be realized by a robot simulator or a physics engine or the like.

[0105] Therefore, the second server 20 can reproduce, in the virtual space model VAM2, the normal operation according to the robot program RP in a state where there is no abnormality in the virtual robot model 40M of the robot 40 for the virtual robot model 40M of the robot 40, in the same manner as the real world.

[0106] In step S209, the second server 20 reproduces the actual operation of the robot 40 according to the log data in the virtual space model VAM2. Specifically, the second server 20 applies the log data and the robot information stored in the storage 20b to the virtual space model VAM2. The log data includes the log data of the robot 40 during actual operation as the operation result of the drive part of the robot 40. The log data of the robot 40 during actual operation includes one or more of the detection results of the sensors that detect the rotation angle of the joints of the drive part, the current value of the motor of the drive part, and the drive command value from the first processing terminal 30 to the robot 40. The log data may include the detection results of the external sensors of the robot 40 as the detection results of the sensors provided in the robot 40A. Examples of the external sensors may include force sensors, tactile sensors, force-tactile sensors, position sensors, distance sensors, vibration sensors, temperature sensors, and vision sensors such as cameras.

[0107] In the virtual space model VAM2, the second server 20 can cause the virtual robot model 40M of the robot 40 to operate in the same manner as the actual operation according to the log data of the robot 40 during actual operation. That is, the second server 20 can reproduce the actual operation of the robot 40 for the virtual robot model 40M. Further, in the virtual space model VAM2, the second server 20 can reflect the detection result of the external sensor on the virtual object model around the virtual robot model 40M or reflect it as environmental information around the robot model 40M. For example, the second server 20 can generate a virtual model of an obstacle that was not included in the virtual space model VAM2 or information about the obstacle. Thereby, the second server 20 can update the state of the virtual space model VAM2.

[0108] In step S210, the second server 20 transmits data representing the virtual space model VAM2 to the remote control device 50. Examples of such data may include data of a three-dimensional model of the virtual space model VAM2, data of a two-dimensional model of the virtual space model VAM2, image data obtained by imaging the three-dimensional model of the virtual space model VAM2 with a virtual camera included in the virtual space model VAM2, and image data obtained by imaging the two-dimensional model of the virtual space model VAM2 with a virtual camera of the virtual space model VAM2. The virtual camera will be described later. In this example, the second server 20 transmits image data representing the virtual space model VAM2 to the remote control device 50. The remote control device 50 displays an image representing the received image data on the display 53. The second server 20 may also display the above image on a local site or a web site that is accessible to the remote control device 50 and can receive input from the remote control device 50.

[0109] The second server 20 generates normal operation image data, which is image data representing the virtual space model VAM2 to which the robot program RP is applied in step S208, and actual operation image data, which is image data representing the virtual space model VAM2 to which the log data is applied in step S209. For both pieces of image data, the second server 20 generates them as data of a plurality of images or videos representing the change in the virtual space model VAM2 from a timing that is a specific time back from the detection timing at which the abnormality of the robot 40 was detected in step S202 to the detection timing. The normal operation image data and the actual operation image data can each represent the normal operation and the same operation as the actual operation of the virtual robot model 40M from a specific timing to the detection timing. The second server 20 may generate normal operation image data and actual operation image data for portions where there are differences between the normal operation and the actual operation.

[0110] The virtual space model VAM2 includes a virtual camera that captures images within the virtual space model VAM2. The virtual space model VAM2 can change the position, orientation, and zoom ratio of the virtual camera by an input operation. The normal operation image data and the actual operation image data are image data captured by the virtual camera. In the present embodiment, since the virtual space model VAM2 is a 3DCG model, the image data is CG image data.

[0111] As shown in, for example, FIG. 5, the second server 20 may transmit to the remote control device 50 image data representing the normal operation image data VN and the actual operation image data VR arranged side by side. As shown in, for example, FIG. 6, the second server 20 may also transmit to the remote control device 50 image data representing the normal operation image data VN and the actual operation image data VR superimposed. In FIG. 6, the normal operation image data VN is shown by a dashed line, and the actual operation image data VR is shown by a solid line. The second server 20 may obtain superimposed image data by performing image processing on the normal operation image data VN and the actual operation image data VR. Alternatively, the second server 20 may obtain superimposed image data by performing steps S208 and S209 in the virtual space model VAM2 in parallel on two virtual robot models 40M. FIGS. 5 and 6 are diagrams showing an example of an image that the second server 20 causes the remote control device 50 to display, and are diagrams showing the operation of the robot arm of the robot model 40M gripping the object W with its tip.

[0112] The second server 20 may obtain text data for explaining the states of the plurality of scenes included in the normal operation image data by inputting the normal operation image data into the language model LM stored in the storage 20b. As shown in, for example, FIG. 7, the second server 20 may generate image data for superimposing the text data on each of the images of the plurality of scenes, and transmit the image data to the remote control device 50. Similarly, the second server 20 may obtain text data for explaining the states of the plurality of scenes included in the actual operation image data by inputting the actual operation image data into the language model LM. As shown in, for example, FIG. 7, the second server 20 may generate image data for superimposing the text data on each of the images of the plurality of scenes, and transmit the image data to the remote control device 50. FIG. 7 is a diagram showing an example of an image in which text data is superimposed on the image of FIG. 5.

[0113] The second server 20 may process the virtual space model VAM2 to clarify the difference between the virtual robot model 40M and the virtual peripheral environment model other than the virtual robot model 40M, and perform processing on the virtual robot model 40M and the virtual peripheral environment model so that one or more of color, shading, transparency, and texture are different from each other. For example, the second server 20 may perform different processing on the peripheral environment model according to whether it is in a fixed state or a movable state and according to the hardness of the surface of the peripheral environment model. The second server 20 may transmit the image data of the processed virtual space model VAM2 to the remote control device 50.

[0114] The operator of the remote control device 50 can confirm the planned operation of the robot 40 to be executed and the process until the abnormality of the actual operation of the robot 40 by visually recognizing the normal operation image data and the actual operation image data in step S210. Thereby, it becomes easy for the operator to find out the cause of the abnormality and find a method to eliminate the abnormality.

[0115] In step S211, the second server 20 allows the reception of the input of the operation to the virtual space model VAM2 by the remote control device 50 and the reception of the input of the manual operation to the robot 40 by the remote control device 50.

[0116] For example, the operator inputs to change the position, orientation, and zoom ratio of the virtual camera in the virtual space model VAM2 using the remote control device 50, and the second server 20 can change the position, orientation, and zoom ratio of the virtual camera according to the input content. Thereby, the operator can visually confirm the robot model 40M and the surrounding situation at various positions, directions, and sizes that cannot be viewed by the cameras arranged on the robot operation area A2 and the robot 40.

[0117] The operator inputs to change one or more of the color, shading, transparency, and texture of the elements in the virtual space model VAM2 using the remote control device 50, and the second server 20 can change the color, shading, transparency, and texture of the element according to the input content. For example, the operator can change the element that is displayed overlapping the robot model 40M or the robot model 40M to be translucent or transparent, and visually confirm the respective relationships.

[0118] The operator inputs to manually operate the robot model 40M in the virtual space model VAM2 using the remote control device 50, and the second server 20 can operate the robot model 40M according to the manual operation. Thereby, the operator can perform a simulation of the operation in advance using the robot model 40M of the virtual space model VAM2 before operating the actual robot 40 to eliminate the abnormality.

[0119] The operator inputs a command to operate the robot model 40M in the virtual space model VAM2 according to the robot program RP using the remote control device 50, and the second server 20 can operate the robot model 40M according to the robot program RP. Thereby, the operator can perform a simulation to complete the task according to the robot program RP using the robot model 40M of the virtual space model VAM2 before operating the actual robot 40 to eliminate the abnormality.

[0120] By operating the virtual space model VAM2 in step S211, the operator of the remote control device 50 can easily identify the cause of the abnormality that has occurred in the robot 40 and find a method to eliminate the abnormality.

[0121] In step S212, the operator inputs a manual operation to the input device 52 of the remote control device 50 to eliminate the abnormality. The remote control device 50 transmits a motion command corresponding to the manual operation to the second server 20, and the second server 20 transmits the motion command to the first processing terminal 30. The first processing terminal 30 causes the robot 40 to execute an operation according to the motion command, that is, an operation according to the manual operation.

[0122] In step S210, the operator visually recognizes the normal operation image data and the actual operation image data, and in step S211, operates the virtual space model VAM2 to perform a simulation, so that the operator can understand the operations that the robot 40 has executed and the operations that the robot 40 should execute. Therefore, even if the operator does not understand the details of the robot program RP, it is easy to engage in the manual operation of the robot 40.

[0123] The second server 20 receives the captured image data of the camera mounted on the robot 40 and the captured image data of the camera arranged in the robot operation area A2 from the first processing terminal 30, and transmits them to the remote control device 50. Thereby, the operator can manually operate the robot 40 using the remote control device 50 while visually recognizing the captured image on the display 53.

[0124] In addition to the above captured image data, the second server 20 may transmit the image data representing the virtual space model VAM2 to the remote control device 50 and receive an input of an operation on the virtual space model VAM2 by the remote control device 50. Thereby, the operator can manually operate the robot 40 while visually recognizing the captured image on the display 53 and operating the virtual space model VAM2 to check the area not represented by the captured image.

[0125] In step S213, when the second server 20 determines that the abnormality of the robot 40 has been resolved, it cuts off the connection between the remote control device 50 and the first processing terminal 30, and sends a command to the first processing terminal 30 to shift from the manual operation mode to the automatic operation mode. As a result, the first processing terminal 30 resumes the control of the robot 40 according to the robot program RP and returns to step S201.

[0126] The second server 20 may determine that the abnormality of the robot 40 has been resolved by receiving a notification that the abnormality has been resolved from the remote control device 50. When the second server 20 determines that the operation of the robot 40 by the remote control device 50 has reached a preset stable state, it may also determine that the abnormality of the robot 40 has been resolved. The stable state is a state in which the autonomous operation of the robot 40 can be restored. For example, it may be a state in which the operation of the robot 40 by the remote control device 50 approximates the operation according to the robot program RP.

[0127] In steps S201 to S213, when an abnormality occurs in the operation of the robot 40 during the autonomous operation of the robot 40, the transition to manual operation by the operator to handle the abnormality becomes smooth.

[0128] In step S210, the second server 20 transmits the normal operation image data and the actual operation image data to the remote control device 50. In addition to these, it may be configured to estimate and transmit the cause of the abnormality. In this case, the second server 20 may store an estimation model, which is a machine learning model, in the storage 20b. The estimation model may be constructed with one or more of the actual operation image data, text data generated by the language model LM for the actual operation image data, robot information, and simulation data of the robot 40 according to the robot program RP as input data, and the cause of the abnormality as output data.

[0129] Instead of text data, parameter values used within the language model LM to generate the text data may be used. The simulation data may be data obtained by simulation using the virtual space model VAM. The estimation model may further include the error code of the autonomous control of the robot as input data. The estimation model may further include a method for dealing with abnormalities as output data. The estimation model performs machine learning using past data corresponding to the input data and past data corresponding to the output data.

[0130] For example, the estimation model may be a model that performs supervised machine learning using past data corresponding to the input data as learning input data and past data corresponding to the output data as teacher data. For example, the estimation model may be constructed by deep learning using a neural network or the like.

[0131] For example, the estimation model may be constructed such that part or all of the log data including the process in which an abnormality occurs in the operation of the robot 40, as used in step S209, is used as input data, and information related to the operation of the robot 40 for dealing with the abnormality is used as output data. The information related to the operation of the robot 40 may be information indicating the operation of the robot 40 for dealing with the abnormality, the manual operation of the robot 40 for dealing with the abnormality, or both of these. For example, the estimation model may be constructed by performing supervised machine learning. In this case, the learning input data may include various log data including the processes of various abnormalities in the operation of the robot 40 that have occurred in the past. The teacher data may include the manual operations input to the remote control device 50 for dealing with each of the abnormalities corresponding to the above log data, the operations of the robot 40 according to the manual operations, or data indicating both of these.

[0132] The second server 20 may store, in association with each other, log data including the process of an abnormality in the operation of the robot 40 that occurred in the past, manual operations input to the remote control device 50 to address the abnormality, operations of the robot 40 according to the manual operations, or information indicating both of them, as machine learning data in the storage 20b. The second server 20 may perform machine learning of an estimation model using the stored machine learning data.

[0133] In step S209, the second server 20 uses a language model LM that outputs text data indicating the state represented by the image data with the image data as input data, but the language model LM is not limited thereto. The language model LM may further include, as input data, simulation data and log data of the robot 40 according to the robot program RP in addition to the normal operation image data or the actual operation image data. The language model LM may further include, as input data, detection results of sensors arranged in the robot operation area A. The language model LM performs machine learning using past data corresponding to the input data and past data corresponding to the output data.

[0134] For example, similar to the estimation model, the language model LM may be a model that performs supervised machine learning using past data corresponding to the input data as learning input data and past data corresponding to the output data as teacher data. For example, the language model LM may be constructed by deep learning using a neural network or the like.

[0135] The second server 20 may store, in association with each other, data generated in the past corresponding to the input data of the language model LM including the image data and text data generated for the data, as machine learning data in the storage 20b. The second server 20 may perform machine learning of an estimation model using the stored machine learning data.

[0136] [Others] As described above, exemplary embodiments of the present disclosure have been explained, but the present disclosure is not limited to the above embodiments. That is, various modifications and improvements are possible within the scope of the present disclosure. For example, those obtained by applying various modifications to the embodiments and those constructed by combining components in different embodiments are also included within the scope of the present disclosure.

[0137] For example, although the robot system 100 of the embodiment includes the first server 10 and the second server 20, the first server 10 and the second server 20 may be integrated into one server. For example, the second server 20 may be configured to have the functions of the first server 10, or the first server 10 may be configured to have the functions of a plurality of second servers 20.

[0138] For example, in the embodiment, the second server 20 stores the machine learning data for the language model LM and the estimation model and performs machine learning on the language model LM and the estimation model using the stored machine learning data, but it is not limited thereto. For example, the second server 20 stores the machine learning data for the language model LM and the estimation model, and the first server 10 may collect the machine learning data from various second servers 20. The first server 10 may perform machine learning on the language model LM and the estimation model using the collected machine learning data. As a result, machine learning of the language model LM and the estimation model using more machine learning data becomes possible, and thus the accuracy of the language model LM and the estimation model is improved.

[0139] Examples of each aspect of the technology of the present disclosure are as follows. The robot system according to the first aspect of the present disclosure includes a robot, a first processing circuit that controls the robot, and a second processing circuit that collects and stores log data including information related to the state of the robot in the process of operating the robot according to an operation program. The robot system further includes a first storage device that stores a virtual space model including a virtual robot model and a virtual surrounding environment model that reproduce the robot and its surrounding environment, a third processing circuit that outputs data representing the virtual space model to a display device, and a fourth processing circuit that monitors the robot and detects an abnormality in the operation of the robot. When an abnormality is detected by the fourth processing circuit, the third processing circuit applies the log data accumulated by the second processing circuit to the virtual space model, reproduces the operation of the robot according to the log data in the virtual space model, and outputs data representing the virtual space model that has received the reproduction according to the log data to the display device.

[0140] According to the first aspect described above, when an abnormality in the operation of a physical robot is detected, the robot system applies log data to a virtual space model, reproduces the operation of the robot according to the log data in the virtual space model, and outputs data representing the virtual space model that has received the reproduction according to the log data to a display device. As a result, an operator dealing with the abnormality of the robot can confirm the reproduction of the operation of the robot by viewing the display device. Therefore, it becomes easier for the operator to respond when the robot performs an unintended operation. The display device may be any device that can display data representing the virtual space model. For example, the display device may be a device including a display or a projector. Examples of data representing the virtual space model may include data of a three-dimensional model of the virtual space model, data of a two-dimensional model of the virtual space model, image data obtained by imaging the three-dimensional model of the virtual space model with a virtual camera of the virtual space model, and image data obtained by imaging the two-dimensional model of the virtual space model with a virtual camera of the virtual space model.

[0141] In the first aspect described above, in the robot system according to the second aspect of the present disclosure, the third processing circuit may reproduce the operation of the robot in the virtual space model from a specific time point before the detection of the abnormality to the time point of the detection of the abnormality according to the log data by applying the log data to the virtual space model.

[0142] According to the second aspect described above, the robot system can reproduce the operation of the robot in the virtual space model from a specific time point before the detection of the abnormality to the time point of the detection of the abnormality. As a result, an operator who deals with the abnormality of the robot can confirm the reproduction of the process in which the operation of the robot leads to the abnormality by looking at the display of the device. Therefore, it becomes easier for the operator to respond when the robot performs an unintended operation.

[0143] In the first or second aspect described above, in the robot system according to the third aspect of the present disclosure, the third processing circuit outputs data representing, together, a first robot model that reproduces the operation of the robot in the virtual space model according to the log data by applying the log data to the virtual space model, and a second robot model that operates according to the operation program by applying the operation program to the virtual space model, to the display device.

[0144] According to the third aspect described above, the robot system can represent together a first robot model that reproduces the operation according to the log data and a second robot model that reproduces the operation according to the operation program. As a result, an operator who deals with the abnormality of the robot can easily identify the cause of the abnormality by comparing the two robot models.

[0145] In any of the first to third aspects described above, the robot system according to the fourth aspect of the present disclosure includes a second storage device that stores a language model that uses image data as input data and outputs text data indicating the state represented by the image data, and inputs the image data representing the virtual space model to the language model to generate text data indicating the state of the virtual space model, and a fifth processing circuit that outputs the text data to the display device.

[0146] According to the fourth aspect described above, the robot system can represent the state of the virtual space model represented by the image data in text data. As a result, an operator who deals with an abnormality of the robot can easily understand the reproduced operation of the robot.

[0147] In any of the first to fourth aspects described above, in the robot system according to the fifth aspect of the present disclosure, the third processing circuit may construct the virtual space model including a virtual camera that can be arranged at an arbitrary position in the virtual space model, and output the image data of the virtual space model captured by the virtual camera to the display device.

[0148] According to the fifth aspect described above, the robot system can output image data representing the virtual space model captured from various positions via a virtual camera. As a result, an operator who deals with an abnormality of the robot can visually recognize the operation of the robot reproduced in the virtual space model from various positions by changing the position of the virtual camera. Therefore, the operator can easily identify the cause of the abnormality.

[0149] In any of the first to fifth aspects described above, in the robot system according to the sixth aspect of the present disclosure, the third processing circuit may output data of the virtual space model in which one or more of color, shading, transparency, and texture are processed so as to be different from each other to the virtual robot model and the virtual surrounding environment model.

[0150] According to the above sixth aspect, the robot system can display a virtual robot model and a virtual surrounding environment model in an easy-to-visualize manner. Therefore, the operator can easily identify the cause of the abnormality.

[0151] In any one of the above first to sixth aspects, the robot system according to the seventh aspect of the present disclosure includes a controller that receives an input of a manual operation for operating the robot according to the input operation, and outputs a control command, which is a command for operating the robot according to the manual operation, to the first processing circuit. The third processing circuit outputs data representing the virtual space model to the controller, and the controller may display the data on a display included in the controller.

[0152] According to the above seventh aspect, the robot system can display data representing a virtual space model on a display of a controller that can manually operate the robot. Therefore, the operator can manually operate the robot to deal with the abnormality while visually recognizing the virtual space model through the display.

[0153] In the above seventh aspect, in the robot system according to the eighth aspect of the present disclosure, when the abnormality is detected, the third processing circuit applies the log data to the virtual space model to reproduce the operation of the robot according to the log data in the virtual space model, and may output data representing the virtual space model that has received the reproduction according to the log data to the controller.

[0154] According to the above eighth aspect, the robot system can display data representing a virtual space model that has received reproduction according to log data on a display of a controller. Therefore, the operator can manually operate the robot while visually recognizing the virtual space model that has received reproduction through the display.

[0155] In any of the first to eighth aspects described above, the robot system according to the ninth aspect of the present disclosure uses, as input data, log data including information related to the state of the robot when an abnormality in the operation of the robot is detected, and uses, as output data, information related to the operation of the robot for dealing with the abnormality. The robot system further includes a third storage device that stores an estimation model that performs machine learning using the log data including information related to the state of the robot when an abnormality in the operation of the robot is detected and information related to the operation of the robot performed to deal with the abnormality, and a sixth processing circuit that, when an abnormality is detected by the fourth processing circuit, inputs the log data accumulated by the second processing circuit to the estimation model and outputs the data output by the estimation model to the display device.

[0156] According to the ninth aspect described above, when the robot system detects an abnormality in the operation of the robot, the robot system uses the estimation model to output, to the display device, information related to the operation of the robot for dealing with the abnormality. Thereby, an operator who deals with the abnormality of the robot can refer to the information for dealing with the abnormality, so that it becomes easy to deal with the abnormality of the robot.

[0157] In any of the first to ninth aspects described above, the robot system according to the tenth aspect of the present disclosure includes a server including the second processing circuit, the third processing circuit, and the first storage device, a processing terminal including the first processing circuit and communicably connected to the server via a communication network, and the display device communicably connected to the server via the communication network. The fourth processing circuit is provided in the server or the processing terminal. The server includes a third storage device that stores a plurality of the operation programs corresponding to a plurality of tasks. The server transmits the operation program corresponding to the requested task to the processing terminal, and the processing terminal may autonomously execute the requested task on the robot using the received operation program.

[0158] According to the above-described tenth aspect, the robot system can cause the robot to execute various operation programs stored in the server via the processing terminal. Thereby, the processing capacity and data volume required for the processing terminal can be kept low. Furthermore, since the server holds the virtual space model and performs processing related to the virtual space model, the processing capacity and data volume required for the processing terminal can be kept low.

[0159] The control method of the robot system according to the eleventh aspect of the present disclosure includes causing the robot to operate according to an operation program, collecting and storing log data including information related to the state of the robot in the process of causing the robot to operate according to the operation program, generating a virtual space model including a virtual robot model and a virtual surrounding environment model that reproduce the robot and the surrounding environment of the robot, detecting an abnormality in the operation of the robot, and when the abnormality is detected, applying the log data to the virtual space model to reproduce the operation of the robot according to the log data in the virtual space model, displaying data representing the virtual space model that has received the reproduction according to the log data on a display included in a controller that receives an input of a manual operation and outputs a control command to cause the robot to operate according to the manual operation, and causing the robot to operate according to the control command output by the controller.

[0160] According to the above-described eleventh aspect, the same effects as those of the robot system according to each aspect of the present disclosure can be obtained. Part or all of the method of the present disclosure may be realized by, for example, a circuit such as a CPU or LSI, an IC card, or a single module. A plurality of elements included in the method of the present disclosure may be realized by one device or may be shared and realized by two or more devices.

[0161] In the eleventh aspect described above, the method for controlling a robot system according to the twelfth aspect of the present disclosure may further include receiving an operation of the virtual space model by the controller and displaying, on the display, data representing the virtual space model that reflects the operation of the virtual robot model according to the received operation.

[0162] According to the twelfth aspect described above, an operator of the controller can operate the virtual space model to operate the virtual robot model while visually recognizing the virtual space model. Thereby, the operator can perform a simulation of an operation for coping with an abnormality before operating the robot.

[0163] The present disclosure may be a computer program that causes a computer to execute the method according to each aspect of the present disclosure. Such a computer program can achieve the same effects as the method according to each aspect of the present disclosure. The computer program may be, for example, a program recorded on a non-transitory computer-readable recording medium, and may be configured to be read from the recording medium using a drive device of the recording medium and installed in the computer. The computer program may be, for example, a program that can be distributed via a transmission medium such as the Internet, and may be configured to be downloaded and installed in the computer.

[0164] The functions of the elements disclosed in this specification can be executed using a circuit or processing circuit that includes a general-purpose processor, a dedicated processor, an integrated circuit, an ASIC, a conventional circuit, and / or a combination thereof configured or programmed to perform the disclosed functions. Since a processor includes transistors and other circuits, it is considered a processing circuit or a circuit. In the present disclosure, a circuit, unit, or means is either hardware that performs the recited functions or hardware programmed to perform the recited functions. The hardware may be the hardware disclosed herein or other known hardware programmed or configured to perform the recited functions. When the hardware is a processor considered a type of circuit, the circuit, means, or unit is a combination of hardware and software, and the software is used in the configuration of the hardware and / or the processor.

[0165] The numbers such as ordinal numbers and quantities used in this specification are all exemplified for specifically explaining the technology of the present disclosure, and the present disclosure is not limited to the exemplified numbers. The connection relationships between components are exemplified for specifically explaining the technology of the present disclosure, and the connection relationships for realizing the functions of the present disclosure are not limited thereto.

[0166] The present disclosure can be implemented in various forms without departing from the scope of its essential features. Since the scope of the present disclosure is defined by the appended claims rather than the description in the specification, the exemplary embodiments and modifications are illustrative and not restrictive. All changes within the claims and their scope, or equivalents of the claims and their scope, are intended to be encompassed by the claims.

Description of Reference Signs

[0167] 20, 20A Second Server (Server) 20a Processing Circuit (Second Processing Circuit, Third Processing Circuit, Fourth Processing Circuit, Fifth Processing Circuit, Sixth Processing Circuit) 20b Memory (First Memory, Second Memory, Third Memory) 30, 30A First processing terminal (processing terminal) 30a Processing circuit (first processing circuit, fourth processing circuit) 40, 40A Robot 50 Remote control device (display device) 100 Robot system

Claims

1. A robot, a first processing circuit for controlling the robot, a second processing circuit that collects and stores log data including information related to the state of the robot in the process of operating the robot according to an operation program, a first storage device that stores a virtual space model including a virtual robot model and a virtual surrounding environment model that reproduce the robot and the surrounding environment of the robot, a third processing circuit that outputs data representing the virtual space model to a display device, a fourth processing circuit that monitors the robot and detects an abnormality in the operation of the robot, and when an abnormality is detected by the fourth processing circuit, the third processing circuit applies the log data accumulated by the second processing circuit to the virtual space model to reproduce the operation of the robot according to the log data in the virtual space model, and outputs data representing the virtual space model that has received the reproduction according to the log data to the display device a robot system.

2. By applying the log data to the virtual space model, the third processing circuit reproduces the operation of the robot in the virtual space model from a specific point in time before the detection of the abnormality to the point in time of the detection of the abnormality according to the log data The robot system according to claim 1.

3. The third processing circuit outputs to the display device data representing, together, a first robot model that reproduces the operation of the robot in the virtual space model according to the log data by applying the log data to the virtual space model, and a second robot model that operates according to the operation program by applying the operation program to the virtual space model The robot system according to claim 1.

4. a second storage device that stores a language model that uses image data as input data and outputs text data indicating the state represented by the image data, and a fifth processing circuit that inputs image data representing the virtual space model to the language model, generates text data indicating the state of the virtual space model, and outputs the text data to the display device The robot system according to claim 1.

5. The third processing circuit constructs the virtual space model including a virtual camera that can be arranged at an arbitrary position within the virtual space model, Output the image data of the virtual space model captured by the virtual camera to the display device The robot system according to claim 1

6. The third processing circuit outputs data of the virtual space model, which is subjected to processing in which one or more of color, shading, transparency, and texture are different from each other, to the virtual robot model and the virtual surrounding environment model The robot system according to claim 1

7. Comprises a controller that receives an input of a manual operation for operating the robot according to the input operation, and outputs a control command, which is a command for operating the robot according to the manual operation, to the first processing circuit The third processing circuit outputs data representing the virtual space model to the controller The controller causes the data to be displayed on a display included in the controller The robot system according to claim 1

8. When the abnormality is detected, the third processing circuit applies the log data to the virtual space model to reproduce the operation of the robot according to the log data in the virtual space model, and outputs data representing the virtual space model that has undergone the reproduction according to the log data to the controller The robot system according to claim 7

9. An estimation model that uses log data including information related to the state of the robot when an abnormality in the operation of the robot is detected as input data, and information related to the operation of the robot for dealing with the abnormality as output data, and is a learning estimation model using the log data including information related to the state of the robot when an abnormality in the operation of the robot is detected and information related to the operation of the robot performed to deal with the abnormality; a third storage device that stores the estimation model Further comprises a sixth processing circuit that, when an abnormality is detected by the fourth processing circuit, inputs the log data accumulated by the second processing circuit into the estimation model, and outputs the data output by the estimation model to the display device The robot system according to claim 1

10. A server comprising the second processing circuit, the third processing circuit, and the first storage device A processing terminal comprising the first processing circuit and communicably connected to the server via a communication network Comprises the display device communicably connected to the server via the communication network The fourth processing circuit is provided in the server or the processing terminal The server includes a third storage device that stores a plurality of the operation programs corresponding to a plurality of tasks. The server transmits the operation program corresponding to the requested task to the processing terminal. The processing terminal uses the received operation program to autonomously execute the requested task on the robot. The robot system according to claim 1.

11. Operating the robot according to an operation program; Collecting and storing log data including information related to the state of the robot in the process of operating the robot according to the operation program; Generating a virtual space model including a virtual robot model and a virtual surrounding environment model that reproduce the robot and the surrounding environment of the robot; Detecting an abnormality in the operation of the robot; When the abnormality is detected, applying the log data to the virtual space model to reproduce the operation of the robot according to the log data in the virtual space model; Displaying data representing the virtual space model that has undergone reproduction according to the log data on a display included in a controller that receives an input of a manual operation and outputs a control command to operate the robot according to the manual operation; Operating the robot according to the control command output by the controller. A control method for a robot system.

12. Further including receiving an operation of the virtual space model by the controller and displaying, on the display, data representing the virtual space model that reflects the operation of the virtual robot model according to the received operation. The control method for the robot system according to claim 11.

Images