Robot system and control method for robot system

Abstract

A robot system (100) comprises: a robot (40); a first processing circuit (30a) for controlling the robot; a second processing circuit (20a) for collecting and accumulating log data related to the state of the robot; a first memory (20b) for storing a virtual space model for reproducing the robot and the surrounding environment; a third processing circuit (20a) for outputting data representing the virtual space model to a display device (50); and a fourth processing circuit (20a) for detecting an abnormality in the operation of the robot. If the abnormality is detected, the third processing circuit applies the log data to the virtual space model to reproduce, in the virtual space model, the operation of the robot which is in accordance with the log data, and the third processing circuit outputs, to the display device, data representing the virtual space model in which the reproduction has been received.

Description

Robot system and control method for robot system

[0001] The present disclosure relates to a robot system and a control method for a robot system.

[0002] Patent Document 1 discloses a robot interference detection device that, based on an operator's input, determines whether interference will occur in a virtual space between three-dimensional models of the robot, workpiece, and surrounding objects while the robot is moving, using 3D CAD that represents approximate shapes of the robot, workpiece, and surrounding objects.

[0003] Japanese Patent Application Laid-Open No. 2019-171501

[0004] Patent Literature 1 discloses a technique for preventing interference between robots, etc. For example, in an environment where an operator cannot directly see the robot, if the robot performs an unintended operation such as interfering with another object and stopping, it is difficult for the operator to resolve such a situation by operating the robot.

[0005] The present disclosure aims to provide a robot system and a control method for a robot system that make it easy for an operator to respond when a robot performs an unintended operation.

[0006] A robot system according to one aspect of the present disclosure comprises a robot, a first processing circuit that controls the robot, a second processing circuit that collects and accumulates log data including information related to the state of the robot while the first processing circuit operates the robot according to an operation program, a first memory that stores a virtual space model including a virtual robot model and a virtual surrounding environment model that reproduce the robot and the robot's surrounding environment, a third processing circuit that outputs data representing the virtual space model to a display device, and a fourth processing circuit that monitors the robot and detects abnormalities in the robot's operation, wherein when an abnormality is detected by the fourth processing circuit, the third processing circuit applies the log data accumulated by the second processing circuit to the virtual space model, thereby reproducing the robot's operation according to the log data in the virtual space model, and outputs data representing the virtual space model reproduced according to the log data to the display device.

[0007] FIG. 1 is a diagram illustrating an example of a robot system according to an exemplary embodiment. FIG. 2 is a diagram illustrating one second server, one first processing terminal, one robot, one remote control device, and one second processing terminal extracted from the robot system of FIG. 1. FIG. 3 is a flowchart illustrating an example of the operation of the robot system according to the embodiment. FIG. 4 is a flowchart illustrating an example of the operation of the robot system according to the embodiment when an abnormality occurs in the operation of the robot. FIG. 5 is a diagram illustrating an example of an image that the second server causes to be displayed on the remote control device. FIG. 6 is a diagram illustrating an example of an image that the second server causes to be displayed on the remote control device. FIG. 7 is a diagram illustrating an example of an image in which text data is superimposed on the image of FIG. 5.

[0008] Exemplary embodiments of the present disclosure will be described below with reference to the drawings. The embodiments described below are all comprehensive or specific examples. Among the components in the following embodiments, components that are not recited in independent claims that represent the highest concepts will be described as optional components. Each figure in the accompanying drawings is a schematic diagram and is not necessarily an exact illustration. In each figure, substantially identical components are assigned the same reference numerals, and redundant explanations may be omitted or simplified. In this specification and claims, the term "device" may refer not only to a single device but also to a system including multiple devices. Below, a robot is exemplified as a mechanical device that operates based on an operating program.

[0009] A robot system 100 according to an exemplary embodiment will be described with reference to FIGS. 1 and 2 . FIG. 1 is a diagram illustrating an example of the robot system 100 according to the exemplary embodiment. FIG. 2 is a diagram illustrating one second server 20, one first processing terminal 30, one robot 40, one remote control device 50, and one second processing terminal 60 extracted from the robot system 100 of FIG. 1 . The robot system 100 includes the first server 10, the second server 20, the first processing terminal 30, the robot 40, and the remote control device 50. The first server 10 is communicatively connected to the second server 20 via a first communication network NA. The first server 10 may be communicatively connected to the first processing terminal 30 via the first communication network NA. The second server 20 is communicatively connected to the first processing terminal 30 via a second communication network NB.

[0010] There are multiple robot operation areas A, and one or more first processing terminals 30 and one or more robots 40 are placed in each of the multiple robot operation areas A. A robot operation area A is an area where one or more robots 40 provide services or perform work. Examples of robot operation areas A include an area set within a building or facility, and an area set based on a map.

[0011] The first processing terminal 30 controls the operation of the robot 40. The first processing terminal 30 may control one robot 40 or two or more robots 40. The first processing terminal 30 may be mounted on the robot 40, or may be located at a position remote from the robot 40 and connected to the robot 40 via wired communication, wireless communication, or a combination thereof. The first processing terminal 30 may be communicably connected to the robot 40 via a communication network. Any wired communication, wireless communication, or communication network may be used.

[0012] The first processing terminal 30 is configured to cause the robot 40 to autonomously perform a predetermined task in accordance with a predetermined computer program. In other words, the first processing terminal 30 is configured to cause the robot 40 to operate autonomously. The robot 40 operates in accordance with commands generated by the first processing terminal 30 by the predetermined computer program. The first processing terminal 30 may be configured to cause the robot 40 to operate in accordance with manual operations manually input by an operator to an operating device disposed away from the robot 40 to cause the robot 40 to operate as intended. In other words, the first processing terminal 30 may be configured to cause the robot 40 to operate manually. In this embodiment, the first processing terminal 30 can cause the robot 40 to operate autonomously and manually.

[0013] The robot 40 may have a structure suitable for any purpose, such as work in a factory or warehouse, loading and unloading, transportation, nursing care, medical care, cleaning, security, guidance, rescue, cooking, and product provision. The robot 40 may have a structure used in a fixed state, a structure that moves when an external force is applied, or a structure that includes a moving means for autonomous movement. Examples of the moving means may include wheels, crawlers, propulsion rotors, flying rotors, jet propulsion devices, and movable legs. The robot 40 may include one or more articulated robot arms. The robot 40 may include one or more articulated robot legs. The robot 40 may have the form of an industrial robot, a humanoid robot, an animal-like robot, or the like.

[0014] There are multiple operating area groups AG. A second server 20 is placed in each operating area group AG. The operating area group AG is set for a specific area including multiple robot operating areas A. The second server 20 manages all robot operating areas A within the operating area group AG with which the second server 20 is associated. The second server 20 is communicatively connected to multiple first processing terminals 30 within the multiple robot operating areas A that it manages via a second communication network NB.

[0015] The second server 20 may be communicatively connected via a second communication network NB to a second processing terminal 60 operated by a user who manages the robot operation area A to be managed. The user may be a user who manages the robots 40 and the first processing terminals 30 in one or more robot operation areas A. The second server 20 is communicatively connected to the remote control device 50 via the second communication network NB or another communication network. The second server 20 is also called an edge server.

[0016] The second server 20 includes a processing circuit 20a and has the functionality of a computer. The processing circuit 20a includes a processor and a memory. The second server 20 includes storage within the processing circuit 20a or separately from the processing circuit 20a. The memory and storage are collectively referred to as a storage 20b. The processing circuit 20a has the functionality of a second processing circuit, a third processing circuit, a fifth processing circuit, and a sixth processing circuit, and may further have the functionality of a fourth processing circuit. The storage 20b may have the functionality of a first storage, a second storage, a third storage, and a fourth storage.

[0017] The processor executes functions, methods, or combinations thereof implemented by code or instructions contained in a program stored in storage. Examples of processors include a central processing unit (CPU), a microprocessing unit (MPU), a graphics processing unit (GPU), a microprocessor, a processor core, a multiprocessor, an application-specific integrated circuit (ASIC), and a field programmable gate array (FPGA). The processor may implement each process described in the present disclosure using a logic circuit formed on an integrated circuit (an integrated circuit (IC) chip, a large scale integration (LSI)), or a dedicated circuit. These circuits may be implemented using one or more integrated circuits. Multiple processes may be implemented using a single integrated circuit.

[0018] The memory temporarily stores programs loaded from storage and provides a working area for the processor. The memory also temporarily stores various data generated while the processor is executing the programs. Examples of memory include semiconductor memories such as RAM (Random Access Memory) and ROM (Read Only Memory).

[0019] The storage stores programs and various data, and examples of the storage include a hard disk drive (HDD), a solid state drive (SSD), and a flash memory.

[0020] For example, the storage stores information on a plurality of robot operation areas A. Each of the robot operation areas A is a robot operation area A managed by the second server 20, and is registered in the second server 20. The storage stores information on the first processing terminal 30 and the robot 40 located in each of the plurality of robot operation areas A, in association with the robot operation area A that includes the first processing terminal 30 and the robot 40. The first processing terminal 30 and the robot 40 are the first processing terminal 30 and the robot 40 located in each of the robot operation areas A registered in the second server 20, and are registered in the second server 20.

[0021] The storage may store information about the second processing terminal 60 in association with a robot operation area A related to the second processing terminal 60. The second processing terminal 60 is the second processing terminal 60 of a user who manages the robot operation area A registered in the second server 20, and is registered in the second server 20. The second server 20 may be communicably connected to the first processing terminal 30 and the second processing terminal 60 registered in the second server 20 via a second communication network NB.

[0022] The storage stores information about the remote control devices 50. The remote control devices 50 are one or more remote control devices 50 set for the operating area group AG managed by the second server 20, and are registered in the second server 20. The second server 20 can be communicably connected to the remote control devices 50 registered in the second server 20 via the second communication network NB.

[0023] The storage stores a virtual space model VAM that forms a virtual space of the robot operation area A and a virtual space program that operates the virtual space model VAM. For example, the virtual space model VAM may be a 3DCG (3-dimensional computer graphics) model. The virtual space model VAM is a model that faithfully reproduces the real-world robot operation area A and includes virtual components that correspond to various components contained in the real-world robot operation area A. For example, by applying information or data about real components acquired from the real-world robot operation area A to the virtual space model VAM, the virtual components in the virtual space model VAM that correspond to the real components can be made to behave in the same way as the real components.

[0024] By executing the virtual space program, the processing circuit 20a can apply various data to the virtual space model VAM, add and delete components to the virtual space model VAM, and operate the components within the virtual space model VAM. The virtual space model VAM includes a virtual robot model of the robot 40 within the robot operation area A and a surrounding environment model of elements other than the robot 40. In the present embodiment, the virtual space model VAM is a virtual space model formed for each robot operation area A registered in the second server 20 using digital twin technology so as to faithfully reproduce the robot operation area A.

[0025] The virtual space model VAM may be generated by the processing circuitry 20a, or may be obtained by the processing circuitry 20a from outside the second server 20. Updating the virtual space model VAM by making changes to the virtual space model VAM may be performed by the processing circuitry 20a, or may be performed by a device outside the second server 20. The processing circuitry 20a may be configured to receive the updated virtual space model VAM from a device outside the second server 20.

[0026] The storage may store a language model LM, which is a machine learning model that uses image data as input data and outputs text data indicating a state represented by the image data. In the present embodiment, the language model LM is a large language model (LLM). The language model LM may be generated by the processing circuitry 20a, or may be acquired by the processing circuitry 20a from outside the second server 20. Machine learning of the language model LM may be performed by the processing circuitry 20a, or may be performed by a device outside the second server 20. The processing circuitry 20a may be configured to receive the machine-learned language model LM from outside the second server 20.

[0027] The storage stores various robot programs RP for causing the robot 40 to autonomously execute tasks. The robot program RP is generated for each task in correspondence with various tasks. The robot program RP can be generated in correspondence with various robots 40 in the robot operating area A registered in the second server 20. When robots 40 with different structures perform the same task, the robot programs RP may be the same or different. The robot programs RP generated for each task in correspondence with various robots 40 are stored in the storage.

[0028] Examples of the second communication network NB may include a LAN (Local Area Network), a local 5G, or other access network. The communication distance between the second server 20 and the first processing terminal 30 via the second communication network NB is significantly shorter than the communication distance between the first server 10 and the first processing terminal 30 via the first communication network NA and the communication distance between the first server 10 and the second server 20 via the first communication network NA. In other words, the communication distance via the second communication network NB is significantly shorter than the communication distance via the first communication network NA. Therefore, communication via the second communication network NB enables large amounts of data to be sent and received at high speed.

[0029] The first server 10 is communicably connected to a plurality of second servers 20 via a first communication network NA. The first server 10 is also called a cloud server. The first server 10 may be communicably connected to a plurality of first processing terminals 30 via the first communication network NA. The first server 10 may be communicably connected to a plurality of second processing terminals 60 via the first communication network NA.

[0030] The first server 10 includes a processing circuit 10a and has computer functions. The processing circuit 10a includes a processor and a memory. The first server 10 includes a storage device within the processing circuit 10a or separate from the processing circuit 10a. The memory and storage device are collectively referred to as a storage device 10b.

[0031] Examples of the processor, memory, and storage of the first server 10 are similar to the examples of the processor, memory, and storage of the second server 20. The functions of the processor, memory, and storage of the first server 10 are similar to the functions of the processor, memory, and storage of the second server 20.

[0032] The storage of the first server 10 stores user information related to the user of the first processing terminal 30 and the user of the second processing terminal 60. The user information is stored for the first processing terminals 30 and second processing terminals 60 associated with all robot operation areas A managed by all second servers 20 connected to the first server 10. In other words, the user information is stored for all first processing terminals 30 and second processing terminals 60 registered in all second servers 20. The user information may include one or more of a user ID, which is user identification information, a login password for the user ID to the system, information on the first processing terminal 30 registered for the user ID, information on the robot 40 controlled by the first processing terminal 30, information on the second processing terminal 60 registered for the user ID, and information on the robot operation area A registered for the user ID.

[0033] The storage of the first server 10 stores area information related to the robot operation area A. The robot operation areas A to be stored are all robot operation areas A that are managed by all second servers 20 connected to the first server 10. The area information for each robot operation area A may include one or more of information on the second servers 20 that manage the robot operation area A, information on the first processing terminals 30 that are placed in the robot operation area A, information on the second processing terminals 60 that are placed in the robot operation area A, and information on the robots 40 that are placed in the robot operation area A.

[0034] The storage of the first server 10 stores program type information for the robot program RP. The program type information represents the type of the robot program RP classified according to one or more of the content of the task executed by the robot program RP and information on the robot 40 that can execute the robot program RP. For example, the program type can be represented as the type of task of transporting an object by a bipedal robot, the type of bipedal robot, or the type of task of transporting an object.

[0035] The first communication network NA is a network that covers a wider area than the second communication network NB. Examples of the first communication network NA may include an open network such as the Internet and an Internet VPN (Virtual Private Network), a closed network such as an IP-VPN (Internet Protocol Virtual Private Network) and a dedicated line, or a combination thereof.

[0036] The first processing terminal 30 includes a processing circuit 30a and has the functionality of a computer. The processing circuit 30a includes a processor and a memory. The first processing terminal 30 may include storage within the processing circuit 30a or separately from the processing circuit 30a. The memory and storage are collectively referred to as a storage device 30b. The processing circuit 30a has the functionality of the first processing circuit, and may also have the functionality of a fourth processing circuit.

[0037] Examples of the processor, memory, and storage of the first processing terminal 30 are similar to the examples of the processor, memory, and storage of the second server 20. The functions of the processor, memory, and storage of the first processing terminal 30 are similar to the functions of the processor, memory, and storage of the second server 20.

[0038] Examples of the first processing terminal 30 may include an electronic circuit board, an electronic control unit, a microcomputer, a desktop computer, a laptop computer, and a smart device such as a tablet. The first processing terminal 30 may be any of a processing terminal dedicated to the robot system 100, a processing terminal dedicated to the robot 40, a processing terminal dedicated to a system other than the robot system 100, such as a system used in the robot operating area A, and a general-purpose processing terminal. For example, the first processing terminal 30 may be a robot controller or a part thereof as a processing terminal dedicated to the robot 40. The first processing terminal 30 is communicatively connected to the robot 40 via wired communication, wireless communication, or a combination thereof. The first processing terminal 30 may be located remotely from the robot 40 or may be mounted on the robot 40.

[0039] The first processing terminal 30 is communicatively connected to the second communication network NB via wired communication, wireless communication, or a combination thereof, and is communicatively connected to the second server 20 via the second communication network NB. The first processing terminal 30 may be communicatively connected to the first communication network NA via wired communication, wireless communication, or a combination thereof, and may be communicatively connected to the first server 10 via the first communication network NA. Any wired communication and wireless communication may be used. The first processing terminal 30 may be connected to either the first communication network NA or the second communication network NB via one or more of a regional IP network, a mobile phone network such as 4G or 5G, a private network, a wired LAN, and a wireless LAN such as wireless Wi-Fi.

[0040] The storage device 30b stores an operation system program OS1 for operating the first processing terminal 30, and an application program AP1 (hereinafter also referred to as "app AP1") that runs on the operation system program OS1. The operation system program OS1 may be a program dedicated to the robot system 100 or a program not dedicated to the robot system 100. The app AP1 is a program dedicated to the robot system 100. The app AP1 may be constructed as an operation system program and incorporated into the first processing terminal 30.

[0041] The app AP1 is incorporated into the first processing terminal 30 by being installed. The app AP1 has a function of generating commands for operating the robot 40 connected to the first processing terminal 30. The app AP1 has a function of operating the robot 40 in an automatic operation mode and a manual operation mode. In the automatic operation mode, the app AP1 functions using the robot program RP.

[0042] The processing circuit 30a executes the automatic operation mode using the application AP1, thereby fetching the robot program RP sent from the second server 20 and executing the robot program RP on the application AP1.

[0043] The robot program RP includes, in an execution order, operation commands that command various target operations to be performed by the robot 40 in the process of executing a task defined in the robot program RP. For example, the operation commands may include commands for target positions and target postures of each part of the robot 40. The processing circuit 30a generates, in an execution order, drive commands that command target operations of the drive parts of the robot 40, and transmits the drive commands to the robot 40, in order to cause the robot 40 to sequentially execute operations in accordance with the operation commands of the robot program RP. The drive commands are commands corresponding to the structure of the robot 40, and may include, for example, commands for target operations of the joints included in the robot 40.

[0044] By executing the manual control mode using the app AP1, the processing circuit 30a causes the robot 40 to perform an action in accordance with a command for manually controlling the robot 40, which is output from a controller of the robot 40, such as the remote control device 50. For example, an operator moves a control on the controller to make the robot 40 perform an intended action. The controller outputs a command to make the robot 40 perform an action corresponding to the movement of the control. For example, the controller outputs a command to make the robot 40 operate in a direction and at a speed indicated by the movement of the control. The processing circuit 30a generates a drive command to make the robot 40 perform the action of the command received from the controller, and transmits it to the robot 40.

[0045] The processing circuit 30a acquires from the robot 40 the operation results of the driving parts of the robot 40 and the detection results of the sensors equipped in the robot 40, and transmits the operation results and detection results as log data to the second server 20. Furthermore, the processing circuit 30a may associate the execution results of the robot program RP with the operation results and detection results and transmit them to the second server 20 as log data.

[0046] For example, the processing circuit 30a may acquire operation results and detection results at predetermined time intervals. Examples of operation results of the driving parts may include an output value or current value of a driving device that drives the driving parts, and a detection value of a sensor that detects the driving amount of the driving parts. Examples of sensors included in the robot 40 may include a camera, an acceleration sensor, a gyro sensor, a position sensor, a speed sensor, a distance sensor, a tactile sensor, a force sensor, a force-tactile sensor, a vibration sensor, and a temperature sensor.

[0047] The first processing terminal 30 may include an input device 31 that accepts input from a user or the like. Examples of the input device 31 may include a button, a lever, a dial, a joystick, a mouse, keys, a touch panel, a microphone, and a motion capture device. For example, the input device 31 accepts a command from a user or the like requesting the robot 40 to perform a task, and outputs the command to the processing circuitry 30a. The processing circuitry 30a may be configured to send the content of the commanded task to the first server 10 via the first communication network NA. The content of the task may include, as task elements, a task subject, a task type, a task object, and a task goal to be achieved by the task.

[0048] For example, the task subject is a robot that executes a task. The task type corresponds to the type of operation that the robot 40 is made to execute, such as a transport operation or an assembly operation. The task object is an object that the robot 40 acts on by the task. The task goal corresponds to the target state of the object that should be reached by executing the task.

[0049] The input device 31 may accept input of task elements by the user's hand or voice selection or language input. The processing circuitry 30a may transmit information on the selected task element or character string data representing the input language to the first server 10. The processing circuitry 30a may transmit to the first server 10, together with the above information or data, either or both of identification information of the first processing terminal 30 and information on the robot 40 that will execute the task.

[0050] The processing circuit 10a of the first server 10 uses information received from the first processing terminal 30 to determine the program type to be executed by the robot 40, and transmits information on the determined program type, etc. to the second server 20 in which the first processing terminal 30 is registered. The processing circuit 20a of the second server 20 extracts a robot program RP corresponding to the program type from the storage device 20b and transmits it to the first processing terminal 30. The processing circuit 30a of the first processing terminal 30 causes the robot 40 to operate in accordance with the received robot program RP.

[0051] The second processing terminal 60 includes a processing circuit 60a and has the functionality of a computer. The processing circuit 60a includes a processor and a memory. The second processing terminal 60 may include a storage device within the processing circuit 60a or separately from the processing circuit 60a. The memory and storage device are collectively referred to as a storage device 60b.

[0052] Examples of the processor, memory, and storage of the second processing terminal 60 are similar to the examples of the processor, memory, and storage of the second server 20. The functions of the processor, memory, and storage of the second processing terminal 60 are similar to the functions of the processor, memory, and storage of the second server 20.

[0053] Examples of the second processing terminal 60 may include smart devices such as smartphones, smart watches, and tablets, desktop computers, and laptop computers. The second processing terminal 60 may be any of a processing terminal dedicated to the robot system 100, a processing terminal dedicated to the robot 40, a processing terminal dedicated to a system other than the robot system 100 such as a system used in the robot operating area A, and a general-purpose processing terminal. For example, the second processing terminal 60 may be a general-purpose processing terminal owned by a user. The second processing terminal 60 is located remotely from the robot 40.

[0054] The second processing terminal 60 is communicatively connected to the first communication network NA via wired communication, wireless communication, or a combination thereof, and is communicatively connected to the first server 10 via the first communication network NA. The second processing terminal 60 may be communicatively connected to the second communication network NB via wired communication, wireless communication, or a combination thereof, and may be communicatively connected to the second server 20 via the second communication network NB. Any wired communication and wireless communication may be used. The second processing terminal 60 may be connected to either the first communication network NA or the second communication network NB via one or more of a regional IP network, a mobile phone network such as 4G or 5G, a private network, and a wireless LAN.

[0055] The second processing terminal 60 is communicatively connected to the first processing terminal 30 via wired communication, wireless communication, or a combination thereof. The second processing terminal 60 may be configured to be communicatively connected to one first processing terminal 30, or may be configured to be communicatively connected to two or more first processing terminals 30.

[0056] The second processing terminal 60 includes an input device 61 that accepts input from a user or the like. Examples of the input device 61 may include a button, a lever, a dial, a joystick, a mouse, keys, a touch panel, a microphone, and a motion capture device. For example, the input device 61 accepts a command from a user or the like requesting the robot 40 to perform a task, and outputs the command to the processing circuit 60a. The processing circuit 60a sends the content of the requested task to the first server 10 via the first communication network NA. The content of the task may include a task subject, a task type, a task object, and a task goal as task elements.

[0057] The input device 61 may accept input of task elements by the user's hand or voice selection or language input. The processing circuitry 60a may transmit information on the selected task element or character string data representing the input language to the first server 10. The processing circuitry 60a may transmit to the first server 10, together with the above information or data, either or both of identification information of the second processing terminal 60 and information on the robot 40 that will execute the task.

[0058] The processing circuit 10a of the first server 10 uses information received from the second processing terminal 60 to determine the program type to be executed by the robot 40, and transmits information on the determined program type, etc. to the second server 20 in which the robot 40 is registered. The processing circuit 20a of the second server 20 extracts a robot program RP corresponding to the program type information from the storage device 20b and transmits it to the first processing terminal 30. The processing circuit 30a of the first processing terminal 30 causes the robot 40 to operate in accordance with the received robot program RP.

[0059] The storage device 60b stores an operation system program OS2 for operating the second processing terminal 60, and an application program AP2 (hereinafter also referred to as "app AP2") that runs on the operation system program OS2. The operation system program OS2 may be a program dedicated to the robot system 100 or a program not dedicated to the robot system 100. The app AP2 is a program dedicated to the robot system 100. The app AP2 may be constructed as an operation system program and incorporated into the second processing terminal 60.

[0060] The app AP2 is incorporated into the second processing terminal 60 by being installed. The app AP2 has a function of mutually transmitting and receiving commands, information, and data to and from the first server 10 and the first processing terminal 30. The app AP2 may also have a function of mutually transmitting and receiving commands, information, and data to and from the second server 20. The app AP2 has a function of receiving task elements of a task to be executed by the robot 40, such as a task subject, a task type, a task object, and a task goal, and transmitting information on the received task elements to the first server 10. The app AP2 may also have a function of transmitting commands to the first processing terminal 30 to cause the robot 40 to perform operations not based on the robot program RP, such as starting, stopping, emergency stopping, and moving to a standby position of the robot 40.

[0061] The second processing terminal 60 may be configured to access a website constructed for the robot system 100. The second processing terminal 60 may be configured to transmit and receive commands, information, and data to and from the first server 10 via the website. The second processing terminal 60 may be configured to accept input of task elements of a task to be executed by the robot 40 on the website, and the first server 10 may be configured to receive information about the task elements via the website. In this case, the app AP2 does not need to be installed in the second processing terminal 60.

[0062] The remote control device 50 includes a control processing terminal 51, an input device 52, a display 53, a camera 54, and an audio device 55. The control processing terminal 51 includes a processing circuit 51a and has the functions of a computer. The processing circuit 51a includes a processor and a memory. The control processing terminal 51 may include storage within the processing circuit 51a or separately from the processing circuit 51a. The memory and storage are collectively referred to as a storage device 51b. The remote control device 50 is an example of a display device.

[0063] Examples of the processor, memory, and storage of the steering processing terminal 51 are the same as the examples of the processor, memory, and storage of the second server 20. The functions of the processor, memory, and storage of the steering processing terminal 51 are the same as the functions of the processor, memory, and storage of the second server 20.

[0064] Examples of the control processing terminal 51 may include an electronic circuit board, an electronic control unit, a microcomputer, a desktop computer, a laptop computer, and a smart device such as a tablet. The control processing terminal 51 may be either a processing terminal dedicated to the robot system 100 or a general-purpose processing terminal.

[0065] The memory 51b stores an operation system program OS3 for operating the control processing terminal 51, and an application program AP3 (hereinafter also referred to as "app AP3") that runs on the operation system program OS3. The operation system program OS3 may be a program dedicated to the robot system 100 or a program not dedicated to the robot system 100. The app AP3 is a program dedicated to the robot system 100. The app AP3 may be constructed as an operation system program and incorporated into the control processing terminal 51. The app AP3 has the function of transmitting and receiving commands, information, and data to and from the second server 20.

[0066] The remote control device 50 is communicatively connected to the second communication network NB via wired communication, wireless communication, or a combination thereof, and is communicatively connected to the second server 20 via the second communication network NB. In the present embodiment, the remote control device 50 is connectable to the second server 20 of one operation area group AG, but may be connectable to the second servers 20 of two or more operation area groups AG. Any wired or wireless communication may be used. The remote control device 50 may be connected to the second communication network NB via one or more of a regional IP network, a mobile phone network such as 4G or 5G, a private network, a wired LAN, and a wireless LAN.

[0067] The remote control device 50 is placed in a location away from the robot operation area A registered in the second server 20 connected to the remote control device 50, for example, in a location where the operator of the remote control device 50 cannot directly see the robot operation area A. In response to a request from the second server 20, the remote control device 50 can operate a robot 40 that has been authorized by the second server 20. The remote control device 50 is connected by the second server 20 to a first processing terminal 30 that controls the authorized robot 40.

[0068] The input device 52 receives input from the operator and outputs the command received from the operator to the operation processing terminal 51. Examples of the input device 52 may include a button, a lever, a dial, a joystick, a mouse, a key, a touch panel, a microphone, and a motion capture device.

[0069] For example, the input device 52 accepts a manual operation that is manually input by the operator to make the robot 40 perform an intended action. The input device 52 outputs a command indicating the content of the manual operation to the operation processing terminal 51. For example, when the operator manually operates the joystick of the input device 52, the input device 52 may output a command indicating the tilt direction and tilt angle of the joystick to the operation processing terminal 51.

[0070] The processing circuit 51a of the operation processing terminal 51 processes the command received from the input device 52, generates a movement command instructing the robot 40 to execute a target movement, and transmits the movement command to the second server 20. When the command received from the input device 52 is a command indicating a tilt direction and tilt angle of the joystick, the processing circuit 51a may generate a movement command instructing a target direction and target speed of movement of the robot 40. The tilt direction of the joystick corresponds to a target direction of movement of the robot 40, and the tilt angle of the joystick may represent a target speed of movement of the robot 40 in that direction.

[0071] The second server 20 transmits the movement command to the first processing terminal 30 that controls the robot 40 that is the target of the movement command. The first processing terminal 30 generates a drive command that commands a target movement of the drive part of the robot 40 and transmits it to the robot 40, in order to make the robot 40 perform the movement in accordance with the movement command.

[0072] The display 53 converts the image signal sent from the control processing terminal 51 into an image and displays it. In this specification and claims, "image" includes both still images and moving images. For example, the processing circuit 51a of the control processing terminal 51 transmits to the display 53 a signal representing the operation screen of the robot system 100 and an image signal received from the second server 20. For example, the processing circuit 51a receives from the second server 20 a signal representing one or more of the following signals: an image of a map of the robot operation area A in which the first processing terminal 30 connected to the control processing terminal 51 is located; an image captured by a camera mounted on the robot 40 to be controlled by the first processing terminal 30; and an image captured by a camera located in the robot operation area A. The remote control device 50 may include a projector that displays images similar to the display 53, instead of or in addition to the display 53.

[0073] The camera 54 captures an image of the operator of the remote control device 50 and outputs a signal of the captured image to the processing circuit 51a of the control processing terminal 51. The processing circuit 51a transmits the signal to the second server 20. The second server 20 transmits the signal to the first processing terminal 30 connected to the control processing terminal 51, to the second processing terminal 60 in the same robot operation area A as the first processing terminal 30, or to both. The first processing terminal 30 and the second processing terminal 60 display the image represented by the received signal on a display or projector that they may each have.

[0074] The acoustic device 55 includes one or more of a sound collector such as a microphone and an audio output device such as a speaker, earphones, or headphones. The sound collector collects the voice of the operator of the remote control device 50, converts it into an audio signal, and outputs it to the processing circuit 51a of the control processing terminal 51. The processing circuit 51a transmits the audio signal to the second server 20. The second server 20 transmits the audio signal to the first processing terminal 30 connected to the control processing terminal 51, to the second processing terminal 60 in the same robot operation area A as the first processing terminal 30, or to both. The first processing terminal 30 and the second processing terminal 60 output the received audio signal as audible sound from an audio output device that each of them may have.

[0075] The audio output device converts audio signals sent from the control processing terminal 51 into audible sounds and outputs the sounds. For example, the processing circuit 51a of the control processing terminal 51 transmits audio signals received from the second server 20 to the audio output device. For example, the processing circuit 51a receives, as the above-mentioned signals, audio signals representing one or more of audio collected by a sound collector provided in the first processing terminal 30 connected to the control processing terminal 51, audio collected by a sound collector provided in the second processing terminal 60 in the same robot operation area A as the first processing terminal 30, and audio collected by a sound collector arranged in the robot operation area A.

[0076] Therefore, the operator of the remote control device 50 can operate the robot 40 while communicating with a user operating a first processing terminal 30 connected to the remote control device 50, or a second processing terminal 60 in the same robot operation area A as the first processing terminal 30.

[0077] An example of the operation of the robot system 100 will be described with reference to Fig. 3. Fig. 3 is a flowchart showing an example of the operation of the robot system 100 according to the embodiment.

[0078] In step S101, a user in the robot operation area A1 inputs a command to the second processing terminal 60 to have the robot 40A execute a task TA. In this example, the robot operation area A1 is one of the robot operation areas A and is a store that provides food and drink to customers, although this is not limited thereto. The user is a service provider at the store. The robot 40A is one of the robots 40 present in the robot operation area A1. The task TA is to have the robot 40A carry a container containing a beverage from a serving location to a table. The task elements of the task TA include a task subject TA1 called the robot 40A, a task type TA2 including a receiving action, a carrying action, and a serving action of an object, a task object TA3 called the beverage container, and a task goal TA4 including a starting location called the serving location and a destination location called the table.

[0079] In step S102, the second processing terminal 60 transmits the input command and user information including the user ID and the like to the first server 10. The user ID may be set for the robot operation area A1, the second processing terminal 60, the robot 40A, or the first processing terminal 30A that controls the robot 40A. The first processing terminal 30A is one of the first processing terminals 30.

[0080] In step S103, the first server 10 authenticates the user using the received user ID, etc. If the first server 10 authenticates the user, it identifies the first processing terminal 30A that controls the robot 40A, the model of the robot 40A, and the robot operation area A1 in which the robot 40A is located, based on the received command and the user information stored in the memory 10b. Furthermore, the first server 10 identifies the second server 20A that manages the robot operation area A1, based on the area information stored in the memory 10b. The second server 20A is one of the second servers 20. If the first server 10 cannot authenticate the user, it ends the series of processes.

[0081] Furthermore, the first server 10 determines the program type of the robot program RP to be executed by the robot 40A based on the program type information of the robot program RP stored in the memory 10b. That is, the first server 10 determines the program type to be executed by the robot 40A and the first processing terminal 30A that executes the robot program RP, and determines the second server 20A to which this information is to be sent.

[0082] In step S104, the first server 10 sends an instruction to the second server 20A to cause the first processing terminal 30A to execute the robot program RP, along with information including information on the program type of the robot program RPA, information on the first processing terminal 30A, and information on the task object TA3 and the task goal TA4.

[0083] In step S105, the second server 20A extracts a robot program RPA that is compatible with the program type and executable by the robot 40A from the robot programs RP stored in the memory 20b. Furthermore, the second server 20A sets the task object and task goal included in the robot program RPA as a task object TA3 and a task goal TA4 for the robot program RPA. In other words, the second server 20A sets task elements for the robot program RPA.

[0084] In step S106, the second server 20A transmits the robot program RPA in which the task object TA3 and the task goal TA4 are set to the first processing terminal 30A designated by the first server 10.

[0085] In step S107, the first processing terminal 30A executes the robot program RPA in which the task object TA3 and the task goal TA4 are set, thereby sequentially generating drive commands and outputting them to the robot 40A. For example, the first processing terminal 30A generates drive commands at points on the trajectory of the robot 40A's operation where the direction or speed of the robot 40A's movement changes, and at predetermined intervals on the trajectory. The predetermined intervals may be time intervals or distance intervals. The robot 40A operates in accordance with the drive commands to sequentially perform the task TA in accordance with the task elements included in the task TA.

[0086] In step S108, each time the robot 40A operates in accordance with the drive command, the first processing terminal 30A acquires from the robot 40A the operation results of the drive parts of the robot 40A and the detection results of the sensors equipped in the robot 40A as log data, and transmits the log data to the second server 20. The second server 20 stores the received log data in the storage device 20b.

[0087] In step S109, if the first processing terminal 30A determines that the robot program RPA has ended (Yes in step S109), the series of processes ends. The first processing terminal 30A may transmit information about the completion of task TA to the second processing terminal 60. If the first processing terminal 30A determines that the robot program RPA is being executed (No in step S109), the process returns to step S107.

[0088] In this way, even if the first processing terminal 30 does not have a robot program RP capable of executing a requested task, it can cause the robot 40 to execute the task by acquiring the robot program RP from the second server 20. The second communication network NB connecting the first processing terminal 30 and the second server 20 can send and receive large amounts of data at high speed, thereby reducing communication delays for large amounts of data such as the robot program RP. Management of information about users and processing using this information are performed by the first server 10, thereby ensuring confidentiality. The amount of processing on the first server 10 and the amount of data sent and received from the first server 10 are small, thereby reducing communication delays.

[0089] In this embodiment, when an abnormality occurs in the operation of the robot 40 in accordance with the robot program RP, the first processing terminal 30 that controls the robot 40 switches the control mode from the automatic operation mode to the manual operation mode. Detection of an abnormality in the operation of the robot 40 is performed by the first processing terminal 30, the second server 20, or both of them.

[0090] For example, the processing circuit 30a of the first processing terminal 30 and the processing circuit 20a of the second server 20 may determine whether or not there is an abnormality in the operation of the robot 40 based on one or more of the results of comparing the robot program RP with the log data of the robot 40, the detection results of sensors equipped in the robot 40, the detection results of sensors placed in the surrounding environment of the robot 40, and the voltage or current values ​​of motors or other driving devices equipped in the robot 40.

[0091] For example, the processing circuits 20a and 30a may determine that there is an abnormality in the operation of the robot 40 when they determine that the position or operation of the robot 40 indicated by the log data deviates from the target position or operation of the robot 40 in accordance with the robot program RP. The processing circuits 20a and 30a may determine that there is an abnormality in the operation of the robot 40 when they determine that the detected value of a sensor or its behavior indicates an abnormal state. The processing circuits 20a and 30a may determine that there is an abnormality in the operation of the robot 40 when they determine that the voltage value or current value of a motor that drives the robot 40 or its behavior indicates an abnormal state. When the processing circuit 30a of the first processing terminal 30 determines that there is an abnormality in the operation of the robot 40, it transmits the determination result to the second server 20.

[0092] 4, the operation of the robot system 100 according to the embodiment when an abnormality occurs in the operation of the robot 40 will be described. Fig. 4 is a flowchart showing an example of the operation of the robot system 100 according to the embodiment when an abnormality occurs in the operation of the robot 40. Fig. 4 shows an example in which the first processing terminal 30 detects an abnormality in the operation of the robot 40.

[0093] During the following steps S201 to S213, while the robot 40 is executing a task in accordance with the robot program RP, the first processing terminal 30 acquires, at predetermined timings, from the robot 40, operation results of the driving parts of the robot 40, such as the current values ​​of the motors of the robot 40, and detection results of the sensors equipped in the robot 40, as log data, and stores the log data in the storage device 30b. For example, the first processing terminal 30 acquires the log data at predetermined time intervals. Furthermore, the first processing terminal 30 transmits the log data to the second server 20, and the second server 20 stores the log data in the storage device 20b.

[0094] In step S201, the first processing terminal 30 causes the robot 40 to operate in accordance with the robot program RP, thereby causing the robot 40 to autonomously perform a task.

[0095] In step S202, the first processing terminal 30 determines whether there is an abnormality in the operation of the robot 40, that is, whether there is an abnormality in the robot 40. If the first processing terminal 30 determines that there is an abnormality (Yes in step S202), the process proceeds to step S203, and if the first processing terminal 30 determines that there is no abnormality (No in step S202), the process proceeds to step S204. The first processing terminal 30 determines whether there is an abnormality using information acquired as log data. The first processing terminal 30 may execute step S202 at predetermined time intervals.

[0096] In step S204, the first processing terminal 30 determines whether the robot program RP has been completed. If the first processing terminal 30 determines that the robot program RP has been completed (Yes in step S204), it ends the series of processes related to the task. If the first processing terminal 30 determines that the robot program RP has not been completed (No in step S204), it returns to step S201.

[0097] In step S203, the first processing terminal 30 stops the operation of the robot 40 in accordance with the robot program RP, and transmits a report of the occurrence of an abnormality in the robot 40 to the second server 20.

[0098] In step S205, the second server 20 determines a remote control device 50 that will deal with the abnormality of the robot 40. For example, based on the information about the remote control devices 50 stored in the memory 20b, the second server 20 selects, from the remote control devices 50 connected to the second server 20, a remote control device 50 that is suitable for the task of the robot program RP being executed by the first processing terminal 30. Examples of the information about the remote control device 50 may include one or more of the type of the input device 52 of the remote control device 50, tasks suitable for manual operation with the remote control device 50, the operator of the remote control device 50, the proficiency level of the operator, tasks for which the operator is good at manual operation, and the operating status of the remote control device 50.

[0099] In step S206, the second server 20 transmits a command to the determined remote control device 50 to deal with the abnormality of the robot 40, and connects the remote control device 50 and the first processing terminal 30 so that they can communicate with each other.

[0100] In step S207, the second server 20 commands the first processing terminal 30 to switch control from the automatic steering mode to the manual steering mode, and the first processing terminal 30 executes control in the manual steering mode.

[0101] In step S208, the second server 20 reproduces normal operations of the robot 40 in accordance with the robot program RP in the virtual space model VAM2 of the robot operation area A2 stored in the memory 20b. The robot operation area A2 is the robot operation area A in which the robot 40 in which the abnormality occurred is located. The second server 20 applies the task element information, the robot program RP, and the robot information stored in the memory 20b to the virtual space model VAM2. The second server 20 may use, as the virtual space model VAM2, a virtual space model of the entire robot operation area A2, or a virtual space model of the operation region of the robot 40 in accordance with the robot program RP.

[0102] The information on the task elements includes information on the task subject, i.e., the robot 40, the task type, the task object, and the task goal. The information on the task object may include one or more of specification information such as shape, dimensions, weight, and material, state information of the object, and three-dimensional model data of the object.

[0103] The robot information may include information about the robot 40, information about sensors mounted on the robot 40, and environmental information. The information about the robot 40 is information for identifying the robot 40 and may include identification information or model information. The environmental information may include one or more of detailed information about the task object, information about obstacles, and information about environmental conditions such as constraints on the robot 40. The obstacles may be obstacles added after the virtual space model VAM2 is constructed. The second server 20 may identify obstacles using detection results from sensors arranged in the robot operation area A2, or may identify obstacles using obstacle information notified by the user via the second processing terminal 60.

[0104] The virtual space model VAM includes virtual object models of various movable objects, immovable objects, and living creatures that exist within the robot operating area A, and also includes a virtual controller for the robot 40. The virtual controller is an engine that operates the virtual robot model using the robot program RP and a controller such as the remote control device 50, and can be realized by a robot simulator or a physics engine, etc.

[0105] Therefore, the second server 20 can cause the virtual robot model 40M of the robot 40 to reproduce normal operation in accordance with the robot program RP when there is no abnormality in the robot model 40M within the virtual space model VAM2, just as in the real world.

[0106] In step S209, the second server 20 causes the virtual space model VAM2 to reproduce the actual movement of the robot 40 in accordance with the log data. Specifically, the second server 20 applies the log data and robot information stored in the memory 20b to the virtual space model VAM2. The log data includes log data of the robot 40 during actual movement as the movement results of the driving parts of the robot 40. The log data of the robot 40 during actual movement includes one or more of the detection results of sensors detecting the rotation angles of the joints of the driving parts, the current values ​​of the motors of the driving parts, and the drive command values ​​sent from the first processing terminal 30 to the robot 40. The log data may include the detection results of external sensors included in the robot 40A as the detection results of the sensors included in the robot 40A. Examples of the external sensors may include force sensors, tactile sensors, force-tactile sensors, position sensors, distance sensors, vibration sensors, temperature sensors, and vision sensors such as cameras.

[0107] The second server 20 can cause the virtual robot model 40M of the robot 40 to operate in the virtual space model VAM2 in the same manner as the actual operation of the robot 40, in accordance with the log data of the robot 40 during the actual operation. In other words, the second server 20 can cause the virtual robot model 40M to reproduce the actual operation of the robot 40. Furthermore, the second server 20 can reflect the detection results of the external sensors in the virtual space model VAM2 in a virtual object model around the virtual robot model 40M or as environmental information around the robot model 40M. For example, the second server 20 can generate a virtual model of an obstacle not included in the virtual space model VAM2 or information about the obstacle. This allows the second server 20 to update the state of the virtual space model VAM2.

[0108] In step S210, the second server 20 transmits data representing the virtual space model VAM2 to the remote control device 50. Examples of the data may include data of a three-dimensional model of the virtual space model VAM2, data of a two-dimensional model of the virtual space model VAM2, image data of the three-dimensional model of the virtual space model VAM2 captured by a virtual camera included in the virtual space model VAM2, and image data of the two-dimensional model of the virtual space model VAM2 captured by a virtual camera included in the virtual space model VAM2. The virtual camera will be described later. In this example, the second server 20 transmits image data representing the virtual space model VAM2 to the remote control device 50. The remote control device 50 displays an image representing the received image data on the display 53. The second server 20 may display the image on a local site or a web site that is accessible to the remote control device 50 and that can accept input from the remote control device 50.

[0109] In step S208, the second server 20 generates normal operation image data, which is image data representing the virtual space model VAM2 to which the robot program RP has been applied, and in step S209, actual operation image data, which is image data representing the virtual space model VAM2 to which the log data has been applied. The second server 20 generates both sets of image data as multiple images or video data representing changes in the virtual space model VAM2 from a timing that is a specific time before the detection timing at which an abnormality in the robot 40 was detected in step S202 to the detection timing. The normal operation image data and actual operation image data can represent normal operation and operation similar to the actual operation of the virtual robot model 40M from the specific timing to the detection timing, respectively. The second server 20 may generate normal operation image data and actual operation image data for portions where there is a difference between the normal operation and the actual operation.

[0110] The virtual space model VAM2 includes a virtual camera that captures images within the virtual space model VAM2. The virtual space model VAM2 can change the position, attitude, and zoom ratio of the virtual camera through input operations. The normal operation image data and actual operation image data are image data captured by the virtual camera. In this embodiment, the virtual space model VAM2 is a 3DCG model, and therefore the image data is CG image data.

[0111] The second server 20 may transmit image data representing the normal operation image data VN and the actual operation image data VR side by side to the remote control device 50, as shown in FIG. 5, or may transmit image data representing the normal operation image data VN and the actual operation image data VR superimposed on each other to the remote control device 50, as shown in FIG. 6. In FIG. 6, the normal operation image data VN is indicated by a dashed line, and the actual operation image data VR is indicated by a solid line. The second server 20 may obtain the superimposed image data by image processing the normal operation image data VN and the actual operation image data VR. Alternatively, the second server 20 may obtain the superimposed image data by performing steps S208 and S209 in parallel on two virtual robot models 40M in the virtual space model VAM2. FIGS. 5 and 6 are diagrams showing examples of images displayed on the remote control device 50 by the second server 20, illustrating the operation of the robot arm of the robot model 40M grasping an object W with its hand.

[0112] The second server 20 may input normal action image data into the language model LM stored in the storage device 20b to obtain text data describing each of the states of multiple scenes included in the normal action image data. The second server 20 may generate image data in which text data is superimposed on each of the images of the multiple scenes, as shown in FIG. 7, for example, and transmit the image data to the remote control device 50. Similarly, the second server 20 may input actual action image data into the language model LM to obtain text data describing each of the states of multiple scenes included in the actual action image data. The second server 20 may generate image data in which text data is superimposed on each of the images of the multiple scenes, as shown in FIG. 7, for example, and transmit the image data to the remote control device 50. FIG. 7 is a diagram illustrating an example of an image in which text data is superimposed on the image of FIG. 5.

[0113] In order to clarify the differences between the virtual robot model 40M and the virtual surrounding environment model other than the virtual robot model 40M, the second server 20 may process the virtual space model VAM2 to make the virtual robot model 40M and the virtual surrounding environment model different from each other in one or more of color, shading, transparency, and texture. For example, the second server 20 may process the surrounding environment model differently depending on whether the surrounding environment model is fixed or movable and on the hardness of the surface of the surrounding environment model. The second server 20 may transmit image data of the processed virtual space model VAM2 to the remote control device 50.

[0114] By visually checking the normal operation image data and the actual operation image data in step S210, the operator of the remote control device 50 can confirm the process leading up to an abnormality in each of the planned operation of the robot 40 to be executed and the actual operation of the robot 40. This makes it easier for the operator to identify the cause of the abnormality and find a way to resolve it.

[0115] In step S211, the second server 20 allows the remote control device 50 to accept input of an operation to the virtual space model VAM2 and the remote control device 50 to accept input of a manual operation to the robot 40.

[0116] For example, the operator can use the remote control device 50 to input changes to the position, attitude, and zoom ratio of the virtual camera in the virtual space model VAM2, and the second server 20 can change the position, attitude, and zoom ratio of the virtual camera in accordance with the input content. This allows the operator to visually check the robot model 40M and its surrounding conditions in various positions, directions, and sizes that cannot be seen by the cameras installed in the robot operating area A2 and the robot 40.

[0117] The operator uses the remote control device 50 to input changes to one or more of the color, shading, transparency, and texture of elements in the virtual space model VAM2, and the second server 20 can change the color, shading, transparency, and texture of the elements in accordance with the input. For example, the operator can change the elements that are displayed overlapping the robot model 40M or the robot model 40M itself to semi-transparent or transparent, allowing the operator to visually confirm the relationship between them.

[0118] The operator uses the remote control device 50 to input manual operations for the robot model 40M in the virtual space model VAM2, and the second server 20 can operate the robot model 40M in accordance with the manual operations. This allows the operator to simulate operations in advance using the robot model 40M in the virtual space model VAM2 before operating the actual robot 40 to resolve an abnormality.

[0119] The operator uses the remote control device 50 to input commands to the robot model 40M in the virtual space model VAM2 to operate in accordance with the robot program RP, and the second server 20 can operate the robot model 40M in accordance with the robot program RP. This allows the operator to perform a simulation of completing a task in accordance with the robot program RP using the robot model 40M in the virtual space model VAM2 before operating the actual robot 40 to resolve an abnormality.

[0120] By manipulating the virtual space model VAM2 in step S211, the operator of the remote control device 50 can easily identify the cause of the abnormality in the robot 40 and find a way to resolve the abnormality.

[0121] In step S212, the operator inputs a manual operation to the input device 52 of the remote control device 50 to resolve the abnormality. The remote control device 50 transmits a movement command corresponding to the manual operation to the second server 20, and the second server 20 transmits the movement command to the first processing terminal 30. The first processing terminal 30 causes the robot 40 to perform an operation in accordance with the movement command, i.e., an operation in accordance with the manual operation.

[0122] The operator visually checks the normal operation image data and the actual operation image data in step S210, and operates the virtual space model VAM2 to perform a simulation in step S211, thereby being able to understand the operations that the robot 40 has performed and the operations that the robot 40 should perform. This makes it easy for the operator to manually operate the robot 40 even if they do not understand the details of the robot program RP.

[0123] The second server 20 receives image data captured by the camera mounted on the robot 40 and image data captured by the camera located in the robot operation area A2 from the first processing terminal 30, and transmits the data to the remote control device 50. This allows the operator to manually operate the robot 40 using the remote control device 50 while viewing the captured images on the display 53.

[0124] The second server 20 may transmit image data representing the virtual space model VAM2 to the remote control device 50 in addition to the captured image data, and may accept input of operations to the virtual space model VAM2 from the remote control device 50. This allows the operator to manually operate the robot 40 while viewing the captured image on the display 53 and operating the virtual space model VAM2 to check areas not represented by the captured image.

[0125] In step S213, when the second server 20 determines that the abnormality in the robot 40 has been resolved, it disconnects the remote control device 50 from the first processing terminal 30 and sends a command to the first processing terminal 30 to switch from manual control mode to automatic control mode. As a result, the first processing terminal 30 resumes control of the robot 40 in accordance with the robot program RP and returns to step S201.

[0126] The second server 20 may determine that the abnormality of the robot 40 has been resolved by receiving a notification from the remote control device 50 that the abnormality has been resolved. The second server 20 may determine that the abnormality of the robot 40 has been resolved when it determines that the operation of the robot 40 controlled by the remote control device 50 has reached a preset stable state. The stable state is a state in which the autonomous operation of the robot 40 can be restored, and may be, for example, a state in which the operation of the robot 40 controlled by the remote control device 50 has become similar to the operation in accordance with the robot program RP.

[0127] Steps S201 to S213 allow a smooth transition to manual operation by the operator to deal with an abnormality that occurs in the operation of the robot 40 during the robot 40's autonomous operation.

[0128] In step S210, the second server 20 transmits the normal operation image data and the actual operation image data to the remote control device 50. In addition to these, the second server 20 may be configured to estimate and transmit the cause of the abnormality. In this case, the second server 20 may store an estimation model, which is a machine learning model, in the storage device 20b. The estimation model may be constructed so that input data is one or more of the actual operation image data, text data generated for the actual operation image data by a language model LM, robot information, and simulation data of the robot 40 in accordance with the robot program RP, and the like, and the cause of the abnormality is output data.

[0129] Instead of text data, parameter values ​​used in the language model LM to generate the text data may be used. The simulation data may be data obtained by a simulation using the virtual space model VAM. The estimation model may further include an error code of the robot's autonomous control as input data. The estimation model may further include a method for dealing with the abnormality as output data. The estimation model is machine-learned using past data corresponding to the input data and past data corresponding to the output data.

[0130] For example, the estimation model may be a model that performs supervised machine learning using past data corresponding to the input data as learning input data and past data corresponding to the output data as training data. For example, the estimation model may be constructed by deep learning using a neural network or the like.

[0131] For example, the estimation model may be constructed so that input data is part or all of the log data, such as that used in step S209, including a process in which an abnormality occurs in the operation of the robot 40, and output data is information related to the operation of the robot 40 to deal with the abnormality. The information related to the operation of the robot 40 may be information indicating the operation of the robot 40 to deal with the abnormality, manual operation of the robot 40 to deal with the abnormality, or both. For example, the estimation model may be constructed by performing supervised machine learning. In this case, the learning input data may include various log data including the process of various abnormalities in the operation of the robot 40 that occurred in the past. The training data may include data indicating manual operations input to the remote control device 50 to deal with each of the abnormalities corresponding to the log data, the operation of the robot 40 in accordance with the manual operation, or both.

[0132] The second server 20 may associate log data including the process of past abnormalities in the operation of the robot 40 with information indicating manual operations input to the remote control device 50 to deal with the abnormalities, the operation of the robot 40 in accordance with the manual operations, or both, and store the associated data in the storage device 20b as machine learning data. The second server 20 may perform machine learning of an estimation model using the stored machine learning data.

[0133] In step S209, the second server 20 uses a language model LM that takes image data as input data and outputs text data indicating the state represented by the image data, but the language model LM is not limited to this. In addition to normal operation image data or actual operation image data, the language model LM may further include simulation data and log data of the robot 40 in accordance with the robot program RP as input data. The language model LM may further include detection results from sensors placed in the robot operation area A as input data. The language model LM performs machine learning using past data corresponding to the input data and past data corresponding to the output data.

[0134] For example, the language model LM may be a model that performs supervised machine learning, similar to the estimation model, using past data corresponding to the input data as learning input data and past data corresponding to the output data as training data. For example, the language model LM may be constructed by deep learning using a neural network or the like.

[0135] The second server 20 may associate previously generated data corresponding to input data of the language model LM including image data with text data generated for the previously generated data, and store the data as machine learning data in the storage device 20 b. The second server 20 may perform machine learning of the language model LM using the stored machine learning data.

[0136] [Others] Illustrative embodiments of the present disclosure have been described above, but the present disclosure is not limited to the above-described embodiments. That is, various modifications and improvements are possible within the scope of the present disclosure. For example, various modifications to the embodiments and embodiments constructed by combining components of different embodiments are also included within the scope of the present disclosure.

[0137] For example, the robot system 100 according to the embodiment includes the first server 10 and the second server 20, but the first server 10 and the second server 20 may be integrated into a single server. For example, the second server 20 may be configured to have the functions of the first server 10, or the first server 10 may be configured to have the functions of multiple second servers 20.

[0138] For example, in the embodiment, the second server 20 accumulates machine learning data for the language model LM and the estimation model, and performs machine learning on the language model LM and the estimation model using the accumulated machine learning data, but this is not limited to this. For example, the second server 20 may accumulate machine learning data for the language model LM and the estimation model, and the first server 10 may collect machine learning data from various second servers 20. The first server 10 may perform machine learning on the language model LM and the estimation model using the collected machine learning data. This enables machine learning on the language model LM and the estimation model using a larger amount of machine learning data, thereby improving the accuracy of the language model LM and the estimation model.

[0139] Examples of aspects of the technology of the present disclosure are as follows: A robot system according to a first aspect of the present disclosure includes a robot, a first processing circuit that controls the robot, a second processing circuit that collects and accumulates log data including information related to a state of the robot during a process in which the first processing circuit causes the robot to operate according to an operation program, a first memory that stores a virtual space model including a virtual robot model and a virtual surrounding environment model that reproduce the robot and the robot's surrounding environment, a third processing circuit that outputs data representing the virtual space model to a display device, and a fourth processing circuit that monitors the robot and detects abnormalities in the robot's operation, wherein when an abnormality is detected by the fourth processing circuit, the third processing circuit applies the log data accumulated by the second processing circuit to the virtual space model to reproduce the robot's operation according to the log data in the virtual space model, and outputs data representing the virtual space model reproduced according to the log data to the display device.

[0140] According to the first aspect, when a robot system detects an abnormality in the behavior of a real robot, the robot system applies log data to a virtual space model to reproduce the robot's behavior according to the log data in the virtual space model, and outputs data representing the virtual space model reproduced according to the log data to a display device. This allows an operator dealing with the robot's abnormality to check the reproduction of the robot's behavior by looking at the display device. This makes it easier for the operator to respond when the robot performs an unintended behavior. The display device may be any device capable of displaying data representing the virtual space model. For example, the display device may be a device equipped with a display or a projector. Examples of the data representing the virtual space model may include data of a three-dimensional model of the virtual space model, data of a two-dimensional model of the virtual space model, image data of the three-dimensional model of the virtual space model captured by a virtual camera of the virtual space model, and image data of the two-dimensional model of the virtual space model captured by a virtual camera of the virtual space model.

[0141] In the first aspect above, in the robot system according to the second aspect of the present disclosure, the third processing circuit may apply the log data to the virtual space model, thereby reproducing in the virtual space model the behavior of the robot during the period from a specific point in time before the detection of the abnormality to the point in time when the abnormality is detected, in accordance with the log data.

[0142] According to the second aspect, the robot system can reproduce in the virtual space model the robot's behavior from a specific point before the abnormality is detected to the point at which the abnormality is detected. This allows an operator dealing with the robot's abnormality to view the display of the display device and confirm a reproduction of the process by which the robot's behavior led to the abnormality. This makes it even easier for the operator to respond when the robot behaves unintentionally.

[0143] In the first or second aspect described above, in a robot system according to a third aspect of the present disclosure, the third processing circuit may output to the display device data representing both a first robot model that reproduces the operation of the robot in the virtual space model according to the log data by applying the log data to the virtual space model, and a second robot model that operates according to the operation program by applying the operation program to the virtual space model.

[0144] According to the third aspect, the robot system can be represented by a first robot model that reproduces operations according to the log data and a second robot model that reproduces operations according to the operation program, allowing an operator dealing with an abnormality in the robot to easily identify the cause of the abnormality by comparing the two robot models.

[0145] In any of the first to third aspects described above, a robot system according to a fourth aspect of the present disclosure may include a second memory device that stores a language model that receives image data as input data and outputs text data indicating a state represented by the image data, and a fifth processing circuit that inputs image data representing the virtual space model into the language model, generates text data indicating a state of the virtual space model, and outputs the text data to the display device.

[0146] According to the fourth aspect, the robot system can express the state of the virtual space model represented by the image data in text data, which makes it easier for an operator dealing with an abnormality in the robot to understand the reproduced robot behavior.

[0147] In any of the first to fourth aspects above, in a robot system according to a fifth aspect of the present disclosure, the third processing circuit may construct the virtual space model including a virtual camera that can be placed at any position within the virtual space model, and output image data of the virtual space model captured by the virtual camera to the display device.

[0148] According to the fifth aspect, the robot system can output image data representing a virtual space model captured from various positions via a virtual camera. This allows an operator dealing with an abnormality in the robot to view the robot's behavior reproduced in the virtual space model from various positions by changing the position of the virtual camera. This allows the operator to easily identify the cause of the abnormality.

[0149] In any of the first to fifth aspects above, in a robot system according to a sixth aspect of the present disclosure, the third processing circuit may output data of the virtual space model in which the virtual robot model and the virtual surrounding environment model have been processed to have different one or more of color, shading, transparency, and texture.

[0150] According to the sixth aspect, the robot system can display the virtual robot model and the virtual surrounding environment model in an easily visible manner, thereby enabling the operator to easily identify the cause of the abnormality.

[0151] In any of the first to sixth aspects above, a robot system according to a seventh aspect of the present disclosure may include a controller that accepts input of a manual operation that causes the robot to operate in accordance with the input operation, and outputs an operation command to the first processing circuit that is a command that causes the robot to operate in accordance with the manual operation, and the third processing circuit may output data representing the virtual space model to the controller, and the controller may display the data on a display provided in the controller.

[0152] According to the seventh aspect, the robot system can display data representing the virtual space model on a display of a controller that can manually operate the robot, thereby allowing an operator to manually operate the robot to deal with an abnormality while visually checking the virtual space model via the display.

[0153] In the seventh aspect above, in a robot system according to an eighth aspect of the present disclosure, when the abnormality is detected, the third processing circuit may apply the log data to the virtual space model to reproduce the behavior of the robot according to the log data in the virtual space model, and output data representing the virtual space model reproduced according to the log data to the controller.

[0154] According to the eighth aspect, the robot system can display data representing the virtual space model that has been reproduced according to the log data on the display of the controller, thereby allowing the operator to manually operate the robot while visually checking the reproduced virtual space model via the display.

[0155] In any of the above first to eighth aspects, a robot system according to a ninth aspect of the present disclosure may further include a third memory that stores an estimation model that uses log data including information related to the state of the robot when an abnormality in the robot's behavior is detected as input data and information related to the behavior of the robot for dealing with the abnormality as output data, the estimation model being machine-learned using the log data including information related to the state of the robot when an abnormality in the robot's behavior is detected and information related to the behavior of the robot performed to deal with the abnormality; and a sixth processing circuit that, when an abnormality is detected by the fourth processing circuit, inputs the log data accumulated by the second processing circuit to the estimation model and outputs data output by the estimation model to the display device.

[0156] According to the ninth aspect, when an abnormality in the robot's operation is detected, the robot system uses the estimation model to output information related to the robot's operation for dealing with the abnormality to the display device. This allows the operator dealing with the robot's abnormality to refer to the information for dealing with the abnormality, making it easier to deal with the robot's abnormality.

[0157] In any of the first to ninth aspects above, a robot system according to a tenth aspect of the present disclosure may include a server having the second processing circuit, the third processing circuit, and the first memory, a processing terminal having the first processing circuit and communicatively connected to the server via a communication network, and the display device communicatively connected to the server via the communication network, wherein the fourth processing circuit is provided on the server or the processing terminal, the server has a fourth memory that stores a plurality of the operation programs corresponding to a plurality of tasks, the server transmits the operation program corresponding to a requested task to the processing terminal, and the processing terminal uses the received operation program to cause the robot to autonomously execute the requested task.

[0158] According to the tenth aspect, the robot system can cause the robot to execute various operation programs stored in the server via the processing terminal. This makes it possible to reduce the processing power and data capacity required of the processing terminal. Furthermore, because the server stores the virtual space model and performs processing related to the virtual space model, it is possible to reduce the processing power and data capacity required of the processing terminal.

[0159] A control method for a robot system according to an eleventh aspect of the present disclosure includes: causing a robot to operate in accordance with an operation program; collecting and accumulating log data including information related to a state of the robot during the process of causing the robot to operate in accordance with the operation program; generating a virtual space model including a virtual robot model and a virtual surrounding environment model that reproduce the robot and the environment surrounding the robot; detecting an abnormality in the operation of the robot; and, when the abnormality is detected, applying the log data to the virtual space model to reproduce the operation of the robot in accordance with the log data in the virtual space model; displaying data representing the virtual space model reproduced in accordance with the log data on a display provided in a controller that accepts input of a manual operation and outputs operation commands to cause the robot to operate in accordance with the manual operation; and causing the robot to operate in accordance with the operation commands output by the controller.

[0160] According to the eleventh aspect, the same effects as those of the robot systems according to the other aspects of the present disclosure can be obtained. Part or all of the method of the present disclosure may be realized, for example, by a CPU, a circuit such as an LSI, an IC card, or a standalone module. Multiple elements included in the method of the present disclosure may be realized by a single device, or may be realized by two or more devices sharing the same functions.

[0161] In the above-mentioned eleventh aspect, the control method for a robot system according to a twelfth aspect of the present disclosure may further include accepting an operation of the virtual space model by the controller, and displaying, on the display, data representing the virtual space model that reflects the movement of the virtual robot model in accordance with the accepted operation.

[0162] According to the twelfth aspect, the operator of the controller can operate the virtual space model to operate the virtual robot model while visually checking the virtual space model, thereby enabling the operator to simulate operations to deal with an abnormality before operating the robot.

[0163] The present disclosure may also be a computer program that causes a computer to execute a method according to each aspect of the present disclosure. Such a computer program can achieve the same effects as the method according to each aspect of the present disclosure. The computer program may, for example, be a program recorded on a non-transitory, tangible, computer-readable recording medium, and may be configured to be read from the recording medium using a recording medium drive device and installed on a computer. The computer program may, for example, be a program that can be distributed via a transmission medium such as the Internet, and may be configured to be downloaded and installed on a computer.

[0164] The functions of the elements disclosed herein can be performed using circuits or processing circuits, including general-purpose processors, special-purpose processors, integrated circuits, ASICs, conventional circuits, and / or combinations thereof, configured or programmed to perform the disclosed functions. A processor is considered a processing circuit or circuit because it includes transistors and other circuitry. In this disclosure, a circuit, unit, or means is hardware that performs the recited functions or hardware that is programmed to perform the recited functions. The hardware may be hardware disclosed herein or other known hardware that is programmed or configured to perform the recited functions. Where the hardware is a processor, which is considered a type of circuit, the circuit, means, or unit is a combination of hardware and software, and the software is used to configure the hardware and / or processor.

[0165] All numbers such as ordinal numbers and quantities used in this specification are provided as examples to specifically explain the technology of the present disclosure, and the present disclosure is not limited to the illustrated numbers. The connection relationships between components are provided as examples to specifically explain the technology of the present disclosure, and the connection relationships that realize the functions of the present disclosure are not limited to these.

[0166] Because the present disclosure may be embodied in various forms without departing from the scope of its essential characteristics, the scope of the present disclosure is defined by the appended claims rather than the description in the specification, and therefore the exemplary embodiments and modifications are intended to be illustrative and not limiting. All modifications within the scope of the claims and their equivalents are intended to be embraced by the claims.

Claims

1. A robot system comprising: a robot; a first processing circuit that controls said robot; a second processing circuit that collects and accumulates log data including information related to the state of the robot during a process in which said first processing circuit causes said robot to operate in accordance with an operating program; a first memory that stores a virtual space model including a virtual robot model and a virtual surrounding environment model that reproduces said robot and the robot's surrounding environment; a third processing circuit that outputs data representing the virtual space model to a display device; and a fourth processing circuit that monitors the robot and detects abnormalities in the operation of the robot, wherein when an abnormality is detected by the fourth processing circuit, the third processing circuit applies the log data accumulated by the second processing circuit to the virtual space model, thereby reproducing the operation of the robot in accordance with the log data in the virtual space model, and outputs data representing the virtual space model that has been reproduced in accordance with the log data to the display device.

2. The robot system according to claim 1, wherein the third processing circuit applies the log data to the virtual space model, thereby reproducing in the virtual space model the operation of the robot during the period from a specific point before the detection of the abnormality to the point of detection of the abnormality, in accordance with the log data.

3. The robot system described in claim 1, wherein the third processing circuit outputs to the display device data representing both a first robot model that reproduces the movement of the robot in the virtual space model in accordance with the log data by applying the log data to the virtual space model, and a second robot model that operates in accordance with the operation program by applying the operation program to the virtual space model.

4. The robot system according to claim 1, further comprising: a second memory device that stores a language model that receives image data as input data and outputs text data indicating the state represented by the image data; and a fifth processing circuit that inputs image data representing the virtual space model into the language model, generates text data indicating the state of the virtual space model, and outputs the text data to the display device.

5. The robot system according to claim 1, wherein the third processing circuit constructs the virtual space model including a virtual camera that can be positioned at any position within the virtual space model, and outputs image data of the virtual space model captured by the virtual camera to the display device.

6. The robot system according to claim 1, wherein the third processing circuit outputs data of the virtual space model in which the virtual robot model and the virtual surrounding environment model have been processed to have one or more of different colors, shading, transparency, and textures.

7. The robot system according to claim 1, further comprising a controller that accepts input of manual operations that cause the robot to operate in accordance with the input operation, and outputs to the first processing circuit a manipulation command that is a command to cause the robot to operate in accordance with the manual operation, the third processing circuit outputs data representing the virtual space model to the controller, and the controller displays the data on a display provided in the controller.

8. The robot system described in claim 7, wherein when the abnormality is detected, the third processing circuit applies the log data to the virtual space model, thereby reproducing the robot's movement in accordance with the log data in the virtual space model, and outputs data representing the virtual space model reproduced in accordance with the log data to the controller.

9. The robot system of claim 1, further comprising: a third memory device that stores an estimation model that uses log data including information related to the state of the robot when an abnormality in the robot's behavior is detected as input data and information related to the robot's behavior to deal with the abnormality as output data, the estimation model being machine-learned using the log data including information related to the robot's state when an abnormality in the robot's behavior is detected and information related to the robot's behavior performed to deal with the abnormality; and a sixth processing circuit that, when an abnormality is detected by the fourth processing circuit, inputs the log data accumulated by the second processing circuit into the estimation model and outputs data output by the estimation model to the display device.

10. A robot system as described in claim 1, comprising: a server having the second processing circuit, the third processing circuit, and the first memory; a processing terminal having the first processing circuit and communicatively connected to the server via a communication network; and the display device communicatively connected to the server via the communication network, wherein the fourth processing circuit is provided in the server or the processing terminal, the server has a fourth memory which stores a plurality of the operation programs corresponding to a plurality of tasks, the server transmits the operation program corresponding to a requested task to the processing terminal, and the processing terminal uses the received operation program to cause the robot to autonomously execute the requested task.

11. A method for controlling a robot system, comprising: causing a robot to operate in accordance with an operating program; collecting and accumulating log data including information related to a state of the robot during the process of causing the robot to operate in accordance with the operating program; generating a virtual space model including a virtual robot model and a virtual surrounding environment model that reproduce the robot and the robot's surrounding environment; detecting abnormalities in the robot's operation; when the abnormality is detected, reproducing the robot's operation in accordance with the log data in the virtual space model by applying the log data to the virtual space model; displaying data representing the virtual space model that has been reproduced in accordance with the log data on a display provided in a controller that accepts manual operation input and outputs operation commands to cause the robot to operate in accordance with the manual operation; and causing the robot to operate in accordance with the operation commands output by the controller.

12. The method of controlling a robot system according to claim 11, further comprising: accepting an operation of the virtual space model by the controller; and displaying on the display data representing the virtual space model that reflects the movement of the virtual robot model in accordance with the accepted operation.

Images