Configuration framework for robot control systems

Abstract

A method, system, and apparatus, including a computer program encoded on a computer storage medium, for a configurability framework that supports the coordination of low-level operations of multiple subsystems. [Solution] One method includes receiving a multi-device application definition for a real-time robotic control system configured to perform operations including controlling a robot execution environment having multiple robotic components, the application definition including one or more custom high-level software modules for each robot component of the multiple robotic components and a respective module interface definition for each custom high-level software module, the module interface definitions of the custom high-level software modules specifying one or more asynchronous or real-time communication channels and a respective device interface definition for each of the multiple robotic components; and processing the application definition to generate output programs to be executed by each of the multiple robotic components.

Description

Background Art

[0001] Background This specification relates to a framework for a software control system.

[0002] A software control system is a software system that controls the operation of physical devices (e.g., actuators or robots). Some software control systems are real-time software control systems that control real-time systems. A real-time software control system needs to execute control routines within strict timing requirements to achieve normal operation. The timing requirements often specify that certain operations need to be performed or outputs need to be generated within a specific time window to avoid the system entering a failure state. In a failure state, the system can take several other measures such as stopping execution or interrupting normal operation.

[0003] Such real-time software control systems are often used to control physical machines with high precision and timing requirements. As an example, the execution environment of industrial robots can be controlled by a real-time software control system that requests each robot to repeatedly receive commands at a specific frequency (e.g., 1, 10, or 100 kHz). If one of the robots does not receive a command during one of the periodic time windows, the robot can enter a failure state by stopping the operation of the robot or automatically executing a recovery procedure to return to a maintenance position.

[0004] Due to such timing requirements, software control systems for physical machines are often implemented using closed software modules specifically configured for highly specialized tasks. For example, a robot that lifts components and places them on a printed circuit board can be controlled by a closed software system that controls each of its low-level pick-and-place operations. The coordination of such software control modules often follows observation, planning, and action patterns, including sensor observations, plan generation, and actions that follow the plan. [Overview of the project] [Means for solving the problem]

[0005] overview This specification describes a configurability framework that assists in coordinating the low-level operation of multiple subsystems, which may include one or more real-time robots. Generally, a framework is a library of software modules that provide several basic functions for building applications in a particular execution environment. In this specification, the execution environment includes multiple subsystems, each having its own computing hardware. The framework provides asynchronous communication channels, low-level performance discovery, and mechanisms for coordination between multiple independent devices. The techniques described below provide a framework for building applications that coordinate multiple subsystems in an execution environment, each of which may have observation, planning, and operation functions.

[0006] Certain embodiments of the subject matter described in this specification can be implemented to achieve one or more of the following advantages:

[0007] Some existing robot application frameworks dictate the interfaces of devices and software modules, preventing users from customizing the interfaces for specific use cases. Several systems described in this application allow users to configure general-purpose low-level software modules and generate high-level software modules tailored to their needs. Users can also form the data interfaces for the software modules. Furthermore, some such systems enable high-level configurability of software modules, allowing custom high-level software modules to be further configured into even higher-level software modules. These high-level software modules can establish communication channels between each of these configuration software modules and coordinate the execution of each of these configuration software modules.

[0008] Some of the systems described in this specification may also provide a discovery mechanism for devices and software modules that declares each data interface of the device and software module according to the user's requirements. Thus, the system provides user-friendly management of software modules by making it easier for the user to invoke software modules and find the data interfaces of the software modules. Therefore, these systems can be applied to a wide range of applications and devices.

[0009] Several systems described in this specification allow a user to establish a communication channel between two or more devices. The communication channel can pass data between devices, and the devices can execute one or more software modules asynchronously.

[0010] Details of one or more embodiments of the subject matter of this specification are described below in the accompanying drawings and specification. Other features, aspects, and advantages of the subject matter will become apparent from the specification, drawings, and claims. [Brief explanation of the drawing]

[0011] Brief explanation of the drawing [Figure 1] This is a diagram illustrating an example of the system. [Figure 2] This shows an example of an execution environment with multiple devices. [Figure 3] Examples of device interface definitions and software modules are shown. [Figure 4] This is a flowchart illustrating an example of the process of generating a program that is to be executed by one or more devices in an execution environment. [Modes for carrying out the invention]

[0012] The same reference numerals and names in various drawings refer to the same element.

[0013] Detailed explanation Figure 1 is a diagram illustrating an example of system 100. System 100 is an example of a system capable of carrying out the techniques described in this specification.

[0014] System 100 includes many functional components, including a planner 120 and a real-time robot control system 150. Each of these components can be implemented as a computer program installed on one or more computers in one or more locations connected to each other via any suitable communication network (e.g., an intranet or the internet, or a combination of networks). System 100 further includes an execution environment 170 containing N robots 170a-n.

[0015] The real-time robot control system 150 is configured to control robots 170a-n in the execution environment 170. The real-time robot control system 150 must satisfy one or more strict time constraints in the operation of the execution environment 170. For example, it may require one of the robots in the execution environment 170 to perform a specific action at equal intervals (e.g., 10 milliseconds), and if the robot is unable to consistently perform the action within a given time window, the robot enters a failure state.

[0016] The real-time robot control system 150 consists of one or more software module loops, each containing a number of software modules that are repeatedly executed in a predetermined order to control robots 170a-n. For simplicity, Figure 1 shows a single software module loop containing M software modules 150a-m. Each software module can perform one or more functions of each robot 170a-n.

[0017] Each software module may have a corresponding data interface that specifies the real-time data that the software module receives as input and supplies as output. During execution, the software modules exchange P real-time messages 152a-p that each software module needs to execute. For example, software module 150a can supply real-time message 152a to software module 150b, and software module 150b can supply real-time message 152p to software module 150m.

[0018] The overall objective of Planner 120 is to generate a schedule 135 that allows the real-time robot control system 150 to control the robot in the execution environment 170 to perform a specific task.

[0019] The planner 120 receives the application definition 110. The application definition 110 can be generated by the user. The application definition 110 can identify a specific task to be completed by the robot in the execution environment 170. To complete the task, the application definition 110 can specify a software module loop, including identifying the software modules 150a-m in the software module loop. Furthermore, the application definition 110 can specify real-time messages 152a-p that are exchanged between software modules according to the data interface of the software modules.

[0020] A portion of the software modules 150a-m may be a high-level software module composed of one or more low-level software modules and data interfaces generated by the user using low-level software modules. In other words, a custom high-level software module may depend on one or more low-level software modules.

[0021] The low-level software modules can perform the basic functions of each robot 170a-n. In some implementations, each low-level software module depends on a custom software module that is customized for each robot. That is, the custom software module can perform specific low-level functions that are designed for a particular robot, for example, tailored for the specific specifications of a particular robot. In some cases, the low-level software modules can depend on custom software modules provided by the robot manufacturer, that is, the custom software modules can be provided to the robot user without additional configuration.

[0022] Each software module can be associated with one or more of the robots in the execution environment 170, and a device interface definition can be used to identify the software modules supported by a given device. This process will be described in more detail later with reference to FIGS. 2 and 3.

[0023] The planner 120 uses the application definition 110 to generate a schedule 135 for the real-time robot control system 150, executes a software module loop, and completes a specific task.

[0024] The planner 120 provides the schedule 135 to the real-time robot control system 150 and executes a software module loop according to the schedule 135. That is, the real-time robot control system 150 executes software modules 150a-m, and the real-time robot control system 150 issues commands 155 to the execution environment 170 to drive the movement of robots 170a-n. While controlling the execution environment 170, the software modules 150a-m exchange real-time messages 152a-p.

[0025] FIG. 2 shows an example of an execution environment 200. The execution environment includes several robot components including a robot 210 having an arm 215, a belt conveyor 220, and a camera 230.

[0026] Each device in the execution environment has a device interface definition that declares which software modules each device can execute. For example, robot 210 has a robot interface 210a that specifies a list of software modules that the robot can execute, including high-level software modules 1A and 1B, and low-level software module 1C. Each high-level software module can be composed of one or more low-level software modules and / or one or more other high-level software modules. The configurable software modules will be described in more detail later with reference to Figure 3.

[0027] Similarly, the belt conveyor 220 has a belt conveyor interface 220a that specifies a list of software modules that the belt conveyor can execute, and the camera 230 has a camera interface 230a that specifies a list of software modules that the camera can execute.

[0028] A user generating application definitions for devices in an execution environment 200 to accomplish a specific task can use device interface definitions 210a-c to determine which software modules are executable by the devices in environment 200 and configure which software modules are available to accomplish the task. As a specific example, an application definition may identify one or more software modules in a conveyor interface 220a that allow a conveyor belt 220 to move an object in front of a robot 210. Furthermore, the application definition may identify one or more software modules in a robot interface 210a that allow the robot 210 to perform pick-and-place operations on an object, where the robot's arm 215 moves the object from one position on the conveyor belt 220 to another. Finally, the application definition may identify one or more software modules in a camera interface 230a that allow a camera to capture an image of the object after it has been moved and use this image to verify that the object is in the correct position. For example, a software module in the camera interface 230a may execute a machine learning recognition algorithm that processes the image and generates a predicted pose of the identified object in the image. The application definition can specify that these software modules should be executed iteratively in a software module loop, meaning that the conveyor belt can move the object further down the line after acknowledging its position and replace it with a new object at the same location in front of the robot 210.

[0029] Furthermore, each software module may have a data interface that identifies a way to communicate with other software modules. For example, a software module in the belt conveyor interface 220a may communicate that a new object has been moved in front of the robot 210. Ruri Altime message , in the software module of the robot interface 210a You may send the data. The data interface will be discussed in more detail later, referring to Figure 3.

[0030] Furthermore, the application definition, which identifies the software modules to be executed by the device in the execution environment 200, can specify that one or more of the software modules should be executed asynchronously. That is, a real-time robot control system executing a plan generated from the application definition, for example, the real-time robot control system 150 shown in Figure 1, can start a software module and perform other operations until it receives a callback indicating that the software module has completed. For example, the real-time robot control system can start the first and second software modules in parallel. In some existing systems, when a software module is started, the system must wait for the software module to complete before proceeding with any other operations. The ability to execute software modules asynchronously is important from a CPU perspective because device operations take a long time to execute, and therefore, by executing many operations in parallel, a lot of time can be saved in a given iteration of the software module loop. For example, a joint movement performed by the robot 210 in the execution environment 200 may take 3 seconds, which is a long time in real-time situations.

[0031] Figure 3 shows an example of a device interface definition 310 and examples of software modules 320 and 350.

[0032] The device interface definition 310 declares software modules that a particular device or a particular group of devices can execute. For example, a particular device can execute high-level software modules 320 and 330, and low-level software module 340. Each high-level software module depends on one or more low-level software modules and / or one or more other high-level software modules. Each low-level software module can perform the basic functions of each device. High-level software module 320 consists of high-level software module 320a and low-level software modules 320b and 320c. That is, the operations performed by high-level software module 320 include operations performed by high-level software module 320a and low-level software modules 320b and 320c, either sequentially or in parallel.

[0033] For example, device interface definition 310 may be for a robotic arm, and high-level software module 320 may perform pick-and-place operations, where the arm picks up an object in the execution environment, moves the object to a different location in the execution environment, and places the object down. High-level software module 320a may perform arm movement, low-level software module 320b may perform suction operation of a suction gripping unit attached to the arm, and low-level software module 320c may perform suction removal operation of a suction gripping unit attached to the arm. Thus, high-level software module 320 can perform pick-and-place by calling high-level software module 320a to move the arm to a location above the object, low-level software module 320b to pick up the object using the suction gripping unit, high-level software module 320a to move the object to a different location again, and low-level software module 320c to release the object using the suction gripping unit.

[0034] Similarly, the high-level software module 320a that moves the arm can be composed of M low-level software modules 321-m. For example, each low-level software module 321a-m can perform arm movement with a specific degree of freedom. In a specific example, the robot arm may have 6 degrees of freedom, and consequently, the high-level software module that moves the arm may consist of 6 low-level software modules, each controlling a specific degree of freedom.

[0035] The high-level software module 320 includes a module interface 322. The module interface specifies one or more communication channels on which the high-level software module 320 can receive data as input or send data as output. The communication channel can be internal, i.e., between two software modules run by the same robot (e.g., between high-level software modules 320 and 330). The communication channel can also be external, i.e., between two software modules run by different robots.

[0036] As a specific example, module interface 322 may specify a communication channel 360 between high-level software module 320 and another high-level software module 350 executed by a different robotic component. Similarly, high-level software module 350 has a module interface 352 that specifies the communication channel 360.

[0037] In some cases, communication channel 360 can be a hard real-time communication channel. That is, communication channel 360 is configured to transmit a specific type of data packet continuously (for example, every 10ms or 100ms) from high-level software module 320 to high-level software module 350. In some cases, high-level software module 350 may enter an error state if high-level software module 320 is unable to transmit data packets.

[0038] In other cases, the communication channel 360 can be asynchronous. That is, the communication channel 360 is configured to transmit a specific type of data packet from the high-level software module 320 to the high-level software module 350 whenever the data packet is available to the high-level software module 320. In other words, the high-level software module 320 generates data packets at irregular intervals, and therefore the high-level software module 350 receives data packets at variable intervals. The high-level software module 350 can then perform a specific action each time the high-level software module 320 transmits a data packet.

[0039] The device interface definition 310 may also include a list of communication channels through which the device can receive input data or transmit output data. That is, the device interface definition 310 can declare external communication channels for each software module in the device interface definition 310, and these external communication channels are communication channels that a software module shares with another software module on a different device.

[0040] As a specific example, device interface definition 310 may be for a camera in the execution environment, and high-level software module 320 may process images of objects in the execution environment captured by the camera and execute a machine learning cognitive algorithm to generate an estimated pose of the objects. Here, the object's pose is data characterizing the object's position and / or orientation in the execution environment. High-level software module 350 may perform pick-and-place operations on objects using a robotic arm in the execution environment. Therefore, software module 350 needs to know the object's pose before picking it up. Accordingly, communication channel 360 can asynchronously transmit the estimated pose from high-level software module 320 to high-level software module 350 when the estimated pose is generated by high-level software module 320. High-level software module 320 can interpolate the received estimated pose and, for example, transmit the interpolated pose in real time over another communication channel to another high-level or low-level software module.

[0041] In some cases, the communication channel specified by the module interface of a software module may be optional; that is, the software module can run without a communication channel. In other cases, a specified communication channel may be required.

[0042] The high-level software module 320 can be generated by the user. That is, the system can provide a number of low-level software modules that can be executed by one or more devices, and a tool for configuring the low-level software modules to generate the high-level software modules. For example, the system can provide a base class that each module interface of the high-level software module can inherit, so that the user can easily design any interface that specifies particular input and output channels. Thus, the user can generate high-level software modules specific to a particular use case if such high-level software modules are not provided by the hardware manufacturer for a given device or group of devices without additional configuration.

[0043] In some implementations, the device interface definition 310 can be for a distributed system of devices. That is, the device interface definition 310 declares a software module executable by a large number of devices across a distributed system, such as a large number of computers or robots connected by a communication channel. In these cases, the communication channel allows the software module to perform asynchronous and real-time operations across the large number of devices.

[0044] Figure 4 is a flowchart of an example of process 400, which generates a program to be executed by one or more devices in the execution environment. Process 400 can be performed by one or more computer programs installed on one or more computers and programmed according to this specification. For example, process 400 can be performed by the planner 120 shown in Figure 1. For convenience, process 400 will be described as being performed by a system of one or more computers.

[0045] The system receives one or more custom high-level software modules for each device in the execution environment (step 410). Some or all of the custom high-level software modules may include low-level software modules for the real-time robot.

[0046] The system receives module interface definitions for each custom high-level software module (step 415). A software module's module interface definition declares a module interface for the software module. In particular, a custom high-level software module's module interface definition can specify one or more asynchronous or real-time communication channels through which the custom high-level software module can receive input data from or send output data to different custom high-level software modules.

[0047] The system receives device interface definitions for each device in the execution environment (step 420). A device interface definition for a particular device can declare the software modules that the particular device can execute. A device interface definition for a particular device may also include one or more module interfaces for each software module that the particular device can execute. That is, if a software module has a module interface that specifies a particular communication channel for different software modules executed by different devices, the device interface definition for a particular device may also specify that particular communication channel.

[0048] The system receives a definition of a communication channel between the first interface of the first device in the execution environment and the second interface of the second device in the execution environment (step 430). In some cases, the execution of one or more software modules of the second device requires information received via the communication channel and therefore waits until it receives information from the first device. That is, the communication channel can be asynchronous. In a particular example, the first device can be a real-time robot, and the information transmitted over the channel can be the joint positions of the real-time robot's joints. In another particular example, the first device can be a camera, and the information transmitted over the channel can be the pose of an object in the execution environment.

[0049] The receiving high-level software module, device interface definition, module interface definition, and asynchronous channel definition can be collectively referred to as the "application definition."

[0050] The system processes the received application definition and generates an output program to be executed by the device (step 440). In some implementations, one or more of the output programs can be executed asynchronously.

[0051] The robot functions described in this specification can be implemented by a hardware-independent software stack that is at least partially hardware-independent, or simply by a software stack. In other words, the software stack can accept commands generated by the planning process described above as input, without requiring commands specifically related to a particular model of robot or a particular robot component. For example, the software stack can be implemented at least partially by the real-time robot control system shown in Figure 1.

[0052] A software stack can include numerous levels that increase hardware specificity in one direction and software abstraction in the other. Robot components, including devices that perform low-level actions and sensors that report low-level states, are at the lowest level of the software stack. For example, a robot can include a variety of low-level components, including motors, encoders, cameras, drivers, gripping parts, application-specific sensors, linear or rotational position sensors, and other peripheral devices. As an example, a motor may receive a command indicating the amount of torque to be applied. In response to receiving a command, the motor may, for example, use an encoder to report the current position of the robot's joint to a higher level of the software stack.

[0053] Each subsequent level in a software stack can implement interfaces that support a number of different basic implementation forms. Generally, each interface between levels supplies state messages from lower levels to higher levels and commands from higher levels to lower levels.

[0054] Ko Command and status messages teeth , Typically, Generated periodically during each control cycle (for example, one status message and one command per control cycle) It will be done Lower levels of the software stack generally have stricter real-time requirements than higher levels of the software stack. At the lowest level of the software stack, for example, a control cycle may have actual real-time requirements. In this specification, real-time means that a command received at one level of the software stack must be executed, and optionally, that status messages be fed back to higher levels of the software stack within a specific control cycle time. If this real-time requirement is not met, the robot may be configured to enter a failure state, for example, by halting all operations.

[0055] At a higher level, the software stack can include software abstractions of specific components called motor feedback controllers. A motor feedback controller can be a software abstraction of any suitable low-level component, not simply a literal motor. Thus, a motor feedback controller receives state via an interface to lower-level hardware components and sends commands back to the lower-level hardware components via the interface based on higher-level commands received from higher levels in the stack. A motor feedback controller can have any suitable control rules to determine how higher-level commands should be interpreted and translated into lower-level commands. For example, a motor feedback controller can use anything from simple logical rules to more advanced machine learning techniques to translate higher-level commands into lower-level commands. Similarly, a motor feedback controller can use any suitable failure rules to determine when a failure condition has been reached. For example, if a motor feedback controller receives a higher-level command and does not receive any low-level state within a particular part of the control cycle, the motor feedback controller can cause the robot to enter a failure condition where all movement stops.

[0056] At a higher level, the software stack can include an actuator feedback controller. The actuator feedback controller can include control logic that controls multiple robot components via each motor feedback controller. For example, several robot components (e.g., an articulated arm) can be controlled by multiple motors. Therefore, the actuator feedback controller can provide a software abstraction of the articulated arm by using the control logic of the articulated arm to send commands to the motor feedback controllers of the multiple motors.

[0057] At a higher level, the software stack can include joint feedback controllers. These controllers can represent joints that map to logical degrees of freedom in a robot. For example, while a robot's wrist can be controlled by a complex network of actuators, a joint feedback controller can abstract away that complexity, making its degrees of freedom manifest as a single joint. Thus, each joint feedback controller can control an arbitrarily complex network of actuator feedback controllers. As an example, a six-degree-of-freedom robot can be controlled by six different joint feedback controllers, each controlling a separate network of actual feedback controllers.

[0058] Each level of the software stack can also enforce level-specific constraints. For example, if a particular torque value received by an actuator feedback controller is outside the acceptable range, the actuator feedback controller can either correct that torque value to within the range or enter a fault condition.

[0059] To drive inputs to the joint feedback controller, the software stack can use a command vector containing command parameters for each component at a low level (e.g., positive, torque, and velocity for each motor in the system). To manifest states from the joint feedback controller, the software stack can use a state vector containing state information for each component at a low level (e.g., position, velocity, and torque for each motor in the system). In some implementations, the command vector further includes some constraint information regarding constraints that should be enforced by the controller at a low level.

[0060] At a higher level, the software stack can include a joint acquisition controller. The joint acquisition controller can handle the issuance of commands and state vectors, which manifest as a set of component abstractions. Each component can include a kinematic model (e.g., performing inverse kinematic calculations), constraint information, and joint state vectors and joint command vectors. For example, a single joint acquisition controller can be used to apply different sets of policies to different subsystems at a lower level. The joint acquisition controller can effectively separate the relationship between how the motors are physically represented and how control policies are associated with these components. Thus, for example, if a robotic arm has a movable base, the joint acquisition controller can be used to implement a set of constraint policies regarding how the arm moves and a different set of constraint policies regarding how the movable base can move.

[0061] At a higher level, the software stack can include a joint selection controller. This controller can dynamically select from commands issued from different sources. In other words, it can receive multiple commands during a control cycle and select one of those commands to be executed during that cycle. The ability to dynamically select from multiple commands during a real-time control cycle significantly increases the control flexibility of conventional robot control systems.

[0062] At a higher level, the software stack can include joint position controllers. Joint position controllers can receive target parameters and dynamically calculate the commands needed to achieve those target parameters. For example, a joint position controller can receive a position target and calculate setpoints to achieve that target.

[0063] At the next higher level, the software stack may include a Cartesian position controller and a Cartesian selection controller. The Cartesian position controller can receive a target in Cartesian space as input and compute an output in joint position space using an inverse kinematics solution. The Cartesian selection controller can then enforce a constraint policy on the result computed by the Cartesian position controller before passing the computed result in joint position space to the joint position controller at the next lower level of the stack. For example, three separate target states in Cartesian coordinates x, y, and z can be given to the Cartesian position controller. In some cases, the target state may be a position, while in other cases, the target state may be a desired velocity.

[0064] Therefore, these functions provided by the software stack offer broad flexibility to control orientation, which should be easily represented as target states in a way that naturally aligns with the high-level planning techniques described above. In other words, when the planning process uses a process definition graph to generate the specific actions to be taken, the actions do not need to be specified with low-level commands for individual robot components. Rather, the actions can be represented as high-level targets accepted by the software stack, which are transformed through various levels until they finally become low-level commands. Furthermore, the actions generated through the planning process can be specified in Cartesian space in a way that human operators can understand, resulting in easier, faster, and more intuitive debugging and analysis of schedules. Moreover, the actions generated through the planning process do not need to be tightly coupled to any particular robot model or low-level command format. Instead, the robot model can have the same degrees of freedom. support And the appropriate level of control is implemented in the software stack. outfit As long as this is the case, the same actions generated during the planning process can actually be performed by different robot models.

[0065] The embodiments and functional operations of the subject matter described in this specification may be implemented in digital electronic circuits, explicitly embodied computer software or firmware, computer hardware, or in combination of one or more thereof, including structures disclosed in this specification and their equivalents. The embodiments of the subject matter described in this specification may be implemented as one or more computer programs, i.e., as one or more modules of computer program instructions encoded in an explicitly persistent storage medium for execution by a data processing device or for controlling the operation of a data processing device. The computer storage medium may be a machine-readable storage device, a machine-readable storage board, a random or sequential access memory device, or in combination of one or more thereof. Alternatively or further, the program instructions may be encoded in artificially generated propagating signals (e.g., machine-generated electrical, optical, or electromagnetic signals) generated to encode information for transmission to a suitable receiving device for execution by a data processing device.

[0066] The term "data processing device" means data processing hardware and includes all types of devices, machines, and equipment that process data, including, for example, a programmable processor, a computer, or a number of processors or computers. A device may also be a dedicated logic circuit (e.g., an FPGA (Field-Programmable Gate Array) or an ASIC (Application-Specific Integrated Circuit)), or may further include such dedicated logic circuits. Optionally, in addition to hardware, a device may include code that generates an execution environment for computer programs, such as processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of these.

[0067] Computer programs, also called programs, software, software applications, applications, modules, software modules, scripts, or code, or described as programs, software, software applications, applications, modules, software modules, scripts, or code, can be written in any form of programming language, including compiler- or interpreter-type languages, or declarative or procedural languages. Computer programs can be deployed in any form, including standalone programs or modules, components, subroutines, or other units suitable for use in a computing environment. A program may correspond to a file in a file system, but may not require a file. A program can be stored in part of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program, or in a number of coordinate files (e.g., a file storing one or more modules, subprograms, or parts of code). Computer programs can be deployed and executed on one computer, or on a number of computers located in one place, or distributed across many locations and interconnected by a data communication network.

[0068] In the case of a system of one or more computers configured to perform a specific operation or action, it means that the system is installed on software, firmware, hardware, or a combination thereof that causes the system to perform the operation or action during operation. In the case of one or more computer programs configured to perform a specific operation or action, if executed by a data processing device, it means that the program contains one or more instructions that cause the device to perform the operation or action.

[0069] As used in this specification, “engine” or “software engine” means a software-implemented input / output system that provides different inputs and outputs. An engine can be a coded block of functionality (e.g., a library, platform, software development kit ("SDK"), or object). Each engine can be implemented on any suitable type of computing device including one or more processors and computer-readable media (e.g., a server, mobile phone, tablet computer, notebook computer, music player, e-book reader, laptop or desktop computer, personal digital assistive device, smartphone, or other fixed or portable device). Furthermore, two or more engines may be implemented on the same computing device or on different computing devices.

[0070] The processing and logic flows described in this specification can be executed by one or more programmable computers that run one or more computer programs that operate on input data and perform their functions by generating outputs. Furthermore, the processing and logic flows can be executed by dedicated logic circuits (e.g., FPGAs or ASICs), or by a combination of dedicated logic circuits and one or more programmable computers.

[0071] A computer suitable for running computer programs can be based on a general-purpose or dedicated microprocessor, or both, or any other type of central processing unit. Generally, the central processing unit receives instructions and data from read-only memory, random-access memory, or both. The essential elements of a computer are the central processing unit that executes or runs instructions, and one or more memory devices that store instructions and data. The central processing unit and memory can be supplemented by or incorporated into dedicated logic circuits. Generally, a computer further includes one or more mass storage devices (e.g., magnetic, magneto-optical disks, or optical disks) that store data, or is operablely coupled to receive data from or transfer data to or receive data from mass storage devices. However, a computer does not necessarily need to have such devices. Furthermore, computers can be integrated into other devices (for example, to name just a few, mobile phones, personal digital assistants (PDAs), mobile voice or video players, game consoles, Global Positioning System (GPS) receivers, or portable storage devices (for example, Universal Serial Bus (USB) flash drives)).

[0072] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices (e.g., EPROM, EEPROM, and flash memory devices), magnetic disks (e.g., internal hard disks or removable disks), magneto-optical disks, and CD-ROM and DVD-ROM disks.

[0073] Embodiments of the subject matter described in this specification can be carried out in a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) that displays information to the user in order to provide interaction with the user, and a keyboard and pointing device (e.g., a mouse, trackball, or a tactile display or other surface that allows the user to provide input to the computer). Interaction with the user can be provided in a similar manner using other types of devices. For example, the feedback given to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback). Input from the user, including acoustic, voice, or tactile input, can be received in any form. Furthermore, the computer can interact with the user by sending documents to a device used by the user and receiving documents from that device, for example, by sending a web page to a web browser on the user's device in response to a request received from a web browser. Furthermore, the computer can interact with the user by sending text messages or other forms of messages to a personal device (e.g., a smartphone) running a messaging application and receiving response messages from the user in return.

[0074] Embodiments of the subject matter described in this specification can be implemented in a computing system that includes backend components (e.g., as a data server), middleware components (e.g., an application server), or frontend components (e.g., a client computer having a graphical user interface, a web browser, or an application that allows a user to interact with an implementation of the subject matter described in this specification), or in any combination of one or more such backend, middleware, or frontend components. Components of the system can be interconnected by digital data communication (e.g., a communication network) in any form or medium. Examples of communication networks include local area networks (LANs) and wide area networks (WANs) (e.g., the Internet).

[0075] A computing system can include clients and servers. Clients and servers are generally geographically distant from each other and typically interact via a communication network. The client-server relationship arises from computer programs running on each computer that have a client-server relationship with each other. In some embodiments, the server sends data (e.g., an HTML page) to a user device for the purpose of displaying data to a user interacting with a device acting as a client and receiving user input from this user. Data generated on the user device (e.g., the results of user interaction) can be received from the device by the server.

[0076] In addition to the embodiments described above, the following embodiments are also innovative.

[0077] Embodiment 1 is, Receiving a multi-device application definition for a real-time robot control system configured to perform actions including controlling a robot execution environment having numerous robotic components, wherein the application definition is: One or more custom high-level software modules for each of a number of robotic components, each custom high-level software module is configured to perform one or more functions of the robotic component, and each custom high-level software module depends on one or more custom high-level software modules that depend on one or more low-level software modules of each robotic component. Each module interface definition for each custom high-level software module, wherein each module interface definition specifies one or more asynchronous or real-time communication channels on which the custom high-level software module can receive input data from or send output data to different custom high-level software modules. Each device interface definition for each of the numerous robot components, wherein each device interface definition declares a software module that the robot component can execute. Including receiving, The process involves processing an application definition to generate output programs to be executed by a number of robot components, each output program containing multiple commands that execute one or more custom high-level software modules, wherein the first output program for a first robot component of the number of robot components causes the first robot component to transmit information about a first communication channel to a second output program for a second robot component of the number of robot components. This method includes [something].

[0078] Embodiment 2 is the method according to Embodiment 1, wherein each low-level software module of the robot component depends on a custom software module that is customized for a particular robot component.

[0079] Embodiment 3 is the method according to Embodiment 1 or 2, wherein the custom software module is provided by the manufacturer of the robot components.

[0080] Embodiment 4 is the method according to any one of Embodiments 1 to 3, wherein the application definition specifies that one or more high-level software modules of the second robot component should wait for information received from the first robot component.

[0081] Embodiment 5 is the method according to any one of Embodiments 1 to 4, wherein the first robot component is a real-time robot, and the information transmitted via the first communication channel is the joint position of the joints of the real-time robot.

[0082] Embodiment 6 is the method according to any one of Embodiments 1 to 5, wherein the first robot component is a camera, and the information transmitted via the first communication channel is the orientation of an object in the execution environment.

[0083] Embodiment 7 is a method according to any one of Embodiments 1 to 6, wherein one or more of the output programs are executable asynchronously.

[0084] Embodiment 8 is a method according to any one of Embodiments 1 to 7, wherein the device interface definition for each of the numerous robotic components includes one or more module interface definitions for each high-level software module that the device can execute.

[0085] Embodiment 9 is the method according to any one of Embodiments 1 to 8, wherein the application definition further includes defining a second communication channel between a first module interface of a first high-level software module of a third robot of a number of robotic components and a second module interface of a second high-level software module of the third robot.

[0086] Embodiment 10 is a system comprising one or more computers and one or more storage devices that, when executed by one or more computers, store instructions that can be operated to cause one or more computers to perform the method described in any one of Embodiments 1 to 9.

[0087] Embodiment 11 is a computer storage medium on which a computer program is encoded that, when executed by a data processing device, includes instructions that can be operated to cause the data processing device to perform the method described in any one of Embodiments 1 to 9.

[0088] Although this specification includes details of many specific implementations, these details should not be interpreted as limiting the scope of any invention or the content of the claims, but rather as descriptions of features that may be specific to a particular embodiment of a particular invention. Specific features described in this specification in the context of individual embodiments may be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may be implemented separately or in any appropriate partial combination in multiple embodiments. Furthermore, although features may be described above as operating in a particular combination and initially described in the claims as such, in some cases, one or more features may be removed from a combination described in the claims, and the combination described in the claims may be directed toward a partial combination or a variation of a partial combination.

[0089] Similarly, although the operations are shown in a specific order in the drawings, this order should not be understood as requiring such operations to be performed in a specific illustrated or sequential order, or to perform all exemplary operations, in order to achieve the desired result. In certain circumstances, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated into a single software product or packaged into multiple software products.

[0090] Specific embodiments of the subject matter are described. Other embodiments are within the scope of the claims below. For example, the operations enumerated in the claims can be performed in different orders and furthermore, the desired results can be achieved. As an example, the processes shown in the accompanying drawings do not necessarily require the specific illustrated order or sequential order to achieve the desired results. In certain cases, multitasking and parallel processing may be advantageous.

Claims

1. A system comprising one or more computers and, when executed by the one or more computers, one or more storage devices that store instructions operable to cause the one or more computers to perform an operation, wherein the operation is Receiving a multi-device application definition for a real-time robot control system to perform a task, wherein the real-time robot control system is configured to perform an operation that includes controlling a robot execution environment having a large number of robot components, and the multi-device application definition is For each of the numerous robot components, one or more custom high-level software modules, each custom high-level software module is configured to perform one or more custom functionalities of the robot components, and is composed of at least one or more low-level software modules of the robot components, each low-level software module of the robot components is configured to perform one or more basic functions of the robot components, and comprises one or more custom high-level software modules. Each module interface definition for each custom high-level software module specifies one or more asynchronous or real-time communication channels configured for the custom high-level software module to receive input data from and / or send output data to different custom high-level software modules, Each device interface definition for each of the aforementioned number of robot components, wherein each device interface definition declares the custom high-level software module configured to be executed by the robot component. This includes receiving multi-device application definitions, Processing the multi-device application definition to generate output programs to be executed by each of the numerous robot components, wherein the output programs include a plurality of commands to execute one or more custom high-level software modules, the numerous robot components include a first robot component corresponding to a first device interface definition of each device interface definition, and a second robot component corresponding to a second device interface definition of each device interface definition, and the multi-device application definition includes the custom high-level software modules of the first robot component and the custom high-level software modules of the second robot component. This defines a first communication channel between the robot and the robot, and the first output program for the first robot component of the plurality of robot components causes the first robot component to transmit information including data characterizing the position and / or orientation of objects in the robot execution environment to the second output program for the second robot component of the plurality of robot components asynchronously with the execution of an output program different from the first output program via the first communication channel, and the first output program and the second output program generate an output program that executes the custom high-level software module for the first robot component and the custom high-level software module for the second robot component in parallel. A system that includes this.

2. The system according to claim 1, wherein at least one low-level software module of a particular robot component of the plurality of robot components is a software module provided by the manufacturer of the particular robot component, or includes such software module.

3. The system according to claim 1 or 2, wherein the multi-device application definition specifies that at least one high-level software module of the second robot component should wait for the information received from the first robot component before executing the custom functionality of the at least one high-level software module.

4. The system according to any one of claims 1 to 3, wherein the first robot component is a real-time robot, and the information transmitted through the first communication channel is the joint position of the joints of the real-time robot.

5. The system according to any one of claims 1 to 3, wherein the first robot component is a camera, and the information transmitted through the first communication channel is the posture of an object in the robot execution environment.

6. The system according to any one of claims 1 to 5, wherein the device interface definition for each of the numerous robot components includes one or more module interface definitions for each high-level software module configured to be executed by the robot component.

7. The system according to any one of claims 1 to 6, wherein the multi-device application definition defines a second communication channel between a first high-level software module of a third robot component of the plurality of robot components and a second high-level software module of the third robot component.

8. A method comprising causing the system according to any one of claims 1 to 7 to perform each of the operations.

9. One or more persistent computer storage media on which computer program instructions are encoded, causing the multiple computers to function as a system according to any one of claims 1 to 7, when executed by multiple computers.

Images