A composability framework for robotic control systems

Abstract

Methods, systems, and apparatus include computer programs encoded on computer storage media for supporting a composability framework for coordinating low-level actions of multiple subsystems. One method includes: receiving a multi-device application definition for a real-time robotic control system configured to perform operations including controlling a robotic execution environment having a plurality of robotic components, wherein the application definition includes: one or more custom high-level software modules for each of the plurality of robotic components; a corresponding module interface definition for each custom high-level software module, wherein the module interface definition of the custom high-level software module specifies one or more asynchronous or real-time communication channels; and a corresponding device interface definition for each of the plurality of robotic components; and processing the application definition to generate output programs to be executed by the plurality of robotic components, respectively.

Description

Background Art

[0001] This specification relates to a framework for software control systems.

[0002] A software control system is a software system that controls the actions of a physical device, such as an actuator or robot. Some software control systems are real-time software control systems that control real-time systems. Real-time software control systems must execute their control routines within strict timing requirements to achieve normal operation. Timing requirements typically dictate that specific actions must be executed or outputs generated within a specific time window to prevent the system from entering a fault state. In a fault state, the system may cease execution or take other actions that disrupt normal operation.

[0003] Such real-time software control systems are often used to control physical machines with high precision and timing requirements. As an example, the execution environment of industrial robots may be controlled by a real-time software control system that requires each robot to repeatedly receive commands at a specific frequency (e.g., 1, 10, or 100 kHz). If one of the robots does not receive a command during one of the periodic time windows, the robot may enter a fault state by either ceasing its operation or automatically executing a recovery process to return to a maintenance position.

[0004] Because of these timing requirements, software control systems for physical machines are often implemented using closed software modules that are specifically configured for highly specialized tasks. For example, a robot that picks components for placement on a printed circuit board might be controlled by a closed software system that manages each low-level pick and place action. Typically, the coordination of these software control modules follows an observe, plan, and act model, which involves making sensor observations, generating a plan, and then acting according to the plan. Summary of the Invention

[0005] This specification describes a composability framework that supports low-level actions that coordinate multiple subsystems, which may include one or more real-time robots. Generally speaking, a framework is a library of software modules that provides some basic functionality for building applications in a specific execution environment. In this specification, the execution environment includes multiple subsystems, each of which has corresponding computing hardware. The framework provides mechanisms for asynchronous communication channels, discovering low-level capabilities, and coordinating between multiple independent devices. The technology described below provides a framework for building applications that coordinate multiple subsystems in an execution environment, each of which can have its own observation, planning, and action functions.

[0006] Particular embodiments of the subject matter described in this specification can be implemented so as to realize one or more of the following advantages.

[0007] Some existing robotics application frameworks specify interfaces between devices and software modules and do not allow users to customize interfaces for specific use cases. Some systems described in this application allow users to compose common low-level software modules to generate high-level software modules tailored to their needs; users can also customize the data interfaces of the software modules. Some such systems also allow high-level composability of software modules, where customized high-level software modules can themselves be combined into higher-level software modules. Such higher-level software modules can establish communication channels between their corresponding component software modules and can coordinate the execution of their corresponding component software modules.

[0008] Some of the systems described in this specification can also provide a discovery mechanism for devices and software modules, whereby devices and software modules declare their corresponding data interfaces upon user request. Thus, these systems provide user-friendly software module management by allowing users to easily invoke software modules and retrieve their data interfaces. Consequently, these systems can be applied to a wide range of applications and devices.

[0009] Some systems described herein allow a user to establish a communication channel between two or more devices. The communication channel can transfer data between the devices, thereby allowing the devices to asynchronously execute one or more software modules.

[0010] The details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 is a diagram illustrating an example system.

[0012] Figure 2 An example execution environment with multiple devices is shown.

[0013] Figure 3 Example device interface definitions and example software modules are shown.

[0014] Figure 4 is a flowchart of an example process for generating a program for execution by one or more devices in an execution environment.

[0015] Like reference numbers and names in the various drawings represent like elements. DETAILED DESCRIPTION

[0016] Figure 1 is a diagram illustrating an example system 100. System 100 is an example of a system that may implement the techniques described herein.

[0017] The system 100 includes several functional components including a planner 120 and a real-time robot control system 150. Each of these components can be implemented as a computer program installed on one or more computers in one or more locations, these computers being connected to each other via any suitable communication network (e.g., an intranet or the Internet or a combination of networks). The system 100 also includes an execution environment 170 having N robots 170a-n.

[0018] The real-time robot control system 150 is configured to control robots 170a-n within an execution environment 170. The real-time robot control system 150 must meet one or more hard time constraints in its operations within the execution environment 170. For example, one of the robots within the execution environment 170 may be required to perform a specific operation every certain period of time (e.g., 10 milliseconds); if the robot fails to perform the operation within the given time window, the robot enters a fault state.

[0019] The real-time robot control system 150 is composed of one or more software module loops, each of which contains multiple software modules that are repeatedly executed in a predetermined order to control the robots 170a-n. Figure 1 A single software module loop is shown comprising M software modules 150a-c. Each software module may perform one or more functions of a corresponding one of the robots 170a-n.

[0020] Each software module may have a corresponding data interface that specifies real-time data, wherein the software module receives real-time data as input and provides real-time data as output. During execution, the software modules exchange P real-time messages 152-154 required for the execution of the respective software modules. For example, software module 150a may provide real-time message 152 to software module 150b, and software module 150b may provide real-time message 154 to software module 150c.

[0021] The overall goal of the planner 120 is to generate a schedule 135 that allows the real-time robot control system 150 to control a robot within the execution environment 170 to perform a specific task.

[0022] The planner 120 receives an application definition 110. The application definition 110 may be user-generated. The application definition 110 may identify a specific task to be completed by a robot within an execution environment 170. To complete the task, the application definition 110 may specify a software module loop, including identifying software modules 150a-c within the software module loop. The application definition 110 may also specify real-time messages 152-154 to be exchanged between the software modules according to their data interfaces.

[0023] Some of the software modules 150a-c may be high-level software modules composed of one or more lower-level software modules and data interfaces, where the data interfaces are generated by the user using the lower-level software modules. In other words, a customized high-level software module may depend on one or more lower-level software modules.

[0024] The low-level software modules can perform the basic functions of the corresponding robot in robots 170a-n. In some embodiments, each low-level software module depends on a custom software module customized for the corresponding robot. That is, the custom software module can perform specific low-level functions designed for a specific robot (e.g., tuned to the specific specifications of the specific robot). In some cases, the low-level software modules can depend on custom software modules provided by the manufacturer of the robot; that is, the custom software modules can be provided to the user of the robot out of the box.

[0025] Each software module can be associated with one or more robots under the execution environment 170; the software modules supported by a given device can be identified using the device interface definition. Figure 2 and Figure 3 Let’s discuss this process in more detail.

[0026] The planner 120 uses the application definition 110 to generate a schedule 135 for causing the real-time robotic control system 150 to execute software module loops to complete a specific task.

[0027] The planner 120 provides the schedule 135 to the real-time robot control system 150, which executes the software module loop according to the schedule 135. That is, the real-time robot control system 150 executes the software modules 150a-c, causing the real-time robot control system 150 to issue commands 155 to the execution environment 170 to drive the movements of the robots 170a-n. While the software modules 150a-c control the execution environment 170, the software modules 150a-c exchange real-time messages 152-154.

[0028] Figure 2 An example execution environment 200 is shown. The execution environment includes several robotic components including a robot 210 with an arm 215, a conveyor belt 220, and a camera 230.

[0029] Each device in the execution environment 200 has a device interface definition that states which software modules the corresponding device can execute. For example, the robot 210 has a robot interface 210a that specifies a list of software modules that the robot can execute, including a high-level software module 1A and a high-level software module 1B, as well as a low-level software module 1C. Each high-level software module can be composed of one or more low-level software modules and / or one or more other high-level software modules. Figure 3 Let's describe the composable software modules in more detail.

[0030] Similarly, the conveyor 220 has a conveyor interface 220a that specifies a list of software modules that the conveyor can execute, and the camera 230 has a camera interface 230a that specifies a list of software modules that the camera can execute.

[0031] A user generating an application definition for enabling a device within execution environment 200 to complete a specific task can use device interface definitions 210a-230a to identify software modules executable by the device within environment 200 and assemble the available software modules to complete the task. As a specific example, an application definition may identify one or more software modules within conveyor interface 220a that allow conveyor 220 to move objects in front of robot 210. The application definition may also identify one or more software modules within robot interface 210a that allow robot 210 to perform pick and place operations on objects, wherein the arm 215 of robot 210 moves an object from one location on conveyor 220 to another location on conveyor 220. Finally, the application definition may identify one or more software modules within camera interface 230a that allow a camera to capture an image of an object after it has been moved by robot 210 and use the image to confirm that the object is correctly positioned. For example, a software module within camera interface 230a may execute a machine learning perception algorithm that processes an image and generates a predicted pose for an object identified in the image. The application definition may specify that these software modules execute iteratively in a software module loop; that is, the conveyor may move an object further down the line after its positioning is approved, replacing the object with a new object in the same position in front of the robot 210.

[0032] Each software module may also have a data interface that identifies how to communicate with other software modules. For example, a software module in the conveyor belt interface 220a may send a real-time message to a software module in the robot interface 210a informing that a new object has been moved in front of the robot 210. Figure 3 Let's discuss data interfaces in more detail.

[0033] An application definition that identifies software modules executed by devices under the execution environment 200 may also specify one or more software modules that should be executed asynchronously. That is, a real-time robotic control system (e.g., Figure 1 The real-time robotic control system 150 depicted can start a software module and perform other operations until a callback is received indicating that the software module has completed. For example, the real-time robotic control system can start a first software module and a second software module in parallel. In some existing systems, when a software module is started, the system must wait for the software module to complete before continuing with other operations. The ability to execute software modules asynchronously can be important because device actuations take a significant amount of time to execute from a CPU perspective, so in a given iteration of a software module loop, a significant amount of time can be saved by executing multiple operations in parallel. For example, a joint movement performed by a robot 210 in the execution environment 200 may take 3 seconds, which is a significant amount of time in a real-time situation.

[0034] Figure 3 An example device interface definition 310 is shown, along with an example software module 320 and an example software module 350 .

[0035] Device interface definition 310 declares the software modules that a specific device or group of devices can execute. For example, a specific device may execute high-level software module 320, high-level software module 330, and low-level software module 340. Each high-level software module depends on one or more low-level software modules and / or one or more other high-level software modules. Each low-level software module can perform the basic functions of the corresponding device. High-level software module 320 consists of high-level software module 320a, low-level software modules 320b, and low-level software modules 320c. In other words, the operations performed by high-level software module 320 include operations performed sequentially or in parallel by high-level software module 320a, low-level software modules 320b, and low-level software modules 320c.

[0036] For example, the device interface definition 310 can be used for a robotic arm, and the high-level software module 320 can perform a pick and place action, wherein the arm picks up an object in an execution environment, moves the object to a different location in the execution environment, and places the object. The high-level software module 320a can perform the movement of the arm, the low-level software module 320b can perform the suction action of a suction gripper attached to the arm, and the low-level software module 320c can perform the desuction action of the suction gripper attached to the arm. Thus, the high-level software module 320 can perform a pick and place by calling the high-level software module 320a to move the arm to the object, calling the low-level software module 320b to pick up the object using the suction gripper, calling the high-level software module 320a again to move the object to a different location, and calling the low-level software module 320c to release the object using the suction gripper.

[0037] Similarly, a high-level software module 320a for moving an arm can be composed of M low-level software modules 321-m. For example, each of the low-level software modules 321a-m can perform arm movement in a specific degree of freedom. As a specific example, a robotic arm can have six degrees of freedom, so the high-level software module for moving the arm can be composed of six low-level software modules, each of which controls a specific degree of freedom.

[0038] High-level software module 320 includes module interface 322. The module interface specifies one or more communication channels through which high-level software module 320 can receive data as input or send data as output. A communication channel can be internal, i.e., between two software modules executed by the same robot, for example, high-level software module 320 and high-level software module 330. A communication channel can also be external, i.e., between two software modules executed by different robots.

[0039] As a specific example, module interface 322 may specify a communication channel 360 between high-level software module 320 and another high-level software module 350 executed by a different robotic component. Similarly, high-level software module 350 has a module interface 352 that specifies communication channel 360.

[0040] In some cases, communication channel 360 may be a hard real-time communication channel. That is, communication channel 360 is configured to continuously (e.g., every 10 ms or every 100 ms) send a specific type of data packet from high-level software module 320 to high-level software module 350. In some cases, if high-level software module 320 is unable to send the data packet, high-level software module 350 may enter an error state.

[0041] In some other cases, communication channel 360 can be asynchronous. That is, communication channel 360 is configured to send a specific type of data packet from high-level software module 320 to high-level software module 350 whenever high-level software module 320 makes a data packet available. In other words, high-level software module 320 generates data packets at an irregular frequency, so high-level software module 350 receives data packets at varying intervals. High-level software module 350 can then perform specific operations each time high-level software module 320 sends a data packet.

[0042] The device interface definition 310 may also include a list of communication channels through which the device may receive input data or transmit output data. In other words, the device interface definition 310 may declare an external communication channel for each corresponding software module in the device interface definition 310, where an external communication channel is a communication channel that a software module shares with another software module on a different device.

[0043] As a specific example, device interface definition 310 may be for a camera within the execution environment, and high-level software module 320 may execute a machine learning perception algorithm that processes images of an object within the execution environment captured by the camera and generates an estimated pose of the object. Here, the pose of an object is data characterizing the position and / or orientation of the object within the execution environment. High-level software module 350 may perform object picking and placement via a robotic arm within the execution environment. Therefore, software module 350 may need to know the pose of the object before picking it up. Therefore, communication channel 360 may asynchronously transmit the estimated pose from high-level software module 320 to high-level software module 350 because the estimated pose is generated by high-level software module 320. High-level software module 320 may interpolate the received estimated pose and, for example, transmit the interpolated pose in real time to another high-level software module or a lower-level software module over another communication channel.

[0044] In some cases, the communication channel specified by the module interface of the software module may be optional; that is, the software module may execute without the communication channel. In some other cases, the specified communication channel may be required.

[0045] High-level software modules 320 can be user-generated. That is, the system can provide multiple low-level software modules that can be executed by one or more devices, as well as tools for combining low-level software modules to generate high-level software modules. For example, the system can provide a base class from which each module interface of a high-level software module can inherit. This allows users to easily design arbitrary interfaces that specify specific input and output channels. Therefore, when a hardware manufacturer does not provide such a high-level software module out-of-the-box for a given device or group of devices, users can generate a high-level software module specific to their particular use case.

[0046] In some implementations, device interface definition 310 can be used in a distributed system of devices. That is, device interface definition 310 declares a software module that can be executed by multiple devices on the distributed system (e.g., multiple computers or robots connected via a communication channel). In these cases, the communication channel can allow the software module to perform asynchronous and real-time operations on multiple devices.

[0047] Figure 4 is a flow chart of an example process 400 for generating a program to be executed by one or more devices in an execution environment. The process 400 may be implemented by one or more computer programs installed on one or more computers and programmed according to the present specification. For example, the process 400 may be implemented by Figure 1 The depicted planner 120 performs For convenience, process 400 will be described as being performed by a system having one or more computers.

[0048] The system receives one or more custom high-level software modules for each device in the execution environment (step 410). Some or all of the custom high-level software modules may include low-level software modules for the real-time robot.

[0049] The system receives a module interface definition for each custom high-level software module (step 415). The module interface definition for a software module declares the module interface for the software module. Specifically, the module interface definition for a custom high-level software module may specify one or more asynchronous or real-time communication channels through which the custom high-level software module may receive input data from different custom high-level software modules or transmit output data to different custom high-level software modules.

[0050] The system receives a device interface definition for each device in the execution environment (step 420). The device interface definition for a particular device may declare the software modules that the particular device can execute. The device interface definition for a particular device may also include one or more module interfaces for the corresponding software modules that can be executed by the particular device. In other words, if a software module has a module interface that specifies a specific communication channel with different software modules executed by different devices, the device interface definition for the particular device may also specify the specific communication channel.

[0051] The system receives a definition of a communication channel between a first interface of a first device in an execution environment and a second interface of a second device in the execution environment (step 430). In some cases, the execution of one or more software modules of the second device requires information received via the communication channel and therefore waits until the information is received from the first device. That is, the communication channel can be asynchronous. As a specific example, the first device can be a real-time robot, and the information transmitted on the channel can be the joint positioning of the joints of the real-time robot. As another specific example, the first device can be a camera, and the information transmitted on the channel can be the pose of an object in the execution environment.

[0052] The received high-level software modules, device interface definitions, module interface definitions, and asynchronous channel definitions may be collectively referred to as "application definitions."

[0053] The system processes the received application definition to generate an output program to be executed by the device (step 440).In some implementations, one or more output programs can be executed asynchronously.

[0054] The robot functionality described in this specification may be implemented by a hardware-agnostic software stack (or, for the sake of brevity, described only as a software stack that is at least partially hardware-agnostic). In other words, the software stack may accept as input the commands generated by the planning process described above without requiring that these commands be specifically associated with a particular robot model or particular robot components. For example, the software stack may be at least partially composed of Figure 1 A real-time robot control system 150 is implemented.

[0055] The software stack can include multiple layers that increase hardware specificity in one direction and software abstraction in another direction. At the lowest level of the software stack are robot components, which include devices that perform low-level actions and sensors that report low-level states. For example, a robot can include various low-level components, including motors, encoders, cameras, drives, grippers, application-specific sensors, linear or rotational positioning sensors, and other peripherals. As an example, a motor can receive a command indicating the amount of torque that should be applied. In response to receiving the command, the motor can report the current positioning of the robot's joints to higher layers of the software stack (e.g., using encoders).

[0056] Each next highest layer in the software stack can implement an interface that supports multiple different underlying layer implementations. Generally speaking, each interface between layers provides status messages from lower layers to upper layers, and provides commands from upper layers to lower layers.

[0057] Typically, commands and status messages are generated cyclically during each control cycle, for example, there is one status message and one command per control cycle. Lower layers of the software stack typically have stricter real-time requirements than higher layers of the software stack. For example, at the lowest layers of the software stack, the control cycle may have actual real-time requirements. In this specification, real-time means that during a specific control cycle time, a command received at one layer of the software stack must be executed and, optionally, a status message provided to an upper layer of the software stack. If this real-time requirement is not met, the robot can be configured to enter a fault state (for example, by freezing all operations).

[0058] At the next highest level, the software stack may include a software abstraction for a particular component, which will be referred to as a motor feedback controller. The motor feedback controller may be a software abstraction for any appropriate lower-level component, and not just a literal motor. Thus, the motor feedback controller receives status through an interface into the lower-level hardware component and, based on upper-level commands received from higher levels in the stack, sends commands back through an interface to the lower-level hardware component. The motor feedback controller may have any appropriate control rules that determine how upper-level commands should be interpreted and converted into lower-level commands. For example, the motor feedback controller may use anything from simple logic rules to more advanced machine learning techniques to convert upper-level commands into lower-level commands. Similarly, the motor feedback controller may use any appropriate fault rules to determine when a fault state has been reached. For example, if the motor feedback controller receives upper-level commands but does not receive lower-level status within a particular portion of a control cycle, the motor feedback controller may cause the robot to enter a fault state where all operations cease.

[0059] At the next highest level, the software stack may include an actuator feedback controller. This actuator feedback controller may include control logic for controlling multiple robotic components via corresponding motor feedback controllers. For example, some robotic components (e.g., an articulated arm) may actually be controlled by multiple motors. Therefore, the actuator feedback controller may provide a software abstraction for an articulated arm by using its control logic to send commands to the motor feedback controllers of the multiple motors.

[0060] At the next highest level, the software stack may include joint feedback controllers. Joint feedback controllers may represent joints that map to logical degrees of freedom in the robot. So, for example, while the wrist of a robot may be controlled by a complex network of actuators, a joint feedback controller may abstract this complexity and expose that degree of freedom as a single joint. Thus, each joint feedback controller may control an arbitrarily complex network of actuator feedback controllers. As an example, a six-degree-of-freedom robot may be controlled by six different joint feedback controllers, each controlling an independent network of actual feedback controllers.

[0061] Each layer of the software stack can also enforce layer-specific constraints. For example, if a specific torque value received by an actuator feedback controller is outside of an acceptable range, the actuator feedback controller can modify the specific torque value to be within the range or enter a fault state.

[0062] In order to drive the input to the joint feedback controller, the software stack can use a command vector, and the command vector includes the command parameters for each assembly in the lower layer, for example, for the positioning, torque and speed of each motor in the system. In order to expose the state from the joint feedback controller, the software stack can use a state vector, and the state vector includes the state information for each assembly in the lower layer, for example, for the positioning, speed and torque of each motor in the system. In some embodiments, the command vector also includes some restriction information about the constraint that will be forced by the controller in the lower layer.

[0063] At the next higher level, the software stack may include a joint collection controller. The joint collection controller may handle the issuance of command vectors and state vectors that are exposed as a collection of component abstractions. Each component may include a kinematic model (e.g., for performing inverse kinematics calculations), constraint information, and a joint state vector and a joint command vector. For example, a single joint collection controller may be used to apply different sets of policies to different subsystems in a lower level. The joint collection controller may effectively decouple the relationship between how the motors are physically represented and how the control policies are associated with these components. Thus, for example, if the robot's arm has a movable base, the joint collection controller may be used to enforce a set of constraint policies regarding how the arm moves, and enforce a different set of constraint policies regarding how the movable base moves.

[0064] At the next highest level, the software stack may include a joint selection controller. This controller may be responsible for dynamically selecting commands from various sources. In other words, the joint selection controller may receive multiple commands during a control cycle and select one of these commands to execute during the control cycle. The ability to dynamically select from multiple commands during a real-time control cycle provides significantly greater control flexibility than traditional robotic control systems.

[0065] At the next highest level, the software stack may include a joint positioning controller. The joint positioning controller may receive target parameters and dynamically calculate the commands required to achieve the target parameters. For example, the joint positioning controller may receive a positioning target and calculate a set point to achieve the target.

[0066] At the next highest level, the software stack may include a Cartesian positioning controller and a Cartesian selection controller. The Cartesian positioning controller may receive a target in Cartesian space as input and use an inverse kinematics solver to calculate an output in joint positioning space. The Cartesian selection controller may then enforce a constraint strategy on the results calculated by the Cartesian positioning controller before passing the results in joint positioning space to the joint positioning controller in the next lowest level of the stack. For example, a Cartesian positioning controller may be assigned three independent target states in Cartesian coordinates x, y, and z. To some extent, the target state may be positioning, while to other extents, the target state may be a desired velocity.

[0067] Thus, these capabilities provided by the software stack provide extensive flexibility for control instructions, allowing them to be easily expressed as target states, thus naturally bridging with the higher-level planning techniques described above. In other words, while the planning process uses a process definition graph to generate specific actions to be taken, these actions do not need to be specified in low-level commands for individual robot components. Instead, they can be expressed as high-level goals accepted by the software stack, which are then transformed through various layers until they become low-level commands. Furthermore, the actions generated by the planning process can be specified in Cartesian space to be understandable to human operators, making debugging and analyzing schedules easier, faster, and more intuitive. Furthermore, the actions generated by the planning process do not need to be tightly coupled to any specific robot model or low-level command format. Instead, the same actions generated during the planning process can actually be executed by different robot models, as long as those models support the same degrees of freedom and the appropriate control layers are implemented in the software stack.

[0068] The embodiments of the subject matter and functional operations described in this specification may be implemented in digital electronic circuits, in tangibly implemented computer software or firmware, in computer hardware including the structures disclosed in this specification and the equivalents of these structures, or in a combination of one or more of them. The embodiments of the subject matter described in this specification may be implemented as one or more computer programs, i.e., one or more computer program instruction modules encoded on a tangible, non-transitory storage medium, which are used to be executed by a data processing device or to control the operation of the data processing device. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them. Alternatively or additionally, the program instructions may be encoded on an artificially generated propagation signal, for example, a machine-generated electrical signal, optical signal, or electromagnetic signal, which is generated to encode information for transmission to a suitable receiving device for execution by the data processing device.

[0069] The term "data processing apparatus" refers to data processing hardware and includes various devices, equipment, and machines for processing data, including, for example, a programmable processor, a computer, or multiple processors or computers. An apparatus may also be or include special-purpose logic circuitry, such as an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). In addition to hardware, an apparatus may optionally include code that creates an execution environment for a computer program, such as code constituting processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of these.

[0070] A computer program (which may also be referred to or described as a program, software, software application, app, module, software module, script, or code) may be written in any form of programming language, including compiled or interpreted languages ​​or declarative or procedural languages, and the computer program may be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program may (but need not) correspond to a file in a file system. A program may be stored in a portion of a file that holds other programs or data (e.g., in one or more scripts in a markup language document), in a single file dedicated to the program in question, or in multiple coordinated files (e.g., in a file that stores one or more modules, subroutines, or portions of code). A computer program may be deployed to execute on one computer or on multiple computers, located at one location or distributed across multiple locations and interconnected by a data communications network.

[0071] For a system of one or more computers to be configured to perform a specific operation or action, this means that the system has installed thereon software, firmware, hardware, or a combination thereof that, in operation, causes the system to perform the operation or action. For one or more computer programs to be configured to perform a specific operation or action, this means that the one or more programs include instructions that, when executed by a data processing apparatus, cause the apparatus to perform the operation or action.

[0072] As used in this specification, "engine" or "software engine" refers to an input / output system implemented by software that provides outputs different from inputs. An engine can be a coded functional block, such as a library, platform, software development kit ("SDK") or object. Each engine can be implemented on any appropriate type of computing device, for example, a server, mobile phone, tablet computer, notebook computer, music player, e-book reader, laptop or desktop computer, PDA, smart phone, or other fixed or portable device including one or more processors and computer-readable media. In addition, two or more engines can be implemented on the same computing device, or can be implemented on different computing devices.

[0073] The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. These processes and logic flows can also be performed by special-purpose logic circuits (e.g., FPGAs or ASICs) or by a combination of special-purpose logic circuits and one or more programmed computers.

[0074] A computer suitable for executing a computer program can be based on a general-purpose microprocessor or a special-purpose microprocessor or both, or based on any other type of central processing unit. Generally speaking, the central processing unit will receive instructions and data from a read-only memory or a random access memory or both. The basic elements of a computer are a central processing unit for executing or running instructions and one or more memory devices for storing instructions and data. The central processing unit and the memory can be supplemented by special-purpose logic circuits or incorporated into special-purpose logic circuits. Generally speaking, a computer will also include one or more mass storage devices (e.g., magnetic disks, magneto-optical disks, or optical disks) for storing data, or by being operably coupled to a mass storage device to receive data from the mass storage device or transfer data to the mass storage device, or both. However, a computer does not need to have such a device. In addition, a computer can be embedded in another device, for example, a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device (e.g., a universal serial bus (USB) flash drive), to name a few.

[0075] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example: semiconductor memory devices, such as EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CD ROM disks and DVD-ROM disks.

[0076] To provide interaction with a user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and pointing device (e.g., a mouse, trackball, or presence sensitive display or other surface) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and the input from the user can be received in any form, including sound, speech, or tactile input. In addition, the computer can interact with the user by sending documents to and receiving documents from a device used by the user; for example, by sending a web page to a web browser on the user's device in response to a request received from the web browser. In addition, the computer can also interact with the user by sending text messages or other forms of messages to a personal device (e.g., a smartphone) running a messaging application and receiving a response message from the user in return.

[0077] Embodiments of the subject matter described in this specification can be implemented in a computing system that includes a back-end component (e.g., as a data server) or includes a middleware component (e.g., an application server) or includes a front-end component (e.g., a client computer having a graphical user interface, a web browser, or an application through which a user can interact with an implementation of the subject matter described in this specification) or any combination of one or more such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include local area networks (LANs) and wide area networks (WANs), such as the Internet.

[0078] A computing system may include a client and a server. The client and server are generally remote from each other and typically interact via a communication network. The relationship between the client and the server arises from the fact that (a plurality of) computer programs are run on respective computers and have a client-server relationship with each other. In some embodiments, the server transmits data (e.g., an HTML page) to a user device, e.g., for displaying data to a user interacting with the device as a client and receiving user input from the user. Data generated at the user device (e.g., the results of the user interaction) can be received from the device at the server.

[0079] In addition to the above embodiments, the following embodiments are also innovative:

[0080] Embodiment 1 is a method comprising:

[0081] A multi-device application definition for a real-time robotic control system is received, the real-time robotic control system being configured to perform operations including controlling a robotic execution environment having a plurality of robotic components, wherein the application definition includes:

[0082] one or more custom high-level software modules for each robotic component of the plurality of robotic components, wherein each custom high-level software module is configured to perform one or more functions of the robotic component, and wherein each custom high-level software module is dependent on one or more low-level software modules of the corresponding robotic component;

[0083] a corresponding module interface definition for each custom high-level software module, wherein the module interface definition of the custom high-level software module specifies one or more asynchronous or real-time communication channels through which the custom high-level software module can receive input data from a different custom high-level software module or transmit output data to a different custom high-level software module; and

[0084] a respective device interface definition for each of the plurality of robotic components, wherein the device interface definition for the robotic component declares software modules that the robotic component can execute; and

[0085] The application definition is processed to generate output programs to be executed by the plurality of robotic assemblies, respectively, wherein the output programs include a plurality of commands to execute one or more custom high-level software modules, and wherein a first output program for a first robotic assembly of the plurality of robotic assemblies causes the first robotic assembly to transmit information on the first communication channel to a second output program for a second robotic assembly of the plurality of robotic assemblies.

[0086] Embodiment 2 is a method according to embodiment 1, wherein each low-level software module of the robotic component depends on a custom software module customized for the specific robotic component.

[0087] Embodiment 3 is a method according to any one of embodiments 1 or 2, wherein the custom software module is provided by the manufacturer of the robotic component.

[0088] Embodiment 4 is a method according to any one of embodiments 1-3, wherein the application definition specifies that one or more high-level software modules of the second robotic component should wait for information received from the first robotic component.

[0089] Embodiment 5 is a method according to any one of embodiments 1-4, wherein the first robotic component is a real-time robot, and wherein the information transmitted on the first communication channel is joint positioning of joints of the real-time robot.

[0090] Embodiment 6 is a method according to any one of embodiments 1-5, wherein the first robotic component is a camera, and wherein the information transmitted on the first communication channel is a pose of an object in the execution environment.

[0091] Embodiment 7 is a method according to any one of embodiments 1-6, wherein one or more output programs can be executed asynchronously.

[0092] Embodiment 8 is a method according to any one of embodiments 1-7, wherein the device interface definition for each of the plurality of robotic components includes one or more module interface definitions of corresponding high-level software modules that the device can execute.

[0093] Example 9 is a method according to any one of Examples 1-8, wherein the application definition also includes the definition of a second communication channel, and the second communication channel is between a first module interface of a first high-level software module of a third robot in the multiple robot components and a second module interface of a second high-level software module of the third robot.

[0094] Embodiment 10 is a system comprising: one or more computers and one or more storage devices storing operable instructions, which, when executed by the one or more computers, cause the one or more computers to perform the method described in any one of embodiments 1 to 9.

[0095] Embodiment 11 is a computer storage medium encoded with a computer program, the program including operable instructions that, when executed by a data processing device, cause the data processing device to perform the method described in any one of embodiments 1 to 9.

[0096] Although this specification contains many specific implementation details, these details should not be understood as limitations on the scope of any invention or the scope of what may be claimed, but rather as descriptions of features that may be specific to a particular embodiment of a particular invention. Specific features described in this specification in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented separately in multiple embodiments, or in any suitable subcombination. Furthermore, although features may be described above as functioning in a particular combination, or even initially claimed as such, in some cases one or more features from a combination may be removed from the claimed combination, and the claimed combination may involve subcombinations or variations of subcombinations.

[0097] Similarly, although the drawings depict operations in a particular order, this should not be understood as requiring that the operations be performed in the particular order shown, or in sequential order, or that all illustrated operations be performed to achieve the desired result. In certain circumstances, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above-described embodiments should not be understood as requiring such separation in all embodiments, but rather that the described program components and systems may generally be integrated into a single software product or packaged into multiple software products.

[0098] While specific embodiments of the subject matter have been described, other embodiments are within the scope of the appended claims. For example, the actions recited in the claims can be performed in a different order and still achieve the desired results. As an example, the processes depicted in the accompanying figures do not necessarily require the specific order shown or sequential order to achieve the desired results. In certain circumstances, multitasking and parallel processing may be advantageous.

Claims

1. A robotic control system of a composable framework, comprising one or more computers and one or more storage devices storing operable instructions, wherein the instructions, when executed by the one or more computers, cause the one or more computers to perform operations comprising: A multi-device application definition for a real-time robotic control system is received, the real-time robotic control system being configured to perform operations including controlling a robotic execution environment having a plurality of robotic components, wherein the application definition includes: one or more customized high-level software modules for each of the plurality of robotic components, wherein each customized high-level software module is configured to perform one or more functions of the robotic component, wherein each customized high-level software module includes one or more low-level software modules of the corresponding robotic component, and wherein each low-level software module of the robotic component is configured to perform one or more basic functions of the robotic component; a corresponding module interface definition for each custom high-level software module, wherein the module interface definition of the custom high-level software module specifies one or more asynchronous or real-time communication channels through which the custom high-level software module can receive input data from a different custom high-level software module or transmit output data to a different custom high-level software module; and a respective device interface definition for each of the plurality of robotic components, wherein the device interface definition for the robotic component declares software modules that the robotic component is capable of executing; and The application definition is processed to generate output programs to be executed by the plurality of robotic assemblies, respectively, wherein the output programs include a plurality of commands to execute one or more custom high-level software modules, and wherein a first output program for a first robotic assembly of the plurality of robotic assemblies causes the first robotic assembly to transmit information on the first communication channel to a second output program for a second robotic assembly of the plurality of robotic assemblies.

2. The system according to claim 1, wherein: Each low-level software module of a robot component depends on custom software modules that are customized for the specific robot component.

3. The system according to claim 2, wherein: Custom software modules are provided by the manufacturers of the robot components.

4. The system according to any one of claims 1 to 3, wherein: The application definition specifies that one or more high-level software modules of the second robotic component should wait for information received from the first robotic component.

5. The system according to claim 1, wherein: The first robotic assembly is a real-time robot, and wherein the information transmitted on the first communication channel is joint positioning of joints of the real-time robot.

6. The system according to claim 1, wherein: The first robotic component is a camera, and wherein the information transmitted on the first communication channel is a pose of an object in the execution environment.

7. The system according to claim 1, wherein: One or more output programs can be executed asynchronously.

8. The system according to claim 1, wherein: The device interface definition for each of the plurality of robotic components includes one or more module interface definitions of corresponding high-level software modules that the device is capable of executing.

9. The system according to claim 1, wherein: The application definition also includes a definition of a second communication channel between a first module interface of a first high-level software module of a third robot in the plurality of robot assemblies and a second module interface of a second high-level software module of the third robot.

10. A robot control method of a composable framework, comprising the operations performed by the system according to any one of claims 1 to 9.

11. One or more non-transitory computer storage media encoded with computer program instructions that, when executed by a plurality of computers, cause the plurality of computers to perform the operations performed by the system of any one of claims 1-9.

Images