Sharing world model objects among multiple autonomous systems

Abstract

This invention relates to a computer-implemented method comprising operating a first autonomous system to perform tasks based on executable code derived from objects in a world model of the first autonomous system. The world model objects of the first autonomous system represent the operating environment of the first autonomous system. The method includes determining a startup trigger when the first autonomous system is about to begin interacting with a second autonomous system. The second autonomous system is operated based on executable code derived from a world model including world model objects representing the operating environment of the second autonomous system. After the startup trigger, the method includes using a world model object shared between the first and second autonomous systems. Subsequently, the method includes continuing to operate the first autonomous system based on an extended world model of the first autonomous system that includes the shared world model object of the second autonomous system.

Description

Technical Field

[0001] This disclosure relates generally to the field of autonomous systems, and more specifically to a technique for communication between autonomous systems. Background Technology

[0002] Most automated systems are not autonomous systems. In conventional automation, a typical strategy for achieving machine-to-machine communication is for programmers to write custom, hand-crafted code for a specific task at hand. In this paradigm, one machine will send a custom message to another, which must then use hand-crafted code to interpret the message and take appropriate action. Typically, automation programs are programmed entirely with raw, custom code. Because the entire program is raw, communication between systems must also be programmed. Automation programs consist of low-level instructions, presented in a recipe-like manner, describing the steps the device must perform. There is usually no direct representation of the task the device is performing or the object the device is manipulating. Summary of the Invention

[0003] In short, various aspects of this disclosure relate to a technique for communicating between autonomous systems based on shared knowledge of the state of the operating environment of each autonomous system.

[0004] According to various aspects of this disclosure, a computer-implemented method includes operating a first autonomous system to perform tasks based on executable code derived from objects in a world model of the first autonomous system. The world model of the first autonomous system includes a plurality of world model objects representing the operating environment of the first autonomous system. The method further includes determining a startup trigger when the first autonomous system is about to begin interacting with a second autonomous system. The second autonomous system is operated based on executable code derived from objects in a world model of the second autonomous system, which includes a plurality of world model objects representing the operating environment of the second autonomous system. The method further includes initiating data communication between the first and second autonomous systems in response to the startup trigger, wherein the data communication includes sharing world model objects between the first and second autonomous systems. The method further includes continuing to operate the first autonomous system based on an extended world model of the first autonomous system, the extended world model including shared world model objects of the second autonomous system.

[0005] Other aspects of this disclosure embody the characteristics of the above-described methods in automated systems and computer program products.

[0006] Additional technical features and benefits can be achieved through the technology disclosed herein. Embodiments and aspects of this disclosure are described in detail herein and are considered part of the claimed subject matter. For a better understanding, refer to the detailed description and accompanying drawings. Attached Figure Description

[0007] The foregoing and other aspects of this disclosure are best understood when the following detailed description is read in conjunction with the accompanying drawings. For ease of identification of any element or action discussed, the most significant digit or more of the reference numerals in the drawings refer to the drawing number in which that element or action is first introduced.

[0008] Figure 1 An example of an autonomous system implemented using a high-level abstraction task based on a world model is shown.

[0009] Figure 2 An example of two autonomous systems interacting based on a shared world model object is shown.

[0010] Figure 3 This illustration illustrates an implementation where each autonomous system locally caches the world model objects of another autonomous system.

[0011] Figure 4 An implementation of accessing world model objects of multiple autonomous systems from a public database is illustrated.

[0012] Figure 5 It demonstrates how the movement of an object can be controlled with a small number of constraints.

[0013] Figures 6A to 6E A series of steps involving the initiation and termination of sharing a world model object among multiple autonomous systems are illustrated according to an exemplary implementation.

[0014] Figure 7 An example of a computing system according to aspects of this disclosure is shown. Detailed Implementation

[0015] Various techniques relating to the systems and methods will now be described with reference to the accompanying drawings, wherein the same reference numerals denote the same elements throughout. In this patent document, the drawings discussed below and the various embodiments used to describe the principles of this disclosure are merely illustrative and should not be construed in any way as limiting the scope of this disclosure. Those skilled in the art will understand that the principles of this disclosure can be implemented in any suitably arranged apparatus. It should be understood that functions described as being performed by certain system elements can be performed by multiple elements. Similarly, for example, elements can be configured to perform functions described as being performed by multiple elements. Numerous innovative teachings of this disclosure will be described with reference to exemplary, non-limiting embodiments.

[0016] The aspects of this disclosure described below relate to autonomous systems designed within a skill-based programming paradigm. In conventional automation, autonomous robots are typically programmed to perform a single, repetitive task, such as positioning a car panel in exactly the same location on every vehicle. In this case, engineers typically participate in programming the entire task from start to finish, usually utilizing low-level code to generate individual commands. In the autonomous systems described herein, skill functions are used instead of individual commands to program physical devices (such as robots) at a higher level of abstraction.

[0017] Autonomous systems perform tasks based on their understanding of their operating environment (or "world"). See also Figure 1 The autonomous system 100 according to various aspects of this disclosure includes controllable physical devices such as robot 102, which operate based on applications running on an associated computing system (see, for example, [link to relevant documentation]). Figure 7 The computing system can include, for example, an industrial PC or a general-purpose computer such as a desktop or laptop computer, which can be connected to the robot 102 via a wired link or wirelessly. The autonomous system 100 uses its world model to automatically track the status of equipment and products in its operating environment. The world model includes digital copies of physical artifacts, which are continuously updated by the application using knowledge from the actual environment of the autonomous system 100.

[0018] exist Figure 1 In the example shown, robot 102 is given the task of picking up working parts 104 and storing them in box 106. This could be an element of a larger task, such as providing a kit of working parts for another process or selecting products for an online order. Robot 102's task is specified as individual skills, such as picking up parts and placing them in the box. Robot 102, working parts 104, box 106, and the surrounding physical elements are all part of the program and are explicitly represented. Knowledge about where working parts 104 are located and how to pick them up is explicitly encoded by the working parts 104 at a certain location in the robot program's virtual world. The task of picking up working parts 104 is not encoded by specifically referencing any particular working part 104, because robot 102 does not know which working parts 104 will be present before it runs. Instead, the task can be implemented as a search of a physical volume. Any working part 104 appearing in the volume is a target to be picked up. Similarly, camera 108, which detects parts, communicates its results by populating robot 102's world model with virtual instances of the working parts 104 to be picked up. Camera 108 does not require a special, custom-coded communication interface, but it can communicate as a tool for manipulating the world model.

[0019] The world model of an autonomous system comprises multiple world model objects representing the state of the autonomous system's operating environment, from which executable code for operating the autonomous system is derived. World model objects are program objects and can include one or more objects corresponding to physical objects in the operating environment, as well as other objects. Physical objects include the controllable physical device itself and physical artifacts that it manipulates or interacts with. World model objects can also include one or more objects corresponding to one or more markers on simulated objects in the autonomous system's operating environment. World model objects can also include one or more objects corresponding to the skill functions of the autonomous system necessary to perform tasks specified by the application. A detailed description of exemplary implementations of the aforementioned world model objects can be found in the applicant's International Patent Application No. PCT / US2017 / 054651, entitled "ENGINEERINGAUTONOMOUS SYSTEMS WITH REUSABLE SKILLS," published as WO 2019 / 066979, which is incorporated herein by reference in its entirety. It should be noted that the actual format of the world objects being shared can differ for different applications.

[0020] By utilizing world models, autonomous systems can perform tasks based on high-level knowledge about themselves and their environment. Instead of programming autonomous systems based on raw electrical signals or fixed movements, applications can be implemented by describing activities based on objects manipulated and transformed by the system. Because applications are designed to possess high-level knowledge of such activities, multiple autonomous systems can interact without extensive custom programming.

[0021] When two or more autonomous systems need to interact, these systems ideally share knowledge about the state of their respective environments. The actions of these autonomous systems are based on the state of environmental objects, such as where work parts are located or which objects need to be assembled first. By understanding the state of nearby peer autonomous systems, autonomous applications can assist and undertake tasks by following their own commands but applying them to objects in their peer systems.

[0022] The embodiments of this disclosure generally relate to a technique for sharing information between peer autonomous systems by formalizing objects used by autonomous systems to represent their environments. In this case, the "represented" environment is a "world model," which represents all objects used by the autonomous system in its functionality. Instead of passing custom messages between autonomous systems, they synchronize on a shared set of world model objects. Different world model objects can originate from different autonomous systems, but once shared, each autonomous system retains information about these world model objects and can use the data for reasoning and task completion.

[0023] exist Figure 2The diagram illustrates an example of two autonomous systems 202 and 204 physically interacting by transferring a working component 214 from one system to the other. The first autonomous system 202 includes a mobile robot 206 with a wheeled cart 208 and a robotic arm 218 with a gripper 220. The second autonomous system 204 includes a stationary robot 210 with a robotic arm 222 and a gripper 224 fixed to a worktable 212. Both the first and second autonomous systems 202 and 204 can be controlled independently, each with its own computing system. Figure 2 Not shown in (for example, see Figure 7 Each autonomous system 202, 204 is operated based on executable code derived from objects in the world model of the corresponding autonomous system. The world model of each autonomous system 202, 204 includes multiple world model objects representing the state of the operating environment of the corresponding autonomous system.

[0024] exist Figure 2 In the example shown, working parts 214 are stacked on a cart surface 216. A mobile robot 206 delivers working parts 214 from the cart surface 216 to a stationary robot 210. Typically, such interactions between robots are difficult to execute, especially because the mobile robot 206 is mobile and its position relative to the stationary robot 210 is not fixed. However, the mobile robot 206 is capable of knowing its own position and using cameras or other positioning techniques to fix its position relative to the stationary robot 210. According to this technology, when autonomous systems 202 and 204 are about to begin interaction, a trigger is determined (e.g., based on the physical extent of the system), thus initiating data communication between autonomous systems 202 and 204. This data communication involves sharing world model objects between the first autonomous system 202 and the second autonomous system 204. Subsequently, each autonomous system 202 and 204 operates based on an extended world model that includes the shared world model objects of the other autonomous systems. When the first autonomous system 202 shares its world model object with the second autonomous system 204, the second autonomous system 204 has up-to-date information about the composition of all mobile robots and the location of their working parts 214. Even if the stationary robot 210 does not have its own sensors or vision technology, the shared world model of the mobile robot 206 will be sufficient to enable the stationary robot to pick up the working part 214 from the mobile robot's gripper. Similarly, by sharing the world model of the second autonomous system 204 with the first autonomous system 202, the first autonomous system 202 is able to know when the working part 214 is grasped by the stationary robot 210 and released by the mobile robot 206.

[0025] Typically, machine-to-machine communication is functional, where one machine invokes a remote function on another machine to perform actions such as annotations upon task completion. According to various aspects of this disclosure, communication is representative. The communicating entities represent physical objects and devices in the world surrounding the system performing automation. This allows for the sharing of objects without requiring automation programmers to write special functions or data formats to implement machine-to-machine communication. Instead, programmers formulate tasks based on work objects, without needing to concern themselves with the existence of work objects provided by the system for the machines performing the work.

[0026] In addition to world model objects that share representations of physical objects in the operating environment (including the controllable physical devices themselves and the physical workpieces that the devices manipulate or interact with), shared world model objects can include application-related contextual information. Instances of world model objects providing contextual information include graphical markers created on simulated objects in the world model based on instructions provided by the user via a graphical user interface. The position and orientation of the graphical markers can be determined relative to the simulated objects. Skill functions (such as picking up work parts) can include functional descriptions for interacting with physical objects using the robot based on the position and orientation of each graphical marker.

[0027] In the described implementation, the shared world model object is implemented via a network or middleware layer. The network or middleware layer can be used to synchronize information about the world model of each system to other systems. Following the shared world model object, each autonomous system has an extended world model that includes its own world model object and additionally includes a synchronized copy of the world model object from another autonomous system. Figure 3 A high-level schematic diagram of an implementation is shown, in which an autonomous system is able to cache world model objects from another autonomous system. Autonomous systems are also able to poll each other to discover shared world model objects. Figure 4 A high-level schematic diagram of an alternative implementation is shown, in which shared world model objects of multiple autonomous systems are collected and viewed from a public database accessible to each autonomous system. In this case, each autonomous system is able to have pointers to world model objects of another autonomous system and can query questions about those world model objects via functions, but does not have a local copy of the actual data transferred to itself.

[0028] In the specific implementation described herein, each autonomous system locally caches a synchronized copy of the shared world model object of other autonomous systems (e.g., ...) at discrete time steps. Figure 3(As described in the text). Several factors can be used to determine whether a shared object is managed by a synchronized local cache as described herein, or whether the object is shared outside a public database. For example, if objects change rapidly, the speed of network devices will be considered. Objects that change infrequently are more suitable for offline storage, or if the database publishes events to a system that follows a set of object changes. If changes are rapid or the system needs access to most world objects in the shared space, caching may be more preferable.

[0029] Application developers can manage how autonomous systems are divided into networked entities. Figure 1 In this example, the camera can be directly integrated into the robot's system and generate world model objects directly within a single system. It can also be divided into local camera systems, where it generates world model objects for itself and shares these objects with other systems, such as those controlling the robot. Figure 2 The system in this example includes a mobile cart equipped with a robot. In this case, the stationary and mobile workstations need to be controlled separately because they are physically separate. Using a network or middleware layer, the world model of each system can still be shared with other systems.

[0030] While no specific network or middleware layer is required by any aspect of this disclosure, it is desirable for the middleware layer to provide the following characteristics. First, the objects in the world model are generally not known a priori. Middleware cannot be configured with a fixed number of objects. In automated network solutions, it is common to fix the data members of all resources (such as symbol tables or HMI labels). Objects are added to and removed from the world model. In middleware implementations, it is preferable to be able to add and remove objects from objects shared by others. Second, it is preferable to be able to reference objects within data, such as those with pointers, to create a graphical structure. The world model is not a flat structure but rather a collection of objects with many different levels of abstraction, such as entities for a machine and entities describing the parts of that machine (such as a robot and its various mobile arms) and physical objects within each part (such as rigid bodies and joints). All these things form an interconnected information graph about the state of the device. Third, it is preferable to have a means of transporting large amounts of data, especially when initially sharing objects and using them for visualization. Various elements of an autonomous system (such as displays and visions) may require datasets of graphical elements, and graphical data often depends heavily on the level of detail of the graph.

[0031] In the described implementation, a middleware data distribution service (DDS) is used. In other implementations, other protocols can be used, such as, but not limited to, TCP / IP using sockets, OPC, REST, etc. In principle, the system can use any number of network technologies and can choose different technologies to use in different situations based on the available and effective technological potential.

[0032] In the described implementation, features for creating data tables are used. In DDS, data tables can be created using data values ​​in a given format, such as numbers, strings, and arrays. The protocol allows a system to "publish" its data to a request broker. The broker can be a public database or a distributed peer-to-peer network. Other systems "subscribe" to the desired data based on search criteria (similar to a database query). The published data can be changed frequently to efficiently stream given data elements, and subscribers receive change notifications and can access updated data. In a non-limiting implementation, the request broker can be implemented on each autonomous system, keeping it synchronized with other request brokers in the network. In one implementation, each autonomous system can publish updated data related to the autonomous system's world model objects through a middleware layer at discrete time steps. After a startup trigger, when the first autonomous system is about to begin interacting with the second autonomous system, each autonomous system in the first and second autonomous systems can begin polling the other autonomous system through the middleware layer to discover the world model objects published by the other autonomous system.

[0033] Shared world model objects typically include static data that depends on classes and data that changes at runtime. The process for achieving world model sharing can be implemented in multiple phases of data transfer. This process is independent of the factors that can cause two systems to begin or terminate sharing, which are discussed separately. Each phase involves transferring different types of data based on the frequency of data changes.

[0034] The first data transfer phase (Phase 1) involves a one-time transfer of all static data related to the shared world model objects. This data varies from object to object, but for a given object, it remains the same throughout the duration of the operation. Examples include values ​​such as object type, object mass, and entity shape, which do not change after the application starts. The second data transfer phase (Phase 2) involves one or more synchronization steps that include the transfer of all data related to the shared world model objects, which change as the respective autonomous system runs. Phase 2 involves values ​​that are members of objects in the first data class, but these values ​​often change during application runtime. Examples include the position and orientation of workpieces, which objects are attached to machines, and which are placed on conveyor belts. The third data transfer phase (Phase 3) involves one or more synchronized maintenance steps that include only the transfer of data related to device control. Phase 3 is dedicated to data for device control. Control data must change like Phase 2 data; however, much of Phase 2 data can be automatically updated in an external system because it is controlled systematically. For example, a robot can rotate at its base. This allows all objects in the robot and any objects carried by the robot to move and change position. However, only one object, the robot's base position controller, truly dictates how all these objects move. Therefore, if the controller remains updated, the other objects in the system will behave correctly. The fourth data transmission phase (Phase 4) involves one or more error correction steps, including transmitting data to correct errors arising from differences in the simulated outputs between the first and second autonomous systems. Phase 4 is necessary because not all simulated behavior occurs as planned. Phase 4 uses data from Phase 2 to determine if the shared systems have become out of sync. If they have diverged, similar to Phase 2, the system will resynchronize.

[0035] The first phase is used to transfer the basic static properties of world model objects. These properties do not change as the application manipulates them. The type of an object (e.g., whether it represents a robot or a box) and its components (such as physical objects, collision surfaces, and motion joints) are instances of static properties. Static object elements form a graph. A robot has parts such as its base, multiple arms, and flanges, and these parts will have sub-parts such as arms with rigid body properties, motion joints such as rotation and linear connections, controllers for position or other properties, collision surfaces, etc. Because DDS does not have a native reference pointing scheme, objects are assigned globally unique identifiers (GUIDs). References between objects are represented using their GUIDs. To publish static object data to DDS, the system creates a flat list of all objects that need to be transferred. These objects are categorized by type. For each type of object, this technique uses a DDS table format. Each row of the table is associated with an object. In addition to storing the object's member data, the table entries also need to store additional data for reconstruction in other systems. For example, the GUIDs of the owners of parts in an assembly are written into the table, while in ordinary objects, the assembly maintains a list of pointers to its parts. Redundant data can be removed to reduce size. Some static data is too large to be published in a DDS table. For example, the shape of a convex collision shape can contain any number of points. For these types of data, streaming network protocols such as TCP / IP can be used.

[0036] In most cases, the types (class definitions) of the shared world model objects are known and will be implemented by all autonomous systems that aim to share world objects derived from them. In this way, the table format of the DDS can be generically qualified by the system library without user intervention. If the library is extended to include new types, the new DDS format will also be included in the extension. Typically, if an autonomous system intends to import objects of types it does not understand, it will be unable to meaningfully interpret their semantics. If this is required in the future, it can be assumed that the corresponding dynamic library code implementation (compiled for the correct machine or interpreter) for the new unknown type can be delivered to the importing system, enabling it to run.

[0037] The second phase involves transmitting synchronization data for all instantaneous values ​​(i.e., values ​​that are potentially changeable during runtime). This includes the positions of components in space, their velocities and angular velocities (typically in all six degrees of freedom), and transformation values ​​such as logical states, on or off states, and numerical values. Reference data can also be transient and require synchronization. For example, when a device picks up a workpiece, it forms a connection between the body of the component and the body of the gripper (or any component performing the picking). This connection can be implemented as a fixed joint, with its "attachment" member set onto the body of the workpiece. The "base" member of the fixed joint is permanently set onto the body of the gripper. Because the "attachment" value changes depending on the object held by the gripper, it needs to be synchronized with the positions of the gripper's body and the workpiece.

[0038] At certain critical moments, values ​​change based on the application's actions. These moments require synchronization to keep each system aligned. For example, when an object is picked up, the gripper's connector changes to attach to the object at the time of pickup. If the connector becomes attached to the object, but the object doesn't move to properly align with the gripper, the object will be attached to the gripper but in the wrong position (like being magically grabbed in space). Changes when object hierarchies or links change always need to be synchronized at the time step in which the change occurs. If synchronization isn't happening here, then full synchronization may be needed to put the object in its proper state.

[0039] The third phase is the maintenance phase, which maintains synchronization between shared world model objects. This phase involves transmitting only values ​​related to device control, such as controller values, switch values, etc. This is a much smaller set of values ​​than in the second phase, and therefore can be updated with lower latency and a higher frequency. Controllers can include, for example, position controllers for controlling the rotational position of a robotic arm, position controllers for controlling the rotational position of a robot base, motion controllers for controlling the linear displacement of the robot, etc. Controllers can be implemented in simulation, for example, as virtual motors.

[0040] The difference between Phase 2 and Phase 3 data transmission is based on the understanding that device objects will operate in a predictable manner, thus eliminating the need for all values ​​to be constantly synchronized. For example, if the structural links between device objects remain unchanged, movement can be linear and predictable. Even when an object is in free fall, its state is predictable until it collides with something (i.e., when constraints change) because the physical properties are calculated to be the same on both the initiator and importer systems. Many critical values ​​exist, such as the destination of the motion / position controller and the values ​​of switches that need to be maintained during this phase. Other values ​​(such as the robot's position and orientation in space) will change as these critical values ​​follow their instructions. For example, an angular position controller can be used to rotate the base 502 of the robot 500, such as... Figure 5 As shown. For simplicity, it is assumed that other joints remain stationary, although this is not necessary. As the base 502 rotates, other objects (such as arms 504, 506, 508), gripper 510, and flange 512 all move in a predictable manner according to their relative positions. Therefore, in this example, most of the time, only the positions of the angular position controller of the base 502 and the other controllers in the arms are needed to position all objects connected to the robot, thus the robot's objects themselves (such as the position and velocity of internal rigid bodies) do not need to be individually synchronized.

[0041] The speed of the controller can be used to influence the quality of synchronization. Two systems typically do not operate at the same speed, especially when they are running on different controllers. The difference between the current value of the external controller (in the physical world) and the replica controller in the local system can be used to adjust the speed of the replica controller so that it matches the value of the external controller over time. This method is also used to match the state of analog equipment to that of physical equipment in a real-world application. Different algorithms can be used to adjust the speed. Proportional / integral / derivative loops or PID loops will be sufficient for the purposes described.

[0042] When the topology of objects changes, external systems are notified that they must perform full synchronization. In another implementation, specific objects can be flagged for full synchronization if it is possible to determine which objects are affected. Topology changes due to application behavior can be particularly important for resynchronization. When a work item is picked up or placed on another device, its fate is critical and should be synchronized. For other types of constraint topology changes (such as collisions), the physics engines in both the current and external systems may be accurate enough that they do not diverge. In this case, it is possible to allow physics properties to run to determine the resulting state. However, objects must still be tracked to determine if they have lost synchronization.

[0043] The fourth phase involves tracking changes between objects that have moved or otherwise changed in simulations instantiated across multiple autonomous systems and determining whether they remain synchronized. Typically, resynchronizing objects at each time step is not desirable, as it would be unreliable and unavailable due to excessively slow network latency. Furthermore, placing objects outside the influence of the physics engine is undesirable, as this allows objects to be effectively "remotely transported" or "popped" to new locations without needing to move between them. If one object pops into the space of another, the physics solver is able to apply a large compensating force to correct for overlap and send all flying objects.

[0044] The data to be synchronized in the fourth stage can be determined based on several considerations. In one implementation, the data to be synchronized can be determined by sampling random values. In another implementation, the data to be synchronized may involve certain key values ​​that are known to be out of sync across multiple systems running the simulation. For example, the movement of controllable devices (such as robots) may be constrained and does not require resynchronization other than that described in stage 3. However, on the other hand, the movement of working parts may be unconstrained and may require resynchronization at stage 4.

[0045] If the position exceeds the range expected by its velocity within the calculated time steps, only synchronization is needed to correct the object. As mentioned above, the controller of the joint is specially handled to control its velocity in an attempt to predict and catch up with the current position of the external controller. If the controller is in the correct range, the object constrained by the controller's position is predicted to be in the range. Freely moving objects (such as unheld working parts) are checked by their absolute position and orientation. If their velocity is zero, they are not expected to move. If they do move, they must be pushed or otherwise affected by another object. This constitutes a topological change in the constraints and facilitates resynchronization.

[0046] The key consideration for synchronization (applicable to phases 2, 3, and 4) is retrieving the state of all objects within a single time step. On the publishing system side, the state of the objects being transferred is read into a cache, which is maintained by a mutex in the runtime engine. This allows the cache to be written to the network layer in a separate thread without unnecessarily delaying the runtime engine. On the user side, the data at the network layer needs to be atomic. That is, the state of a subset of objects cannot be overwritten or otherwise altered until the reader has finished reading all the state of that particular system. If data can be changed before the reader reaches a subset of objects, those objects will be out of sync with the rest. On the reader side, data is cached for a complete time step, and the runtime engine is then paused only when all values ​​need to be changed simultaneously or when new objects are added. Memory allocation and data collection from the network are performed in separate threads to avoid latency.

[0047] The shared world model between peer-to-peer autonomous systems can be initiated in different ways and can also be terminated without disabling the autonomous systems. For example, the initiation trigger can be determined based on the physical extent of the autonomous systems or any other factor requiring interaction between them. Figures 6A to 6E An example of transferring working components between multiple autonomous systems is shown (similar to...) Figure 2In this example, three autonomous systems are involved: a first autonomous system 602, a second autonomous system 604, and a third autonomous system 606. The first autonomous system 602 includes a mobile robot 608, the second autonomous system 604 includes a stationary robot 610, and the third autonomous system 606 also includes a stationary robot 612. Each of the autonomous systems 602, 604, and 606 is capable of operating independently based on a corresponding application running on its own respective computing system (see, for example...). Figure 7 The first autonomous system 602, the second autonomous system 604, and the third autonomous system 606 are each associated with a physical extent indicated by regions R1, R2, and R3, respectively. Sharing of world model objects can be initiated when the first autonomous system 602 becomes close to one of the fixed autonomous systems 604 and 606 in its physical extent.

[0048] When these autonomous systems are not nearby, such as in Figure 6A In this context, each autonomous system (AS) has its own separate world model, which is not shared. Each AS in ASs 602, 604, and 606 can publish data about its world model objects without polling data from other ASs. From the perspective of the mobile robot 608 of the first AS 602, the fixed ASs 604 and 606 are represented only by their physical extent. The mobile robot 608 cannot see the state of the fixed ASs unless the world model is shared. It is possible to mark locations where sharing occurs and network information that allows ASs to connect within the world model of the first AS 602.

[0049] exist Figure 6B In this process, the mobile robot 608 has moved closer to the stationary robot 610. A trigger is determined when the physical extents associated with the first autonomous system 602 and the second autonomous system 604 overlap, as shown in the merged region C1. In response to the trigger, the first autonomous system 602 initiates the sharing of world model objects between itself and the second autonomous system 604. With the world model shared, objects in the world model of the second autonomous system 604 can be handed over to the mobile robot 608 of the first autonomous system 602, such as the working part 614. Because the world model is shared, the working part 614 can be manipulated by both the mobile robot 608 and the stationary robot 610.

[0050] exist Figure 6CIn this process, the mobile robot 608 has moved away from the stationary robot 610. When the physical boundaries associated with the first autonomous system 602 and the second autonomous system 604 no longer overlap, a termination trigger is determined, and the sharing of world model objects between the first autonomous system 602 and the second autonomous system 604 terminates based on the termination trigger. At this point, there is no longer a need to share world model objects, and objects from the external system can be cleared from the local world model of each of the autonomous systems 602 and 604.

[0051] When the sharing of world model objects terminates, one or more world model objects can be transferred between the first autonomous system 602 and the second autonomous system 604. In this example, the working part 614 used with the second autonomous system 604 is transferred to the first autonomous system 602. In one embodiment, the transfer of one or more world model objects can be determined if, when the sharing of world model objects terminates, those one or more world model objects are topologically connected to an autonomous system different from the autonomous system initially associated with the one or more world model objects. In this case, the rule can be that when the world is disconnected and cleared, objects topologically connected to the autonomous system (such as the working part 614 that contacts the surface of the mobile robot 608) are not deleted. In another embodiment, the transfer of one or more world model objects can be determined if, when the sharing of world model objects terminates, those one or more world model objects are contained in a defined volume associated with an autonomous system different from the autonomous system initially associated with those one or more world model objects. In this case, the volume of the system region can determine which objects are retained when the world is divided.

[0052] exist Figure 6D In the next step, the process is repeated, but this time using the first autonomous system 602 and the third autonomous system 606. When the world model is shared, the mobile robot 608 transfers the working part 614 to the stationary robot 612. Figure 6E In this process, when the mobile robot 608 leaves, the working part 614 remains attached to the third autonomous system 606 because it maintains contact with the table of that third autonomous system. In this way, shared objects can be transferred from one autonomous system to another in a manner similar to exchanging actual physical objects. The advantage is that application programmers do not need special programming to perform such exchanges. It is a natural consequence of the interaction between world models of different autonomous systems.

[0053] Figure 7An exemplary computing environment including a computer system 702 is shown, in which aspects of this disclosure can be implemented. The computing system 702 can be configured, for example, as a dedicated computer for an autonomous system. Computers and computing environments (such as computer system 702 and computing environment 700) are known to those skilled in the art and are therefore briefly described herein.

[0054] like Figure 7 As shown, computer system 702 may include communication mechanisms, such as system bus 704 or other communication mechanisms for transmitting information within computer system 702. Computer system 702 also includes one or more processors 706 coupled to system bus 704 for processing information. Processor 706 may include one or more central processing units (CPUs), graphics processing units (GPUs), or any other processor known in the art.

[0055] Computer system 702 also includes system memory 708, which is coupled to system bus 704 for storing information and instructions executed by processor 706. System memory 708 may include computer-readable storage media in the form of volatile and / or non-volatile memory, such as read-only memory (ROM) 710 and / or random access memory (RAM) 712. System memory RAM 712 may include other dynamic storage devices (e.g., dynamic RAM, static RAM, and synchronous DRAM). System memory ROM 710 may include other static storage devices (e.g., programmable ROM, erasable PROM, and electrically erasable PROM). Furthermore, system memory 708 may be used to store temporary variables or other intermediate information during instruction execution by processor 706. A basic input / output system 714 (BIOS), containing basic routines such as those facilitating the transfer of information between elements within computer system 702 during startup, may be stored in system memory ROM 710. System memory RAM 712 may contain data and / or program modules that are immediately accessible and / or currently operating to processor 706. The system memory 708 may additionally include, for example, an operating system 716, an application program 718, other program modules 720, and program data 722.

[0056] Computer system 702 also includes a disk controller 724 coupled to system bus 704 to control one or more storage devices, such as magnetic hard disks 726 and removable media drives 728 (e.g., floppy disk drives, optical disk drives, magnetic tape drives, and / or solid-state drives). Storage devices can be added to computer system 702 using appropriate device interfaces, such as Small Computer System Interface (SCSI), Integrated Device Electronics (IDE), Universal Serial Bus (USB), or FireWire.

[0057] Computer system 702 may also include a display controller 730 coupled to system bus 704 to control display 732, such as a cathode ray tube (CRT) or liquid crystal display (LCD), for displaying information to a computer user. Computer system 702 includes a user input interface 734 and one or more input devices (such as a keyboard 736 and a clicking device 738) for interacting with the computer user and providing information to one or more processors 706. Clicking device 738 may be, for example, a mouse, light pen, trackball, or pointer, for transmitting directional information and command selections to one or more processors 706 and for controlling cursor movement on display 732. Display 732 may provide a touchscreen interface that allows input to supplement or replace the communication of directional information and command selections by clicking device 738.

[0058] The computing system 702 also includes an input / output adapter 746 coupled to a system bus 704 to connect the computing system 702 to a controllable physical device, such as a robot. Figure 7 In the example shown, the input / output adapter 746 is connected to the robot controller 748, which includes one or more motors for controlling the linear and / or angular position of various parts of a physical device or robot (e.g., arm, base, etc.).

[0059] Computer system 702 is capable of performing some or all of the processing steps of embodiments of the present disclosure in response to one or more processors 706 executing one or more sequences of one or more instructions contained in memory (such as system memory 708). Such instructions can be read into system memory 708 from another computer-readable storage medium (such as magnetic hard disk 726 or removable media drive 728). Magnetic hard disk 726 can contain one or more data storages and data files used by embodiments of the present disclosure. The data storage contents and data files can be encrypted to enhance security. Processor 706 can also be configured in a multiprocessing arrangement to execute one or more sequences of instructions contained in system memory 708. In alternative embodiments, hardwired circuitry can be used in place of or in combination with software instructions. Therefore, the embodiments are not limited to any particular combination of hardware circuitry and software.

[0060] Computer system 702 may include at least one computer-readable storage medium or memory for holding instructions programmed according to embodiments of the present disclosure and for containing data structures, tables, records, or other data described herein. As used herein, the term "computer-readable storage medium" refers to any medium that participates in providing instructions to one or more processors 706 for execution. Computer-readable storage media may take many forms, including but not limited to non-transient, non-volatile media, volatile media, and transmission media. Non-limiting examples of non-volatile media include optical discs, solid-state drives, magnetic disks, and magneto-optical discs, such as magnetic hard disk 726 or removable media drive 728. Non-limiting examples of volatile media include dynamic memory, such as system memory 708. Non-limiting examples of transmission media include coaxial cables, copper wires, and optical fibers, including wires constituting system bus 704. Transmission media may also take the form of sound waves or light waves, such as those generated during radio wave and infrared data communication.

[0061] The computing environment 700 may also include a computer system 702 operating in a networked environment using logical connections to one or more remote computers (such as remote computing device 744). The remote computing device 744 may be a personal computer (laptop or desktop), mobile device, server, router, network PC, peer-to-peer device, or other public network node, and typically includes many or all of the elements described above with respect to computer system 702. When used in a networked environment, computer system 702 may include a modem 742 for establishing communication over a network 740 (such as the Internet). The modem 742 may be connected to system bus 704 via network interface 745 or via another suitable mechanism.

[0062] Network 740 can be any network or system commonly known in the art, including the Internet, intranet, local area network (LAN), wide area network (WAN), metropolitan area network (MAN), direct connection or a series of connections, cellular telephone network, or any other network or medium that facilitates communication between computer system 702 and other computers (e.g., remote computing device 744). Network 740 can be wired, wireless, or a combination thereof. Wired connections can be implemented using Ethernet, Universal Serial Bus (USB), RJ-6, or any other wired connection commonly known in the art. Wireless connections can be implemented using Wi-Fi, WiMAX and Bluetooth, infrared, cellular networks, satellite, or any other wireless connection method commonly known in the art. Furthermore, multiple networks can operate independently or communicate with each other to facilitate communication within network 740.

[0063] Embodiments of this disclosure can be implemented using any combination of hardware and software. Furthermore, embodiments of this disclosure can be included in an article of manufacture (e.g., one or more computer program products) having, for example, a non-volatile computer-readable storage medium. The computer-readable storage medium contains, for example, computer-readable program instructions for providing and facilitating mechanisms for embodiments of this disclosure. This article of manufacture can be included as part of a computer system or sold separately.

[0064] Computer-readable storage media can include tangible devices capable of retaining and storing instructions used by an instruction execution device. Computer-readable storage media can be, for example, but not limited to, electronic storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. The computer-readable program instructions described herein can be downloaded from the computer-readable storage media to a corresponding computing / processing device, or to an external computer or external storage device, via a network (e.g., the Internet, a local area network, a wide area network, and / or a wireless network).

[0065] While different aspects and embodiments have been disclosed herein, other aspects and embodiments will be apparent to those skilled in the art. The various aspects and embodiments disclosed herein are for illustrative purposes and are not intended to be limiting, wherein the true scope and subject matter are indicated by the appended claims.

Claims

1. A computer-implemented method, the method comprising: The first autonomous system is operated to perform tasks based on executable code derived from objects in the world model of the first autonomous system, wherein the world model of the first autonomous system includes a plurality of world model objects representing the state of the operating environment of the first autonomous system. When the first autonomous system is about to begin interacting with the second autonomous system, a startup trigger is determined. This trigger operates the second autonomous system based on executable code derived from objects in the world model of the second autonomous system. The world model includes multiple world model objects representing the state of the operating environment of the second autonomous system. In response to the startup trigger, data communication is initiated between the first autonomous system and the second autonomous system, wherein the data communication includes sharing the world model object between the first autonomous system and the second autonomous system, and The first autonomous system continues to operate based on the extended world model of the first autonomous system, which includes the shared world model object of the second autonomous system.

2. The method according to claim 1, wherein, The shared world model object of the first autonomous system and the second autonomous system in each case includes one or more world model objects corresponding to one or more physical objects in the operating environment of the respective autonomous system.

3. The method according to any one of claims 1 and 2, wherein, The shared world model object of the first autonomous system and the second autonomous system in each case also includes one or more world model objects with one or more tags corresponding to the simulated objects in the operating environment of the respective autonomous system.

4. The method according to any one of claims 1 and 2, wherein, The activation trigger is determined when the physical range associated with the first autonomous system and the physical range associated with the second autonomous system overlap.

5. The method according to any one of claims 1 and 2, further comprising: In response to the determination of termination trigger, the sharing of the world model object between the first autonomous system and the second autonomous system is terminated.

6. The method according to claim 5, wherein, The termination trigger is determined when the physical extent associated with the first autonomous system no longer overlaps with the physical extent associated with the second autonomous system.

7. The method of claim 5, further comprising transferring one or more world model objects between the first autonomous system and the second autonomous system when terminating the sharing of the world model objects.

8. The method according to claim 7, wherein, If, upon termination of sharing of the world model objects, one or more of the world model objects are topologically connected to an autonomous system different from the autonomous system originally associated with the one or more of the world model objects, then a transfer of the one or more world model objects is determined.

9. The method according to claim 7, wherein, If, upon termination of sharing of the world model objects, one or more of the world model objects are contained within a defined volume associated with an autonomous system different from the autonomous system initially associated with the one or more of the world model objects, then the transfer of the one or more world model objects is determined.

10. The method according to any one of claims 1 and 2, wherein, After sharing the world model object, the second autonomous system is operated based on an extended world model of the second autonomous system, the extended world model including the shared world model object of the first autonomous system.

11. The method according to any one of claims 1 and 2, wherein, The sharing of the world model objects between the first autonomous system and the second autonomous system is achieved through a middleware layer.

12. The method of claim 11, comprising: Each autonomous system publishes updated data related to the world model object of the autonomous system at discrete time steps through the middleware layer, and After the startup is triggered, each autonomous system polls other autonomous systems through the middleware layer to discover world model objects published by other autonomous systems.

13. The method of claim 11, further comprising each autonomous system locally caching synchronized copies of the shared world model object of other autonomous systems at discrete time steps.

14. The method according to claim 11, wherein, The shared world model objects of the first autonomous system and the second autonomous system are collected in a public database that is accessible to each of the first autonomous system and each of the second autonomous systems.

15. The method according to any one of claims 1 and 2, wherein the shared world model object comprises static data that depends on the class and data that changes at runtime.

16. The method according to any one of claims 1 and 2, wherein, The sharing of the world model objects includes data transfer implemented in multiple phases, which include: The first phase involves a one-time transfer of all static data related to the shared world model object. The second phase involves one or more synchronization steps, which include the transfer of all data related to the shared world model object, which changes during the operation of the corresponding autonomous system. The third phase involves one or more synchronous maintenance steps, which only include the transmission of data related to equipment control. The fourth stage involves one or more error correction steps, which include data transmission to correct errors caused by differences in the analog outputs between the first autonomous system and the second autonomous system.

17. A non-transitory computer-readable storage medium comprising instructions that, when executed by a computer of an autonomous system, cause the computer to: The autonomous system's controllable physical devices perform tasks based on executable code derived from objects in the autonomous system's world model, wherein... The world model of the autonomous system includes multiple world model objects representing the state of the operating environment of the autonomous system. When the autonomous system is about to begin interacting with a peer autonomous system, a trigger is determined, and the peer autonomous system is operated based on executable code derived from objects in the peer autonomous system's world model, which includes multiple world model objects representing the state of the peer autonomous system's operating environment. In response to the startup trigger, data communication is initiated between the autonomous system and the peer autonomous system, wherein the data communication includes sharing the world model object between the autonomous system and the peer autonomous system, and Based on the extended world model of the autonomous system, which includes the shared world model object of the peer autonomous system, the controllable physical devices continue to operate.

18. An autonomous system, comprising: Controllable physical devices; as well as A computer, comprising a processor and a memory storing instructions, which, when executed by the processor, configure the computer to: The controllable physical device is operated to perform tasks based on executable code derived from objects in the world model of the autonomous system, wherein the world model of the autonomous system includes multiple world model objects representing the state of the operating environment of the autonomous system; When the autonomous system is about to begin interacting with a peer autonomous system, a trigger is determined, and the peer autonomous system is operated based on executable code derived from objects in the peer autonomous system's world model, which includes multiple world model objects representing the state of the peer autonomous system's operating environment. In response to the startup trigger, data communication is initiated between the autonomous system and the peer autonomous system, wherein the data communication includes sharing the world model object between the autonomous system and the peer autonomous system, and The controllable physical devices continue to be operated based on the extended world model of the autonomous system, the extended world model including the shared world model objects of the peer autonomous systems.

Images