Programming automation equipment using goal-based programming with semantic annotation

Abstract

In a computer-implemented method for engineering an automation system for a production process, a simulation environment is generated comprising 3D graphical objects representing physical objects of the automation system. The physical objects include material items and devices to manipulate the material items. Markers including semantic annotations are attached to the graphical objects based on user instructions received via a graphical user interface. Markers include placement markers associated with material items, where at least some of the placement markers define hierarchical placement relationships between the associated material items. Operation of each device is specified as a goal or a set of goals parameterized with the markers and groupings of material items based on user instructions received via the graphical user interface. Code is generated that is executable to perform the specified operations by the respective devices.

Description

[0001] PROGRAMMING AUTOMATION EQUIPMENT USING GOAL-BASED PROGRAMMING WITH SEMANTIC ANNOTATION

[0002] Technical Field

[0003] The present disclosure relates to engineering automation systems. In particular, disclosed embodiments are directed to programming of automation devices, such as robots, conveyors, among others, for carrying out complex industrial tasks, using goal-based programming with semantic annotation.

[0004] Background

[0005] Programming automation devices for industrial tasks, such as assembly or object manipulation tasks, can be complex, often involving an interplay between multiple automation devices. For instance, conveyors may be required to move material items (e.g., parts), which may be placed in containers, to a robotic cell. Multiple material items may be required to be placed precisely based on the contents of containers that may originate from different sources and directions. At the robotic cell, a robot may perform operations on those material items to construct new material items (e.g., an assemblage of parts). The constructed material items may also be further processed and be relayed to containers that likewise are transported via conveyors.

[0006] With numerous parts, connections, and destinations, the number of factors that need to be managed can be large. For example, a robot may perform hundreds of unique motions for various pick and place operations. Also, in some examples, containers may store dozens of parts, each with a different position. Furthermore, in some examples, parts that need to be grasped may have specific requirements where to be grasped and what kind of end-effector is needed to perform the action.

[0007] Methods for programming automation devices are varied but most involve some kind of textual language. For example, for programming robots, a developer produces a fixed number set of paths that the robot may traverse. These end positions of these paths are called “teach points” and represent the destinations where the robot may travel. The teach points are given textual names so that they may be activated in the programming language. The controller that runs the language is then able to call out to the robot’s network interface to either activate a particular teach point or even activate a full path for the robot to follow. conveyor may be based on a network interface but can also be primitive such as sending Boolean values representing voltages to discrete wires that are custom built for the automation device. The device programmer is expected to know the interface and write specially tailored code that translates the intended behavior to interface signals.

[0008] Standard programming systems for automation thus typically use text language-like instructions to represent the operations of devices and how they respond to input events. Programs are separated from the specification of the devices and are usually written in device specific scripts that do not reflect the overall state of the system or devices beyond the one being programmed. This makes it difficult to identify and correct mistakes in the program, especially for complex tasks including a long sequence of instructions. Furthermore, standard programing systems may not be scalable to allow programming for different types of devices (e.g., robots and conveyors) to participate in the task.

[0009] Summary

[0010] The present disclosure addresses and overcomes one or more of the described-herein shortcomings or technical problems by providing methods, systems, and apparatuses for programming automation devices for carrying out industrial tasks using goal-based programming with semantic annotation.

[0011] A first aspect provides a computer-implemented method for engineering an automation system for a production process. The automation system comprises a number of physical objects that include material items and devices operable to manipulate the material items. The method comprises generating a simulation environment comprising a plurality of three-dimensional graphical objects. Each graphical object respectively represents a physical object of the automation system. The method further comprises attaching markers including semantic annotations to the graphical objects based on user instructions received via a graphical user interface. The markers include a number of placement markers, each describing a placement position of a material item relative to the physical object represented by the graphical object to which that marker is attached. At least some of the placement markers define hierarchical placement relationships between associated material items. The method further comprises specifying an operation of each device as a goal or a set of goals parameterized with the markers and groupings of material items based on user instructions received via the graphical user interface. The method further comprises generating code executable to perform the specified operations by the respective devices. program products for implementing the above-described method.

[0012] Additional technical features and benefits may be realized through the techniques of the present disclosure. Embodiments and aspects of the disclosure are described in detail herein and are considered a part of the claimed subject matter. For a better understanding, refer to the detailed description and to the drawings.

[0013] Brief Description of the Drawings

[0014] The foregoing and other aspects of the present disclosure are best understood from the following detailed description when read in connection with the accompanying drawings. To easily identify the discussion of any element or act, the most significant digit or digits in a reference number refer to the figure number in which the element or act is first introduced.

[0015] FIG. 1 illustrates a non-limiting example use case of a goal-based automation task, for assembling parts, in a simulation environment.

[0016] FIG. 2 illustrates 3D representations of material items in the example use case.

[0017] FIG. 3A-3C illustrate placement markers attached to a bottom assembled part of a PLC via a GUI in the example use case.

[0018] FIG. 4A-4B illustrate additional placement markers attached to parts of the PLC via the GUI in the example use case.

[0019] FIG. 5 illustrates placing of region markers via the GUI in the example use case.

[0020] FIG. 6A-6D illustrate placement markers indicating assembly positions for PLC parts in the example use case.

[0021] FIG. 7 illustrates an assemblage dialog box in the GUI created from relationships between placement markers associated with PLC parts in the example use case.

[0022] FIG. 8 illustrates placement markers being used to produce a container with filled slots in the example use case. associated with a container in the example use case.

[0023] FIG. 10 illustrates container placement markers at different positions of a front conveyor showing a fill state at each position in the example use case.

[0024] FIG. 11 illustrates container placement markers at different positions of a rear conveyor showing a fill state at each position in the example use case.

[0025] FIG. 12 illustrates a region marker at a starting location of the first conveyor to provide gaps between containers in the example use case.

[0026] FIG. 13 illustrates specifying operations of different devices as goals parameterized with markers via the GUI in the disclosed use case.

[0027] FIG. 14 illustrates a computing environment within which embodiments of the present disclosure may be implemented.

[0028] Detailed Description

[0029] In conventional automation, an automated device, such as a robot, is typically programmed to perform a single, repetitive task, such as positioning a car panel in exactly the same place on each vehicle. In this case, an engineer is usually involved in programming an entire task from start to finish, typically utilizing low-level code to generate individual commands. Simulation systems can run these programs to produce an animation for what the devices would do under different circumstances, but the programs do not intrinsically describe the process that the devices perform. For example, anything representing the materials or how they are transformed is typically not a part of the programmed language. Any transformation applied to materials is implied and not directly modeled. Likewise, the devices or fixtures that are used to handle the materials are also implied and not directly modeled. One may infer that some equipment may be present based on there being code that implements some function for them, but generally the code only describes the device operations and does not describe what actually happens in the physical world. production process in a goal-based programming paradigm using semantic annotations. Semantic annotations may be attached as markers to graphical objects in a simulation environment, where the graphical objects represent physical objects of the automation system. The International Patent Application filed by the present Applicant, published as WO 2024049466 A1, describes a graphical user interface to produce and use semantic markers, the contents of which are incorporated by reference herein in its entirety. The methodology proposed in the present disclosure builds on the concept to allow a user (e.g., an automation programmer) to program interactions between a number of automation devices using high-level goals that are parameterized with markers.

[0030] In this description, an “automation device” or simply “device” refers to a piece of physical equipment that can be controlled via a controller. A “material item” refers to a physical object that can be manipulated by a device. Material items can include, for example, parts of a product to be assembled, as well as containers that can be used to carry the parts. In addition to devices and material items, the layout of the automation system may typically include other physical objects such as tools (e.g., grippers, fasteners, etc.), fixtures (e.g., jigs, clamps, etc.), and so on.

[0031] The proposed methodology uses a simulation environment of the automation system as its basis. The simulation environment includes a number of three-dimensional (3D) graphical objects. Each graphical object is a data object representing a particular physical object of the automation system, such as a device, material item, fixture, tool, etc. A user may add annotations and material assembly descriptions directly into the simulation by selecting the relevant graphical objects and adding markers attached to those graphical objects via a graphical user interface (GUI). Markers may also be 3D graphical objects but are rendered stylistically differently from representations of physical objects. Markers can be of several types, as described hereinafter. In particular, the markers may include a number of placement markers, each describing a placement position of a material item relative to the physical object represented by the graphical object to which that marker is attached, where at least some of the placement markers define hierarchical placement relationships between the associated material items. The desired purpose of the automation may be specified as goals that are assigned to each active piece of equipment (“device”) in the simulation. Each type of device may have the kinds of goals that are particular to the kinds of actions the device can perform. Goals may be parameterized with the markers and groupings of material items (e.g., defining an assemblage of material items or filling material items in a container) that may be specified by the user via the operations by the respective devices.

[0032] The generated code may be transferred to a controller of the respective devices. The actual implemented device behavior in the physical world may consist of search algorithms to recognize key patterns when material items are ready to be operated upon (e.g., presence of particular material items in a container or an assemblage) and using the markers and goals to decide those actions to perform that change the state of the material items to complete the automation devices’ purpose.

[0033] The order and operation of the goals need not be defined by the user, which makes them different from ordinary instructions that are programmed step by step. For example, the automation system can autonomously decide when to “fill a container” and may interleave these operations with other goals, depending on the complexity of the overall task. Also, the manner in which a goal is achieved may be searched by the automation system and need not be explicitly stated by the user, such as which way to grip an object if there is more than one way.

[0034] The proposed methodology provides a programming framework that can be used to program interactions between different types of automation devices, including both robots, as well as non-robotic devices, such as conveyors. Using conveyors is a common method to feed material items to other devices such as robots as well as to transport material items away to other functions such as shipping or other processes. Conveyors can include anything, from simple belts with servos, to control speed up to complex track-based styles where individual parts may be routed independently. Conveyors may transport material items directly or they may transport containers that separately hold the material items. The containers may be simple bins with a pile of material items within or engineered trays with special slots that fit particular material items precisely.

[0035] The proposed methodology can describe the application state as a whole, including the layout of all relevant devices, and describe the function of each device based on the goals of the application in general. Specific input and output events need not be noted, and the actions of the devices may be based on recognizing important states between the devices and the material items being modified. For example, a conveyor in a traditional system would be programmed to start and stop based on an input event, such as a position sensor being tripped. The proposed methodology may describe a conveyor’s activity based on the start and end states of conveyed material items and where these material items should be located when important operations may be applied to them. An important consequence of describing the whole application state is that it makes it easier to restart operations when the process is interrupted. The state of material items in the process can be automatically restored to the program state, whereas a command-based program would typically have to start from the beginning if reset. Also, the methodology makes it is easier to recognize erroneous situations as the whole process is known, so that undesired situations can be recognized by comparing the actual state of material items to their virtual counterparts.

[0036] Referring now to the drawings, FIG. 1 illustrates an example use case implementing embodiments of the proposed methodology. The example is not intended to be limiting. That is, the types of devices, the layout, the materials, and operations may be changed or otherwise be different for different kinds of automation tasks.

[0037] The example shown in FIG. 1 demonstrates an assembly task in a production process. Trays (first containers) filled with material items enter a work cell via a front conveyor 102. In this example, the material items constitute parts of a programmable logic controller (PLC). The front conveyor 102 pauses the trays, one by one, at an area in front of a robot 104 that then assembles the parts on the respective tray. In this example, the parts snap together using press fit, but in other examples, the robot 104 may use other processes such as gluing, welding, or fastening, and these processes may involve various tools that the robot 104 may change. In some examples, there may be more than one device such as two robots or other kinds of devices for assembly. When a PLC is completely assembled on the tray, the robot 104 moves the constructed PLC to a rear conveyor 106, where the constructed PLCs are fitted to slots on bins (second containers). When a bin is fully loaded with constructed items, the rear conveyor 106 moves the bin to the next workstation that is outside the scope of the present work cell. Though the example, as described, focuses on the robot 104 and two conveyors 102, 106, other devices in the work cell could also be included in the programming framework, such as the testing station 108, the rejected component cart 110, among others.

[0038] The proposed methodology uses descriptions of the devices and material items as the means to convey the activity and purpose of the automation. The elements of the work cell are composed using 3D representations as would be performed in a 3D simulation of the devices. All parts of the work cell are specified, including the devices like the robot and conveyors, tools such as grippers or fasteners, fixtures such as jigs or clamps, and material items such as parts and containers (trays and bins). In the present example, there is a front conveyor 102 that transports the trays containing unassembled parts coming into the work cell, and a rear conveyor 106 that transports the bins containing constructed PLCs. There is a robot 104 (in this case, a UR5eTM machined to hold various parts of the assembly in different cut out sections, so that though the parts are differently sized, they all may be gripped firmly. Each of these parts may be arranged using a 3D editor in the configuration that they would have in the physical work cell.

[0039] Instead of writing explicit code for each device, the user / automation programmer may specify the task in a more abstract, descriptive form by laying out markings on the devices, assigning materials to groupings to show how they are assembled or held in a container, and by noting the overall tasks or goals that the devices must perform.

[0040] FIG. 2 shows 3D representations of material items in the example use case. Material items are physical objects that are manipulated by a device. As shown, there are nine material items in this example. Six of the material items are parts of a PLC that are to be assembled. These include a lower exterior case 202, a circuit board 204, a display unit 206, an upper case 208, an insert 210 and a keypad 212. Three of the material items are parts of containers that hold the other material items when being transported on a conveyor. These include a tray or first container 214, an outer portion 218 of a bin or second container 220, and an insert 216 of the bin 220 that defines slots for placing the finished PLC. In some taxonomies, materials and containers can be considered different entities because the containers are not really converted into a product. However, in the described embodiment, any object that is manipulated by a device is being considered a material item for simplicity.

[0041] The methodology in the described use case involves at least three kinds of annotations or markers that mark out locations of special semantic importance. These markers include placement markers, grip markers and region markers.

[0042] A placement marker describes a placement position of a material item relative to the physical object (which could be another material item, or a device, fixture or other object) represented by the graphical object to which that marker is attached. Some placement markers can also describe the position of a material item relative to the 3D space of the work cell. Placement markers may be attached to devices and material items so that if the connected object (device / material item) moves, the marker will remain together with the moving object in the same relative position. Placement markers may be generic in many ways, in that they are not used for a specific purpose but can be used for different purposes when applied to different goals. For example, a placement marker may be used to show where an object is placed in a container, it can show where objects are connected when assembled, it can show where an object is placed inside a device, and many other possible meanings. At least some of the placement markers may define hierarchical placement relationships between the associated material items. This may be explained referring to FIG 3A-3C which illustrate placement markers attached to a bottom assembled part of a PLC. FIG. 3A shows an assemblage 302 of parts, including the lower exterior case 202 and the circuit board 204. In FIG. 3B, a placement marker 206-p associated with the display unit 206 is shown. The placement marker 206-p describes where the display unit 206 becomes attached to the lower exterior case assemblage 302 shown in FIG. 3A. In FIG. 3C, the placement marker 208-p associated with the upper case 208 is shown. The placement marker 208-p describes where the upper case 208 becomes attached to the lower exterior case assemblage 302.

[0043] FIG. 4A-4B illustrate additional placement markers attached to parts of the PLC via the GUI in the example use case. FIG. 4A shows a placement marker 212-p associated with the keypad 212, describing where the keypad 212 becomes attached to the display unit 206. FIG. 4B shows a placement marker 210-p associated with the insert 210, describing where the insert 210 becomes attached to the upper case 208.

[0044] A grip marker is similar to a placement marker but describes the position of a device relative to a physical object (such as a material item, another device, a tool or a fixture). A device often has moving parts that interact with material items or other devices. For example, a robot may pick an object up for assembly, it may pick up a tool such as a fastener, or it may manipulate another piece of equipment like open up a door. Grip markers may be attached to graphical objects based on user instructions provided via a GUI as described in the publication WO 2024049466 A1.

[0045] A region marker defines an exclusive area in a simulated workspace (e.g., the 3D space of the described work cell) to execute a goal parameterized with the region marker. Region markers may be used to indicate places where operations are expected to occur and to track moving objects that may pass through a region, such as keeping track of their number. Referring to FIG. 5, three region markers are shown by box regions 502, 504 and 506. The region markers 502 and 506 may be used to control the number of simulated objects that are placed on the front conveyor 102 and the rear conveyor 106 respectively. For example, the region markers 502, 506 may be used to prevent multiple containers from appearing in the demarcated regions, thus maintaining a desired spacing between containers. The region marker 504 may be used to control the area where the robot 104 performs assembly operations. be specified in the engineering tool or derived from other information such as Bill of Process or Bill of Material specifications. An assemblage is a group of material items that are assembled together to make a new material item. In the present example, the whole PLC with all six parts shown in their proper final arrangement is an assemblage. When the PLC becomes assembled, it may act as a new material item that can be manipulated by automation devices. For example, the PLC assemblage may be marked with a grip mark to show how a robot would pick it up. In case of a container, the contents of a container (i.e., a grouping or collection of material items) define its “manifest.” When a container is specified, the contents of the container make up its state, which is relevant to operations that manipulate either the container or material items within the container.

[0046] The proposed methodology allows the user to create an assemblage based on placement markers. In one embodiment, to create an assemblage, a tree of material items may be generated by reading the placement markers associated with these material items that define hierarchical placement relationships. A dialog box may then be created in the GUI to receive user instructions to select material items from the tree and to specify a build order.

[0047] To illustrate using the present example, the placement of parts of the PLC onto the parts on which they are assembled are described with placement markers. Referring to FIG. 6A-6D, four placement markers are shown that are used to show the positions where parts become connected to other parts in the assembly process. FIG. 6A shows the placement marker 208-p for the upper case 208, describing the position where the upper case 208 is connected to the lower exterior case assemblage 302 (which includes the lower exterior case 202 and the circuit board 204 shown in FIG. 3A). FIG. 6B shows the placement marker 206-p for the display unit 206, describing the position where the display unit 206 is connected to the lower exterior case assemblage 302. FIG. 6C shows the placement marker 212-p for the keypad 212, describing the position where the keypad 212 is connected to the display unit 206. FIG. 6D shows the placement marker 210-p for the insert 210, describing the position where the insert 210 is connected to the upper case 208.

[0048] By reading all the associated placement markers for each material item, a dialog box may be automatically created for specifying an assembly by the user. In FIG. 7, a dialog box 700 for creating the final constructed PLC assemblage is shown. A tree including a hierarchy of parts is generated from the relationships of the placement markers shown in FIG. 6A-6D. The dialog box 700 includes a first portion 702 indicating the tree, in which the user may select which parts are to be included in the assemblage. The dialog box 700 includes a second portion 704 where at 702) are to be assembled.

[0049] The proposed methodology also allows the user to create manifest information for containers based on placement markers. The manifest information defines a grouping of material items forming contents of a container. In one embodiment, to create the manifest information, one or more placement markers may be attached at one or more placement positions on a graphical object representing the container. A dialog box may then be created in the GUI to receive user instructions to assign material items to the one or more placement positions based on the placement markers attached thereto, and to specify a fill order. In some embodiments, a set of options may be automatically generated for each placement position indicated in the dialog box, the options being generated based on specified assemblages that are inclusive of the material item associated with the placement marker at that placement position. In some embodiments, the options generated for a placement position may be binary, allowing a user to either assign a part associated with the placement marker attached thereto, or leave the placement position empty.

[0050] To illustrate using the present example, FIG. 8A-8B show more collections of placement markers, this time to produce a container filled with parts. FIG. 8A shows the graphical object for the tray 214. The tray 214 has five slots 802, 804, 806, 808 and 810, which define placement positions. On each of these placement positions, a placement marker for a part is attached, which describes a position of the part when that part is placed in the tray 214. As shown, the placement marker 212-p for the keypad 212 is attached to the placement position defined by the slot 802, the placement marker 206-p for the display unit 206 is attached to the placement position defined by the slot 804, the placement marker 210-p for the insert 210 is attached to the placement position defined by the slot 806, the placement marker 202-p for the lower exterior case 202 is attached to the placement position defined by the slot 808, and the placement marker 208-p for the upper case 208 is attached to the placement position defined by the slot 810. In FIG. 8B, a placement marker 214-p for the tray 214 is shown at the starting location of the front conveyor 102, which, in turn, includes placement markers as described in FIG. 8A.

[0051] The tray 214 itself has placement markers associated with it. By reading the placement marker associated with the tray 214, a dialog box may be automatically created for specifying manifest information by assigning material items to placement positions (slots) on the tray 214, based on the placement markers attached thereto. In FIG. 9, a dialog box 900 for building a container by specifying manifest information is shown. The dialog box 900 includes a first portion 902, assemblage) to that slot. From this dialog, the user may select which parts are loaded into the tray object at the respective slots shown by the tray’s placement marker. The options for each tray slot are automatically populated based on the assemblages that are inclusive of the part associated with the placement marker shown in FIG. 8A. For example, assemblages inclusive of the lower exterior case 202, which is the part in the tray slot 808, can be the lower exterior case 202 itself, the lower exterior case assemblage 302 (see FIG. 3A), or the whole finished PLC assemblage. This can allow different kinds of material that would fit to be placed in that slot and would allow assembly operations to be carried out on the tray slot by a robot such that the state of material in the slot changes, as described below. Selecting the parts for each slot produces the filled container placement marker, as shown in FIG. 8B. Note the options for each tray slot also include an ‘empty’ option to allow the user to leave the tray slot empty. The dialog box 900 includes a second portion 904 where the user may specify a fill order, i.e. , the order in which the assigned parts (i.e., parts assigned at 902) are to be filled in the container.

[0052] The operation of each device of the automation system is specified with “goals” that are essentially simple directives that are parameterized with the markers and groupings of material items that describe the specific activity of each device. An automation system can include a number of devices including one or more robots, conveyors, among others, for carrying out an automation task.

[0053] A conveyor is operable to transport material items. When a conveyor is present, the operation of the conveyor is specified with one or more conveyor goals. Each conveyor goal is respectively parameterized with a material placement marker attached at a defined location on the graphical object representing the conveyor. The material placement marker indicates an expressed material state at the defined location. In some embodiments, for example, as described herein, a conveyor may be operable to transport material items held in a container. The container itself is also considered a material item. In this case, each material placement marker is a container placement marker indicating an expressed container fill state described by manifest information created for the defined location. In other embodiments, the material being conveyed can just be plain material items that are not held in a container. In some cases, a conveyor goal may be additionally parameterized with a region marker defining an exclusion region where the conveyor goal is to be performed.

[0054] For example, a conveyor may have an “Enter” goal, an “Exit” goal and, in some cases, a “Fill” goal. An Enter goal may be used to bring new containers onto the conveyor at a starting location of the conveyor. An Exit goal may be used to remove containers at an end location of instead come in from or move on to other processes beyond the scope of the work cell. A Fill goal may be used to place containers at other defined locations along the conveyor. The locations must be feasible for the conveyor to position the containers. For example, for a onedimensional straight conveyor, these locations would be along a line. The Enter goal may take a container placement marker and a region marker as parameters. The container placement marker determines the fill state of the container at the starting location of the conveyor. The region marker parameter provides an exclusion area around the origin point to allow multiple containers to appear on the conveyor without overlapping. The semantic is that only one container will appear inside the region. If no region is specified, the whole workspace can be considered the region and only one container appears on the conveyor at a time.

[0055] The semantic for the Fill and Exit goals for a conveyor are similar. Each of these goals takes a container placement marker as its parameter. The overall goal is to move a container that matches the expressed fill state of the container placement marker to the location on the conveyor where that container placement marker is attached. When the container reaches the Fill location, the conveyor is stopped, and the system will wait until the fill state of the container matches another goal. When the container reaches the Exit location, it is removed from the simulation.

[0056] In the present example, the operation of the front conveyor 102 includes producing trays with parts at the head of the conveyor 102, bringing the parts carried in the trays in front of the robot 104, and finally taking away the empty trays at the end of the conveyor 102. To indicate these states, the user may attach container placement markers at the key locations on the front conveyor 102 as shown in FIG. 10. The container state is set to show what parts would be in the tray when a given position is reached. At the starting location (head) of the conveyor 102, the tray is full, and it is still full when the tray reaches the location of the robot 104. At the end location of the conveyor 102, the tray is empty.

[0057] Referring to FIG. 10, the user may attach a first tray placement marker 214-p-1 for the tray 214 at the starting location of the graphical representation of the conveyor 102. The first tray placement marker 214-p-1 indicates a fill state of the tray 214 at the head of the conveyor 102. The fill state is expressed by manifest information for this location, which may be created by assigning parts to slots as described above in FIG. 9. In this case (see also FIG. 8A), the fill state is defined by assigning the keypad 212 to the slot 802, the display unit 206 to the slot 804, the insert 210 to the slot 806, the lower exterior case assemblage 303 to the slot 808 and the upper case 208 to the slot 810 via the dialog box 900. The user may likewise attach a second location where the robot 104 performs the assembly operation. The expressed fill state defined by the second tray placement marker 214-p-2 is similar to that defined by the first tray placement marker 214-p-1 , since the conveyor 102 is expected to transport the tray in the same fill state to the robot 104. The user may attach a third tray placement marker 214-p-3 to the graphical representation of the conveyor 102 at an end location of the conveyor 102. The expressed fill state defined by the third tray placement marker 214-p-3 is that all slots are empty. The manifest information describing this fill state may be created by selecting the option ‘empty’ for each slot in the dialog box of FIG. 9. In this manner, manifest information may be created for each of the three locations on the conveyor 102 based on tray placement markers that describe the expressed fill state of the tray at the respective locations.

[0058] The operation of the front conveyor 102 may be specified by a set of goals including an Enter goal, a Fill goal and an Exit goal.

[0059] Still referring to FIG. 10, the Enter goal for the conveyor 102 is parameterized with the tray placement marker 214-p-1 , which is attached to the graphical representation of the conveyor 102 at the starting or Enter location, as well as with the region marker 502. The Enter goal produces trays 214 at the head of the conveyor 102 in simulation, based on the expressed fill state defined by the tray placement marker 214-p-1. That is, the Enter goal creates an instance of a tray 214 at the starting location of the conveyor 102 with all the parts in the tray 214, as described by the manifest information for this location. The region marker 502 (see also FIG. 12) defines an exclusion area around the origin point to allow multiple trays 214 to appear on the conveyor 102 without overlapping. The region marker 502 can thus be used to control the number of trays appearing at the head of the conveyor 102, and depending on the size of the region marker 502, control a spacing between trays 214 on the conveyor 102 when the conveyor 102 is run.

[0060] The Fill goal for the conveyor 102 places a tray 214 at a Fill location where the robot 104 performs the assembly operation. The Fill goal for the conveyor 102 is parameterized with the tray placement marker 214-p-2 attached to the graphical representation of the conveyor 102 at the Fill location. The tray placement marker 214-p-2 defines the same fill state as the tray placement marker 214-p-1. The Fill goal works by determining if a tray 214 in its current fill state matches the fill state defined by the tray placement marker 214-p-2. Only in case of a match, the tray 214 is placed at the Fill location. For example, if the tray has different parts (does not match with expressed fill state at the Fill location), then the Fill location will not be filled with that tray. Such a mismatch may occur, for example, because the Fill location for that tray may be user error.

[0061] The Exit goal for the conveyor 102 removes a tray 214 from simulation at the end or Exit location of the conveyor 102. The Exit goal for the conveyor 102 is parameterized with the tray placement marker 214-p-3 attached to the graphical representation of the conveyor 102 at the Exit location. As noted earlier, the expressed fill state for the tray placement marker 214-p-3 is that the tray is completely empty. At the Fill location, the robot 104 changes the state of the material items in the tray 214 (assembles parts and places finished PLC on rear conveyor 106), that allows the fill state of the tray 214 to match the expressed fill state defined by the tray placement marker 214-p-3, whereby the Exit goal is achieved.

[0062] The operation of the rear conveyor 102 includes receiving finished PLCs from the robot 104 into empty bins 220 and transporting the bins 220 away from the robot 104, for example, to a next workstation. Referring to FIG. 11 , the user may attach a first bin placement marker 220-p-1 for a bin 220 at a starting or Enter location of the graphical representation of the conveyor 106. The first bin placement marker 220-p-1 indicates a fill state of the bin at the head of the conveyor 106. The fill state is expressed by manifest information for this location, which may be created by assigning parts to slots similar to that described above in FIG. 9 in connection with the tray 214. In this case the parts are all identical, being finished PLCs. The expressed fill state defined by the first bin placement marker 220-p-1 is that the bin is completely empty. The user may attach a second bin placement marker 220-p-2 for a bin 220 at an end or Exit location of the graphical representation of the conveyor 106. The expressed fill state defined by the second bin placement marker 220-p-2 is that some or all of the slots in the bin are assigned respectively with a finished PLC. Note that the bin is not completely full in this example. In this case, the user only wants three finished PLCs in the bin when it is conveyed away by the conveyor 106.

[0063] The operation of the rear conveyor 106 may be specified by a set of goals including an Enter goal and an Exit goal. For the rear conveyor 106, the manipulation occurs immediately when the bin is brought onto the conveyor 106 by the robot 104 filling it in. Since the rear conveyor 106 does not feed parts to any intermediate location where material operations occur, there is no Fill goal in this case. The Enter goal is parameterized with the bin placement marker 220-p-1, which is attached to the graphical representation of the conveyor 106 at the Enter location, as well as with the region marker 506. The Exit goal for the conveyor 106 is parameterized with the bin placement marker 220-p-2 attached to the graphical representation of the conveyor 106 at the Exit location. The Enter and Exit goals for the rear conveyor 106 operate similarly to that of the front conveyor 102, and hence will not be described further. The operation of a robot may be specified with one or more robot goals. The robot goals may involve high-level directives, where each goal may involve performing pick and place tasks parameterized with a recipe defined by the created assemblage or manifest information. Each individual pick or place task may be performed based on grip markers attached to the material items.

[0064] In the present example, the robot 104 has at least two types of goals: an Assemble goal, and a Fill goal. Note that the Fill goal for the robot is the same name as one of the conveyor goals because they have a similar effect. The Fill goal causes material items to be placed at a particular location based on a placement marker as its parameter. The means to fulfill the Fill goal is quite different for the robot 104 compared to the conveyors 102, 106, since a conveyor only has a single direction of movement while a robot can pick up objects and place them in arbitrary locations.

[0065] The Assemble goal for the robot 104 is to assemble a PLC according to its assemblage instructions, which is specified as illustrated in FIG. 7. The order of assembly and the precise locations for all the parts is already given in the assemblage instructions. The same placement markers that were used to produce the assembly tree in the assemblage may be used by the robot 104 to pick up parts and place them into the correct positions. The Assemble goal for the robot 104 is further parameterized with a region marker 504 that allows the robot 104 to focus the assembly task to a restricted region (see FIG. 5). The assemblage instructions include a recipe which specifies the parts to be assembled and also the order in which the parts are assembled to construct a finished PLC, so that the robot 104 may place objects (i.e. , parts of a PLC) in the right sequence.

[0066] When the PLC is fully assembled, the fill state of the tray 214 still does not match the expressed fill state defined the tray placement marker 214-p-3 which parameterizes the Exit goal of the front conveyor 102 (see FIG. 10). This is because the PLC is still in the tray, and the tray is not empty as required by the fill state defined by the tray placement marker 214-p-3. To move the finished PLC, a Fill goal is specified for the robot 104 to place finished PLCs into the bin 220 on the rear conveyor 106. The Fill goal can take the bin placement marker 220-p-1 (see FIG. 11) as its parameter, and it may also take the contents of a bin as its parameters. Like the assemblage, the bin manifest information includes a recipe that specifies the contents of the bin and also the order of filling so that the robot 104 may place objects (i.e., finished PLCs) in the right sequence. system as goals parameterized with markers. As shown, for each device, such as the robot 104 (“UR5”), the front conveyor 102 (“long conveyor”) and the rear conveyor 106 (“conveyor2”), goals may be added that are particular to the kinds of actions that the device can perform. For example, the goals can include one or more of: Enter, Exit and Fill Goals for conveyors and Assemble and Fill goals for the robot as described above. Once a goal is added for a device, a dialog may be automatically created allowing the user to specify parameters, such as placement and (if applicable) region markers, for that goal.

[0067] After the device goals have been specified, such as described above, the engineering system may generate executable code for performing the goals by the respective devices. In one embodiment, the generated code may be used to simulate the operation of each device in the 3D simulation environment. The user may start and stop the simulation, reverse time, play the simulation in real time or at different speeds, and other possible visualizations. When a desired result is indicated by the simulation, i.e. , the user is satisfied with the application’s function, the code may be transferred to one or more physical controllers configured to control the respective devices, for example, using specialized interfaces such as described in the United States Patent No. 10,620,917 assigned to the present Applicant. The actual device behavior in the physical world may be created, for example, by creating a digital twin of the device state in the controller and deploying a search algorithm to recognize key patterns when material items are ready to be operated upon (e.g., presence of particular material items in a container or an assemblage) and using the markers and goals on the digital twin to perform those actions that change the state of the material items to complete the automation devices’ purpose.

[0068] FIG. 14 illustrates an exemplary computing environment 1400 within which embodiments of the disclosure may be implemented. Computers and computing environments, such as computer system 1410 and computing environment 1400, are known to those of skill in the art and thus are described briefly here.

[0069] As shown in FIG. 14, the computer system 1410 may include a communication mechanism such as a system bus 1421 or other communication mechanism for communicating information within the computer system 1410. The computer system 1410 further includes one or more processors 1420 coupled with the system bus 1421 for processing the information.

[0070] The processors 1420 may include one or more central processing units (CPUs), graphical processing units (GPUs), or any other processor known in the art. More generally, a processor as used herein is a device for executing machine-readable instructions stored on a computer and firmware. A processor may also comprise memory storing machine-readable instructions executable for performing tasks. A processor acts upon information by manipulating, analyzing, modifying, converting or transmitting information for use by an executable procedure or an information device, and / or by routing the information to an output device. A processor may use or comprise the capabilities of a computer, controller or microprocessor, for example, and be conditioned using executable instructions to perform special purpose functions not performed by a general-purpose computer. A processor may be coupled (electrically and / or as comprising executable components) with any other processor enabling interaction and / or communication there-between. A user interface processor or generator is a known element comprising electronic circuitry or software or a combination of both for generating display images or portions thereof. A user interface comprises one or more display images enabling user interaction with a processor or other device.

[0071] Continuing with reference to FIG. 14, the computer system 1410 also includes a system memory 1430 coupled to the system bus 1421 for storing information and instructions to be executed by processors 1420. The system memory 1430 may include computer readable storage media in the form of volatile and / or nonvolatile memory, such as read only memory (ROM) 1431 and / or random-access memory (RAM) 1432. The RAM 1432 may include other dynamic storage device(s) (e.g., dynamic RAM, static RAM, and synchronous DRAM). The ROM 1431 may include other static storage device(s) (e.g., programmable ROM, erasable PROM, and electrically erasable PROM). In addition, the system memory 1430 may be used for storing temporary variables or other intermediate information during the execution of instructions by the processors 1420. A basic input / output system 1433 (BIOS) containing the basic routines that help to transfer information between elements within computer system 1410, such as during start-up, may be stored in the ROM 1431. RAM 1432 may contain data and / or program modules that are immediately accessible to and / or presently being operated on by the processors 1420. System memory 1430 may additionally include, for example, operating system 1434, application programs 1435, other program modules 1436 and program data 1437.

[0072] The computer system 1410 also includes a disk controller 1440 coupled to the system bus 1421 to control one or more storage devices for storing information and instructions, such as a magnetic hard disk 1441 and a removable media drive 1442 (e.g., floppy disk drive, compact disc drive, tape drive, and / or solid-state drive). Storage devices may be added to the computer system 1410 using an appropriate device interface (e.g., a small computer system interface (SCSI), integrated device electronics (IDE), Universal Serial Bus (USB), or FireWire). bus 1421 to control a display or monitor 1466, such as a cathode ray tube (CRT) or liquid crystal display (LCD), for displaying information to a computer user. The computer system includes an input interface 1460 and one or more input devices, such as a keyboard 1462 and a pointing device 1461 , for interacting with a computer user and providing information to the processors 1420. The pointing device 1461 , for example, may be a mouse, a light pen, a trackball, or a pointing stick for communicating direction information and command selections to the processors 1420 and for controlling cursor movement on the display 1466. The display 1466 may provide a touch screen interface which allows input to supplement or replace the communication of direction information and command selections by the pointing device 1461. In some embodiments, an augmented reality device 1467 that is wearable by a user, may provide input / output functionality allowing a user to interact with both a physical and virtual world. The augmented reality device 1467 is in communication with the display controller 1465 and the user input interface 1460 allowing a user to interact with virtual items generated in the augmented reality device 1467 by the display controller 1465. The user may also provide gestures that are detected by the augmented reality device 1467 and transmitted to the user input interface 1460 as input signals.

[0073] The computer system 1410 may perform a portion or all of the processing steps of embodiments of the invention in response to the processors 1420 executing one or more sequences of one or more instructions contained in a memory, such as the system memory 1430. Such instructions may be read into the system memory 1430 from another computer readable medium, such as a magnetic hard disk 1441 or a removable media drive 1442. The magnetic hard disk 1441 may contain one or more datastores and data files used by embodiments of the present invention. Datastore contents and data files may be encrypted to improve security. The processors 1420 may also be employed in a multi-processing arrangement to execute the one or more sequences of instructions contained in system memory 1430. In alternative embodiments, hard-wired circuitry may be used in place of or in combination with software instructions. Thus, embodiments are not limited to any specific combination of hardware circuitry and software.

[0074] As stated above, the computer system 1410 may include at least one computer readable medium or memory for holding instructions programmed according to embodiments of the invention and for containing data structures, tables, records, or other data described herein. The term “computer readable medium” as used herein refers to any medium that participates in providing instructions to the processors 1420 for execution. A computer readable medium may take many forms including, but not limited to, non-transitory, non-volatile media, volatile media, state drives, magnetic disks, and magneto-optical disks, such as magnetic hard disk 1441 or removable media drive 1442. Non-limiting examples of volatile media include dynamic memory, such as system memory 1430. Non-limiting examples of transmission media include coaxial cables, copper wire, and fiber optics, including the wires that make up the system bus 1421 . T ransmission media may also take the form of acoustic or light waves, such as those generated during radio wave and infrared data communications.

[0075] The computing environment 1400 may further include the computer system 1410 operating in a networked environment using logical connections to one or more remote computers, such as remote computing device 1480. Remote computing device 1480 may be a personal computer (laptop or desktop), a mobile device, a server, a router, a network PC, a peer device or other common network node, and typically includes many or all of the elements described above relative to computer system 1410. When used in a networking environment, computer system 1410 may include modem 1472 for establishing communications over a network 1471, such as the Internet. Modem 1472 may be connected to system bus 1421 via user network interface 1470, or via another appropriate mechanism.

[0076] Network 1471 may be any network or system generally known in the art, including the Internet, an intranet, a local area network (LAN), a wide area network (WAN), a metropolitan area network (MAN), a direct connection or series of connections, a cellular telephone network, or any other network or medium capable of facilitating communication between computer system 1410 and other computers (e.g., remote computing device 1480). The network 1471 may be wired, wireless or a combination thereof. Wired connections may be implemented using Ethernet, Universal Serial Bus (USB), RJ-6, or any other wired connection generally known in the art. Wireless connections may be implemented using Wi-Fi, WiMAX, and Bluetooth, infrared, cellular networks, satellite or any other wireless connection methodology generally known in the art. Additionally, several networks may work alone or in communication with each other to facilitate communication in the network 1471.

[0077] An executable application, as used herein, comprises code or machine-readable instructions for conditioning the processor to implement predetermined functions, such as those of an operating system, a context data acquisition system or other information processing system, for example, in response to user command or input. An executable procedure is a segment of code or machine-readable instruction, sub-routine, or other distinct section of code or portion of an executable application for performing one or more particular processes. These processes may include receiving input data and / or parameters, performing operations on received input data output data and / or parameters.

[0078] A graphical user interface (GUI), as used herein, comprises one or more display images, generated by a display processor and enabling user interaction with a processor or other device and associated data acquisition and processing functions. The GUI also includes an executable procedure or executable application. The executable procedure or executable application conditions the display processor to generate signals representing the GUI display images.

[0079] These signals are supplied to a display device which displays the image for viewing by the user. The processor, under control of an executable procedure or executable application, manipulates the GUI display images in response to signals received from the input devices. In this way, the user may interact with the display image using the input devices, enabling user interaction with the processor or other device.

[0080] The functions and process steps herein may be performed automatically or wholly or partially in response to user command. An activity (including a step) performed automatically is performed in response to one or more executable instructions or device operation without user direct initiation of the activity.

[0081] The system and processes of the figures are not exclusive. Other systems, processes and menus may be derived in accordance with the principles of the invention to accomplish the same objectives. Although this invention has been described with reference to particular embodiments, it is to be understood that the embodiments and variations shown and described herein are for illustration purposes only. Modifications to the current design may be implemented by those skilled in the art, without departing from the scope of the invention. As described herein, the various systems, subsystems, agents, managers and processes can be implemented using hardware components, software components, and / or combinations thereof.

Claims

Claims1. A computer-implemented method for engineering an automation system for a production process, the automation system comprising a number of physical objects that include material items and devices operable to manipulate the material items, the method comprising: generating a simulation environment comprising a plurality of three-dimensional graphical objects, each graphical object respectively representing a physical object of the automation system, attaching markers including semantic annotations to the graphical objects based on user instructions received via a graphical user interface, the markers including a number of placement markers, each describing a placement position of a material item relative to the physical object represented by the graphical object to which that marker is attached, wherein at least some of the placement markers define hierarchical placement relationships between associated material items, specifying an operation of each device as a goal or a set of goals parameterized with the markers and groupings of material items based on user instructions received via the graphical user interface, and generating code executable to perform the specified operations by the respective devices.

2. The method according to claim 1, comprising creating an assemblage defining a grouping of material items, by: generating a tree of material items by reading the placement markers associated therewith that define hierarchical placement relationships, and creating a first dialog box in the graphical user interface to receive user instructions to select material items from the tree and to specify a build order.

3. The method according to any of claims 1 and 2, wherein the material items include containers for holding other material items, the method comprising creating manifest information defining a grouping of material items forming contents of a container, by: attaching one or more placement markers at one or more placement positions on a graphical object representing the container, creating a second dialog box in the graphical user interface to receive user instructions to assign material items to the one or more placement positions based on the placement markers attached thereto, and to specify a fill order.

4. The method according to claim 3, comprising generating a set of options for each placement position indicated in the second dialog box, wherein the options are generated based on specified assemblages that are inclusive of the material item associated with the placement marker at that placement position.

5. The method according to any of claims 1 to 4, wherein the devices include at least one conveyor operable to transport material items, and wherein the operation of the conveyor is specified with one or more conveyor goals, each conveyor goal being respectively parameterized with a material placement marker attached at a defined location on the graphical object representing the conveyor, the material placement marker indicating an expressed material state at the defined location.

6. The method according to claim 5, wherein the at least one conveyor is operable to transport material items in containers, wherein each material placement marker is a container placement marker indicating an expressed container fill state described by manifest information created for the defined location.

7. The method according to any of claims 2 to 4, wherein the devices include at least one robot, and wherein the operation of the robot is specified with one or more robot goals including pick and place tasks parameterized with a recipe defined by the created assemblage and / or manifest information.

8. The method according to any of claims 1 to 7, wherein the markers further comprise a region marker to define an exclusive area in a simulated workspace to execute a goal parameterized with the region marker.

9. The method according to any of claims 1 to 8, comprising: simulating the operation of each device in the simulation environment based on the generated code, and if a desired result is indicated by the simulation, transferring the code to one or more physical controllers configured to control the respective devices.

10. A non-transitory computer-readable storage medium including instructions that, when processed by one or more processors, configure the one or more processors to perform the method according to any one of claims 1 to 9.

11. An automation engineering system comprising: one or more processors, and memory storing instructions executable by the one or more processors to perform a method according to any of claims 1 to 9.

Images