Target-oriented control of a robot arm

Abstract

Methods and systems for improving control of a robotic arm (114, 600) are presented. The methods include pre-defining a plurality of meta-actions (120, 200, 210, 508), which can include one or more pre-conditions (122, 202, 204, 212, 214, 216, 504) and effects (124, 222, 224, 226, 228, 506). A target state of a plurality of workpieces (108, 304, 306, 308, 310) can be determined, as well as an initial state of the plurality of workpieces (108, 304, 306, 308, 310). A series of operations (206, 208, 217, 218, 220) can be generated from the pre-conditions (122, 202, 204, 212, 214, 216, 504) and / or effects (124, 222, 224, 226, 228, 506) of the meta-actions (120, 200, 210, 508), as well as the target state and the initial state. Executing the series of operations (206, 208, 217, 218, 220) is capable of changing the plurality of workpieces (108, 304, 306, 308, 310) from the initial state to the target state.

Description

Background Technology

[0001] Over the years, the use of robotic arms has impacted many industries. Robotic arms can vary in function, ranging from performing simple movements to executing complex tasks, and the same robotic arm can be used for multiple types of tasks. These movements and tasks require programming the robotic arm to perform actions in a specific sequence according to the given movements or tasks. Summary of the Invention

[0002] This disclosure presents novel and innovative systems and methods for controlling the movement of a robotic arm. In a first aspect, a method is provided for controlling a robotic arm to manipulate multiple workpieces. This method includes predefining multiple meta-actions of the robotic arm, each of which includes one or more preconditions and one or more effects, and acquiring a target state of the multiple workpieces. The method may further include acquiring an initial state of the multiple workpieces and generating a series of operations based on one or more preconditions and one or more effects of the multiple meta-actions, the target state of the multiple workpieces, and the initial state. These operations can be selected from the multiple meta-actions, and executing the series of operations can change the multiple workpieces from the initial state to the target state.

[0003] According to the second aspect of the first aspect, obtaining the target state of multiple workpieces includes identifying the state of the prototype corresponding to the target state of the multiple workpieces.

[0004] According to the third aspect of the second aspect, the state of the prototype includes the relative positions and relative connections between the prototype's components.

[0005] In the fourth aspect, which is based on either the second or third aspect, the state of the prototype is identified using a visual acquisition device.

[0006] In the fifth aspect according to any one of the first to fourth aspects, acquiring the initial state of the plurality of workpieces includes acquiring the initial state of the plurality of workpieces using a vision acquisition device.

[0007] In the sixth aspect according to any one of the first to fifth aspects, the initial state of the multiple workpieces includes the orientation of the multiple workpieces, and a series of operations are generated based on the orientation of the multiple workpieces.

[0008] In the seventh aspect according to any one of the first to sixth aspects, the initial state of the plurality of workpieces includes the relative positions and relative connections between the plurality of workpieces, and a series of operations generated based on the relative positions and relative connections between the plurality of workpieces.

[0009] In the eighth aspect according to any one of the first to seventh aspects, generating a series of operations based on one or more preconditions and one or more effects of multiple meta-actions, the target state and initial state of multiple workpieces includes generating a series of operations in such a way that each precondition of the starting operation in the operation satisfies the initial state of multiple workpieces; and each effect of the last operation in the operation satisfies the target state of multiple workpieces.

[0010] In accordance with the ninth aspect of the eighth aspect, generating a series of operations based on one or more preconditions and one or more effects of multiple meta-actions, and the target state and initial state of multiple workpieces, further includes generating a series of operations in such a way that one or more effects of a previous operation in the operation can change multiple workpieces from a previous state to a subsequent state. One or more preconditions of a subsequent operation in the operation can satisfy the subsequent state.

[0011] In the tenth aspect, which is based on any one of the eighth to ninth aspects, a series of operations are generated by using a graph traversal algorithm.

[0012] In the eleventh aspect according to any one of the first to tenth aspects, the method further includes performing a collision check to determine whether a series of operations are feasible, and in response to determining that a series of operations are not feasible, regenerating a new series of operations based on one or more preconditions and one or more effects of a plurality of meta-actions, and target states and initial states of a plurality of artifacts.

[0013] In the twelfth aspect according to any one of the first to eleventh aspects, the method further includes performing a collision check to determine whether a series of operations are feasible, and performing the series of operations in response to determining that the series of operations are feasible.

[0014] In a thirteenth aspect, a system for controlling a robotic arm is provided, including a processor and a memory storing instructions that, when executed by the processor, cause the processor to perform a method for controlling the robotic arm to manipulate a plurality of workpieces. The method may include predefined multiple meta-actions of the robotic arm, each of the multiple meta-actions including one or more preconditions and one or more effects, and acquiring a target state of the multiple workpieces; the method may further include acquiring an initial state of the multiple workpieces and generating a series of operations based on one or more preconditions and one or more effects of the multiple meta-actions, the target state of the multiple workpieces, and the initial state. These operations can be selected from the multiple meta-actions, and executing the series of operations can change the multiple workpieces from the initial state to the target state.

[0015] According to aspect fourteen of aspect thirteen, the system also includes a vision acquisition device. The state of the prototype can be identified by the vision acquisition device, as can the initial state of multiple workpieces.

[0016] In the fifteenth aspect, according to either the thirteenth or fourteenth aspect, the initial state of the plurality of workpieces includes the orientation of the plurality of workpieces, as well as the relative positions and relative connections between the plurality of workpieces. A series of operations are generated based on the orientation of the plurality of workpieces, as well as the relative positions and relative connections between the plurality of workpieces.

[0017] In the sixteenth aspect according to any one of aspects thirteen to fifteen, generating a series of operations based on one or more preconditions and one or more effects of a plurality of meta-actions, a target state and an initial state of a plurality of workpieces includes generating a series of operations in such a way that each precondition of the starting operation in the operation satisfies the initial state of the plurality of workpieces; and each effect of the last operation in the operation satisfies the target state of the plurality of workpieces; and one or more effects of the previous operation in the operation are capable of changing the plurality of workpieces from a previous state to a subsequent state, wherein one or more preconditions of the subsequent operation in the operation satisfy the subsequent state.

[0018] In the seventeenth aspect according to any one of aspects thirteen through sixteen, the method further includes performing a collision check to determine whether a series of operations are feasible, and in response to determining that a series of operations are not feasible, regenerating a new series of operations based on one or more preconditions and one or more effects of a plurality of meta-actions, and target states and initial states of a plurality of artifacts.

[0019] In an eighteenth aspect, a non-transitory computer-readable medium is provided for storing instructions that, when executed by a processor, cause the processor to perform a method for controlling a robotic arm to manipulate a plurality of workpieces. The method may include predefined meta-actions of the robotic arm, each of the meta-actions including one or more preconditions and one or more effects, and acquiring a target state of the plurality of workpieces. The method may further include acquiring an initial state of the plurality of workpieces and generating a series of operations based on one or more preconditions and one or more effects of the meta-actions, the target state of the plurality of workpieces, and the initial state. These operations can be selected from the meta-actions, and performing the series of operations can change the plurality of workpieces from the initial state to the target state.

[0020] In accordance with the nineteenth aspect of the eighteenth aspect, generating a series of operations based on one or more preconditions and one or more effects of a plurality of meta-actions, a target state and an initial state of a plurality of workpieces includes generating a series of operations in such a way that each precondition of the starting operation in the operation satisfies the initial state of the plurality of workpieces; and each effect of the last operation in the operation satisfies the target state of the plurality of workpieces; and one or more effects of the previous operation in the operation are capable of changing the plurality of workpieces from a previous state to a subsequent state, wherein one or more preconditions of the subsequent operation in the operation satisfy the subsequent state.

[0021] In aspect 20, according to either aspect eighteen or nineteen, the state of the prototype includes the relative positions and connections between the prototype's components. The initial state of the multiple components includes the orientation of the multiple components, as well as the relative positions and connections between the multiple components. Based on the relative positions and connections between the prototype's components, the orientation of the multiple components, and the relative positions and connections between the multiple components, a series of operations can also be generated.

[0022] The features and advantages described herein are not exhaustive, and in particular, many additional features and advantages will be apparent to those skilled in the art based on the figures and description. Attached Figure Description

[0023] Figure 1 A system according to an exemplary embodiment of the present disclosure is shown.

[0024] Figure 2A-2B Meta-actions according to exemplary embodiments of this disclosure are shown.

[0025] Figure 3 A view of the workpiece arrangement according to an exemplary embodiment of the present disclosure is shown.

[0026] Figure 4 A method for assembling and disassembling a workpiece according to an exemplary embodiment of the present disclosure is shown.

[0027] Figure 5 A method for creating and storing meta-actions according to exemplary embodiments of the present disclosure is shown.

[0028] Figure 6 A robotic arm according to an exemplary embodiment of the present disclosure is shown. Detailed Implementation

[0029] When a robotic arm is programmed to perform a task, the actions it performs and the sequence of those actions can be extremely important for the overall success of the task. For example, when assembling a component from multiple parts, the order in which the parts are connected can be crucial to ensuring the component functions correctly. Correctly executing a task often requires an operator to manually program specific actions and sequences of actions based on the given task. However, a task is not always performed the same way. For example, changes in the position of a part can alter the specific actions and / or the sequence of actions performed (e.g., speed, location, size, shape, etc.). Therefore, a robotic arm may need to be programmed to perform every potential sequence of actions required for the task in multiple different scenarios, and / or may need to be programmed to account for variables that may change.

[0030] In some cases, the correct execution of a robotic arm may also require an operator. The operator can monitor the conditions in the robotic arm's environment to ensure it is not hindered in completing its task. Since each task may be unique, the operator may need to continuously input different instructions to prevent potential errors in the robotic arm. This can be very difficult and may require many operators to periodically supervise operations and modify the robotic arm's code. Furthermore, the operator may need to manually construct or otherwise program the sequence of actions the robotic arm must perform to add new capabilities (e.g., handling new workpieces) and / or consider new variables that may alter the sequence of actions to be performed (e.g., a new arrangement of workpieces). Adding capabilities in this way can be overly cumbersome, as the operator may need to specify the actions that the robotic arm must perform. Therefore, there is a need for robotic arms to be able to accurately and efficiently construct a series of actions to accomplish new tasks and / or respond to new environmental variables.

[0031] One solution to this problem is to configure the robotic arm to automatically generate a series of actions to perform a specified task. These actions can be selected from predefined meta-actions that the robot is capable of taking. The robotic arm may be able to select a series of meta-actions to generate a set of actions to perform the specified task, without requiring the operator to repeatedly construct the actions the robotic arm will perform when executing the task. The robotic arm can consider multiple variables before selecting meta-actions. For example, the robotic arm can communicate with a vision sensing system that can identify information about the surrounding environment and workpieces that the robotic arm can handle. By considering different variables, such as the information identified by the vision sensing system and information about the current state of the robotic arm, the robotic arm can select a range of effective meta-actions to accomplish the objective.

[0032] Figure 1 A system 100 according to an exemplary embodiment of the present disclosure is illustrated. The system 100 can be configured to analyze information about the environment to select a series of meta-actions to generate a series of operations to perform a specified task. The system 100 includes a computing device 102, which can be configured to process information about one or more artifacts and the surrounding environment before selecting the meta-actions to be performed.

[0033] The computing device 102 includes a processor 104, a memory 105, and a controller 106. The computing device 102 can be configured to perform various functions. The processor 104 and memory 105 can be configured to perform one or more functions of the computing device 102. For example, the memory 105 can store instructions that, when executed by the processor 104, cause the processor 104 to perform one or more operational features of the computing device 102. The memory 105 can also store performance information from the system 100, including environmental information and a history of tasks completed by the system 100. The computing device 102 can be configured to detect and process information about the environment. For example, the computing device 102 and the vision sensing device 116 can constitute a vision sensing system that can identify one or more workpieces 108 in the environment and can identify information about the workpieces 108, such as the position of the workpieces 108 and the relative connections between workpieces 108. For example, the workpieces 108 may have positional information indicating the current orientation and orientation of the workpieces 108 relative to each other. One workpiece can be connected to other workpieces, and the connections between workpieces 108 can also be identified by the vision sensing system, so that the system 100 can know whether the workpieces 108 are connected to each other and the type of connection between the workpieces 108. Other information about the workpieces 108 (e.g., shape, size, color) can be input by the operator of the system 100 and stored in the memory 105, so that the vision sensing system can identify each workpiece 108.

[0034] Information about workpiece 108 can be collected via vision sensing device 116. Vision sensing device 116 may include one or more cameras or other vision sensors. Data from vision sensing device 116 (e.g., images or other visual data) can be analyzed to identify workpiece 108 and related information (e.g., position 110 and / or relative connection 112). Vision sensing device 116 can be communicatively connected to computing device 102 to transmit information about workpiece 108 for processing by processor 104. For example, computing device 102 can receive image data from vision sensing device 116 and analyze that image data to identify workpiece 108 and related information. Vision sensing device 116 can also be attached to robotic arm 114 to accurately determine the position of workpiece 108 relative to robotic arm 114. Furthermore / or, certain relevant information about workpiece 108 can be specified by an operator (e.g., the operator specifies the task that robotic arm 114 is programmed to perform).

[0035] The robotic arm 114 can be configured to perform various tasks based on meta-actions 120 selected by the computing device 102. The meta-actions 120 can be stored in a database 118, which is communicatively connected to the computing device 102 and the robotic arm 114. The database 118 can be configured to receive new meta-actions 120 created by a user or transmitted from other systems or devices. The meta-actions 120 stored in the database 118 can specify all or at least some of the actions that the robotic arm 114 is capable of performing. Each meta-action 120 can include one or more preconditions 122 that are required before the robotic arm 114 performs the meta-action 120. Each meta-action 120 can also include one or more effects 124 that indicate the corresponding changes in the state of the workpiece 108 and the robotic arm 114 resulting from completing the meta-action 120. For example, an exemplary meta-action might be opening a gripper. Its preconditions could include that the gripper is closed, and its effects could include that the object held by the gripper is no longer held by the gripper. In some implementations, the robotic arm 114 may receive commands from the computing device 102. For example, the computing device 102 may provide commands to the robotic arm 114 based on a meta-action 120 selected from a database. In a further implementation, commands may be transmitted from the computing device 102 to the robotic arm 114 via a controller 106, which may be communicatively connected to the robotic arm 114.

[0036] Figure 2A and 2BExemplary meta-actions 200 and 210 that can be used to control the operation of a robotic arm are described. Meta-action 200 can be executed to move a workpiece, while meta-action 210 can be executed to screw into a workpiece. In particular, meta-actions 200 and 210 can be exemplary implementations of meta-action 120 stored in database 118 and can be utilized by computing device 102 to generate a series of operations to control the movement of robotic arm 114. Meta-actions 200 and 210 include preconditions 202, 204, 212, 214, and 216, operations 206, 208, 218, and 220, and effects 222, 224, 226, and 228. For example, meta-action 200 includes two preconditions 202 and 204. Preconditions 202 and 204 can specify one or more conditions or states that must be met before meta-action 200 is executed. In particular, precondition 202 may require that the gripper of the robotic arm 114 is currently grasping the workpiece 108, and precondition 204 may require that the motion path (e.g., the motion path of the workpiece 108) is unobstructed. The computing device 102 may be configured to determine whether preconditions 202 and 204 are met. For example, to determine whether preconditions 202 and 204 are met, the vision sensing device 116 may determine the initial state of the workpiece 108 by collecting and sending, for example, image data of the motion path of the workpiece 108 and / or the gripper of the robotic arm 114 to the computing device 102. As another example, the system 100 may use sensors (e.g., force sensors) in the gripper or the joints of the robotic arm 114 to determine whether precondition 202 is met. As a specific example, if the gripping force exceeds a predetermined threshold, the computing device 102 may determine that the gripper is grasping the workpiece 108. In another embodiment, the computing device 102 may not require data input from the sensors of the vision sensing system 116 or the robotic arm 114. In such an embodiment, computing device 102 may rely on information stored in memory 105. For example, operations performed by system 100 may be stored in memory 105. Computing device 102 may access memory 105 to determine whether preconditions 202, 204 are satisfied based on previously performed operations. In a further implementation, preconditions 202, 204 may be verified based on the effects of earlier meta-actions.

[0037] If the computing device 102 determines that all preconditions 122 are met, then operations 206 and 208 can be allowed to be executed. Meta-action 200 comprises two operations 206 and 208. Operations 206 and 208 may need to be executed to complete meta-action 200. In meta-action 200, the computing device 102 may allow operation 206 to move workpiece 108 to a specific destination. Operation 208 can then be executed to release the gripper of the robotic arm 114. Since workpiece 108 has been moved to the desired destination via operation 206, the computing device 102 can instruct the robotic arm 114 to release workpiece 108 by releasing the gripper. After completing meta-action 200, one or more effects 222 and 228 may occur. In particular, after completing meta-action 200, as shown by effect 222, workpiece 108 is no longer gripped, and as shown by effect 228, workpiece 108 is moved to the specific destination.

[0038] Figure 2BA second exemplary meta-action 210 is described. Meta-action 210 demonstrates operations 217, 218, and 220 that can be performed by robotic arm 114 to screw into workpiece 108 (e.g., to install workpiece 108 using a threaded connection). Meta-action 210 includes three preconditions 212, 214, and 216. All three preconditions 212, 214, and 216 may be required to be satisfied before operations 217, 218, and 220 are performed. In particular, precondition 212 may require that the gripper of robotic arm 114 is currently gripping workpiece 108. Precondition 214 requires that workpiece 108 is positioned on a separate parent workpiece. Precondition 216 requires that the connection type of workpiece 108 (e.g., workpiece 108 gripped by robotic arm 114 and the parent workpiece) is compatible. To determine whether precondition 216 is satisfied, vision sensing device 116 may collect image data of workpiece 108 and send it to computing device 102. Workpiece 108 may have specific connection types. For example, a workpiece may have connection types such as male and / or female connectors (e.g., threaded / screw connections, plug / socket connections, clamps, etc.). Because workpiece 108 may have specific connection types, computing device 102 may need to analyze image data sent from vision sensing device 116 to determine the specific connection type of each workpiece 108. To satisfy precondition 216, computing device 102 can use the connection type information of each workpiece 108 to determine whether the workpiece 108 is compatible. For example, compatible workpiece 108 may include a workpiece 108 with a threaded male connector and a female workpiece with a threaded opening, or a female connector. Examples of incompatible workpiece 108 may include a workpiece 108 with a threaded male connector and a female workpiece with a threaded male connector. The connection types of at least some workpieces 108 may be input by the operator of robotic arm 114 and pre-stored in the memory 105 of computing device 102, rather than using vision sensing device 116.

[0039] If computing device 102 determines that preconditions 212, 214, and 216 for meta-action 210 are met, operations 217, 218, and 220 can be allowed to be executed. Because preconditions 212, 214, and 216 are met, robotic arm 114 can grip workpiece 108, and workpiece 108 can be positioned on a parent workpiece that is aligned and compatible with the relative connection 112. Meta-action 210 comprises three operations 217, 218, and 220. For example, operations 217, 218, and 220 may need to be executed to complete meta-action 210. Specifically, operation 217 may require robotic arm 114 to move workpiece 108 to contact the parent workpiece. For example, computing device 102 may instruct robotic arm 114 to move the gripped workpiece 108 toward the parent workpiece until the connection portion of the two workpieces contacts. Operation 218 may include rotating the gripper of robotic arm 114 until a threshold torque level is reached, screwing workpiece 108 into the parent workpiece. Operation 220 may include releasing workpiece 108 via the gripper of robotic arm 114. After meta-action 210 is completed, one or more effects 224, 226 may occur. In particular, after meta-action 200 is completed, workpiece 108 is not gripped, as shown in effect 222, and workpiece 108 is mounted on the parent workpiece, as shown in effect 226.

[0040] In some implementations, one or two of the preconditions 202, 204, 212, 214, 216 and the effects 222, 224, 226, 228 can be stored as logical expressions. For example, preconditions 202 and 212 can be stored as WorkpieceGrasped, precondition 204 can be stored as Not PathObstructed, and precondition 214 can be stored as Workpiece1 OVER Workpiece2. As another example, effect 228 of meta-action 200 can be stored as WorkpieceMoved, and effect 226 of meta-action 210 can be stored as Workpiece1 ON Workpiece2.

[0041] Figure 3Views 320, 330, and 340 depict a workpiece arrangement 300 according to exemplary embodiments of the present disclosure. View 320 may be a front view of the workpiece arrangement 300, and views 330 and 340 may be side views of the left and right sides of view 320, respectively. The workpiece arrangement 300 includes workpieces 304, 306, 308, and 310 of different sizes and dimensions arranged on a base 302. Specifically, workpieces 304 and 306 are directly positioned on the base 302 (e.g., the base 302 is the "parent" of workpieces 304 and 306). Workpiece 308 is positioned on top of workpiece 306 (e.g., workpiece 306 is the parent of workpiece 308), and workpiece 310 is positioned on workpieces 304 and 308 (e.g., workpieces 304 and 308 are both the parent of workpiece 310). Workpieces 304, 306, 308, 310 and base 302 may be assembled or otherwise attached together (e.g., by snap-fit, insert mount, etc.), or may be positioned without attachment as described. In some cases, workpiece arrangement 300 may be represented as a logical expression, such as (workpiece 306 on base 302) && (workpiece 304 on base 302) && (workpiece 308 on workpiece 306) && (workpiece 310 on workpiece 308) && (workpiece 310 on workpiece 304).

[0042] Figure 4 A method 420 according to an exemplary embodiment of system 100 is illustrated. Method 420 can be performed to control a robotic arm 114 to perform a desired task. For example, method 420 can be performed to accomplish a goal or process relating to assembling or disassembling a workpiece through predefined meta-actions. Method 420 can be implemented on a computer system, such as system 100. The method can also be implemented by a set of instructions stored on a computer-readable medium, which, when executed by a processor, cause the computer system to perform method 420. For example, all or part of method 420 can be implemented by processor 104 and memory 105. Although the following examples are for illustrative purposes only... Figure 4 The flowchart shown is described, but many other methods can also be used. Figure 4 The execution methods of related behaviors. For example, the order of some blocks can be changed, some blocks can be combined with other blocks, one or more blocks can be repeated, and some of the blocks described can be optional.

[0043] Method 420 may begin with a predefined meta-action 120 (block 422). Meta-action 120 may be stored in the memory 105 of system 100. Meta-action 120 may be accessed by computing device 102 to complete a target or desired process. Meta-action 120 may correspond to an operation to be performed by system 100. As described above, meta-action 120 may include preconditions 122 and / or effects 124. The creation of meta-actions will be discussed in more detail below in conjunction with method 500.

[0044] Then, system 100 can acquire the target state of workpiece 108 (block 424). For example, the target or desired process may include the movement, reordering, or alteration of one or more workpieces 108 to the target state. The target state may include the position and connection of one or more workpieces 108 relative to other workpieces 108. The target state may be directly input by the operator. For example, the operator may input that workpieces 108 need to be moved to a specific orientation relative to each other and / or positioned in a specific direction relative to each other. Alternatively, the target state of the workpieces can be acquired using a vision sensing device 116. For example, system 100 may present a prototype or specific arrangement of workpieces 108 that system 100 must replicate. Computing device 102 may instruct vision sensing device 116 to acquire image data of workpieces 108, including position 110 and / or relative connection 112. Vision sensing device 116 may send the image data to computing device 102 for processing by processor 104. Processor 104 may determine the target state of each workpiece 108. Information about the target state may be stored in memory 105. In a specific example, the target state of workpiece 108 can be workpiece arrangement 300.

[0045] Data regarding the initial state of workpiece 108 can then be determined (block 426). For example, after acquiring the target state of workpiece 108, system 100 can present workpiece 108 in its initial state. Although the target may be to move or position workpiece 108 in a specific manner corresponding to the target state, workpiece 108 may begin in a different orientation and position 110 and / or direction than the target state. Computing device 102 can receive data regarding the initial state of workpiece 108 using vision sensing device 116. For example, vision sensing device 116 can acquire image data of the current setup of workpiece 108, representing the initial state of workpiece 108. Vision sensing device 116 can send the image data of the initial state to computing device 102. Computing device 102 can process the image data using processor 104. Alternatively, the initial state of workpiece 108 can be input by an operator. In the example where the target state is workpiece arrangement 300, the initial state may indicate that workpieces 304, 306, 308, and 310 are all connected to base 302 (e.g., at orientations separate from each other). In some cases, the initial state may include the orientation of workpieces 108 relative to each other. Continuing with the previous example, the initial state may indicate that workpieces 310 and 308 are perpendicular to workpiece 306 (instead of being parallel as in workpiece arrangement 300), and workpiece 304 is parallel to workpiece 306 (instead of being perpendicular as in workpiece arrangement 300).

[0046] Based on the initial and target states of workpiece 108, system 100 can generate a sequence of corresponding operations to complete the objective and reach the target state of workpiece 108 (block 428). This series of operations can be further based on meta-actions 120, preconditions 122, and effects 124. For example, computing device 102 can select meta-actions 120 based on the target state and initial state of workpiece 108. In particular, computing device 102 can select corresponding operations such that the initial state satisfies at least the precondition 122 of a first operation, and that the final result of the effect 124 of the series of operations satisfies the target state of workpiece 108. In some implementations, completing the effect 124 of the first operation may result in the satisfaction of the precondition 122 of a subsequent second operation. In particular, completing the first operation can transition workpiece 108 from a previous state to a subsequent state, where the previous state does not satisfy at least one of the preconditions 122 of the second operation, while the subsequent state satisfies each of the preconditions 122 of the second operation. In some implementations, a graph traversal algorithm, such as the A* search algorithm or a similar algorithm, can be used at least in part to determine the series of operations. In particular, this search can be performed on the database 118 of the meta-action 120 to determine a series of operations. Continuing with the target state of workpiece 108, which is an example of workpiece arrangement 300, the series of operations may include (1) picking up workpiece 304 and placing it on the base 302 near workpiece 306, (2) picking up workpiece 308 and placing it on workpiece 306, and (3) picking up workpiece 310 and placing it on workpieces 304 and 308.

[0047] The computing device 102 can perform the series of operations (block 430). For example, the computing device 102 can perform the operations in the order specified by the series of operations (e.g., by performing one or more motion steps of a meta-action included in the series of operations). In some cases, before performing the series of operations, the computing device 102 can perform a collision check analysis on the feasibility of successfully completing the series of operations. For example, the computing device 102 can analyze the series of operations to determine whether a particular workpiece is inaccessible (e.g., because other workpieces or equipment prevent the robotic arm 114 from accessing workpiece 108). As another example, the computing device 102 can analyze the sequence of operations to determine whether performing a particular operation (e.g., moving workpiece 108) would cause workpiece 108 to collide with other equipment or workpieces. If the computing device 102 determines that a collision is possible, or that the series of operations is otherwise infeasible, a new series of operations can be generated (e.g., by repeating block 428).

[0048] Figure 5A method 500 according to an exemplary embodiment of system 100 is illustrated. Method 500 can be performed to program and store meta-actions 508. For example, method 500 can be performed to store meta-actions in database 118 for use in method 420 (e.g., during block 422). Method 500 can be implemented on a computer system, such as system 100. Method 500 can also be implemented by a set of instructions stored on a computer-readable medium that, when executed by a processor, cause the computer system to perform method 500. For example, all or part of method 500 can be implemented by processor 104 and memory 105.

[0049] The method 500 may begin by receiving one or more motion steps 502. For example, computing device 102 may receive one or more motion steps 502 to be performed when executing meta-action 508. Motion steps 502 may specify actions and / or movements performed by robotic arm 114. The method 500 may also include receiving one or more preconditions 504 and one or more effects 506. Preconditions 504 and effects 506 may be received in conjunction with motion steps 502. For example, preconditions 504, effects 506, and / or motion steps 502 may be received from a user. As a specific example, a user may use computing device 102 or another computing device to create and / or input motion steps 502, preconditions 504, and effects 506.

[0050] Meta-action 508 can then be created. For example, computing device 102 can create meta-action 508 including motion step 502, prerequisite 504, and effect 506. Meta-action 508 can then be stored in database 510, which can be an exemplary implementation of database 118. For example, computing device 102 can transmit meta-action 508 to database 510 for storage. Alternatively, meta-action 508 can be stored in database 510 implemented within memory 105 of computing device 102.

[0051] Figure 6An exemplary robotic arm 600 is described. The robotic arm 600 can be communicatively connected to system 100 and can be an exemplary implementation of robot 114. The robotic arm 600 may include multiple robotic arm segments 602 and an end effector 604. The end effector 604 may include a gripper in which one or more sensors (e.g., force and / or torque sensors) may be mounted. For example, the end effector 604 may be a robotic gripper that utilizes sensors to detect changes in force and / or torque, as described above in relation to operations 217, 218. In other examples, the end effector 604 may be any other suitable tool utilizing any suitable sensors described in this disclosure. In some embodiments, sensors for detecting changes in force and / or torque may alternatively or additionally be mounted in the joint actuators of the robotic arm segments 602.

[0052] All the methods and processes described in this disclosure can be implemented using one or more computer programs or components. These components can be provided as a set of computer instructions to any conventional computer-readable or machine-readable medium, including volatile and non-volatile memories such as RAM, ROM, flash memory, magnetic disks or optical disks, optical storage, or other storage media. The instructions can be provided in the form of software or firmware and can be implemented, wholly or partially, in hardware components such as ASICs, FPGAs, DSPs, or any other similar devices. The instructions can be configured to be executed by one or more processors, which, when executed, perform or facilitate the execution of all or part of the disclosed methods and processes.

[0053] It should be understood that various changes and modifications to the examples described herein will be apparent to those skilled in the art. Such changes and modifications can be made without departing from the spirit and scope of the subject matter and without diminishing its intended advantages. Therefore, the appended claims cover these changes and modifications.

Claims

1. A method for controlling a robotic arm to manipulate multiple workpieces, comprising: Multiple meta-movements of the robotic arm are predefined, each of which includes one or more preconditions and one or more effects; Obtain the target state of the plurality of workpieces; Obtain the initial state of the plurality of workpieces, the initial state of the plurality of workpieces including the relative position and relative connection between the plurality of workpieces; as well as A series of operations are generated based on one or more preconditions and one or more effects of the plurality of meta-actions, the target state and the initial state of the plurality of workpieces; The operation is selected from the plurality of meta-actions; The series of operations is generated based on the relative positions and relative connections between the plurality of workpieces, and the execution of the series of operations can change the plurality of workpieces from the initial state to the target state.

2. The method according to claim 1, wherein obtaining the target state of the plurality of workpieces includes: Identify the state of the prototype corresponding to the target state of the plurality of workpieces.

3. The method according to claim 2, wherein the state of the prototype includes the relative position and relative connection between the workpieces of the prototype.

4. The method according to claim 2, wherein the state of the prototype is identified by a visual acquisition device.

5. The method according to claim 1, wherein acquiring the initial state of the plurality of workpieces includes acquiring the initial state of the plurality of workpieces using a vision acquisition device.

6. The method according to claim 1, wherein the initial state of the plurality of workpieces includes the orientation of the plurality of workpieces, and the series of operations are generated based on the orientation of the plurality of workpieces.

7. The method according to claim 1, wherein, The process of generating the series of operations based on one or more preconditions and one or more effects of the plurality of meta-actions, the target state and the initial state of the plurality of workpieces includes: The series of operations is generated in such a manner that each precondition of the start operation in the operation satisfies the initial state of the plurality of workpieces; and each effect of the last operation in the operation satisfies the target state of the plurality of workpieces.

8. The method according to claim 7, wherein, The process of generating the series of operations based on the one or more preconditions and one or more effects of the plurality of meta-actions, the target state and the initial state of the plurality of workpieces further includes: The series of operations is generated in such a way that one or more effects of a previous operation in the operation can change the plurality of workpieces from a previous state to a subsequent state, wherein one or more preconditions of the subsequent operation in the operation satisfy the subsequent state.

9. The method of claim 7, wherein the series of operations is generated by using a graph traversal algorithm.

10. The method according to claim 1, further comprising: Perform a collision check to determine if the series of operations are feasible; as well as In response to determining that the series of operations is not feasible, a new series of operations is generated based on the one or more preconditions and one or more effects of the plurality of meta-actions, the target state and the initial state of the plurality of workpieces.

11. The method according to claim 1, further comprising: Perform a collision check to determine if the series of operations are feasible; as well as In response to determining that the series of operations is feasible, the series of operations are performed.

12. A system for controlling a robotic arm, comprising: processor; as well as A memory storing instructions that, when executed by the processor, cause the processor to perform any one of claims 1 to 11 for controlling the robotic arm to manipulate a plurality of workpieces.

13. A non-transitory computer-readable medium storing instructions that, when executed by a processor, cause the processor to perform any one of claims 1 to 11, a method for controlling a robotic arm to manipulate a plurality of workpieces.

Images