Systems and methods for natural task allocation to one or more robots

Abstract

Embodiments of the present disclosure relate to robotic systems and methods that may include one or more robots. The system may include a robotic system having a maximum number of degrees of freedom. The system may also include a graphical user interface configured to receive a natural robotic task having at least one natural workpiece constraint associated with the natural robotic task. The system may also include a processor configured to identify a minimum number of degrees of freedom required to perform the natural robotic task, wherein the minimum number of degrees of freedom is based at least in part on at least one natural workpiece constraint associated with the natural robotic task.

Description

[0001] Cross-references

[0002] This application claims the benefit of U.S. Non-Provisional Application Serial No. 16 / 025,544, filed on July 2, 2018. The entire contents of which are incorporated herein by reference.

[0003] This invention was supported by Air Force SBIR Contract FA8650-17-P-5043. The U.S. Government has certain rights in this invention. Technical Field

[0004] The present invention relates generally to robots, and more particularly to systems and methods for under-constraining a robot or robotic system. Background Art

[0005] Assigning tasks to robots is difficult, and this difficulty is exacerbated by assigning tasks to robots in an unnatural, over-constrained way. Natural task assignment is a comprehensive approach to easily specify robot tasks without over-constraining or under-constraining.

[0006] For example, rather than modeling the exact position where the hand is placed on the bar, a gymnast models the bar as having one degree of freedom when it is placed. Similarly, tasks like welding and painting have additional degrees of freedom in how the electrode or nozzle rotates around its main axis, and the operator who assigns tasks to the robot benefits by working with these natural constraints. Similarly, a part to be gripped by an industrial robot may have symmetries that allow it to be gripped in a variety of ways. Natural task assignment specifies the constraints on the robot in their smallest and least restrictive way. Summary of the invention

[0007] In one or more embodiments of the present disclosure, a system is provided. The system may include a robotic system having a maximum number of degrees of freedom. The system may also include a graphical user interface configured to receive a natural robotic task having at least one natural workpiece constraint associated with the natural robotic task. The system may also include a processor configured to identify a minimum number of degrees of freedom required to perform the natural robotic task, wherein the minimum number of degrees of freedom is based at least in part on at least one natural workpiece constraint associated with the natural robotic task.

[0008] One or more of the following features may be included. The at least one natural workpiece constraint associated with the natural robotic task may be selected from the group consisting of: an orientation constraint, an assignment to a curve or surface, a distance constraint, a joint position, a combination of linear and nonlinear functions of joint positions, a workpiece geometry, a workpiece feature, a center of mass, and linear and angular momentum. The system may include a scanner configured to identify at least one natural workpiece constraint associated with the natural robotic task. The system may include a simulator configured to determine a feasible or optimized implementation of the natural robotic task. In some embodiments, receiving the natural robotic task may be received in conjunction with a learning demonstration. The minimum number of degrees of freedom may be less than the maximum number of degrees of freedom. The identification of at least one natural workpiece constraint or the minimum number of degrees of freedom associated with the natural robotic task may be specified by a user. The identification of at least one natural workpiece constraint or the minimum number of degrees of freedom associated with the natural robotic task may be automatically determined in response to scanner identification. The identification of at least one natural workpiece constraint or the minimum number of degrees of freedom associated with the natural robotic task may be updated in real time during operation of the robotic system. The minimum number of degrees of freedom may be based at least in part on sensor data. The identification of at least one natural workpiece constraint or a minimum number of degrees of freedom associated with the natural robotic task may be updated using a communication channel before, during, or after operation of the robotic system. The graphical user interface may be at least one of a two-dimensional interface or a three-dimensional interface. The minimum number of degrees of freedom may be identified without reference to a robotic system frame used to control the robotic system. The minimum number of degrees of freedom may be identified by reference to only one or more features and a geometric shape of a workpiece. The processor may be configured to control the robotic system to perform the natural robotic task by exploring additional degrees of freedom, wherein the additional degrees of freedom are based at least in part on a difference between the maximum number of degrees of freedom and the minimum number of degrees of freedom. Exploring the additional degrees of freedom may include optimizing a criterion for using the additional degrees of freedom to solve a gradient-based problem that attempts to minimize the criterion. The processor and the scanner may be configured to define a workspace before performing the robotic task. The natural robotic task may include at least one natural environment constraint associated with the natural robotic task. The at least one natural workpiece constraint may include one or more geometric features associated with a particular workpiece. The at least one natural environment constraint includes: shaft hole insertion, line following, camera pointing, grasping a symmetrical object, placing an object on a flat surface, contacting a flat object with a suction cup, manipulating a mobile base, or controlling the center of mass of the mobile robot to produce stability.

[0009] In another embodiment of the present disclosure, a method is provided. The method may include providing a robotic system having a maximum number of degrees of freedom associated therewith. The method may also include receiving a natural robotic task at a graphical user interface, the natural robotic task having at least one natural workpiece constraint associated with the natural robotic task. The method may also include using at least one processor to identify a minimum number of degrees of freedom required to perform the natural robotic task, wherein the minimum number of degrees of freedom is based at least in part on at least one natural workpiece constraint associated with the natural robotic task.

[0010] One or more of the following features may be included. At least one natural workpiece constraint associated with the natural robotic task may be selected from the group consisting of: orientation constraints, assignments to curves or surfaces, distance constraints, joint positions, linear and nonlinear function combinations of joint positions, workpiece geometry, workpiece features, center of mass, and linear and angular momentum. The method may include identifying at least one natural workpiece constraint associated with the natural robotic task based at least in part on a scanner. The method may also include determining a feasible or optimized implementation of the natural robotic task based at least in part on a simulation. In some embodiments, receiving the natural robotic task may be received in conjunction with a learning demonstration. The minimum number of degrees of freedom may be less than the maximum number of degrees of freedom. The identification of at least one natural workpiece constraint or the minimum number of degrees of freedom associated with the natural robotic task may be specified by a user. The identification of at least one natural workpiece constraint or the minimum number of degrees of freedom associated with the natural robotic task may be automatically determined in response to scanner identification. The identification of at least one natural workpiece constraint or the minimum number of degrees of freedom associated with the natural robotic task may be updated in real time during operation of the robotic system. The minimum number of degrees of freedom may be based at least in part on sensor data. The identification of at least one natural workpiece constraint or a minimum number of degrees of freedom associated with the natural robotic task may be updated using a communication channel before, during, or after operation of the robotic system. The graphical user interface may be at least one of a two-dimensional interface or a three-dimensional interface. The minimum number of degrees of freedom may be identified without reference to a robotic method coordinate system used to control the robotic system. The minimum number of degrees of freedom may be identified by reference to only one or more features and geometries of the workpiece. The method may also include controlling the robotic system to perform the natural robotic task by exploring additional degrees of freedom using the at least one processor, wherein the additional degrees of freedom are based at least in part on a difference between the maximum number of degrees of freedom and the minimum number of degrees of freedom. The method may also include defining a workspace before performing the robotic task, wherein the definition is based at least in part on data from a processor and a scanner. The natural robotic task may include at least one natural environment constraint associated with the natural robotic task. The at least one natural workpiece constraint may include a geometric feature associated with a particular workpiece. The at least one natural environment constraint includes: shaft hole insertion, line following, camera pointing, grasping a symmetrical object, placing an object on a flat surface, contacting a flat object with a suction cup, manipulating a mobile base, or controlling the center of mass of a mobile robot to produce stability. The motion associated with the robotic system may be calculated, at least in part, using a manipulator Jacobian. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In order to better understand the nature and purpose of the present disclosure, reference is made to the following detailed description in conjunction with the following drawings, in which:

[0012] Figure 1 is a block diagram of a natural task allocation robotic system according to an embodiment of the present disclosure;

[0013] Figure 2 is a graphical user interface illustrating multiple degrees of freedom of a natural task allocation robotic system according to an embodiment of the present disclosure;

[0014] Figure 3 is a block diagram of a natural task allocation robotic system according to an embodiment of the present invention;

[0015] Figure 4 is a block diagram of a speed control method for a natural task allocation robot system according to an embodiment of the present disclosure;

[0016] Figure 5 is a flow chart of a natural task assignment robotic method according to an embodiment of the present disclosure;

[0017] Figure 6 is a graphical user interface of a natural task assignment robotic system according to an embodiment of the present disclosure;

[0018] Figure 7 is a graphical user interface of a natural task assignment robotic system according to an embodiment of the present disclosure;

[0019] Figure 8 is a graphical user interface of a natural task assignment robotic system according to an embodiment of the present disclosure;

[0020] Fig. 9 is a graphical user interface of a natural task assignment robotic system according to an embodiment of the present disclosure;

[0021] Fig.10 is a graphical user interface of a natural task assignment robotic system according to an embodiment of the present disclosure;

[0022] Fig.11 is a graphical user interface of a natural task assignment robotic system according to an embodiment of the present disclosure;

[0023] Fig.12 is a graphical user interface of a natural task assignment robotic system according to an embodiment of the present disclosure;

[0024] Fig.13 is a graphical user interface of a natural task allocation robotic system in a welding application according to an embodiment of the present disclosure;

[0025] Fig.14 is a graphical user interface of a natural task allocation robotic system in a welding application according to an embodiment of the present disclosure;

[0026] Fig.15 is a graphical user interface of a natural task allocation robotic system in a welding application according to an embodiment of the present disclosure;

[0027] Fig.16 is a graphical user interface of a natural task allocation robotic system in a welding application according to an embodiment of the present disclosure;

[0028] Fig.17 is a graphical user interface of a natural task allocation robotic system in a welding application according to an embodiment of the present disclosure;

[0029] Fig.18 is a graphical user interface of a natural task allocation robotic system in a welding application according to an embodiment of the present disclosure;

[0030] Fig.19 is a graphical user interface of a natural task allocation robotic system in a welding application according to an embodiment of the present disclosure;

[0031] Fig. 20 is a graphical user interface of a natural task allocation robotic system in a robotic assembly application according to an embodiment of the present disclosure;

[0032] Fig.21 is a graphical user interface of a natural task allocation robotic system in a robotic assembly application according to an embodiment of the present disclosure;

[0033] Fig. 22 is a graphical user interface of a natural task allocation robotic system in a robotic assembly application according to an embodiment of the present disclosure; and

[0034] Fig.23 is a graphical user interface of a natural task allocation robotic system in a robotic assembly application according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0035] Implementations of the subject application may include concepts from U.S. Pat. No. 6,757,587, U.S. Pat. No. 7,680,300, U.S. Pat. No. 8,301,421, U.S. Pat. No. 8,408,918, U.S. Pat. No. 8,428,781, U.S. Pat. No. 9,357,708, U.S. Publication No. 2015 / 0199458, U.S. Publication No. 2016 / 0321381, and U.S. Publication No. 2018 / 0060459, the entire contents of each of which are incorporated herein by reference in their entirety.

[0036] See now Figure 1, an embodiment of a natural task allocation robot system 100 is provided. The system 100 may include multiple components, some of which may be designed for specific applications and / or tasks. The first component of the system may include a software system 102 for adding new processes to a database 104. Once built, the database 104 can be reused by an operator on-site or remotely. The operator can use a graphical user interface 106 to select elements from the database 104 for execution by the control software 108, such as Figure 1 As shown. Procedures for specific applications and / or tasks (e.g., welding, robotic assembly, etc.) can be added to the database 104 by an expert offline. The database 104 can be used online with a graphical user interface 106 and task assignment software to develop each process for each task. The software modules may include, but are not limited to, training, task assignment, and executing specific tasks. All of these can be used to control the operation of the robotic hardware 110. The robotic hardware 110 can respond to controls received from the control software 108, however, it should be noted that the robotic hardware itself may be limited by its own maximum number of degrees of freedom, as discussed in further detail below.

[0037] In robotic systems, the phrase "degrees of freedom" can refer to specific, defined patterns in which a mechanical device or system can move. The number of degrees of freedom can be equal to the total number of independent displacements or aspects of motion. For example, a six degree of freedom ("6DOF") scenario can refer to the freedom of a rigid body to move in three-dimensional space. Specifically, the body can freely change position, such as translating forward / reverse (forward), up / down (vertical), and left / right (lateral) in three perpendicular axes, while changing orientation by rotating about three perpendicular axes, commonly referred to as yaw (normal axis), pitch (lateral axis), and roll (longitudinal axis). In contrast, placing points in space can correspond to three degrees of freedom, specifying the distance between two points on different links is one degree of freedom, etc.

[0038] In some embodiments, as used herein, the phrase "robotic system" may include a system of one, two, and / or any number of robots. Thus, the total robotic system DOF may refer to the sum of the DOF on each of the individual robots. This may include one DOF for each single-axis joint and six DOF for the freely moving base. For example, for a robotic system including two robots, one with 6 DOF and the other with 5 DOF, the available total robotic system degrees of freedom may be 11 DOF.

[0039] Embodiments included herein may be configured to explore additional numbers of degrees of freedom of one or more robots to naturally perform robotic tasks. In this example, the entire robotic system may include a maximum number of degrees of freedom DOF ("N") associated therewith, and may also include a minimum number of DOF ("M") subtracted therefrom.

[0040] For example, see now Figure 2 , provides a 3D rendering from a graphical user interface 200 that depicts a welding tool with five degrees of freedom compared to a welding tool with six degrees of freedom in full 3D rotation and orientation. The difference between the two is that the five degrees of freedom can be obtained by relaxing the rotation about one axis. This particular model is adjusted to optimize the scanning and welding process. This involves configuring a bounding volume around the scanner tool, and aligning the system with the base principal coordinate system so that the tool path and point cloud appear in the correct location. The tool offset is also configured based on how the tool path is created from the point cloud. The path was created along the bottom of the seam, so the offset from the top is configured so that there is no collision. As shown in FIG. Figure 2 As shown, the arm can be constrained using two different constraint setups, each using a different tool offset, one for scanning the part and one for welding. The degrees of freedom setup for scanning uses a six degrees of freedom coordinate system, and the welding setup uses a five degrees of freedom coordinate system that allows the tip of the spray gun to rotate freely about the tip. This allows the part to be scanned first, and then the degrees of freedom of the weld path are relaxed, which is more difficult for a robot to achieve given its workspace envelope.

[0041] In some embodiments, see now Figure 3 , provides an embodiment of a flowchart depicting a plurality of operations, which are consistent with a natural task allocation robot process. The natural task allocation process 300 may include defining 302 robots, tools, parts, and robot environments. The process 300 may also include defining 304 one or more constraints based on tasks, tool geometries, part geometries, etc. The process 300 may also include using the constraints to define 306 one or more task motions. The process 300 may be configured to define the minimum number of constraints naturally by analyzing geometry (e.g., tool geometry), such as by finding symmetric or nearly symmetric axes. In this way, the naturalness of the constraints and the ability to explore additional degrees of freedom in robot motion allow the robot to be naturally controlled. Natural control may be based at least in part on both natural constraints and the natural optimization of these constraints, as will be discussed in further detail below. In operation, once naturally optimized, the Jacobian-based velocity control method described below may be used to perform 308 tasks.

[0042] In some embodiments, see now Figure 4 , a block diagram 400 consistent with an embodiment of a natural task allocation robot system of the present disclosure is provided. Control of the end effector of a robotic arm in the system may be performed using a plurality of components (e.g., using Energid's The end effector can be controlled by a program (eg, a program that can be used to control the end effector) using a control system 400. These components may include, but are not limited to, an external position controller 402 and an internal velocity controller 404. The combined control system 400 allows the program to "fly" the end effector throughout the task space by specifying the desired placement, and the joint velocities and positions can be automatically calculated. This allows the operator to focus on the task without being constrained, rather than focusing on the low-level control of the robot.

[0043] In some embodiments, the core velocity architecture may be based on the manipulator Jacobian equation:

[0044]

[0045] Here V is an m-length vector representation of the motion of the hand or hands (typically some combination of linear and angular velocities referenced to a point of a component rigidly attached to the manipulator); q is an n-length vector of joint positions (where q is its time derivative); and J is an m×n manipulator Jacobian, a function of q. (For a space arm with a single end effector, V is typically a coordinate system velocity with three linear components and three angular components. As used herein, it assumes a larger meaning, including a cascade of points, coordinate systems, or other motions of multiple end effectors. The Jacobian J(q) is a matrix that makes () true for all possible values ​​of q. Note that V can represent a cascade of values ​​for multiple end effectors, thereby achieving coordinated motion of multiple points on the manipulator.

[0046] The velocity control problem is as follows: given a desired tool motion V, what are the joint velocities q that best achieve that motion? To answer this, the architecture is based in part on an approach that uses a scalar α, a matrix function W(q), and a scalar function f(q) to solve for q given V using the following formula:

[0047]

[0048] in is the gradient of f, and N J is the set of n×(n×m) vectors spanning the null space of J. That is, JN J =0, and N J Has rank (n×m). and N J It is usually a function of q. By changing the values ​​of α, W and f, many new and most deterministic speed control techniques can be achieved.

[0049] However, embodiments of the present disclosure go beyond the above formula to form a more general framework. Instead of sticking with the gradient of the function, a general column vector F(q) is used. Not all vector functions are gradients. This minor but important modification yields the following:

[0050]

[0051] This achieves the desired V while minimizing:

[0052]

[0053] The parameters α, W and F can be defined using XML to give many different types of speed control.

[0054] In some embodiments, the position control system 402 can be built on the velocity control system 404 to provide robust position control to the end effector. A position control module can be provided as an interface to the position control system 402 to enable the addition of new position control algorithms. The position control system 402 can provide basic position control for the end effector by building on the velocity control system 404. Taking into account the joint velocities from the velocity control system 404, the position control system 402 can use Euler integration to calculate the joint positions. Once the joint positions are calculated, the end effector position is known. The control system can check for joint limit exceeds and collisions as it iterates forward in time. If a singularity is detected, it can also zero the joint velocity commands.

[0055] For example, a key component of the robot control, which may involve multiple mechanisms, is to avoid collisions with the environment. To this end, a collision avoidance algorithm may be used based on a defined bounding volume for each shape of interest in the simulation. The algorithm employs a gradient-based approach that attempts to minimize a function of the following general form:

[0056]

[0057] Here, N is the number of links in the manipulator, B is the number of obstacles in the environment, P is a user-defined exponent, and F(i,j) is a measure of the proximity of i's bounding volume to the bounding volume of link j. When the distance is greater than a user-specified threshold, F(i,j) is zero, and within the threshold it is simply a scalar constant times the minimum distance required to move one bounding volume to bring it beyond the threshold distance from another bounding volume.

[0058] In some embodiments, this function can be used with finite differences to calculate its gradient, which can then be used as a vector parameter to the core control system to drive the manipulator in a direction to avoid collisions. Here, obstacles in the environment can be explicitly defined, such as other manipulators broadcasting the state of the obstacle, or they can be determined in real time from vision or other sensing mechanisms.

[0059] In some embodiments, the control methods described above may be used, at least in part, to allow a controller to associate the overall robotic system DOF (e.g., the system's available DOF, "N") with the minimum number of degrees of freedom ("M") required that is identified based solely on workpiece constraints. In other words, the robotic systems and methods described herein may only require workpiece data in order to perform natural robotic tasks. In some cases, the DOF may also be based on other factors, including but not limited to other factors or constraints, however this information may not be necessary. Embodiments of the natural task allocation robotic systems and methods provided herein may be used to provide better control and / or improved performance of robotic systems, more feasible and / or optimized solutions to robotic tasks, etc.

[0060] In some embodiments, see now Figure 5 , an embodiment is provided depicting a flow chart having a plurality of operations consistent with a natural task assignment robot process 500. The process 500 may include providing 502 a first robot of one or more robots, the first robot having a maximum number of degrees of freedom associated therewith. The process 500 may also include receiving (504) a natural robot task at a graphical user interface, the natural robot task having at least one natural workpiece constraint associated with the natural robot task. The process 500 may also include identifying (506) using at least one processor a minimum number of degrees of freedom required to perform the natural robot task, wherein the minimum number of degrees of freedom is based at least in part on at least one natural workpiece constraint associated with the natural robot task.

[0061] The phrase "maximum number of degrees of freedom" may refer to a specific, defined pattern in which a mechanism, robot, and / or system can move a maximum number of specific, defined patterns. The phrase "minimum number of degrees of freedom" may refer to a specific, defined pattern in which a mechanism, robot, and / or system moves a minimum number of specific, defined patterns in which a mechanism, robot, and / or system moves to naturally perform a task. The phrase "natural robotic task" may refer to a task that can be performed using a reduced and / or simplified amount of motion. In some cases, specifying a natural robotic task may be contrasted with a mathematically intensive way of specifying a task that may require a user to specify in the coordinates / coordinate system of a robotic system. A natural robotic task may be specified / presented to a robotic system using one or more robots or tools, and a minimum number of degrees of freedom may be determined based at least in part on the natural robotic task. The minimum number of degrees of freedom may be determined by analyzing the tool, robot, and / or workpiece geometry. It may be specified using demonstrations, teaching learning robotics techniques, using a GUI, and the like.

[0062] In some embodiments, the naturalness of a task can refer to the degree of agreement between 1) the commanded robot task and 2) the actual desired thing to be accomplished. Some examples are provided below:

[0063] Desired Task (1): Place the cup on the table.

[0064] Unnatural: The cup is placed at x=30 cm, y=50 cm from the corner, with the handle pointing to the right.

[0065] Natural: Place the cup anywhere within the boundaries of the table with the handle pointing in any direction.

[0066] Desired task (2): Make the camera on the robot point to the 3D point x=1, y=2, z=2 in space.

[0067] Unnatural: Place the camera at x=0, y=0.2, z=0.3 and point it at x=1, y=2, z=2.

[0068] Unnatural: Let the camera point from any position to {1, 2, 2}, where upward in the image is consistent with upward in the real world.

[0069] Natural: With no constraints on the camera position, point the camera to {1, 2, 2} in any orientation.

[0070] Desired Task (3): Spray the part using a nozzle from 1 m away and pointing the nozzle directly at the center of the part.

[0071] Unnatural: Spraying from 1m away with the nozzle pointed towards the part and specifically oriented around the axis of the part.

[0072] Natural: sprays from a distance of 1m with no special orientation of the nozzle other than pointing it towards the part.

[0073] It should be noted that while these unnatural constraints may appear strange in how they overly restrict the robot's motion, they are in fact how these types of tasks are typically performed. Embodiments of the natural robotic task allocation system included herein allow for the use and solving of natural representations.

[0074] In some embodiments, the phrase "natural workpiece constraints" may refer to geometric features, material properties, and / or any other aspects associated with a particular workpiece. The phrase "natural environment constraints" may refer to geometric features, obstacles, material properties, and / or any other aspects associated with the environment that may be associated with a natural robotic task.

[0075] In some embodiments, workpiece constraints, environmental constraints, and manipulator constraints may form a closed set in the method, and multiple natural constraints may be combined to define comprehensive task allocation motion. Some of these restrictions may include, but are not limited to, direction constraints, orientation constraints, allocation to curves or surfaces, distance constraints, joint positions, linear and nonlinear function combinations of joint positions, center of mass, and linear and angular momentum. These constraints may vary depending on the robot configuration. Tool offsets, static objects or obstacles, and dynamic objects or obstacles may be used to define and configure the constraints. In some embodiments, the identification of the minimum number of degrees of freedom may be based on the workpiece constraint geometry and / or its features, some of which may include, but are not limited to, holes, shapes, materials, sizes, etc.

[0076] In some embodiments, one or more constraints may be edited by a user using a visual editor such as Figure 6-Figure 12, to create a complex combination of time-varying constraints to best implement real-world tasks. It should be noted that some or all of these constraints may change during the execution of the robot task or operation, even in time steps, and may be combined with optimizations such as collision avoidance and joint-limit avoidance. In this way, the number and type of constraints can be updated dynamically in real time. For example, in some embodiments, some or all constraints can be automatically determined and / or identified using a scanner, which can be configured to scan the robot environment to identify, allow necessary processing and subsequent determination. Various sensors and devices can be used to obtain this data, some of which may include but are not limited to data from laser radar, stereo cameras, color cameras, multi-spectral cameras, etc. In addition and / or alternatively, some or all constraints can be remotely updated via a communication channel (e.g., radio, laser or network communication, etc.). Constraints are set using a 2D graphical user interface or a 3D interface (e.g., augmented reality, etc.). The constraint can be a "less than or equal to" type, not just "equal to". Thus, for example, the natural tasking robotics systems included herein may allow a user to constrain points above the floor rather than just on it.

[0077] Thus, embodiments of the natural tasking robotics systems described herein may include one or more robots with a maximum number of degrees of freedom "N". In some embodiments, the system may allow a user to naturally specify tasks (e.g., by way of example, to teach a robotic system, to perform robot "seeing" and learning using one or more sensors, to perform demonstrations using a GUI such as a teach pendant, etc.) by indicating only what needs to be done, such as inserting a peg into a hole (using five degrees of freedom under constraints), following a line with the tip of a welding tool (using three degrees of freedom under constraints), or pointing a camera at a location in space (using two degrees of freedom under constraints), etc. Naturally specifying tasks may include specifying tasks using demonstrations, e.g., to teach a robot or to have a robot learn how to perform a task. Additionally and / or alternatively, a user may not specify how a robotic system may perform a task, and the system may determine how to perform the task automatically or with minimal input.

[0078] In some embodiments, the natural task allocation robotics system described herein may be configured to determine the minimum number of degrees of freedom "M" that may be required to perform a natural robotics task, where M<=N. The natural task allocation robotics system may be configured to determine the most natural or efficient way to perform a task by exploring additional (NM) degrees of freedom. This may involve optimizing auxiliary criteria that use the additional degrees of freedom, such as the criteria in (4). For example, a controller (e.g., Actin) may help enable exploration of additional degrees of freedom (NM) as described above. The additional number of degrees of freedom can be used to determine a more convenient, faster, energy-efficient, and / or feasible "method" to perform a task. The method or manner of execution may be based at least in part on the determination of a collision-free path, coordination of robots (if more than one), etc. In some embodiments, various software methods may be used (e.g., using Energid's Software) to at least partially apply to perform or fulfill a natural robotic task. The method or manner of naturally determining to perform a task may also include using one or more simulators that can be configured to perform simulations. The results of such simulations can be used to determine how to perform the task.

[0079] In some embodiments, exploring additional degrees of freedom (NM) may include fully relaxing some DOFs (e.g., the "Z" dimension). Additionally and / or alternatively, embodiments included herein may be configured to partially relax DOFs, e.g., allowing Z to rotate freely within +-15 degrees. The same concept may be applied to any DOF to provide flexibility when specifying a natural task. Thus, this may allow the robotic system to find a solution in a more efficient manner.

[0080] See now Figure 6-Figure 12 , provides an embodiment of a graphical user interface depicting a natural task allocation robotic system. As mentioned above with reference Figure 3 and Figure 5 As discussed, embodiments of the present disclosure may be configured to define how one or more robots are constrained based on the task. This may involve constraining only what is necessary. For example, referring to Figure 6 GUI 600, the user can define which links are constrained and how to use general spatial constraints. This may involve selecting constraints based on geometry, specifying speeds, and / or defining tool offsets. The user can also use Figure 7 The GUI 700 is used to configure one or more tools or robot offsets, such as Figure 8 shown.

[0081] See now Figures 9 and 10, provide embodiments depicting graphical user interfaces 900, 1000 configured to define tool paths and part relative waypoints, respectively. In this example, a user may define a task relative to an object to be manipulated (e.g., a task natural reference coordinate system). This may be accomplished using one or more waypoint poses, paths, and / or other customized motions. Each motion may include its own separate GUI. The final step may involve scripting the sequence of operations to be performed.

[0082] See now Figure 11 to Figure 12 , which illustrate embodiments depicting graphical user interfaces 1100 , 1200 configured to edit constraints and manage tool offsets, respectively. Fig.11 Includes the Constraint Editor, which includes an Edit Tool Offsets option. The Edit Tool Offsets option allows the user to edit tool offsets in context, such as Fig.12 In operation, select this option to display the Tool Offset Editor, which pre-selects the offsets by the constraints in use.

[0083] See now Figures 13 to 19 , embodiments of a natural task allocation robotic system for welding processes are provided. In some embodiments, minimal constraints can be specified in their natural reference coordinate system, further simplifying the task process. An example is a task performed by one robot on a part held by another robot, such as Fig.13 The welding task shown. In this example, the welding task can be defined in the reference coordinate system of the parts held by the two robots. This constraint approach is natural in terms of specifying exactly what needs to be done and allowing the motion of all the robots to achieve it.

[0084] In operation, Fig.13 As shown, the system can be configured to constrain the gripper relative to the part it is intended to grip. Wherever the parts are located in the workspace, the script commands them relative to their positions. Fig.14 An example is shown where two parts may be mated together. This may involve setting the tool offset on one arm to the mating coordinates of the part it holds, and setting the target to the mating coordinates of another part that may be held by the other arm. The arms may be constrained relative to each other so there is no specific position in the system coordinate system where these parts will mate.

[0085] See now Figures 15 to 19, in this particular example, a third arm can move a welder into position relative to an assembly that may be held by the other two arms. Again, the position of the assembly in the workspace may be unconstrained, only the welding gun may be constrained relative to the assembly, and its roll and pitch may be constrained in the system. The welder may move along a path, and the other two arms may automatically manipulate the component to achieve the path motion. It should be noted that the assembly appears to travel randomly around the workspace. After the path is completed, one gripper may retract relative to the assembly, again unconstrained in this system coordinate system. It is then moved to the nominal system position. This example shows how minimal constraints are specified in their natural reference coordinate system, and can be used to accomplish very complex motion coordinated between three arms.

[0086] See now Figure 20 to Figure 23 , an embodiment of a natural task allocation robotic system applied to a robotic assembly process is provided. In this specific example, depending on the geometry of the component, positioning the component in the assembly task may not require full 6-DOF. This specific example shows positioning the shaft by a wheel and a carriage. In this example, the arm holding the shaft bolt can rotate freely about the shaft bolt axis, and the arm holding the wheel also rotates freely about the wheel axis (e.g., Y axis).

[0087] like Fig. 20 As shown, the first step in the process of inserting the shaft may include grasping the part and then setting the tool offset to the relevant coordinates. In this specific example, one arm is the center of the wheel and the other arm is the center of the shaft. The wheel can then be positioned relative to the housing. At this point, the arm holding the wheel can rotate freely about the wheel axis, and the arm holding the shaft can rotate freely about the shaft axis. Figure 21 to Figure 22 As shown, the next step is insertion, which may involve commanding the hand to move the pin into position according to its current constraints. As the shaft is inserted, the arm holding the wheel is free to rotate about its axis and avoid colliding with the other arm. Fig.23 As shown, the final step may include release and retraction relative to the manipulated component. This example shows how minimal constraints are specified in their natural reference frame and how they are used to accomplish coordinated assembly motion between two robotic arms.

[0088] As will be appreciated by those skilled in the art, aspects of the present disclosure may be embodied as systems, methods, or computer program products. Thus, aspects of the present disclosure may take the form of a complete hardware implementation, a complete software implementation (including firmware, resident software, microcode, etc.), or an implementation combining software and hardware aspects, which may be collectively referred to herein as circuit "modules" or "systems." In addition, aspects of the present disclosure may take the form of a computer program product embodied in one or more computer-readable media having computer-readable program code embodied thereon.

[0089] Any combination of one or more computer-readable media can be utilized. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. For example, a computer-readable storage medium can be, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example (incomplete list) of a computer-readable storage medium will include the following: an electrical connector with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the context of this document, a computer-readable storage medium can be any tangible medium that can contain or store a program for use by an instruction execution system, a device or an equipment or for combining with the instruction execution system, a device or an equipment.

[0090] A computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave.

[0091] Such propagated signals may take any of a variety of forms, including, but not limited to, electromagnetic, optical, or any suitable combination thereof. A computer readable signal medium may be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including, but not limited to, wireless, wired, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0092] Computer program code for carrying out operations of aspects of the present disclosure may be written in any combination of one or more programming languages, including object oriented programming languages ​​such as Smallrix, C++, and Python. ++etc., and conventional procedural programming languages ​​such as the "C" programming language or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter case, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0093] The various aspects of the present disclosure are described below with reference to the flowchart and / or block diagram of the method, device (system) and computer program product according to the embodiments of the present disclosure. It should be understood that each frame of the flow chart and / or block diagram and the combination of frames in the flow chart and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device create a device for implementing the function / action specified in one or more frames of the flow chart and / or block diagram.

[0094] These computer program instructions may also be stored in a computer-readable medium, which may instruct a computer, other programmable data processing apparatus, or other device to operate in a specific manner so that the instructions stored in the computer-readable medium produce an article of manufacture including instructions for implementing the functions / actions specified in one or more boxes of the flowchart and / or block diagram.

[0095] The computer program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device, so that a series of operational steps are performed on the computer, other programmable apparatus, or other device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable apparatus provide a process for implementing the functions / actions specified in one or more boxes of the flowchart and / or block diagram.

[0096] The flowcharts and block diagrams in the accompanying drawings illustrate the possible specific implementation architecture, functions and operations of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram may, but not always, represent a module, section or part of a code including one or more executable instructions for implementing a specified logical function. It should also be noted that in some optional specific implementations, the functions indicated in the box may not occur in the order described in the accompanying drawings. For example, depending on the functions involved, the two boxes shown in succession may actually be executed substantially simultaneously, or these boxes may sometimes be executed in the opposite order. It should also be noted that each box in the block diagram and / or flowchart illustration, as well as the combination of boxes in the block diagram and / or flowchart illustration, can be implemented by a system based on dedicated hardware that performs a specified function or action, or a combination of dedicated hardware and computer instructions.

[0097] The terms used herein are only used to describe the purpose of specific implementations and are not intended to limit the present disclosure. As used herein, unless the context clearly indicates otherwise, the singular forms "a", "a kind of" and "the" are intended to also include plural forms. It should also be understood that when used in this specification, the terms "include" and / or "comprise" specify the features, integers, steps (not necessarily in a specific order), operations, elements, and / or parts, but do not exclude the presence or addition of one or more other features, integers, steps (not necessarily in a specific order), operations, elements, parts and / or groups thereof.

[0098] All means or steps that may be present in the claims below plus the corresponding structures, materials, actions and equivalent forms of the functional elements are intended to include any structure, material or action for performing a function in combination with other claimed elements as specifically claimed. The specification of the present disclosure has been presented for the purpose of illustration and description, but is not intended to be exhaustive or limited to the disclosure in the disclosed form. Without departing from the scope and essence of the present disclosure, many modifications, variations, substitutions and any combination thereof will be apparent to those of ordinary skill in the art. One or more specific implementations are selected and described in order to best explain the principles and practical applications of the present disclosure, and to enable other persons of ordinary skill in the art to understand the present disclosure for any combination of one or more various specific implementations and / or one or more specific implementations suitable for the envisioned specific purpose.

[0099] Thus, having described the disclosure of the present application in detail and with reference to one or more specific implementations of the present application, it will be apparent that modification, variation and any combination (including any modification, variation, substitution and combination thereof) of one or more specific implementations are possible without departing from the scope of the present disclosure as defined in the appended claims.

Claims

1. A system comprising: a robotic system having a maximum number of degrees of freedom associated therewith; a graphical user interface configured to receive a natural robotic task having at least one natural workpiece constraint associated with the natural robotic task, the natural robotic task comprising a task to be performed using a reduced amount of motion; and A processor configured to identify a minimum number of degrees of freedom required to perform the natural robotic task, wherein the minimum number of degrees of freedom is based at least in part on the at least one natural workpiece constraint associated with the natural robotic task, wherein the maximum number of degrees of freedom is a maximum number of patterns in which the robotic system can move and the minimum number of degrees of freedom is a minimum number of patterns in which the robotic system can move to perform the natural robotic task, and wherein the at least one natural workpiece constraint is defined based on a definition of an environment for the robotic system and motion of the natural robotic task is defined using the at least one natural workpiece constraint.

2. The system of claim 1 , wherein the at least one natural workpiece constraint associated with the natural robotic task is selected from the group consisting of: orientation constraints, assignments to curves or surfaces, distance constraints, joint positions, combinations of linear and nonlinear functions of joint positions, workpiece geometry, workpiece features, center of mass, and linear and angular momentum.

3. The system according to claim 1, further comprising: A scanner is configured to identify the at least one natural workpiece constraint associated with the natural robotic task.

4. The system according to claim 1, further comprising: A simulator is configured to determine a feasible or optimized implementation of the natural robotics task.

5. The system of claim 1, wherein receiving the natural robotic task is received in conjunction with a learning demonstration.

6. The system of claim 1, wherein the minimum number of degrees of freedom is less than the maximum number of degrees of freedom.

7. The system of claim 1, wherein the identification of the at least one natural workpiece constraint or minimum number of degrees of freedom associated with the natural robotic task is specified by a user.

8. The system of claim 3, wherein the identification of the at least one natural workpiece constraint or minimum number of degrees of freedom associated with the natural robotic task is automatically determined in response to the scanner identification.

9. The system of claim 1, wherein the identification of the at least one natural workpiece constraint or minimum number of degrees of freedom associated with the natural robotic task is updated in real time during operation of the robotic system.

10. The system of claim 1, wherein the minimum number of degrees of freedom is based at least in part on sensor data.

11. The system of claim 1 , wherein the identification of the at least one natural workpiece constraint or minimum number of degrees of freedom associated with the natural robotic task is updated using a communication channel before, during, or after operation of the robotic system.

12. The system of claim 1, wherein the graphical user interface is at least one of a two-dimensional interface or a three-dimensional interface.

13. The system of claim 1, wherein the natural robotic task is received by the system without reference to a robotic system coordinate system used to control the robotic system.

14. The system of claim 1, wherein the minimum number of degrees of freedom is identified by reference to only one or more features and geometries of the workpiece.

15. The system of claim 1, wherein the processor is configured to control the robotic system to perform the natural robotic task by exploring additional degrees of freedom, wherein the additional degrees of freedom are based at least in part on a difference between the maximum number of degrees of freedom and the minimum number of degrees of freedom.

16. The system of claim 15, wherein exploring the additional degrees of freedom comprises optimizing a criterion for using the additional degrees of freedom to employ a gradient-based approach that attempts to minimize the criterion.

17. The system of claim 3, wherein the processor and the scanner are configured to define a workspace prior to performing a robotic task.

18. The system of claim 1, wherein the natural robotic task includes at least one natural environmental constraint associated with the natural robotic task.

19. The system of claim 1, wherein the at least one natural workpiece constraint comprises one or more geometric features associated with a particular workpiece.

20. The system of claim 17, wherein the at least one natural environmental constraint comprises: Axle hole insertion, line following, camera pointing, gripping symmetrical objects, placing objects on flat surfaces, contacting flat objects with suction cups, manipulating mobile bases, or controlling the center of mass of a mobile robot to create stability.

21. A method comprising: providing a robotic system having a maximum number of degrees of freedom associated therewith; receiving, at a graphical user interface, a natural robotic task having at least one natural workpiece constraint associated with the natural robotic task, the natural robotic task comprising a task to be performed using a reduced amount of motion; as well as Using at least one processor, identifying a minimum number of degrees of freedom required to perform the natural robotic task, wherein the minimum number of degrees of freedom is based at least in part on the at least one natural workpiece constraint associated with the natural robotic task, wherein the maximum number of degrees of freedom is a maximum number of patterns in which the robotic system can move and the minimum number of degrees of freedom is a minimum number of patterns in which the robotic system can move to perform the natural robotic task, and wherein the at least one natural workpiece constraint is defined based on a definition of an environment for the robotic system and motion of the natural robotic task is defined using the at least one natural workpiece constraint.

22. The method of claim 21, wherein the at least one natural workpiece constraint associated with the natural robotic task is selected from the group consisting of: orientation constraints, assignments to curves or surfaces, distance constraints, joint positions, combinations of linear and nonlinear functions of joint positions, workpiece geometry, workpiece features, center of mass, and linear and angular momentum.

23. The method according to claim 21, further comprising: The at least one natural workpiece constraint associated with the natural robotic task is identified based at least in part on a scanner.

24. The method of claim 21, further comprising: A feasible or optimized implementation of the natural robotic task is determined based at least in part on the simulation.

25. The method of claim 21, wherein receiving the natural robotic task is received in conjunction with a learning demonstration.

26. The method of claim 21, wherein the minimum number of degrees of freedom is less than the maximum number of degrees of freedom.

27. The method of claim 21, wherein the identification of the at least one natural workpiece constraint or minimum number of degrees of freedom associated with the natural robotic task is specified by a user.

28. The method of claim 23, wherein the identification of the at least one natural workpiece constraint or minimum number of degrees of freedom associated with the natural robotic task is automatically determined in response to the scanner identification.

29. The method of claim 21, wherein the identification of the at least one natural workpiece constraint or minimum number of degrees of freedom associated with the natural robotic task is updated in real time during operation of the robotic system.

30. The method of claim 21, wherein the minimum number of degrees of freedom is based at least in part on sensor data.

31. The method of claim 21, wherein the identification of the at least one natural workpiece constraint or minimum number of degrees of freedom associated with the natural robotic task is updated using a communication channel before, during, or after operation of the robotic system.

32. The method of claim 21, wherein the graphical user interface is at least one of a two-dimensional interface or a three-dimensional interface.

33. The method of claim 21, wherein the minimum number of degrees of freedom is identified without reference to a robot approach coordinate system used to control the robot system.

34. The method of claim 21, wherein the minimum number of degrees of freedom is identified by reference to only one or more features and geometries of the workpiece.

35. The method of claim 21, further comprising: The robotic system is controlled, using the at least one processor, to perform the natural robotic task by exploring additional degrees of freedom, wherein the additional degrees of freedom are based at least in part on a difference between the maximum number of degrees of freedom and the minimum number of degrees of freedom.

36. The method of claim 23, further comprising: A workspace is defined prior to performing a robotic task, wherein the definition is based at least in part on data from the processor and the scanner.

37. The method of claim 21, wherein the natural robotic task includes at least one natural environmental constraint associated with the natural robotic task.

38. The method of claim 21, wherein the at least one natural workpiece constraint comprises a geometric feature associated with a particular workpiece.

39. The method of claim 37, wherein the at least one natural environmental constraint comprises: Axle hole insertion, line following, camera pointing, gripping symmetrical objects, placing objects on flat surfaces, contacting flat objects with suction cups, manipulating mobile bases, or controlling the center of mass of a mobile robot to create stability.

40. The method of claim 21, wherein the at least one natural workpiece constraint includes not only a constraint type equal to the at least one natural workpiece constraint, but also a constraint type less than the at least one natural workpiece constraint.

41. The method of claim 21, wherein a manipulator Jacobian is used at least in part to calculate motion associated with the robotic system.

42. The system of claim 1, wherein the processor is configured to control the robotic system to perform the natural robotic task by exploring relaxed degrees of freedom.

Images