Human skill-based path generation

Abstract

A method for robot path planning uses skills extracted from a motion program of human teaches applied to a new obstacle environment. A three-dimensional convolutional neural network is used to extract features representing an obstacle environment, wherein feature vector representation of an obstacle overcomes problems encountered when point cloud obstacle data is used. Obstacle feature data and robot path start and target points are provided to an encoder / decoder neural network system trained to extract skills from a database of human-generated motion programs. An encoder / decoder neural network system generates a waypoint distribution for a current obstacle environment and a start / target point. The waypoint distribution is used to perform final collision-free path generation using a fast exploring random tree (RRT) technique or an optimization-based technique.

Description

Technical Field

[0001] The present disclosure relates generally to the field of industrial robot motion programming, and more specifically, to a method for generating a path for a robot, which uses a convolutional neural network to extract features characterizing an obstacle environment, uses an encoder / decoder neural network system to extract skills and generate a distribution of waypoints for the current obstacle environment from a database of human-generated motion programs, and performs final collision-free path generation based on the distribution of waypoints. Background Art

[0002] It is well known to use industrial robots to repeatedly perform a wide range of manufacturing, assembly and material movement operations. There are various techniques for teaching robots to move from a starting point to a target point. However, existing path generation techniques all exhibit certain shortcomings when there are obstacles between the starting and target points.

[0003] One known technique for path generation is to use a teach pendant. The teach pendant communicates with the robot controller and is operated by a human operator. The operator uses the teach pendant to instruct the robot to make incremental moves—like “jog in the x-direction” or “rotate the gripper around the local z-axis.” The robot motions are recorded by the robot controller and stored as motion programs. These types of teach pendant instructions and robot motions are satisfactory for simple paths with uncomplicated motion and few or no obstacles. However, programming a robot using a teach pendant in a complex obstacle environment is often difficult, error-prone, and time-consuming.

[0004] Another known technique for path generation is to use a collaborative robot in a "guiding" process. In the guiding process, a human operator manually grabs a tool or workpiece at the end of the robot arm and moves the tool or workpiece from a starting point to a target point. The guiding process has the advantage of capturing human expertise in selecting a path and allows collision avoidance assessment of all parts of the robot during motion - which is critical in applications where any part of the robot (not just the tool or workpiece) may come into contact with an obstacle. Unfortunately, when the obstacle environment is complex, it is difficult or impossible for a human operator to operate the entire robot (including all intermediate joint positions) to avoid collisions.

[0005] Robot teaching by human demonstration is also known, in which a human demonstrator manually grasps and moves a workpiece from a starting position to a target position. However, path generation by human demonstration may lack the positional accuracy required for precise movement of the workpiece, and path generation by human demonstration does not consider collision avoidance of the robot arm itself with obstacles in the workspace.

[0006] Automatic path generation techniques are also known - where a start point and a target point are provided, as well as a geometric definition of obstacles in the environment - and a path generation calculation is attempted. These automatic path generation techniques also have some disadvantages, which are discussed further below.

[0007] In view of the above, there is a need for an improved robotic path generation technique that captures the essence of the skills taught by humans generating paths and applies these skills to generate collision-free paths in new obstacle environments. Summary of the invention

[0008] The present disclosure describes a method for robot path planning that uses skills extracted from human-taught motion programs applied to new obstacle environments. A three-dimensional convolutional neural network is used to extract features that describe the obstacle environment, where a feature vector representation of the obstacles overcomes the problems encountered when using point cloud obstacle data. The obstacle feature data and the robot path start and target points are provided to an encoder / decoder neural network system that is trained to extract skills from a database of human-generated motion programs. The encoder / decoder neural network produces a waypoint distribution for the current obstacle environment and start / target points. The waypoint distribution is used to perform the final collision-free path generation using a rapidly exploring random tree (RRT) technique or an optimization-based technique.

[0009] Additional features of the presently disclosed apparatus and methods will become apparent from the following description and appended claims, taken in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1A is an illustration of automatic path generation using the Rapidly Exploring Random Tree (RRT) technique as known in the art, and Figure 1B is an illustration of automatic path generation using optimization-based techniques as known in the art;

[0011] Figure 2 is a block diagram illustration of a system for human skill-based robot path generation according to an embodiment of the present disclosure;

[0012] Figure 3 is a diagram depicting how a point cloud takes on different mathematical representations when the order in which the points in the point cloud are numbered is changed;

[0013] Figure 4 is a block diagram illustration depicting extraction of features from obstacle point cloud data and use of the feature vector data as input for human skill-based path generation according to an embodiment of the present disclosure;

[0014] Figure 5is a diagram depicting how to exploit human skills to extract a distribution of waypoints for creating a robot path using encoder and decoder neural networks, using start and goal point and obstacle feature data as inputs along with a database of human-generated paths for training, according to an embodiment of the present disclosure;

[0015] Figure 6 is a description of how to use RRT or optimization-based techniques according to an embodiment of the present disclosure Figure 5 Block diagram illustration of the distribution of waypoints for collision-free path generation;

[0016] Figure 7 is a schematic diagram illustration of a system for human skill-based robotic path generation according to an embodiment of the present disclosure, depicting robots and obstacle geometries from an example implementation; and

[0017] Figure 8 It is a flowchart of a method for human skill-based robot path generation according to an embodiment of the present disclosure, including obstacle feature extraction and human skill extraction in a neural network system, and collision-free path generation based on the distribution of waypoints. DETAILED DESCRIPTION

[0018] The following discussion of the disclosed embodiments of methods for human skill-based robotic path generation is merely exemplary in nature and is in no way intended to limit the disclosed devices and techniques or their applications or uses.

[0019] The use of industrial robots for various manufacturing, assembly, and material movement operations is well known. It has long been a goal to develop simple techniques for generating robot motion programs that are efficient and avoid collisions with any obstacles present in the working environment. However, existing path generation techniques all exhibit certain shortcomings.

[0020] It is recognized that human intuition and vision are powerful tools that can be employed in robot path generation. Therefore, various path generation techniques for using human input have been developed. These techniques - including teach pendant manipulation, collaborative robot guidance, and human demonstration of workpiece pick and place operations - can be very effective in path generation for certain types of operations. However, when complex obstacle environments are involved, such as a robot mounted on one side of a workpiece and must pass through a hole in the workpiece to perform an operation on the other side, these existing path generation techniques often fall short of the capabilities required to generate efficient and collision-free paths.

[0021] Automatic path generation techniques are also known, where a start point and a target point are provided, along with a geometric definition of obstacles in the environment, and automatic calculation of a collision-free path is attempted. Two such techniques and their limitations are discussed below.

[0022] Figure 1A is an illustration of automatic path generation using the Rapidly Exploring Random Tree (RRT) technique as known in the art, Figure 1B is an illustration of automatic path generation using optimization-based techniques as known in the art. Figure 1A , an RRT path generation scenario 100 is depicted. In the scenario 100, a collision-free path is desired from a start point 110 to a target point 112. There are a number of obstacles 120 in the workspace between the start point 110 and the target point 112. The RRT generated path 130 is depicted midway through its growth process.

[0023] As known to those skilled in the art, the RRT method proposes a new waypoint within an increasing distance from a previous waypoint (or starting point 110), and evaluates the feasibility of the path segment from the previous waypoint to the new waypoint. If the path segment is collision-free, the new waypoint is added to the path, and another new waypoint is evaluated. Many branches are formed in the RRT path, and a complete path from the starting point 110 to the target point 112 can be found eventually. However, the path generated by RRT is unnatural in shape in characteristics, with many short path segments of zigzags back and forth. Due to this and other reasons, the path generated using pure RRT technology is often found to be less ideal than the path generated in other ways.

[0024] exist Figure 1B , scene 140 contains the same environment of starting point 110 and target point 112 and the same obstacles 120 as in scene 100 described above. In scene 140, optimization-based techniques are used to attempt to find a collision-free path from starting point 110 to target point 112. Path 150 extends from starting point 110 midway through the obstacle environment of scene 140. Optimization-based path generation methods can generate paths using objective functions that encourage smoothness and efficiency of path routing and constraints that ensure collision avoidance throughout the path. However, in complex obstacle environments, optimization-based methods may not be able to find a feasible solution. This situation is depicted in scene 140, in which case path 150 extends to point 152, at which point it is stuck behind two obstacles 120 and surrounded by the outer boundary of the workspace.

[0025] The disclosed techniques have been developed to overcome the limitations and shortcomings of existing path generation methods described above. The techniques discussed below capture the skills embodied in human-generated paths while using advanced computation to apply human path generation skills to find collision-free paths through new obstacle environments.

[0026] Figure 2is a block diagram illustration of a system 200 for human skill-based robot path generation according to an embodiment of the present disclosure. At block 210, obstacle data of the workspace environment is provided, as well as the start and target point locations of the new path to be calculated. These inputs correspond to Figure 1A / 1B and previously discussed scenes 100 and 140. Instead of directly calculating the path, the input from block 210 is provided to an encoder / decoder block 220. The encoder / decoder block 220 includes encoder and decoder neural networks that are configured to extract human skills from existing robot motion programs and apply these skills to calculate paths in new obstacle environments. The encoder / decoder block 220 is discussed in detail below.

[0027] A database 230 of existing motion programs is also provided to the encoder / decoder block 220. The database 230 includes robot motion programs generated using any technology that incorporates human skills. These technologies include, for example, using a teach pendant to define robot motion programs as a series of incremental motions, using collaborative robots by humans in "guided" motion capture techniques, and human demonstration of workpiece pick-up and placement operations. The database 230 includes multiple human-taught motion programs (robot paths), and for each motion program, a definition of an obstacle environment corresponding to the robot path. In this way, the database 230 characterizes the way human experts solve motion programming problems for various obstacle environments. That is, the database 230 captures human skills when tracking a path from a starting point to a target point, such as path curvature and obstacle avoidance distance. The database 230 is used to train the encoder / decoder block 220. As discussed below, the human skills embodied in the database 230 are extracted by the encoder / decoder block 220 and ultimately used to generate paths for new obstacle environments.

[0028] The encoder / decoder block 220 generates a waypoint distribution 240, which contains the distribution of waypoints (in a sense, a cloud path) in a workspace environment that contains the obstacles defined in the input block 210. The waypoint distribution 240 is then used to generate a collision-free path 250 in one of two ways. The system 200 captures the path generation skills of human experts, applies these skills to draw a path through a new obstacle environment, and uses the power of automatic path generation technology in a way that overcomes the limitations of existing path generation methods. Each of the various components of the system 200 and their interactions with each other are discussed further below.

[0029] Figure 3A diagram depicting how a point cloud presents different mathematical representations when the order in which the points in the point cloud are numbered is changed. Many existing path generation techniques use a point cloud representation of an obstacle environment. As known to those skilled in the art, a point cloud is a collection of points representing an object or objects, where the points are typically defined in a three-dimensional (3D) coordinate system. The points in a point cloud may be located only on the outer surface of an object, or the points may be located on both the exterior and interior of an object.

[0030] Although point clouds are a convenient form of 3D obstacle representation, they exhibit some properties that are problematic in subsequent calculations using them. For example, consider a simple point cloud 310 containing five points in a spatial environment, which are numbered 1-5. Point cloud 320 includes five points in a spatial environment, whose positions are the same as the points in point cloud 310, but the points in point cloud 320 have been numbered 1-5 in a different order. Point cloud 310 corresponds to coordinate matrix 330, where point 1 of coordinate matrix 330 has coordinates (x1, y1, z1), and so on. Point cloud 320 corresponds to coordinate matrix 340, where point 1 of coordinate matrix 340 has coordinates (x1, y1, z1) that are different from (x1, y1, z1) of coordinate matrix 330. In other words, even though point cloud 310 and point cloud 320 define five points in the same position in space, coordinate matrix 330 is completely different from coordinate matrix 340. Therefore, point clouds do not exhibit permutation invariance; that is, the point cloud definition changes based on the order in which the items are listed.

[0031] Another characteristic of point clouds is that they do not exhibit transformation invariance. That is, a point cloud representing an object oriented one way (e.g., "right side up") is completely different from a point cloud representing the same object oriented another way (e.g., "upside down"). Both permutation variation and transformation variation create problems and inefficiencies when using point clouds for obstacle definition, as both phenomena make it difficult or impossible to effectively identify and process objects, especially in machine learning systems.

[0032] Figure 4 is a block diagram illustration depicting the extraction of features from obstacle point cloud data and the use of feature vector data as input for human skill-based path generation according to an embodiment of the present disclosure. Figure 2 , block 210 is shown as providing obstacle data input, as well as definition of start and target point locations. Figure 4 shows how obstacle data can be provided as input to the Figure 2 system.

[0033] The point cloud 410 represents an object (in this example a chair, which can be considered an obstacle) in a known manner and as described above. A 3D convolutional neural network (CNN) 420 is used to extract features from the obstacle point cloud 410. A feature extractor - such as CNN 420 - takes a point cloud or other data (e.g., a solid model or image) as input and provides a set of feature vectors characterizing the input as output. The feature extractor CNN 420 significantly reduces the amount of data required for subsequent processing by replacing a point cloud with thousands of point coordinates with a set of feature vectors that can be defined in hundreds of numbers. The feature extractor CNN 420 also overcomes the problems inherent in the point cloud obstacle definition discussed above by using a feature vector representation of the obstacle rather than the original point cloud data.

[0034] As will be appreciated by those skilled in the art, CNN is a common architecture for feature extraction, and in practice, pre-trained feature extractor CNNs are available where the network structure, including the number of layers and the size of each layer output, is fixed and the network parameters are trained. Feature extractor CNN 420 may be a pre-trained feature extractor CNN of this type. The CNN 420 to be used as the encoder / decoder block 220 (from Figure 2 ) is an input to a CNN 420 and the final output is a set of feature vectors shown in block 430. The output depicted in block 430 is an array of feature vectors having a size of "X" by "Y", each feature vector having a dimension of {NNx1}. The number of feature vectors (X·Y) and the dimension of each feature vector can be selected to be suitable for a particular application and to correspond to the design of CNN 420.

[0035] The feature vector from block 430 characterizing the obstacle from point cloud 410 is provided as Figure 2 The encoder / decoder block 220 shown earlier in FIG. 4 is provided as input to the encoder / decoder block 220. In block 440, the start and goal point locations of the robot path to be generated are also provided as input to the encoder / decoder block 220. After training with the database 230 of existing motion programs, the encoder / decoder block 220 uses the obstacle and start / goal inputs from blocks 430 and 440 to calculate the previously referenced Figure 2 The waypoint distribution 240. The training and calculation of the waypoint distribution are discussed below.

[0036] Figure 5 is a block diagram illustration of how human skills can be exploited to extract the distribution of waypoints for creating a robot path using encoder and decoder neural networks according to an embodiment of the present disclosure. The encoder / decoder block 220 is Figure 5, receiving as inputs an obstacle feature vector 430 and the start and goal points in block 440. A database 230 of human-generated motion programs with corresponding obstacle data is also provided as input. The obstacle and motion data from the human-generated path database are used to train encoder and decoder neural networks that capture human skills, where the encoder defines a Gaussian distribution of probabilities associated with a set of state (start and goal point locations, obstacles) and action (motion) data, and the decoder determines an action (waypoint) associated with a set of state (start and goal point locations, obstacles) data and corresponding probabilities.

[0037] The encoder / decoder block 220 includes an encoder neural network 510. Data from a database 230 of human-generated motor programs provides a sequence of corresponding “state” and “action” data—(s0, a1), (s1, a2), (s2, a3), etc.—for training the encoder neural network 510. Each “state” s i Each of the following is an obstacle characteristic and start and target point location from one of the various motion programs in database 230, and the corresponding "action" a i+1 is the motion characteristic (waypoint from the motion program) resulting from the state. The encoder neural network 510 generates a distribution q (shown at 520) of probabilities z associated with the state s and the action a. The distribution q(z|s,a) shown at 520 captures human skills from human-generated motion programs in the database 230. The database 230 of human-generated motion programs, including both obstacle environment data and corresponding human-generated motion programs, is used as labeled data for training the encoder neural network 510 and the decoder neural network 530 to exhibit the desired path generation behavior.

[0038] After training, the decoder neural network 530 is then used to determine the action a corresponding to the state s and the probability z. This is done using the probability function π(a|s,z) as shown on line 540. The decoder neural network 530 receives as input the feature vector 430 that defines the obstacle environment of the path to be generated and the start and target point locations of the path to be generated from box 440. Then, using the probability function π, the decoder neural network 530 calculates a set of actions a (waypoints) corresponding to the state s (start and target point locations, and obstacles) and the probability z. This results in a waypoint distribution 240, which is not a complete and deterministic path (motion program), but a waypoint distribution based on the obstacle environment of the path being generated and the probabilities encoded in the human skills from the database 230.

[0039] The training of the encoder / decoder block 220 is accomplished using a database 230 of human-generated motion programs, which includes an obstacle environment for each motion program. Using the database 230 as a labeled data set for training, the encoder neural network 510 learns a distribution q that captures human skill, while the decoder learns a probability function π that produces a distribution of waypoints along the motion program used as input. The training of the encoder neural network 220 can be accomplished using known loss function methods or another technique as determined to be most appropriate.

[0040] To summarize the above discussion; a database 230 of human-generated motion programs (including an obstacle environment for each motion program) is used to train the encoder neural network 510 and the decoder neural network 530 to capture human path generation skills applied to many different obstacle scenarios. Then, once trained, the decoder neural network 530 receives the start / target location and obstacle environment (obstacle feature vector data) of a new path to be generated, and uses the distribution q to calculate a probability function π that generates a waypoint distribution 240, where the waypoint distribution 240 is a collection of multiple points along a path extending from a start point to a target point while generally avoiding obstacles along the way.

[0041] In an experimental evaluation of the method of the present disclosure, the training data set (in database 230) included 300 motion programs (each with a corresponding obstacle environment). This training data set demonstrated the ability to fully train the encoder / decoder block 220. After training with the 300 motion program data set, a test data set of 50 obstacle environments was provided for evaluation, where a waypoint distribution was calculated for each of the 50 examples in the test data set. More than half of the test examples resulted in a collision-free waypoint distribution throughout the obstacle environment of the particular example. However, the remaining test examples resulted in a waypoint distribution that included some waypoint interference with obstacles. Therefore, although the waypoint distribution 240 does a very good job of capturing the general motions that represent human skills in the training database, additional method steps are required to produce reliable collision-free paths for each start / target constraint and obstacle environment.

[0042] Figure 6 is a description of how to use RRT or optimization-based techniques according to an embodiment of the present disclosure to convert Figure 5 The distribution of waypoints 240 is used for a block diagram illustration of collision-free path generation. As explained above, the waypoint distribution 240 does not guarantee that the path from the starting point to the target point is collision-free. Therefore, the currently disclosed technology adds a method step of performing collision-free path generation based on the waypoint distribution 240.

[0043] The first technique for collision-free path generation based on waypoint distribution 240 uses the RRT method. Figure 6 In block 610, the waypoint distribution 240 is shown, as well as the start of the RRT path indicated at 612. Figure 1A Unlike conventional RRT path generation discussed above, the path generation technique of the present disclosure does not randomly select suggested new waypoints from anywhere in the workspace. Instead, the disclosed path generation technique selects suggested new waypoints only based on the waypoint distribution 240. This ensures that the RRT generated path follows the path arranged by the encoder / decoder block 220, which captures human path generation skills. It also makes the RRT path generation very efficient because most of the suggested new waypoints will be collision-free, and thus the RRT path will be quickly constructed from the start point to the target point, following the waypoint distribution based on human skills, while avoiding all obstacles along the way (because each RRT path segment is checked for collision avoidance). After all poorly performing branches have been pruned, the resulting collision-free path 250 is Figure 6 , superimposed on waypoint distribution 240 .

[0044] A second technique for collision-free path generation based on waypoint distribution 240 uses optimization calculations. Figure 6 In block 620, the waypoint distribution 240 is used in the following optimization calculation. The optimization model is defined using an objective function that penalizes deviations from the waypoint distribution and an inequality constraint that ensures that the path remains collision-free. A preferred non-limiting embodiment of the objective function is defined in the following equation (1): min[λ 1( wp robot -wp human )+λ2∑d i +λ3∑t i ](1) Among them wp robot is the waypoint of the robot in the path being calculated, wp human is the waypoint distribution 240 provided from the encoder / decoder block 220 and captures the human path generation skill, ∑d i is the total distance of the generated path (the sum of the distances for each path point i), ∑t i is the total cycle time of the robot motion using the generated path, and λ1-λ3 are weighting factors. The first term of the objective function in equation (1) is used to minimize the deviation of the calculated robot path from the waypoint distribution 240, where the center or mean position of the waypoint distribution is preferably used for this calculation. The weighting factors λ1-λ3 can be selected to achieve the desired path generation performance in any given application.

[0045] Together with the objective function of equation (1), the optimization model includes at least one constraint, such as an inequality constraint, which computes the minimum distance between the robot and any obstacle for each iteration of the waypoint calculation, and the constraint stipulates that the minimum distance is greater than a predetermined threshold (the threshold can be zero, or some positive distance value such as 15 mm). In this way, before each robot waypoint i is added to the path, it is required that the robot waypoint i is collision-free. In addition to the collision avoidance inequality constraint, other constraints can be included in the optimization model, such as constraints on the robot joint positions, which ensure that the robot configuration at each waypoint i is feasible.

[0046] The optimization model defined above ensures that the generated path follows the waypoint distribution arranged by the encoder / decoder block 220 that captures the human path generation skills. The optimization model also makes the generated path very efficient while avoiding all obstacles along the way. Figure 6 Also shown at 250 in FIG. 2 is a collision-free path resulting from optimization-based path generation, which is superimposed on the waypoint distribution 240 .

[0047] Figure 7 is a schematic illustration of a system 700 for human skill-based robotic path generation, depicting robots and obstacle geometries from an example implementation, according to an embodiment of the present disclosure. Figure 7 is provided as a means of applying the above-described method and system (regarding Figure 2 and Figure 4-6 ) is applied to an actual robot in a complex 3D obstacle environment rather than just tracing a curve through a simple set of 2D obstacles.

[0048] In the first step (indicated at ①), the encoder / decoder neural network system 720 is trained using a database 710 of human-generated robot motion programs. This step refers to Figure 5 A detailed description is given, in which a database 710 (equivalent to database 230) includes many (e.g., hundreds) of motion programs created by a human teaching method, and corresponding obstacle data for each motion program, and the data is used to train an encoder / decoder system 720 (distribution q(z|s,a) and distribution probability function π(a|s,z)) to simulate the human skill of defining path points that avoid obstacles in a spatial environment.

[0049] In the second step (indicated at ②), the start and target positions of the new path to be generated are provided to the trained encoder / decoder neural network system 720 along with the obstacle environment in which the path must be found. The obstacle geometry data is provided to the 3D CNN 730 which extracts obstacle feature data; the obstacle feature data is provided to the encoder / decoder system 720 instead of the original obstacle geometry or point cloud data. This step refers to Figure 4 Detailed description is given. An illustration of a robot and a door frame is provided in box 702. This depicts an example of the type of actual robot path generation scenario to which the currently disclosed methods and systems are applied. In this example, the starting point has a tool center point located on the "near" side of the door frame (the same side on which the robot base is mounted), while the target point has a tool center point located on the "far" side of the door frame. This would, for example, represent the type of path that needs to be generated for a spot welding robot, where welding operations are required on both sides of the workpiece (door frame). In addition, the door frame itself is an obstacle that the robot must avoid. Other obstacles may also exist in the workspace environment.

[0050] In the third step (indicated at ③), the decoder neural network uses the start and target positions and obstacle feature data to generate a waypoint distribution 740. This step is also described previously with reference to Figure 5 . Waypoint distribution 740 includes a distribution of points calculated by probability function π based on the start / goal position and obstacle environment. In the example shown, waypoint distribution 740 follows a path through an opening or hole in a door frame. Waypoint distribution 740 embodies the human skill captured in encoder / decoder system 720 for collision avoidance path generation. Waypoint distribution 740 is not guaranteed to be collision-free and therefore includes the final step of path generation.

[0051] In the fourth step (indicated at ④), a collision-free path 750 is generated based on the waypoint distribution 740. This step is also previously referred to Figure 6 As previously discussed, the collision-free path 750 can be calculated from the waypoint distribution 740 using either the RRT method or the optimization-based method. Both the RRT method and the optimization-based method use the waypoint distribution 740 in the selection of points along the final collision-free path.

[0052] refer to Figure 7, some important points will be observed. First, the obstacles involved in robot motion programming are typically 3D objects that may have complex shapes (such as car door frames), and may also include fixtures and platforms with simpler prismatic shapes. In addition, each waypoint or robot pathpoint is not just a point in 2D or 3D space, but is defined by a complete robot configuration, including the positions of all joints of the robot arm. In this way, the collision avoidance calculation (in the RRT method or optimization-based method for path generation) calculates the minimum 3D distance between all parts of the robot arm (including the tool, and including any workpiece that the tool may be grasping) and all obstacles. The minimum distance calculation can be performed in any appropriate manner, such as using a signed distance field method or other appropriate method. Figure 7 depicts the actual complexity involved in all the data and calculations, whereas the previous figures depict simplified examples for conceptual explanation.

[0053] Figure 8 800 is a flowchart of a method for human skill-based robot path generation according to an embodiment of the present disclosure, the method including obstacle feature extraction and human skill extraction in a neural network system, and collision-free path generation based on the distribution of waypoints. At box 802, a training database of human-generated motion programs is provided. This is the database 230 (or 710) discussed earlier, which includes robot motion programs generated using any technology that incorporates human skills, such as using a teach pendant to define the robot motion program as a series of incremental moves, humans using collaborative robots in "guided" motion capture technology, and humans demonstrating workpiece pick-up and placement operations. The database 230 includes multiple human-taught motion programs (robot paths) and, for each motion program, includes a definition of the obstacle environment corresponding to the robot path.

[0054] At block 804, the encoder / decoder neural network system is trained using the training database. As previously described, the encoder / decoder neural network system (block 220) learns to encode or extract human skills for motion program generation when faced with an obstacle environment. Training teaches the encoder / decoder neural network system (via the distribution q(z|s,a) and the probability function π(a|s,z)) to emulate human skills in defining waypoints that avoid obstacles in a spatial environment.

[0055] At block 806, the new path to be generated is provided with start and target locations and obstacle feature data. In a preferred embodiment, a 3D convolutional neural network (3D CNN) is used to extract feature vector data from obstacle 3D geometry data (e.g., point cloud or solid model).

[0056] At block 808, a waypoint distribution is calculated for the new path by the encoder / decoder neural network system. Specifically, the trained decoder neural network uses the probability function π(a|s, z) to calculate the waypoint distribution based on the start and target positions of the new path and the obstacle feature data. Because the decoder neural network is trained with the encoder neural network in conjunction with the human skill in the encoding distribution q, the resulting waypoint distribution simulates human skill in defining waypoints for navigating around obstacles in a spatial environment.

[0057] At box 810, a collision-free path is generated based on the waypoint distribution. The collision-free path generation uses the waypoint distribution for guidance and includes a complete interference check analysis (for the entire robot) at each waypoint. At box 812, the RRT method can be used to generate the collision-free path, or at box 814, an optimization-based method is used to generate the collision-free path. The resulting collision-free path embodies the human skill of finding a path through an obstacle environment as taught by the training database. The collision-free path is a complete robot motion program (defining all joint positions at each waypoint), or it can be easily converted into a complete motion program through inverse kinematics calculations understood by technicians in the field of robotics.

[0058] Throughout the foregoing discussion, various computers and controllers are described and implied. It should be understood that the software applications and modules of these computers and controllers are executed on one or more computing devices having a processor and a storage module. In particular, this includes a processor in a computer or robot controller, where the controller / computer is configured to perform human skill-based path generation calculations in the manner described above. The computing device may include a specialized computing device that is specifically configured to execute a 3D CNN and / or encoder / decoder neural network system.

[0059] As described above, the disclosed human skill-based robotic path generation technique provides several advantages over existing robotic path generation methods. The disclosed technique captures the intuition of human path generation in a trained neural network, uses feature vector representations of obstacles to improve consistency, applies human skills to waypoint distribution generation for new paths, and performs path generation based on the waypoint distribution to ensure that the final path is collision-free.

[0060] Although multiple exemplary aspects and embodiments of human skill-based path generation have been discussed above, those skilled in the art will recognize modifications, permutations, additions, and sub-combinations thereof. Therefore, the following appended claims and claims introduced thereafter are to be interpreted as including all such modifications, permutations, additions, and sub-combinations within their true spirit and scope.

Claims

1. A method for human skill-based robot path generation, the method comprising: providing a training database comprising a plurality of human-generated robotic motion programs including data defining obstacles to be avoided during robotic execution of each motion program; Using the training database to train an encoder / decoder neural network system comprising an encoder neural network and a corresponding decoder neural network running on a computing device; providing the encoder / decoder neural network system with obstacle signature data and start and goal locations for the new path; Calculating a waypoint distribution of the new path by the encoder / decoder neural network system, wherein the waypoint distribution is a set of candidate waypoints of the new path; and The new path is generated based on the waypoint distribution, wherein the new path defines a motion of a robot to move a tool center from the start position to the target position, including performing a robot-obstacle collision avoidance calculation for each waypoint.

2. The method according to claim 1, wherein: The training database includes robot motion programs generated by one or more methods selected from the group consisting of: human-operated teach pendant programming, human-guided path demonstration by a collaborative robot, and demonstration of robot operation by human grasping and manipulation of a tool or workpiece.

3. The method according to claim 1, wherein: The encoder neural network determines a distribution of probabilities associated with states and actions, wherein the states and actions are represented by the obstacles and corresponding steps in the plurality of motion programs in the training database.

4. The method of claim 3, wherein the decoder neural network determines a probability function of an action corresponding to a state and a probability, wherein the action is defined as the candidate waypoint in the waypoint distribution.

5. The method according to claim 4, wherein: Training the encoder / decoder neural network system includes using the training database as a labeled training data set, causing the encoder neural network to learn the distribution and causing the decoder neural network to learn the probability function that captures human path generation skills from the plurality of human-generated motion programs in the training database.

6. The method according to claim 1, wherein: Providing obstacle feature data includes providing a point cloud or solid model representation of the obstacle in the workspace of the new path to a three-dimensional convolutional neural network (3D CNN), using the 3D CNN to extract a feature vector representing the obstacle, and providing the feature vector to the encoder / decoder neural network system.

7. The method according to claim 1, wherein: Generating the new path based on the waypoint distribution includes using a rapidly exploring random tree (RRT) computation, wherein proposed waypoints evaluated in the RRT computation are selected from the candidate waypoints in the waypoint distribution.

8. The method according to claim 1, wherein: Generating the new path based on the waypoint distribution includes using an optimization calculation, wherein the optimization calculation minimizes an objective function including a term that calculates a distance between a waypoint in the new path and an average position of the candidate waypoints in the waypoint distribution.

9. The method according to claim 8, wherein: The objective function also includes terms for calculating the total distance of the new path and the total time to execute the new path, and the optimization calculation further includes an inequality constraint that stipulates that the minimum robot-obstacle distance at each waypoint is greater than a predetermined non-negative threshold.

10. The method according to claim 1, wherein: For each waypoint in the new path, a robot configuration is calculated using inverse kinematics calculations and used in the robot-obstacle collision avoidance calculations.

11. The method according to claim 1, further comprising: The new path is provided to a controller of the robot, and the new path is executed by the robot based on motion commands from the controller.

12. A method for human skill-based robot path generation, the method comprising: providing a training database comprising a plurality of human-generated robotic motion programs including data defining obstacles to be avoided during robotic execution of each motion program; training an encoder / decoder neural network system including an encoder neural network and a corresponding decoder neural network running on a computing device using the training database, wherein the training causes the encoder neural network to learn a distribution of probabilities associated with states and actions and the decoder neural network to determine a probability function for an action corresponding to the states and probabilities; providing obstacle feature data and start and goal locations for the new path to the encoder / decoder neural network system, including providing a point cloud or solid model representation of obstacles in a workspace of the new path to a three-dimensional convolutional neural network (3D CNN), extracting a feature vector characterizing the obstacle using the 3D CNN, and providing the feature vector to the encoder / decoder neural network system; calculating a waypoint distribution of the new path by the encoder / decoder neural network system, wherein the waypoint distribution is a set of candidate waypoints of the new path; and The new path is generated based on the waypoint distribution, wherein the new path defines the motion of the robot to move the tool center from a start position to a target position, wherein generating the new path includes using a rapidly exploring random tree (RRT) calculation or an optimization-based calculation, and includes performing a robot-obstacle collision avoidance calculation for each waypoint.

13. A system for human skill-based robot path generation, the system comprising: a robot communicating with a robot controller; as well as A computing device having a processor and a memory, the computing device being configured with one or more algorithms to perform steps comprising: accessing a training database comprising a plurality of human-generated robotic motion programs including data defining obstacles to be avoided during robotic execution of each motion program; Using the training database to train an encoder / decoder neural network system, which includes an encoder neural network and a corresponding decoder neural network; providing the encoder / decoder neural network system with obstacle signature data and start and goal locations for the new path; calculating a waypoint distribution of the new path by the encoder / decoder neural network system, wherein the waypoint distribution is a set of candidate waypoints of the new path; and generating the new path based on the waypoint distribution, wherein the new path defines the motion of the robot to move a tool center from the starting position to the target position, including performing a robot-obstacle collision avoidance calculation for each waypoint, Wherein, the new path is provided to the controller, and the robot executes the new path based on motion commands from the controller.

14. The system according to claim 13, wherein: The training database includes robot motion programs generated by one or more methods selected from the group consisting of: human-operated teach pendant programming, human-guided path demonstration by collaborative robots, and demonstration of robot operation by human grasping and manipulation of tools or workpieces.

15. The system of claim 13, wherein: The encoder neural network determines a distribution of probabilities associated with states and actions, wherein the states and actions are represented by the obstacles and corresponding steps in the plurality of motion programs in the training database.

16. The system of claim 15, wherein: The decoder neural network determines a probability function for an action corresponding to a state and a probability, wherein the action is defined as the candidate waypoint in the waypoint distribution.

17. The system of claim 16, wherein: Training the encoder / decoder neural network system includes using the training database as a labeled training data set, causing the encoder neural network to learn the distribution and causing the decoder neural network to learn the probability function that captures human path generation skills from the plurality of human-generated motion programs in the training database.

18. The system of claim 13, wherein: Providing obstacle feature data includes providing a point cloud or solid model representation of the obstacle in the workspace of the new path to a three-dimensional convolutional neural network (3D CNN), extracting a feature vector representing the obstacle using the 3D CNN, and providing the feature vector to the encoder / decoder neural network system.

19. The system of claim 13, wherein generating the new path based on the waypoint distribution comprises using a rapidly exploring random tree (RRT) computation, wherein the proposed waypoints evaluated in the RRT computation are selected from the candidate waypoints in the waypoint distribution.

20. The system of claim 13, wherein generating the new path based on the waypoint distribution comprises using an optimization calculation, wherein the optimization calculation minimizes an objective function, the objective function comprising a term that calculates a distance between the waypoints in the new path and an average position of the candidate waypoints in the waypoint distribution, wherein the objective function further comprises a term that calculates a total distance of the new path and a total time to execute the new path, and the optimization calculation further comprises an inequality constraint that stipulates that a minimum robot-obstacle distance at each waypoint is greater than a predetermined non-negative threshold.

Images