Route generation based human technique

Abstract

To provide a method for a robot route plan using a technique extracted from a movement program taught by humans that has been applied to new obstacle environments.SOLUTION: A three-dimensional convolutional neural network is used for extracting features characterizing obstacle environment, obstacle feature vector expression overcomes a problem encountered when using point group obstacle data. Obstacle feature data and robot route departure and goal points are provided to an encoder / decoder neural network system trained so as to extract a technique from database of movement programs generated by humans. The encoder / decoder neural network system generates a distribution of intermediate points for current obstacle environment and departure / goal points. The distribution of intermediate points is used for executing route generation with no final collision by using either a high-speed retrieval random tree (RRT) technique or a technique based on optimization.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present disclosure relates generally to the field of motion programming for industrial robots, and more particularly to a method for generating paths for a robot that uses a convolutional neural network to extract features characterizing an obstacle environment, an encoder / decoder neural network to extract techniques from a database of human-generated motion programs, and a final collision-free path generation from the distribution of waypoints. [Background technology]

[0002] The use of industrial robots to perform a wide range of repetitive manufacturing, assembly, and material moving operations is well known. A variety of techniques exist for teaching robots to move from a start point to a goal point. However, when obstacles exist between the start and goal points, all existing path generation techniques exhibit certain shortcomings.

[0003] One known technique for path generation is to use a teach pendant. The teach pendant communicates with the robot controller and is operated by a human operator. The teach pendant is used by the operator to instruct the robot to make incremental movements such as "move a little in the X direction" or "rotate the gripper around the local Z axis." The robot movements are recorded by the robot controller and stored as a motion program. These types of teach pendant commands and robot movements are sufficient for linear movements and simple paths with few or no obstacles. However, using a teach pendant to program a robot in a complex obstacle environment is often difficult, error prone, and time consuming.

[0004] Another known technique for path generation is to use a human-collaborative robot in a "lead-through" process. In the lead-through process, a human operator manually holds a tool or workpiece at the end of the robot arm and moves the tool or workpiece from a start point to a goal point. The lead-through process has the advantage of capturing human expertise in choosing paths, and also allows collision avoidance evaluation of all parts of the robot during the movement, which is important in applications where any part of the robot (not just the tool or workpiece) may come into contact with obstacles. Unfortunately, it may be difficult or impossible for a human operator to deftly manipulate the entire robot (including all intermediate joint locations) to avoid collisions when the obstacle environment is complex.

[0005] Robot teaching by human demonstration is also known, where a human demonstrator manually grasps a workpiece and moves it from a start position to a goal position. However, path generation by human demonstration can lack the positional accuracy required for precise movement of the workpiece, and path generation by human demonstration does not consider collision avoidance of the robot arm itself with obstacles in the workspace.

[0006] Automatic path generation techniques are also known in which start and finish points are provided along with a definition of the geometry of obstacles in the environment and a path generation computation is attempted. These automatic path generation techniques also suffer from certain drawbacks, which are discussed further below.

[0007] In view of the above, there is a need for improved robotic path generation techniques that capture the essence of techniques taught by human-generated paths and apply those techniques to generate collision-free paths in new obstacle environments. Summary of the Invention

[0008] This disclosure describes a method for robot path planning using techniques extracted from human-taught motion programs applied to new obstacle environments. A 3D convolutional neural network is used to extract features that characterize the obstacle environment, and a feature vector representation of the obstacles overcomes the problems encountered when using point cloud obstacle data. The obstacle feature data and the robot path start and goal points are provided to an encoder / decoder neural network that is trained to extract techniques from a database of human-generated motion programs. The encoder / decoder neural network generates a distribution of waypoints relative to the current obstacle environment and the start / goal points. The distribution of waypoints is used to perform the final collision-free path generation using either a fast-searching random tree (RRT) technique or an optimization-based technique.

[0009] Additional features of the presently disclosed apparatus and methods will become apparent from the following description and appended claims, taken in conjunction with the accompanying drawings. [Brief description of the drawings]

[0010] [Figure 1A] FIG. 1A is an illustration of automated route generation using the Rapid Searching Random Trees (RRT) technique, which is known in the art. [Figure 1B] FIG. 1B is an illustration of automated path generation using optimization-based techniques, as known in the art.

[0011] [Diagram 2] FIG. 2 is an illustration of a block diagram of a system for robotic path generation based on human techniques, according to an embodiment of the present disclosure.

[0012] [Diagram 3] FIG. 3 is an illustration showing how a point cloud can take on different mathematical representations when the order of the numbering of the points in the cloud is changed.

[0013] [Figure 4] FIG. 4 is an example block diagram illustrating extraction of features from obstacle point cloud data and using the feature vector data as input for path generation based on human techniques according to an embodiment of the present disclosure.

[0014] [Diagram 5] FIG. 5 is an example block diagram showing how an encoder and decoder neural network is used to generate a distribution of waypoints for a robot path using start and finish points and obstacle feature data as inputs along with a database of human-generated paths for training, according to an embodiment of the present disclosure.

[0015] [Figure 6] FIG. 6 is an example block diagram showing how the distribution of waypoints from FIG. 5 is used for collision-free path generation using either the RRT or optimization-based techniques, according to an embodiment of the present disclosure.

[0016] [Figure 7] FIG. 7 is an illustration of a block diagram of a system for robot path generation based on human techniques showing a robot and obstacle geometry from an example implementation according to an embodiment of the present disclosure.

[0017] [Figure 8] FIG. 8 is a flowchart diagram of a method for robot path generation based on human skills, including obstacle feature extraction and human skill extraction in a neural network system, and collision-free path generation from a distribution of waypoints, according to an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0018] The following discussion of disclosed embodiments directed to methods for robotic path generation based on human techniques is merely exemplary in nature and is not intended to limit in any way the disclosed apparatus and techniques or their applications or uses.

[0019] The use of industrial robots for a variety of manufacturing, assembly, and material moving operations is well known. It has long been a goal to develop a simple technique for generating robot motion programs that are efficient and avoid collisions with any obstacles present in the working environment. However, all existing path generation techniques exhibit certain shortcomings.

[0020] It has been recognized that human intuition and visualization are powerful tools that can be employed in robot path generation. Therefore, various techniques have been developed for path generation using human input. These techniques, including dexterity with a teaching pendant, human-collaborative robot lead-through, and human demonstration of workpiece pick-up and place operations (hereinafter referred to as "pick-and-place operations"), can be very effective in path generation for certain types of operations. However, when complex obstacle environments are involved, such as a robot mounted on one side of a workpiece that must reach the other side through an opening in the workpiece to perform an operation on the other side, these existing path generation techniques often lack the required capabilities to generate efficient and collision-free paths.

[0021] Automatic path generation techniques are also known in which start and finish points are provided along with a definition of the geometry of obstacles in the environment, and which attempt to automatically compute a collision-free path. Two such techniques are discussed below along with their limitations.

[0022] FIGURE 1A is an illustration of automated path generation using a rapid searching random tree (RRT) technique, and FIGURE 1B is an illustration of automated path generation using an optimization-based technique, both known in the art. FIGURE 1A illustrates an RRT path generation scenario 100. In scenario 100, a collision-free path is desired from a start point 110 to a goal point 112. A number of obstacles 120 exist in the workspace between start point 110 and goal point 112. An RRT-generated path 130 is shown partway through its growth.

[0023] As known to those skilled in the art, the RRT method proposes a new waypoint within a small distance from the previous pathpoint (or starting point 110) and evaluates the feasibility of a path segment from the previous pathpoint to the new waypoint. If there are no collisions in the path segment, the new waypoint is added to the path and other new waypoints are evaluated. Many branches develop in the RRT path, and eventually a complete path can be found from the starting point 110 to the goal point 112. However, paths generated by RRT have a characteristic artifact in their shape, with many short path segments that zigzag back and forth. For this and other reasons, paths generated using pure RRT techniques are often less desirable than paths generated by other methods.

[0024] In FIG. 1B, scenario 140, like scenario 100 discussed above, The scenario 140 includes a start point 110 and a goal point 112, and the same obstacle environment 120. In the scenario 140, optimization-based techniques are used to attempt to find a collision-free path from the start point 110 to the goal point 112. A path 150 extends from the start point 110 partway through the obstacle environment of the scenario 140. Optimization-based path generation methods can generate a path using an objective function that promotes smoothness and efficiency in routing, and constraints that ensure collision avoidance throughout the path. However, in complex obstacle environments, optimization-based methods may fail to find a feasible solution. This situation is illustrated in the scenario 140, where the path 150 extends to a point 152 where it gets stuck behind two of the obstacles 120 and is trapped by the external boundary of the workspace.

[0025] The techniques of this disclosure have been developed to overcome the limitations and shortcomings of existing path generation methods discussed above. The techniques discussed below capture the techniques embodied in human generated paths, using advanced computation to apply human path generation techniques to find collision-free paths through new obstacle environments.

[0026] FIG. 2 is an example block diagram of a system 200 for robot path generation based on human techniques, according to an embodiment of the present disclosure. In block 210, obstacle data for the workspace environment is provided along with start and finish point locations for the new path to be calculated. These inputs correspond to scenarios 100 and 140 shown in FIGS. 1A and 1B and discussed above. Instead of directly calculating a path, the inputs from block 210 are provided to an encoder / decoder block 220. The encoder / decoder block 220 includes an encoder and decoder neural network that is configured to extract human techniques from existing robot motion programs and apply those techniques to calculate a path in a new obstacle environment. The encoder / decoder block 220 is discussed in detail below.

[0027] A database 230 of existing motion programs is also provided to the encoder / decoder block 220. The database 230 contains robot motion programs that have been generated using any technique that incorporates human techniques. These techniques include, for example, the use of a teaching pendant to define a robot motion program as a series of incremental motions, the use of a human-cooperative robot by a human in a "lead-through" motion capture technique, and human demonstrations of workpiece pick-and-place operations. The database 230 contains a number of motion programs (robot paths) that have been taught by humans, and for each motion program, includes a definition of the obstacle environment that corresponds to the robot path. In this way, the database 230 characterizes how human experts solve motion programming problems for various obstacle environments. That is, the database 230 captures human techniques such as path curvature and obstacle avoidance distance when tracking a path from a start point to a goal point. The database 230 is used to train the encoder / decoder block 220. The human skills embodied in the database 230 are extracted by the encoder / decoder block 220 and are ultimately used to generate paths for new obstacle environments, as discussed below.

[0028] The encoder / decoder block 220 generates a waypoint distribution 240 that includes a distribution of waypoints (a type of cloud path) in the obstacle-containing workspace environment defined in the input block 210. The waypoint distribution 240 is then used to generate a collision-free path 250 in one of two ways. The system 200 captures the path generation techniques of human experts and applies those techniques to chart paths through new obstacle environments to overcome the limitations of existing path generation methods, employing the power of automated path generation techniques. Each of the elements of the system 200 and the interactions between them are discussed further below.

[0029] FIG. 3 is an illustration showing how a point cloud takes on different mathematical representations when the order of numbering of the points in the cloud is changed. Many existing path generation techniques use a point cloud representation of an obstacle environment. As known to those skilled in the art, a point cloud is a collection of points that represent an object or objects, where the points are typically defined in a three-dimensional (3D) coordinate frame. The points in the point cloud can be located only on the exterior surface of the object, or the points can be located both on the exterior and interior of the object.

[0030] Point clouds are a convenient form of three-dimensional obstacle representation, but they exhibit several problematic properties in subsequent computations in which they are used. For example, consider a simple point cloud 310 that includes five points numbered 1 through 5 in a spatial environment. Point cloud 320 includes five points in the same locations in the spatial environment as the points in point cloud 310, but the points in point cloud 320 are numbered 1 through 5 in a different order. Point cloud 310 corresponds to coordinate matrix 330, where point 1 in coordinate matrix 330 has coordinates (x1, y1, z1), and so on. Point cloud 320 corresponds to coordinate matrix 340, where point 1 in coordinate matrix 340 has coordinates (x1, y1, z1), which are different from (x1, y1, z1) in coordinate matrix 330. In other words, even though point cloud 310 and point cloud 320 define five identically located points in space, coordinate matrix 330 is completely different from coordinate matrix 340. As such, point clouds do not exhibit permutation invariance, i.e., the definition of a point cloud varies based on the order in which the items (in this case, points) are listed.

[0031] Another property of point clouds is that they do not exhibit transformation invariance: a point cloud representing an object oriented in one direction (e.g., "right side up") is quite different from a point cloud representing the same object oriented in the other direction (e.g., "upside down"). Both permutation variability and transformation variability lead to problems and inefficiencies in the use of point clouds for obstacle definition, as both these phenomena make it difficult or impossible for objects to be recognized and processed efficiently, especially in machine learning systems.

[0032] FIG. 4 is an example block diagram illustrating feature extraction from obstacle point cloud data and using the feature vector data as input to human skill based path generation according to an embodiment of the present disclosure. In FIG. 2, block 210 was shown as providing obstacle data input along with definition of start and finish point locations. FIG. 4 illustrates how obstacle data is provided as input to the system of FIG. 2 to overcome the problems with point cloud data discussed above.

[0033] The point cloud 410 represents an object (in this example, a chair, which can be thought of as an obstacle) in a known manner and as discussed above. A 3D convolutional neural network (CNN) 420 is used to extract features from the obstacle point cloud 410. A feature extractor such as the CNN 420 takes a point cloud or other data (such as a solid model or an image) as input and provides as output a set of feature vectors that characterize the input. The feature extractor CNN 420 dramatically reduces the amount of data required for subsequent processing by replacing a point cloud with thousands of point coordinates with a set of numerically defined feature vectors that only need a few hundred. The feature extractor CNN 420 also overcomes the problems inherent in defining obstacles by point clouds, discussed above, by using a feature vector representation of the obstacles rather than the raw point cloud data.

[0034] As will be appreciated by those skilled in the art, CNNs are a common architecture used for feature extraction, and in fact pre-trained feature extractor CNNs are available where the network structure, including the number of layers and the output size of each layer, is fixed and the network parameters are trained. The feature extractor CNN 420 may be this type of pre-trained feature extractor CNN. The final output of the CNN 420, which is used as input to the encoder / decoder block 220 (from FIG. 2), is a set of feature vectors, shown in block 430. The output, shown in block 430, is an array of feature vectors, where the array has a size of width “X” and depth “Y”, and each feature vector has dimensions of {NN×1}. The number of feature vectors (X·Y) and the dimensions of each feature vector can be selected as appropriate for a particular application, corresponding to the design of the CNN 420.

[0035] Feature vectors from block 430 characterizing the obstacles from the point cloud 410 are provided as inputs to the encoder / decoder block 220 shown above in Figure 2. In block 440, the locations of start and goal points for the robot path to be generated are also provided as inputs to the encoder / decoder block 220. The encoder / decoder block 220 is trained on a database 230 of existing motion programs and then uses the obstacle and start / goal inputs from blocks 430 and 440 to compute the waypoint distribution 240 described above with respect to Figure 2. Training and computation of the waypoint distribution are discussed below.

[0036] FIG. 5 is an example block diagram showing how an encoder and decoder neural network is used to create a distribution of waypoints for a robot path from a human skill extraction according to an embodiment of the present disclosure. The encoder / decoder block 220 is shown expanded in FIG. 5 and receives as inputs obstacle feature vectors 430 and start and finish points in block 440. A database of human-generated motion programs 230 along with corresponding obstacle data are also provided as inputs. The obstacle and motion data from the human-generated path database are used to train an encoder and decoder neural network to capture human skills, where the encoder defines a Gaussian distribution of probabilities associated with a set of state (start and finish point locations, obstacles) and action (motion) data, and the decoder determines actions (waypoints) associated with a set of state (start and finish point locations, obstacles) data and corresponding probabilities.

[0037] The encoder / decoder block 220 includes an encoder neural network 510. Data from a database 230 of human-generated motion programs provides a set of corresponding "state" and "action" data (s0,a1), (s1,a2), (s2,a3), etc., that is used to train the encoder neural network 510. i Each of the obstacles is associated with a corresponding "action" a from one of the motion programs in the database 230, along with the location of the start and finish points. i+1are the motion characteristics that are the result of states (waypoints from the motion program). The encoder neural network 510 generates a distribution q (shown in 520) of probabilities z associated with states s and actions a. The distribution q(z|s,a) shown in 520 captures human technique from human-generated motion programs in the database 230. The database 230 of human-generated motion programs contains obstacle environment data and corresponding human-generated motion programs and serves as tagged data for training the encoder neural network 510 and the decoder neural network 530 to indicate desired path generation behavior.

[0038] Then, after training, the decoder neural network 530 is used to determine the actions a corresponding to state s and probability z. This is done using a probability function π(a|s,z) as shown on line 540. The decoder neural network 530 receives as input the feature vector 430 that defines the obstacle environment for the path to be generated along with the start and finish point locations of the path to be generated from block 440. Then, using the probability function π, the decoder neural network 530 calculates a set of actions a (waypoints) corresponding to state s (start and finish point locations and obstacles) and probability z. This results in a waypoint distribution 240 that is not a complete or even final path (movement program), but a distribution of waypoints based on the obstacle environment of the path being generated and the probabilities encoded in the human skill from the database 230.

[0039] Training of the encoder / decoder block 220 is accomplished using a database 230 of human-generated motion programs, including obstacle environments for each of the motion programs. Using the database 230 as a tagged data set for training, the encoder neural network 510 learns a distribution q that captures human technique, while the decoder learns a probability function π that generates a waypoint distribution along the motion programs used as input. Training of the encoder neural network 510 can be accomplished using known loss function approaches, or other techniques as deemed most appropriate.

[0040] To summarize what has been discussed above, a database 230 of human-generated motion programs (including the obstacle environment for each of the motion programs) is used to train the Encoder Neural Network 510 and the Decoder Neural Network 530 to capture human path generation techniques applied to a number of different obstacle scenarios. Once trained, the Decoder Neural Network 530 then receives the start / goal locations and obstacle environment (obstacle feature vector data) for a new path to be generated, and uses the distribution q to calculate a probability function π to generate a waypoint distribution 240, which is a set of points along the path that extends from the start location to the goal location while generally avoiding obstacles along the path.

[0041] In an experimental evaluation of the method disclosed herein, the training data set (in the database 230) included 300 motion programs (each with a corresponding obstacle environment). This training data set demonstrated the ability to sufficiently train the encoder / decoder block 220. After training with the 300 motion program data set, a test data set of 50 obstacle environments was provided for evaluation, and waypoint distributions were calculated for each of the 50 examples in the test data set. More than half of the test examples resulted in collision-free waypoint distributions through the obstacle environment of the particular example. However, the remainder of these test examples resulted in waypoint distributions that included some waypoint interference with the obstacles. Therefore, even though the waypoint distributions 240 did a very good job of capturing the overall motions representative of human skill in the training database, additional method steps are required to generate reliable collision-free paths for each start / goal constraint and obstacle environment.

[0042] 6 is an example block diagram showing how the waypoint distribution from FIG. 5 is used for collision-free path generation using either the RRT or optimization-based techniques, according to an embodiment of the present disclosure. As explained above, the waypoint distribution 240 is not guaranteed to be collision-free along the path from the start point to the goal point. Therefore, the techniques disclosed herein add a method step of collision-free path generation from the waypoint distribution 240.

[0043] The first technique for collision-free path generation from waypoint distribution 240 uses the RRT method. This is shown on the upper track ((1)) in FIG. 6. In block 610, waypoint distribution 240 is shown with the start of the RRT path shown in 612. Unlike conventional RRT path generation as discussed with respect to FIG. 1A, the path generation technique of the present disclosure does not randomly select proposed new waypoints from anywhere in the workspace. Instead, the disclosed path generation technique selects proposed new waypoints only from waypoint distribution 240. This ensures that the path generated by the RRT follows the path laid out by the encoder / decoder block 220, which captures the human path generation technique. It also makes the RRT path generation very efficient because most of the proposed new waypoints are collision-free, so the RRT path is quickly constructed from the start point to the goal point according to the waypoint distribution based on the human technique while avoiding all obstacles along the path (because each RRT path segment checks for collision avoidance). The resulting collision-free path 250, after all branching paths that do not yield good results have been removed, is shown on the right side of FIG. 6, superimposed on the waypoint distribution 240.

[0044] A second technique for collision-free path generation from the waypoint distribution 240 uses an optimization calculation. This is shown on the lower track ((2)) in FIG. 6. In block 620, the waypoint distribution 240 is used in an optimization calculation as follows. An optimization model is defined using an objective function that penalizes deviations from the waypoint distribution along with inequality constraints that ensure the path remains collision-free. A preferred, non-limiting embodiment of the objective function is defined in equation (1) below. min[λ1(wp robot -wp human )+λ2Σd i +λ3Σt i ] (1) Here, wp robot is the waypoint of the robot on the path being calculated, and wphuman is the waypoint distribution 240 provided by the encoder / decoder block 220 that captures human path generation techniques, and Σd i is the total distance of the generated route (the sum of the distances to each route point i), and Σt i is the total cycle time of the robot's movement using the generated path, and λ1-λ3 are weighting factors. The first term of the objective function in equation (1) serves to minimize the deviation of the computed robot path from the waypoint distribution 240, where the center or mean position of the waypoint distribution is preferably used in this calculation. The weighting factors λ1-λ3 can be selected to achieve the path generation performance desired in any given application.

[0045] Along with the objective function of Equation (1), the optimization model includes at least one constraint, such as an inequality constraint that calculates the minimum distance between the robot and any obstacle for each iteration of the waypoint calculation, which constraint dictates that the minimum distance be greater than a predefined threshold (the threshold may be zero or some positive distance value, such as 15 mm). In this way, each robot path point i is required to be collision-free before it is added to the path. In addition to the collision avoidance inequality constraint, other constraints can be included in the optimization model, such as constraints on robot joint locations that ensure that the robot configuration at each path point i is feasible.

[0046] The optimization model defined above ensures that the generated path follows the waypoint distribution laid out by the encoder / decoder block 220, which captures human path generation techniques. The optimization model also ensures that the generated path is highly efficient while avoiding all obstacles along the path. The resulting collision-free path from the optimization-based path generation is also shown at 250 in FIG. 6, superimposed on the waypoint distribution 240.

[0047] 7 is a schematic illustration of a system 700 for robot path generation based on human techniques showing the robot and obstacle geometries from an example implementation according to an embodiment of the present disclosure. FIG. 7 is provided as a more realistic visual example of how the methods and systems discussed above (with respect to FIGS. 2 and 4-6) are applied to an actual robot in a complex 3-D obstacle environment, rather than simply tracing a curve through a simple group of 2-D obstacles.

[0048] In the first step (shown as (1)), a database 710 of human-generated robot motion programs is used to train an encoder / decoder neural network system 720. This step is described in detail with respect to Fig. 5, which shows that the database 710 (equivalent to database 230) contains a large number (e.g., hundreds) of motion programs created by human teaching methods, along with corresponding obstacle data for each motion program, and this data is used to train the encoder / decoder system 720 (via the distribution q(z|s,a) and probability function π(a|s,z)) to mimic human skills in defining path points to avoid obstacles in a spatial environment.

[0049] In the second step (shown as (2)), the start and goal locations for the new path to be generated are provided to a trained encoder / decoder neural network system 720 along with the obstacle environment through which the path must be found. The obstacle geometry data is provided to a 3D CNN 730 which extracts obstacle feature data, and the obstacle feature data, rather than the raw obstacle geometry or point cloud data, is provided to the encoder / decoder system 720. This step is described in detail with respect to FIG. 4. An illustration of a robot and a door frame is provided in box 702. This shows an example of the type of practical robot path generation scenario in which the disclosed method and system can be applied. In this example, the start point has a tool center point located on the "near" side of the door frame (the same side on which the robot base is mounted), while the goal point has a tool center point located on the "far" side of the door frame. This may represent the type of path that needs to be generated for a spot welding robot, for example, when welding operations are required on both sides of the workpiece (door frame). Furthermore, the door frame itself is an obstacle that must be avoided by the robot. There may be other obstacles in the workspace environment.

[0050] In the third step (shown as (3)), the start and goal locations and obstacle feature data are used by the decoder neural network to generate a waypoint distribution 740. This step is also described above with respect to FIG. 5. The waypoint distribution 740 includes a distribution of points calculated by a probability function π based on the start / goal locations and the obstacle environment. In the example shown, the waypoint distribution 740 follows a path through an opening or gap in a door frame. The waypoint distribution 740 embodies human techniques for collision-free path generation captured in the encoder / decoder system 720. The final step of path generation is included since the waypoint distribution 740 is not guaranteed to be collision-free.

[0051] In the fourth step (shown as (4)), a collision-free path 750 is generated from the waypoint distribution 740. This step was previously described with respect to FIG. 6. The collision-free path 750 can be calculated from the waypoint distribution 740 using either the RRT method or the optimization-based method, as discussed above. Both the RRT method and the optimization-based method use the waypoint distribution 740 in selecting points along the final collision-free path.

[0052] With reference to the example provided in FIG. 7, several important points should be noted. First, the obstacles included in the robot motion programming are typically three-dimensional objects that may have complex shapes (such as the door frames of an automobile) and may also include fixtures and platform-shaped structures with simpler prismatic shapes. In addition, each waypoint or robot pathpoint is not just a point in two- or three-dimensional space, but a point defined by the complete robot configuration, including the locations of all joints of the robot arm. Thus, the collision avoidance calculation (in either the RRT method or the optimization-based method for path generation) calculates the minimum three-dimensional distance between all parts of the robot arm (including the tool and any workpiece that the tool may be gripping) and all obstacles. The minimum distance calculation can be performed in any suitable manner, such as using a signed distance field method, or other methods as appropriate. While the previous figures showed simplified examples for the purposes of explaining the concept, FIG. 7 shows the complexity that is actually involved in all the data and calculations.

[0053] 8 is a flow chart diagram 800 of a method for human skill-based robot path generation including obstacle feature extraction and human skill extraction in a neural network system, and collision-free path generation from a distribution of waypoints, according to an embodiment of the present disclosure. In box 802, a training database of human-generated motion programs is provided. This is the database 230 (or 710) discussed above, and includes robot motion programs generated using any technique incorporating human skills, such as using a teaching pendant to define a robot motion program as a sequence of incremental movements, using a human-cooperative robot in a "lead-through" motion capture technique, and human demonstration of a workpiece pick-and-place operation. The database 230 includes a plurality of motion programs (robot paths) taught by humans, and for each motion program, a definition of the obstacle environment corresponding to the robot path.

[0054] In box 804, the training database is used to train the encoder / decoder neural network system. As discussed above, the encoder / decoder neural network system (block 220) learns to encode or extract human techniques for motion program generation when encountering an obstacle environment. This training teaches the encoder / decoder neural network system (via the distribution q(z|s,a) and probability function π(a|s,z)) to mimic human techniques in defining path points that avoid obstacles in a spatial environment.

[0055] Start and goal locations and obstacle feature data are provided for the new path to be generated in box 806. In a preferred embodiment, a three-dimensional convolutional neural network (3D CNN) is used to extract feature vector data from the obstacle three-dimensional geometry data (e.g., point cloud or solid model).

[0056] In box 808, a waypoint distribution is computed for the new path by the encoder / decoder neural network system. Specifically, the trained decoder neural network uses a probability function π(a|s,z) to compute the waypoint distribution based on the start and goal locations and obstacle feature data for the new path. The decoder neural network is trained in conjunction with the encoder neural network to encode human skill in the distribution q, so that the resulting waypoint distribution mimics human skill in defining path points to navigate while avoiding obstacles in a spatial environment.

[0057] In box 810, a collision-free path is generated from the waypoint distribution. The collision-free path generation uses the waypoint distribution as a guideline and includes a full collision checking analysis (for the entire robot) at each path point. The collision-free path can be generated using an RRT method in box 812 or an optimization-based method in box 814. The resulting collision-free path embodies the human skill for finding a path through an obstacle environment, as taught by the training database. The collision-free path is a complete robot motion program (defining all joint positions at each path point) and can be easily converted to a complete motion program by inverse kinematics calculations, as will be appreciated by those skilled in robotics.

[0058] Throughout the above discussion, various computers and controllers are described and implied. It should be understood that these computer and controller software applications and modules are executed on one or more computing devices having a processor and memory modules. In particular, the computing device includes a processor in a computer or robot controller, the controller / computer configured to perform human skill based path generation calculations in the manner discussed above. The computing device can include specialized computing devices that are specifically configured for execution of 3D CNN and / or encoder / decoder neural network systems.

[0059] As outlined above, the disclosed techniques for robot path generation based on human techniques offer several advantages over existing robot path generation methods: the disclosed techniques capture the intuition of human path generation in a trained neural network, use feature vector representations of obstacles for improved consistency, apply human techniques in waypoint distribution generation for new paths, and perform path generation from waypoint distributions to ensure that the final path is collision-free.

[0060] While numerous example aspects and embodiments of human technology based route generation have been discussed above, those skilled in the art will recognize modifications, permutations, additions, and subcombinations thereof, and it is therefore intended that the following appended claims and any subsequent introduced claims be construed as including all such modifications, permutations, additions, and subcombinations as are within their true spirit and scope.

Claims

1. 1. A method for human technique based robot path generation, comprising: providing a training database containing a plurality of human-generated robot motion programs including data defining obstacles to be avoided during robotic execution of each motion program; training an encoder / decoder neural network system including an encoder neural network and a corresponding decoder neural network running on a computing device using the training database; providing obstacle feature data and start and finish locations for a new path to said encoder / decoder neural network system; calculating, for the new path, a waypoint distribution, the waypoint distribution being a set of candidate waypoints for the new path, by the encoder / decoder neural network system; generating the new path based on the waypoint distribution, the new path defining a movement for the robot to move the center of the tool from the start position to the goal position, the new path including performing a robot-obstacle collision avoidance calculation for each path point.

2. 2. The method of claim 1, wherein the training database includes robot motion programs generated by one or more methods selected from the group consisting of programming by a human operated teach pendant, path demonstration by human lead-through of a human collaborative robot, and demonstration of robot motion by human grasping and manipulating a tool or workpiece.

3. 2. The method of claim 1 , wherein the encoder neural network determines distributions of probabilities associated with states and actions represented by the obstacles in the training database and corresponding steps in the motion program.

4. 4. The method of claim 3, wherein the decoder neural network determines a probability function of actions corresponding to states and probabilities defined as the candidate waypoints in the waypoint distribution.

5. 5. The method of claim 4, wherein training the encoder / decoder neural network system includes training a pre-encoder neural network to learn the distribution using the training database as a labeled training data set, and training the decoder neural network to learn the probability function that captures human path generation techniques from the motion programs generated by humans in the training database.

6. 2. The method of claim 1, wherein providing obstacle feature data comprises providing a point cloud or solid model representation of obstacles in a workspace of the new path to a three-dimensional convolutional neural network (3D CNN), extracting feature vectors characterizing the obstacles using the 3D CNN, and providing the feature vectors to the encoder / decoder neural network system.

7. 2. The method of claim 1, wherein generating the new route based on the waypoint distribution includes using a Rapid Searching Random Trees (RRT) calculation, and the proposed waypoints evaluated in the RRT calculation are selected from the candidate waypoints in the waypoint distribution.

8. 2. The method of claim 1 , wherein generating the new route based on the waypoint distribution includes using an optimization calculation to minimize an objective function that includes a term that calculates a distance between the route points in the new route and an average position of the candidate waypoints in the waypoint distribution.

9. 9. The method of claim 8, wherein the objective function also includes terms that calculate a total distance of the new path and a total time for execution of the new path, and the optimization calculation further includes an inequality constraint that dictates that a minimum distance between the robot and an obstacle at each path point is greater than a predetermined non-negative threshold.

10. 2. The method of claim 1, wherein for each path point in the new path, a robot configuration is calculated using inverse kinematics calculations, and the robot configuration is used in collision avoidance calculations between the robot and an obstacle.

11. providing the new path to a controller of the robot; 2. The method of claim 1, further comprising executing the new path with the robot based on movement commands from the controller.

12. 1. A method for human technique based robot path generation, comprising: providing a training database containing a plurality of human-generated robot motion programs including data defining obstacles to be avoided during robotic execution of each motion program; training an encoder / decoder neural network system running on a computing device, the encoder / decoder neural network system including an encoder neural network and a corresponding decoder neural network, using the training database to train the encoder neural network to learn distributions of probabilities associated with states and actions, and the decoder neural network to determine probability functions of actions corresponding to the states and probabilities; providing obstacle feature data and start and finish positions for the new path to the encoder / decoder neural network system, including providing a point cloud or solid model representation of obstacles in a workspace of the new path to a three-dimensional convolutional neural network (3D CNN), extracting feature vectors characterizing the obstacles using the 3D CNN, and providing the feature vectors to the encoder / decoder neural network system; calculating, for the new path, a waypoint distribution, the waypoint distribution being a set of candidate waypoints for the new path, by the encoder / decoder neural network system; generating the new path defining movements for a robot to move a center of a tool from the start position to the goal position based on the waypoint distribution, the new path defining movements for a robot to move a center of a tool from the start position to the goal position ...

13. 1. A system for human technique based robot path generation, comprising: a robot in communication with a robot controller; A computing device having a processor and a memory, accessing a training database containing a plurality of human-generated robot motion programs including data defining obstacles to be avoided during robotic execution of each motion program; training an encoder / decoder neural network system including an encoder neural network and a corresponding decoder neural network using the training database; providing obstacle feature data and start and finish locations for a new path to said encoder / decoder neural network system; calculating, for the new path, a waypoint distribution, the waypoint distribution being a set of candidate waypoints for the new path, by the encoder / decoder neural network system; generating the new path based on the waypoint distribution, the new path defining a movement of the robot to move a tool center from the start position to the goal position, the new path including performing a robot-obstacle collision avoidance calculation for each waypoint; The new path is provided to the controller, and the robot executes the new path based on movement commands from the controller.

14. 14. The system of claim 13, wherein the training database contains robot motion programs generated by one or more methods selected from the group consisting of programming by a human operated teach pendant, path demonstration by human lead-through of a human collaborative robot, and demonstration of robot manipulation by human grasping or dexterity of a tool or workpiece.

15. 14. The system of claim 13, wherein the encoder neural network determines distributions of probabilities associated with states and actions represented by the obstacles in the training database and corresponding steps in the motion program.

16. 16. The system of claim 15, wherein the decoder neural network determines a probability function of actions corresponding to states and probabilities, the actions being defined as the candidate waypoints in the waypoint distribution.

17. 17. The system of claim 16, wherein training the encoder / decoder neural network system includes using the training database as a labeled training database, training the encoder neural network to learn the distribution, and training the decoder neural network to learn the probability function that captures human path generation techniques from the motion programs generated by humans in the training database.

18. 14. The system of claim 13, wherein providing obstacle feature data includes providing a point cloud or solid model representation of obstacles in a workspace of the new path to a three-dimensional convolutional neural network (3D CNN), extracting feature vectors characterizing the obstacles using the 3D CNN, and providing the feature vectors to the encoder / decoder neural network system.

19. 14. The system of claim 13, wherein generating the new route based on the waypoint distribution includes using a Rapid Searching Random Tree (RRT) calculation, and proposed waypoints evaluated in the RRT calculation are selected from the candidate waypoints in the waypoint distribution.

20. 14. The system of claim 13, wherein generating the new path based on the waypoint distribution includes using an optimization calculation to minimize an objective function including terms that calculate distances between the path points in the new path and an average location of the candidate waypoints in the waypoint distribution, the objective function also including terms that calculate a total distance of the new path and a total time for execution of the new path, and the optimization calculation further includes an inequality constraint that dictates that a minimum distance between a robot and an obstacle at each path point is greater than a predetermined non-negative threshold.

Images