Path dataset generation

Abstract

To provide a method for generating a dataset of robot motion programs for training a path generation neural network.SOLUTION: A large language model uses a robot task library and an asset geometry database as inputs. The large language model breaks down a task into steps, and then generates code describing a robot and object motion to complete the task. The generated code produces robot motions for the task, and a corresponding robot motion program is created and executed in simulation. The simulated robot motion programs are used to generate collision-free robot paths via RRT and / or optimization, and the collision-free paths are validated for robot reachability and object placement success. The validated motion programs are added to a dataset and used for training a path generation neural network.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present disclosure relates generally to the field of industrial robot motion programming, and more particularly to a method for generating a dataset of robot motion programs that can be used to train a path generation neural network, using a large language model to generate code that configures a task environment and creates a robot simulation, checking the simulated robot motion programs for collision avoidance, and adding the verified robot motion programs to the dataset. [Background technology]

[0002] Discussion of related technologies The use of industrial robots to perform a wide range of manufacturing, assembly, and material transfer tasks is well known. Various techniques exist for teaching robots to move from a start point to a destination point. However, all existing path generation techniques exhibit certain drawbacks when obstacles exist between the start point and the destination point.

[0003] One known technique for path generation is the use of a teach pendant. The teach pendant communicates with the robot controller and is operated by a human operator to instruct the robot on incremental movements such as "jog in the X direction" or "rotate the gripper around the local Z axis." However, using a teach pendant to program a robot in a complex obstacle environment often proves difficult, error-prone, and time-consuming. Another known technique for path generation is the use of a collaborative robot in a "lead-through" process, in which a human operator manually grasps the end of the robot arm and moves the tool or workpiece from a start point to a target point. The lead-through process has the advantage of capturing human expertise in path selection and also allows for collision-avoidance evaluation of all parts of the robot during operation. Unfortunately, it may be impossible for an operator to manipulate the entire robot to avoid collisions in a complex obstacle environment.

[0004] Robot teaching by human demonstration is also known, in which a human demonstrator manually grasps a workpiece and moves it from a start position to a target position using visual motion capture. However, path generation by human demonstration may lack the positional accuracy required for accurate movement of the workpiece, and path generation by human demonstration does not consider collision avoidance between the robot arm itself and obstacles in the workspace.

[0005] Automatic path generation techniques are also known in which a start point and a destination point are provided along with a geometric definition of obstacles in the environment, and a path generation calculation is performed. One path generation technique uses an encoder / decoder neural network model to calculate waypoints based on the obstacle environment and the path start / end points. However, the database of motion programs required to train the encoder / decoder neural network is difficult and time-consuming to create. Summary of the Invention [Problem to be solved by the invention]

[0006] In light of the above, there is a need for improved robot path generation techniques that can automatically generate a large number of collision-free paths to populate the dataset used to train the encoder / decoder neural network. [Means for solving the problem]

[0007] This disclosure describes a method for automatically generating a dataset of robot motion programs that can be used to train a path-generation neural network. A large language model is used to generate code that breaks down and configures a task environment to create a robot simulation. The large language model uses a robot task library and an asset geometry database as input. Based on task and asset inputs and task instructions, the large language model breaks down the task into steps and then generates code that describes the robot and object movements to complete the task. The generated code generates robot movements for the task, and a corresponding robot motion program is created and executed in simulation. The simulated robot motion program is used to generate a collision-free robot path via RRT or optimization-based calculations, and the collision-free path is verified for robot reachability and object placement success. The verified robot motion program is added to the dataset used for subsequent training of the path-generation neural network.

[0008] Further features of the devices and methods of the present disclosure will become apparent from the following description and appended claims, taken in conjunction with the accompanying drawings. [Brief explanation of the drawings]

[0009] [Figure 1A] FIG. 1A is a diagram illustrating automatic path generation using rapidly-exploring random tree (RRT) technology. [Figure 1B] FIG. 1B is a diagram illustrating automatic path generation using optimization-based techniques as known in the art. [Figure 2] FIG. 2 is a block diagram illustrating a system for robot path generation according to an embodiment of the previous disclosure. [Figure 3] FIG. 3 is a flowchart illustrating a technique for automatic path dataset generation that uses a large language model to generate code that simulates a robotic task, where validated paths are added to a training database, according to an embodiment of the present disclosure. [Figure 4] FIG. 4 is a diagram illustrating the inputs and outputs of the large language model of the path dataset generation pipeline of FIG. 3, including asset and task library inputs along with task instructions, and task breakdown and code generation output, according to an embodiment of the present disclosure. [Figure 5] FIG. 5 illustrates the simulation and path generation steps of the path dataset generation pipeline of FIG. 3 and validation of the generated path, according to an embodiment of the present disclosure. [Figure 6] FIG. 6 is a block diagram illustrating a system for automatic path dataset generation, training of an encoder / decoder neural network using the generated dataset, and robot path generation and execution, according to an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0010] The following discussion of embodiments of the present disclosure directed to methods for automated robotic path dataset generation is merely exemplary in nature and is in no way intended to limit the disclosed devices and techniques or their applications or uses.

[0011] The use of industrial robots for a variety of manufacturing, assembly, and material transfer tasks is well known. It has long been a goal to develop simple techniques for generating efficient robot motion programs that avoid collisions with any obstacles present in the work environment. However, existing path generation techniques all exhibit certain drawbacks.

[0012] Human intuition and visualization are recognized as powerful tools that can be used in robot path generation. Therefore, various techniques for path generation using human input have been developed. These techniques, including teach pendant manipulation, collaborative robot lead-through, and human demonstration of workpiece pick-and-place tasks, can be very effective in path generation for certain types of tasks. However, when complex obstacle environments are involved, such as when a robot is attached to one side of a workpiece or must perform a task on the other side through an aperture in the workpiece, these existing path generation techniques often fall short of the capabilities required to generate efficient, collision-free paths.

[0013] Automatic path generation techniques are also known in which a start point and a destination point are provided along with a geometric definition of obstacles in the environment and an attempt is made to automatically calculate a collision-free path. Two such techniques, along with their limitations, are discussed below.

[0014] Figure 1A illustrates automated path generation using a rapid-searching random tree (RRT) technique, and Figure 1B illustrates automated path generation using an optimization-based technique as known in the art. Figure 1A shows an RRT path generation scenario 100. In scenario 100, a collision-free path is desired from a start point 110 to a target point 112. A number of obstacles 120 exist in the workspace between start point 110 and target point 112. A path 130 generated by the RRT is depicted midway through its growth process.

[0015] As known to those skilled in the art, RRT techniques propose new waypoints within incremental distances from a previous path point (or starting point 110) and evaluate the feasibility of a path segment from the previous path point to the new waypoint. If the path segment is collision-free, a new waypoint is added to the path, and another new waypoint is evaluated. Many branches develop within the RRT path, and eventually, a complete path from the starting point 110 to the destination point 112 can be found. However, paths generated by RRT are characteristically unnatural in shape, with many short path segments that zigzag back and forth. For this and other reasons, paths generated using pure RRT techniques often prove less desirable than paths generated by other methods.

[0016] In FIG. 1B , scenario 140 includes the same start point 110 and destination point 112 as scenario 100 described above, as well as the same environment of obstacles 120. In scenario 140, optimization-based techniques are used to attempt to find a collision-free path from start point 110 to destination point 112. Path 150 extends from start point 110 partway through the obstacle environment of scenario 140. Optimization-based path generation methods can generate paths using objective functions that promote smoothness and efficiency in path routing and constraints that ensure collision avoidance throughout the path. However, in complex obstacle environments, optimization-based methods may not be able to find a feasible solution. This situation is illustrated in scenario 140, where path 150 dead-ends behind two of the obstacles 120 and extends to point 152, where it is bounded by the outer boundary of the workspace.

[0017] 1A and 1B are shown in two dimensions for clarity, it should be appreciated that both the RRT and optimization-based path generation techniques are applicable to three-dimensional objects having six degrees of freedom. In fact, the RRT and optimization techniques can be applied to the entire robot, with all parts of the robot arm and workpiece being checked for interference in a three-dimensional obstacle environment.

[0018] While the RRT and optimization-based techniques described above can have difficulty finding collision-free paths through complex obstacle environments, either or both of these techniques can be useful in combination with other methods for path generation, as discussed further below.

[0019] Another automated path generation technique is disclosed in U.S. patent application Ser. No. 18 / 355,914, entitled "HUMAN SKILL BASED PATH GENERATION," filed July 20, 2023, which is commonly assigned with the present application and is incorporated herein by reference in its entirety. The aforementioned patent application (hereinafter the "'914 application") discloses a technique that uses an encoder-decoder neural network trained to calculate a distribution of waypoints through an obstacle environment and performs additional calculations to generate a collision-free path for a robot moving a tool or workpiece.

[0020] FIG. 2 is a block diagram illustrating a system 200 for robot path generation as disclosed in the '914 application. In block 210, obstacle data for the workspace environment is provided along with the locations of start and destination points for a new path to be calculated. These inputs correspond to scenarios 100 and 140 shown in FIGS. 1A / 1B and discussed above. Instead of directly calculating a path, the inputs from block 210 are provided to an encoder / decoder block 220. The encoder / decoder block 220 includes an encoder and decoder neural network configured to extract skills from an existing robot motion program and apply those skills to calculate a path in a new obstacle environment. The encoder / decoder block 220 is discussed in detail in the '914 application.

[0021] A database 230 of existing motion programs is also provided to the encoder / decoder block 220. The database 230 contains robot motion programs generated using any suitable technique, such as human demonstration. The database 230 includes multiple motion programs (robot paths) and, for each motion program, a definition of the obstacle environment corresponding to the robot path. In this way, the database 230 characterizes how human experts or other suitable training sources solve motion programming problems for various obstacle environments. That is, the database 230 captures behaviors such as path curvatures and obstacle avoidance distances when tracing a path from a start point to a goal point. The database 230 is used to train the encoder / decoder block 220. The skills embodied in the database 230 are extracted by the encoder / decoder block 220 and ultimately used to generate paths for new obstacle environments.

[0022] The encoder / decoder block 220 generates a waypoint distribution 240 that includes a distribution of waypoints (cloud path types) in the obstacle-containing workspace environment defined in the input block 210. The waypoint distribution 240 is then used to generate a collision-free path 250 using either the RRT technique or the optimization-based technique described above. The system 200 captures the path generation skills embodied in the database 230 and applies those skills to chart paths through new obstacle environments, using the power of automatic path generation technology in a way that overcomes the limitations of existing path generation methods.

[0023] In an ideal situation, the database 230 would be populated with robot motions from human demonstrations, with collision-free robot paths that capture the human skill and intuition in moving a robot from a start point to a goal point through a complex obstacle environment. However, generating a sufficient number of motion programs through human demonstrations for training the encoder / decoder block 220 can be difficult and time-consuming. The techniques of this disclosure are developed to automatically generate a large number of collision-free robot paths, thus providing a convenient method of populating the database 230 for training the encoder / decoder block.

[0024] Figure 3 is a flowchart 300 illustrating a technique for automated path dataset generation that uses a large language model to generate code that simulates a robotic task, where verified paths are added to a training database, according to one embodiment of the present disclosure. Figure 3 shows a high-level overview of a "pipeline" of steps for creating and verifying motion programs using large language model artificial neural networks and other techniques. The motion programs generated by the pipeline are added to a database for encoder / decoder training. Details of the steps and elements of Figure 3 are described below in conjunction with subsequent figures.

[0025] In box 310, inputs and task instructions are provided. The provided inputs include an asset library and a task library. An asset library is essentially a database of computer-aided design (CAD) models of objects that can be used in task simulations. Objects in an asset library can include geometry primitives (e.g., cube, sphere), workpiece designs, fixtures, and other obstacles known to exist in the robot workspace. The asset library can also include robot arm elements and grippers. The task library contains primitive robotic tasks (e.g., pick object, grasp object, move object, place object) that can be combined to perform more complex tasks. Task libraries can be populated with tasks specific to specific industrial robots, such as a specific model of a six-axis articulated robot (with known arm geometry, kinematics, etc.); these libraries are discussed further below.

[0026] The task instructions provided in box 310 contain a description of the robot task to be performed. A simple example is "Place blocks on a stack based on color." Task instructions can be thought of as including additional information needed to perform the robot task, such as a start position and a target position, as well as the identification of the workpiece to be operated on.

[0027] The large language model (LLM) block 320 receives input from box 310 and decomposes the overall task into a set of tasks from the task library and corresponding objects from the asset library. The large language model is an artificial neural network configured for the ability to achieve general-purpose language generation and understanding. The large language model acquires these capabilities by learning statistical relationships from instructional documents during a training process that may include self-supervised and semi-supervised learning. The task breakdown from the large language model 320, including a task description of the sequence of steps and object names, is provided to box 330.

[0028] Large language model block 340 takes the task description and object names from box 330 and generates code (in a programming language such as Python®) to simulate the work in box 350. The large language model in block 340 is the same as the large language model in block 320, essentially performing a second operation (code generation) after the previous operation (task decomposition). Using the code generated in box 350, the work task is performed in simulation in box 360, which includes simulating the articulation of the robot arm to move the workpiece according to the input task instructions. The simulated motion of the robot arm and workpiece from box 360 is provided to box 370, where RRT techniques and / or optimization-based techniques are used to generate a collision-free path in box 380.

[0029] Final validation of the generated path is performed in box 390, which includes ensuring that the generated path is collision-free and feasible from a robot reachability perspective, and that the workpiece or object being moved reaches and stabilizes the target location. At decision diamond 392, it is determined whether all validation criteria have been met. Paths that do not pass all aspects of validation are discarded at terminal 394. Paths that pass all aspects of validation are added to the operating program database 230 for use in training the encoder / decoder neural network, as shown in FIG. 2. FIG. 3 provides a high-level overview of the path dataset generation database pipeline of the present disclosure. The steps of the pipeline are discussed in detail below.

[0030] Figure 4 is a diagram illustrating the inputs and outputs of the large language model of the path dataset generation pipeline of Figure 3, including asset and task library inputs along with task instructions, and task decomposition and code generation outputs, according to an embodiment of the present disclosure. Large language model 440 is shown in the center of Figure 4. The large language model is shown twice in Figure 3 because large language model 440 corresponds to both box 320 and box 340 in Figure 3, and Figure 3 is a block diagram of steps and elements in the path dataset generation pipeline.

[0031] In FIG. 4 , inputs to large language model 440 are at the left and top left, and outputs from large language model 440 are at the right. Asset library 410 contains CAD models of any objects that may be referenced in path generation. This includes workpieces, tools, fixtures, tables, bins, and any other objects that may be moved by a robot or that may pose obstacles to the robot during movement. Asset library 410 is shown with many very small graphical icons, which are not meant to be individually significant when viewing FIG. 4 but simply represent a large library of object or asset data. Enlarged view 412 shows more detail of some items in asset library 410; these simply represent the CAD solid model data of the many objects in asset library 410.

[0032] Task library 420 contains robot tasks that can be selected by large language model 440 to perform individual steps in a complete robot task. Tasks in task library 420 can be defined for a specific robot architecture, such as a five-axis articulated robot. Task library 420 includes many primitive and basic robot tasks, such as reposition gripper, grasp object, move object to target point, move object vertically in the positive Z-axis direction, place / ungrasp object, etc. Task library 420 can also include tasks more complex than simple primitive movement or grasping, such as inserting an object into an aperture. Task library 420 is shown with many very small graphical icons, which are not meant to be individually significant when viewing FIG. 4 but simply represent a large library of robot tasks. Close-up 422 shows more detail on some items in task library 420; these are simply visual representations of the tasks in task library 420.

[0033] The final input to the large language model 440 is the task instruction 430. A non-limiting, illustrative example of a task instruction is "Move object W from point S (start point) out of bin B and place object W in fixture F at point G (destination point)." The task instruction 430 is defined in plain language (e.g., English) to be spoken. It can be observed that the example task instruction listed above requires the large language model 440 to identify the objects (workpiece W, bin B, fixture F), understand the geometric information (start point and destination point), and figure out how to perform the entire task using tasks in the task library and taking into account implicit constraints (move the workpiece onto the bin, then move it to the destination point). Other examples of task instructions include "insert matching objects of the same color into the object" and human-directed multi-step instructions such as "build a cabinet using vertical block piece B5 and horizontal shelf piece S7. Then place objects of type X on the lower shelf and objects of type Y on the upper shelf."

[0034] Given the above inputs, large language model 440 interprets task instructions 430 and decomposes it into individual steps from task library 420, which contains objects from asset library 410. The result of task decomposition is shown in box 450, where objects are identified and a narrative for the entire task operation is written. For example, for the cited task instructions 430, the large language model has determined that first the gripper (identified by the tool center point, or TCP, at the end of the robot arm) must be moved to start point S and grasp the workpiece W, then the tool center point must be moved vertically upward to clear the bin, and then the tool center point must be further repositioned before the workpiece can be moved to the target point.

[0035] As indicated by U-turn arrow 460, the task decomposition in box 450 is then used by large language model 440 for the next step, indicated by arrow 470, which is to generate code (programming instructions) that objectively executes the task decomposition as described in the narrative. Box 480 shows a representation of the resulting code, including variable definitions, programming logic, and directives. In one embodiment, the code in box 480 programmatically defines the movements of the robot gripper and workpiece as the complete task operation is performed.

[0036] Figure 5 illustrates the simulation and path generation steps of the path dataset generation pipeline of Figure 3 and validation of the generated path, according to an embodiment of the present disclosure. Figure 5 begins where Figure 4 left off, with the code in box 480 programmatically defining the motion of the robot gripper and workpiece as the complete task operation is performed. Based on the gripper motion, the complete robot motion program can be calculated, for example, by using inverse kinematics. The complete robot motion program, including the motion of all joints in the articulated robot, allows for simulation of the robot and workpiece motion, as shown in box 510.

[0037] The simulation in box 510 is created based on the motion of the gripper and workpiece, which is then synthesized by the large language model to complete the defined task. Therefore, the simulated robot motion in box 510 is not guaranteed to be collision-free, especially when considering all elements of the robot arm along with the workpiece, and further when the obstacle environment is complex. Therefore, the simulated robot motion from box 510 is used as input to path generation box 520, where RRT techniques and / or optimization-based techniques are used to generate a collision-free path.

[0038] In one embodiment, an optimization-based method may be employed in box 520 that takes the simulated robot motion from box 510 as the initial path and performs an optimization calculation with an objective function formulated to maximize path quality (or minimize path length and cycle time) and constraints that penalize interference between any part of the robot or workpiece and any obstacle. The optimization solution may determine that the robot arm interferes with an obstacle, for example, when passing through or reaching around the obstacle, and find a solution that eliminates the interference by positioning the inner and outer robot arms differently while keeping the tool center point path the same. In this way, the optimization calculation generates a collision-free path while embodying the general path shape characteristics defined by the large language model.

[0039] The RRT method may be used to find a collision-free path in box 520 in a manner similar to that described above for the optimization-based method, starting with the robot's motion from box 510 and searching for new waypoints that resolve the detected collisions. Both the RRT method and the optimization method may also be used, e.g., using the RRT to search the configuration space to find a collision-free path and using optimization to improve the quality of the collision-free path found by the RRT method.

[0040] The collision-free path generated in box 520 undergoes final validation in box 530. In one embodiment, validation includes ensuring that the path is collision-free, that robot reachability is feasible for all points in the path, and that the object (workpiece) is stable at the target point. Some of these validation parameters may already be known if the path generation in box 520 is successfully completed. The stability of the workpiece object at the target point may be a separate calculation to ensure that the workpiece is not placed upside down or in any unstable pose or position. If all validation parameters are met, the generated path is added to the training database, as described above.

[0041] Referring again to Figure 3, along with the details of the steps shown in Figures 4 and 5, the overall path generation pipeline can be used to automatically add many collision-free paths (motion programs) to database 230 by varying the locations of the start and target points, varying the pose of the starting workpiece, varying the obstacle environment, etc. Once many collision-free motion programs with desired path characteristics have been added, database 230 can be used to train an encoder / decoder neural network system, as shown in Figure 2. It may be desirable to fine-tune the pre-trained neural network system with a smaller number of paths from another source, such as human demonstrations.

[0042] 6 is a block diagram illustrating a system 600 for automated path data set generation, training an encoder / decoder neural network using the generated data set, and robot path generation and execution, according to an embodiment of the present disclosure. System 600 generally includes a computer or computing device 610, a robot controller 620, and a robot work cell 630.

[0043] The computer 610 is any computing device suitable for performing the path dataset generation tasks shown in Figures 4-6 and described above. The computer 610 may be a general-purpose computer or a server, or the computer 610 may be a device selected for its performance in executing the large language model 440.

[0044] Large language model 440 runs on computer 610 and receives the aforementioned inputs: asset library 410, task library 420, and task description 430. Large language model 440 decomposes the task and generates code 480. All of these operations are shown in FIGS. 4-5 and have been discussed previously. Code 480 is used to create simulation 510, which is used to generate a path at 520 using RRT and / or optimization techniques. Path 520 is validated at 530, and the validated path is added to database 230, as also shown in FIGS. 4 and 6.

[0045] The database 230 is used to train the encoder / decoder neural network 220, as discussed above with respect to Figure 2. In a manufacturing robot operation, inputs 210 for a robot task are provided to the encoder / decoder 220 running on a controller 620, which calculates a robot path at 250. The controller 620 provides the calculated robot path to a robot operating in a robot work cell 630. The robot in the work cell 630 performs operations using an operation program provided by the controller 620.

[0046] The division of tasks between computer 610 and controller 620 may differ slightly from that shown in Figure 6. For example, in addition to creating training database 230, computer 610 may also train and run encoder / decoder 220 to compute paths from input 210. In this scenario, computer 610 communicates the computed paths to controller 620, which sends commands to the robots to execute motion programs in robotic workcell 630.

[0047] Various computers and controllers have been described or implied throughout the foregoing discussion. It should be understood that the software applications and modules of these computers and controllers execute on one or more computing devices having a processor and memory modules. In particular, this includes the processors within computer 610 and robot controller 620 of FIG. 6, where the controller / computer is configured to perform the path dataset generation calculations in the manner described above. The computing devices may include dedicated devices specifically configured for the execution of convolutional and / or encoder / decoder neural network systems.

[0048] As outlined above, the disclosed technique for path dataset generation offers several advantages over existing robot path generation methods. The disclosed technique enables the automatic and rapid creation of a database for training an encoder / decoder neural network system without the need to generate each training path by human demonstration. The paths in the training dataset are validated as collision-free and also define the obstacle environment within which each path was computed, making them useful for neural network training.

[0049] While several exemplary aspects and embodiments of route dataset generation have been described above, those skilled in the art will recognize modifications, permutations, additions, and combinations thereof, and it is therefore intended that the appended claims, and any claims hereafter introduced, be interpreted as including all such variations, permutations, additions, and subcombinations as fall within their true spirit and scope.

Claims

1. 1. A method for robot path dataset generation, comprising: providing the task library, the object asset library, and the text instructions for the tasks to a large language model (LLM) running on a computer having a processor and a memory; generating, using the LLM, code containing programming instructions for a robot to perform the task; running a simulation of the robot performing the task using the code; generating a collision-free robot path from the simulation using a collision-avoidance path generation algorithm; validating the robot path against a set of path quality criteria; adding the robot path and data defining an obstacle environment to the path dataset if the robot path satisfies the path quality criteria; A method comprising:

2. The method of claim 1 , wherein the task library includes definitions of primitive tasks and task sub-combinations that can be combined to perform the task.

3. The method of claim 1 , wherein the object asset library includes three-dimensional models of objects involved in the task, including workpieces, robot arm parts, grippers, fixtures, and obstacles.

4. The method of claim 1 , wherein generating the code includes writing programming instructions that define the movement of a tool center point at the end of a robotic arm required to perform the task.

5. 5. The method of claim 4, wherein the LLM first writes a story of the steps required to perform the task based on the text instructions, and then writes the programming instructions corresponding to the story of the steps.

6. The method of claim 4 , wherein performing the simulation includes calculating motions of all parts of the robot corresponding to motions of the tool center point.

7. The method of claim 6 , wherein calculating the motion of all parts of the robot includes using an inverse kinematics calculation algorithm.

8. 2. The method of claim 1 , wherein generating the collision-free robot path includes using a simulation of the robot as an initial path and using either a fast-searching random trees (RRT) algorithm or an optimization-based algorithm to generate the collision-free robot path.

9. 2. The method of claim 1, wherein validating the robot path includes verifying that the robot path is collision-free, that all movements of the robot in the robot path are executable, and that the task is completed successfully.

10. The method of claim 1 , wherein the path dataset is populated with multiple validated robot paths, each generated based on a different combination of the text instructions, start and destination points, and the obstacle environment.

11. 11. The method of claim 10, further comprising using the pathway dataset to train a neural network system, including supervised learning training of a neural network.

12. 12. The method of claim 11, further comprising: using the neural network system to generate a collision-free robot motion program based on inputs for a motion; and using the collision-free robot motion program to send instructions to the robot to perform the motion.

13. 1. A method for controlling a robot, comprising: generating a collision-free robot motion program based on inputs for a motion using a neural network system executed on a computing device; and using the collision-free robot motion program to send instructions to the robot to perform the motion; Equipped with the neural network system is trained using a pathway dataset in a supervised learning process; the path dataset is populated with a plurality of validated paths, each path being generated using steps based on a different combination of text instructions, start and end points, and obstacle environments; The steps include: providing a task library, an object asset library, and text instructions for the task to a large language model (LLM); generating code containing programming instructions for a robot to perform the task using the LLM; running a simulation of the robot performing the task using the code; generating a collision-free robot path from the simulation using a collision-avoidance path generation algorithm; Validating the robot path against a set of path quality criteria; adding the robot path and data defining an obstacle environment to the path dataset if the robot path satisfies a path quality criterion; A method comprising:

14. 1. A system for robot path dataset generation, comprising: The system includes a computer having a processor and memory, the computer executing the large language model (LLM) and the steps; The steps include: providing a task library, an object asset library, and text instructions for performing the tasks; generating, by said LLM, code containing programming instructions for a robot to perform said task using a task library, an object asset library, and text instructions for the task; running a simulation of the robot performing the task using the code; generating a collision-free robot path from the simulation using a collision-avoidance path generation algorithm; validating the robot path against a set of path quality criteria; adding the robot path and data defining an obstacle environment to the path dataset if the robot path satisfies the path quality criteria; Including, the system.

15. 15. The system of claim 14, wherein the task library includes definitions of primitive tasks and task sub-combinations that can be combined to perform the tasks, and the object asset library includes three-dimensional models of objects involved in the tasks, including workpieces, robot arm parts, grippers, fixtures, and obstacles.

16. 15. The system of claim 14, wherein generating the code includes writing programming instructions defining the movement of a tool center point at the end of a robotic arm required to perform the task, and wherein the LLM first writes a story of steps required to perform the task based on the text instructions, and then writes the programming instructions corresponding to the story of steps.

17. The system of claim 16 , wherein performing the simulation includes calculating motions of all parts of the robot corresponding to motions of the tool center point.

18. 15. The system of claim 14, wherein generating the collision-free robot path includes using a simulation of the robot as an initial path and using either a rapid-searching random trees (RRT) algorithm or an optimization-based algorithm to generate the collision-free robot path.

19. 15. The system of claim 14, wherein validating the robot path includes verifying that the robot path is collision-free, that all movements of the robot in the robot path are executable, and that the task is completed successfully.

20. 15. The system of claim 14, wherein the path dataset is populated with multiple validated robot paths, each generated based on a different combination of the text instructions, start and destination points, and the obstacle environment.

21. 21. The system of claim 20, wherein the pathway dataset is used to train a neural network, including supervised learning training of the neural network.

22. 22. The system of claim 21, further comprising a robot in communication with a robot controller, wherein the neural network system is used to generate a collision-free robot motion program based on inputs for motion, and the robot controller sends instructions to the robot to perform the motion using the collision-free robot motion program.

Images