Trajectory path planning and management of robot arm movements

Abstract

To provide a trajectory generating method for moving a robot arm in the fastest and smoothest path.SOLUTION: The method of the present invention achieves time saving and smooth movements by implementing a planning, sequence and trajectory path manager to plan movements ahead of a predetermined time, and by concurrently creating a database of trajectory paths that is subsequently used for storing and retrieving of repetitive movements of a robot arm. In a dynamically changing operating environment, endpoints are calculated through implementation of an intelligence module. The intelligence module captures a three-dimensional image of the processing area and predicts a three-dimensional pose. The three-dimensional pose, in turn, is translated to positional coordinates for a robot to utilize, to move to the target endpoint. The planning manager working in sync with the intelligence module and the sequence manager enables a very favorable environment to seamlessly and efficiently move objects smoothly at controlled speeds, mimicking human movement.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a trajectory generation and storage method for smooth and shortest path movement of a robot arm using a planning, sequence and trajectory path manager in conjunction with an intelligent module. The present invention also provides parallel processing of trajectory paths of future robot arm movements to minimize operation time resulting in faster manufacturing process. [Background technology]

[0002] In an industrial environment, trajectories are taught by a human expert and then played back in a typical teach-and-playback manufacturing environment. One or more trajectories are generated for each robotic arm by connecting two or more waypoints, where each waypoint is typically taught by a human operator based on the pick-and-placement pose or position of the object.

[0003] According to the conventional method, an operator prepares the robot arm by teaching it through trial and error pass-points that satisfy the condition that the robot arm does not enter an area where it cannot operate or does not interfere with peripheral devices or other pass-points. In a given situation, there is a possibility that a robot trajectory path that allows the robot arm to operate at an optimal motion speed is not generated quickly enough, thus delaying the cycle time. In such a case, manual intervention is required to reconsider and reset the pass-points of the robot arm to achieve the optimal cycle time.

[0004] In other conventional methods, an operator may prepare the robot arm by teaching one or more waypoints that may require abrupt turns or direction changes, which may cause the arm to generate trajectories with sudden acceleration or deceleration movements, and object pick-up or placement operations may be affected due to vibrations caused by such movements of the robot arm, which may potentially impact the manufacturing environment, resulting in inefficient operations as well as increased cycle times and poor quality of the final product.

[0005] A playback robot arm is typically designed such that a robot controller teaches the robot arm multiple waypoints at one end of the robot arm (manipulator). These waypoints may include (but are not limited to) a start point, an end point, and intermediate points in between. A robot hand is suitably integrated at one end of the manipulator. The robot controller is programmed to perform interpolation on the taught command points to determine a trajectory that passes through the waypoints. The robot controller is then programmed to move the robot arm smoothly along the determined trajectory. While this may work in most environments, there is a high probability that the robot arm will encounter collisions with predefined obstacles or encounter dynamically changing environments where the waypoints may move depending on the task at hand and the object to be machined.

[0006] Trajectory path generation aims to create a motion-controlled path for the robot arm that allows it to execute the planned trajectory smoothly, without unstable jerks, and to cover the distance at the highest possible speed without colliding with any of the waypoints around the working area.

[0007] Advanced trajectory path generation along with planning manager is required to implement forward-looking strategies to perform specific trajectory path movements without adding overhead to timing and movements that may include trajectory paths that have been planned well in advance to speed up the robot arm. Managing the robot arm trajectory by implementing an intelligence module, avoiding obstacles, navigating complex mazes, and considering dynamically changing waypoints in some complex cases, ensures more efficient, safe and fast operation. There are many machine motion laws that must be taken into account during trajectory management and advanced planning, such as saturation limits of motor drive speed and acceleration, resonance points of actuator structures, when arriving at a smooth trajectory path in practical implementations using electric / hydraulic or pneumatic drive modes. Trajectory path involves path determination using multiple waypoints, and routing through those waypoints is planned to determine the shortest path with collision avoidance. The path is then used by the robot controller to move the robot arm on the planned trajectory path. The trajectory path uses a geometric path planned for a specific task, and determines the speed and acceleration around each waypoint according to the turning angle, minimizing the effect of resonance to smooth the movement of the robot arm. In simple industrial machines, the trajectory path is usually trained by a human operator through trial and error. However, as the machines become more complex and the performance level increases, it is necessary to incorporate an automated method of trajectory path planning. Here, a trajectory path planning manager alone may not be sufficient. In such a case, an image-based device and method in cooperation with an intelligence module is required to dynamically plan trajectory paths by using a forward lookup algorithm and pre-stored trajectory paths, or when the robot environment changes, it is essential to quickly calculate and communicate a new trajectory path to the robot controller to ensure high-speed performance. Furthermore, by incorporating artificial intelligence and machine learning techniques, all frequently used trajectory paths are stored in a database related to a specific product type to minimize machine configuration settings during product changeover.Therefore, these methods have become a necessity rather than an option in robotic controlled industrial environments.

[0008] In certain cases, trajectory path planning algorithms have limitations that may result in failure in planning the robot arm movement from one waypoint to another in a trajectory due to obstacle avoidance. In such cases, when the robot arm needs to carefully maneuver around an obstacle, it is essential to plan multiple trajectories while handling the object, in which case two or more intermediate points need to be added between two waypoints and multiple trajectories need to be planned and combined to reach the target position. This results in start / stop motions of the robot motion executing multiple trajectories, and multiple deceleration / acceleration motions for each trajectory will generate vibrations due to the resonance characteristics involved. Summary of the Invention

[0009] In view of the background, it is an object of one aspect of the present invention to provide a trajectory path planning manager in a motion controller that uses a planning manager in conjunction with a sequence manager for standard practical applications where waypoints do not change.

[0010] Another objective of one aspect of the present invention is to provide a trajectory path planning manager for a motion controller by providing a planning and sequence manager with an intelligence module that captures images and provides a mapping to the trajectory path managing module for a dynamically changing environment where waypoints may change. The intelligence module helps to deal with complex situations that typically arise in multiple robotic arm applications or obstacles. Using the task layout map provided by the intelligence module, the trajectory path managing module utilizes a class of polynomial or spline interpolation to generate the desired path waypoints and arrive at a time-based control sequence that smoothly moves the robot arm from its current position to its destination. The trajectory path manager module deals with exponential complexity for a given robot degree of freedom (controllable joints) by applying artificial intelligence that follows machine learning techniques. The trajectory path manager further includes a means for calculating from a spline interpolation implementation to generate smooth motion between multiple path segments without discontinuities in position, velocity and acceleration to avoid the Runge phenomenon. The Runge phenomenon causes oscillations or ringing at the beginning or end of the robot arm movement that is typical in polynomial interpolation implementations when manipulating equidistant waypoints.

[0011] According to a further aspect of the present invention, a method is provided for determining a single motion by implementing a chain motion mode in which multiple trajectories are planned together and then assembled into one. This reduces pauses between two trajectories, since the robot arm does not need to wait for the next trajectory to be planned for each segment of the other partial paths. While merging multiple trajectories, deceleration usually occurs during multiple trajectory transitions due to deceleration and acceleration at each waypoint, especially when the planned trajectory requires abrupt direction changes. To overcome this problem, a blending concept is implemented for trajectories with smooth curves, where a blending radius is calculated to merge with the path of the next trajectory to effectively reduce or minimize the effect of sudden deceleration and acceleration rather than reaching the waypoint. The blending is controlled by a blending radius that determines the amount of curvature required to smooth the path near the transition, making the motion as smooth as human motion.

[0012] According to a further aspect of the invention, a method is provided in which trajectories between all known waypoints are planned in advance by a planning manager. When a trajectory sequence needs to be executed, the robot arm retrieves the trajectory from the planning manager without waiting time, since it is pre-planned and pre-stored. In the event that a pre-planned and stored trajectory path is not available for a set of waypoints, the planning manager together with the sequence manager determines a trajectory path from the start state to the nearest waypoint for which a pre-planned trajectory path exists. This significantly reduces the time compared to the case where a pre-planned trajectory path is not available. The same logic is also performed in the event that a pre-stored trajectory path does not exist until the final state. In this case, a new trajectory path is determined from the nearest waypoint to the final state. The flexibility of the pre-stored trajectory path database is enhanced, making it efficient and fast.

[0013] According to a further aspect of the present invention, a method is provided where a 3D (three dimensional) image of the working environment is captured by an intelligence module which predicts a 3D pose or map and the coordinates of all way points and obstacles are determined to guide the robot arm to a specific end point according to instructions received from a robot controller. Further, parameters such as rotational coordinates of row pitch and yaw are additionally determined according to the 3D map, resulting in a 6D (six dimensional) calculation of the trajectory path to move the robot arm from a start state to a final state.

[0014] According to a further aspect of the invention, a method is provided in which an intelligence manager handles imaging server operations in one pipeline to determine object positions and communicate non-linear trajectory sequences to a robot controller in real time.

[0015] According to a further aspect of the invention, a method is provided for rapidly calculating motion trajectories for automated systems such as robots, cranes and vehicles. Given a set of waypoints and associated obstacles and constraints, one or more starting states of the robot, and one or more desired end states of the robot, the method initiates a novel iterative procedure to calculate fast sample motion trajectories to meet the objective of the robot's goal state. Each iteration improves the trajectory by implementing PID (Proportional-integral-derivative), MFC (Model predictive control) and adaptive control algorithms until it meets all criteria up to the desired tolerances of acceleration and velocity for the payload, making the robot arm move faster and smoother. Offline (prior to execution), the procedure is used to generate multiple trajectories in several starting and ending states of the robot, followed by applying artificial intelligence and machine learning techniques to create a database of all frequently used trajectories. This process allows for a fast response to commands to move the robot from one state to another, since the database has most of the trajectories pre-calculated and stored ahead of time. Further, by implementing deep learning techniques, the robot's processes and movements are stored and, optionally, fine-tuned through frequent calculations involving various input and output states of the robot.

[0016] The planning manager and sequence manager working with the intelligence module facilitate most trajectories to be executed in minimal computation time. This process of maneuvering a visually guided robot arm through complex balanced paths is fast, seamless and smooth, mimicking human movement. [Brief description of the drawings]

[0017] The present disclosure will be better understood from reading the following description of non-limiting embodiments with reference to the accompanying drawings, in which:

[0018] [Figure 1]A block diagram of the computational system including the various involved subsystems is shown, which determine the optimized trajectory path and provide commands to the robot controller for the respective robot arm movements. [Diagram 2] FIG. 1 shows a flow diagram illustrating trajectory motion planning for a robotic arm with pre-determined way-points, according to one embodiment. [Diagram 3] FIG. 1 shows a flow diagram illustrating trajectory motion planning for a robotic arm with pre-determined and dynamic way-points according to one embodiment. [Figure 4] 1 shows a flowchart of Sequence A in which an intelligence manager predicts 3D waypoints, calculates rotational coordinates for roll, pitch and yaw, resulting in a six-dimensional orbital path for a working area, according to one embodiment. [Diagram 5] 1 shows an example of a trajectory path between six waypoints using spline interpolation. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0019] The following description pertains to the calculation of motion trajectory paths for automated systems that consider the robot's payload and balance while avoiding obstacles, reduce stress to minimize wear on the mechanical mechanisms involved, and generate rapid, jerky-free movement from initial to final positions.

[0020] In one embodiment of the present disclosure, the system and method provided herein are used to speed up and improve the motion of a robot for loading and unloading contact lens blister packs from an input loader of an automated line to an inspection location. In this case, a set of starting states is defined by the desired position of a suction cup or pick-up end effector integrated in the robot to pick up the blister packs. A set of end states is defined by the position of the pick-up end effector at the inspection station, where accurate and fine positioning is performed by intelligent imaging and subsequent deactivation of the end effector suction cup holding the contact lens blister pack resulting in unloading at the inspection station. At the end state of unloading the contact lens blister pack, there is a high degree of positional accuracy in the X, Y and Z axes and orientation. To increase system throughput and reduce wear on the robot, it is desirable to quickly calculate a motion trajectory path from one of the starting states to one of the end states. This motion trajectory path avoids both fixed and dynamically moving obstacles using the chain mode concept. The chain mode is implemented by a planning manager that links multiple trajectories together by adopting smooth movements in terms of velocity and acceleration to ensure vibration- and jerky-free movement through all given waypoints.

[0021] Referring to FIG. 1, a block diagram 50 is shown as an example of a computing system 52 for jerky and vibration-free motion planning according to one embodiment. The computing system 52 includes a sequence planning subsystem 106, a path planning subsystem 108, and a trajectory planning subsystem 110 suitably linked to a database, and is communicatively coupled to a robot controller 104 to control a robot arm 102 or other system adapted for autonomous movement in an automated machine 112. Although the computing system 52 is depicted as a separate system from the robot controller 104, it should be understood that in some examples, one or more components of the computing system 52 may be integrated into the robot controller 104 or may be standalone modules to operate the robot arm 102. The computing components may include, but are not limited to, artificial intelligence, machine learning, reinforcement learning, and deep learning algorithms to build a powerful and efficient database of readily available trajectory paths instead of calculating the trajectory paths in real time once the waypoint map is received from the computing system. The database serves as a readily available trajectory path storage subsystem that can be adapted to different configurations of industrial machines used to process and manufacture different products.

[0022] Referring to Fig. 2, a flow chart 100 illustrates diagrammatically the operations performed by the robot controller. The flow chart 100 illustrates the interactions between various modules in the computing system 52 of Fig. 1. The flow chart focuses specifically on automated machines, where the planned trajectory path is based on a pre-determined set of waypoints. This is applicable to automated machines such as packaging, sorting, or labeling of processed products, where most of the processing points for the products are fixed.

[0023] The flow typically starts with an input command 116 to the robot controller. Subsequently, the robot controller communicates the waypoints (initial waypoint and final waypoint) to the planned trajectory path database 110. In return, the database sends the desired trajectory path to the robot controller if a pre-planned or pre-defined path is available for that waypoint. The robot controller 104 drives the robot arm 102 to the final destination waypoint with the specified motion speed, velocity, and trajectory path. However, in the event that a pre-determined trajectory path does not exist or is not available, the robot controller 104 obtains information about the absence of a pre-determined trajectory path from the planned trajectory path database 110, which then starts the process of planning a new trajectory path, followed up by a command by the robot controller 104 to the sequence manager 106. The sequence manager 106 further checks the waypoint positions and determines the sequence of trajectory paths to be executed after implementing the blending of the trajectory path with all intermediate waypoints, taking into account a blend radius parameter that determines the amount of curvature required to smooth the trajectory path near the transition from one waypoint to the next. The calculated sequence is subsequently communicated to the path manager 108, which determines the fastest trajectory path to the final destination waypoint and uploads the information to the planned trajectory path database 110 for storage and further communication to the robot controller 104. The robot controller 104 then executes the movements of the robot arm 102. The executed movements result in movements as smooth as human movements.

[0024] In Fig. 3, a flow chart 130 shows a schematic of the operations performed by the robot controller, where a dynamic waypoint is encountered. The flow chart 130 shows the interaction between various modules in the computational system 52 of Fig. 1, where the planned trajectory path is based on both a variable set of waypoints and a fixed set of waypoints. This is adaptable to automated machines where interactions between several processing points are involved resulting in unpredictable environments and changing conditions. The complexity of such operations requires a series of pre-planned sequence operations, on-the-fly calculation of trajectories, and a path manager that can quickly rearrange the sequences and trajectory paths to achieve smooth and accurate movements of the robot arm.

[0025] The flow chart of Fig. 3 starts with an input command 116 to the robot controller. Subsequently, the robot controller 104 communicates the waypoints (initial waypoint, intermediate waypoint and final waypoint) to the planned trajectory path database 110. In return, the database sends the desired trajectory path to the robot controller if a pre-planned or pre-determined trajectory path is available for the waypoint. The robot controller 104 drives the robot arm 102 with the specified motion speed, velocity and curvature to the final destination waypoint. However, in the event that a pre-determined trajectory path does not exist or is not available, or if an obstacle exists in the pre-planned trajectory path, the robot controller 104 sends a command to the sequence manager 106 to start the process of planning a new trajectory. The sequence manager 106 further checks the waypoint locations, including obstacles and intermediate waypoints. This information is communicated to the planning manager 108. The planning manager, aware of the new environment after the change, sends a query to branch 112(A). The trajectory path manager determines the trajectory path by implementing PID (Proportional Integral Derivative), MFC (Model Predictive Control) and adaptive control algorithms to ensure jerky and resonance-free movement of the robot arm from the start state or waypoint to the end state or waypoint through all intermediate waypoints. Furthermore, the calculated trajectory path is uploaded to 116 for further characterization of the trajectory path using artificial intelligence, machine learning and reinforcement learning, and subsequently updated to the database 110 residing in the computing system 52 of FIG. 1 or the cloud. The database 110 is iteratively fine-tuned and updated to ensure that the robot arm can operate with minimal delay for a particular command by utilizing the pre-planned trajectory path.

[0026] In Fig. 4, flow chart A starts with the intelligence manager 195 sending a command to the imaging system to capture the 3D pose of the working environment in step 200. In step 205, the software algorithm identifies and calculates the X, Y and Z coordinates and determines the rotational values ​​related to roll, pitch and yaw, followed by returning a total of six values ​​in step 210, namely the X, Y and Z coordinates of the current state of the waypoint positions, which may include obstacles and intermediate waypoint positions, and the determined values ​​of roll, pitch and yaw. In step 210, the trajectory path. Further, parameters such as rotational coordinates such as pitch, roll and yaw are determined in 205 considering all the waypoints, the starting state and the final state, and the trajectory path is determined in 210. In the next step, the six-dimensional trajectory path determined in 210 is relayed to the path manager 107 in Fig. 3, followed by the end of the flow in 220.

[0027] In FIG. 5, an example of a trajectory path calculated using spline interpolation between waypoints P0-P6 is shown. Spline interpolation fits a low order polynomial between two adjacent waypoints. For example, instead of fitting one 6th order polynomial between each pair of 6 waypoints, 5 3rd order or piecewise polynomials are fitted. This minimizes the interpolation error and also eliminates the problems related to vibration (wobbling), resulting in a smooth movement of the robot arm. In the diagram 300 of FIG. 5, 6 waypoints, namely P0, P1, P2, P3...P6, are shown as an example. C1-C5 are the result curves blended to form a spline curve between each pair of waypoints. Obviously, the robot controller moves from the start state P0 to the end state P6 by following the trajectory path of C1-C5. The mathematical formulas or mathematical analysis between the waypoints, acceleration, deceleration and peak speed are not described as they are outside the scope of the present invention.

[0028] It will be understood that various modifications not yet described may be made, and it is intended in the appended claims to cover all such modifications that are within the scope of the present invention. The written specification uses typical examples to disclose the invention and to enable one skilled in the art to practice the invention, including making and using any device or system, and implementing any incorporated methods. The patentable scope of the invention is defined by the claims, and may include other examples that may occur to those skilled in the art.

Claims

1. 1. A method comprising: calculating a trajectory by a plan manager, a sequence manager and a trajectory manager in combination with an intelligence module utilizing an imaging system, using spline interpolation and fitting a plurality of low order polynomials between two adjacent waypoints from a start state to an end state through one or more intermediate waypoints; Rapid search for a trajectory path of an object in the driving environment of the robot arm according to kinematic and dynamic constraints of the automation system to obtain a final trajectory, and updating said trajectory against a database in a computing system or cloud for fast processing purposes; moving said robot arm from a start state through one or more waypoints to a finish state according to a final trajectory communicated by a robot controller; A method comprising:

2. 2. The method of claim 1, wherein calculating the trajectories includes calculating a plurality of trajectories for input commands specifying one or more starting states of the robot arm and one or more ending states of the robot arm through PID (Proportional Integral Derivative), MFC (Model Predictive Control) and adaptive control algorithms, each trajectory path in the plurality of trajectories having a different duration and a different distance.

3. The method of claim 2 , wherein the plan manager and sequence manager calculate the plurality of trajectories according to degrees of freedom associated with the robot arm at the one or more starting states and the one or more ending states.

4. The method of claim 2 , wherein the plan manager and sequence manager, in cooperation with the intelligence module, calculates the plurality of trajectories as a function of degrees of freedom associated with the robot arm at the one or more starting states and the one or more ending states.

5. 3. The method of claim 2, further comprising updating the trajectory path to minimize jerks of the robotic arm with iterative fine-tuning via application of artificial intelligence and machine learning combined with reinforcement learning techniques to achieve optimal speed and accuracy over time.

6. 3. The method of claim 2, further comprising updating the orbital path to a database for storage and retrieval in response to respective input and exit states.

7. The method of claim 1 , wherein the work environment includes a robotic arm attached to an industrial machine configured for repetitive and non-repetitive motion.

8. The method of claim 7 , wherein the orbital path of the repetitive motion is obtained from a pre-stored orbital path present in the database on the computing system or the cloud.

9. 8. The method of claim 7, wherein a trajectory path of a non-repetitive motion is calculated by utilizing 3D positions of all way-points of the working environment acquired by the imaging system and determining the roll, pitch and yaw required for the robot arm to move from a start state to the end state.

10. 8. The method of claim 7, wherein if a trajectory path of a non-repetitive motion from a given start state to an end state does not exist in the database, the computing system first identifies a waypoint closest to the start state and a waypoint closest to the end state, and subsequently obtains a trajectory path between these two repetitive waypoints, which may be pre-stored in the database.

11. 11. The method of claim 10, wherein the computing system calculates a trajectory path from the start state to a nearest waypoint, and then calculates another trajectory path from the nearest waypoint to the end state to arrive at a complete trajectory path from the start state to the end state, effectively combining a set of calculated trajectory paths with a pre-determined trajectory path obtained from the database.

12. 1. A system comprising: An automated machine utilizing a robot, including an automated machine operated by a robot controller system configured for automated repetitive and non-repetitive movements; The robot controller, when executed, causes the controller to: calculating trajectory paths from one or more start states to one or more end states of the working environment by PID (proportional integral derivative), MFC (model predictive control) and adaptive control algorithms; updating the orbital path with associated constraints and restrictions to a database residing in the computing system or in a cloud; automatically controlling the automated machine from a start state of the one or more start states to an end state of the one or more end states according to the trajectory path; The system is configured to have a software application that causes

13. The controller further comprises: an algorithm for computing a plurality of trajectory paths in response to inputs specifying the one or more starting states of the automated machine and the one or more ending states of the automated machine; The system of claim 8 , wherein each orbital path of the plurality of orbital paths has a different duration, velocity, acceleration and curvature.

14. 1. A system comprising: An automated machine utilizing a robot, the automated machine including an automated machine operated by a robot controller incorporating an imaging system and configured for automated repetitive and non-repetitive motion; The robot controller, when executed, causes the controller to: acquiring an image of the work environment using the imaging system to identify the waypoint; calculating a trajectory path from one or more start states to one or more end states of the working environment that is guided through the waypoints using PID (Proportional Integral Derivative), MFC (Model Predictive Control) and adaptive control algorithms; updating the orbital path with associated constraints and restrictions to the database residing in the computing system or in the cloud; automatically controlling the automated machine from a start state of the one or more start states to an end state of the one or more end states according to the trajectory path; The system is configured to have a software application that causes

Images