Method and system for robotic arm guidance for autonomously performing object pick and place tasks with adaptability

The method enables efficient kinesthetic guidance and autonomous operation of robotic arms by recording and optimizing motion profiles during demonstration, allowing adaptability in changing environments and reducing programming time, addressing the limitations of current robotic systems in production lines.

WO2024241119A1PCT designated stage expired Publication Date: 2024-11-28ARISTOTLE UNIVERSITY OF THESSALONIKI RESEARCH COMMITTEE ELKE
View PDF 14 Cites 0 Cited by

Patent Information

Application Number
PCT/IB2024/054175
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-05-19
Filing Date
2024-04-30
Publication Date
2024-11-28

AI Technical Summary

Technical Problem

Current robotic systems require specialized training and lack adaptability in changing environments, as they struggle to transition from kinesthetic demonstration to autonomous operation efficiently, especially in unstructured settings like production lines, where they need to pick and place objects randomly located on conveyor belts.

Method used

A method that allows a robotic arm to record its motion during kinesthetic demonstration, including initial, final, and intermediate poses, velocity, and acceleration profiles, and then automatically switch to autonomous operation using image sensors for online motion generation, enabling flexibility and adaptability in picking and placing objects.

Benefits of technology

This approach significantly reduces programming time, allows non-experienced users to operate robots efficiently, and enhances adaptability in variable environments by optimizing the robotic arm's motion, enabling it to adjust trajectories based on new object positions and orientations, even if they are moving.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure IB2024054175_28112024_PF_FP_ABST
    Figure IB2024054175_28112024_PF_FP_ABST
Patent Text Reader

Abstract

Method and system for robotic arm guidance for autonomously performing object pick and place tasks with adaptability The present invention relates to a method for autonomous execution of pick and place of objects without programming after kinesthetic demonstration, intended for use by robots in production line operations such as packaging and palletizing. The stages are: a) kinesthetic guidance of the arm (1) to the initial position, b) activation of recording, c) kinesthetic guidance of the arm (1) to the final position with simultaneous recording of the initial position, of the trajectory and of the final position in the recording system, d) completion of the recording at the final position and encoding of the demonstration, and e) simultaneous transition to autonomous mode with optimized trajectory generation, similar to that of the demonstration, to pick new objects from different positions and place them in order, according to the demonstration.
Need to check novelty before this filing date? Find Prior Art

Description

DESCRIPTIONMethod and system for robotic arm guidance for autonomously performing object pick and place tasks with adaptabilityTechnical field of the invention

[0001] The present invention relates to a method for autonomously performing object pick and place tasks without programming after kinesthetic demonstration, intended for use by robots in production line operations such as packaging, palletizing, and others. The method simplifies the use of robots in production lines so that the user does not need special training. It also increases the flexibility of the system so that the robot optimally adapts its movement to new pick and place positions and to changes in the environment, according to the information conveyed by an image processing system.

[0002] The use of robots in modern industries is widespread. However, many times, using a robot is complex and requires special training. Moreover, during autonomous operation, the robot provides little adaptability, not being able to perceive and cope with changes in the environment for which it has not been programmed. Thus, although their use solves many problems, it creates new ones.State of the art

[0003] The traditional way of programming a robotic arm involves the use of the provided teaching pendant, with which the user moves either each joint individually or the end-effector with respect to a Cartesian coordinate system, stores some points through which the robot's trajectory must pass and then connects them with predefined types of motions (e.g. linear, circular, etc.) to obtain the robot's final trajectory. This method is very time consuming, requires specialization, and offers no flexibility to changes in the robot's environment.Teaching robots with kinesthetic demonstration is a feature offered in many modern collaborative robotic arms, where instead of using the robot's teaching pendant to move the robot, the human can grasp the robot arm with their hands and guide it to specific positions, ki Aesthetically, by pushing or pulling it. Although the programming time can be reduced in this way, the user must save each intermediate position by frequently using the teaching pendant, spending time. In addition, kinesthetic guidance is provided at the joint level without coordination of the joints to achieve a certain dynamic behaviour of the endeffector that could increase the accuracy of the demonstration. Recording the entire trajectory from a demonstration without manual saving of intermediate positions using the teaching pendant, also does not offer flexibility, as the robotexactly reproduces the trajectory of the demonstration without the possibility of adjusting its motion to different initial, final, or intermediate positions.To enable a robotic arm to adapt its motion when operating in a not fully structured environment, such as to pick up objects that are randomly located on a conveyor belt and place them in an ordered manner, image sensors can be used to detect the objects and the environment. After automatic processing of each image, the processing system transfers information about the position of the objects on the robotic arm to change its trajectory. Existing solutions use simple image processing techniques to extract object features e.g. hole detection for assembly. Recently, solutions have been developed that use machine learning techniques to identify objects, but without extracting any information on how the robot should manipulate the objects which have been identified by the vision system in order to place them in an orderly manner.

[0004] The method of kinesthetic demonstration is mainly used to demonstrate points through which the robot's trajectory must pass, rather than to learn the entire trajectory, let alone the velocity and acceleration profile. The transition from demonstration mode to autonomous mode requires several time-consuming steps of interaction with the robot's teaching pendant. Also, commercially available object detection methods are not combined with automatic motion generation methods, but only with methods of prescribed robot motion, where only the initial or final position of the trajectory is changed, offering little flexibility.Therefore, the need arises for a method that allows the robot to record its motion after demonstration by even a non-experienced user, with little to no interaction with interfaces (physical or graphical), and then automatically transition to autonomous operation, which will offer flexibility to the robotic system through online motion generation according to data received from image sensors and having partial knowledge of the type of task it will perform.

[0005] US patent application number US 2022 / 0402139 is for a system that uses a machine learning model to predict the best position for placing objects. The system uses camera data to evaluate a possible object placement on a pallet, given the current state of the pallet and a set of previously placed objects. Only the autonomous mode stage is involved in calculating the final position of the object placement. There is no human robot interaction with physical contact or motion capture. The trajectory that the robot will perform to reach the final position is not calculated.International patent application number 2022 / 107943 is for a device that measures the position of objects in space for a robotic arm to pick them up. The camera in the device is moved to make the measurement. It concerns only the autonomous operation stage for calculating the position of grasping objects.There is no human robot interaction with physical contact or motion capture. The trajectory that the robot will perform to reach the final position is not calculated.The Greek patent application GR20170100133 is for a method of programming a humanoid robot with audiovisual examples from an operator. The robot uses a camera sensor, microphone and light sensors. The operator shows the robot the task by grasping an object himself and moving it, without being in physical contact with the robot. Starting and ending the recording is done by voice commands, with uncertain results within a noisy industrial environment. There is no optimization or encoding of the trajectory that was demonstrated but only simple playback.The Greek patent application GR20160100022 concerns a method of guiding the robot via a teaching pendant, where the relative orientation between the coordinate system of the teaching pendant and of the robot is automatically calculated for greater ease of use in storing points through which the robot's trajectory must pass. There is no demonstration of motion or trajectory recording in a kinesthetic manner. There is no flexibility in autonomous execution.Chinese patent application number CN113927593A is for a method of learning skills from a robotic arm based on task decomposition. There is no demonstration of motion or trajectory recording. Information about the robot's task is provided by the user in written text at a semantic level. The method requires the use of a simulation environment with an exact copy of the real robot environment, which is time consuming.Chinese patent application number CN114131603A is for a method for computing object picking using deep reinforcement learning. The method takes as input color-depth images and extracts the position for picking an object with the robot. It only involves the autonomous operation stage for calculating the object grasping position. There is no human robot interaction with physical contact or motion capture. The trajectory that the robot will perform to reach the pick position is not calculated.Chinese patent application number CN110561450A is for a method of learning assembly from a robotic arm through human motion capture. An optical motion capture device is used to collect motion data of the human hand while performing the assembly. This data is processed in non-real time and a trajectory is extracted for the robot to perform the assembly. There is no optimization or encoding of the trajectory that was demonstrated, but only simple reproduction. There is no human robot interaction with physical contact. No automatic transition from demonstration mode to autonomous mode is possible due to the processing needed for the data.Chinese patent application number CN110561430A is for a method of optimizing a trajectory of a robot for assembly with human demonstration via a camera.The human performs the assembly process, the camera detects the human's arms and encodes the human's movement in order to extract an assembly motion model, which is used by the robot to perform the assembly process. There is no human-robot interaction with physical contact. It is not possible to automatically transition from demonstration mode to autonomous mode due to the processing required for the data.Chinese patent application number CN110561431A is for a method of optimizing a robotic arm trajectory for assembly with human demonstration via a camera. The human performs the assembly process, the camera detects the human's arms and encodes the human's movement in order to extract an assembly motion model, which is used by the robot to perform the assembly process. There is no human-robot interaction with physical contact. It is not possible to automatically transition from demonstration mode to autonomous mode due to the processing required for the data.Chinese patent application number CN 111923041A is for a method of optically guiding a robotic arm to avoid interference from an external light source. The system recognizes the material being manipulated by the robot and can protect it from external light sources, ensuring the manipulation accuracy by the robotic arm during the guidance operation. There is no human robot interaction with physical contact or motion capture. No calculation of the trajectory the robot will perform to reach the position of picking or placing objects.Chinese patent application number CN215920461U is for a device for human gesture recognition, wireless data transfer in a motion capture system. It concerns only the demonstration stage. There is no optimization or encoding of the trajectory demonstrated but only simple reproduction. No flexibility is offered.Chinese patent application number CN 107253195A relates to a method and system for controlling a robotic arm mounted on an autonomous robotic mobile platform for transporting people. It does not relate to performing operations of positioning and grasping objects.

[0006] All of the above inventions relate to methods and / or devices that focus on other functionalities. Some relate to using robots for assembly, some relate to controlling and programming robots with a teaching pendant or other methods. However, in many of the above inventions the robot is blind, and in none of the above inventions, there is physical interaction between the human and the robot for kinesthetic guidance of the robot's motion, so that the latter records and encodes not only the basic trajectory to be followed, but also the speed and acceleration profiles in order to generate an optimized trajectory during autonomous operation, which is activated by pressing a button. Thus, the robot can take an object from its initial position, e.g. while moving on a conveyor belt,and move it to a final position, e.g. a carton, even if the initial and final positions are different from those of the demonstration.Presentation of the invention

[0007] The invention relates to a method for use in a robotic arm to enable it to autonomously perform pick and place tasks with adaptability, following the kinesthetic demonstration of the task by a human. For the kinesthetic demonstration, the user grasps the robot with one or both hands and, by exerting forces, moves the robot to the initial position (i.e., the object's pick position), activates the recording, which is signaled by either a physical button or a virtual button in a graphical interface, and then moves the robot to the final position (i.e., the place position). As the user demonstrates the trajectory to the robot, the recording system stores the initial, final and all intermediate poses of the robotic arm, including the velocity and acceleration profiles from the demonstration. Upon completion of the demonstration, which is signaled in the same way as the start, the robotic system encodes the demonstrated task and then automatically switches to autonomous operation. During autonomous operation, the robotic system can autonomously adjust the task even if the initial, final or even intermediate positions are changed, according to the information conveyed by an image processing system.By means of at least one image sensor, which is either mounted in a fixed position relative to the robot base or in a variable position on the arm, the detection system shall calculate the relative position of the object with respect to the end-effector's frame when the object's pick is demonstrated (start of recording) and the layout of the space surrounding the object when the object's placement is demonstrated. Upon completion of the demonstration, the robotic system shall encode the demonstrated task and then switch to autonomous operation. During autonomous operation, the detection system identifies new objects to be picked by calculating their pose relative to the robot base and their placement pose. Once the pick is calculated, the motion generation system encodes the recorded motion, optimizing it and generating online the trajectory of the robotic arm to pick the new objects (whether they are static or moving) and then position them appropriately according to the robot's predefined task (e.g., packaging, palletizing, machine tending, etc.).The aim is to reduce the programming time of the robotic arm through kinesthetic demonstration of the motion by the human, to automatically switch from demonstration mode to autonomous mode by pressing a button, and to increase the flexibility offered by the robotic arm through the detection system and through online motion generation based on the encoded demonstration. The technical problem solved by the invention is the ability to demonstrate the work both by experienced users who are familiar with robots and by inexperienced and occasional users, with great efficiency. The programming time is alsoreduced, because only one demonstration from the pick position to the placement position is sufficient, since the motion generation system will perform the reverse motion for picking the next objects. In addition, it is sufficient to press a button to signal the completion of the demonstration and the start of autonomous operation. Finally, the robot's trajectory during autonomous operation will be calculated onlince according to the demonstration, without being predetermined, increasing the flexibility of the robotic system.

[0008] The present invention has an advantage over competing technologies and products as it achieves trajectory demonstration in a minimum of time, with minimal effort from the user, without the need for special training, and offers high flexibility for the robotic system through the combination of task encoding, automatic motion generation and detection system from an image sensor. It also achieves improved robot adaptability in variable environments and enhances their ability to perform complex tasks.Human motion demonstration is used to a very small extent in industry and is usually limited to storing control points, through which the robot must pass. The novelty of this patent is in the demonstration and recording of the entire trajectory, while recording both the position of the robot and its velocity and acceleration. These are then appropriately encoded to optimize the motion that the robot will perform autonomously, in terms of one or more criteria such as reducing the change in acceleration to make the robot's motion smoother in autonomous mode.

[0009] A new feature is that it only takes the press of a button to signal the completion of the demonstration, to encode the motion and for the robot to starts the autonomous operation. The trajectory generated by the motion generation system and executed by the robot will be identical to the demonstration and optimized with respect to one or more criteria, offering flexibility, since it can generalize the demonstrated trajectory from a different initial to a different final point, even if they are in motion (e.g., picking up an object from or placing it on a moving belt). It is also possible for the generated trajectory to pass through some dynamic points that are calculated during execution (e.g. to avoid obstacles).

[0010] Finally, the placement of objects during autonomous operation of the robot is not always in the same position as the demonstration, because the motion generation system uses information from both the detection system and from the human regarding the type of task to be performed. If, for example, a tactical positioning task has been selected (such as in product boxing), the motion generation system selects the placement positions according to their dimensions so that the objects are arranged in their positioning locations (which may either be predefined or identified by the detection system) and rotates themappropriately so that they all have the same orientation, even if they were picked up from random orientations.[Oil] Summarizing the above, the invention consists of a method for kinesthetic guidance of a robotic arm for autonomously performing pick and place tasks of objects with adaptability characterized by a) kinesthetic guidance of the arm to the initial position (grasping position), b) activation of the recording, c) kinesthetic guidance of the arm to the final position (placement position) with simultaneous recording of the initial position, of the trajectory (position, velocity, and acceleration) and of the final position in the recording system; d) completion of the recording at the final position and encoding of the demonstration; e) and simultaneous transition to autonomous operation with generation of an optimized trajectory similar to that of the demonstration, for picking new objects from different positions and placing them according to the demonstration.In the initial position (pick position) the robotic arm picks an object and then places it at the final position (placement position). The method generates motion in which the trajectory that the robot will perform in autonomous operation is adjusted according to the initial and final position of each object, even if those have changed. Also, the method generates motion that takes into account the passage through intermediate spatial points that did not belong to the trajectory of the demonstration. During autonomous operation, new objects to be picked are detected by calculating their position and orientation with respect to the robot base and their position and orientation for placement by the robotic arm.The method is performed by means of a system, consisting of a) a computer, b) an object detection system with at least one image sensor, c) a system for recording and encoding the movements of the robotic arm, d) a task (movement) database, e) a system for generating the trajectory of the robotic arm, f) a button, physical or virtual in a graphical environment, g) an interface for selecting the type of work of the robot. The image sensor is placed either in a fixed position relative to the robot base or in a variable position, on the arm.The method is used, for example, for packaging of products, boxing, palletizing or de-palletizing of boxes of the same or different types and dimensions, machine tending, handling of objects for the purpose of classification, sorting etc.Description of the figures

[0012] Figure 1 shows a robotic arm (1) with an end-effector (2) and a coordinate system mounted on its base (3). Through a camera (4), the detection system detects objects to be picked in a predefined area (5) in order to place them in an appropriate manner in another user-defined area (6). The method is implemented on a computer (7) to which the camera (4) and the graphical interface (8) are connected. The computer (7) is connected to the robot control unit (9). The button (10) is connected to the control unit (9) via the arm (1).

[0013] Figure 2 shows this robotic arm (1) with an end-effector (2) and a coordinate system mounted on its base (3), in order to demonstrate to the user (9) how to pick objects (7) randomly located in one area (5) and place them in an ordered manner in another area (6). Initially the user (9) determines via the graphical interface (8) the type of task and sets the parameters. By selecting the task type, the detection system is activated and the positions of the objects (7) are detected by the camera (4). At the same time, the hand-guidance mode is activated for the user to be able to guide the robot kinesthetically.

[0014] Figure 3 shows this robotic arm (1) with an end-effector (2) and a coordinate system mounted on its base (3), which has been guided by the user over an object (9) in the predefined area (5). By pressing the button (8), the end-effector (2) is activated to pick the object (9) and the motion recording is activated.

[0015] Figure 4 shows this robotic arm (1) being moved by the user while the end-effector (2) carries the picked object (9) taken from its initial position (10) to the placement area (6), and while the movement is recorded (11).

[0016] Figure 5 shows this robotic arm (1) that has been moved by the user to the placement area (6) and the end-effector (2) carries the picked object (9). By pressing the button (8), the end-effector (2) releases the object (9) and the motion recording (11) is completed. Then, the motion is encoded and the trajectory can be generated so that the robot can pick up the remaining objects (7) in area (5) and place them in an arranged manner in area (6).

[0017] Figure 6 shows this robotic arm (1) in autonomous operation after a few repetitions of the movement (11) that was demonstrated by the user, with its end-effector (2) carrying an object taken from the position (12) of the picking area (5), which is continuously fed with new objects (7). The robotic arm (1) is moving by performing a trajectory (13) similar to the demonstrated one, for place the object at the position (15) of the placement area (6) in an ordered sequence, next to the objects (9) it has previously placed. The trajectory is adapted to pass through an intermediate point (16) located over an obstacle (14) located between the pick position (12) and the placement position (15).

[0018] Figure 7 shows the connection diagram of the functions of the method.Realisation of the invention

[0019] Example 1 (Figures 1 and 2) shows this robotic arm (1) with an endeffector suitable for picking objects (two fingers) (2) and a coordinate system mounted on its base (3), in order to be demonstrated by the user (9) how to pick objects (7) that are randomly located in one area (5) and place them in an ordered manner in another area (6). Initially, the user (9) determines via the graphical interface (8) that a task of orderly placement of objects in the same orientation is to be performed and sets the parameters. By selecting the type of task, the detection system is activated and the poses of the objects (7) aredetected by the image sensor (4). At the same time, the hand-guidance mode is activated for the user to be able to guide the robot kinesthetically.The user (figure 3) demonstrates the task to be performed by the robot (1) by guiding it, initially, over the object (9) to be picked. By pressing a button (8) located on the arm, the human signals the activation of the end-effector (closing of the fingers (2)) for the robot to pick the object (9) and, at the same time, the start of the demonstration. At the same time, the detection system calculates the relative position of the object with respect to the end-effector during the demonstration of the pick and the task recording system stores it.The user then demonstrates the trajectory to the robot (1) (Figure 4) by moving it from the pick area (5) to the placement area (6). The recording system stores the initial (1), final and all intermediate poses of the robotic arm (11), including the velocity and acceleration profiles from the demonstration.(Figure 5) When the user has moved the arm (1) and the object (9) to the desired place position within the placement area (6), they signal the completion of the demonstration by pressing the same button (8). By pressing the button (8) the recorded movement (11) is encoded in the task database. The encoding involves optimizing the demonstration by reducing changes in the acceleration profile, smoothing the demonstrated motion. Once the encoding and storage in the database is completed, the autonomous operation of the robot is activated.In autonomous mode (Figure 6), the detection system detects the objects to be picked (7) in the pick area (5) that are fed by a moving conveyor belt, calculating their position and orientation with respect to the robot base. Once at least one object (7) is detected, the motion generation system generates a trajectory similar to the demonstrated one to move the arm from the pick position (6) to the position of the closest object (12) to be picked. After picking the object (2), the motion generation system generates the motion of the robot to place the new object (2) in a position (15) according to the preselected task and based on the information (either at the demonstrated placement position or in the available space next to / above the previous objects) provided by the detection system, rotating it if necessary. The movement (13) of the robot is calculated according to the new pick (12) and placement positions (15) and intermediate points (16) located over an obstacle (14) to be avoided. The process is repeated until the object feeding is stopped or until the available placement positions have been completed.

Claims

CLAIMSClaim 1: A method for guiding a robotic arm (1) for autonomously performing object pick and place tasks with adaptive capabilities characterized by a) kinesthetic guidance of the arm (1) to the initial position (pick position) (12), b) activation of the recording, c) kinesthetic guidance of the arm (1) to the final position (placement position) (13) with simultaneous recording of the initial position, the trajectory (position, velocity and acceleration) and the final position (d) completing the recording at the final position and encoding the demonstration with machine learning; and (e) simultaneously switching to autonomous operation by generating an optimized trajectory, similar to that of the demonstration, for picking new objects from different positions and placing them in an arranged manner according to the demonstration.Claim 2: A method for guiding a robotic arm (1) for autonomously performing object pick and place tasks with adaptability, as described in claim 1, characterized in that in the initial position (pick position) (12) the arm picks an object (7) and places it in the final position (placement position) (13).Claim 3: A method for guiding a robotic arm (1) for autonomously performing object pick and place tasks with adaptability, as described in claims 1 and 2, characterized in that it generates motion to which the trajectory to be performed by the robot in autonomous operation is adapted according to the initial and final position of each object, even if they have changed.Claim 4: A method for guiding a robotic arm (1) for autonomously performing object pick and place tasks with adaptability, as described in claim 1, characterized in that it generates motion that takes into account passing through intermediate points that were not part of the trajectory of the demonstration.Claim 5: A method for guiding a robotic arm (1) for autonomously performing object pick and place tasks with adaptability, as described in the preceding claims, characterized in that, during autonomous operation, new objects (7) to be picked are identified by calculating their position and orientation with respect to the robot base (3) and their placement position and orientation by the robotic arm (1).Claim 6: A system for performing the method for guiding a robotic arm (1) as described in claims 1 to 4, comprising a) a computer (8), b) an object detectionsystem with at least one image sensor (4), c) a system for recording and encoding the movements of the robotic arm (1), d) a task (movement) database, e) a system for generating the trajectory of the robotic arm, f) a button (8), physical or virtual in a graphical environment, g) an interface for selecting the type of work of the robot.Claim 7: A system for performing the robotic arm guidance method as described in claim 5, characterized in that the image sensor (4) is placed either in a fixed position relative to the robot base (3) or in a variable position on the arm.Claim 8: Use of the robotic arm guidance method as described in claims 1 to 5, characterized in that it is used for packaging products, boxing, palletizing or depalletizing boxes of the same or different types and dimensions, machine tending, handling objects for the purpose of classification, sorting.

Citation Information

Patent Citations

  • Loading robot arm control self-adaption blended learning mapping intelligent control method and system

    CN107253195A

  • Robot assembly trajectory optimization method and device for offline example learning

    CN110561430A

  • Robot assembly demonstration track extraction method and device for offline example learning

    CN110561431A

  • Example learning system and method of robot assembly off-line for motion capture

    CN110561450A

  • Mechanical arm optical guidance system for automatically shielding external light source interference

    CN111923041A