Method for a robot arm to learn a sequence of movements

Abstract

The invention relates to a method for a robot arm (10) to learn a sequence of movements, comprising the following method steps: a user moving the robot arm (10) in such a way that its distal end (12), and in particular a tool (16) fastened to this end, touches a workpiece (14), wherein only part of the surface (14a) of the workpiece (14), that is to say not the entire surface of the workpiece, is touched, sensing contact points between the distal end (12) of the robot arm (10) and the workpiece (14) so as to produce a contact-point cloud, scanning the surface (14a) of the workpiece (14) by a scanning process so as to produce a scanning-point cloud, assigning the sensed contact-point cloud and the scanning-point cloud to a virtual model of the surface (14a) of the workpiece (14), and extrapolating the movement performed on the part of the surface (14a) to the entire surface to be worked of the virtual model.

Description

[0001] Method for teaching a movement sequence for a robot arm

[0002] The invention relates to a method for teaching a movement sequence for a robot arm.

[0003] Surface processing, such as grinding or polishing, is largely carried out manually in industrial / crafts applications. These tasks are often monotonous, strenuous, unergonomic, or even harmful to health. Especially in small and medium-sized companies, and with batch sizes in the single or double digit range, the use of robots to carry out surface processing is not yet possible due to the lack of intuitive robot programming methods or automated programs. For larger quantities, an economical solution to the problem is possible by having a robot follow a predefined (programmed) trajectory, whereby a defined surface of a constantly identical component can be processed. This process requires specialists in handling robots and cannot be implemented economically in small series due to cost-benefit considerations.

[0004] Various methods for using robots for surface processing can be found in the following state of the art:

[0005] [1] GrayMatter Robotics: System and method for autonomously scanning and processing apart (US-20230126085-A1, US-11613014-B2, US-20220388115-AI, etc.) https: / / image-ppubs.uspto.gov / dirsearch-publidprint / downloadPdf / 202301 26085 , https: / / image-ppubs.uspto.gov / dirsearch-publidprint / downloadPdf / 20220388115, https: / / image-ppubs.uspto.gov / dirsearch-publidprint / down-loadPdf / 20220388115

[0006] [2] Mobile robot, machining primitive: KUKA, surface machining of a component using a mobile robot (DE10201500414683) https: / / depatisnet.dpma.de / DepatisNet / depatis- net?action = bibdat&docid = DE10201500414683

[0007] [3] Surface treatment, robotics: Artiminds, METHOD AND SYSTEM FOR DETERMINING

[0008] Optimized program parameters for a robot program (DE102020209511B3, W0002022022784A1) https: / / depatis- net.dpma.de / DepatisNet / depatisnet?action=bib- dat&docid=W0002022022784Al

[0009] [4] Grinding, Robotics: DLR, Method for performing wiping movements by a robot arm (DE10201621912884) https: / / depati- snet.dpma.de / DepatisNet / depatisnet?action=pdf&docid=DE102016219128B4

[0010] [5] Automated surface processing: Fraunhofer, Processes for the automated processing of workpiece surfaces by polishing, grinding or painting (DE102021204215A1) https: / / depatis- net.dpma.de / DepatisNet / depatisnet?action=bib- dat&docid=DE102021204215Al

[0011] Commercial solutions:

[0012] [6] Robotiq Sanding Kit: https: / / www.youtube.com / watch?v=ewxYBsqvYbc, https: / / robotiq.com / de / pro- dukte / schleifkit Scientific papers:

[0013] [7] Stefan Schneyer et al., DLIVTUM, Segmentation and Coverage Planning of Freeform Geometry for Robotic Surface, IEEE RA-L 2023 Finishinghttps: / / ieeexplore.ieee.org / stamp / stamp.jsp?tp=&arnumber=l 0175386

[0014] [8] Yalun Wen et al., Uniform Coverage Tool Path Generation for Robotic Surface Finishing of Curved Surfaces IEEE RA-L 2022 iittps: / / ieeexplore.ieee.org / stamp / stamp.jsp?tp=8zarnumber=971819 Z

[0015] The current state of the art does not provide an intuitive learning or programming process that would enable users without robotics expertise to instruct a robot to process components autonomously, in particular to grind or polish them.

[0016] The solution in [6] offers a combination of graphical programming and parameterization via demonstration. With this solution, a program flow must be manually created, the surface primitives for different sections of a component must be selected independently by the user, and boundaries must then be parameterized via demonstration. The system requires experience in robotics and is not intuitive to use. In [2], an approach is described that aligns a mobile robot relative to a component and uses machining primitives to machine the surface of the component. Similar to [6], however, the machining primitive must be manually specified and parameterized.

[0017] In approach [3], a graphical interface is used to combine program blocks that represent the structure of the task. The individual blocks can then be parameterized via the graphical user interface. This is followed by an exploration phase of the robot, which serves to optimize the selected parameters. During this exploration phase on the component, training data is collected, with the execution being repeated 100 to 1,000 times. This process requires a very long training or exploration phase, which could result in damage to the component. This approach is therefore not suitable for small batches or for single-piece production.

[0018] The approaches in [1], [3], [7], [8] use CAD data or a surface scan of a component and generate a robot trajectory based on this data to uniformly machine the entire surface of the component. However, different surfaces or edges of the component may require different strategies or parameters for surface processing.

[0019] In [4], an approach based on depth image data and contact forces is used to monitor a robotic wiping motion during execution. This approach does not involve programming or learning a task using the robot. [5] describes an adaptation of the machining strategy based on measurement results, which requires additional sensors to measure the surface texture.

[0020] The object of the invention is to provide an easy-to-use method for teaching a movement sequence for a robot arm, which nevertheless enables a high level of accuracy when teaching the desired task.

[0021] The object is achieved according to the invention by a method according to claim 1.

[0022] In the method according to the invention for teaching a movement sequence for a robot arm, the following steps are carried out:

[0023] The robot arm is moved by a user such that its distal end, and in particular a tool attached to it, touches a workpiece. These contact points or contact point clouds can also be generated without the robot, for example, by the user moving their finger along a surface while at least one camera, in particular the HoloLens, registers the contact points between the finger and the object. Touching the workpiece only occurs on a portion of the surface, i.e., not on its entire surface. The workpiece can have various surfaces, for example, flat surfaces, spherical surfaces, etc.

[0024] Contact points between the distal end of the robot arm or the user's finger and the workpiece are recorded, creating a contact point cloud. Furthermore, the surface of the workpiece is scanned using a scanning process, creating a scan point cloud. A depth camera or a lidar sensor, for example, can be used for this purpose.

[0025] The acquired contact point cloud and scan point cloud are assigned to a virtual model of the workpiece surface. This can be done by selecting from existing virtual models for various surfaces, such as a sphere, a plane, or a straight line.

[0026] The movement demonstrated on the part of the surface is then extrapolated to the entire surface of the virtual model to be machined.

[0027] In the method according to the invention, the information from the scan point cloud and the contact point cloud is thus combined to create the virtual model, wherein the contact point cloud preferably has fewer points than the scan point cloud. A user only has to demonstrate the desired movement on a small part of the surface so that it can then be carried out independently by the robot arm on the entire surface represented by the virtual model. It is preferable to select the virtual model that has the greatest correspondence with the contact points between tool and workpiece. If the workpiece is a sphere, for example, a virtual model of a sphere will have the greatest correspondence, i.e. the most common points, with the contact point cloud, while other models, for example the virtual model of a plane, will have fewer common points.

[0028] It is preferred that any surfaces and / or edges of the workpiece can be converted into a virtual model by capturing the contact point cloud and the scan point cloud.

[0029] Furthermore, it is preferred that the RANSAC method is used as the surface segmentation method, which receives the contact point cloud and the scan point cloud as input.

[0030] Furthermore, it is preferred that the user's demonstration of the movement be used to define parameters for the desired movement. These can be the contact force and / or the tilt angle between the tool and the workpiece. Furthermore, a sub-strategy, in particular a circular movement and / or a deflection normal to the direction of movement, can be additionally or alternatively defined through the demonstration of the movement.

[0031] In a preferred embodiment, the determined virtual model is displayed to a user, in particular as a point cloud directly on the workpiece or on a model of the workpiece in an extended reality representation.

[0032] In this extended reality representation, the user can also be presented with an interface through which inputs can be provided. For example, after the user has demonstrated the movement, the virtual model can be adjusted via the user interface. For example, its boundaries can be adjusted and / or zones can be defined that should not be processed and / or should be processed using a different strategy and / or parameterization.

[0033] It is further preferred that an approach based on the geometry and pose of the workpiece and tool, as well as the recorded contact force, be used to estimate the contact between tool and workpiece at any time during the motion demonstration.

[0034] In the following, preferred embodiments of the invention are explained with reference to figures.

[0035] Figure 1 shows the user teaching a desired movement of the robot arm.

[0036] Figures 2a and 2b show how a trajectory for the robot arm is generated from this.

[0037] To do this, the user moves the robot arm 10 toward the workpiece 14 such that a tool 16 attached to its distal end 12 touches the surface 14a of the workpiece 14. On this surface 14a, the user performs a desired movement with the tool 16. This can be, for example, a grinding movement or a polishing movement in which the tool is moved along the surface 14a of the workpiece 14.

[0038] The workpiece 16 was previously scanned using a scanning process to capture its surface. This scan point cloud is merged with the contact point cloud to generate a virtual model of the surface 14a of the workpiece 14. The movement demonstrated on the portion of the surface 14a of the workpiece 14 is extrapolated to the entire surface of the virtual model to be machined, allowing the robot arm to independently execute the desired movement on the entire surface 14a of the workpiece 14 to be machined.

[0039] Figure 2a shows the segmented surface primitive 14a of the workpiece 14, with contact points 18 visible at its upper right corner, which were detected by the contact between the tool 16 and the workpiece 14.

[0040] Figure 2b shows the resulting robot trajectory, which runs along the segmented surface primitive 14a. Based on the detected surface primitive, which in this example is a plane, a strategy is selected to process the surface. This strategy is parameterized using the demonstration.

[0041] By using the contact points from the demonstration, the user implicitly defines the surface primitives (line, plane, sphere, etc.) that will be detected on the component to be machined using the segmentation approach. Since the segmentation approach automatically detects a surface primitive, the user only needs to demonstrate a limited section of the surface to be machined. The strategy can then be extrapolated to the entire surface. To do this, the robot can have a specific strategy available for each surface primitive, ensuring that the entire segmented surface is machined. This strategy can be defined by an expert or learned from a demonstration and can also include impedance or force control of the robot.This enables the method to be used even for single-piece production, as implementation time can be significantly reduced by extrapolating the strategy based on the automatically detected surface primitive. In practice, this takes just a few seconds. In a second step, the demonstration can also be used to parameterize the strategy. The extracted parameters are, for example, the contact force or the tilt angle between tool and workpiece, or a sub-strategy (circular motion, deflection normal to the direction of motion, etc.). Existing approaches from 1.3 do not use an approach to segment the surface primitives and differentiate them in order to potentially use different strategies or parameters for the different segments. Publication [4] uses a segmentation approach, but this is not used for programming, but only for control during execution.In [6], a user must independently identify the different surface primitives and select a suitable machining strategy. This strategy must then be parameterized in a cumbersome manner that is incomprehensible to a layperson. None of the known approaches allows the parameterization of the strategy used via demonstration. For a layperson, it is easier to demonstrate the contact force, the tilt angle, or a sub-strategy than to specify or determine exact values ​​for them via a graphical interface.

[0042] The invention enables a continuous bidirectional flow of information between robot and user. The developed method includes a graphical user interface that can be implemented via Extended Reality (XR) interfaces or a classic GUI. For example, the user interface can continuously display the segmented surface primitive directly on the component or in a virtual model of the component. The segmentation of a surface primitive only begins with the user demonstration and continuously improves over the course of the demonstration with more training data (contact points between tool and workpiece). Thus, while programming the task, a user can constantly verify the model currently learned by the robot, recognize the influence of their demonstration, and influence the result if necessary.After the demonstration, the user can use the user interface to adjust the segmentation (move boundaries) or define zones that should not be processed or should be processed with a different strategy / parameterization. The XR interface can be used to specify this information directly on the component. A specific zone can be defined using gesture control, for example. Alternatively, a classic GUI with a virtual component can be used. After the programming / training phase is complete, the execution on the robot is planned using the known strategy for the identified surface primitive. The result of this planning can be displayed to the user via the interface. The planned trajectory can be visualized as a contact line on the component or simulated directly in the workspace using a virtual robot.After reviewing the planned execution, the user can adjust the trajectory or parameters again and send the refined plan to the robot for autonomous processing.

[0043] Compared to existing approaches, the invention enables the user to be continuously informed about the robot's learned model. The discrepancy between the model actually learned by the robot and the model the user believes the robot has learned is a known problem and the subject of current research. By informing the user about the model actually learned by the robot, this discrepancy can be reduced. This leads to faster robot programming with better results, as any errors that may occur are immediately visually displayed and cannot only be observed during execution. Furthermore, it improves the user's trust in the robot and enables targeted influence on the learned model by adapting the demonstration.

Claims

Patent claims 1. Method for teaching a movement sequence for a robot arm (10), comprising the following method steps: Moving the robot arm (10) by a user such that its distal end (12) and in particular a tool (16) attached thereto touches a workpiece (14), or touching the workpiece with the finger of a user and detecting the position of the finger by at least one camera, wherein the touching of the workpiece (14) occurs only on a part of the surface (14a) of the workpiece (14), i.e. not on the entire surface of the workpiece, Detecting contact points between the distal end (12) of the robot arm (10) or the user's finger and the workpiece (14), thereby creating a contact point cloud, Scanning the surface (14a) of the workpiece (14) by a scanning process, thereby creating a scan point cloud, Assigning the acquired contact point cloud and the scan point cloud to a virtual model of the surface (14a) of the workpiece (14), Extrapolating the movement demonstrated on the part of the surface (14a) to the entire surface of the virtual model to be machined.

2. Method according to claim 1, characterized in that the virtual model is selected which has the highest correspondence with the contact points between the tool (16) and the workpiece (14).

3. Method according to claim 2, characterized in that any surfaces and / or edges of the workpiece (14) can be converted into a virtual model by detecting the contact point cloud and the scan point cloud.

4. Method according to claims 1 - 3, characterized in that the RANSAC method is used as the method for surface segmentation, which receives the contact point cloud and the scan point cloud as input.

5. Method according to claims 1 - 4, characterized in that the demonstration of the movement by the user is used to define parameters of the desired movement, in particular the contact force and / or the tilt angle between the tool (16) and the workpiece (14) and / or a sub-strategy, in particular a circular movement and / or a deflection normal to the direction of movement.

6. Method according to claims 1 - 5, characterized in that the determined virtual model, in particular as a point cloud, is displayed to a user directly on the workpiece (14) or on a model of the workpiece (14) in an extended reality representation.

7. Method according to claims 1 - 6, characterized in that a user interface is shown in the extended reality representation.

8. Method according to claim 7, characterized in that, after the user has demonstrated the movement, the boundaries of the virtual model can be adjusted via the user interface and / or zones can be defined which are not to be processed and / or are to be processed in a different strategy and / or parameterization.

9. Method according to claims 1 - 8, characterized in that to estimate the contact between tool (16) and workpiece (14) at each time point of the demonstration of the movement, an approach based on the geometry, the pose of the workpiece (14) and the tool (16) and the recorded contact force is used.

Images