Method for teaching a robot arm a movement sequence

Abstract

The invention relates to a method for teaching a reinforcement sequence to a robot arm (10), comprising the following method steps: Moving the robot arm (10) by a user in such a way that its distal end (12) and in particular a tool (16) attached to it touches a workpiece (14), wherein the workpiece (14) is only touched on a part of the surface (14a) of the workpiece (14), that is, not on the entire surface of the workpiece, Capturing contact points between the distal end (12) of the robot arm (10) and the workpiece (14), so that a contact point cloud is created, Scanning the surface (14a) of the workpiece (14) using a scanning method, so that a scan point cloud is created, Assigning the captured contact point cloud and the scan point cloud to a virtual model of the surface (14a) of the workpiece (14), Extrapolating the motion demonstrated on part of the surface (14a) to the entire surface of the virtual model to be processed.

Description

[0001] The invention relates to a method for teaching a robot arm a movement sequence.

[0002] Surface finishing, such as grinding or polishing, is still largely performed manually in industrial and craft applications. These tasks are often monotonous, strenuous, unergonomic, or even hazardous to health. Especially in small and medium-sized enterprises (SMEs) and with batch sizes in the single or double digits, the use of robots for surface finishing is not yet feasible due to a lack of intuitive robot programming methods or automated programs. For larger production runs, an economical solution is possible by having a robot follow a predefined (programmed) trajectory, thus processing a defined area of ​​an identical component. This method requires specialists in robot operation and, due to cost-benefit considerations, is not economically viable for small production runs.

[0003] Various methods for using robots for surface processing can be found in the following state of the art: [1] US-20230126085-A1, US-11613014-B2, US 2022 / 0 388 115 A1 [2] DE 10 2015 004 146 B3 [3] DE 10 2020 209 511 B3, WO 2022 / 022 784 A1 [4] DE 10 2016 219 128 B4 [5] DE 10 2021 204 215 A1 Commercial solutions: [6] Robotiq Sanding Kit: https: / / www.y-outube.com / watch?v=ewxYBsqvYbc, https: / / robotiq.com / de / produkte / schleifkit Scientific papers: [7] Stefan Schneyer et al., DLIVTUM, Segmentation and Coverage Planning of Freeform Geometry for Robotic Surface, IEEE RA-L 2023 Finishinghttps: / / ieeexplore.ieee.org / stamp / stamp.jsp?tp=&arnumber=1 0175386 [8] Yalun Wen et al., Uniform Coverage Tool Path Generation for Robotic Surface Finishing of Curved Surfaces IEEE RA-L 2022 iittps: / / ieeexplore.ieee.org / stamp / stamp.jsp?tp=8zarnumber=971819 Z

[0004] Further state of the art is known from the following publications: DE 10 2022 122 685 A1, DE 10 2020 110 650 A1, DE 10 2019 001 207 A1, DE 10 2015 119 240 B3, DE 10 2011 082 800 B4 and US 11 529 733 B2.

[0005] The current state of the art does not allow for an intuitive learning or programming method that enables users without robotics expertise to make a robot autonomously process components, in particular to grind or polish them.

[0006] The solution in [6] offers a combination of graphical programming and parameterization via demonstration. With this solution, a program sequence must be created manually, the user must independently select the surface primitives for different sections of a component, and the boundaries must then be parameterized via demonstration. The system requires experience in robotics and is not intuitive to operate. In [2], an approach is described that aligns a mobile robot relative to a component and uses machining primitives to process the component's surface. Similar to [6], however, the machining primitive must be manually specified and parameterized.

[0007] Approach [3] uses a graphical interface to combine program modules that represent the structure of the task. The individual modules can then be parameterized via the graphical user interface. This is followed by an exploration phase of the robot, which serves to optimize the selected parameters. During this exploration phase on the component, training data is collected, with the execution being repeated 100 to 1000 times. This process requires a very long training and exploration phase, during which the component could be damaged. This approach is therefore not suitable for small production runs or for single-unit production.

[0008] The approaches in [1], [3], [7], [8] use CAD data or a surface scan of a component and generate a robot trajectory based on this to process the entire surface of the component uniformly. However, different surfaces or edges of the components may require different strategies or parameters for surface processing.

[0009] In [4], an approach based on depth image data and contact forces is used to monitor a robot's wiping motion during execution. This approach is not intended for programming or training the robot to perform a task. In [5], an adaptation of the processing strategy based on measurement results is described; this requires additional sensors to measure surface texture.

[0010] The object of the invention is to provide an easy-to-use method for teaching a robot arm a movement sequence, which nevertheless enables high accuracy when teaching the desired task.

[0011] The object is achieved according to the invention by a method according to claim 1.

[0012] In the inventive method for teaching a robot arm a movement sequence, the following steps are performed: The robot arm is moved by a user in such a way that its distal end, and in particular a tool attached to it, touches a workpiece. This contact point or contact point cloud can also be generated without the robot, for example, by the user running their finger along a surface while at least one camera, especially the HoloLens, registers the contact points between the finger and the object. The workpiece is only touched on a portion of its surface, not its entire surface. The workpiece can have various surface shapes, such as flat surfaces, spherical surfaces, etc.

[0013] Contact points between the distal end of the robot arm or the user's finger and the workpiece are recorded, creating a contact point cloud. Additionally, the workpiece surface is scanned, generating a scan point cloud. A depth camera or lidar sensor, for example, can be used for this purpose.

[0014] The captured contact point cloud and scan point cloud are assigned to a virtual model of the workpiece surface. A selection can be made from existing virtual models for different surfaces, such as a sphere, a plane, or a straight line.

[0015] The movement demonstrated on the part of the surface is then extrapolated to the entire surface of the virtual model to be processed.

[0016] The tool thus touches a surface of a workpiece, and a desired movement is performed on this surface with the tool. Alternatively, the workpiece can be touched with a user's finger, with the finger being moved along the surface of the workpiece.

[0017] In the method according to the invention, the information from the scan point cloud and the contact point cloud are combined to create the virtual model, wherein the contact point cloud preferably has fewer points than the scan point cloud. A user only needs to demonstrate the desired movement on a small part of the surface, so that it can then be executed independently by the robot arm on the entire surface represented by the virtual model.

[0018] It is preferable to choose the virtual model that exhibits the highest degree of correspondence with the contact points between the tool and the workpiece. For example, if the workpiece is a sphere, a virtual model of a sphere will have the greatest correspondence, meaning the most points in common with the contact point cloud, while other models, such as a virtual model of a plane, will have fewer points in common.

[0019] It is preferred that any surfaces and / or edges of the workpiece can be converted into a virtual model by capturing the contact point cloud and the scan point cloud.

[0020] Furthermore, it is preferred that the RANSAC method be used as the surface segmentation method, which receives the contact point cloud and the scan point cloud as input.

[0021] Furthermore, it is preferred that the user's demonstration of the movement is used to define parameters of the desired movement. These parameters can include the contact force and / or the tilt angle between the tool and the workpiece. Additionally or alternatively, a sub-strategy, in particular a circular movement and / or a deflection perpendicular to the direction of movement, can be defined through the demonstration of the movement.

[0022] In a preferred embodiment, the determined virtual model, in particular as a point cloud, is displayed to a user directly on the workpiece or on a model of the workpiece in an extended reality representation.

[0023] In this extended reality display, an interface can also be presented to the user through which input can be made. For example, after the user has demonstrated movement, the virtual model can be adjusted via the user interface. For instance, its boundaries can be adjusted and / or zones can be defined that should not be processed and / or should be processed using a different strategy and / or parameterization.

[0024] It is still preferred that an approach based on the geometry and pose of the workpiece and tool, as well as the recorded contact force, be used to estimate the contact between tool and workpiece at each point in time during the demonstration of the movement.

[0025] Preferred embodiments of the invention are explained below with reference to figures. Fig. Figure 1 shows how a user teaches the robot arm a desired movement. Fig. 2a and Fig. Figure 2b shows how a trajectory for the robot arm is generated from this.

[0026] For this purpose, the user moves the robot arm 10 towards the workpiece 14 in such a way that a tool 16 attached to its distal end 12 touches the surface 14a of the workpiece 14. On this surface 14a, the user performs a desired movement with the tool 16. This could be, for example, a grinding or polishing movement, in which the tool is moved along the surface 14a of the workpiece 14.

[0027] Workpiece 16 was previously scanned using a scanning process, so that its surface was captured. This scan point cloud is merged with the contact point cloud to generate a virtual model of the surface 14a of workpiece 14.

[0028] The movement demonstrated on part of the surface 14a of the workpiece 14 is extrapolated to the entire surface to be machined of the virtual model, so that the robot arm can independently perform the desired movement on the entire surface 14a to be machined of the workpiece 14.

[0029] The Fig. Figure 2a shows the segmented surface primitive 14a of the workpiece 14, with contact points 18 visible at its upper right corner, which were detected by the contact between the tool 16 and the workpiece 14.

[0030] In Fig. Figure 2b shows the robot trajectory generated from this, which runs along the segmented surface primitives 14a. Based on the recognized surface primitive, which in this example is a plane, a strategy is chosen to manipulate the surface. This strategy is parameterized using the demonstration.

[0031] By using the contact points from the demonstration, the user implicitly defines the surface primitives (line, plane, sphere, etc.) that are detected on the workpiece using the segmentation approach. Since the segmentation approach automatically detects a surface primitive, the user only needs to demonstrate a limited section of the area to be machined. The strategy can then be extrapolated to the entire surface. For this purpose, the robot can be provided with a specific strategy for each surface primitive, ensuring that the entire segmented surface is machined. This strategy can be defined by an expert or learned from a demonstration and can also include impedance or force control of the robot.This enables the method to be used even for single-unit production, as the implementation time can be significantly reduced by extrapolating the strategy based on the automatically detected surface primitives. In practice, it amounts to just a few seconds.

[0032] In a second step, the demonstration can also be used to parameterize the strategy. The extracted parameters include, for example, the contact force or the tilt angle between the tool and the workpiece, or a sub-strategy (circular motion, deflection normal to the direction of movement, etc.). Existing approaches from 1.3 do not use an approach to segment and differentiate the surface primitives in order to potentially use different strategies or parameters for the different segments. Publication [4] uses a segmentation approach, but it is not used for programming, only for monitoring during execution. In [6], a user must independently identify the different surface primitives and select a suitable machining strategy. This strategy then has to be parameterized in a cumbersome way that is not comprehensible to a layperson.None of the known approaches allow the parameterization of the strategy used via demonstration. For a layperson, it is easier to demonstrate the contact force, the tilt angle, or a sub-strategy than to specify them via a graphical interface or to determine exact values ​​for them.

[0033] The invention enables a continuous, bidirectional flow of information between the robot and the user. The developed method includes a graphical user interface that can be implemented via Extended Reality (XR) interfaces or a classic GUI. Through this interface, the user can, for example, continuously and directly view the segmented surface primitives on the component or in a virtual model of the component. The segmentation of a surface primitive begins with the user demonstration and continuously improves as more training data (contact points between the tool and the workpiece) is added. Thus, while programming the task, the user can constantly verify the model currently being learned by the robot, recognize the influence of their demonstration, and, if necessary, influence the result.Using the user interface, the user can adjust the segmentation (move boundaries) or define zones that should not be processed or should be processed with a different strategy / parameterization after the demonstration. The XR interface can be used to specify this information directly on the component. For example, a specific zone can be defined using gesture control. Alternatively, a classic GUI with a virtual component can be used. After the programming / training phase is complete, the execution on the robot is planned using the known strategy for the identified surface primitives. The result of this planning can be displayed to the user via the interface. The planned trajectory on the component can be visualized as a contact line or simulated directly in the workspace using a virtual robot.After reviewing the planned execution, the user can adjust the trajectory or parameters again and send the refined plan to the robot for autonomous processing.

[0034] Compared to existing approaches, this invention allows the user to be continuously informed about the robot's learned model. The discrepancy between the model actually learned by the robot and the model the user believes the robot has learned is a known problem and the subject of current research. By informing the user about the model actually learned by the robot, this discrepancy can be reduced. This leads to faster robot programming with better results, as any errors that may occur are immediately displayed visually and are not only observed during execution. Furthermore, it improves the user's confidence in the robot and allows for targeted influence on the learned model by adjusting the demonstration.

Claims

[1] Method for teaching a movement sequence for a robot arm (10), comprising the following method steps: Moving the robot arm (10) by a user such that its distal end (12) and in particular a tool (16) attached thereto touches a surface of a workpiece (14), and on this surface (14a) a desired movement is carried out with the tool (16) or touching the workpiece with the finger of a user, moving the finger along the surface and detecting the position of the finger by at least one camera, wherein the contact points between finger and object are registered, wherein the contact of the workpiece (14) occurs only on a part of the surface (14a) of the workpiece (14), i.e. not on the entire surface of the workpiece, Detecting contact points between the distal end (12) of the robot arm (10) or the user's finger and the workpiece (14), thereby creating a contact point cloud, Scanning the surface (14a) of the workpiece (14) by a scanning process, thereby creating a scan point cloud, Assigning the acquired contact point cloud and the scan point cloud to a virtual model of the surface (14a) of the workpiece (14), Extrapolating the movement demonstrated on the part of the surface (14a) to the entire surface of the virtual model to be machined. [2] Method according to claim 1, characterized by that the virtual model is selected which has the highest correspondence with the contact points between the tool (16) and the workpiece (14). [3] Method according to claim 2, characterized bythat any surfaces and / or edges of the workpiece (14) can be converted into a virtual model by capturing the contact point cloud and the scan point cloud. [4] Method according to claims 1-3, characterized by that the RANSAC method is used as the surface segmentation method, which receives the contact point cloud and the scan point cloud as input. [5] Method according to claims 1-4, characterized by that the demonstration of the movement by the user is used to define parameters of the desired movement, in particular the contact force and / or the tilt angle between the tool (16) and the workpiece (14) and / or a sub-strategy, in particular a circular movement and / or a deflection normal to the direction of movement. [6] Method according to claims 1-5, characterized bythat the determined virtual model, in particular as a point cloud, is displayed to a user directly on the workpiece (14) or on a model of the workpiece (14) in an extended reality representation. [7] Method according to claims 1-6, characterized by that a user interface is shown in the extended reality representation. [8] Method according to claim 7, characterized by that, after the user has demonstrated the movement, the limits of the virtual model can be adjusted via the user interface and / or zones can be defined that should not be processed and / or should be processed using a different strategy and / or parameterization. [9] Method according to claims 1-8, characterized bythat to estimate the contact between tool (16) and workpiece (14) at each time point of the demonstration of the movement, an approach based on the geometry, the pose of the workpiece (14) and the tool (16) and the recorded contact force is used.

Images