Method for programming and executing movements of a robotic system

Abstract

The present invention relates to a method for programming and executing movements of a robotic system, comprising a robotic manipulator, an input device with a screen, and a database of skills that the robotic manipulator can perform. By drawing symbols with an input device displayed on a screen, robotic systems, preferably industrial robots and / or cobots, can be programmed easily and remotely.

Description

The invention relates to a method for programming and executing movement of a robotic system.It is known from the prior art that industrial robots are usually programmed by manually writing program code. To simplify this process, what are known as teaching pendants are frequently used, portable devices with a screen and keyboard. These devices usually record points in the robot program by first driving the robot to the desired point and then pressing a button.Collaborative robots-also called robots-are often programmed by manual guidance and by input in a graphical user interface. In this case, reusable program blocks are often used, which can be adapted to different tasks by parameterization. Known as skils or apps, these blocks are sold. The robot capabilities programmed in these blocks are referred to as skils below.The fact that industrial robots must be programmed by manually writing program code or with the aid of so-called teaching pendants results in additional complexity and inflexibility in the application. These methods require specific technical knowledge. In addition, hand-held programming presupposes the physical presence of the programmer on site.Background art on the present invention can be found in the following publications:DE 10 2023 125 994 A1DE 11 2021 007 517 T5DE 11 2021 005 036 T5DE 10 2020 200 024 A1DE 10 2015 209 900 A1DE 696 18 606 T2It is an object of the invention to provide a method for programming and executing movement of a robotic system that enables more intuitive user handling for programming. A corresponding robot system is also to be provided.The object is achieved according to the invention by claims 1 and 14.The method according to the invention has a robotic manipulator, in particular for manipulating objects. A robotic manipulator can be, for example, a gripper. A preferred robotic manipulator can be, for example, a gripper on a robot, such as an industrial robot or Cobot, which is used to grip, move, stitch or otherwise manipulate objects.The method according to the invention also has an input device with a screen.The input device is preferably used by the user in order to provide the robotic system with an input such as drawing, swiping, mouse movement, hand gesture or pressing a key. An input device is preferably a touchscreen for drawing with fingers and / or stylus; furthermore, a mouse or a keyboard or a camera for tracking hand movements and gestures in three-dimensional space is preferably used as input device. A screen can also be designed as a VR / AR headset. The display-screen input device can be both attached to and detached from the robot and exchange information with the robot via cable or radio or the Internet.The method according to the invention further comprises a database with skils, which the robotic manipulator can perform. The database is configured, among other things, to store the code for the skils. A skil refers to a programmable action block. These reusable program blocks can be adapted to different tasks with the aid of different skip parameters and form basic functions or skills which a robot can perform. A skil could be, for example, to grasp a particular object, to kick, or release, or to perform a specific manipulation or task.The method according to the invention has a method step for programming the robotic system. Manual management and graphical user inputs may alternate or supplement each other during the programming phase. Typically, these programming environments offer a fixed set of ski boots that the user can reuse in different task areas. The graphical user interfaces can be operated, for example, with a mouse and keyboard or via a touchscreen with a finger or stylus.The method step according to the invention for programming the robotic system comprises a method step for displaying the position of one or more objects to be manipulated. The screen of the robotic system serves as a display for the position of the object(s) to be manipulated. In a preferred embodiment, a virtual model of the robotic manipulator is presented, including its environment and objects to be manipulated. In a further preferred embodiment, a camera image / video of the environment of the robotic manipulator is displayed.The method step according to the invention for programming the robotic system further comprises a method step for drawing a symbol with an input device, which is drawn on the screen and reproduced at the drawn position. It is preferred that the symbol is drawn and reproduced on or on an object to be manipulated on the screen. A user thus preferably sees a virtual 3D environment or a camera video with an object to be manipulated on the screen and is capable of making an input, for example on a touchscreen on which the camera video is displayed. The input preferably corresponds to a symbol, for example a "Z". The user thus records a "Z" on or on the object to be manipulated, which is displayed on the screen at a corresponding position. It is furthermore preferred that the input takes place on, on or in the vicinity of an object to be manipulated.A first position is the position that was first designated in time order. A second position is the position that has been referred to as second in time order, etc.The method step according to the invention for programming the robotic system further comprises a method step for comparing the symbol with a series of symbols stored in the database. The skils stored in the database are associated with one or a series of symbols which the user can draw in order to carry out this skil. As soon as the user draws a symbol on or on an object to be manipulated or he has finished his input, the method according to the invention compares the drawn symbol with all or a subset of all symbols in the database and calculates the difference between the symbols as a similarity value. The difference between the symbols can be effected, for example, by "embedding" with the aid of a neural network; the vectors resulting therefrom can be compared, for example, with cosine similarity. In a further preferred embodiment, the similarity value could be determined by a convolutional neural network. A convolutional neural network (CNN) analyzes images by using various filters to recognize patterns. As a result, it can ascertain similarities between symbols. In a further preferred embodiment, a support vector machine could be used to determine the similarity value. A support vector machine (SVM) is a classifier that represents a hyper-dimensional space to separate patterns. It finds the optimal boundary separating the classes and maximizes the distance to the next data points, the so-called support vectors.The method step according to the invention for programming the robotic system further comprises a method step for assigning the drawn symbol to a symbol stored in the database and to the corresponding skil, as soon as a similarity value between stored and drawn symbol is higher than a confidence threshold. The system thus calculates a similarity value between each symbol in the database and the drawn symbol and assigns the drawn symbol to a skil as soon as the drawn symbol and one of the symbols in the database have a similarity value preferably between 80%-100%, or 90%-100%. If several symbols meet this condition, the symbol having the highest similarity value to the drawn symbol is chosen.The method according to the invention comprises, among other things, a method step for executing the movement of the robotic system.The method step according to the invention for executing the movement of the robotic system has a method step for moving the robotic manipulator to the position of the drawn symbol. Once programming is complete, the user preferably presses a button that causes the robotic system to execute the corresponding ski in the corresponding position. For this purpose, the robotic manipulator first moves to the position at which the symbol was drawn.The method step according to the invention for executing the movement of the robotic system further comprises a method step for executing the skil recognized by the drawn symbol. The skil is applied to the object which is closest to the symbol or on or on which the symbol has been drawn. A symbol is designed such that it can be assigned to the ski to be executed. Letters, numbers, or pictograms or drawings of the manipulator, as the corresponding skil is executed, are suitable for this purpose.It is preferred that the object on which the skil is executed is the next object from the drawn symbol. The next object is preferably determined by the minimum distance between the center point of the drawn symbol and the center point of the object to be manipulated. Should no object be in the vicinity of the drawn symbol, or the minimum distance to the next object exceeds a threshold, the skil is not executed and the user is asked to make a new input before the movement execution of the robotic system begins.It is preferred that the method according to the invention further comprises an object detection device which recognizes objects, obstacles, surfaces and generally the environment of the robotic manipulator to be manipulated. Such an object detection device could preferably be a camera, a lidar, radar, ultrasound or a comparable sensor. An object detection device could likewise be an object detection system which consists of a plurality of sensors, in particular a combination of the sensor types mentioned above.It is furthermore preferred that the method step for programming the robotic system further comprises a method step for detecting one or more manipulatable objects and / or detecting the robotic manipulator with the aid of the object detection device. This step serves to continuously update the display on the screen in order to continuously display the correct position of all objects and / or the robotic manipulator to the user.It is furthermore preferred that the method step for programming the robotic system takes into account the shape, color / reflectivity, and / or the surface condition and / or the material of the object to be manipulated when selecting the skil parameters. The shape of the object can be used, for example, to identify at which location or from which direction a specific object can be easily gripped. The color / reflectivity of the object could, for example, provide information about whether an object is clean or could be dirty / slippery. Based on this, the force with which the manipulator grasps the object could be adjusted. The same applies to the surface condition. If an object is jammed, or provides much grip, less force is required to close the gripper than a smooth / slippery object. The material may also be considered to determine the ski parameters. A soft object requires other ski parameters than a hard object. For example, a soft object could be gripped more quickly and with more force than a hard object, or vice versa.In a preferred embodiment, the object to be manipulated is already known to the robotic system and all or some ski parameters are already stored in a database. These skip parameters may be read to determine the skip parameters to interact with the object. In a preferred embodiment, the read ski parameters are equal to the interaction ski parameters.It is further preferred that the method step for programming the robotic system further takes into account the shape and / or the position relative to one or more nearby object(s) and / or the size of the drawn symbol and, based thereon, adjusts the selected skip parameters. The shape and / or position of the drawn symbol relative to one or more nearby objects could preferably be interpreted to grasp multiple objects simultaneously, provided the objects are close enough to each other. For example, whether a normal Z or a italic Z is drawn could control how fast access or how fast a skil is performed. The size of the drawn symbol could be used, for example, to define how powerful the manipulator is to access.It is furthermore preferred that the method step for programming the robotic system provides for the drawing of a trajectory with an input device, wherein the trajectory is displayed on the screen. The user can decide whether to draw a trajectory at any position in space with the input device so that it is displayed on the screen or whether the trajectory is derived from a drawn symbol so that it is displayed on the screen. The drawing of the trajectory can take place both before the drawing of the first symbol or after the drawing of the first symbol and before the drawing of the next symbol or after the drawing of the last symbol.It is furthermore preferred that the method step for executing the movement of the robotic system comprises the method step for moving the robotic manipulator to the starting point of the trajectory and traversing the trajectory by the robotic manipulator. The orientation of the robotic manipulator remains constant relative to the last joint of the robotic system or constant in a spatial direction, for example, or has been defined by the user or a skil or a skil parameter.It is furthermore preferred that the method step for programming the robotic system comprises drawing a second or further symbol on the screen with an input device at a second or further position, which is displayed on the screen. This ensures that the user can program a series of skils in sequence. The assignment of the drawn second or further symbol to a symbol stored in the database and to the corresponding skil takes place equivalently to the procedure for the first drawn symbol.It is furthermore preferred that the method step for executing the movement of the robotic system comprises traversing a trajectory of the robotic manipulator from the position of the first drawn symbol to the position of the second or further drawn symbol. It is further preferred that the skil recognized by the second drawn symbol is executed.It is furthermore preferred that the method step for programming the robotic system or for executing the movement of the robotic system has a method step in which all trajectories that are not explicitly defined by the user or the trajectory between the first drawn symbol and the second drawn symbol are automatically calculated. The automatic calculation preferably takes into account:the shortest path and / or the most efficient path. The aim here can be efficiently to minimize the energy consumption or to minimize the wear of the robotic system. The energy consumption can be reduced, for example, by moving as little mass as possible of the robot, wherein the wear can be minimized if as few parts as possible are moved with a short service life with frequent movement. Optionally, the trajectory can be calculated based on a user's preset. For example, the user could define to always come from point A to point B using a parabolic trajectory.It is furthermore preferred that the method step for programming the robotic system or for executing the movement of the robotic system has a method step in which the departure direction of the trajectory is defined. The departure direction could be dependent on:the drawing direction of the trajectory or of the sequence in which the symbols have been drawn from a fixed logical sequence of the two selected ski boots, such as first grasping object, then placing object.It is furthermore preferred that the database entry of each ski comprises an pictogram. This preferably represents the embodiment of the ski boot, optionally as a moving pictogram. It is furthermore preferred that the method step for programming the robotic system comprises a method step for replacing the drawn symbol with the pictogram. This makes it easier for the user to recognize that the correct ski has been recognized. This step is preferably carried out after the symbol drawn has been assigned to a skil. Optionally, this step may be accompanied by a message asking the user whether to replace the drawn icon with the detected skil icon. When the user does not wish to adapt his drawing further until the method has detected the correct skil.It is furthermore preferred that the method step for programming the robotic system has an eraser tool, with which drawn symbols can be partially or completely removed.Preferred embodiments of the invention are explained below with reference to figures.The following are shown: FIG. 1 is an exemplary view of the screen during an exemplary programming step FIG. 2 is an exemplary view of the screen in an optional execution of the programming stepFIG. 1 shows the screen content of a touchscreen 1 during a preferred method step for programming the robotic system. With the aid of the input device 2, the user is able to draw symbols "Z" and "A" and trajectory 7 in the 3D scene 3.According to FIG. 1, a plurality of objects 4 and 5 are displayed in the 3D scene including the environment. FIG. 1 shows a "Z" symbol on object 4. "Z" could exemplarily stand for accessing. The symbol should be designed such that it can be assigned to the ski to be executed. Letters, numbers, but also pictograms or drawings of the manipulator, such as the corresponding skil is executed, are of course suitable for this purpose. Furthermore, a trajectory 7 has been drawn which defines the trajectory along which the robotic manipulator propagates in the method step of executing the movement. Furthermore, a second symbol "A" has been drawn to the position in the 3D scene on the screen where the robotic manipulator is to make a second skil. In Fig. 1, "A" could stand for settling. In the method step of executing the movement, the program therefore executes the following method steps:capturing the object 4 at the position of the drawn "Z", moving the robotic manipulator with object 4 along the trajectory 7, placing the object 4 on the point 6.Figure 2 shows an optional embodiment according to claim 10, in which the drawn symbols "Z" and "A" are replaced by recognised skils. The symbol "Z" has been replaced in FIG. 2 by the pictogram 9 for the access skil. Likewise, the "A" symbol has been replaced by the drop-ski icon 10. The recognized skils together with the displayed trajectory 7 result in the recognized program 8. The order in which the skils are executed in the program is shown in the list and can be changed by moving the skils. For example, the top ski may be executed first, followed by the second top, and so on. At the same time, an embodiment of the ski boots from bottom to top or in an alternative representation of the list from right to left or left to right is also conceivable.

Claims

Method for programming and executing movement of a robotic system, comprising a robotic manipulator, an input device (2) with screen and a database with skils, which the robotic manipulator can perform, with the following method steps: a. wherein the programming comprises the following method steps: i. displaying the position of one or more objects (4, 5) to be manipulated, ii. drawing a symbol with an input device (2), which is drawn on the screen and reproduced at the drawn position, on or on an object (4, 5), iii. comparing the symbol with a series of symbols stored in the database iv. assigning the drawn symbol to a symbol stored in the database and to the corresponding skil, once a similarity value between stored and drawn symbol is higher than a confidence threshold, b. wherein the movement execution has the following method steps: i. moving the robotic manipulator to the position of the drawn symbol ii. executing the skil recognized by the drawn symbol.Method for programming and executing movement of a robotic system according to claim 1, characterized in that the robotic system further comprises an object detection device, a. wherein the programming comprises the following further method steps: i. detecting one or more manipulatable objects (4, 5), and / or detecting the robotic manipulator with the aid of the object detection device.Method according to claims 1-2, characterized in that the input device (2) with screen comprises a touch-sensitive screen, which can be operated either via finger touch or via "pen", or a screen with a separate interface, such as a mouse, joystick, character tablet, 3D scanner or camera, for visually detecting hand gestures.Method according to Claims 1-3, characterized in that the method recognizes the shape and / or the colour / reflectivity and / or the surface condition and / or the material of the object (4, 5) and, on the basis thereof, adapts the selected ski life parameters.Method according to claims 1-4, characterized in that the method takes into account the shape and / or the position relative to one or more nearby objects (4, 5) and / or the size of the drawn symbol and, based thereon, adjusts the selected skip parameters.Method according to Claims 1 - 5, a. wherein the programming has the following method steps: i. drawing a trajectory (7) with an input device (2) which is displayed on the screen b. wherein the movement execution has the following method steps: i. moving the robotic manipulator to the starting point of the trajectory (7) ii. traversing the trajectory (7) by the robotic manipulatorMethod according to claims 1 - 6, a. wherein the programming comprises the following method steps: i. drawing a second symbol on the screen with an input device (2), at a second position, which is reproduced on the screen, ii. comparing the second symbol with a series of symbols stored in the database, iii. assigning the drawn second symbol to a symbol stored in the database and to the corresponding skil as soon as a similarity value between stored and second drawn symbol is higher than a confidence threshold b. wherein the movement execution comprises the following method steps: i. traversing a trajectory (7) of the robotic manipulator from the first position to the second position ii. executing the ski-IIs recognized by the second drawn symbol.Method according to claim 7, a. wherein the programming comprises the following method steps: i. calculating the trajectory (7) between the first drawn symbol and the second drawn symbol based on: the shortest path and / or, the most efficient path (energy / wear), a user presetMethod according to Claim 4 or 5, characterized in that the departure direction, the trajectory (7) depends on the drawing direction of the trajectory (7), or depends on the order in which the symbols have been drawn, or depends on a fixed logical order of the two selected skils, or depends on a user input.Method according to Claims 1 - 9, characterized in that the database has at least one pictogram (9, 10) for each ski boot, a. wherein the programming has the following method steps: i. replacing the drawn symbol with the pictogram (9, 10).Method according to claims 1-10, characterized in that the similarity value is calculated by the cosine similarity of a vector representation of the symbols, or the similarity value is determined by a convolutional neural network, or the similarity value is determined by a support vector machine.Method according to Claims 1-11, characterized in that the confidence threshold is between 80-100%.Method according to Claims 1 - 12, characterized in that symbols which have been drawn can be removed either completely or partially with the aid of an eraser tool.Robot system comprising a robotic manipulator, an input device (2) with screen and a database with skils that the robotic manipulator can perform with control configured to perform a method according to any of claims 1-12.

Images