Apparatus and method for programming a robot by demonstration - Patents.com

Abstract

The present invention relates to an apparatus for programming a robot based on a novel passive pointing device including fiducial markers that allow interaction with one or more cameras through a computer-implemented method. By utilizing a camera integrated in the robot, the computer-implemented method and the pointing device enable a tracking mechanism that allows the position of the robot to be modified in real time during robot programming in order to keep the relative pose between the on-board camera and the pointing device substantially constant. In this way, a human operator can advantageously set the pose of the robot tool by demonstration, i.e. by placing the pointer in a desired position and orientation during the execution of the computer-implemented method. Thus, a larger workspace and a higher degree of precision, adjustable according to needs, are achieved compared to known systems. The apparatus may include a passive pointer and a camera attached to the wrist of the robot.

Description

[Technical field]

[0001] The present invention relates to a device and method for programming robots, in particular collaborative robots, or cobots. [Background technology]

[0002] Robots are becoming indispensable machines to perform a variety of often dangerous activities in industry and other sectors. Currently, different types of robots are available: humanoid, SCARA, parallel, humanoid, etc. They can be subdivided by their size, load capacity and speed. To show how broad the term "robot" is, the familiar autonomous household devices used for mowing the grass and vacuuming are in fact robots.

[0003] For purposes of describing the present invention, robots can be substantially divided into two broad families: traditional robots and collaborative robots, also known as "cobots."

[0004] The former operates in a controlled environment requiring strict safety measures, in particular the operator must maintain a safe distance while the work cycle is running.

[0005] Conversely, cobots will need to operate in close proximity to an active collaborating operator, and clearly measures will need to be adopted to ensure the safety of the human operator.

[0006] These are usually equipped with force sensors so the cobot can detect collisions and stop when it bumps into a person.

[0007] In the past and still today, the programming steps of collaborative robots (and traditional robots) are carried out according to a standard mode, which boils down to the need to move a tool attached to the robot, verify its position and record the pose of the tool, i.e. its position and inclination. This procedure is generally very time-consuming, since a significant number of poses are required to model even an industrial process of moderate complexity.

[0008] In conventional robots, positioning the tool in various poses according to a predefined sequence of movements is done by means of a specific device named "teach pendant". Said device is generally attached to the robot and consists of a portable console with an interface that the operator uses to move the tool in space, set each single pose of the sequence, and store or delete a specific pose. This method of commanding a robot is often called "point-to-point" programming.

[0009] The teach pendant may in some cases be equipped with a camera to facilitate remote assistance (as claimed in document WO2010130289A1 in the name of ABB Research Ltd) or to generate an augmented reality environment to facilitate robot programming (as claimed in applications US2021023694A1 and US2020384647A1 in the names of Qingdao University and Fanuc Corp, respectively).

[0010] However, for collaborative robots, a teach pendant is needed to position the cobot in predefined poses, or a special programming mode called "kinesthetic teaching" is needed, where the operator can manually drag the cobot and place it in various poses with little effort.

[0011] Kinesthetic teaching (as described, for example, in application WO2017 / 178469 in the name of Universal Robots A / S) is very time-consuming, since the operator needs to position the cobot in all poses required by the sequence of movements in the program. In addition, fine positioning, i.e. very precise tool positioning, usually requires further adjustments via the teach pendant.

[0012] Cameras and fiducial markers are widely used in robotics. A solution using both of these is described for example in application WO2021050646A1 in the name of DMG Mori Co Ltd and Skylla Tech. This document concerns a device attached to a robot that can move around a machine tool. A camera attached to the robot arm detects an "identification figure" placed on a flat ceramic surface. The purpose is to allow the use of the robot at multiple workstations and to ensure that after the robot is moved from one station to another, it is positioned in the same way as it was positioned at the first station. However, the use of fiducial markers disclosed in WO2021050646A1 is not useful for setting up devices and methods for programming robots, in particular cobots.

[0013] Cameras and fiducial markers are also widely used for the calibration of robotic arms (known examples are provided, for example, in US2020198145A1 in the name of Industrial Technology Research Institute and US7945349B2 in the name of ABB Technology AB), the purpose of which is quite different from the purpose of the present invention and would not be useful to a person skilled in the art who wishes to develop devices and methods for programming robots.

[0014] A prominent solution dealing with robot programming relevant to the present invention is disclosed in US patent US7353081B2 (or its equivalent family application US2005251290A1) in the name of ABB Research Ltd.

[0015] More specifically, US7353081B2 discloses a method for programming a robot with a pointer and a camera. The method includes the steps of acquiring an image detected by the camera of an object to be processed and acquiring information about the position of a pen-shaped pointer placed on or in the vicinity of the object. It is noteworthy that the pointer is an active device, i.e. it includes a dedicated electronic unit and associated keys for acquiring the position and orientation of the pointer. In addition, the camera is held at a suitable fixed point in the workspace or attached to the operator's head (or somewhere else on the body). However, this patent specification does not suggest fixing the camera directly to the robot to be trained, in particular to the wrist of the robot.

[0016] In particular, one embodiment of US7353081B2 refers to a method for programming a robot based on image recognition in which two markers are used (FIG. 1): a so-called "fiducial marker" that is in a fixed relationship with the object, and a "pointing marker" that is attached to the pointer. In fact, the function of the fiducial marker is to know where the pointer is relative to the reference coordinate system given by the fiducial marker. In this regard, it is worth noting that for the purposes of implementing US7353081B2, the markers must be clear and asymmetric so that they can be recognized from any angle using image recognition algorithms.

[0017] Despite being an improvement in the art with regard to tools and methods for programming robots based on cameras and pointers, it is clear that US7353081B2 has significant shortcomings.

[0018] The first drawback is that the cost is relatively high, since the pointer is an active device. Moreover, the pointer localization procedure is based on the "world reference coordinate system", so the implementation of the method and system according to US7353081B2 requires at least two markers, both of which must be within the field of view of the camera. If the camera cannot frame one of the patterns, the available workspace is reduced and the operator must stop programming the robot, move it to the appropriate position, and continue programming.

[0019] A further drawback is that the camera is not attached to the body of the robot, which essentially determines a decrease in performance in terms of positioning accuracy, as well as the constancy and optimality of the camera viewpoint.

[0020] Finally, US7353081B2 does not teach or suggest any pattern tracking mechanism to one skilled in the art.

[0021] In summary, the limitations of current techniques relate to cost, installation difficulties (e.g., regarding positioning of illuminators or other cameras), available workspace, and low accuracy of pose acquisition.

[0022] In conclusion, for the reasons mentioned above, the problem of programming robots, in particular cobots, has not yet been fully addressed and it would be desirable to have an improved solution based on the use of one or more cameras and pointer devices, in particular passive pointing devices. Summary of the Invention

[0023] Purpose / Scope of the Invention In view of the above, the present invention intends to overcome the existing shortcomings and drawbacks of the prior art by providing a new and innovative apparatus and method for programming robots, particularly collaborative robots.

[0024] Therefore, a first main objective of the present invention is to provide an apparatus for programming a robot that is easy to use and based on a pointing device, which does not use any electronic system for calculating poses or contain any electronic system for communicating with a computing unit.

[0025] A second important object of the present invention is to provide such an apparatus with a high degree of accuracy that can be adjusted according to the needs of a particular application.

[0026] A third important object of the present invention is to provide an apparatus and method for programming a robot that can take advantage of the inherent repeatability of the robot.

[0027] A fourth important object of the present invention is to provide a method for expanding the educational workspace without repositioning external sensors or cameras.

[0028] Finally, a final object of the present invention is to provide an apparatus and method for programming a collaborative robot that can be manufactured or implemented in a simple and economical manner using known technologies. Technical Solutions and Inventive Concepts

[0029] These and further other objects, as more clearly indicated in the following specification, are achieved by an apparatus for programming a robot, a robot including said apparatus, and a method for programming a robot.

[0030] The present invention is defined by the attached independent claims 1, 12 and 15, and advantageous features are set out in the attached dependent claims. The claims, to which reference should be made for brevity, are specifically defined below and are intended to be an integral part of this specification.

[0031] In summary, the first object of this patent is an apparatus for programming a robot based on a substantially passive, or completely passive, pointing device that includes a single target consisting of one or more markers arranged in a pattern, and one or more cameras that interact with said target.

[0032] Preferably, the apparatus is based on a single camera integrated with the robot and a fully passive pointing device with a pattern of multiple markers, preferably six markers. Furthermore, in this specification, the term "substantially passive" in reference to a pointing device is intended to mean a pointing device including an electronic unit. However, such an electronic unit is not used to calculate the pose of the pointer in space or to transmit signals (e.g. electromagnetic signals) useful for such purposes, but only allows communication with a processing unit where such calculations are performed.

[0033] Similarly, the term "fully passive" is intended to mean a pointing device that does not include any electronic unit or electronics for communicating with a computing unit where the pose is calculated.

[0034] For clarity, in this specification, the term "pose" is intended to mean the position and orientation of the pointing device in space, and thus the position and orientation of the robotic tool.

[0035] For clarity, in this specification, the term "workspace" is intended to mean the set of poses that the robot can reach, and the term "teaching workspace" is intended to mean the locations where the pose of the pointing device can be detected, i.e., where the robot can be instructed according to the method of the present invention.

[0036] The robot may be of another type, such as an industrial robot or a humanoid robot, but is preferably a collaborative robot, or cobot.

[0037] In a preferred embodiment, the camera is mounted on the robot's wrist or other moving part of the robot, such as an external axis or carriage. In an alternative embodiment, in addition to the on-board camera, the environment in which the robot operates includes one or more fixed cameras.

[0038] Typically, the pointing device or pointer is a pen-like device that is placed in the workspace by the operator to determine the pose of the pointer, and therefore the pose of the robot tool, at a particular step of the work cycle, as interpreted in the program for the robot.

[0039] The pose of the pointing device is estimated by computational vision algorithms based on one or more cameras and "fiducial markers." Such algorithms enable pointer tracking mechanisms and allow the robot to change its position so that the relative pose between the camera and pointing device during programming remains constant, or variably within a predefined range.

[0040] Such a tracking mechanism allows the programming device to achieve high accuracy, consistency, and optimality, as well as a larger teaching workspace.

[0041] These salient features depend on geometric parameters such as the distance between the marker and the pointer tip, the distance from the marker to the camera, the distance between the markers in the pattern, etc. Said parameters are taken into account during the design phase of the device in order to adjust the accuracy according to the requirements of a particular application. For example, limited space available for manipulation in the workspace may require a large distance between the camera and the target, which can be compensated for by a larger configuration of markers or by placing the target closer to the tip of the pointer.

[0042] The pointing device, the robot, or the processing device on which the pose estimation and tracking code runs may be equipped with interaction means (e.g., keys, buttons, gestures or a voice interaction system) that allow interaction with an operator, to obtain and store the current pose, to delete the last stored pose, and to perform other operations aimed at constructing and manipulating a sequence of poses that form a "pose path." Such a pose path constitutes a robot program or represents a starting point for generating or modifying a robot program.

[0043] In a preferred embodiment, the interaction with the operator advantageously excludes the use of electronic devices dedicated to communication with the computing unit: the interaction is based on artificial intelligence (AI) algorithms, designed firstly to interpret the operator's hand movements (or other gestures) and secondly to trigger operations on the poses.

[0044] Advantageously, in preferred embodiments, the programming method according to the present invention does not require a "teach pendant" device or off-line programming, and is therefore extremely fast and intuitive.

[0045] In an alternative embodiment, the method can also be used in conjunction with a "teach pendant" device to ensure maximum stability of the pointing device in high precision applications. Moreover, unlike known robot programming systems (notably US7353081B2), the solution provided herein requires the use of a single target integrated with the pointer, since the pose determination does not depend on the pointer's position relative to the "world reference frame", but is limited to its position relative to the camera. This feature significantly improves the accuracy of the pose calculation.

[0046] When the camera is fixed to the wrist of the robot, the target tracking mechanism makes it possible to overcome the problem of limited teaching workspace related to the need to detect multiple targets simultaneously in conventional robot programming systems. Furthermore, the tracking system avoids visual occlusion of the markers, which is a serious problem when fixed camera configurations are used, since it is very time-consuming to program. In fact, when this event occurs, the instructor is forced to move the robot or remove the occlusion and restart the teaching session.

[0047] Finally, when the device includes one or more additional cameras for framing the working area in fixed positions relative to the robot, it is also possible to alert the user to any dangerous positions or dangerous behaviors, as well as anomalies in the teaching method according to the invention.

[0048] The present invention will be more fully understood by reference to the following drawings, which are provided for illustrative purposes only and are not limiting. [Brief description of the drawings]

[0049] [Figure 1] FIG. 1 illustrates a first preferred embodiment of the present invention, in which the device for programming includes a single on-board camera. [Diagram 2] FIG. 1A shows a pointing device according to the present invention showing a target with a marker; FIG. 1B shows a pointing device with a target fixed on top to improve ergonomics; and FIG. 1C shows a pointing device with a target fixed on the bottom of the device to improve accuracy. [Diagram 3] FIG. 13 illustrates a second embodiment of the present invention in which the device for programming includes one or more fixed cameras other than a single on-board camera.

[0050] These figures illustrate and demonstrate various features and embodiments of the invention, but are not to be construed as limiting the invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0051] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Detailed descriptions of certain preferred, but not exclusive, embodiments of the apparatus and methods according to the present invention are provided below to illustrate, but not to limit, the invention. Preferred Embodiments Best Mode A device for programming cobots

[0052] A first object of the invention is a device for programming a robot, which will be described in detail with reference to a first preferred embodiment.

[0053] In a first preferred embodiment, the apparatus according to the invention comprises a single camera mounted (i.e. "eye-in-hand") on a collaborative robot (or cobot). The cobot tracks the pointing device in real time at a constant distance to maintain the best shot.

[0054] As shown in the accompanying FIG. 1, which is illustrative of the present invention and is not meant to be limiting thereof, the apparatus is generally designated by the reference numeral (1) and comprises a pointing device (10) or pointer having a body (102) shaped to be easily grasped by an operator, a target (11) integral with said pointer (10), a single camera (20) mounted on the wrist (51) of a collaborative robot (50), and a processing unit (30) associated with the device (10) on which software controlling the programming operations of the cobot is executed.

[0055] The body of the pointer (10) is equipped with buttons (12) allowing interaction with the processing unit (30) via wireless technologies such as WiFi or Bluetooth® (trademark of Bluetooth Special Interest Group Inc.). However, the events to be communicated can also be activated by a button or key located on a separate portable device, preferably on the same teach pendant, which the operator can for example hold in the other hand to ensure maximum stability of pointer positioning and orientation, especially in high-precision applications.

[0056] As shown in the accompanying Figure 2, which is illustrative of the invention and is not meant to be limiting thereof, the target (11) has a body (111) presenting a number of markers (113) of known geometric shape, suitably arranged to form a pattern. The body (111) of the target (11) is preferably a parallelepiped made of plastic, although different shapes and materials may be used, provided they provide sufficient stability and rigidity to the pointer.

[0057] The markers (113) are arranged on the target (11) according to a pattern so as to present a high contrast with the background of the surface of the body (111) to which they are fixed.

[0058] Preferably, the markers are black circles on a white background or white circles on a black background, however, various shapes and colors can be used depending on the needs.

[0059] For purposes of illustrating the invention and not meant to be limiting thereof, the markers (113) may be square or diamond shaped and may have a color that is very far from the background in color space. The markers (113) may also be reflective of infrared or ultraviolet light, as long as they contrast with the background.

[0060] In any case, each marker (113) is associated with a point in space, for example relative to the centre of each circular marker.

[0061] The target (11) of the device (1) for programming a robot according to the invention has at least one protruding element (112), i.e. a support, on which at least one marker (113') is fixed so as to be located outside the plane of the remaining markers (113).

[0062] For the purposes of the implementation of the present invention, the configuration of the geometric elements of the pattern is not constrained, and the number of elements, as well as the relative distance and the number of posts (112) can be varied. For example, the number of markers (13, 113') can be six, and the spacing between the markers (113, 113') can be a few millimeters, for example 3-4 millimeters. Furthermore, the height of the posts (112) can be chosen as required, typically 2-10 millimeters, preferably 5 millimeters.

[0063] In the device (1) for programming a robot according to the invention, a number of markers (113, 113') are integral with the pointing device (10) and fixed to the body (11) of the pointing device by means of a connecting means (114).

[0064] Preferably, the target (11) is fixed to the top (101) of the pointing device (10). This configuration is suitable when ergonomics of the handle are the primary need. However, the target (11) can also be fixed to the bottom (103) of the pointing device (10). This configuration is suitable when precision is the primary need.

[0065] To allow for increased flexibility during cobot programming, the target (11) may be slidably engaged with the body (102) of the pointing device (10). In this configuration, the operator may move the target (11) between a lower portion (103) near the tip of the body (102) and an upper portion (101) near the opposite end.

[0066] A mechanism of this type, which ensures sufficient displacement accuracy, can be manufactured at low cost in a simple manner using known techniques.

[0067] The device (1) for programming a robot according to the invention comprises a single camera (20) mounted on the wrist (51) of a collaborative robot (50).

[0068] In a first embodiment, provided herein for the purposes of illustrating the invention and not for the purposes of limitation, the robot (50) is a cobot, but it may also be of a different type, for example a humanoid robot or a SCARA robot.

[0069] In any case, the robot (50) has an arm with a wrist (51) on which is mounted a single camera (20), the camera (20) having capabilities that may be found in commercially available products, preferably a grey level camera, but may also be an RGB camera or an infrared sensitive camera depending on the characteristics of the target (11), which will depend on the particular application for which the device (1) is intended.

[0070] In this embodiment, the camera (20) is preferably mounted on the wrist (51) of the robot, but other positions integral to the robot (50) can be conveniently chosen to obtain an optimal framing of the pattern (113). For example, the camera (20) can be fixed to any carriage on which the robot is mounted, or more generally to a moving axis integral to the robot, but not part of it. Finally, the device (1) according to the invention comprises a processing unit (30) associated to the pointing device (10), on which software is executed that controls the operation of the apparatus for programming the cobot.

[0071] In a preferred embodiment, the processing unit (30) is external and resides in a computer external to the cobot (50). However, other options are possible, the processing unit (30) may be a "plug-in" module in the robot controller, the unit (30) may be integrated with the electronics of the camera (20), and combinations of these embodiments are also possible.

[0072] The computer program is executed in the processing unit (30). The software essentially comprises two modules that are important for the purposes of implementing the invention: a first module (31) that locates the pattern (113, 113') and provides an estimate of the pose of the pattern (113, 113') relative to the camera (20) and thus of the pointer (10), and a second module (32) that allows dynamic tracking of the marker pattern (113).

[0073] Finally, a processing unit (30) containing a graphic interface (33) interacts with the activation means (12) in a manner that will be described below.

[0074] The tip of the pointer (10) is first placed in a desired position and orientation by the operator, and then by executing a first software module (31), the marker pattern (113, 113') and therefore the pose (i.e., position and orientation) of the tip can be accurately detected.

[0075] By operating a button (12) on the handle of the pointer body (10), the operator communicates a selected option to the processing unit (30). For purposes of illustration and not limitation of the invention, said options include storing the current pose, storing a series of poses in continuous mode, the last pose, or deleting a stored series of poses. In continuous acquisition mode, the processing unit (30) in conjunction with software (31) detects poses at predetermined time intervals (which can be set by the user) as the operator moves the pointer (10) along a path.

[0076] In this way, multiple poses can be recorded while the pointing device 10 is moving, and the trajectory, i.e. the individual positions and the speed between one position and another, can be recorded. This mode is particularly useful when it is necessary to reproduce the movements performed by a human operator, for example in a work process such as painting.

[0077] Obviously, the pose path can be obtained by attaching the pointing device (10) to an arm or robot that is moved by an actuator. This mode is useful in applications that take place in dangerous environments where an operator should not be present. Moreover, the pose path instructed by a "teacher" robot can be easily and automatically copied to a "learner" robot that is also equipped with the pointing device (10).

[0078] The coordinates of the points stored up to a given moment of the programming step can be displayed on the graphic interface (33) of the processing unit (30), which also allows the adjustment of the functional parameters required by the software modules (31, 32).

[0079] Advantageously, the graphical interface (33) can superimpose in real time onto the images acquired by the camera (20) points that have already been stored and recalculated based on the current position of the robot, for example so that they are displayed integrally with the object to be processed.

[0080] As shown in the attached FIG. 1, which illustrates the invention but does not imply its limitation, a camera (20) mounted on the wrist (51) of a cobot (50) can communicate with a processing unit (30) via wires or wirelessly, depending on the amount of data required and the disturbances of the industrial environment.

[0081] The camera (20) frames the marker pattern (113, 113') and, thanks to the second software module (32), tracks the pointer (10), i.e. moves the robot (50) so that the pattern is always in frame while the pointer (10) is maintained at the desired distance and angle.

[0082] This allows the pointer (10) to be tracked while the operator is using it, keeping it at an ideal distance for localization in order to reduce the overall errors in positioning and orientation. All this will be explained in more detail below, but it will be pointed out that there are further preferred angles for framing, for example by considering the camera optics with an ideal working distance. Finally, the device (1) according to the invention can use a first calibration tool for the standard camera and a second calibration tool for the pointer (10) that implements some functions similar to known calibration tools. In particular, such a tool automatically estimates some characteristic distances of the pointer (10) based on a series of acquired images, such as the position of the tip (101, 103) relative to the reference system of the pattern (113, 113') or the position of the tool (52) of the robot (50) relative to the reference system of the camera (20).

[0083] From the description provided, it will be clear to the skilled person how the device (1) for programming a robot according to the invention utilizes very low-cost components. In addition, the software modules (31, 32) can be executed on a processing unit already present in the system, further reducing the costs. Thanks to the software modules (31, 32) and the camera mounted on the wrist of the cobot, the device (1) achieves a much higher accuracy than other solutions already present on the market.

[0084] In a prototype of a cobot according to the invention, with a standard setup, the inventors achieved an accuracy of about 0.4 mm. More precisely, tests carried out on such a prototype made it possible to quantify in 0.4 mm the maximum distance between the programmed position of the pointer tip set in the programming step and the actual position reached by the tool when the robot was positioned at the programmed position after retrieval from memory.

[0085] However, as will be explained in more detail below, it is possible to achieve even higher accuracy while still complying with the inherent precision (repeatability) of the robot (50), which represents a practical limit for further improvement. In fact, the accuracy depends on several parameters, such as the calibration of the camera (20), the framing distance, the size and the position of the patterns (113, 113') relative to the tips (101, 103). All parameters can be adjusted according to the requirements of a particular application. How to Program a Cobot

[0086] A second object of the invention is a method for programming a robot, which will be described in detail with reference to a first preferred embodiment.

[0087] From the above description, the above-mentioned device (1) allows an operator to carry out the following steps: a) integrating a mobile camera (20) with a robot (50), preferably by fixing it to the wrist (51) of said robot (50); b) placing a pointing device (10) having a marker pattern (113, 113') in the workspace; c) framing a marker pattern (113, 113') on the pointing device (10) with the mobile camera (20); d) executing software in a processing unit (30) associated with the pointing device (10), the software including instructions for causing a movement of the robot (50) such as to track a pattern (113, 113') while an operator or other means holds the pointing device (10), i.e., such that the pattern (113, 113') is framed by the mobile camera (20) at a predetermined distance and inclination while an operator or other means holds the pointing device (10), the predetermined distance and inclination being selected to reduce an overall error in positioning and orientation; e) optionally varying the positioning of the patterns (113, 113') relative to the tip of the pointing device (10) to reduce the overall error in the positioning and orientation of the pointing device (10); f) acting on the actuation means (12) of said pointing device (10) to detect the coordinates (x, y, z) and the orientation (α, β, γ) of the pattern (113, 113') relative to said camera (20, 60) and to calculate a transformation T obj calculating a pose ((x0, y0, z0), (α0, β0, γ0)) of the pointing device (10) using g) calculating a pose ((x3, y3, z3), (α3, β3, γ3)) of the pointing device (10) relative to the robot base (50) based on the relative pose ((x1, y1, z1), (α1, β1, γ1)) of the camera (20) relative to the tool (52) of the robot (50) and the pose ((x2, y2, z2), (α2, β2, γ2)) of the tool (52) relative to the base of the robot (50), and performing a transformation T cam / tool and h) storing the pose ((x3, y3, z3), (α3, β3, γ3)) calculated in step g) and making it available for use by an operator when necessary; It will be apparent that the position and orientation in space of the tool of the robot (50) can be set using a method including:

[0088] Advantageously, using this method it is possible to track the pointer 10 while the operator is using it, and at the same time detect the coordinates (x,y,z) and orientation (α,β,γ) by keeping the pointer 10 at an optimal distance, i.e. a distance where the overall error in pose is reduced, for the following reasons:

[0089] It is well known that industrial robots generally have a very high repeatability, which can be defined as the accuracy of positioning the robot tool in the same pose. Typical repeatability values ​​for industrial robots are less than 0.1 millimeters. Conversely, the accuracy, which can be defined as the ability to place the robot's tool in a desired point in the workspace, is usually much worse and depends on the point in the workspace considered.

[0090] For known robot programming systems based on fixed cameras, a priori knowledge of the following rigid body transformations (or experimental estimation) is required: T ob : Transforming a camera to a predefined pose T cam : World RS to Camera Conversion T rob : World RS to RS robot conversion (usually placed at the base of the robot) T tool : Robot RS to Robot Tool Conversion T cam / tool : Transformation from camera to robot tool.

[0091] Such transformations are necessary to position the robot tool relative to objects detected by the camera, or to selected points in the workspace (RS stands for "reference system" for short). Each of these transformations is subject to uncertainties.

[0092] In detail, the steps performed in a fixed camera based robot programming system are as follows: 1. Conversion T obj , T cam to localize the object with respect to the world RS. 2. Conversion T rob , T tool to place the robot at the previously found point.

[0093] In particular, step 2 depends on the accuracy of the robot (i.e., its ability to accurately locate a point in space specified with respect to the world RS) and is therefore subject to significant error.

[0094] On the other hand, by using the on-board camera and tracking mechanism according to the present invention, only two transformations are involved: obj (same as for fixed cameras) and T cam / tool which represents the pose of the camera relative to the tool.

[0095] Therefore, to calculate a pose in a robot programming system based on an on-board camera, the following steps need to be performed: 1. Conversion T obj , T cam , and T tool is applied to locate the object with respect to the tool. 2. Perform a relative movement of the tool from its current pose (which can be repeated very accurately) to the new pose calculated in step 1.

[0096] It is worth pointing out that this motion is relative and does not involve transformations with respect to the world reference frame. Advantageously, the uncertainties that affect such transformations do not affect it. Furthermore, the accuracy of the robot is limited since it only involves small movements of relative motion.

[0097] In other words, the accuracy errors that would occur if the robot were instructed to place the tool in exact coordinates in 3D space (called an external reference system) are avoided. Thus, because tracking at the optimal distance involves local transformations, if the work path is executed by the same robot used during the teaching operation, the errors can be reduced almost to the iteration error, i.e. to the intrinsic accuracy of the robot.

[0098] Thus, the apparatus and method according to the invention utilises an on-board camera and pointer tracking mechanism fixed to the wrist of the robot, making it possible to utilise the inherent repeatability of the robot, in contrast to known solutions based on fixed cameras.

[0099] The tracking actions performed by the robot arm to track the pattern can be designed according to known techniques, such as for example "Position-Based Visual Servoing" (PBVS). In summary, with PBVS the processing unit (30) performs the following operations: the current pose of the camera relative to the pointer is calculated for the current image, then the calculated pose is compared with the intended pose (i.e. the preferred pose that ensures framing the pattern on the pointer from an optimal distance and angle), and finally a rotation-translation transformation is calculated to be applied to the camera to obtain the intended pose.

[0100] This rotation-translation transformation is used as an assigned fiducial to the robot's control system, which tracks the assigned fiducial (fiducial tracking) according to the well-known paradigm of feedback control by using knowledge of the relative fixed pose between the robot's arm and the camera, calculated via known hand-eye calibration algorithms.

[0101] In the design step, the optimal distance is calculated based on the camera optics to minimize the overall positioning and orientation error.

[0102] The angles are calculated within a range that avoids both frames that are orthogonal to the pattern plane and frames with angles that are too large: obviously, orthogonal frames make the pose calculation very sensitive to errors in locating the pattern points, while angled frames make the pattern points difficult to identify.

[0103] Within reasonable limits selected according to the aforementioned criteria, the optimum angle is established by non-trivial systematic laboratory testing and error analysis. Other Preferred Embodiments

[0104] Further features and advantages of the present invention will become apparent from the description of three preferred, but non-exclusive, embodiments. Example 2: Single on-board camera and fixed camera

[0105] Referring to FIG. 3, attached hereto for purposes of illustration of the present invention and not meant to be limiting thereof, an apparatus (1) according to a second preferred embodiment includes one or more fixed cameras (61) in addition to a camera (20) mounted on the wrist of a cobot (50).

[0106] In said embodiment, the fixed cameras (61) include at least one fixed surveillance camera (611) for operator safety and optionally a fixed tracking camera (612). Preferably, there are two and one fixed cameras (611, 612) respectively.

[0107] Two fixed surveillance cameras (611) are placed at suitable positions to frame the entire workspace, with the purpose of alerting the operator of any abnormal behavior and avoiding collisions between the cobot (50) and the operator, or between the cobot (50) and objects in the workspace.

[0108] A single fixed tracking camera (612) assists the pattern tracking software (32) when, for any reason, the on-board camera's (20) frame is no longer locked-in to the marker pattern (113, 113') of the target (11).

[0109] The advantage of this configuration is that it increases the safety and flexibility of the application in the tracking step. Example 3: A single on-board camera on a carriage-mounted robot

[0110] The apparatus (1) according to the third embodiment of the present invention is described for the purpose of illustration of the present invention and is not meant to be limiting thereof, and may have a similar configuration to the previous embodiments 1 or 2. However, the cobot or robot (50) is mounted on a carriage (also called the "external axis") that can move along a predefined trajectory or path within the workspace.

[0111] In this way, the pointing device (10) is tracked in real time by a camera (20) integral to the cobot, preferably mounted on the cobot's wrist, such that the cobot (50) or camera (2) is moved a distance (calculated in real time by the processing unit) to maintain an optimal shot while the cobot (50) moves along the carriage path.

[0112] The main advantage of this configuration of the device (1) is the increased working space. Embodiment 4: Gesture-Based Input

[0113] A fourth embodiment of the apparatus (1) according to the present invention is described for purposes of illustration of the invention and is not meant to be limiting thereof, and includes an activation means that excludes a button (12) or key within the body (102) of the pointing device (10).

[0114] The activation means according to this embodiment may be used in any of the devices previously described.

[0115] In this embodiment, a "clickless" mode, i.e. a "gesture-based input" or "virtual click", allows the pointer (10) to interact with the processing unit (30) through the same camera (20). This mode is based on a third software module based on an artificial intelligence algorithm of the type known in the state of the art. The algorithm interprets a gesture or a movement of the hand (or another anatomical part of the operator), for example the movement of one of the fingers of the operator holding the pointer (10). The gesture or movement can thus be associated with an action, which is illustrative and not meant to be limiting, including one or more of the following: storing the current pose, storing a series of poses in a continuous mode, deleting the last pose, or storing one or more of the series of stored poses.

[0116] As is evident from the above, by using "gesture-based input" as the activation means (12), the pen becomes a completely passive item with negligible manufacturing costs.

[0117] To facilitate visual and passive detection of gestures or movements, the operator can advantageously utilize aids such as, for example, gloves, rings or bracelets worn on the hand, which have "target" elements easily recognizable by the camera associated with the third software module. Obviously, in this way it is also possible to increase the number of achievable gestures or signs, thus offering greater freedom in demonstration programming (i.e. through demonstrations) through this fully passive mode. Embodiment 5: Processing on the camera

[0118] Finally, a fifth embodiment of the device (1) according to the invention, described for the purposes of illustration of the invention and without implying its limitation, refers to a computing unit (30) integrated with a camera (20) mounted on the wrist (51) of a robot (50). For example, a smart camera (30, 20) can be used, i.e. a camera equipped with an operating system and a computing processor to carry out the necessary processing.

[0119] The computing unit (30) according to this embodiment may be used in combination with any of the devices previously described.

[0120] The advantage for the user is the overall simplicity of the device (1) in this configuration.

[0121] It comprises only two components, namely the pointing device (10) and the camera (20). The world's most important cobot manufacturers provide access to software development kits (SDKs) for the teach pendant, so that the interface (33) for adjusting the pointer parameters (10) can be advantageously transferred to the teach pendant attached to the robot (50). In this case, the "teach pendant" is used exclusively for the adjustment of some parameters and not for programming the robot (50), the programming being carried out exclusively via the pointing device (10) according to the disclosure of the present invention. Advantages and Industrial Applications

[0122] Many advantages will be apparent to those skilled in the art from the description of an apparatus and method for programming a robot.

[0123] Clearly, the apparatus and method according to the present invention simplifies and reduces the robot programming task.

[0124] A second notable advantage of the present invention is the reduced cost, since the device contains very low-cost components and the two software modules can run on a low-cost processing unit that may already be part of the robot to be trained or be an integrated component of the on-board camera, thus facilitating the widespread adoption of cobots even in very small businesses.

[0125] A further advantage is the wide range of industrial and non-industrial sectors in which the apparatus and method for programming a robot according to the present invention can be used, such as surface finishing, welding, textiles, to name just a few.

[0126] However, the main advantage comes from the combination of mounting a camera on the robot's wrist and using pure vision algorithms, which enables the pointer tracking mechanism based on the "fiducial markers" mentioned above.

[0127] Furthermore, the tracking mechanism provides three additional advantages.

[0128] First, it can maintain the optimal frame of the pointer in terms of focus and field of view, the latter being a key parameter for avoiding ambiguity in reconstructing the target pose.

[0129] The second, and more important, advantage of the pointer tracking mechanism is that the positioning error of the cobot in a memorized pose depends primarily on the precision of the cobot, and not on its accuracy as it would if the pose were achieved by physically moving a work tool over a 3D point in space.

[0130] A third advantage comes from tracking, which inherently guarantees avoidance of visual occlusion.

[0131] In summary, by combining a camera mounted on the robot's wrist with the use of pure vision algorithms, it is possible to fully exploit the intrinsic repeatability of the cobot and significantly improve the precision of the device compared to systems already on the market. It is also possible to design the pointing device depending on the required precision by using its configurable shape. Indeed, as mentioned before, since the tracking mechanism involves local transformations, it is possible to limit the error almost to the iteration error (i.e. the intrinsic precision of the robot). In this way, the precision errors that would arise if the robot were instead instructed to place the tool in precise coordinates in 3D space, called an external reference system, are avoided.

[0132] Furthermore, the accuracy of the device is uniform in space and does not depend on fixed sensors or distances between emitters, which can vary.

[0133] Finally, unlike known solutions, the accuracy is not affected by noise and other disturbance sources that usually affect industrial environments: in fact, the calculation of the pose does not involve emitters or receivers, for example of electromagnetic or ultrasonic signals, and furthermore, the device and method according to the invention are not even affected by the known drift problems of inertial systems.

[0134] In conclusion, from the description provided, the advantages of the present invention in terms of positioning accuracy over known systems will be apparent to one skilled in the art. conclusion

[0135] It has been found that the invention described herein fully accomplishes its intended objects and aims.

[0136] In particular, an apparatus and method for programming a robot, preferably a cobot, is disclosed that is easy to install and implement, highly accurate, and low cost. It will be apparent to those skilled in the art that said apparatus and method represent a significant improvement in robotics technology that is the result of non-obvious inventive efforts. In conclusion, it is understood that the present invention is not limited to the exemplary embodiments shown and described herein, and that while the description and examples provided contain many details, these should not be construed as limiting the scope of the present invention, but merely as illustrations of some embodiments of the present invention.

[0137] Accordingly, any modification of the present invention that comes within the scope of the following claims is considered to be part of the present invention.

[0138] When reference signs are applied after features and techniques mentioned in any claim, these are applied solely for the purpose of enhancing the comprehension of the claims, and as a result, these reference signs do not have a limiting effect on the interpretation of the respective elements identified by these reference signs as examples.

Claims

1. A device (1) for programming a robot (50), comprising: a pointing device (10) positionable at a point in a workspace in which the robot (50) operates; A target (11) integrated with the pointing device (10), a target body (111) having protruding elements (112); and a plurality of markers (113, 113') arranged according to a pattern on the surfaces of said body (111) and said projecting elements (112); a target (11) comprising: one or more mobile video cameras (20) and optionally one or more fixed video cameras (60) integrated with said robot (50); at least one processing unit (30) associated with said one or more cameras (20, 60) or said pointing device (10), said processing unit (30) storing executable software; an activation means (12) which, when activated by an operator, enables the camera (20, 60) in conjunction with the software to detect in real time either a single position and orientation in the workspace of the target (11) to be programmed, or multiple positions and orientations constituting a trajectory to be programmed; 10. A device characterized by the fact that it comprises:

2. 2. The apparatus of claim 1, wherein the pointing device (10) has a body (102) designed to facilitate gripping by the operator, and the target (11) is fixed to a lower portion (103) or an upper portion (101) of the body (102) of the pointing device (10), or the target (11) is slidably constrained to the body (102) to allow an operator to move the target (11) to a position between the lower portion (103) and the upper portion (101) on the body (102).

3. The apparatus described in claim 1, wherein the activation means (12) is a button that allows detection of a single position or multiple positions, and the pointing device (10) is substantially passive, i.e., includes an electronic unit configured to only allow communication of commands given by the operator to the processing unit (30) via the activation means (12).

4. The apparatus described in claim 2, wherein the activation means (12) is a button that allows detection of a single position or multiple positions, and the pointing device (10) is substantially passive, i.e., includes an electronic unit configured to only allow communication of commands given by the operator to the processing unit (30) via the activation means (12).

5. The device described in claim 3, wherein the button (12) is located on the main body (102) or on a separate portable device and is preferably adapted to be activated by the hand not holding the pointing device (10) to ensure maximum stability of the pointing device (10) in high-precision applications.

6. The device described in claim 4, wherein the button (12) is located on the main body (102) or on a separate portable device and is preferably adapted to be activated by the hand not holding the pointing device (10) to ensure maximum stability of the pointing device (10) in high-precision applications.

7. The apparatus described in claim 5, wherein the separate portable device is a robot teach pendant.

8. The apparatus described in claim 6, wherein the separate portable device is a robot teach pendant.

9. An apparatus as described in claim 1 or 2, wherein the activation means (12) is a third software module for implementing a gesture-based input mode suitable for interpreting signs or gestures of an operator holding the pointing device (10), the pointing device (10) being completely passive, and optionally the activation means (12) being assisted by a glove, ring, or bracelet.

10. The device described in claim 1, wherein the mobile video camera (20) is fixed to the wrist or movement axis of the robot (50).

11. The device described in claim 2, wherein the mobile video camera (20) is fixed to the wrist or movement axis of the robot (50).

12. The device described in claim 3, wherein the mobile video camera (20) is fixed to the wrist or movement axis of the robot (50).

13. The device described in claim 4, wherein the mobile video camera (20) is fixed to the wrist or movement axis of the robot (50).

14. The apparatus described in claim 5, wherein the mobile video camera (20) is fixed to the wrist or movement axis of the robot (50).

15. The device described in claim 6, wherein the mobile video camera (20) is fixed to the wrist or movement axis of the robot (50).

16. The apparatus described in claim 7, wherein the mobile video camera (20) is fixed to the wrist or movement axis of the robot (50).

17. The apparatus described in claim 8, wherein the mobile video camera (20) is fixed to the wrist or movement axis of the robot (50).

18. The device described in Claim 9, wherein the mobile video camera (20) is fixed to the wrist or movement axis of the robot (50).

19. The device described in claim 1, wherein the one or more fixed cameras (60) include a main fixed camera (611) for safety and / or an additional fixed camera (612) for target location identification.

20. The device described in claim 2, wherein the one or more fixed cameras (60) include a main fixed camera (611) for safety and / or an additional fixed camera (612) for target location identification.

21. The device described in claim 3, wherein the one or more fixed cameras (60) include a main fixed camera (611) for safety and / or an additional fixed camera (612) for target location identification.

22. The device described in claim 4, wherein the one or more fixed cameras (60) include a main fixed camera (611) for safety and / or an additional fixed camera (612) for target location identification.

23. The device described in claim 5, wherein the one or more fixed cameras (60) include a main fixed camera (611) for safety and / or an additional fixed camera (612) for target location identification.

24. The apparatus described in claim 6, wherein the one or more fixed cameras (60) include a main fixed camera (611) for safety and / or an additional fixed camera (612) for target location identification.

25. The apparatus described in claim 7, wherein the one or more fixed cameras (60) include a main fixed camera (611) for safety and / or an additional fixed camera (612) for target location identification.

26. The apparatus described in claim 8, wherein the one or more fixed cameras (60) include a main fixed camera (611) for safety and / or an additional fixed camera (612) for target location identification.

27. ​​The device described in claim 9, wherein the one or more fixed cameras (60) include a main fixed camera (611) for safety and / or an additional fixed camera (612) for target location identification.

28. The apparatus described in claim 1, wherein the apparatus comprises a single mobile camera fixed to the wrist of the robot (50).

29. The apparatus described in claim 2, wherein the apparatus comprises a single mobile camera fixed to the wrist of the robot (50).

30. The apparatus described in claim 3, wherein the apparatus comprises a single mobile camera fixed to the wrist of the robot (50).

31. The apparatus described in claim 4, wherein the apparatus comprises a single mobile camera fixed to the wrist of the robot (50).

32. The apparatus described in claim 5, wherein the apparatus comprises a single mobile camera fixed to the wrist of the robot (50).

33. The apparatus described in claim 6, wherein the apparatus comprises a single mobile camera fixed to the wrist of the robot (50).

34. The apparatus described in claim 7, wherein the apparatus comprises a single mobile camera fixed to the wrist of the robot (50).

35. The apparatus described in claim 8, wherein the apparatus comprises a single mobile camera fixed to the wrist of the robot (50).

36. The apparatus described in Claim 9, wherein the apparatus comprises a single mobile camera fixed to the wrist of the robot (50).

37. The device described in claim 1, comprising a mobile wrist camera (20) and a single fixed camera (60).

38. The device described in claim 2, comprising a mobile wrist camera (20) and a single fixed camera (60).

39. The device described in claim 3, comprising a mobile wrist camera (20) and a single fixed camera (60).

40. The device described in claim 4, comprising a mobile wrist camera (20) and a single fixed camera (60).

41. The device described in claim 5, comprising a mobile wrist camera (20) and a single fixed camera (60).

42. The device described in claim 6, comprising a mobile wrist camera (20) and a single fixed camera (60).

43. The device described in claim 7, comprising a mobile wrist camera (20) and a single fixed camera (60).

44. The device described in claim 8, comprising a mobile wrist camera (20) and a single fixed camera (60).

45. The device described in claim 9, comprising a mobile wrist camera (20) and a single fixed camera (60).

46. The device described in claim 1, wherein the computing unit (30) is integrated with the mobile video camera (20).

47. The device described in claim 2, wherein the computing unit (30) is integrated with the mobile video camera (20).

48. The device described in claim 3, wherein the computing unit (30) is integrated with the mobile video camera (20).

49. The device described in claim 4, wherein the computing unit (30) is integrated with the mobile video camera (20).

50. The device described in claim 5, wherein the computing unit (30) is integrated with the mobile video camera (20).

51. The device described in claim 6, wherein the computing unit (30) is integrated with the mobile video camera (20).

52. The device described in claim 7, wherein the computing unit (30) is integrated with the mobile video camera (20).

53. The device described in claim 8, wherein the computing unit (30) is integrated with the mobile video camera (20).

54. The device described in claim 9, wherein the computing unit (30) is integrated with the mobile video camera (20).

55. A robot, a pointing device (10) having a body (102) designed to be easily grasped by an operator; A target (11) integrated with the pointing device (10), a target body (111) having protruding elements (112); and a plurality of markers (113, 113') arranged according to a pattern on the surfaces of said body (111) and said projecting elements (112); a target (11) comprising: a single mobile camera (20) integrated with the robot, preferably attached to the wrist (51) of the robot; a processing unit (30) associated with said pointing device (10) or said single mobile camera (20), said processing unit (30) storing executable software; an activation means (12) which, when activated by an operator, enables the camera (20) in conjunction with the software to detect in real time either a single position and orientation in the workspace of the target (11) to be programmed, or multiple positions and orientations constituting a trajectory to be programmed; A robot equipped with:

56. 56. The robot of claim 55, wherein the robot is mounted on a carriage such that the robot can move along a track within the workspace.

57. 55. A method for programming the position and orientation of a tool (52) of a robot (50) in a workspace using an apparatus according to any one of claims 1 to 54, said method comprising: a) integrating a mobile camera (20) with said robot (50), preferably by fixing it to the wrist (51) of said robot (50); b) placing a pointing device (10) having a marker pattern (113, 113') in said workspace; c) framing the marker pattern (113, 113') on the pointing device (10) using the mobile camera (20); d) executing software in a processing unit (30) associated with the pointing device (10), the software including instructions for causing the robot (50) to move, such as to track the pattern (113, 113') while the operator or other means holds the pointing device (10), i.e., to frame the pattern (113, 113') with the mobile camera (20) at a predetermined distance and tilt while the operator or other means holds the pointing device (10), the predetermined distance and tilt being selected to reduce overall errors in positioning and orientation; e) optionally varying the positioning of said patterns (113, 113') relative to the tip of said pointing device (10) to reduce the overall error in the positioning and orientation of said pointing device (10); f) detecting the coordinates (x, y, z) and the orientation (α, β, γ) of the pattern (113, 113′) relative to the camera (20, 60) by acting on the activation means (12) of the pointing device (10), and calculating the transformation T ｏｂｊ to change the pose (x ０ , y ０ , z ０ ), (α ０ , β ０ , γ ０ ) and g) the relative pose (x) of the camera (20) with respect to the tool (52) of the robot (50) １ , y １ , z １ ), (α １ , β １ , γ １ )) and the pose of the tool (52) relative to the base of the robot (50) ((x ２ , y ２ , z ２ ), (α ２ , β ２ , γ ２ )) and the pose ((x ３ , y ３ , z ３ ), (α ３ , β ３ , γ ３ ) and to obtain the pose to be detected, the transformation T ｃａｍ／ｔｏｏｌ and h) The pose ((x ３ , y ３ , z ３ ), (α ３ , β ３ , γ ３ )) and store it in the storage device so that the operator can call it up and use it as needed; A method comprising:

58. A method for programming the spatial position and orientation of a tool of a robot (50) as described in claim 55, wherein the means for holding the pointing device (10) is an arm moved by an actuator or a second robot.

Images