METHOD FOR CARRYING OUT AT LEAST ONE ACTIVITY PROCESS BY MEANS OF A ROBOT

DE502020009590D1Active Publication Date: 2024-11-21BATTENBERG GUNTHER
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
DE502020009590
Authority / Receiving Office
DE · DE
Patent Type
Patents
Current Assignee / Owner
Priority Date
2019-03-22
Filing Date
2020-03-18
Publication Date
2024-11-21
Estimated Expiration
2040-03-18

AI Technical Summary

Technical Problem

In automated production plants, accurately aligning a robot with an object for quality checks and activities is challenging due to potential deviations in object positioning, which can lead to incorrect execution of tasks.

Method used

A method using contactless localization via optical or ultrasound sensors, such as 2D cameras and 3D scanners, to determine the 6D position of the object based on CAD data, allowing precise alignment of the robot for activity execution, with continuous localization if the object is moving, and using a multi-sensor head mounted on the robot flange for edge or surface detection.

Benefits of technology

Ensures accurate and cost-effective alignment of robots with objects, enabling precise execution of activities like quality checks and measurements without the need for frequent recalibration, even when objects are in motion, and allows for feedback loops to adjust CAD models based on real measurements.

✦ Generated by Eureka AI based on patent content.
Patent Text Reader
Need to check novelty before this filing date? Find Prior Art

Description

[0001] The invention relates to a method for carrying out at least one activity process in the form of a measurement by means of a robot on an object in the form of a vehicle, wherein the measurement is a quality inspection.

[0002] Particularly in automated production facilities where robots perform tasks such as assembly work or quality inspections, the exact location of the object or an object area on or in which a task is to be performed must be known in robot coordinates. This does not pose any problems if the position of the object in relation to the robot is clearly defined. However, any deviations can lead to tasks being performed incorrectly.

[0003] The subject of DE 690 32 185 T2 is a method and a device for controlling the machining track of an industrial robot.

[0004] According to DE 10 2015 007 829 B4, a workpiece is held by a robot in order to carry out work based on the CAD data of the workpiece.

[0005] DE 10 2016 114 337 A1 relates to a system and a method for linking coordinate spaces.

[0006] An industrial robot according to DE 10 2007 09 851 B3 has a 2D camera that is moved in at least two different positions in order to determine the position relative to an object.

[0007] A device for calibrating an industrial robot system is known from EP 2 350 750 B1.

[0008] DE 10 2006 005 990 A1 describes a workpiece measurement for 3D position detection in several multi-robot systems.

[0009] DE 10 2010 015 031 A1 describes a method and a device for carrying out a process on a motor vehicle component with precise positioning.

[0010] EP 1 043 642 A2 describes a robot system in which coarse and fine localisation are carried out to detect randomly arranged objects.

[0011] EP 2 216 144 A2 discloses a method and system for controlling components and / or functional units.

[0012] The present invention is based on the object of developing a method of the type mentioned at the outset in such a way that a clear alignment of the robot to the object or object area can be achieved with simple measures and at low cost in order to carry out the necessary activities.

[0013] To solve the problem, a method according to claim 1 is proposed.

[0014] In particular, the localization for determining the second data is carried out contactlessly, in particular optically or by means of ultrasound, using one or more sensors, wherein a test or measuring head emanating from the robot, in particular a multi-sensor measuring head, is used for localization, wherein the at least one first sensor is a sensor from the group of optical sensors, such as a camera, in particular a 2D camera, 3D scanner, time-of-flight sensor, laser line sensor, distance sensor, such as a laser distance sensor, ultrasonic distance sensor.

[0015] According to one aspect of the invention, it is provided that from a 6D position of the object or object area determined by the first and second data, the robot carrying out the activity process or part of it is moved to an area of ​​the object determined by first data and carries out the activity process, wherein preferably when the object is moving, the object is continuously localized or if the movement vector and speed of a moving object are known after localization, further localizations are not required or are required to a small extent.

[0016] The position of an object or object area – hereinafter referred to simply as an object – is determined based on a CAD model of the object. This involves comparing the data from the CAD model, referred to as the first data, with the image or measurement point cloud of the real object, which is the second data. This localizes the position of the object relative to the robot, allowing the robot or a part of it, such as a robot arm, to be aligned to the area of ​​the object in which an activity is to be performed.

[0017] For example, an edge or surface of the object can be detected and created using the corresponding known CAD data.

[0018] For localization, a multi-sensor head is used in particular, which extends from a robot flange plate, whereby the multi-sensor head has one or more mounting locations for optical sensors such as laser distance sensors, 3D scanners, time-of-flight sensors, 2D line sensors, 2D cameras, which perform the function of localization sensors in order to detect the object and compare it with the CAD data.

[0019] To carry out the activity process, the position and orientation of the object in robot coordinates must be available at all times.

[0020] While the localization sensor preferably comes from the robot itself, it is also possible to position an optical sensor such as a 3D camera stationary in the area where the object is located. The point cloud provided by the 3D camera—as well as the localization sensor(s) provided by the robot—enables the 6D position of the object to be determined using suitable software. This can be achieved, for example, by edge detection or area detection with a 3D camera, edge detection with 2D sensors, area detection using a 3D scanner, or by image processing.

[0021] To perform the task, initial data—meaning measurement positions defined in the CAD model—are provided in robot coordinates, i.e., second coordinates, with sufficient accuracy for the measurement task. The actual data thus determined—i.e., second data—are then used to perform the task without requiring any changes to the measuring head, since this head, in addition to one or more localization sensors, also has a tool such as a suction cup or gripper, or a gap measuring tool, such as a laser line sensor. This tool, particularly with regard to the robot flange plate, is positioned centrally relative to the robot flange axis.

[0022] The measurement position defined in the CAD model is linked to the measurement results determined by at least one localization sensor using suitable software.

[0023] Localization is performed in several steps. In the first step, a rough localization of the object relative to the robot is performed by comparing the CAD model with the image or point cloud from the localization sensor(s). A model-based fit is then performed.

[0024] Fine localization is achieved by measuring one or more local features on the object, e.g., center point determination, plane determination by distance measurement, angle measurement. This allows the element performing the activity, which originates from the robot, to be aligned with high precision to the area of ​​the object in or on which the activity is to be performed.

[0025] The data from the localization sensor on the robot or mounted on the robot flange can be linked to one or more external sensors to achieve high accuracy. Alternatively, data from multiple localization sensors on the robot can be used.

[0026] According to the invention, a robot, in particular a six-axis industrial robot, is used to perform a task. Model-based localization of the robot relative to an object on which a task is to be performed is performed. The results of the task can be visualized. For example, the completed tasks can be superimposed as an image on the CAD model of the area in which the task was to be performed. A real image of the area can also be superimposed on the CAD model of the area.

[0027] Augmented Reality (AR) or Virtual Reality (VR) visualization is possible.

[0028] The teaching of the invention is implemented through the activity process of measuring. Measuring can be the quality inspection of, for example, vehicle interior components, such as haptic measurements and functional testing of climate control units, infotainment systems, vents, touch panel sets, etc. A force / torque sensor of the robot, from which the sensor head extends, can be used. The force / torque sensor is arranged, in particular, between the adapter plate and the sensor head or the robot flange.

[0029] Other measurements such as stiffness measurements on car bodies with force / torque sensors, gap measurements on car bodies with laser line sensors, door and flap testing with force / torque sensors and line sensors on vehicle exterior components are also possible.

[0030] However, it is also possible to use the data determined by the at least one first sensor emanating from the robot to control another robot, e.g. to carry out another activity process, which, for example, carries out the fastening after the first robot has inserted a vehicle light.

[0031] If the object is moving, it is continuously localized. If the motion vector and speed of the moving object are known, it is generally sufficient to omit further measurements after localization has been performed once. However, to detect possible deviations in the motion vector or speed, localization can be performed again at regular intervals.

[0032] To carry out the method, a test head can be used which is fastened to a robot or robot flange, wherein the test head used is one with a preferably cylindrical base body which is detachably connected to the robot or the robot flange, wherein an adapter plate with a tool, such as a haptic tool and / or a suction cup and / or a gripper and / or an optical sensor, extends from the base body or its front area, wherein in particular a force / torque sensor is arranged between the adapter plate and the base body, and wherein the tool is peripherally surrounded by a plurality of mounting surfaces for lighting and / or one for one or more optical sensors such as a 2D camera, 3D scanner, time-of-flight sensor, laser distance sensor, 2D line sensor.

[0033] Furthermore, one or more compressed air connections for connecting to the tool, such as suction cups and / or grippers, can extend from the base body, in particular its peripheral wall.

[0034] According to the invention, a 6D position detection of a possibly moving object is carried out in order to carry out activities on the object on the basis of CAD data of the object, wherein the position detection is carried out by linking data of the real object, which are determined by one or more sensors emitted by the robot, with CAD data of the object.

[0035] It is possible that, after the activity has been performed, second data from the area in which the activity was performed can be determined, e.g., using a laser line sensor. Taking the second data into account, the first data relating to the area in the CAD model can be influenced in such a way that, when performing activities on other objects, any previously detected deviations of the object from the CAD model are taken into account. This essentially creates a feedback loop.

[0036] Further details, advantages and features of the invention will become apparent from the following description and drawings of preferred embodiments.

[0037] They show: Fig. 1 shows a schematic diagram of an arrangement with an object on which at least one activity is to be carried out by means of a robot, Fig. 2 shows a test head in perspective view, Fig. 3 shows the test head in front view, Fig. 4 shows a process diagram for carrying out an activity and Fig. 5 shows a process diagram for representing the activity carried out.

[0038] The figures are intended to illustrate, purely in principle, a method according to which one or more activities in the form of measurements are to be carried out on a vehicle 10 (object) by means of a robot 12, such as a six-axis robot.

[0039] Is in Fig. 1 Although object 10 is depicted as a motor vehicle, this is not intended to limit the teachings of the invention. Therefore, reference is made below to both "object" and "vehicle." These terms also include areas of the object or vehicle.

[0040] According to the invention, an activity is performed with the six-axis robot or industrial robot 12 on the object, i.e., vehicle 10, taking into account the data from the CAD model of the object 10, which is transformed into the robot data as second data. To perform the activity, an area is specified in the CAD model in order to align the robot, or a tool or sensor emanating from it, to this area and then perform the desired activity. However, this requires that the unique 6D position of the object 10 is known. For this purpose, localization sensors are used, which preferably originate from the robot 12, namely from a multi-sensor test or measuring head 14 – hereinafter referred to simply as a multi-sensor test head – which is connected to a robot flange 16.The multi-sensor test head 14 has a base body 16 which is cylindrical in the exemplary embodiment, from whose front region 22, which preferably has a conical geometry, several mounting locations emanate, some of which are identified by the reference numerals 24, 26, 28, 30.

[0041] The assembly station 30 is centrally located and the robot flange axis passes through it. An adapter plate 32 is connected to the assembly station 30, with a force / torque sensor 34 arranged between it and the mounting surface. In the exemplary embodiment, a tool, such as a haptic measuring tool, suction cup, gripper, or an optical sensor, extends from the adapter plate 32 to perform the task. Localization sensors such as a laser distance sensor 38 or a 2D laser line sensor, or an optical sensor 40, such as a 2D camera, 3D scanner, time-of-flight sensor, or camera, can be arranged with one or more of the other assembly stations 24, 26, 28. A lighting source 42 emanates from one assembly station, specifically from the assembly station 26 in the exemplary embodiment. However, a localization sensor, such as sensor 38 and / or sensor 40, emanates from at least one of the assembly stations.The localization sensor is designed to localize the position of the object or an area in order to then determine the 6D position of the object 10, taking into account the CAD model of the object 10. For localization, at least one 2D method, possibly a combination of a 2D and a 3D method, is carried out, as shown in the . Fig. 4 is clarified purely in principle.

[0042] For example, a 2D camera, which is directly or indirectly attached to the robot flange 14, is specified as a localization sensor (Box 42).

[0043] A snapshot of object 10 is taken using the 2D camera (Box 44). A CAD model of object 10 is available, or the area in or near which an activity is to be performed, such as an air conditioning control unit (Box 48). Using the software, the data from the CAD model can then be linked as first data with the data determined by one or more localization sensors as second data in order to provide the 6D position of the object in robot coordinates. The area (position Px) in which an activity is to be performed is defined in the CAD coordinate system and is transformed into robot coordinates by the software after the object's position has been determined. The robot can then be aligned to position Px (Box 52).

[0044] Additionally or alternatively, a 3D camera can also be used (box 54), which is, for example, stationary in the room in which the object 10 is located. The 3D camera provides a point cloud, i.e. a snapshot of the object (box 56), in order to then determine the 6D position of the object (box 60) on the basis of the relevant data and taking into account the CAD data using software (box 58) in order to align the robot 12 or the multi-sensor head 14 to the area of ​​the object 10 in or on which an activity is to be carried out, in accordance with the above explanations.

[0045] In order to check the activities carried out or to carry out measurements independently of them, Fig. 5After approaching the area of ​​the coordinates Px (box 52) by means of the robot 12, for example, measurement data and quality characteristics of the object (box 54) are determined, e.g. force-displacement curves for determining the stiffness, e.g. on the body of a vehicle.

[0046] Using an app, for example, the data from the CAD model (Box 56) and the data (Box 58) of the real object area in or on which activities were carried out, such as measurement data and quality characteristics such as gap dimensions, can be used to enable AR (Augmented Reality) or VR (Virtual Reality) visualization, for example, in which activities were carried out using an optical sensor such as a localization sensor or webcam. This is illustrated in Boxes 62 and 64. Box 62 shows the option of visualizing the measurement data, such as quality characteristics, on the CAD model, for example on a tablet. Box 64 shows the visualization of the quality characteristics on the real object according to the CAD model, i.e. the real image is overlaid with virtual information.

[0047] According to the invention, an activity in the form of a measurement is carried out with a six-axis robot, whereby a CAD model-based localization of the six-axis robot relative to an object is carried out in order to then enable, for example, a user-adjustable visualization of the results in AR or VR.

[0048] The measured deviations between the characteristics on the real object, i.e. the actual values ​​(second data), are compared with the target data obtained from the CAD model (first data). For example, if consistent deviations occur after measuring several objects corresponding to the CAD model, the CAD data can be corrected in order to achieve optimization in subsequent activities on the same object.

[0049] Another advantage of the inventive teaching is that quality characteristics from different production facilities can be compared with each other, ensuring equally good work results regardless of the assembly location. Remote monitoring of quality characteristics is possible.

[0050] Another advantage of the teaching according to the invention is that with one test head both the localization of the object and activities can be carried out on it without the need for replacement.

[0051] The teaching according to the invention offers the possibility of combining measuring robotics with automatic position detection.

Claims

1. A method for performing at least one activity process, in the form of measurement using a robot (12) on an object in the form of a vehicle (10), wherein the measurement is a quality test comprising the method steps - providing first data in the form of data of a CAD model of the object or of an object area, or of a model of the object or the object area in a first coordinate system, - localising the object or the object area to determine second data in a second coordinate system by means of a probe (14) extending from the robot and having at least one first sensor, wherein the second coordinate system is that of the robot (12), - transforming the first data into the second coordinate system, - moving the robot or a part thereof in the second coordinate system and - performing the at least one activity process on the object or on the object area by presetting first data transformed into the second coordinate system, characterized in that localization is performed in multiple stages such that initially a rough localization and then a fine localization is performed, wherein the rough localization is performed by comparing data of the CAD model with measured values of at least one localization sensor from the group of first sensor (38, 40) and second sensor (54) which is stationarily arranged in a space surrounding the object (10), and fine calibration is performed by measuring at least one local feature of the object by means of the at least one first sensor, wherein center point determination, plane determination by distance measurement or angle measurement is performed at the object.

2. The method according to claim 1, characterized in that localization to determine the second data is performed contactlessly, in particular optically or ultrasonically, using one or more sensors, wherein a probe or measuring head (14), in particular a multisensor measuring head, extending from the robot (12) is used for localization, wherein the at least one first sensor (38, 40) is a sensor from the group of optical sensor such as a camera, in particular a 2D camera, a 3D scanner, a time-of-flight sensor, a laser line sensor or a distance sensor such as a laser distance sensor or ultrasonic distance sensor.

3. The method according to claim 1 or 2, characterized in that the robot or a part thereof performing the activity process is moved from a 6D position of the object (10) or object area determined from the first and second data to an area of the object determined by first data and performs the activity process, wherein the object (10) is continuously localized preferably with the moving object or, if the movement vector and the speed of a moving object (10) are known after localization thereof is completed, further localizations are not required, or if so only to a minor extent.

4. The method according to at least one of the preceding claims, characterized in that a quality test of vehicle interior components, such as haptic measurement, function testing of, for example, air conditioning control units, infotainment systems, air diffusers, touch panels, seats and / or vehicle interior components, stiffness measurement, gap measurement, door and flap testing, is performed as the quality test.

5. The method according to at least one of the preceding claims, characterized in that second data of the activity process performed in the area of the object (10) is compared with first data defined by the activity process to be performed, and if required the comparison is visualized and / or recorded.

6. The method according to at least one of the preceding claims, characterized in that the second data determined by the at least one first sensor is used to control a further robot (12) for performing a further activity process, preferably in the area of the object (10).

7. The method according to at least one of the preceding claims, characterized in that the second data of the area determined after performance of the at least one activity process in the area of the object (10) is used to influence first data for the activity process to be performed subsequently.

8. The method according to at least one of the preceding claims, characterized in that the measurement is performed using a force / torque sensor (34) and / or at least one first sensor, in particular a laser line sensor.

9. The method according to at least one of the preceding claims, characterized in that a first image, formed from the first data, of the area of the object in the CAD model is optically covered by a real image, formed from the second data, of the object (10) or of the area of the object after the process activity, or vice versa, wherein preferably an Augmented Reality or Virtual Reality visualization is performed.

10. The method according to at least one of the preceding claims, characterized in that the probe (14) used is one with a preferably cylindrical base body (20) which is detachably connected to the robot (12) or to a robot flange (16), wherein an adapter plate (32) with a tool (36), such as a haptic tool and / or a suction element and / or a gripper and / or an optical sensor, extends in particular from the central area of the base body or end area thereof, and a force / torque sensor (34) is arranged preferably between the adapter plate and the base body, and wherein one or more assembly areas or surfaces (24, 26, 28) for lighting and / or for an optical sensor, such as a 2D camera (40), a 3D scanner (40), a time-of-flight sensor (40), a laser distance sensor (38) or a 2D laser line sensor (38), are provided preferably from the end-side edge area of the end area of the base body, wherein compressed air connections for the suction unit and / or for the gripper extend preferably from the base body (20), in particular from its circumferential wall.