Method and device for process control of work task on object by means of tool

Abstract

The invention relates to a method and a device for the process control of a work task on at least one object (2) by means of at least one tool (4), wherein the work task is to be carried out in a 3D work area (10) defined on the object (2). By means of a position recognition system (5), the position and orientation of the tool (4) in an advanced coordinate system of the position recognition system (5) and the position and orientation of the object (3) are optically determined over time. Depending on the determined position and orientation of the object (2), a defined 3D working area (10) on the object (2) is obtained in each case. By comparison, when the tool (4) is located in one of the 3D work areas (10) on the object (2) is confirmed over time, and then the execution of the work task for the object (2) is released, parameterized and / or detected in the 3D work areas (10). In particular, a LIDAR sensor or a TOF sensor is provided as a sensor (9).

Description

Technical Field

[0001] The invention relates to a method and a device for process control of a work task on at least one object by means of at least one tool. The object can preferably be a workpiece, such as a motor vehicle or another product, which is in particular an object produced in a production line in a plurality of work steps in the processing of the work task. The terms "workpiece" and "object" are used synonymously in this document. The work task is to be performed in a 3D workspace defined on the object.

[0002] According to the invention, by means of a position recognition system, the position and orientation of the tool in the higher-level coordinate system of the position recognition system is determined optically over time, i.e. the position (position and orientation of the tool) is recognized in six spatial degrees of freedom. To this end, at least one marker of the position recognition system attached to the tool is detected by means of an image by means of a camera. Background Art

[0003] An apparatus and method for identifying the orientation and position of a marker in three-dimensional space comprises a marking device (marker) having a marking unit, which for example comprises an illumination device (e.g. an infrared diode) arranged along a line, which has at least one optical image detection unit (camera) configured to record an image of the marking device, and an evaluation unit (as part of a position recognition system) configured to clearly determine the orientation (orientation) and position of the marking device based on exactly one image of the optical image detection unit. Such an example is described in WO2022 / 012899A2. In this example, each marking unit has at least three illumination devices, which are configured as markings and / or communication elements. At least one of the marking units belongs to a first type of marking unit having at least three markings, and at least one other of the marking units belongs to a second type of marking unit having exactly two markings and at least one communication element, the communication element being arranged between two markings (21). At least one of the first type of marking units and at least one of the second type of marking units are not arranged coplanarly. This arrangement is particularly suitable for clearly determining the orientation and position of the marker, wherein the markings of the first type of marking unit and the second type of marking unit are identified and placed relative to each other. According to the invention, the method can also be used to identify the position and orientation of markers.

[0004] A similar system is also known from WO2021 / 259523A1. According to the present invention, the system is also particularly suitable for identifying the position and orientation of a marker. It includes at least one marking unit (marker) having a plurality of marks, an optical image detection unit configured to record an image of the marking unit, and an evaluation unit configured to clearly determine the orientation (orientation) and position of the marking unit. The marking unit has at least five marks for orientation and position determination and at least one communication element for encoding the marking unit. The evaluation unit determines the orientation and position of the marking unit according to the mark in the recorded image and determines the identity of the marking unit according to at least one communication element.

[0005] These and other possibilities for determining the position and orientation of a marker are known to the person skilled in the art. All these known possibilities can in principle be used in the present invention.

[0006] According to the invention, the position recognition system is intended to optically determine the position and orientation of an object (i.e. its position in six spatial degrees of freedom) over time. This can be done in the superordinate coordinate system of the position recognition system or in the native coordinate system of a sensor used for this purpose, which is located in the superordinate coordinate system of the position recognition system.

[0007] Determining the position and orientation of tools and objects over time means that the position identification system records the movements of the tools and objects and always knows the position and orientation of the objects and the position and orientation of the tools.

[0008] Based on the determined (in the sense of acquisition) position and orientation of the object, a 3D workspace defined on the object in a high-level coordinate system is acquired in each case (i.e. each time the position and orientation of the object is determined over time). This can be done in the position recognition system or in the downstream process control system, where both systems can also be combined into one physical unit. By comparison over time, it is determined when the tool is located in one of the 3D workspaces on the object, where in this case the working position of the tool is signaled. This is particularly true for specific tools that are to perform work tasks in a 3D working position, in particular when multiple work tasks controlled by the process control have to be performed by different tools.

[0009] When the working position of the tool is signaled or indicated, the execution of the working task of the object is released, parameterized and / or checked in the 3D working area by means of the process control system. This process is also called process control.

[0010] It is crucial here that the system is able to recognize where the object (workpiece) is currently located and where the tool is located relative to the object.

[0011] For this purpose, a system for markerless identification of the six degrees of freedom of a workpiece is known from the prior art DE202018105197U1, wherein the workpiece moves along a defined, fixedly preset motion path. The ranging laser is positioned and guided in the system so that it irradiates a workpiece point in each position of the workpiece along a fixed motion path, and measures the distance between the workpiece and the ranging laser by means of a sensor device. After the position and orientation (orientation) of the workpiece in space are detected once by the marker system, as described exemplarily above, and after associating the detected position and orientation of the workpiece with the corresponding distance on the fixed preset motion trajectory obtained by the ranging laser, the position and orientation of the workpiece can be specified according to the measured distance by simply measuring the distance with the ranging laser. This method works well if the workpiece moves along a particularly straight but at least precisely defined motion trajectory and the orientation of the workpiece relative to the carrier of the workpiece for the purpose of movement is reproducibly and precisely defined.

[0012] However, different workpieces cannot be automatically identified. Automatic identification is also difficult when new workpieces enter the motion trajectory detected by the ranging laser. If the laser point on the object leaves the detection area during the motion trajectory, for example by shielding when a subsequent workpiece enters the projection space of the laser, a reflector must usually be attached to the moving workpiece. The ranging laser is essentially limited to one object in the detection area. Errors can also occur if the laser beam is blocked by an object in the detection area. Summary of the invention

[0013] Against this background, the object of the present invention is to provide a more robust and more flexibly applicable possibility for process control of a work task on at least one object (workpiece) by means of at least one tool.

[0014] This object is achieved by a method having the features of claim 1 and by a device having the features of claim 10. The process comprises providing at least one sensor in a position recognition system for determining the position and orientation of an object, with which a scene with at least one object and at least one tool is optically detected over time and distance information is assigned to each image point as a 3D point cloud, wherein the object is recognized and the position and orientation of the object is determined based on the image points and the distance information.

[0015] Preferably, the sensor can be a LIDAR sensor or a TOF sensor. LIDAR stands for "Light Detection And Ranging" and is a distance measurement method similar to radar. LIDAR sensors usually emit laser pulses and detect backscattered light. TOF stands for Time Of Flight and represents a 3D camera system that measures distance using the transmission time method. To this end, the observed scene is illuminated by means of light pulses, and the camera measures the transmission time of light to the object and back for each image point. Compared with laser scanners that are often used with LIDAR sensors, TOF sensors have the advantage that the entire scene can be recorded at once. The illumination can be an LED or a laser diode in the visible or infrared wavelength range. This means that the system is easier to integrate into the production line than the systems known in the prior art, because only the sensor needs to be installed to generate a complete 3D point cloud of the scene with the object, in which the object can be clearly identified and its position determined. The time-consuming installation of distance readers and the attachment of reflectors and / or the positionally accurate attachment of markers can be eliminated.

[0016] Thus, objects can be identified without external specification of which object should be tracked. Identification can be done with the help of a few characteristic points recorded during the initial movement of the workpiece, without the need to acquire the exact position and orientation and without the need for the entire object to be visible. Position identification and identification of the object in the process control system can be used to activate the area control of the workpiece variant responsible for the identification. Otherwise, if the identified object does not need to be processed in this processing area or processing subarea, further processing of the data by the process control system can be aborted.

[0017] The sensor proposed by the invention makes it possible to define a movement channel or area in the processing area (and corresponding to the sensor field) in a 3D point cloud, wherein the data required for calculating at least one unique measurement value is filtered, which identifies the object and describes the actual position and orientation of the object (localization signal). Compared to the situation where only one measurement point is available, for example in a distance measuring laser according to the prior art, by detecting the 3D point cloud using the sensor according to the invention, a more robust localization signal can be generated based on many parallel measurement points. In addition, all image points of the 3D point cloud (with assigned distance information) can be evaluated in parallel, so that multiple objects in the processing area can be recognized simultaneously and taken into account by the process control.

[0018] According to a preferred embodiment, multiple (flat or curved) faces can be identified in the 3D point cloud and placed relative to each other in order to identify the object and determine its position and orientation over time. In other words, the objects in the image are identified by faces in the image, which faces are identified by the image evaluation and have a defined relationship to each other. By identifying the object in each image, the movement trajectory of the object can also be described over time (i.e. in images recorded at different times). The identification of faces can be limited to describing the movement channel of the object's actual processing area (area within the processing area).

[0019] According to the present invention, different objects can be automatically recognized and identified by associating different feature faces for description.

[0020] In a further development of this embodiment, the edge of the surface of the object can be identified by an edge, at which the orientation of the surface changes, and / or by a brightness change in the image. In this sense, an edge is a change in the surface, at which an abrupt (discontinuous) transition from a first orientation of the surface to a second orientation occurs. In an optical camera image, such edges usually appear as lines and / or abrupt contrast changes, which can be optically recognized in a two-dimensional image using conventional image recognition methods. Due to the additional distance information in the 3D point cloud, edges can also be recognized by large changes in the distance between image points in the vicinity of the image points (or in other words, by changes in the distance gradient between image points in the vicinity).

[0021] Image brightness changes between image points in the vicinity can occur at different orientations of the surface due to changes in the direction of light reflection. However, if the properties of the surface are different (type of surface material, different surface properties), significant brightness changes can occur even if the orientation of the surface does not change. This is due to the different reflective properties of different materials.

[0022] An important advantage of using a sensor according to the invention, in particular a LIDAR and / or TOF sensor, is that the optical information about the surface properties (brightness of the image point) is combined with the distance information (indicative of the distance of the object from the sensor in the image point). In this way, information about the position and orientation of the object can be detected directly from the image point data of the recognized object, without having to carry out complex calculations, such as triangulation methods, without having to know the construction data of the object, and / or without having to use a stereo camera system and calibrate it in a complex manner.

[0023] In addition, the sensors used according to the invention, in particular LIDAR and / or TOF sensors, themselves emit light for detection after reflection. This means that, in contrast to purely optical image recording using ambient light, the detection method of these sensors is independent of ambient light. Preferably, the spectrum of the light is known, so that wavelength-selective filtering can be performed during detection in order to minimize the noise caused by ambient light. According to one embodiment, if the light used has red, yellow and blue components, the color information of the surface can also be used when determining the face of the object. If the light used has a filterable infrared component, the distance information can be obtained independently of the light component in the visible wavelength range, whereby the background noise can be further reduced. Combinations of the above options are also possible.

[0024] In general, using the sensor according to the invention, the object can be characterized by many different features and can therefore be identified with a very high degree of accuracy. Its position and orientation can accordingly be described very precisely. This also enables the identification and orientation determination (position and orientation) of the object if parts of the object are obscured in the scene (e.g. by a worker guiding and operating a tool to perform a work task) and are not shown in the image. This makes the method according to the invention very robust compared to solutions known from the prior art.

[0025] Mathematically, in a simple embodiment, the identified faces of an object can be described by a flat plane with straight edges, wherein the orientation of the plane is described, for example, by a normal vector perpendicular to the plane. This makes it mathematically easy to form relationships between the identified faces of an object. In more complex solutions, faces can also be described by adjusting (fitting) geometric shapes, wherein the position and orientation of the geometric figures and the parameters fitted to describe the figures can be used to form the relationship between the faces. This also enables the description of non-flat faces and / or non-straight edges. In this respect, the invention is not limited to the previously described embodiments, even though these embodiments provide easy-to-implement and robust possibilities for identifying and describing objects.

[0026] In another preferred embodiment, the position and orientation of the object can be described by a vector which always acts on the same position of the object. The position can be determined automatically by image recognition (software) in a position recognition system or process control system.

[0027] According to the invention, the movement of an object relative to a position detection system can be determined based on the change in the position and / or orientation of the object over time (for example, by acquiring the direction of movement and / or the speed of movement). In process control, it can also be checked whether certain work tasks are performed in a specified process part or processing (sub) area, and if the object continues to move uniformly, whether all the work tasks to be completed can also be achieved in the process part. This can be used to flexibly control the movement of objects in the process line, so that it can be controlled or adjusted so that the specified work tasks are performed in the process part or processing (sub) area (using the length of the specified process part or processing (sub) area). This dynamically optimizes the use of available resources and is very robust because problems arising in the process can be dynamically considered. If the signal is lost, the known object movement information can also be used to predict possible future positions.

[0028] According to the invention, particularly simple operation and parameterization can be achieved if the determination of a 3D workspace defined on an object is accomplished by measuring the object using a method in which the object detects a 3D point cloud with the aid of at least one sensor of a position recognition system, a plurality of (planar or curved) faces are recognized and identified as faces of the object (e.g., by manual selection or by presetting an image area in which at least one object is displayed). The faces identified as faces of the object are placed relative to one another in order to unambiguously describe the object. During the measurement, one or more markers of the position recognition system are additionally attached to the object. The position and orientation of the markers on the object can also be detected by a camera of the position detection system, just like the markers on the tool. In order to identify the 3D workspace on the object, a tool with a marker attached to the tool and / or a position tracker with a marker attached to the position tracker are guided to one (or successively to each) working point to perform a work task on the object.

[0029] Then, the 3D point cloud and the image of the object's marker, the image of the tool's marker and / or the image of the position tracker's marker are detected synchronously in time by at least one sensor, wherein: (a) the position and orientation of the object are determined from the 3D point cloud; (b) the position and orientation of the tool's marker and / or the position tracker's marker are determined from the camera's image and (c) they can be associated with each other via the position and orientation of the object's marker. A 3D workspace around the object's workspace is thereby defined (wherein a tolerance range can be specified for the 3D workspace).

[0030] This correlation enables the 3D workspace to be preset according to the position and orientation of the object in the advanced coordinate system without explicitly determining the position and orientation of the object in space in the advanced coordinate system (by specifying specific coordinates). The position and orientation of the object recorded in the 3D point cloud and described, for example, by a vector on the object are transmitted to the advanced coordinate system by association through markers on the object during the measurement process, that is, the position and orientation of the markers attached to the position tracker and / or the tool are known relative to the position and orientation of the object. This allows the 3D working position to be described in the advanced coordinate system of the tracker. This correlation also leads to the optical determination of the position and orientation of the object in the advanced coordinate system of the position recognition system, but there is no need to explicitly describe the coordinates of the position and orientation of the object in the coordinate system. This represents a preferred embodiment of the present invention because the parameterization and teaching of the object are particularly easy. With this solution, it is not necessary to explicitly calibrate the coordinate system of the sensor to adapt to the advanced coordinate system describing the marker.

[0031] However, the invention is not limited to this particularly preferred solution. According to the invention, it is also possible to explicitly calibrate the coordinate system of the sensor and the higher-level coordinate system of the marker to one another and to describe the position and orientation of the object in this higher-level coordinate system explicitly (specifying coordinates). It is known to the person skilled in the art to carry out such a calibration. However, this involves considerable effort when setting up the system and usually has to be recalibrated regularly in operation, i.e. during the execution of the process control method.

[0032] Another advantage of the advantageous teaching method described in this embodiment is that the construction data of the object is not known or does not need to be known. The 3D workspace can be taught, which is also called teaching, by simply marking the work points using tools and / or position trackers (and markers attached thereto).

[0033] This means that once the position and orientation of the object in the high-level coordinate system is determined (preferably by the above-mentioned correlation), the 3D workspace relative to the object position and orientation is also known and the actual process control of the work task can be performed.

[0034] By determining the position and orientation of an object according to the invention, which is parameterized by simply recording the object by a sensor and teaching the 3D workspace, no further information about the object (3D construction data) is required or the movement trajectory of the object relative to a higher-level coordinate system or position detection system during the execution of the work task must be known. The system can be quickly configured in practice and can be used flexibly, for example when the position of the object is not always the same during the production process and when different objects are processed in the process line. After teaching one or each different object processed in the production line, the objects in the production line can be easily identified and their position and orientation can be easily and reliably determined and used for process control.

[0035] In a particularly preferred embodiment of the proposed method, during the measurement, the object is moved from a starting point to an end point of the processing region of the object and detected by the sensor on the processing region, wherein the position and orientation of the object are determined in each case. Preferably, the position and orientation of the object are stored together with the association of the identified surfaces, so that comparative data of the object are present for different positions of the object in the processing region. The data of at least one sensor are thus present in the form of a 3D point cloud covering every working position in the processing region of the object. This makes it easier to find the object and to identify its orientation when executing a working step during process control.

[0036] According to the invention, measuring the movement trajectory of the object also makes it possible to identify deviations in the position and orientation of the object in operation, i.e. when the method is carried out after the measurement. According to the invention, this can be used to check whether a preset movement sequence of the object is adhered to during process control, for example by presetting tolerances for the position and orientation during the measurement. Since, according to the invention, the 3D workspace on the object for performing the work task by the tool is related to the position and orientation of the object during read-in (i.e., in other words, they are determined in the object coordinate system), the current orientation and position are automatically adapted to the data of the 3D point cloud. The 3D workspace on the object is therefore known in a high-level coordinate system relative to the object and the position and orientation of the tool are automatically taken into account by the process control.

[0037] In this embodiment, when the object is measured by the motion trajectory from the starting point to the end point, the starting point and the end point of the object in the processing area can be determined accordingly. According to the invention, when the method is carried out (after the measurement), these starting points and end points of the object can be specifically monitored (special motion channels), that is, each starting point and each end point of the object can be checked in each image to see whether an object related to this position as the starting point or end point can be identified at this position. At the starting point, the process control of the object is activated, and at the end point, the process control of the object is deactivated again.

[0038] According to a particularly preferred embodiment, a scene with at least one object and at least one tool is recorded from different directions by multiple sensors (and possibly also cameras). The data from different sensors can be merged, since partially redundant data can be present in the 3D point cloud and can be correlated. This allows a more robust identification of the orientation of the object even if parts of the object are covered by the sensor field of one or more sensors. When the body of a worker (who guides and operates the tool to perform the work task) is located between the sensor and the object, the area of ​​the object is usually covered. Having redundant data makes it possible to identify and correlate the individual faces on the object with each other even if the entire identified face is not fully detected by a single sensor.

[0039] In other words, 3D point clouds from multiple sensors can be fused in order to expand the motion space and, for example, detect the entire processing area from at least one viewing direction, preferably from multiple viewing directions. The 3D point clouds can be registered with each other and map the desired motion space. If an object is partially visible in multiple sensors, this also makes it possible to locate the object more accurately than using only one sensor.

[0040] In principle, each sensor (especially when teaching) can at least initially form its own coordinate system with the detected 3D point cloud, so that there are multiple different coordinate systems. These can be concentrated by the position identification system or the process control system into exactly one higher-level coordinate system of the position identification and process control system, preferably by means of the above-mentioned correlation.

[0041] However, the process control system may also basically work with additional coordinate systems of various sensors and select one of the coordinate systems (or the corresponding sensor) to perform a work task during the process control.

[0042] Accordingly, multiple cameras may be provided for position recognition of the tool and recording of the scene. According to the invention, the recording of the camera and the sensor is preferably performed synchronously with each other, so that the position of the object and the position of the tool are known at the same time.

[0043] When executing the process control method according to the invention, multiple objects and / or tools can be located in the working area at the same time, and multiple work tasks can be performed in parallel independently of each other. In particular, different types of objects (provided that each object has been taught or taught once as described) can also be detected in parallel at the same time and processed by the process control. Objects and tools detected simultaneously in the scene and processed by the process control can each perform their own movement in different directions and / or speeds. Therefore, the distances between different objects in the detected motion space can change and be different at different measurement time points.

[0044] This is a major advantage of monitoring a scene using a camera, which can acquire the orientation and position of a marker (in a known manner) based on the image of a specially designed marker (as described above), in conjunction with a sensor that produces a 3D point cloud of image points and distance information for the entire sensor field. This enables data from multiple objects and tools to be evaluated quickly and efficiently, even at the same time.

[0045] The moving object can be an object that moves relative to a stationary work environment. It is also possible that the work environment (tools and workers) passes by a stationary object or an object that moves with another movement. For example, when using a track system, the tool (such as a screwdriver with a torque arm) moves along the track. This approach is often used when tools are used in parallel on different vehicles in continuous production. The belt then moves with the tool and the worker can carry out subsequent processing in the processing area (e.g. several meters long) along the line.

[0046] The position detection system with camera and sensor can be arranged in a fixed system or a moving system, and preferably detects at least one object moving relative to the position detection system and / or a working environment moving relative to the position detection system. In a typical application, the position detection system and the working environment are located in a common system (i.e., fixed relative to each other), and the object system moves relative to it.

[0047] In principle, the method can also be used if both the object and the position detection and process control system move or can move at different speeds and / or with different movement paths compared to a spatially fixed system.

[0048] The evaluation of the 3D point cloud and the images from the camera (hereinafter also referred to as "data") which is particularly preferred according to the invention can apply AI methods (artificial intelligence methods) for the recognition of surfaces, for the allocation of 3D workspaces and / or for linking the position and orientation of objects with the 3D point cloud and the markers of the objects, and for accessing the 3D point cloud recorded over time by at least one sensor. This can include in particular data recorded during teaching and data detected over time during operation. This enables a continuous updating of the position detection system and the process control system, thereby adapting to changing workflows, etc. According to the invention, in addition to objects, the position of workers when performing work tasks can also be evaluated, for example as part of quality control and workflow evaluation.

[0049] The invention also relates to a device for process control of a work task on at least one object with the aid of at least one tool, having the features of claim 10. The device has a position recognition system for acquiring the position and orientation of the tool, comprising at least one marker that can be attached (or attached) to the tool and a camera configured to detect an image of the marker, and for acquiring the position and orientation of the object, comprising at least one sensor. In addition, a process control system for releasing, parameterizing and / or detecting the execution of a work task for an object is provided. The computing unit of the position detection and process control system is equipped with at least one correspondingly configured processor (e.g., a common processor for the position detection and process control system) to control the position detection and process control system (including its components, such as in particular cameras and sensors). At least one sensor is configured to detect a scene with at least one object and at least one tool as a 3D point cloud with image points, wherein each image point is assigned distance information. At least one processor is configured to perform the method according to any one of claims 1 to 9 or parts thereof.

[0050] The position detection and process control system are considered and described as one logical unit, regardless of whether they are actually implemented in one physical computing unit or in multiple physical computing units, which exchange data as a communication network of processors and perform individual method steps in this and / or other computing units. A person skilled in the art can easily divide certain method steps into different physical computing units or implement the position detection and process control system in one computing unit, each computing unit having one or more processors.

[0051] According to a preferred embodiment, at least one sensor is a LIDAR sensor or a TOF sensor, wherein in the device according to the invention having a plurality of sensors, some sensors may also be designed as LIDAR sensors and another part of the sensors may be designed as TOF sensors.

[0052] Further advantages, features and application possibilities of the invention also emerge from the following description of the embodiments and the drawings. All features described and / or illustrated therein together or in any combination that makes sense from a technical point of view belong to the subject matter of the invention, regardless of how they are summarized in the described or illustrated embodiments or in the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] The accompanying drawings show:

[0054] Figure 1 a schematic diagram showing an embodiment of the device according to the present invention; and

[0055] Figure 2 Show according to Figure 1Schematic diagram of a portion of a device when measuring an object. DETAILED DESCRIPTION

[0056] exist Figure 1 , an embodiment of a device 1 according to the invention is schematically shown, simplified for the sake of clarity, for process control of a work task on an object 2, also called a workpiece. During processing, the object 2 in the example shown here moves on a movement path 3 through a processing area from a starting point S to an end point E. The movement speed is variable. The work task is performed with the aid of a tool 4, which is guided and operated, for example, by a worker.

[0057] The workpiece 2 can also stop during the movement. The system according to the invention is able to detect this without reinitializing the process control. For this purpose, the device 1 for detecting the processing area has a position recognition system 5 and, in particular, a downstream process control system 6, which are shown here as a common computing unit with a processor, which is set up for controlling the device and executing the method according to the invention.

[0058] In the example shown here, two cameras 7 are connected to the position recognition system 5, which detect the entire scene from different viewing directions. In practical applications, more cameras 7 are usually provided to record images of the entire scene from different viewing directions in a redundant manner. Based on the images from the cameras 7, the position recognition system determines the position and orientation of the tool 4 in the high-level coordinate system over time, i.e. during the movement of the workpiece 2 along the movement trajectory 3. To this end, at least one camera 7 detects an image of the tool 4 to which a marker 8 is attached. As explained at the outset and known to the person skilled in the art from the prior art, the position recognition system 5 is configured to determine the position and orientation of the marker 8 of the tool 4 based on the images from the cameras 7.

[0059] Furthermore, the position recognition system 5 is also connected to sensors 9, in the example shown here two sensors 9, which detect the entire scene from different viewing directions. In practical applications, more sensors are usually provided to detect the entire scene from different viewing directions in a redundant manner. The sensor 9 is a LIDAR sensor or a TOF sensor or a similar sensor 9, which is able to optically detect a scene with an object 2 and at least one tool 4 over time and to assign distance information to each image point as a 3D point cloud, wherein the object 2 is recognized and the position and orientation of the object is determined based on the image point and the distance information.

[0060] By comparison (for example in the position recognition system 5 or the process control system 6), it is determined over time when the tool 4 is located in one of the 3D working areas 10 on the object 2 (precisely for this tool 4). In this case, the working position of the tool 4 is signaled. Figure 2, it will be described later how to confirm the 3D working position on the workpiece 2 in a high-level coordinate system, in which the position and orientation of the tool are also described. The high-level coordinate system of the position recognition system 5 is used to accurately locate the tool 4 and the person, the body part of the person (such as the hand, arm, head) or other part to which the marker 8 is fixed in position and orientation. The position recognition system 5 is able to recognize the movement of one or more tools 4 (or markers 8) in space in six degrees of freedom (position and orientation) very accurately.

[0061] When the working position of the tool 4 is signaled, the execution of the work task on the object 2 in the 3D workspace 10 is released, parameterized and / or detected by means of the process control system 6. The worker can then perform the work task on the workpiece 2 in the 3D workspace 10 using the tool 4, wherein the parameters of the tool can also be set or the execution of the work step can be detected for quality assurance. This is an example of the subject of process control.

[0062] In a specific example, the tool 4 can be a screwdriver or a drill. The work on the workpiece 2 (object) should be carried out in a 3D workspace 10 defined on the workpiece 2. The 3D workspace 10 can be very small; for example, a spherical area with a radius of 1 cm should be monitored. Even if the entire processing area may be several meters long, the smallest spatial area over the entire movement trajectory of the workpiece 2 can be clearly identified. In the automotive industry, the distance over which the workpiece 2 (vehicle) is processed is usually 7 meters.

[0063] The screws are usually attached to invisible locations on the workpiece 2 and can only be reached by extending the screwdriver bit (part of the tool 4) because these screws are recessed in the hole. The position and orientation of one or more handheld screwdrivers (tools 4) in space are clearly identified by the position identification system 5. For example, the position identification system 5 calculates the position of the screwdriver bit in space through the vector to the screwdriver tip (part of the tool 4), and sends a signal back to the process control system 6 about whether the position of the screwdriver (tool 4) bit has reached the 3D work area 10 on the workpiece 2. The process control can use this information to individually parameterize the screwdriver (tool 4) for subsequent tightening processes in this area and allow the tightening process to be carried out in this position. The previously locked screwdriver is now released and the worker can tighten the screw exclusively in this position.

[0064] When the method is carried out, the workpiece 2 and the carrier of the workpiece 2 (not shown) are not equipped with a marker 8. This is advantageous because the process of retrofitting the marker 4 to each workpiece is both cost-intensive and labor-intensive. The marker 4 must be precisely manufactured and classifiable. In addition, the exact fastening point must be found on the workpiece or object 2 and attached to the object 2 by means of a calibration process, or each incoming workpiece is inserted into a precise position by a worker. The 3D construction data of the workpiece 2 are also not necessary and correspondingly are not known in typical applications at least to the position recognition system 6 and the process control system 6.

[0065] By means of the proposed system (device and method), a 3D workspace 10 on one or more objects 2 (workpieces) can be precisely located at any time along the trajectory of the object 2. The 3D workspace 10 can be, for example, an assembly area for screwing, drilling, reaming, rolling, gluing, lubricating, snapping, pressing, which does not move synchronously with the tool 4. The movements of one or more tools 4 and one or more workers in different positions are clearly identified and assigned to each other.

[0066] Figure 2 Shown according to Figure 1 The device 1 is part of a device 1 which has a position detection system 5, a camera 7 and a sensor 9 as well as a process control system 6.

[0067] The (still unknown) object 2 is arranged in the detection area of ​​the position recognition system 5 and is to be taught into the system by measurement (teaching). For this purpose, the object 2 is detected as a 3D point cloud by means of the sensor 9 of the position recognition system 5. If multiple sensors 9 detect the object 2, the multiple 3D point clouds can be registered with each other, so that a common 3D point cloud (in the sense of a common 3D space) is created.

[0068] In an image region where an object 2 is identified (e.g. manually or automatically with the aid of image recognition software) and displayed, a plurality of (flat or curved) faces 11, 12, 13 are identified (shown in the figures with different textures) and identified as faces 11, 12, 13 of the object. The faces 11, 12, 13 identified as faces 11, 12, 13 of the object 2 are placed relative to each other in order to describe the object 2. The description of the object 2 can be expressed, for example, as a position and orientation vector V of the object 2, which is derived from the relationship of the identifiable faces 11, 12, 13.

[0069] Specifically, in the example shown, the following steps can be performed. Filtered 3D data channels from a 3D point cloud, which form or describe surfaces 11, 12, 13 in space, are converted into edges. Edges are described by straight lines in 3D space with a start point and an end point and are related to each other. The sum of the calculated orientations of the surfaces 11, 12, 13 in space generates a feature value, which can be expressed as a position and an orientation vector V of the object 2 in space and acts on a specific position of the object 2 (preferably determined by the program for evaluation), for example, on the intersection of certain identified surfaces 11, 12, 13. Other methods for determining feature values ​​are also possible, as described at the beginning.

[0070] According to the invention, the change of the characteristic value (for example the vector V) can be used in all described embodiments of the invention to determine the speed and direction of movement of the object 2. If an interpolation of the movement along the detected data signal is assumed in the event of a signal loss, a linear increment value can be defined, which can also be used according to the invention in all described embodiments of the invention.

[0071] In addition, at least one marker 8 of the position identification system 5 is attached to the object 2. The marker or markers 8 define a coordinate system on the object 2 (workpiece) which corresponds to the high-level coordinate system.

[0072] In addition, a tool 4 having a marker 8 and / or (such as Figure 2 In the embodiment shown in FIG. 1 , a position tracker 14 with a marker 8 is guided to a work point of an object 2 for a work task. Then, a 3D point cloud is detected by the sensor 2 and images of the marker 8 of the object 2 and the tool 4 and / or the position tracker 14 are detected by the camera 7 in a time-synchronous manner, wherein the position and orientation of the object 2 (e.g., as described by the position and orientation vector V) and the position and orientation of the markers of the tool 4 and / or the position tracker 14 are related to each other via the position and orientation of the marker 8 on the object 2 to define one or more 3D workspaces 10 of the object 2. Therefore, the position of the 3D workspace 10 of the object 2 is known in the high-level coordinate system without quantitatively knowing the position and orientation vector V of the object 2 in the high-level coordinate system (or more generally its position and orientation).

[0073] When teaching is applied in the described manner, each product variant must be taught once. Furthermore, it is advantageous to move the workpiece 2 along the motion trajectory 3 through the processing area during the measurement in order to facilitate and accelerate the identification of the workpiece 2 when the process control method is applied. The speed of the movement is not important here. For example, the recording process can also be stopped or interrupted. Multiple workpieces 2 can also be moved in parallel in the processing area. The data of the individual workpieces 2 can be extracted and used for training (for example using AI methods) in order to obtain more robust results through redundant detection. In this way, multiple product variants can also be taught simultaneously, additionally or alternatively.

[0074] The device 1 described here by way of example may be configured to carry out all method steps described in this application and may additionally have further components which are not shown in the figures for the sake of clarity.

[0075] Reference numerals list

[0076] 1Equipment for process control of work tasks on objects

[0077] 2 Object (workpiece)

[0078] 3 Movement trajectory

[0079] 4 Tools

[0080] 5 Location Identification System

[0081] 6 Process Control Systems

[0082] 7 Camera

[0083] 8 Markers with labels

[0084] 9 Sensors

[0085] 10 3D Workspace

[0086] 11. Surface of an object

[0087] 12 Object faces

[0088] 13 Object faces

[0089] 14 Location Tracker

[0090] S is the starting point of the object on its trajectory

[0091] E The end point of the object on its trajectory

[0092] V The position and orientation vector of the object

Claims

1. A method for process control of a work task on at least one object (2) by means of at least one tool (4), wherein the work task is performed in a 3D workspace (10) defined on the object (2), the method comprising: optically determining over time the position and orientation of the tool (4) in a higher-order coordinate system of the position identification system (5) by means of a position identification system (5), for which purpose at least one marker (8) of the position identification system (5) attached to the tool is detected by means of an image by means of a camera (7); optically determining the position and orientation of the object (2) over time by means of the position recognition system (5); Determining, in each case, the defined 3D workspace (10) on the object (2) in the high-level coordinate system, based on the determined position and orientation of the object (2); by comparing over time to identify when the tool (4) is located in one of the 3D working areas (10) on the object (2), wherein in this case the working position of the tool (4) is indicated by a signal; and When the working position of the tool (4) is indicated by a signal, the execution of the working task of the object (2) is released, parameterized and / or detected in the 3D working area (10) by means of a process control system (6); It is characterized in that At least one sensor (9), in particular a LIDAR sensor or a TOF sensor, is provided in the position recognition system (5) for determining the position and orientation of the object (2), with which a scene with the at least one object (2) and the at least one tool (4) is optically detected over time and distance information is assigned to each image point as a 3D point cloud, wherein the object (2) is recognized and the position and orientation of the object (2) is determined based on the image points and the distance information.

2. The method according to claim 1, characterized in that A plurality of faces (11, 12, 13) are identified in the 3D point cloud and placed relative to each other in order to identify the object (2) and determine the position and orientation of the object (2) over time.

3. The method according to claim 2, characterized in that The edges of the surfaces (11, 12, 13) of the object (2) are detected by edges at which the orientation of the surfaces (11, 12, 13) changes and / or by brightness changes in the image.

4. The method according to claim 2 or 3, characterized in that: The movement of the object (2) relative to the position detection system (5) is determined based on changes in the position and / or orientation of the object (2) over time.

5. The method according to any one of the preceding claims, characterized in that Determination of the 3D working area (10) defined on the object (2) is accomplished by measuring the object (2), wherein The object (2) detects a 3D point cloud by means of the at least one sensor (9) of the position recognition system (5), identifies a plurality of faces (11, 12, 13) and identifies the plurality of faces as faces (11, 12, 13) of the object (2); placing the faces (11, 12, 13) identified as faces (11, 12, 13) of the object (2) relative to one another so as to describe the object (2); attaching at least one marker (8) of the position identification system (5) to the object (2), and The tool (4) with the marker (8) and / or the position tracker (14) with the marker (8) are guided to a work point for a work task, and the 3D point cloud and the image of the marker (8) of the object (2), the image of the marker (8) of the tool (4) and / or the position tracker (14) are detected in a time-synchronous manner, wherein The position and orientation of the object (2) and the position and orientation of the marker (8) of the tool (4) and / or the position tracker (14) are associated with each other via the position and orientation of the marker (8) of the object (2) to define the 3D workspace (10) of the object (2).

6. The method according to claim 5, characterized in that The object (2) is moved during the measurement process from a starting point (S) to an end point (E) of a processing area of ​​the object (2), and the object is detected via the processing area by means of the sensor (9) and the position and orientation are determined in each case.

7. The method according to any one of the preceding claims, characterized in that A scene having at least one object (2) and at least one tool (4) is recorded from different directions by a plurality of sensors (9).

8. The method according to any one of the preceding claims, characterized in that During process control, a plurality of objects (2) and / or tools (4) are present simultaneously in the processing area and a plurality of work tasks are performed independently of one another and in parallel.

9. The method according to any one of the preceding claims, characterized in that An AI method is used to evaluate the 3D point cloud and the image of the camera (7) for identifying the surfaces (11, 12, 13), for allocating the 3D workspace (1) and / or linking the position and orientation of the object (2) with the 3D point cloud and markers of the object (2), and for accessing the 3D point cloud recorded over time by the at least one sensor (9).

10. A device for process control of a work task on at least one object (2) by means of at least one tool (4), comprising: a position identification system (5) for acquiring the position and orientation of the tool (4), the position identification system comprising at least one marker (8) which can be attached to the tool (4), a camera (7) and at least one sensor (9), the marker (8) being attachable to the tool (4), the camera (7) being configured to detect an image of the marker and to acquire the position and orientation of the object (2); a process control system (6) for releasing, parameterizing and / or monitoring the execution of a work task of the object (2); A computing unit of a position identification system (5) and a process control system (6), configured to control the position identification system (5) and the process control system (6) using at least one processor; It is characterized in that The at least one sensor (9) is configured to detect a scene with the at least one object (2) and the at least one tool (4) as a 3D point cloud with image points, wherein each image point is assigned a distance information, and The at least one processor is configured to perform the method according to any one of claims 1 to 9.

11. The device according to claim 10, characterized in that The at least one sensor (9) is a LIDAR sensor or a TOF sensor.

Images